{"thread":{"id":"39497","subject":"[PATCH v4 0/4] submodule config lookup API","startedAt":"2015-06-02T14:24:36Z","lastAt":"2015-06-03T19:31:26Z","messageCount":7,"participants":["Heiko Voigt","Junio C Hamano"],"isPatch":true,"patchVersion":4,"patchTotal":4},"messages":[{"id":"262726","messageId":"20150602142436.GA7713@book.hvoigt.net","threadId":"39497","inReplyTo":null,"subject":"[PATCH v4 0/4] submodule config lookup API","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2015-06-02T14:24:36Z","receivedAt":"2015-06-02T14:24:36Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"Since there were no other comments than style fixes here is another\niteration. The last iteration can be found here:\n\nhttp://article.gmane.org/gmane.comp.version-control.git/269611\n\nI found there is a merge conflict with master so I rebased this series\nto master.\n\nThe interdiff to last iteration contains nothing spectacular:\n\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 177767d..199692b 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -27,7 +27,7 @@ struct submodule_entry {\n \n enum lookup_type {\n \tlookup_name,\n-\tlookup_path,\n+\tlookup_path\n };\n \n static struct submodule_cache cache;\n@@ -390,12 +390,12 @@ static const struct submodule *config_from(struct submodule_cache *cache,\n \t\treturn submodule;\n \n \tswitch (lookup_type) {\n-\t\tcase lookup_name:\n-\t\t\tsubmodule = cache_lookup_name(cache, sha1, key);\n-\t\t\tbreak;\n-\t\tcase lookup_path:\n-\t\t\tsubmodule = cache_lookup_path(cache, sha1, key);\n-\t\t\tbreak;\n+\tcase lookup_name:\n+\t\tsubmodule = cache_lookup_name(cache, sha1, key);\n+\t\tbreak;\n+\tcase lookup_path:\n+\t\tsubmodule = cache_lookup_path(cache, sha1, key);\n+\t\tbreak;\n \t}\n \tif (submodule)\n \t\treturn submodule;\n@@ -419,12 +419,12 @@ static const struct submodule *config_from(struct submodule_cache *cache,\n \tfree(config);\n \n \tswitch (lookup_type) {\n-\t\tcase lookup_name:\n-\t\t\tsubmodule = cache_lookup_name(cache, sha1, key);\n-\t\t\tbreak;\n-\t\tcase lookup_path:\n-\t\t\tsubmodule = cache_lookup_path(cache, sha1, key);\n-\t\t\tbreak;\n+\tcase lookup_name:\n+\t\tsubmodule = cache_lookup_name(cache, sha1, key);\n+\t\tbreak;\n+\tcase lookup_path:\n+\t\tsubmodule = cache_lookup_path(cache, sha1, key);\n+\t\tbreak;\n \t}\n \n \treturn submodule;\ndiff --git a/submodule-config.h b/submodule-config.h\nindex 58afc83..9061e4e 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -24,6 +24,6 @@ const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\n const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n \t\tconst char *path);\n-void submodule_free();\n+void submodule_free(void);\n \n #endif /* SUBMODULE_CONFIG_H */\n\n\nHeiko Voigt (4):\n implement submodule config cache for lookup of submodule names\n extract functions for submodule config set and lookup\n use new config API for worktree configurations of submodules\n do not die on error of parsing fetchrecursesubmodules option\n\n .gitignore | 1 +\n Documentation/technical/api-submodule-config.txt | 63 +++\n Makefile | 2 +\n builtin/checkout.c | 1 +\n builtin/fetch.c | 1 +\n diff.c | 1 +\n submodule-config.c | 484 +++++++++++++++++++++++\n submodule-config.h | 29 ++\n submodule.c | 122 ++----\n submodule.h | 4 +-\n t/t7411-submodule-config.sh | 153 +++++++\n test-submodule-config.c | 76 ++++\n 12 files changed, 839 insertions(+), 98 deletions(-)\n create mode 100644 Documentation/technical/api-submodule-config.txt\n create mode 100644 submodule-config.c\n create mode 100644 submodule-config.h\n create mode 100755 t/t7411-submodule-config.sh\n create mode 100644 test-submodule-config.c\n\n-- \n2.4.2.391.g2979c89\n"},{"id":"262732","messageId":"20150602142538.GB7713@book.hvoigt.net","threadId":"39497","inReplyTo":"20150602142436.GA7713@book.hvoigt.net","subject":"[PATCH v4 1/4] implement submodule config cache for lookup of submodule names","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2015-06-02T14:25:39Z","receivedAt":"2015-06-02T14:25:39Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"This submodule configuration cache allows us to lazily read .gitmodules\nconfigurations by commit into a runtime cache which can then be used to\neasily lookup values from it. Currently only the values for path or name\nare stored but it can be extended for any value needed.\n\nIt is expected that .gitmodules files do not change often between\ncommits. Thats why we lookup the .gitmodules sha1 from a commit and then\neither lookup an already parsed configuration or parse and cache an\nunknown one for each sha1. The cache is lazily build on demand for each\nrequested commit.\n\nThis cache can be used for all purposes which need knowledge about\nsubmodule configurations. Example use cases are:\n\n * Recursive submodule checkout needs lookup a submodule name from its\n path when a submodule first appears. This needs be done before this\n configuration exists in the worktree.\n\n * The implementation of submodule support for 'git archive' needs to\n lookup the submodule name to generate the archive when given a\n revision that is not checked out.\n\n * 'git fetch' when given the --recurse-submodules=on-demand option (or\n configuration) needs to lookup submodule names by path from the\n database rather than reading from the worktree. For new submodule it\n needs to lookup the name from its path to allow cloning new\n submodules into the .git folder so they can be checked out without\n any network interaction when the user does a checkout of that\n revision.\n\nSigned-off-by: Heiko Voigt \n---\n .gitignore | 1 +\n Documentation/technical/api-submodule-config.txt | 46 +++\n Makefile | 2 +\n submodule-config.c | 445 +++++++++++++++++++++++\n submodule-config.h | 27 ++\n submodule.c | 1 +\n submodule.h | 1 +\n t/t7411-submodule-config.sh | 85 +++++\n test-submodule-config.c | 66 ++++\n 9 files changed, 674 insertions(+)\n create mode 100644 Documentation/technical/api-submodule-config.txt\n create mode 100644 submodule-config.c\n create mode 100644 submodule-config.h\n create mode 100755 t/t7411-submodule-config.sh\n create mode 100644 test-submodule-config.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 422c538..337c121 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -204,6 +204,7 @@\n /test-sha1-array\n /test-sigchain\n /test-string-list\n+/test-submodule-config\n /test-subprocess\n /test-svn-fe\n /test-urlmatch-normalization\ndiff --git a/Documentation/technical/api-submodule-config.txt b/Documentation/technical/api-submodule-config.txt\nnew file mode 100644\nindex 0000000..2ff4907\n--- /dev/null\n+++ b/Documentation/technical/api-submodule-config.txt\n@@ -0,0 +1,46 @@\n+submodule config cache API\n+==========================\n+\n+The submodule config cache API allows to read submodule\n+configurations/information from specified revisions. Internally\n+information is lazily read into a cache that is used to avoid\n+unnecessary parsing of the same .gitmodule files. Lookups can be done by\n+submodule path or name.\n+\n+Usage\n+-----\n+\n+The caller can look up information about submodules by using the\n+`submodule_from_path()` or `submodule_from_name()` functions. They return\n+a `struct submodule` which contains the values. The API automatically\n+initializes and allocates the needed infrastructure on-demand.\n+\n+If the internal cache might grow too big or when the caller is done with\n+the API, all internally cached values can be freed with submodule_free().\n+\n+Data Structures\n+---------------\n+\n+`struct submodule`::\n+\n+\tThis structure is used to return the information about one\n+\tsubmodule for a certain revision. It is returned by the lookup\n+\tfunctions.\n+\n+Functions\n+---------\n+\n+`void submodule_free()`::\n+\n+\tUse these to free the internally cached values.\n+\n+`const struct submodule *submodule_from_path(const unsigned char *commit_sha1, const char *path)`::\n+\n+\tLookup values for one submodule by its commit_sha1 and path or\n+\tname.\n+\n+`const struct submodule *submodule_from_name(const unsigned char *commit_sha1, const char *name)`::\n+\n+\tThe same as above but lookup by name.\n+\n+For an example usage see test-submodule-config.c.\ndiff --git a/Makefile b/Makefile\nindex 54ec511..5d9a63f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -594,6 +594,7 @@ TEST_PROGRAMS_NEED_X += test-sha1\n TEST_PROGRAMS_NEED_X += test-sha1-array\n TEST_PROGRAMS_NEED_X += test-sigchain\n TEST_PROGRAMS_NEED_X += test-string-list\n+TEST_PROGRAMS_NEED_X += test-submodule-config\n TEST_PROGRAMS_NEED_X += test-subprocess\n TEST_PROGRAMS_NEED_X += test-svn-fe\n TEST_PROGRAMS_NEED_X += test-urlmatch-normalization\n@@ -784,6 +785,7 @@ LIB_OBJS += strbuf.o\n LIB_OBJS += streaming.o\n LIB_OBJS += string-list.o\n LIB_OBJS += submodule.o\n+LIB_OBJS += submodule-config.o\n LIB_OBJS += symlinks.o\n LIB_OBJS += tag.o\n LIB_OBJS += trace.o\ndiff --git a/submodule-config.c b/submodule-config.c\nnew file mode 100644\nindex 0000000..97f4a04\n--- /dev/null\n+++ b/submodule-config.c\n@@ -0,0 +1,445 @@\n+#include \"cache.h\"\n+#include \"submodule-config.h\"\n+#include \"submodule.h\"\n+#include \"strbuf.h\"\n+\n+/*\n+ * submodule cache lookup structure\n+ * There is one shared set of 'struct submodule' entries which can be\n+ * looked up by their sha1 blob id of the .gitmodule file and either\n+ * using path or name as key.\n+ * for_path stores submodule entries with path as key\n+ * for_name stores submodule entries with name as key\n+ */\n+struct submodule_cache {\n+\tstruct hashmap for_path;\n+\tstruct hashmap for_name;\n+};\n+\n+/*\n+ * thin wrapper struct needed to insert 'struct submodule' entries to\n+ * the hashmap\n+ */\n+struct submodule_entry {\n+\tstruct hashmap_entry ent;\n+\tstruct submodule *config;\n+};\n+\n+enum lookup_type {\n+\tlookup_name,\n+\tlookup_path\n+};\n+\n+static struct submodule_cache cache;\n+static int is_cache_init;\n+\n+static int config_path_cmp(const struct submodule_entry *a,\n+\t\t\t const struct submodule_entry *b,\n+\t\t\t const void *unused)\n+{\n+\treturn strcmp(a->config->path, b->config->path) ||\n+\t hashcmp(a->config->gitmodules_sha1, b->config->gitmodules_sha1);\n+}\n+\n+static int config_name_cmp(const struct submodule_entry *a,\n+\t\t\t const struct submodule_entry *b,\n+\t\t\t const void *unused)\n+{\n+\treturn strcmp(a->config->name, b->config->name) ||\n+\t hashcmp(a->config->gitmodules_sha1, b->config->gitmodules_sha1);\n+}\n+\n+static void cache_init(struct submodule_cache *cache)\n+{\n+\thashmap_init(&cache->for_path, (hashmap_cmp_fn) config_path_cmp, 0);\n+\thashmap_init(&cache->for_name, (hashmap_cmp_fn) config_name_cmp, 0);\n+}\n+\n+static void free_one_config(struct submodule_entry *entry)\n+{\n+\tfree((void *) entry->config->path);\n+\tfree((void *) entry->config->name);\n+\tfree(entry->config);\n+}\n+\n+static void cache_free(struct submodule_cache *cache)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct submodule_entry *entry;\n+\n+\t/*\n+\t * We iterate over the name hash here to be symmetric with the\n+\t * allocation of struct submodule entries. Each is allocated by\n+\t * their .gitmodule blob sha1 and submodule name.\n+\t */\n+\thashmap_iter_init(&cache->for_name, &iter);\n+\twhile ((entry = hashmap_iter_next(&iter)))\n+\t\tfree_one_config(entry);\n+\n+\thashmap_free(&cache->for_path, 1);\n+\thashmap_free(&cache->for_name, 1);\n+}\n+\n+static unsigned int hash_sha1_string(const unsigned char *sha1,\n+\t\t\t\t const char *string)\n+{\n+\treturn memhash(sha1, 20) + strhash(string);\n+}\n+\n+static void cache_put_path(struct submodule_cache *cache,\n+\t\t\t struct submodule *submodule)\n+{\n+\tunsigned int hash = hash_sha1_string(submodule->gitmodules_sha1,\n+\t\t\t\t\t submodule->path);\n+\tstruct submodule_entry *e = xmalloc(sizeof(*e));\n+\thashmap_entry_init(e, hash);\n+\te->config = submodule;\n+\thashmap_put(&cache->for_path, e);\n+}\n+\n+static void cache_remove_path(struct submodule_cache *cache,\n+\t\t\t struct submodule *submodule)\n+{\n+\tunsigned int hash = hash_sha1_string(submodule->gitmodules_sha1,\n+\t\t\t\t\t submodule->path);\n+\tstruct submodule_entry e;\n+\tstruct submodule_entry *removed;\n+\thashmap_entry_init(&e, hash);\n+\te.config = submodule;\n+\tremoved = hashmap_remove(&cache->for_path, &e, NULL);\n+\tfree(removed);\n+}\n+\n+static void cache_add(struct submodule_cache *cache,\n+\t\t struct submodule *submodule)\n+{\n+\tunsigned int hash = hash_sha1_string(submodule->gitmodules_sha1,\n+\t\t\t\t\t submodule->name);\n+\tstruct submodule_entry *e = xmalloc(sizeof(*e));\n+\thashmap_entry_init(e, hash);\n+\te->config = submodule;\n+\thashmap_add(&cache->for_name, e);\n+}\n+\n+static const struct submodule *cache_lookup_path(struct submodule_cache *cache,\n+\t\tconst unsigned char *gitmodules_sha1, const char *path)\n+{\n+\tstruct submodule_entry *entry;\n+\tunsigned int hash = hash_sha1_string(gitmodules_sha1, path);\n+\tstruct submodule_entry key;\n+\tstruct submodule key_config;\n+\n+\thashcpy(key_config.gitmodules_sha1, gitmodules_sha1);\n+\tkey_config.path = path;\n+\n+\thashmap_entry_init(&key, hash);\n+\tkey.config = &key_config;\n+\n+\tentry = hashmap_get(&cache->for_path, &key, NULL);\n+\tif (entry)\n+\t\treturn entry->config;\n+\treturn NULL;\n+}\n+\n+static struct submodule *cache_lookup_name(struct submodule_cache *cache,\n+\t\tconst unsigned char *gitmodules_sha1, const char *name)\n+{\n+\tstruct submodule_entry *entry;\n+\tunsigned int hash = hash_sha1_string(gitmodules_sha1, name);\n+\tstruct submodule_entry key;\n+\tstruct submodule key_config;\n+\n+\thashcpy(key_config.gitmodules_sha1, gitmodules_sha1);\n+\tkey_config.name = name;\n+\n+\thashmap_entry_init(&key, hash);\n+\tkey.config = &key_config;\n+\n+\tentry = hashmap_get(&cache->for_name, &key, NULL);\n+\tif (entry)\n+\t\treturn entry->config;\n+\treturn NULL;\n+}\n+\n+static int name_and_item_from_var(const char *var, struct strbuf *name,\n+\t\t\t\t struct strbuf *item)\n+{\n+\tconst char *subsection, *key;\n+\tint subsection_len, parse;\n+\tparse = parse_config_key(var, \"submodule\", &subsection,\n+\t\t\t&subsection_len, &key);\n+\tif (parse < 0 || !subsection)\n+\t\treturn 0;\n+\n+\tstrbuf_add(name, subsection, subsection_len);\n+\tstrbuf_addstr(item, key);\n+\n+\treturn 1;\n+}\n+\n+static struct submodule *lookup_or_create_by_name(struct submodule_cache *cache,\n+\t\tconst unsigned char *gitmodules_sha1, const char *name)\n+{\n+\tstruct submodule *submodule;\n+\tstruct strbuf name_buf = STRBUF_INIT;\n+\n+\tsubmodule = cache_lookup_name(cache, gitmodules_sha1, name);\n+\tif (submodule)\n+\t\treturn submodule;\n+\n+\tsubmodule = xmalloc(sizeof(*submodule));\n+\n+\tstrbuf_addstr(&name_buf, name);\n+\tsubmodule->name = strbuf_detach(&name_buf, NULL);\n+\n+\tsubmodule->path = NULL;\n+\tsubmodule->url = NULL;\n+\tsubmodule->fetch_recurse = RECURSE_SUBMODULES_NONE;\n+\tsubmodule->ignore = NULL;\n+\n+\thashcpy(submodule->gitmodules_sha1, gitmodules_sha1);\n+\n+\tcache_add(cache, submodule);\n+\n+\treturn submodule;\n+}\n+\n+static void warn_multiple_config(const unsigned char *commit_sha1,\n+\t\t\t\t const char *name, const char *option)\n+{\n+\tconst char *commit_string = \"WORKTREE\";\n+\tif (commit_sha1)\n+\t\tcommit_string = sha1_to_hex(commit_sha1);\n+\twarning(\"%s:.gitmodules, multiple configurations found for \"\n+\t\t\t\"'submodule.%s.%s'. Skipping second one!\",\n+\t\t\tcommit_string, name, option);\n+}\n+\n+struct parse_config_parameter {\n+\tstruct submodule_cache *cache;\n+\tconst unsigned char *commit_sha1;\n+\tconst unsigned char *gitmodules_sha1;\n+\tint overwrite;\n+};\n+\n+static int parse_config(const char *var, const char *value, void *data)\n+{\n+\tstruct parse_config_parameter *me = data;\n+\tstruct submodule *submodule;\n+\tstruct strbuf name = STRBUF_INIT, item = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\t/* this also ensures that we only parse submodule entries */\n+\tif (!name_and_item_from_var(var, &name, &item))\n+\t\treturn 0;\n+\n+\tsubmodule = lookup_or_create_by_name(me->cache, me->gitmodules_sha1,\n+\t\t\tname.buf);\n+\n+\tif (!strcmp(item.buf, \"path\")) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\t\tif (!value) {\n+\t\t\tret = config_error_nonbool(var);\n+\t\t\tgoto release_return;\n+\t\t}\n+\t\tif (!me->overwrite && submodule->path != NULL) {\n+\t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n+\t\t\t\t\t\"path\");\n+\t\t\tgoto release_return;\n+\t\t}\n+\n+\t\tif (submodule->path)\n+\t\t\tcache_remove_path(me->cache, submodule);\n+\t\tfree((void *) submodule->path);\n+\t\tstrbuf_addstr(&path, value);\n+\t\tsubmodule->path = strbuf_detach(&path, NULL);\n+\t\tcache_put_path(me->cache, submodule);\n+\t} else if (!strcmp(item.buf, \"fetchrecursesubmodules\")) {\n+\t\tif (!me->overwrite &&\n+\t\t submodule->fetch_recurse != RECURSE_SUBMODULES_NONE) {\n+\t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n+\t\t\t\t\t\"fetchrecursesubmodules\");\n+\t\t\tgoto release_return;\n+\t\t}\n+\n+\t\tsubmodule->fetch_recurse = parse_fetch_recurse_submodules_arg(var, value);\n+\t} else if (!strcmp(item.buf, \"ignore\")) {\n+\t\tstruct strbuf ignore = STRBUF_INIT;\n+\t\tif (!me->overwrite && submodule->ignore != NULL) {\n+\t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n+\t\t\t\t\t\"ignore\");\n+\t\t\tgoto release_return;\n+\t\t}\n+\t\tif (!value) {\n+\t\t\tret = config_error_nonbool(var);\n+\t\t\tgoto release_return;\n+\t\t}\n+\t\tif (strcmp(value, \"untracked\") && strcmp(value, \"dirty\") &&\n+\t\t strcmp(value, \"all\") && strcmp(value, \"none\")) {\n+\t\t\twarning(\"Invalid parameter '%s' for config option \"\n+\t\t\t\t\t\"'submodule.%s.ignore'\", value, var);\n+\t\t\tgoto release_return;\n+\t\t}\n+\n+\t\tfree((void *) submodule->ignore);\n+\t\tstrbuf_addstr(&ignore, value);\n+\t\tsubmodule->ignore = strbuf_detach(&ignore, NULL);\n+\t} else if (!strcmp(item.buf, \"url\")) {\n+\t\tstruct strbuf url = STRBUF_INIT;\n+\t\tif (!value) {\n+\t\t\tret = config_error_nonbool(var);\n+\t\t\tgoto release_return;\n+\t\t}\n+\t\tif (!me->overwrite && submodule->url != NULL) {\n+\t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n+\t\t\t\t\t\"url\");\n+\t\t\tgoto release_return;\n+\t\t}\n+\n+\t\tfree((void *) submodule->url);\n+\t\tstrbuf_addstr(&url, value);\n+\t\tsubmodule->url = strbuf_detach(&url, NULL);\n+\t}\n+\n+release_return:\n+\tstrbuf_release(&name);\n+\tstrbuf_release(&item);\n+\n+\treturn ret;\n+}\n+\n+static int gitmodule_sha1_from_commit(const unsigned char *commit_sha1,\n+\t\t\t\t unsigned char *gitmodules_sha1)\n+{\n+\tstruct strbuf rev = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tif (is_null_sha1(commit_sha1)) {\n+\t\thashcpy(gitmodules_sha1, null_sha1);\n+\t\treturn 1;\n+\t}\n+\n+\tstrbuf_addf(&rev, \"%s:.gitmodules\", sha1_to_hex(commit_sha1));\n+\tif (get_sha1(rev.buf, gitmodules_sha1) >= 0)\n+\t\tret = 1;\n+\n+\tstrbuf_release(&rev);\n+\treturn ret;\n+}\n+\n+/* This does a lookup of a submodule configuration by name or by path\n+ * (key) with on-demand reading of the appropriate .gitmodules from\n+ * revisions.\n+ */\n+static const struct submodule *config_from(struct submodule_cache *cache,\n+\t\tconst unsigned char *commit_sha1, const char *key,\n+\t\tenum lookup_type lookup_type)\n+{\n+\tstruct strbuf rev = STRBUF_INIT;\n+\tunsigned long config_size;\n+\tchar *config;\n+\tunsigned char sha1[20];\n+\tenum object_type type;\n+\tconst struct submodule *submodule = NULL;\n+\tstruct parse_config_parameter parameter;\n+\n+\t/*\n+\t * If any parameter except the cache is a NULL pointer just\n+\t * return the first submodule. Can be used to check whether\n+\t * there are any submodules parsed.\n+\t */\n+\tif (!commit_sha1 || !key) {\n+\t\tstruct hashmap_iter iter;\n+\t\tstruct submodule_entry *entry;\n+\n+\t\thashmap_iter_init(&cache->for_name, &iter);\n+\t\tentry = hashmap_iter_next(&iter);\n+\t\tif (!entry)\n+\t\t\treturn NULL;\n+\t\treturn entry->config;\n+\t}\n+\n+\tif (!gitmodule_sha1_from_commit(commit_sha1, sha1))\n+\t\treturn submodule;\n+\n+\tswitch (lookup_type) {\n+\tcase lookup_name:\n+\t\tsubmodule = cache_lookup_name(cache, sha1, key);\n+\t\tbreak;\n+\tcase lookup_path:\n+\t\tsubmodule = cache_lookup_path(cache, sha1, key);\n+\t\tbreak;\n+\t}\n+\tif (submodule)\n+\t\treturn submodule;\n+\n+\tconfig = read_sha1_file(sha1, &type, &config_size);\n+\tif (!config)\n+\t\treturn submodule;\n+\n+\tif (type != OBJ_BLOB) {\n+\t\tfree(config);\n+\t\treturn submodule;\n+\t}\n+\n+\t/* fill the submodule config into the cache */\n+\tparameter.cache = cache;\n+\tparameter.commit_sha1 = commit_sha1;\n+\tparameter.gitmodules_sha1 = sha1;\n+\tparameter.overwrite = 0;\n+\tgit_config_from_buf(parse_config, rev.buf, config, config_size,\n+\t\t\t¶meter);\n+\tfree(config);\n+\n+\tswitch (lookup_type) {\n+\tcase lookup_name:\n+\t\tsubmodule = cache_lookup_name(cache, sha1, key);\n+\t\tbreak;\n+\tcase lookup_path:\n+\t\tsubmodule = cache_lookup_path(cache, sha1, key);\n+\t\tbreak;\n+\t}\n+\n+\treturn submodule;\n+}\n+\n+static const struct submodule *config_from_path(struct submodule_cache *cache,\n+\t\tconst unsigned char *commit_sha1, const char *path)\n+{\n+\treturn config_from(cache, commit_sha1, path, lookup_path);\n+}\n+\n+static const struct submodule *config_from_name(struct submodule_cache *cache,\n+\t\tconst unsigned char *commit_sha1, const char *name)\n+{\n+\treturn config_from(cache, commit_sha1, name, lookup_name);\n+}\n+\n+static void ensure_cache_init(void)\n+{\n+\tif (is_cache_init)\n+\t\treturn;\n+\n+\tcache_init(&cache);\n+\tis_cache_init = 1;\n+}\n+\n+const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n+\t\tconst char *name)\n+{\n+\tensure_cache_init();\n+\treturn config_from_name(&cache, commit_sha1, name);\n+}\n+\n+const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n+\t\tconst char *path)\n+{\n+\tensure_cache_init();\n+\treturn config_from_path(&cache, commit_sha1, path);\n+}\n+\n+void submodule_free(void)\n+{\n+\tcache_free(&cache);\n+\tis_cache_init = 0;\n+}\ndiff --git a/submodule-config.h b/submodule-config.h\nnew file mode 100644\nindex 0000000..cd68030\n--- /dev/null\n+++ b/submodule-config.h\n@@ -0,0 +1,27 @@\n+#ifndef SUBMODULE_CONFIG_CACHE_H\n+#define SUBMODULE_CONFIG_CACHE_H\n+\n+#include \"hashmap.h\"\n+#include \"strbuf.h\"\n+\n+/*\n+ * Submodule entry containing the information about a certain submodule\n+ * in a certain revision.\n+ */\n+struct submodule {\n+\tconst char *path;\n+\tconst char *name;\n+\tconst char *url;\n+\tint fetch_recurse;\n+\tconst char *ignore;\n+\t/* the sha1 blob id of the responsible .gitmodules file */\n+\tunsigned char gitmodules_sha1[20];\n+};\n+\n+const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n+\t\tconst char *name);\n+const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\n+\t\tconst char *path);\n+void submodule_free(void);\n+\n+#endif /* SUBMODULE_CONFIG_H */\ndiff --git a/submodule.c b/submodule.c\nindex b8747f5..7822dc5 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -355,6 +355,7 @@ int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg)\n \tdefault:\n \t\tif (!strcmp(arg, \"on-demand\"))\n \t\t\treturn RECURSE_SUBMODULES_ON_DEMAND;\n+\t\t/* TODO: remove the die for history parsing here */\n \t\tdie(\"bad %s argument: %s\", opt, arg);\n \t}\n }\ndiff --git a/submodule.h b/submodule.h\nindex 7beec48..920fef3 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -5,6 +5,7 @@ struct diff_options;\n struct argv_array;\n \n enum {\n+\tRECURSE_SUBMODULES_NONE = -2,\n \tRECURSE_SUBMODULES_ON_DEMAND = -1,\n \tRECURSE_SUBMODULES_OFF = 0,\n \tRECURSE_SUBMODULES_DEFAULT = 1,\ndiff --git a/t/t7411-submodule-config.sh b/t/t7411-submodule-config.sh\nnew file mode 100755\nindex 0000000..2602bc5\n--- /dev/null\n+++ b/t/t7411-submodule-config.sh\n@@ -0,0 +1,85 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2014 Heiko Voigt\n+#\n+\n+test_description='Test submodules config cache infrastructure\n+\n+This test verifies that parsing .gitmodules configuration directly\n+from the database works.\n+'\n+\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+\n+test_expect_success 'submodule config cache setup' '\n+\tmkdir submodule &&\n+\t(cd submodule &&\n+\t\tgit init &&\n+\t\techo a >a &&\n+\t\tgit add . &&\n+\t\tgit commit -ma\n+\t) &&\n+\tmkdir super &&\n+\t(cd super &&\n+\t\tgit init &&\n+\t\tgit submodule add ../submodule &&\n+\t\tgit submodule add ../submodule a &&\n+\t\tgit commit -m \"add as submodule and as a\" &&\n+\t\tgit mv a b &&\n+\t\tgit commit -m \"move a to b\"\n+\t)\n+'\n+\n+cat >super/expect <actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'test parsing and lookup of submodule config by name' '\n+\t(cd super &&\n+\t\ttest-submodule-config --name \\\n+\t\t\tHEAD^ a \\\n+\t\t\tHEAD a \\\n+\t\t\tHEAD^ submodule \\\n+\t\t\tHEAD submodule \\\n+\t\t\t\t>actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+cat >super/expect_error <>.gitmodules &&\n+\t\tgit add .gitmodules &&\n+\t\tmv .gitmodules.bak .gitmodules &&\n+\t\tgit commit -m \"add error\" &&\n+\t\ttest-submodule-config \\\n+\t\t\tHEAD b \\\n+\t\t\tHEAD submodule \\\n+\t\t\t\t>actual &&\n+\t\ttest_cmp expect_error actual\n+\t)\n+'\n+\n+test_done\ndiff --git a/test-submodule-config.c b/test-submodule-config.c\nnew file mode 100644\nindex 0000000..f3c3918\n--- /dev/null\n+++ b/test-submodule-config.c\n@@ -0,0 +1,66 @@\n+#include \"cache.h\"\n+#include \"submodule-config.h\"\n+\n+static void die_usage(int argc, char **argv, const char *msg)\n+{\n+\tfprintf(stderr, \"%s\\n\", msg);\n+\tfprintf(stderr, \"Usage: %s [ ] ...\\n\", argv[0]);\n+\texit(1);\n+}\n+\n+int main(int argc, char **argv)\n+{\n+\tchar **arg = argv;\n+\tint my_argc = argc;\n+\tint output_url = 0;\n+\tint lookup_name = 0;\n+\n+\targ++;\n+\tmy_argc--;\n+\twhile (starts_with(arg[0], \"--\")) {\n+\t\tif (!strcmp(arg[0], \"--url\"))\n+\t\t\toutput_url = 1;\n+\t\tif (!strcmp(arg[0], \"--name\"))\n+\t\t\tlookup_name = 1;\n+\t\targ++;\n+\t\tmy_argc--;\n+\t}\n+\n+\tif (my_argc % 2 != 0)\n+\t\tdie_usage(argc, argv, \"Wrong number of arguments.\");\n+\n+\twhile (*arg) {\n+\t\tunsigned char commit_sha1[20];\n+\t\tconst struct submodule *submodule;\n+\t\tconst char *commit;\n+\t\tconst char *path_or_name;\n+\n+\t\tcommit = arg[0];\n+\t\tpath_or_name = arg[1];\n+\n+\t\tif (commit[0] == '\\0')\n+\t\t\thashcpy(commit_sha1, null_sha1);\n+\t\telse if (get_sha1(commit, commit_sha1) < 0)\n+\t\t\tdie_usage(argc, argv, \"Commit not found.\");\n+\n+\t\tif (lookup_name) {\n+\t\t\tsubmodule = submodule_from_name(commit_sha1, path_or_name);\n+\t\t} else\n+\t\t\tsubmodule = submodule_from_path(commit_sha1, path_or_name);\n+\t\tif (!submodule)\n+\t\t\tdie_usage(argc, argv, \"Submodule not found.\");\n+\n+\t\tif (output_url)\n+\t\t\tprintf(\"Submodule url: '%s' for path '%s'\\n\",\n+\t\t\t\t\tsubmodule->url, submodule->path);\n+\t\telse\n+\t\t\tprintf(\"Submodule name: '%s' for path '%s'\\n\",\n+\t\t\t\t\tsubmodule->name, submodule->path);\n+\n+\t\targ += 2;\n+\t}\n+\n+\tsubmodule_free();\n+\n+\treturn 0;\n+}\n-- \n2.4.2.391.g2979c89\n"},{"id":"262727","messageId":"20150602142631.GC7713@book.hvoigt.net","threadId":"39497","inReplyTo":"20150602142436.GA7713@book.hvoigt.net","subject":"[PATCH v4 2/4] extract functions for submodule config set and lookup","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2015-06-02T14:26:31Z","receivedAt":"2015-06-02T14:26:31Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"This is one step towards using the new configuration API. We just\nextract these functions to make replacing the actual code easier.\n\nSigned-off-by: Heiko Voigt \n---\n submodule.c | 142 +++++++++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 97 insertions(+), 45 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 7822dc5..c3b5f44 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -41,6 +41,76 @@ static int gitmodules_is_unmerged;\n */\n static int gitmodules_is_modified;\n \n+static const char *get_name_for_path(const char *path)\n+{\n+\tstruct string_list_item *path_option;\n+\tif (path == NULL) {\n+\t\tif (config_name_for_path.nr > 0)\n+\t\t\treturn config_name_for_path.items[0].util;\n+\t\telse\n+\t\t\treturn NULL;\n+\t}\n+\tpath_option = unsorted_string_list_lookup(&config_name_for_path, path);\n+\tif (!path_option)\n+\t\treturn NULL;\n+\treturn path_option->util;\n+}\n+\n+static void set_name_for_path(const char *path, const char *name, int namelen)\n+{\n+\tstruct string_list_item *config;\n+\tconfig = unsorted_string_list_lookup(&config_name_for_path, path);\n+\tif (config)\n+\t\tfree(config->util);\n+\telse\n+\t\tconfig = string_list_append(&config_name_for_path, xstrdup(path));\n+\tconfig->util = xmemdupz(name, namelen);\n+}\n+\n+static const char *get_ignore_for_name(const char *name)\n+{\n+\tstruct string_list_item *ignore_option;\n+\tignore_option = unsorted_string_list_lookup(&config_ignore_for_name, name);\n+\tif (!ignore_option)\n+\t\treturn NULL;\n+\n+\treturn ignore_option->util;\n+}\n+\n+static void set_ignore_for_name(const char *name, int namelen, const char *ignore)\n+{\n+\tstruct string_list_item *config;\n+\tchar *name_cstr = xmemdupz(name, namelen);\n+\tconfig = unsorted_string_list_lookup(&config_ignore_for_name, name_cstr);\n+\tif (config) {\n+\t\tfree(config->util);\n+\t\tfree(name_cstr);\n+\t} else\n+\t\tconfig = string_list_append(&config_ignore_for_name, name_cstr);\n+\tconfig->util = xstrdup(ignore);\n+}\n+\n+static int get_fetch_recurse_for_name(const char *name)\n+{\n+\tstruct string_list_item *fetch_recurse;\n+\tfetch_recurse = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name);\n+\tif (!fetch_recurse)\n+\t\treturn RECURSE_SUBMODULES_NONE;\n+\n+\treturn (intptr_t) fetch_recurse->util;\n+}\n+\n+static void set_fetch_recurse_for_name(const char *name, int namelen, int fetch_recurse)\n+{\n+\tstruct string_list_item *config;\n+\tchar *name_cstr = xmemdupz(name, namelen);\n+\tconfig = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name_cstr);\n+\tif (!config)\n+\t\tconfig = string_list_append(&config_fetch_recurse_submodules_for_name, name_cstr);\n+\telse\n+\t\tfree(name_cstr);\n+\tconfig->util = (void *)(intptr_t) fetch_recurse;\n+}\n \n int is_staging_gitmodules_ok(void)\n {\n@@ -55,7 +125,7 @@ int is_staging_gitmodules_ok(void)\n int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n {\n \tstruct strbuf entry = STRBUF_INIT;\n-\tstruct string_list_item *path_option;\n+\tconst char *path;\n \n \tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n \t\treturn -1;\n@@ -63,13 +133,13 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n \tif (gitmodules_is_unmerged)\n \t\tdie(_(\"Cannot change unmerged .gitmodules, resolve merge conflicts first\"));\n \n-\tpath_option = unsorted_string_list_lookup(&config_name_for_path, oldpath);\n-\tif (!path_option) {\n+\tpath = get_name_for_path(oldpath);\n+\tif (!path) {\n \t\twarning(_(\"Could not find section in .gitmodules where path=%s\"), oldpath);\n \t\treturn -1;\n \t}\n \tstrbuf_addstr(&entry, \"submodule.\");\n-\tstrbuf_addstr(&entry, path_option->util);\n+\tstrbuf_addstr(&entry, path);\n \tstrbuf_addstr(&entry, \".path\");\n \tif (git_config_set_in_file(\".gitmodules\", entry.buf, newpath) < 0) {\n \t\t/* Maybe the user already did that, don't error out here */\n@@ -89,7 +159,7 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n int remove_path_from_gitmodules(const char *path)\n {\n \tstruct strbuf sect = STRBUF_INIT;\n-\tstruct string_list_item *path_option;\n+\tconst char *path_option;\n \n \tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n \t\treturn -1;\n@@ -97,13 +167,13 @@ int remove_path_from_gitmodules(const char *path)\n \tif (gitmodules_is_unmerged)\n \t\tdie(_(\"Cannot change unmerged .gitmodules, resolve merge conflicts first\"));\n \n-\tpath_option = unsorted_string_list_lookup(&config_name_for_path, path);\n+\tpath_option = get_name_for_path(path);\n \tif (!path_option) {\n \t\twarning(_(\"Could not find section in .gitmodules where path=%s\"), path);\n \t\treturn -1;\n \t}\n \tstrbuf_addstr(§, \"submodule.\");\n-\tstrbuf_addstr(§, path_option->util);\n+\tstrbuf_addstr(§, path_option);\n \tif (git_config_rename_section_in_file(\".gitmodules\", sect.buf, NULL) < 0) {\n \t\t/* Maybe the user already did that, don't error out here */\n \t\twarning(_(\"Could not remove .gitmodules entry for %s\"), path);\n@@ -165,12 +235,11 @@ done:\n void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\t\t\t\t const char *path)\n {\n-\tstruct string_list_item *path_option, *ignore_option;\n-\tpath_option = unsorted_string_list_lookup(&config_name_for_path, path);\n-\tif (path_option) {\n-\t\tignore_option = unsorted_string_list_lookup(&config_ignore_for_name, path_option->util);\n-\t\tif (ignore_option)\n-\t\t\thandle_ignore_submodules_arg(diffopt, ignore_option->util);\n+\tconst char *name = get_name_for_path(path);\n+\tif (name) {\n+\t\tconst char *ignore = get_ignore_for_name(name);\n+\t\tif (ignore)\n+\t\t\thandle_ignore_submodules_arg(diffopt, ignore);\n \t\telse if (gitmodules_is_unmerged)\n \t\t\tDIFF_OPT_SET(diffopt, IGNORE_SUBMODULES);\n \t}\n@@ -221,7 +290,6 @@ void gitmodules_config(void)\n \n int parse_submodule_config_option(const char *var, const char *value)\n {\n-\tstruct string_list_item *config;\n \tconst char *name, *key;\n \tint namelen;\n \n@@ -232,22 +300,14 @@ int parse_submodule_config_option(const char *var, const char *value)\n \t\tif (!value)\n \t\t\treturn config_error_nonbool(var);\n \n-\t\tconfig = unsorted_string_list_lookup(&config_name_for_path, value);\n-\t\tif (config)\n-\t\t\tfree(config->util);\n-\t\telse\n-\t\t\tconfig = string_list_append(&config_name_for_path, xstrdup(value));\n-\t\tconfig->util = xmemdupz(name, namelen);\n+\t\tset_name_for_path(value, name, namelen);\n+\n \t} else if (!strcmp(key, \"fetchrecursesubmodules\")) {\n-\t\tchar *name_cstr = xmemdupz(name, namelen);\n-\t\tconfig = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name_cstr);\n-\t\tif (!config)\n-\t\t\tconfig = string_list_append(&config_fetch_recurse_submodules_for_name, name_cstr);\n-\t\telse\n-\t\t\tfree(name_cstr);\n-\t\tconfig->util = (void *)(intptr_t)parse_fetch_recurse_submodules_arg(var, value);\n+\t\tint fetch_recurse = parse_fetch_recurse_submodules_arg(var, value);\n+\n+\t\tset_fetch_recurse_for_name(name, namelen, fetch_recurse);\n+\n \t} else if (!strcmp(key, \"ignore\")) {\n-\t\tchar *name_cstr;\n \n \t\tif (!value)\n \t\t\treturn config_error_nonbool(var);\n@@ -258,14 +318,7 @@ int parse_submodule_config_option(const char *var, const char *value)\n \t\t\treturn 0;\n \t\t}\n \n-\t\tname_cstr = xmemdupz(name, namelen);\n-\t\tconfig = unsorted_string_list_lookup(&config_ignore_for_name, name_cstr);\n-\t\tif (config) {\n-\t\t\tfree(config->util);\n-\t\t\tfree(name_cstr);\n-\t\t} else\n-\t\t\tconfig = string_list_append(&config_ignore_for_name, name_cstr);\n-\t\tconfig->util = xstrdup(value);\n+\t\tset_ignore_for_name(name, namelen, value);\n \t\treturn 0;\n \t}\n \treturn 0;\n@@ -646,7 +699,7 @@ static void calculate_changed_submodule_paths(void)\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \t/* No need to check if there are no submodules configured */\n-\tif (!config_name_for_path.nr)\n+\tif (!get_name_for_path(NULL))\n \t\treturn;\n \n \tinit_revisions(&rev, NULL);\n@@ -693,7 +746,7 @@ int fetch_populated_submodules(const struct argv_array *options,\n \tint i, result = 0;\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n-\tstruct string_list_item *name_for_path;\n+\tconst char *name_for_path;\n \tconst char *work_tree = get_git_work_tree();\n \tif (!work_tree)\n \t\tgoto out;\n@@ -724,18 +777,17 @@ int fetch_populated_submodules(const struct argv_array *options,\n \t\t\tcontinue;\n \n \t\tname = ce->name;\n-\t\tname_for_path = unsorted_string_list_lookup(&config_name_for_path, ce->name);\n+\t\tname_for_path = get_name_for_path(ce->name);\n \t\tif (name_for_path)\n-\t\t\tname = name_for_path->util;\n+\t\t\tname = name_for_path;\n \n \t\tdefault_argv = \"yes\";\n \t\tif (command_line_option == RECURSE_SUBMODULES_DEFAULT) {\n-\t\t\tstruct string_list_item *fetch_recurse_submodules_option;\n-\t\t\tfetch_recurse_submodules_option = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name);\n-\t\t\tif (fetch_recurse_submodules_option) {\n-\t\t\t\tif ((intptr_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_OFF)\n+\t\t\tint fetch_recurse_option = get_fetch_recurse_for_name(name);\n+\t\t\tif (fetch_recurse_option != RECURSE_SUBMODULES_NONE) {\n+\t\t\t\tif (fetch_recurse_option == RECURSE_SUBMODULES_OFF)\n \t\t\t\t\tcontinue;\n-\t\t\t\tif ((intptr_t)fetch_recurse_submodules_option->util == RECURSE_SUBMODULES_ON_DEMAND) {\n+\t\t\t\tif (fetch_recurse_option == RECURSE_SUBMODULES_ON_DEMAND) {\n \t\t\t\t\tif (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))\n \t\t\t\t\t\tcontinue;\n \t\t\t\t\tdefault_argv = \"on-demand\";\n-- \n2.4.2.391.g2979c89\n"},{"id":"262728","messageId":"20150602142714.GD7713@book.hvoigt.net","threadId":"39497","inReplyTo":"20150602142436.GA7713@book.hvoigt.net","subject":"[PATCH v4 3/4] use new config API for worktree configurations of submodules","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2015-06-02T14:27:15Z","receivedAt":"2015-06-02T14:27:15Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"We remove the extracted functions and directly parse into and read out\nof the cache. This allows us to have one unified way of accessing\nsubmodule configuration values specific to single submodules. Regardless\nwhether we need to access a configuration from history or from the\nworktree.\n\nSigned-off-by: Heiko Voigt \n---\n Documentation/technical/api-submodule-config.txt | 19 ++-\n builtin/checkout.c | 1 +\n diff.c | 1 +\n submodule-config.c | 12 ++\n submodule-config.h | 1 +\n submodule.c | 160 ++++-------------------\n submodule.h | 1 -\n t/t7411-submodule-config.sh | 37 +++++-\n test-submodule-config.c | 10 ++\n 9 files changed, 104 insertions(+), 138 deletions(-)\n\ndiff --git a/Documentation/technical/api-submodule-config.txt b/Documentation/technical/api-submodule-config.txt\nindex 2ff4907..2ea520a 100644\n--- a/Documentation/technical/api-submodule-config.txt\n+++ b/Documentation/technical/api-submodule-config.txt\n@@ -10,10 +10,18 @@ submodule path or name.\n Usage\n -----\n \n+To initialize the cache with configurations from the worktree the caller\n+typically first calls `gitmodules_config()` to read values from the\n+worktree .gitmodules and then to overlay the local git config values\n+`parse_submodule_config_option()` from the config parsing\n+infrastructure.\n+\n The caller can look up information about submodules by using the\n `submodule_from_path()` or `submodule_from_name()` functions. They return\n a `struct submodule` which contains the values. The API automatically\n-initializes and allocates the needed infrastructure on-demand.\n+initializes and allocates the needed infrastructure on-demand. If the\n+caller does only want to lookup values from revisions the initialization\n+can be skipped.\n \n If the internal cache might grow too big or when the caller is done with\n the API, all internally cached values can be freed with submodule_free().\n@@ -34,6 +42,11 @@ Functions\n \n \tUse these to free the internally cached values.\n \n+`int parse_submodule_config_option(const char *var, const char *value)`::\n+\n+\tCan be passed to the config parsing infrastructure to parse\n+\tlocal (worktree) submodule configurations.\n+\n `const struct submodule *submodule_from_path(const unsigned char *commit_sha1, const char *path)`::\n \n \tLookup values for one submodule by its commit_sha1 and path or\n@@ -43,4 +56,8 @@ Functions\n \n \tThe same as above but lookup by name.\n \n+If given the null_sha1 as commit_sha1 the local configuration of a\n+submodule will be returned (e.g. consolidated values from local git\n+configuration and the .gitmodules file in the worktree).\n+\n For an example usage see test-submodule-config.c.\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 2f92328..f1f168d 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -18,6 +18,7 @@\n #include \"xdiff-interface.h\"\n #include \"ll-merge.h\"\n #include \"resolve-undo.h\"\n+#include \"submodule-config.h\"\n #include \"submodule.h\"\n #include \"argv-array.h\"\n #include \"sigchain.h\"\ndiff --git a/diff.c b/diff.c\nindex 7500c55..d0be279 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -13,6 +13,7 @@\n #include \"utf8.h\"\n #include \"userdiff.h\"\n #include \"sigchain.h\"\n+#include \"submodule-config.h\"\n #include \"submodule.h\"\n #include \"ll-merge.h\"\n #include \"string-list.h\"\ndiff --git a/submodule-config.c b/submodule-config.c\nindex 97f4a04..fc7bf40 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -424,6 +424,18 @@ static void ensure_cache_init(void)\n \tis_cache_init = 1;\n }\n \n+int parse_submodule_config_option(const char *var, const char *value)\n+{\n+\tstruct parse_config_parameter parameter;\n+\tparameter.cache = &cache;\n+\tparameter.commit_sha1 = NULL;\n+\tparameter.gitmodules_sha1 = null_sha1;\n+\tparameter.overwrite = 1;\n+\n+\tensure_cache_init();\n+\treturn parse_config(var, value, ¶meter);\n+}\n+\n const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name)\n {\ndiff --git a/submodule-config.h b/submodule-config.h\nindex cd68030..5fe44ce 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -18,6 +18,7 @@ struct submodule {\n \tunsigned char gitmodules_sha1[20];\n };\n \n+int parse_submodule_config_option(const char *var, const char *value);\n const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\n const struct submodule *submodule_from_path(const unsigned char *commit_sha1,\ndiff --git a/submodule.c b/submodule.c\nindex c3b5f44..97355eb 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"submodule-config.h\"\n #include \"submodule.h\"\n #include \"dir.h\"\n #include \"diff.h\"\n@@ -12,9 +13,6 @@\n #include \"argv-array.h\"\n #include \"blob.h\"\n \n-static struct string_list config_name_for_path;\n-static struct string_list config_fetch_recurse_submodules_for_name;\n-static struct string_list config_ignore_for_name;\n static int config_fetch_recurse_submodules = RECURSE_SUBMODULES_ON_DEMAND;\n static struct string_list changed_submodule_paths;\n static int initialized_fetch_ref_tips;\n@@ -41,77 +39,6 @@ static int gitmodules_is_unmerged;\n */\n static int gitmodules_is_modified;\n \n-static const char *get_name_for_path(const char *path)\n-{\n-\tstruct string_list_item *path_option;\n-\tif (path == NULL) {\n-\t\tif (config_name_for_path.nr > 0)\n-\t\t\treturn config_name_for_path.items[0].util;\n-\t\telse\n-\t\t\treturn NULL;\n-\t}\n-\tpath_option = unsorted_string_list_lookup(&config_name_for_path, path);\n-\tif (!path_option)\n-\t\treturn NULL;\n-\treturn path_option->util;\n-}\n-\n-static void set_name_for_path(const char *path, const char *name, int namelen)\n-{\n-\tstruct string_list_item *config;\n-\tconfig = unsorted_string_list_lookup(&config_name_for_path, path);\n-\tif (config)\n-\t\tfree(config->util);\n-\telse\n-\t\tconfig = string_list_append(&config_name_for_path, xstrdup(path));\n-\tconfig->util = xmemdupz(name, namelen);\n-}\n-\n-static const char *get_ignore_for_name(const char *name)\n-{\n-\tstruct string_list_item *ignore_option;\n-\tignore_option = unsorted_string_list_lookup(&config_ignore_for_name, name);\n-\tif (!ignore_option)\n-\t\treturn NULL;\n-\n-\treturn ignore_option->util;\n-}\n-\n-static void set_ignore_for_name(const char *name, int namelen, const char *ignore)\n-{\n-\tstruct string_list_item *config;\n-\tchar *name_cstr = xmemdupz(name, namelen);\n-\tconfig = unsorted_string_list_lookup(&config_ignore_for_name, name_cstr);\n-\tif (config) {\n-\t\tfree(config->util);\n-\t\tfree(name_cstr);\n-\t} else\n-\t\tconfig = string_list_append(&config_ignore_for_name, name_cstr);\n-\tconfig->util = xstrdup(ignore);\n-}\n-\n-static int get_fetch_recurse_for_name(const char *name)\n-{\n-\tstruct string_list_item *fetch_recurse;\n-\tfetch_recurse = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name);\n-\tif (!fetch_recurse)\n-\t\treturn RECURSE_SUBMODULES_NONE;\n-\n-\treturn (intptr_t) fetch_recurse->util;\n-}\n-\n-static void set_fetch_recurse_for_name(const char *name, int namelen, int fetch_recurse)\n-{\n-\tstruct string_list_item *config;\n-\tchar *name_cstr = xmemdupz(name, namelen);\n-\tconfig = unsorted_string_list_lookup(&config_fetch_recurse_submodules_for_name, name_cstr);\n-\tif (!config)\n-\t\tconfig = string_list_append(&config_fetch_recurse_submodules_for_name, name_cstr);\n-\telse\n-\t\tfree(name_cstr);\n-\tconfig->util = (void *)(intptr_t) fetch_recurse;\n-}\n-\n int is_staging_gitmodules_ok(void)\n {\n \treturn !gitmodules_is_modified;\n@@ -125,7 +52,7 @@ int is_staging_gitmodules_ok(void)\n int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n {\n \tstruct strbuf entry = STRBUF_INIT;\n-\tconst char *path;\n+\tconst struct submodule *submodule;\n \n \tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n \t\treturn -1;\n@@ -133,13 +60,13 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n \tif (gitmodules_is_unmerged)\n \t\tdie(_(\"Cannot change unmerged .gitmodules, resolve merge conflicts first\"));\n \n-\tpath = get_name_for_path(oldpath);\n-\tif (!path) {\n+\tsubmodule = submodule_from_path(null_sha1, oldpath);\n+\tif (!submodule || !submodule->name) {\n \t\twarning(_(\"Could not find section in .gitmodules where path=%s\"), oldpath);\n \t\treturn -1;\n \t}\n \tstrbuf_addstr(&entry, \"submodule.\");\n-\tstrbuf_addstr(&entry, path);\n+\tstrbuf_addstr(&entry, submodule->name);\n \tstrbuf_addstr(&entry, \".path\");\n \tif (git_config_set_in_file(\".gitmodules\", entry.buf, newpath) < 0) {\n \t\t/* Maybe the user already did that, don't error out here */\n@@ -159,7 +86,7 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)\n int remove_path_from_gitmodules(const char *path)\n {\n \tstruct strbuf sect = STRBUF_INIT;\n-\tconst char *path_option;\n+\tconst struct submodule *submodule;\n \n \tif (!file_exists(\".gitmodules\")) /* Do nothing without .gitmodules */\n \t\treturn -1;\n@@ -167,13 +94,13 @@ int remove_path_from_gitmodules(const char *path)\n \tif (gitmodules_is_unmerged)\n \t\tdie(_(\"Cannot change unmerged .gitmodules, resolve merge conflicts first\"));\n \n-\tpath_option = get_name_for_path(path);\n-\tif (!path_option) {\n+\tsubmodule = submodule_from_path(null_sha1, path);\n+\tif (!submodule || !submodule->name) {\n \t\twarning(_(\"Could not find section in .gitmodules where path=%s\"), path);\n \t\treturn -1;\n \t}\n \tstrbuf_addstr(§, \"submodule.\");\n-\tstrbuf_addstr(§, path_option);\n+\tstrbuf_addstr(§, submodule->name);\n \tif (git_config_rename_section_in_file(\".gitmodules\", sect.buf, NULL) < 0) {\n \t\t/* Maybe the user already did that, don't error out here */\n \t\twarning(_(\"Could not remove .gitmodules entry for %s\"), path);\n@@ -235,11 +162,10 @@ done:\n void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\t\t\t\t const char *path)\n {\n-\tconst char *name = get_name_for_path(path);\n-\tif (name) {\n-\t\tconst char *ignore = get_ignore_for_name(name);\n-\t\tif (ignore)\n-\t\t\thandle_ignore_submodules_arg(diffopt, ignore);\n+\tconst struct submodule *submodule = submodule_from_path(null_sha1, path);\n+\tif (submodule) {\n+\t\tif (submodule->ignore)\n+\t\t\thandle_ignore_submodules_arg(diffopt, submodule->ignore);\n \t\telse if (gitmodules_is_unmerged)\n \t\t\tDIFF_OPT_SET(diffopt, IGNORE_SUBMODULES);\n \t}\n@@ -288,42 +214,6 @@ void gitmodules_config(void)\n \t}\n }\n \n-int parse_submodule_config_option(const char *var, const char *value)\n-{\n-\tconst char *name, *key;\n-\tint namelen;\n-\n-\tif (parse_config_key(var, \"submodule\", &name, &namelen, &key) < 0 || !name)\n-\t\treturn 0;\n-\n-\tif (!strcmp(key, \"path\")) {\n-\t\tif (!value)\n-\t\t\treturn config_error_nonbool(var);\n-\n-\t\tset_name_for_path(value, name, namelen);\n-\n-\t} else if (!strcmp(key, \"fetchrecursesubmodules\")) {\n-\t\tint fetch_recurse = parse_fetch_recurse_submodules_arg(var, value);\n-\n-\t\tset_fetch_recurse_for_name(name, namelen, fetch_recurse);\n-\n-\t} else if (!strcmp(key, \"ignore\")) {\n-\n-\t\tif (!value)\n-\t\t\treturn config_error_nonbool(var);\n-\n-\t\tif (strcmp(value, \"untracked\") && strcmp(value, \"dirty\") &&\n-\t\t strcmp(value, \"all\") && strcmp(value, \"none\")) {\n-\t\t\twarning(\"Invalid parameter \\\"%s\\\" for config option \\\"submodule.%s.ignore\\\"\", value, var);\n-\t\t\treturn 0;\n-\t\t}\n-\n-\t\tset_ignore_for_name(name, namelen, value);\n-\t\treturn 0;\n-\t}\n-\treturn 0;\n-}\n-\n void handle_ignore_submodules_arg(struct diff_options *diffopt,\n \t\t\t\t const char *arg)\n {\n@@ -699,7 +589,7 @@ static void calculate_changed_submodule_paths(void)\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n \t/* No need to check if there are no submodules configured */\n-\tif (!get_name_for_path(NULL))\n+\tif (!submodule_from_path(NULL, NULL))\n \t\treturn;\n \n \tinit_revisions(&rev, NULL);\n@@ -746,7 +636,6 @@ int fetch_populated_submodules(const struct argv_array *options,\n \tint i, result = 0;\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n-\tconst char *name_for_path;\n \tconst char *work_tree = get_git_work_tree();\n \tif (!work_tree)\n \t\tgoto out;\n@@ -771,23 +660,26 @@ int fetch_populated_submodules(const struct argv_array *options,\n \t\tstruct strbuf submodule_git_dir = STRBUF_INIT;\n \t\tstruct strbuf submodule_prefix = STRBUF_INIT;\n \t\tconst struct cache_entry *ce = active_cache[i];\n-\t\tconst char *git_dir, *name, *default_argv;\n+\t\tconst char *git_dir, *default_argv;\n+\t\tconst struct submodule *submodule;\n \n \t\tif (!S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n \n-\t\tname = ce->name;\n-\t\tname_for_path = get_name_for_path(ce->name);\n-\t\tif (name_for_path)\n-\t\t\tname = name_for_path;\n+\t\tsubmodule = submodule_from_path(null_sha1, ce->name);\n+\t\tif (!submodule)\n+\t\t\tsubmodule = submodule_from_name(null_sha1, ce->name);\n \n \t\tdefault_argv = \"yes\";\n \t\tif (command_line_option == RECURSE_SUBMODULES_DEFAULT) {\n-\t\t\tint fetch_recurse_option = get_fetch_recurse_for_name(name);\n-\t\t\tif (fetch_recurse_option != RECURSE_SUBMODULES_NONE) {\n-\t\t\t\tif (fetch_recurse_option == RECURSE_SUBMODULES_OFF)\n+\t\t\tif (submodule &&\n+\t\t\t submodule->fetch_recurse !=\n+\t\t\t\t\t\tRECURSE_SUBMODULES_NONE) {\n+\t\t\t\tif (submodule->fetch_recurse ==\n+\t\t\t\t\t\tRECURSE_SUBMODULES_OFF)\n \t\t\t\t\tcontinue;\n-\t\t\t\tif (fetch_recurse_option == RECURSE_SUBMODULES_ON_DEMAND) {\n+\t\t\t\tif (submodule->fetch_recurse ==\n+\t\t\t\t\t\tRECURSE_SUBMODULES_ON_DEMAND) {\n \t\t\t\t\tif (!unsorted_string_list_lookup(&changed_submodule_paths, ce->name))\n \t\t\t\t\t\tcontinue;\n \t\t\t\t\tdefault_argv = \"on-demand\";\ndiff --git a/submodule.h b/submodule.h\nindex 920fef3..547219d 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -20,7 +20,6 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n \t\tconst char *path);\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n-int parse_submodule_config_option(const char *var, const char *value);\n void handle_ignore_submodules_arg(struct diff_options *diffopt, const char *);\n int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);\n void show_submodule_summary(FILE *f, const char *path,\ndiff --git a/t/t7411-submodule-config.sh b/t/t7411-submodule-config.sh\nindex 2602bc5..7229978 100755\n--- a/t/t7411-submodule-config.sh\n+++ b/t/t7411-submodule-config.sh\n@@ -5,8 +5,8 @@\n \n test_description='Test submodules config cache infrastructure\n \n-This test verifies that parsing .gitmodules configuration directly\n-from the database works.\n+This test verifies that parsing .gitmodules configurations directly\n+from the database and from the worktree works.\n '\n \n TEST_NO_CREATE_REPO=1\n@@ -82,4 +82,37 @@ test_expect_success 'error in one submodule config lets continue' '\n \t)\n '\n \n+cat >super/expect_url <super/expect_local_path <actual &&\n+\t\ttest_cmp expect_url actual &&\n+\t\tgit config submodule.a.path c &&\n+\t\ttest-submodule-config \\\n+\t\t\t\"\" c \\\n+\t\t\t\"\" submodule \\\n+\t\t\t\t>actual &&\n+\t\ttest_cmp expect_local_path actual &&\n+\t\tgit config submodule.a.url $old_a &&\n+\t\tgit config submodule.submodule.url $old_submodule &&\n+\t\tgit config --unset submodule.a.path c\n+\t)\n+'\n+\n test_done\ndiff --git a/test-submodule-config.c b/test-submodule-config.c\nindex f3c3918..dab8c27 100644\n--- a/test-submodule-config.c\n+++ b/test-submodule-config.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"submodule-config.h\"\n+#include \"submodule.h\"\n \n static void die_usage(int argc, char **argv, const char *msg)\n {\n@@ -8,6 +9,11 @@ static void die_usage(int argc, char **argv, const char *msg)\n \texit(1);\n }\n \n+static int git_test_config(const char *var, const char *value, void *cb)\n+{\n+\treturn parse_submodule_config_option(var, value);\n+}\n+\n int main(int argc, char **argv)\n {\n \tchar **arg = argv;\n@@ -29,6 +35,10 @@ int main(int argc, char **argv)\n \tif (my_argc % 2 != 0)\n \t\tdie_usage(argc, argv, \"Wrong number of arguments.\");\n \n+\tsetup_git_directory();\n+\tgitmodules_config();\n+\tgit_config(git_test_config, NULL);\n+\n \twhile (*arg) {\n \t\tunsigned char commit_sha1[20];\n \t\tconst struct submodule *submodule;\n-- \n2.4.2.391.g2979c89\n"},{"id":"262730","messageId":"20150602142807.GE7713@book.hvoigt.net","threadId":"39497","inReplyTo":"20150602142436.GA7713@book.hvoigt.net","subject":"[PATCH v4 4/4] do not die on error of parsing fetchrecursesubmodules option","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2015-06-02T14:28:07Z","receivedAt":"2015-06-02T14:28:07Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"We should not die when reading the submodule config cache since the user\nmight not be able to get out of that situation when the configuration is\npart of the history.\n\nWe should handle this condition later when the value is about to be\nused.\n\nSigned-off-by: Heiko Voigt \n---\n builtin/fetch.c | 1 +\n submodule-config.c | 29 ++++++++++++++++++++++++++++-\n submodule-config.h | 1 +\n submodule.c | 15 ---------------\n submodule.h | 2 +-\n t/t7411-submodule-config.sh | 35 +++++++++++++++++++++++++++++++++++\n 6 files changed, 66 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7910419..faae548 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -11,6 +11,7 @@\n #include \"run-command.h\"\n #include \"parse-options.h\"\n #include \"sigchain.h\"\n+#include \"submodule-config.h\"\n #include \"submodule.h\"\n #include \"connected.h\"\n #include \"argv-array.h\"\ndiff --git a/submodule-config.c b/submodule-config.c\nindex fc7bf40..199692b 100644\n--- a/submodule-config.c\n+++ b/submodule-config.c\n@@ -204,6 +204,30 @@ static struct submodule *lookup_or_create_by_name(struct submodule_cache *cache,\n \treturn submodule;\n }\n \n+static int parse_fetch_recurse(const char *opt, const char *arg,\n+\t\t\t int die_on_error)\n+{\n+\tswitch (git_config_maybe_bool(opt, arg)) {\n+\tcase 1:\n+\t\treturn RECURSE_SUBMODULES_ON;\n+\tcase 0:\n+\t\treturn RECURSE_SUBMODULES_OFF;\n+\tdefault:\n+\t\tif (!strcmp(arg, \"on-demand\"))\n+\t\t\treturn RECURSE_SUBMODULES_ON_DEMAND;\n+\n+\t\tif (die_on_error)\n+\t\t\tdie(\"bad %s argument: %s\", opt, arg);\n+\t\telse\n+\t\t\treturn RECURSE_SUBMODULES_ERROR;\n+\t}\n+}\n+\n+int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg)\n+{\n+\treturn parse_fetch_recurse(opt, arg, 1);\n+}\n+\n static void warn_multiple_config(const unsigned char *commit_sha1,\n \t\t\t\t const char *name, const char *option)\n {\n@@ -255,6 +279,8 @@ static int parse_config(const char *var, const char *value, void *data)\n \t\tsubmodule->path = strbuf_detach(&path, NULL);\n \t\tcache_put_path(me->cache, submodule);\n \t} else if (!strcmp(item.buf, \"fetchrecursesubmodules\")) {\n+\t\t/* when parsing worktree configurations we can die early */\n+\t\tint die_on_error = is_null_sha1(me->gitmodules_sha1);\n \t\tif (!me->overwrite &&\n \t\t submodule->fetch_recurse != RECURSE_SUBMODULES_NONE) {\n \t\t\twarn_multiple_config(me->commit_sha1, submodule->name,\n@@ -262,7 +288,8 @@ static int parse_config(const char *var, const char *value, void *data)\n \t\t\tgoto release_return;\n \t\t}\n \n-\t\tsubmodule->fetch_recurse = parse_fetch_recurse_submodules_arg(var, value);\n+\t\tsubmodule->fetch_recurse = parse_fetch_recurse(var, value,\n+\t\t\t\t\t\t\t\tdie_on_error);\n \t} else if (!strcmp(item.buf, \"ignore\")) {\n \t\tstruct strbuf ignore = STRBUF_INIT;\n \t\tif (!me->overwrite && submodule->ignore != NULL) {\ndiff --git a/submodule-config.h b/submodule-config.h\nindex 5fe44ce..9061e4e 100644\n--- a/submodule-config.h\n+++ b/submodule-config.h\n@@ -18,6 +18,7 @@ struct submodule {\n \tunsigned char gitmodules_sha1[20];\n };\n \n+int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);\n int parse_submodule_config_option(const char *var, const char *value);\n const struct submodule *submodule_from_name(const unsigned char *commit_sha1,\n \t\tconst char *name);\ndiff --git a/submodule.c b/submodule.c\nindex 97355eb..4822559 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -288,21 +288,6 @@ static void print_submodule_summary(struct rev_info *rev, FILE *f,\n \tstrbuf_release(&sb);\n }\n \n-int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg)\n-{\n-\tswitch (git_config_maybe_bool(opt, arg)) {\n-\tcase 1:\n-\t\treturn RECURSE_SUBMODULES_ON;\n-\tcase 0:\n-\t\treturn RECURSE_SUBMODULES_OFF;\n-\tdefault:\n-\t\tif (!strcmp(arg, \"on-demand\"))\n-\t\t\treturn RECURSE_SUBMODULES_ON_DEMAND;\n-\t\t/* TODO: remove the die for history parsing here */\n-\t\tdie(\"bad %s argument: %s\", opt, arg);\n-\t}\n-}\n-\n void show_submodule_summary(FILE *f, const char *path,\n \t\tconst char *line_prefix,\n \t\tunsigned char one[20], unsigned char two[20],\ndiff --git a/submodule.h b/submodule.h\nindex 547219d..5507c3d 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -5,6 +5,7 @@ struct diff_options;\n struct argv_array;\n \n enum {\n+\tRECURSE_SUBMODULES_ERROR = -3,\n \tRECURSE_SUBMODULES_NONE = -2,\n \tRECURSE_SUBMODULES_ON_DEMAND = -1,\n \tRECURSE_SUBMODULES_OFF = 0,\n@@ -21,7 +22,6 @@ void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,\n int submodule_config(const char *var, const char *value, void *cb);\n void gitmodules_config(void);\n void handle_ignore_submodules_arg(struct diff_options *diffopt, const char *);\n-int parse_fetch_recurse_submodules_arg(const char *opt, const char *arg);\n void show_submodule_summary(FILE *f, const char *path,\n \t\tconst char *line_prefix,\n \t\tunsigned char one[20], unsigned char two[20],\ndiff --git a/t/t7411-submodule-config.sh b/t/t7411-submodule-config.sh\nindex 7229978..fc97c33 100755\n--- a/t/t7411-submodule-config.sh\n+++ b/t/t7411-submodule-config.sh\n@@ -115,4 +115,39 @@ test_expect_success 'reading of local configuration' '\n \t)\n '\n \n+cat >super/expect_fetchrecurse_die.err <actual.out 2>actual.err &&\n+\t\ttouch expect_fetchrecurse_die.out &&\n+\t\ttest_cmp expect_fetchrecurse_die.out actual.out &&\n+\t\ttest_cmp expect_fetchrecurse_die.err actual.err &&\n+\t\tgit config --unset submodule.submodule.fetchrecursesubmodules\n+\t)\n+'\n+\n+test_expect_success 'error in history in fetchrecursesubmodule lets continue' '\n+\t(cd super &&\n+\t\tgit config -f .gitmodules \\\n+\t\t\tsubmodule.submodule.fetchrecursesubmodules blabla &&\n+\t\tgit add .gitmodules &&\n+\t\tgit config --unset -f .gitmodules \\\n+\t\t\tsubmodule.submodule.fetchrecursesubmodules &&\n+\t\tgit commit -m \"add error in fetchrecursesubmodules\" &&\n+\t\ttest-submodule-config \\\n+\t\t\tHEAD b \\\n+\t\t\tHEAD submodule \\\n+\t\t\t\t>actual &&\n+\t\ttest_cmp expect_error actual &&\n+\t\tgit reset --hard HEAD^\n+\t)\n+'\n+\n test_done\n-- \n2.4.2.391.g2979c89\n"},{"id":"262769","messageId":"xmqqbngx3muj.fsf@gitster.dls.corp.google.com","threadId":"39497","inReplyTo":"20150602142538.GB7713@book.hvoigt.net","subject":"Re: [PATCH v4 1/4] implement submodule config cache for lookup of submodule names","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-06-02T19:57:08Z","receivedAt":"2015-06-02T19:57:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Heiko Voigt writes:\n\n> This submodule configuration cache allows us to lazily read .gitmodules\n> configurations by commit into a runtime cache which can then be used to\n> easily lookup values from it. Currently only the values for path or name\n> are stored but it can be extended for any value needed.\n>\n> It is expected that .gitmodules files do not change often between\n> commits. Thats why we lookup the .gitmodules sha1 from a commit and then\n> either lookup an already parsed configuration or parse and cache an\n> unknown one for each sha1. The cache is lazily build on demand for each\n> requested commit.\n>\n> This cache can be used for all purposes which need knowledge about\n> submodule configurations. Example use cases are:\n>\n> * Recursive submodule checkout needs lookup a submodule name from its\n> path when a submodule first appears. This needs be done before this\n> configuration exists in the worktree.\n>\n> * The implementation of submodule support for 'git archive' needs to\n> lookup the submodule name to generate the archive when given a\n> revision that is not checked out.\n>\n> * 'git fetch' when given the --recurse-submodules=on-demand option (or\n> configuration) needs to lookup submodule names by path from the\n> database rather than reading from the worktree. For new submodule it\n> needs to lookup the name from its path to allow cloning new\n> submodules into the .git folder so they can be checked out without\n> any network interaction when the user does a checkout of that\n> revision.\n\nWhat is unclear to me after reading the above twice is what this\nthing is meant to achieve. Is it efficiency by doing lazy lookups\nand caching to avoid asking the same thing more than once from\neither the filesystem or read_sha1_file()? Is it expected that\nreading through this \"cache\" will be the _only_ way callers would\ninteract with the .gitmodules data, or is it an opt-in feature that\nsome callers that do not see the benefit (why they may want to\nignore is totally unclear, because what the \"cache\" system wants to\nachieve is) can safely ignore and bypass?\n\nBecause the above talks about looking up \".gitmodules from a\ncommit\", I am guessing that the \"commit\" used as one of the lookup\nkeys throughout the system is a commit in the superproject, not from\nsubmodules, but you may want to state that more explicitly.\n\nWhat, if anything, should be done for .gitmodules that are not yet\ncommitted? Are there cases that the callers that usually interact\nwith .gitmodules via this \"cache\" system need to use .gitmodules\nimmediately after adding a new submodule but before committing that\nchange to the superproject? Do they code something like this?\n\n\tif (cached)\n \tread .gitmodules from the index and fabricate\n\t\tstruct submodule;\n\telse if (worktree)\n \tread .gitmodules from the working tree and fabricate\n\t\tstruct submodule;\n\telse\n\t\tcall submodule_from_name(\"HEAD\", ...) and receive\n struct submodule;\n\n\tuse the struct submodule to learn from the module;\n\nYes, I am wondering if submodule_from_name() should be extended to\nallow the former two cases, so that the caller can make a single\ncall above and then use resulting \"struct submodule\" throughout its\ncode after doing so. And I also am hoping that the answer to my\nquestions above to be \"This is not just an opt-in 'cache' API, but\nwe want to make it the unified API for C code to learn about what is\nin .gitmodule\".\n\n> diff --git a/Documentation/technical/api-submodule-config.txt b/Documentation/technical/api-submodule-config.txt\n> new file mode 100644\n> index 0000000..2ff4907\n> --- /dev/null\n> +++ b/Documentation/technical/api-submodule-config.txt\n> @@ -0,0 +1,46 @@\n> +submodule config cache API\n> +==========================\n> +\n> +The submodule config cache API allows to read submodule\n> +configurations/information from specified revisions. Internally\n> +information is lazily read into a cache that is used to avoid\n> +unnecessary parsing of the same .gitmodule files. Lookups can be done by\n> +submodule path or name.\n> +\n> +Usage\n> +-----\n> +\n> +The caller can look up information about submodules by using the\n> +`submodule_from_path()` or `submodule_from_name()` functions. They return\n> +a `struct submodule` which contains the values. The API automatically\n> +initializes and allocates the needed infrastructure on-demand.\n> +\n> +If the internal cache might grow too big or when the caller is done with\n> +the API, all internally cached values can be freed with submodule_free().\n> +\n> +Data Structures\n> +---------------\n> +\n> +`struct submodule`::\n> +\n> +\tThis structure is used to return the information about one\n> +\tsubmodule for a certain revision. It is returned by the lookup\n> +\tfunctions.\n\nHopefully this will not stay an opaque structure as we read later\npatches ;-).\n\n> +Functions\n> +---------\n> +\n> +`void submodule_free()`::\n> +\n> +\tUse these to free the internally cached values.\n\n\"These\" meaning \"this single function\", or are there variants of it?\n\n> diff --git a/submodule-config.c b/submodule-config.c\n> new file mode 100644\n> index 0000000..97f4a04\n> --- /dev/null\n> +++ b/submodule-config.c\n> @@ -0,0 +1,445 @@\n> +#include \"cache.h\"\n> +#include \"submodule-config.h\"\n> +#include \"submodule.h\"\n> +#include \"strbuf.h\"\n> +\n> +/*\n> + * submodule cache lookup structure\n> + * There is one shared set of 'struct submodule' entries which can be\n> + * looked up by their sha1 blob id of the .gitmodule file and either\n> + * using path or name as key.\n> + * for_path stores submodule entries with path as key\n> + * for_name stores submodule entries with name as key\n> + */\n> +struct submodule_cache {\n> +\tstruct hashmap for_path;\n> +\tstruct hashmap for_name;\n> +};\n> +\n> +/*\n> + * thin wrapper struct needed to insert 'struct submodule' entries to\n> + * the hashmap\n> + */\n> +struct submodule_entry {\n> +\tstruct hashmap_entry ent;\n> +\tstruct submodule *config;\n> +};\n\nThe above, and the singleton-ness of the \"cache\", implies that we\ncan have only one \"struct submodule\" for a given path (or a name).\nDoes that mean the subsystem implicitly is tied to a single commit\nat the superproject level?\n\nWhat happens when I call submodule_from_path() for a single\nsubmodule at one commit in the superproject, and then ask about that\nsame submodule for another commit in the superproject, which may\nhave a different version of .gitmodules, by calling the same\nfunction again?\n\n\tSide note: I think I know the answer to these questions,\n\tafter reading the hash function. for_path does not store\n\tsubmodule entries with path as key. It uses the commit and\n\tthe path as a combined key, so both HEAD:.gitmodules and\n\tHEAD^:.gitmodules can be cached and looked up separatedly if\n\ttheir contents are different. The comment and field names\n\tof \"struct submodule_cache\" may want to be improved.\n\nWhen do we evict the cache? I am wondering what would happen when\nyou do \"git log --recursive\" at the superproject level, which may\ngrow the cache in an unbounded way without some eviction policy.\n"},{"id":"262910","messageId":"20150603193126.GA94875@book.hvoigt.net","threadId":"39497","inReplyTo":"xmqqbngx3muj.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH v4 1/4] implement submodule config cache for lookup of submodule names","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2015-06-03T19:31:26Z","receivedAt":"2015-06-03T19:31:26Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Tue, Jun 02, 2015 at 12:57:08PM -0700, Junio C Hamano wrote:\n> Heiko Voigt writes:\n> \n> > This submodule configuration cache allows us to lazily read .gitmodules\n> > configurations by commit into a runtime cache which can then be used to\n> > easily lookup values from it. Currently only the values for path or name\n> > are stored but it can be extended for any value needed.\n> >\n> > It is expected that .gitmodules files do not change often between\n> > commits. Thats why we lookup the .gitmodules sha1 from a commit and then\n> > either lookup an already parsed configuration or parse and cache an\n> > unknown one for each sha1. The cache is lazily build on demand for each\n> > requested commit.\n> >\n> > This cache can be used for all purposes which need knowledge about\n> > submodule configurations. Example use cases are:\n> >\n> > * Recursive submodule checkout needs lookup a submodule name from its\n> > path when a submodule first appears. This needs be done before this\n> > configuration exists in the worktree.\n> >\n> > * The implementation of submodule support for 'git archive' needs to\n> > lookup the submodule name to generate the archive when given a\n> > revision that is not checked out.\n> >\n> > * 'git fetch' when given the --recurse-submodules=on-demand option (or\n> > configuration) needs to lookup submodule names by path from the\n> > database rather than reading from the worktree. For new submodule it\n> > needs to lookup the name from its path to allow cloning new\n> > submodules into the .git folder so they can be checked out without\n> > any network interaction when the user does a checkout of that\n> > revision.\n> \n> What is unclear to me after reading the above twice is what this\n> thing is meant to achieve. Is it efficiency by doing lazy lookups\n> and caching to avoid asking the same thing more than once from\n> either the filesystem or read_sha1_file()? Is it expected that\n> reading through this \"cache\" will be the _only_ way callers would\n> interact with the .gitmodules data, or is it an opt-in feature that\n> some callers that do not see the benefit (why they may want to\n> ignore is totally unclear, because what the \"cache\" system wants to\n> achieve is) can safely ignore and bypass?\n\nMaybe I should add a paragraph like this on top:\n\n In a superproject some commands need to interact with submodules. They\n need to query values from the .gitmodules file either from the\n worktree of from certain revisions. At the moment this is quite hard\n since a caller would need to read the .gitmodules file from the\n history and then parse the values. We want to provide an API for this\n so we have one place to get values from .gitmodules from any revision\n (including the worktree).\n\nSo yes it will be the only way callers would read .gitmodules data.\nSince you abstractly wrote \"interact\" I realize that I have not thought\nabout writing values.\n\n> Because the above talks about looking up \".gitmodules from a\n> commit\", I am guessing that the \"commit\" used as one of the lookup\n> keys throughout the system is a commit in the superproject, not from\n> submodules, but you may want to state that more explicitly.\n\nYes. Well it is a commit in the current project which should be the\nsuperproject to other submodules. Assuming we have the additional\nparagraph above. Does that make it more clear?\n\n> What, if anything, should be done for .gitmodules that are not yet\n> committed? Are there cases that the callers that usually interact\n> with .gitmodules via this \"cache\" system need to use .gitmodules\n> immediately after adding a new submodule but before committing that\n> change to the superproject? Do they code something like this?\n> \n> \tif (cached)\n> \tread .gitmodules from the index and fabricate\n> \t\tstruct submodule;\n> \telse if (worktree)\n> \tread .gitmodules from the working tree and fabricate\n> \t\tstruct submodule;\n> \telse\n> \t\tcall submodule_from_name(\"HEAD\", ...) and receive\n> struct submodule;\n> \n> \tuse the struct submodule to learn from the module;\n> \n> Yes, I am wondering if submodule_from_name() should be extended to\n> allow the former two cases, so that the caller can make a single\n> call above and then use resulting \"struct submodule\" throughout its\n> code after doing so.\n\nReading from the worktree is supported by passing in null_sha1 (as\ndocumented). That said it may also contain values from the local\nconfiguration (overlaid in the typical priority).\n\nThe index case is the only one that is missing. I am not sure whether we\nwill need that. The use cases I have described above do not require this\n(AFAICS). But you are right maybe it makes sense to keep this open so we\ncan at least allow it. Since we already use the NULL pointer and the\nnull_sha1 as special values we could either add another function pair\nand a separate internal cache for them or define some special sha1 value\nthat is very unlikely to collide (e.g. an empty commit sha1 at epoch\nwith some special author).\n\nEither way I think there is nothing blocking us from extending it but I\nwould not do it now without any users.\n\nThe current use cases are all about interrogating an existing history\nand the worktree. I currently see no need to interrogate .gitmodules\ndirectly after adding a submodule in the same process. We also should be\nable to add it later in case needed.\n\n> And I also am hoping that the answer to my\n> questions above to be \"This is not just an opt-in 'cache' API, but\n> we want to make it the unified API for C code to learn about what is\n> in .gitmodule\".\n\nYes as stated above I am planning to make this the unified API for C\ncode. If you look at the next two commits you see that I am already\nreplacing the existing lookups from the worktree to use this cache.\nWith this series all places that read .gitmodules values use this cache.\nAnything else would be a bug in my code :)\n\n> > diff --git a/Documentation/technical/api-submodule-config.txt b/Documentation/technical/api-submodule-config.txt\n> > new file mode 100644\n> > index 0000000..2ff4907\n> > --- /dev/null\n> > +++ b/Documentation/technical/api-submodule-config.txt\n> > @@ -0,0 +1,46 @@\n> > +submodule config cache API\n> > +==========================\n> > +\n> > +The submodule config cache API allows to read submodule\n> > +configurations/information from specified revisions. Internally\n> > +information is lazily read into a cache that is used to avoid\n> > +unnecessary parsing of the same .gitmodule files. Lookups can be done by\n> > +submodule path or name.\n> > +\n> > +Usage\n> > +-----\n> > +\n> > +The caller can look up information about submodules by using the\n> > +`submodule_from_path()` or `submodule_from_name()` functions. They return\n> > +a `struct submodule` which contains the values. The API automatically\n> > +initializes and allocates the needed infrastructure on-demand.\n> > +\n> > +If the internal cache might grow too big or when the caller is done with\n> > +the API, all internally cached values can be freed with submodule_free().\n> > +\n> > +Data Structures\n> > +---------------\n> > +\n> > +`struct submodule`::\n> > +\n> > +\tThis structure is used to return the information about one\n> > +\tsubmodule for a certain revision. It is returned by the lookup\n> > +\tfunctions.\n> \n> Hopefully this will not stay an opaque structure as we read later\n> patches ;-).\n\nOk will add some documentation for the members here :)\n\n> > +Functions\n> > +---------\n> > +\n> > +`void submodule_free()`::\n> > +\n> > +\tUse these to free the internally cached values.\n> \n> \"These\" meaning \"this single function\", or are there variants of it?\n\nI think I had multiple free routines at some point. This seems like a\nleftover. Will change to \"this\".\n\n> > diff --git a/submodule-config.c b/submodule-config.c\n> > new file mode 100644\n> > index 0000000..97f4a04\n> > --- /dev/null\n> > +++ b/submodule-config.c\n> > @@ -0,0 +1,445 @@\n> > +#include \"cache.h\"\n> > +#include \"submodule-config.h\"\n> > +#include \"submodule.h\"\n> > +#include \"strbuf.h\"\n> > +\n> > +/*\n> > + * submodule cache lookup structure\n> > + * There is one shared set of 'struct submodule' entries which can be\n> > + * looked up by their sha1 blob id of the .gitmodule file and either\n> > + * using path or name as key.\n> > + * for_path stores submodule entries with path as key\n> > + * for_name stores submodule entries with name as key\n> > + */\n> > +struct submodule_cache {\n> > +\tstruct hashmap for_path;\n> > +\tstruct hashmap for_name;\n> > +};\n> > +\n> > +/*\n> > + * thin wrapper struct needed to insert 'struct submodule' entries to\n> > + * the hashmap\n> > + */\n> > +struct submodule_entry {\n> > +\tstruct hashmap_entry ent;\n> > +\tstruct submodule *config;\n> > +};\n> \n> The above, and the singleton-ness of the \"cache\", implies that we\n> can have only one \"struct submodule\" for a given path (or a name).\n> Does that mean the subsystem implicitly is tied to a single commit\n> at the superproject level?\n\nNo the keys in the hashmap are a combination of the sha1 blob id of the\n.gitmodules file and its name or path respectively. I should make that\nmore clear here.\n\n> What happens when I call submodule_from_path() for a single\n> submodule at one commit in the superproject, and then ask about that\n> same submodule for another commit in the superproject, which may\n> have a different version of .gitmodules, by calling the same\n> function again?\n> \n> \tSide note: I think I know the answer to these questions,\n> \tafter reading the hash function. for_path does not store\n> \tsubmodule entries with path as key. It uses the commit and\n> \tthe path as a combined key, so both HEAD:.gitmodules and\n> \tHEAD^:.gitmodules can be cached and looked up separatedly if\n> \ttheir contents are different. The comment and field names\n> \tof \"struct submodule_cache\" may want to be improved.\n\nThat is correct.\n\n> When do we evict the cache? I am wondering what would happen when\n> you do \"git log --recursive\" at the superproject level, which may\n> grow the cache in an unbounded way without some eviction policy.\n\nI left this as an exercise for the caller to figure out ;-) I think we\nhave to leave it to the caller how much of the cache he wants to use.\nThe only thing I can currently think of, that might help, is that we\ncould provide the caller with the amount of .gitmodules files read so he\nhas some simple measure to determine when he wants to call\nsubmodule_free().\n\nAll use-cases I currently have in mind are expected to use only a few\n.gitmodules. But only in the typical situation. There might be extreme\ncases were this is not true anymore. I am not sure how theoretical this\nis and thus not sure whether we should take countermeasures now.\n\nWhat would you expect \"git log --recursive\" to do?\n\nCheers Heiko\n"}]}