{"thread":{"id":"33386","subject":"[RFC/PATCH 0/7] Rework git core for native submodules","startedAt":"2013-04-04T18:30:36Z","lastAt":"2013-04-19T17:08:56Z","messageCount":138,"participants":["Ramkumar Ramachandra","Linus Torvalds","Jonathan Nieder","Junio C Hamano","Jens Lehmann","John Keeping","Duy Nguyen","Jeff King","Jakub Narębski","Philip Oakley"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"213144","messageId":"1365100243-13676-1-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":null,"subject":"[RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T18:30:36Z","receivedAt":"2013-04-04T18:30:36Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nThe purpose of this series is to convince you that we've made a lot of\nfundamental mistakes while designing submodules, and that we should\nfix them now. [1/7] argues for a new object type, and this is the\ncore of the idea.\n\nIt's an entire beautiful design + UI/UX package. To demo it now:\n\n # Switch your git to https://github.com/artagnon/git/tree/link\n\n $ git config --global url.\"git://github.com/\".insteadOf gh:\n $ git config --global clone.submodulegitdir /home//bare\n $ cd /tmp\n $ git clone gh:artagnon/varlog\n $ cd varlog\n $ git clone gh:artagnon/clayoven\n # Notice how it puts clayoven.git in ~/bare\n $ git add clayoven\n # Just works!\n $ git commit -a -m \"add subproject clayoven\"\n $ git ls-tree HEAD\n # Try to cat-file or show the link object\n $ cd clayoven/lib\n $ git clone gh:artagnon/sandbox\n $ git add sandbox\n # Again, just works! No cd-to-toplevel nonsense\n $ git ci -a -m \"add subproject sandbox\"\n $ cd ../..\n $ git rm clayoven\n $ git ci -a -m \"remove subproject clayoven\"\n # start hacking: read the rest of this email\n # note that git diff is broken\n\n>From a design perspective, the goal is to make possible various kinds\nof compositions, essentially replacing repo, mr, gitslave,\ngit-subtree, (the old) git-submodule, and other similar tools. All\nsubmdodules are not equal: each one will have tweakable parameters\nthat will change how git-core treats them.\n\n>From the UI/UX perspective, the goal is to get existing git commands\nto work seamlessly with submodules without introducing any\nsubmodule-specific commands (or atleast keeping it to a minimum). The\ndeprecation path for git-submodule is clear: first, we have to strip\nit down to be a very thin wrapper around existing git commands, and\nthen announce that it's no longer necessary.\n\nThis series is two days of unedited unrebased work ([5/7] is dead code\nfor instance). I've written in a big hurry, and it's meant to be a\nproof-of-concept only. I discovered lots of core bugs along the way\nthat need to be fixed first. Off the top of my head:\n\n1. 'git add' should not go past submodule boundaries. I should not be\n able to 'git add clayoven/' or 'git add clayoven/LICENSE'. In\n addition, the shell completion also needs to be fixed.\n\n2. An empty directory containing a .git file is a perfectly valid\n worktree, but does not show up in the superproject's 'git status'\n output. How can it be treated like an empty directory?\n\n3. sha1_file.c:index_path() should not return paths with trailing\n slashes, I think.\n\n4. There really must be a better way to figure out if I'm in a\n worktree than setup_git_directory_gently().\n\nAlso, I'm going to need your help to finish this. I was trying to\nwrite the 8th patch when I got stuck. I'm guessing I need to\nunderstand how wt-status extracts the differences between two trees,\nand filter it for added/removed link objects. Then, I have to follow\nthe example of updating the HEAD ref to update my\nrefs/modules//* refs.\n\nIf you think this is all a big waste of time, and that we should focus\non improving git-submodule.sh, you're probably deranged. Because it's\n_very_ clear to me.\n\nThank you for reading.\n\nRamkumar Ramachandra (7):\n link.c, link.h: introduce fifth object type\n sha1_file, link: write link objects to the database\n teach ce_compare_gitlink() about OBJ_LINK\n builtin/log: teach show about OBJ_LINK\n edit-link: add new builtin\n clone: introduce clone.submodulegitdir\n sha1_file: write ref_name to link object\n\n Makefile | 2 ++\n alloc.c | 3 ++\n builtin/clone.c | 29 +++++++++++++++++++\n builtin/log.c | 17 +++++++++++\n builtin/ls-tree.c | 4 +--\n cache.h | 3 +-\n environment.c | 11 -------\n git-edit-link.sh | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n link.c | 49 +++++++++++++++++++++++++++++++\n link.h | 26 +++++++++++++++++\n object.c | 9 ++++++\n read-cache.c | 33 ++++++++++++++-------\n sha1_file.c | 48 +++++++++++++++++++++++++++++-\n 13 files changed, 296 insertions(+), 25 deletions(-)\n create mode 100644 git-edit-link.sh\n create mode 100644 link.c\n create mode 100644 link.h\n\n-- \n1.8.2.380.g0d4e79b\n"},{"id":"213146","messageId":"1365100243-13676-2-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 1/7] link.c, link.h: introduce fifth object type","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T18:30:37Z","receivedAt":"2013-04-04T18:30:37Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Submodules suffer from one major design flaw: they are represented as\ncommit objects in the tree. There are several problems with this:\n\n1. Since the object is actually part of the submodule's object\n store (you can't even cat-file the commit), the superproject knows\n very little about the submodule. We currently work around this by\n having an ugly .gitmodules file in the toplevel directory mapping\n upstream URLs to submodule paths.\n\n2. We are restricted to having a concrete SHA-1 to convey what the\n subproject's HEAD should point to. As a result, it is impossible\n to have true floating submodules.\n\n3. It is impossible to initialize a nested submodule without\n initializing the containing submodule first. This is a consequence\n of our .gitmodules hack, and we should really fix it.\n\n4. Always stat'ing all subproject worktrees can be problematic if\n there are very large subprojects; we should be able to turn it off\n for some submodules selectively. A good future direction would be\n to add more submodule-specific properties, instead of assuming that\n all submodules are equal. Stuffing more properties into\n .gitmodules is really not a solution.\n\n5. Finally, the git-submodule shell script has lots of horrible warts\n (like cd-to-toplevel for any operation) that are non-trivial to\n fix, and introduces a very unnatural layer of abstraction.\n\nThere are various specialized tools like mr, repo, gitslave, and\ngit-subtree, which offer different compositions solving some problems\nwhile introducing limitations of their own.\n\nWe propose to fix the problem for good. More specifically, introduce\na new object type corresponding to mode 16000 (gitlink). This new\nobject will convey all the required information about the individual\nsubmodules to the superproject. Rework git core to get rid of\n.gitmodules and git-submodule altogether.\n\nThis patch doesn't do anything by itself: although it is possible to\ncreate link objects using 'git hash-object -t link',\nparse_link_buffer() is unimplemented. In future patches, we intend to\nflesh out how core git will handle this object.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n Makefile | 2 ++\n alloc.c | 3 +++\n cache.h | 3 ++-\n link.c | 27 +++++++++++++++++++++++++++\n link.h | 26 ++++++++++++++++++++++++++\n object.c | 9 +++++++++\n 6 files changed, 69 insertions(+), 1 deletion(-)\n create mode 100644 link.c\n create mode 100644 link.h\n\ndiff --git a/Makefile b/Makefile\nindex 0f931a2..cd4b6f9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -673,6 +673,7 @@ LIB_H += help.h\n LIB_H += http.h\n LIB_H += kwset.h\n LIB_H += levenshtein.h\n+LIB_H += link.h\n LIB_H += list-objects.h\n LIB_H += ll-merge.h\n LIB_H += log-tree.h\n@@ -801,6 +802,7 @@ LIB_OBJS += hex.o\n LIB_OBJS += ident.o\n LIB_OBJS += kwset.o\n LIB_OBJS += levenshtein.o\n+LIB_OBJS += link.o\n LIB_OBJS += list-objects.o\n LIB_OBJS += ll-merge.o\n LIB_OBJS += lockfile.o\ndiff --git a/alloc.c b/alloc.c\nindex aeae55c..1445879 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -15,6 +15,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"link.h\"\n \n #define BLOCKING 1024\n \n@@ -49,6 +50,7 @@ DEFINE_ALLOCATOR(blob, struct blob)\n DEFINE_ALLOCATOR(tree, struct tree)\n DEFINE_ALLOCATOR(commit, struct commit)\n DEFINE_ALLOCATOR(tag, struct tag)\n+DEFINE_ALLOCATOR(link, struct link)\n DEFINE_ALLOCATOR(object, union any_object)\n \n static void report(const char *name, unsigned int count, size_t size)\n@@ -66,4 +68,5 @@ void alloc_report(void)\n \tREPORT(tree);\n \tREPORT(commit);\n \tREPORT(tag);\n+\tREPORT(link);\n }\ndiff --git a/cache.h b/cache.h\nindex ec2fd7a..ca0583f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -317,7 +317,7 @@ enum object_type {\n \tOBJ_TREE = 2,\n \tOBJ_BLOB = 3,\n \tOBJ_TAG = 4,\n-\t/* 5 for future expansion */\n+\tOBJ_LINK = 5,\n \tOBJ_OFS_DELTA = 6,\n \tOBJ_REF_DELTA = 7,\n \tOBJ_ANY,\n@@ -1241,6 +1241,7 @@ extern void *alloc_blob_node(void);\n extern void *alloc_tree_node(void);\n extern void *alloc_commit_node(void);\n extern void *alloc_tag_node(void);\n+extern void *alloc_link_node(void);\n extern void *alloc_object_node(void);\n extern void alloc_report(void);\n \ndiff --git a/link.c b/link.c\nnew file mode 100644\nindex 0000000..bb20a51\n--- /dev/null\n+++ b/link.c\n@@ -0,0 +1,27 @@\n+#include \"cache.h\"\n+#include \"link.h\"\n+\n+const char *link_type = \"link\";\n+\n+struct link *lookup_link(const unsigned char *sha1)\n+{\n+\tstruct object *obj = lookup_object(sha1);\n+\tif (!obj)\n+\t\treturn create_object(sha1, OBJ_LINK, alloc_link_node());\n+\tif (!obj->type)\n+\t\tobj->type = OBJ_LINK;\n+\tif (obj->type != OBJ_LINK) {\n+\t\terror(\"Object %s is a %s, not a link\",\n+\t\t sha1_to_hex(sha1), typename(obj->type));\n+\t\treturn NULL;\n+\t}\n+\treturn (struct link *) obj;\n+}\n+\n+int parse_link_buffer(struct link *item, void *buffer, unsigned long size)\n+{\n+\tif (item->object.parsed)\n+\t\treturn 0;\n+\titem->object.parsed = 1;\n+\treturn 0;\n+}\ndiff --git a/link.h b/link.h\nnew file mode 100644\nindex 0000000..64dd19d\n--- /dev/null\n+++ b/link.h\n@@ -0,0 +1,26 @@\n+#ifndef LINK_H\n+#define LINK_H\n+\n+#include \"object.h\"\n+\n+extern const char *link_type;\n+\n+struct link {\n+\tstruct object object;\n+\tconst char *upstream_url;\n+\tconst char *checkout_rev;\n+\tconst char *ref_name;\n+\tunsigned int floating:1;\n+\tunsigned int statthrough:1;\n+};\n+\n+struct link *lookup_link(const unsigned char *sha1);\n+\n+int parse_link_buffer(struct link *item, void *buffer, unsigned long size);\n+\n+/**\n+ * Links do not contain references to other objects, but have\n+ * structured data that needs parsing.\n+ **/\n+\n+#endif /* LINK_H */\ndiff --git a/object.c b/object.c\nindex 20703f5..d3674ea 100644\n--- a/object.c\n+++ b/object.c\n@@ -4,6 +4,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"link.h\"\n \n static struct object **obj_hash;\n static int nr_objs, obj_hash_size;\n@@ -24,6 +25,7 @@ static const char *object_type_strings[] = {\n \t\"tree\",\t\t/* OBJ_TREE = 2 */\n \t\"blob\",\t\t/* OBJ_BLOB = 3 */\n \t\"tag\",\t\t/* OBJ_TAG = 4 */\n+\t\"link\",\t\t/* OBJ_LINK = 5 */\n };\n \n const char *typename(unsigned int type)\n@@ -175,6 +177,13 @@ struct object *parse_object_buffer(const unsigned char *sha1, enum object_type t\n \t\t\t return NULL;\n \t\t\tobj = &tag->object;\n \t\t}\n+\t} else if (type == OBJ_LINK) {\n+\t\tstruct link *link = lookup_link(sha1);\n+\t\tif (link) {\n+\t\t\tif (parse_link_buffer(link, buffer, size))\n+\t\t\t return NULL;\n+\t\t\tobj = &link->object;\n+\t\t}\n \t} else {\n \t\twarning(\"object %s has unknown type id %d\", sha1_to_hex(sha1), type);\n \t\tobj = NULL;\n-- \n1.8.2.380.g0d4e79b\n"},{"id":"213147","messageId":"1365100243-13676-3-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 2/7] sha1_file, link: write link objects to the database","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T18:30:38Z","receivedAt":"2013-04-04T18:30:38Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"On a 'git add', instead of returning the SHA-1 of the subproject\ncommit, write a real link object to the object database. Also\nimplement parse_link_buffer() correspondingly.\n\nindex_path() determines the upstream_url and checkout_rev from a\npre-cloned submodule. The checkout_rev is set to the SHA-1 of the\nHEAD, and we get a non-floating submodule.\n\nWhile at it, fix the 'ls-tree' output to correctly show a link object.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n builtin/ls-tree.c | 4 ++--\n link.c | 22 ++++++++++++++++++++++\n sha1_file.c | 38 +++++++++++++++++++++++++++++++++++++-\n 3 files changed, 61 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/ls-tree.c b/builtin/ls-tree.c\nindex fb76e38..ab17fb5 100644\n--- a/builtin/ls-tree.c\n+++ b/builtin/ls-tree.c\n@@ -6,7 +6,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"tree.h\"\n-#include \"commit.h\"\n+#include \"link.h\"\n #include \"quote.h\"\n #include \"builtin.h\"\n #include \"parse-options.h\"\n@@ -76,7 +76,7 @@ static int show_tree(const unsigned char *sha1, const char *base, int baselen,\n \t\t\tretval = READ_TREE_RECURSIVE;\n \t\t *\n \t\t */\n-\t\ttype = commit_type;\n+\t\ttype = link_type;\n \t} else if (S_ISDIR(mode)) {\n \t\tif (show_recursive(base, baselen, pathname)) {\n \t\t\tretval = READ_TREE_RECURSIVE;\ndiff --git a/link.c b/link.c\nindex bb20a51..349646d 100644\n--- a/link.c\n+++ b/link.c\n@@ -20,8 +20,30 @@ struct link *lookup_link(const unsigned char *sha1)\n \n int parse_link_buffer(struct link *item, void *buffer, unsigned long size)\n {\n+\tchar *bufptr = buffer;\n+\tchar *tail = buffer + size;\n+\tchar *eol;\n+\n \tif (item->object.parsed)\n \t\treturn 0;\n \titem->object.parsed = 1;\n+\twhile (bufptr < tail) {\n+\t\teol = strchr(bufptr, '\\n');\n+\t\t*eol = '\\0';\n+\t\tif (!prefixcmp(bufptr, \"upstream_url = \"))\n+\t\t\titem->upstream_url = xstrdup(bufptr + 15);\n+\t\telse if (!prefixcmp(bufptr, \"checkout_rev = \"))\n+\t\t\titem->checkout_rev = xstrdup(bufptr + 15);\n+\t\telse if (!prefixcmp(bufptr, \"ref_name = \"))\n+\t\t\titem->ref_name = xstrdup(bufptr + 11);\n+\t\telse if (!prefixcmp(bufptr, \"floating = \"))\n+\t\t\titem->floating = atoi(bufptr + 11);\n+\t\telse if (!prefixcmp(bufptr, \"statthrough = \"))\n+\t\t\titem->statthrough = atoi(bufptr + 14);\n+\t\telse\n+\t\t\treturn error(\"Parse error in link buffer\");\n+\n+\t\tbufptr = eol + 1;\n+\t}\n \treturn 0;\n }\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 5f573d9..a8a6d72 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -12,6 +12,7 @@\n #include \"pack.h\"\n #include \"blob.h\"\n #include \"commit.h\"\n+#include \"link.h\"\n #include \"run-command.h\"\n #include \"tag.h\"\n #include \"tree.h\"\n@@ -35,6 +36,7 @@\n static inline uintmax_t sz_fmt(size_t s) { return s; }\n \n const unsigned char null_sha1[20];\n+void *upstream_url = NULL;\n \n /*\n * This is meant to hold a *small* number of objects that you would\n@@ -2859,10 +2861,19 @@ int index_fd(unsigned char *sha1, int fd, struct stat *st,\n \treturn ret;\n }\n \n+static int parse_origin_url(const char *key, const char *value, void *cb) {\n+\tif (!strcmp(key, \"remote.origin.url\"))\n+\t\tupstream_url = xstrdup(value);\n+\treturn 0;\n+}\n+\n int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned flags)\n {\n \tint fd;\n \tstruct strbuf sb = STRBUF_INIT;\n+\tchar pathbuf[PATH_MAX];\n+\tconst char *submodule_gitdir;\n+\tunsigned char checkout_rev[20];\n \n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFREG:\n@@ -2888,7 +2899,32 @@ int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned\n \t\tstrbuf_release(&sb);\n \t\tbreak;\n \tcase S_IFDIR:\n-\t\treturn resolve_gitlink_ref(path, \"HEAD\", sha1);\n+\t\t/* gitlink. Prepare and write a new link object to\n+\t\t * the database.\n+\t\t */\n+\n+\t\t/* Figure out upstream_url */\n+\t\tsprintf(pathbuf, \"%s/%s\", path, \".git\");\n+\t\tsubmodule_gitdir = resolve_gitdir(pathbuf);\n+\t\tsprintf(pathbuf, \"%s/%s\", submodule_gitdir, \"config\");\n+\t\tgit_config_from_file(parse_origin_url, pathbuf, NULL);\n+\t\tif (!upstream_url)\n+\t\t\tdie(\"Unable to read remote.origin.url from submodule\");\n+\n+\t\t/* Figure out checkout_rev */\n+\t\tif (resolve_gitlink_ref(path, \"HEAD\", checkout_rev) < 0)\n+\t\t\tdie(\"Unable to resolve submodule HEAD\");\n+\n+\t\t/* Add fields to the strbuf */\n+\t\tstrbuf_addf(&sb, \"upstream_url = %s\\n\", (char *) upstream_url);\n+\t\tstrbuf_addf(&sb, \"checkout_rev = %s\\n\", sha1_to_hex(checkout_rev));\n+\t\tif (!(flags & HASH_WRITE_OBJECT))\n+\t\t\thash_sha1_file(sb.buf, sb.len, link_type, sha1);\n+\t\telse if (write_sha1_file(sb.buf, sb.len, link_type, sha1))\n+\t\t\treturn error(\"%s: failed to insert into database\",\n+\t\t\t\t path);\n+\t\tstrbuf_release(&sb);\n+\t\tbreak;\n \tdefault:\n \t\treturn error(\"%s: unsupported file type\", path);\n \t}\n-- \n1.8.2.380.g0d4e79b\n"},{"id":"213145","messageId":"1365100243-13676-4-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 3/7] teach ce_compare_gitlink() about OBJ_LINK","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T18:30:39Z","receivedAt":"2013-04-04T18:30:39Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"This simply requires parsing out the checkout_rev from the link\nobject, and comparing its SHA-1 with that of HEAD.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n read-cache.c | 33 +++++++++++++++++++++++----------\n 1 file changed, 23 insertions(+), 10 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 5a9704f..f22c1c0 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -11,6 +11,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"blob.h\"\n+#include \"link.h\"\n #include \"resolve-undo.h\"\n #include \"strbuf.h\"\n #include \"varint.h\"\n@@ -128,19 +129,31 @@ static int ce_compare_link(struct cache_entry *ce, size_t expected_size)\n \n static int ce_compare_gitlink(struct cache_entry *ce)\n {\n-\tunsigned char sha1[20];\n+\tunsigned char checkout_rev_sha1[20], head_sha1[20];\n+\tvoid *buffer;\n+\tunsigned long size;\n+\tenum object_type type;\n+\tstruct link link;\n \n-\t/*\n-\t * We don't actually require that the .git directory\n-\t * under GITLINK directory be a valid git directory. It\n-\t * might even be missing (in case nobody populated that\n-\t * sub-project).\n-\t *\n-\t * If so, we consider it always to match.\n+\tbuffer = read_sha1_file(ce->sha1, &type, &size);\n+\n+\t/* For compatibility with an older version: earlier, gitlinks\n+\t * were represented as commit SHA-1s (that wouldn't resolve)\n+\t * in the cache.\n \t */\n-\tif (resolve_gitlink_ref(ce->name, \"HEAD\", sha1) < 0)\n+\tif (!buffer) {\n+\t\tif (resolve_gitlink_ref(ce->name, \"HEAD\", head_sha1) < 0)\n+\t\t\treturn 0;\n+\t\treturn hashcmp(head_sha1, ce->sha1);\n+\t}\n+\n+\tmemset(&link, 0, sizeof(struct link));\n+\tif (parse_link_buffer(&link, buffer, size) < 0)\n+\t\tdie(\"Cannot continue.\");\n+\tif (resolve_gitlink_ref(ce->name, link.checkout_rev, checkout_rev_sha1) < 0 ||\n+\t\tresolve_gitlink_ref(ce->name, \"HEAD\", head_sha1) < 0)\n \t\treturn 0;\n-\treturn hashcmp(sha1, ce->sha1);\n+\treturn hashcmp(head_sha1, checkout_rev_sha1);\n }\n \n static int ce_modified_check_fs(struct cache_entry *ce, struct stat *st)\n-- \n1.8.2.380.g0d4e79b\n"},{"id":"213148","messageId":"1365100243-13676-5-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 4/7] builtin/log: teach show about OBJ_LINK","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T18:30:40Z","receivedAt":"2013-04-04T18:30:40Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"'git show' now works with link objects.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n builtin/log.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/log.c b/builtin/log.c\nindex 0f31810..a170df9 100644\n--- a/builtin/log.c\n+++ b/builtin/log.c\n@@ -411,6 +411,20 @@ static int show_blob_object(const unsigned char *sha1, struct rev_info *rev)\n \treturn stream_blob_to_fd(1, sha1, NULL, 0);\n }\n \n+static int show_link_object(const unsigned char *sha1, struct rev_info *rev)\n+{\n+\tunsigned long size;\n+\tenum object_type type;\n+\tchar *buf = read_sha1_file(sha1, &type, &size);\n+\n+\tif (!buf)\n+\t\treturn error(_(\"Could not read object %s\"), sha1_to_hex(sha1));\n+\n+\tassert(type == OBJ_LINK);\n+\tprintf(\"%s\", buf);\n+\treturn 0;\n+}\n+\n static int show_tag_object(const unsigned char *sha1, struct rev_info *rev)\n {\n \tunsigned long size;\n@@ -534,6 +548,9 @@ int cmd_show(int argc, const char **argv, const char *prefix)\n \t\t\tadd_object_array(o, name, &rev.pending);\n \t\t\tret = cmd_log_walk(&rev);\n \t\t\tbreak;\n+\t\tcase OBJ_LINK:\n+\t\t\tret = show_link_object(o->sha1, NULL);\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\tret = error(_(\"Unknown type: %d\"), o->type);\n \t\t}\n-- \n1.8.2.380.g0d4e79b\n"},{"id":"213149","messageId":"1365100243-13676-6-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 5/7] edit-link: add new builtin","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T18:30:41Z","receivedAt":"2013-04-04T18:30:41Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"This is a WIP.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n git-edit-link.sh | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 87 insertions(+)\n create mode 100644 git-edit-link.sh\n\ndiff --git a/git-edit-link.sh b/git-edit-link.sh\nnew file mode 100644\nindex 0000000..3ff0e84\n--- /dev/null\n+++ b/git-edit-link.sh\n@@ -0,0 +1,87 @@\n+#!/bin/sh\n+# Copyright (c) 2013 Ramkumar Ramachandra\n+\n+dashless=$(basename \"$0\" | sed -e 's/-/ /')\n+USAGE='[--floating] [--ref-name ] [--checkout-rev ] [--statthrough] '\n+\n+SUBDIRECTORY_OK=Yes\n+OPTIONS_SPEC=\n+START_DIR=`pwd`\n+. git-sh-setup\n+. git-sh-i18n\n+require_work_tree\n+cd_to_toplevel\n+\n+link_spec=\"$GIT_DIR/LINK_SPEC\"\n+\n+read_and_verify_link_spec () {\n+\ttest -f \"$link_spec\" || die \"fatal: could not open $link_spec.\"\n+\n+\t## TODO\n+\t## Rules:\n+\t# upstream_url is a mandatory field; others are optional\n+\t# if floating is false, checkout_rev has to be a SHA-1 hex\n+\t# ref_name must be a valid name, and not conflict with an existing ref\n+\n+\treturn 0;\n+}\n+\n+# Prepare an initial link_spec using the command-line options\n+rm -f \"$link_spec\"\n+touch \"$link_spec\" || die \"fatal: could not create $link_spec.\"\n+\n+total_argc=$#\n+while test $# != 0\n+do\n+\tcase \"$1\" in\n+\t\t--checkout-rev)\n+\t\t\tshift\n+\t\t\tcat >>\"$link_spec\" <<-EOF\n+\t\t\tcheckout_ref = $1\n+\t\t\tEOF\n+\t\t\t;;\n+\t\t--ref-name)\n+\t\t\tshift\n+\t\t\tcat >>\"$link_spec\" <<-EOF\n+\t\t\tref_name = $1\n+\t\t\tEOF\n+\t\t\t;;\n+\t\t--floating)\n+\t\t\tcat >>\"$link_spec\" <<-\\EOF\n+\t\t\tfloating = 1\n+\t\t\tEOF\n+\t\t\t;;\n+\t\t--statthrough)\n+\t\t\tcat >>\"$link_spec\" <<-\\EOF\n+\t\t\tstatthrough = 1\n+\t\t\tEOF\n+\t\t\t;;\n+\t\t--)\n+\t\t\tshift\n+\t\t\tbreak\n+\t\t\t;;\n+\tesac\n+\tshift\n+done\n+\n+link_directory=\"$1\"\n+test -d \"$link_directory\" || die \"fatal: $link_directory is not a valid directory.\"\n+\n+cd \"$link_directory\" && {\n+\ttest -f \"$link_directory/.git\" &&\n+\ttest \"$(git rev-parse --is-inside-work-tree 2>/dev/null)\" = true ||\n+\tdie \"fatal: $link_directory is not a valid bare working tree.\"\n+\n+\t# Determine the upstream_url\n+\tupstream_url=$(git config --get remote.origin.url)\n+}\n+\n+test -z $upstream_url &&\n+die \"fatal: $link_directory does not have a configured upstream remote origin.\"\n+cat >>\"$link_spec\" <<-\\EOF\n+upstream_url = $upstream_url\n+EOF\n+\n+# Launch the editor\n+git_editor \"$link_spec\" || die \"$(gettext \"Could not execute editor\")\"\n+read_and_verify_link_spec\n-- \n1.8.2.380.g0d4e79b\n"},{"id":"213150","messageId":"1365100243-13676-7-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 6/7] clone: introduce clone.submodulegitdir","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T18:30:42Z","receivedAt":"2013-04-04T18:30:42Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"This configuration variable comes into effect when 'git clone' is\ninvoked in an existing git repository. Instead of cloning the\nrepository as-is, it relocates the gitdir of the repository to the\npath specified by this variable. Arguably, it does the right thing\nwhen working with submodules.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n builtin/clone.c | 29 +++++++++++++++++++++++++++++\n environment.c | 11 -----------\n 2 files changed, 29 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex e0aaf13..1b798e6 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -43,6 +43,7 @@ static char *option_template, *option_depth;\n static char *option_origin = NULL;\n static char *option_branch = NULL;\n static const char *real_git_dir;\n+static const char *submodule_gitdir;\n static char *option_upload_pack = \"git-upload-pack\";\n static int option_verbosity;\n static int option_progress = -1;\n@@ -658,11 +659,22 @@ static void write_refspec_config(const char* src_ref_prefix,\n \tstrbuf_release(&value);\n }\n \n+static int git_clone_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, \"clone.submodulegitdir\")) {\n+\t\tgit_config_string(&submodule_gitdir, var, value);\n+\t\treturn 0;\n+\t}\n+\treturn git_default_config(var, value, cb);\n+}\n+\n int cmd_clone(int argc, const char **argv, const char *prefix)\n {\n \tint is_bundle = 0, is_local;\n \tstruct stat buf;\n \tconst char *repo_name, *repo, *work_tree, *git_dir;\n+\tchar dest_git_dir[PATH_MAX];\n+\tchar cwd[PATH_MAX];\n \tchar *path, *dir;\n \tint dest_exists;\n \tconst struct ref *refs, *remote_head;\n@@ -676,6 +688,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tconst char *src_ref_prefix = \"refs/heads/\";\n \tstruct remote *remote;\n \tint err = 0, complete_refs_before_fetch = 1;\n+\tint nongit = 1;\n \n \tstruct refspec *refspec;\n \tconst char *fetch_pattern;\n@@ -683,6 +696,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tjunk_pid = getpid();\n \n \tpacket_trace_identity(\"clone\");\n+\n+\t/* setup_git_directory_gently without changing directories */\n+\tgetcwd(cwd, sizeof(cwd) - 1);\n+\tsetup_git_directory_gently(&nongit);\n+\tchdir(cwd);\n+\n+\tgit_config(git_clone_config, NULL);\n+\n \targc = parse_options(argc, argv, prefix, builtin_clone_options,\n \t\t\t builtin_clone_usage, 0);\n \n@@ -736,6 +757,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\tdie(_(\"destination path '%s' already exists and is not \"\n \t\t\t\"an empty directory.\"), dir);\n \n+\tif (!nongit && submodule_gitdir) {\n+\t\tsprintf(dest_git_dir, \"%s/%s.git\", real_path(submodule_gitdir), dir);\n+\t\tif (!stat(dest_git_dir, &buf) && !is_empty_dir(dest_git_dir))\n+\t\t\tdie(_(\"destination path '%s' already exists and is not \"\n+\t\t\t\t\t\"an empty directory.\"), dest_git_dir);\n+\t\treal_git_dir = dest_git_dir;\n+\t}\n+\n \tstrbuf_addf(&reflog_msg, \"clone: from %s\", repo);\n \n \tif (option_bare)\ndiff --git a/environment.c b/environment.c\nindex e2e75c1..9dce4c7 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -182,8 +182,6 @@ const char *strip_namespace(const char *namespaced_ref)\n \treturn namespaced_ref + namespace_len;\n }\n \n-static int git_work_tree_initialized;\n-\n /*\n * Note. This works only before you used a work tree. This was added\n * primarily to support git-clone to work in a new repository it just\n@@ -191,15 +189,6 @@ static int git_work_tree_initialized;\n */\n void set_git_work_tree(const char *new_work_tree)\n {\n-\tif (git_work_tree_initialized) {\n-\t\tnew_work_tree = real_path(new_work_tree);\n-\t\tif (strcmp(new_work_tree, work_tree))\n-\t\t\tdie(\"internal error: work tree has already been set\\n\"\n-\t\t\t \"Current worktree: %s\\nNew worktree: %s\",\n-\t\t\t work_tree, new_work_tree);\n-\t\treturn;\n-\t}\n-\tgit_work_tree_initialized = 1;\n \twork_tree = xstrdup(real_path(new_work_tree));\n }\n \n-- \n1.8.2.380.g0d4e79b\n"},{"id":"213151","messageId":"1365100243-13676-8-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 7/7] sha1_file: write ref_name to link object","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T18:30:43Z","receivedAt":"2013-04-04T18:30:43Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Great. Now, we just have to write refs/modules//* at\ncommit-time.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n sha1_file.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex a8a6d72..2ea101a 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2874,6 +2874,7 @@ int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned\n \tchar pathbuf[PATH_MAX];\n \tconst char *submodule_gitdir;\n \tunsigned char checkout_rev[20];\n+\tchar *ref_name;\n \n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFREG:\n@@ -2915,9 +2916,18 @@ int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned\n \t\tif (resolve_gitlink_ref(path, \"HEAD\", checkout_rev) < 0)\n \t\t\tdie(\"Unable to resolve submodule HEAD\");\n \n+\t\t/* Construct a ref_name from path */\n+\t\tsprintf(pathbuf, \"%s\", path);\n+\t\tpathbuf[strlen(pathbuf) - 1] = '\\0'; /* Remove trailing slash */\n+\t\tif (strchr(pathbuf, '/'))\n+\t\t\tref_name = xstrdup(strrchr(pathbuf, '/') + 1);\n+\t\telse\n+\t\t\tref_name = xstrdup(pathbuf);\n+\n \t\t/* Add fields to the strbuf */\n \t\tstrbuf_addf(&sb, \"upstream_url = %s\\n\", (char *) upstream_url);\n \t\tstrbuf_addf(&sb, \"checkout_rev = %s\\n\", sha1_to_hex(checkout_rev));\n+\t\tstrbuf_addf(&sb, \"ref_name = %s\\n\", ref_name);\n \t\tif (!(flags & HASH_WRITE_OBJECT))\n \t\t\thash_sha1_file(sb.buf, sb.len, link_type, sha1);\n \t\telse if (write_sha1_file(sb.buf, sb.len, link_type, sha1))\n-- \n1.8.2.380.g0d4e79b\n"},{"id":"213154","messageId":"CA+55aFz1D_dMtMHHMpiGi3KL=Y-m4DVxHVr=1ZX8zYWQ2TPvwA@mail.gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2013-04-04T18:40:32Z","receivedAt":"2013-04-04T18:40:32Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Thu, Apr 4, 2013 at 11:30 AM, Ramkumar Ramachandra\n wrote:\n>\n> The purpose of this series is to convince you that we've made a lot of\n> fundamental mistakes while designing submodules, and that we should\n> fix them now. [1/7] argues for a new object type, and this is the\n> core of the idea.\n\nI don't dispute that a new link object might be a good idea, but\nthere's no explanation of the actual format of this thing anywhere,\nand what the real advantages would be. A clearer \"this is the design,\nthis is the format of the link object, and this is what it buys us\"\nwould be a good idea. Also, one of the arguments against using link\nobjects originally was that the format wasn't stable, and in\nparticular the address of the actual submodule repository might differ\nfor different people. So when adding a new object type, explaining\n*why* the format of such an object is globally stable (since it will\nbe part of the SHA1 of the object) is a big deal.\n\n Linus\n"},{"id":"213157","messageId":"20130404184730.GL30308@google.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-04-04T18:47:30Z","receivedAt":"2013-04-04T18:47:30Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n\n> The purpose of this series is to convince you that we've made a lot of\n> fundamental mistakes while designing submodules, and that we should\n> fix them now. [1/7] argues for a new object type, and this is the\n> core of the idea.\n\nOh, dear.\n\nShouldn't it be possible to explain the same thing using a test\nscript illustrating intended UI?\n\n[...]\n> $ git clone gh:artagnon/varlog\n> $ cd varlog\n> $ git clone gh:artagnon/clayoven\n> # Notice how it puts clayoven.git in ~/bare\n\nI really would like to be able to continue doing something like\n\n\tgit clone --recurse-submodules git://repo.or.cz/cgit.git\n\t# never mind!\n\trm -fr cgit\n\nwithout leaving any clutter behind. I have used systems that kept\nstate in my home directory before and found them a pain in the neck to\ndebug. Others may disagree, though.\n\n[...]\n> # Again, just works! No cd-to-toplevel nonsense\n\nDidn't Jens mention that git-submodule requiring that one work\nat the toplevel is just a (presumably easily fixable) bug?\n\n[...]\n> If you think this is all a big waste of time, and that we should focus\n> on improving git-submodule.sh, you're probably deranged. Because it's\n\nI don't think that *you* should focus on improving git-submodule, as\nlong as you are not using it and dislike its design. But I do think\nit's strange to at the same time\n\n 1) tell me I'm deranged for liking submodules\n 2) dismiss other experiments that have been created as alternatives\n\nI like experimentation, which means sometimes having tools whose\npurposes overlap, and I like when it's possible to help something\nevolve to be better, even far enough to interoperate with or replace\nuses of another tool.\n\nI also believe in \"live and let live\". That means that even if\nsomeone is a little crazy, if they are not actively harmful, I do not\ndestroy their tools.\n\nThat probably marks me as deranged.\n\nHope that helps,\nJonathan\n"},{"id":"213158","messageId":"CALkWK0nNjvV5VGvT_eaubFoOhMnJ-N8FECAayd5A2K3BzeRh6Q@mail.gmail.com","threadId":"33386","inReplyTo":"CA+55aFz1D_dMtMHHMpiGi3KL=Y-m4DVxHVr=1ZX8zYWQ2TPvwA@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T18:52:37Z","receivedAt":"2013-04-04T18:52:37Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Linus Torvalds wrote:\n> I don't dispute that a new link object might be a good idea, but\n> there's no explanation of the actual format of this thing anywhere,\n> and what the real advantages would be. A clearer \"this is the design,\n> this is the format of the link object, and this is what it buys us\"\n> would be a good idea.\n\nYeah, I need help with that. I've just stuffed in whatever fields\npopped into my mind first. The current ones are:\n\n1. upstream_url: this records the upstream URL. No need to keep a .gitmodules.\n\n2. checkout_rev: this records the ref to check out the submodule to.\nAs opposed to a concrete SHA-1, this allows for more flexibility; you\ncan put refs/heads/master and have truly floating submodules.\n\n3. ref_name: this specifies what name the ref under\nrefs/modules// should use.\n\n4. floating: this bit specifies whether to record a concrete SHA-1 in\ncheckout_rev.\n\n5. statthrough: this bit specifies whether git should stat() through\nthe worktree. We can turn it off on big repositories for performance\nreasons.\n\n> Also, one of the arguments against using link\n> objects originally was that the format wasn't stable, and in\n> particular the address of the actual submodule repository might differ\n> for different people. So when adding a new object type, explaining\n> *why* the format of such an object is globally stable (since it will\n> be part of the SHA1 of the object) is a big deal.\n\nAfter some discussion, I hope to be able to finalize a list of fields\nthat will suffice for (nearly) everything.\n"},{"id":"213159","messageId":"20130404185522.GM30308@google.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-04-04T18:55:22Z","receivedAt":"2013-04-04T18:55:22Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n\n> 1. 'git add' should not go past submodule boundaries. I should not be\n> able to 'git add clayoven/' or 'git add clayoven/LICENSE'. In\n> addition, the shell completion also needs to be fixed.\n\nYep. This is a bug.\n\n> 2. An empty directory containing a .git file is a perfectly valid\n> worktree, but does not show up in the superproject's 'git status'\n> output. How can it be treated like an empty directory?\n\nStated like that, it doesn't sound like a bug. Git since very early\nhas deliberately not tracked files or directories named .git.\n\nDo you need this as a way of importing from a foreign VCS when someone\nhas accidentally checked in a .git directory along with everything\nelse?\n\nThanks,\nJonathan\n"},{"id":"213161","messageId":"20130404185839.GN30308@google.com","threadId":"33386","inReplyTo":"20130404184730.GL30308@google.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-04-04T18:58:39Z","receivedAt":"2013-04-04T18:58:39Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n> Ramkumar Ramachandra wrote:\n\n>> The purpose of this series is to convince you that we've made a lot of\n>> fundamental mistakes while designing submodules, and\n[...]\n> Shouldn't it be possible to explain the same thing using a test\n> script illustrating intended UI?\n\nSorry, I sent this reply too quickly. Your explanation to Linus\nclarified the idea.\n\nRegards,\nJonathan\n"},{"id":"213166","messageId":"CA+55aFyQwJfiYo06y1bRNpKT6wOquhG9a9M_4YvLG_UT3b34-w@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0nNjvV5VGvT_eaubFoOhMnJ-N8FECAayd5A2K3BzeRh6Q@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2013-04-04T19:04:05Z","receivedAt":"2013-04-04T19:04:05Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Thu, Apr 4, 2013 at 11:52 AM, Ramkumar Ramachandra\n wrote:\n>\n> 1. upstream_url: this records the upstream URL. No need to keep a .gitmodules.\n>\n> 2. checkout_rev: this records the ref to check out the submodule to.\n> As opposed to a concrete SHA-1, this allows for more flexibility; you\n> can put refs/heads/master and have truly floating submodules.\n>\n> 3. ref_name: this specifies what name the ref under\n> refs/modules// should use.\n>\n> 4. floating: this bit specifies whether to record a concrete SHA-1 in\n> checkout_rev.\n>\n> 5. statthrough: this bit specifies whether git should stat() through\n> the worktree. We can turn it off on big repositories for performance\n> reasons.\n\nSo the thing is (and this was pretty much the original basis for\n.gitmodules) that pretty much *all* of the above fields are quite\npossibly site-specific, rather than globally stable.\n\nSo I actually conceptually like (and liked) the notion of a link\nobject, but I just don't think it is necessarily practically useful,\nexactly because different installations of the *same* supermodule\nmight well want to have different setups wrt these submodule fields.\n\nMy gut feel is that yes, .gitmodules was always a bit of a hack, but\nit's a *working* hack, and it does have advantages exactly because\nit's more fluid than an actual git object (which by definition has to\nbe set 100% in stone). If there are things you feel it does wrong\n(like the \"git add\" bug that is being discussed elsewhere), I wonder\nif it's not best to at least try to fix/extend them in the current\nmodel. The features you seem to be after (ie that whole\nfloating/refname thing) don't seem fundamentally antithetical to the\ncurrent model (a \"commit\" SHA1 of all zeroes for floating, with a new\nrefname field in .submodules? I dunno)..\n\n Linus\n"},{"id":"213170","messageId":"7va9pehzql.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CA+55aFyQwJfiYo06y1bRNpKT6wOquhG9a9M_4YvLG_UT3b34-w@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-04T19:17:38Z","receivedAt":"2013-04-04T19:17:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> ... The features you seem to be after (ie that whole\n> floating/refname thing) don't seem fundamentally antithetical to the\n> current model (a \"commit\" SHA1 of all zeroes for floating, with a new\n> refname field in .submodules? I dunno)..\n\nJust on this part.\n\nI think Heiko and Jens's (by the way, why aren't they on the Cc:\nlist when this topic is clearly discussing submodules? Don't we\nwant to learn how the current submodule subsystem is used to solve\nwhat real-world problems?) .gitmodules updates is exactly going in\nthat direction.\n\n - A submodule can be marked as floating in .gitmodules and be\n specified how (typially, \"use the tip of this branch in the\n submodule\");\n\n - Running \"submodule update\" a floating submodule does not detach\n the submodule working tree to commit in the index of the\n superproject; instead it will use the specified branch tip;\n\n - A floating submodule records a concrete commit object name in the\n index of the superproject (no need to stuff an unusual SHA-1\n there to signal that the submodule is floating---it is recorded\n in the .gitmodules). Thanks to this, a release out of the\n top-level can still describe the state of the entire tree;\n\n - It would be normal for the commit recorded in the index of the\n superproject not to match what is checked out in the submodule\n working tree (i.e. the tip of the branch in the submodule may\n have advanced). A traditional non-floating submodule has many\n mechanisms to be noisy about this situation to prevent users from\n making an incomplete commits, but they may have to be toned down\n or squelched for floating submodules.\n\nAnything I missed, Jens, Heiko?\n"},{"id":"213172","messageId":"CALkWK0mQt7cE0zUECDF4ZC2-9Q+pEL=XGPdaOksyWzae2_W1CA@mail.gmail.com","threadId":"33386","inReplyTo":"CA+55aFyQwJfiYo06y1bRNpKT6wOquhG9a9M_4YvLG_UT3b34-w@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T19:36:40Z","receivedAt":"2013-04-04T19:36:40Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Linus Torvalds wrote:\n> So the thing is (and this was pretty much the original basis for\n> .gitmodules) that pretty much *all* of the above fields are quite\n> possibly site-specific, rather than globally stable.\n>\n> So I actually conceptually like (and liked) the notion of a link\n> object, but I just don't think it is necessarily practically useful,\n> exactly because different installations of the *same* supermodule\n> might well want to have different setups wrt these submodule fields.\n>\n> My gut feel is that yes, .gitmodules was always a bit of a hack, but\n> it's a *working* hack, and it does have advantages exactly because\n> it's more fluid than an actual git object (which by definition has to\n> be set 100% in stone). If there are things you feel it does wrong\n> (like the \"git add\" bug that is being discussed elsewhere), I wonder\n> if it's not best to at least try to fix/extend them in the current\n> model. The features you seem to be after (ie that whole\n> floating/refname thing) don't seem fundamentally antithetical to the\n> current model (a \"commit\" SHA1 of all zeroes for floating, with a new\n> refname field in .submodules? I dunno)..\n\nLet's compare the two alternatives: .gitmodules versus link object.\nIf I want my fork of .gitmodules, I create a commit on top. If I want\nmy fork of the link object, I create a link object, plus tree object,\nplus commit object on top of that. But the commit still rebases fine.\n\nOn malleability, have you looked at [5/7], where I create edit-link\n(dead code; half done)? The buffer looks just like a .gitmodules\nbuffer. Fundamentally, what is the difference between this and a\nblob? git-core can parse it into structured data that it can slurp\neasily.\n\nI don't want full float or nothing. I want in-betweens too, and refs are great.\n"},{"id":"213173","messageId":"CALkWK0moEg1iFd5HUQq3-P2ZV+xf98vL1+WEotNHEpL3egSLCA@mail.gmail.com","threadId":"33386","inReplyTo":"CA+55aFyQwJfiYo06y1bRNpKT6wOquhG9a9M_4YvLG_UT3b34-w@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T19:42:10Z","receivedAt":"2013-04-04T19:42:10Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Linus Torvalds wrote:\n> .... don't seem fundamentally antithetical to the\n> current model\n\nI don't think it's fundamentally antithetical either. This basically\nmakes the life of git-submodule much simpler, and eventually obsolete\nit away completely.\n"},{"id":"213175","messageId":"CA+55aFw+2wgUufJn7BcXACBQqywAGBAcZWS6N_UV8UA91G447Q@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0mQt7cE0zUECDF4ZC2-9Q+pEL=XGPdaOksyWzae2_W1CA@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2013-04-04T19:44:22Z","receivedAt":"2013-04-04T19:44:22Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Thu, Apr 4, 2013 at 12:36 PM, Ramkumar Ramachandra\n wrote:\n>\n> Let's compare the two alternatives: .gitmodules versus link object.\n> If I want my fork of .gitmodules, I create a commit on top.\n\nOr you could also just edit and carry a dirty .gitmodules around for\nyour personal use-case.\n\nI don't know if anybody does that, but it should work fine.\n\nAnd I don't see what you can do with the link objects that you cannot\ndo with .gitmodules. That's what it really boils down to. .gitmodules\ndo actually work. Your extensions would work with them too.\n\n Linus\n"},{"id":"213177","messageId":"CALkWK0mMQJfJ7M=NGJdEs3np3Y32DwKUMVEYJf7xh+X6toWwNg@mail.gmail.com","threadId":"33386","inReplyTo":"CA+55aFw+2wgUufJn7BcXACBQqywAGBAcZWS6N_UV8UA91G447Q@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T19:52:45Z","receivedAt":"2013-04-04T19:52:45Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Linus Torvalds wrote:\n> On Thu, Apr 4, 2013 at 12:36 PM, Ramkumar Ramachandra\n> wrote:\n>>\n>> Let's compare the two alternatives: .gitmodules versus link object.\n>> If I want my fork of .gitmodules, I create a commit on top.\n>\n> Or you could also just edit and carry a dirty .gitmodules around for\n> your personal use-case.\n\nJust take the link's buffer with you everywhere. All you have to do\nis git edit-link and paste the file's contents there, instead\nof opening .gitmodules directly in your editor.\n\n> And I don't see what you can do with the link objects that you cannot\n> do with .gitmodules. That's what it really boils down to. .gitmodules\n> do actually work. Your extensions would work with them too.\n\nIf it came to that, you could write a huge Perl script to solve\neverything with a .githack. It breaks the internal symmetry of the\nrepository, which is why git-submodule is having such a field day.\nI'm trying to prove, in my series, that making fundamental changes\nlets us get rid of a huge amount of complexity.\n"},{"id":"213180","messageId":"CALkWK0nwdgjiNnQo+1OY90BYGbKdUeLhqqCvQEtnS3q3bQtHFA@mail.gmail.com","threadId":"33386","inReplyTo":"7va9pehzql.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T19:59:02Z","receivedAt":"2013-04-04T19:59:02Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> I think Heiko and Jens's (by the way, why aren't they on the Cc:\n> list when this topic is clearly discussing submodules? Don't we\n> want to learn how the current submodule subsystem is used to solve\n> what real-world problems?) .gitmodules updates is exactly going in\n> that direction.\n\nBecause it's pointless. We're not discussing a git-submodule\nalternative. We're discussing how to fix git-core so that\ngit-submodule becomes much simpler; to the extent that it will be\nunnecessary soon.\n\ngit-submodule is years of hard work and it can do a limited version of\nfloating with great difficulty. Mine is two days of work, and can\nalready do true floating submodules. What is going on?\n"},{"id":"213181","messageId":"CALkWK0n0uKTsPv2d20ESVd5OV5yfK5V1HecGWePYU=UFw5Jkfg@mail.gmail.com","threadId":"33386","inReplyTo":"CA+55aFw+2wgUufJn7BcXACBQqywAGBAcZWS6N_UV8UA91G447Q@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T20:04:06Z","receivedAt":"2013-04-04T20:04:06Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Linus Torvalds wrote:\n> Or you could also just edit and carry a dirty .gitmodules around for\n> your personal use-case.\n\nI'm sorry, but a dirty worktree is unnecessarily painful to work with.\n I don't think anyone objects to committing, if they can understand\nbasic rebase.\n"},{"id":"213182","messageId":"CALkWK0nS74ZiHkEk7uCg=yPB7m6FkmfmfU7vs7u-AENjPu-GCw@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0mMQJfJ7M=NGJdEs3np3Y32DwKUMVEYJf7xh+X6toWwNg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T20:08:11Z","receivedAt":"2013-04-04T20:08:11Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Ramkumar Ramachandra wrote:\n> Just take the link's buffer with you everywhere. All you have to do\n> is git edit-link and paste the file's contents there, instead\n> of opening .gitmodules directly in your editor.\n\nOn this. The buffer doesn't have to conform to a tight spec: we can\njust expose a .gitconfig-like buffer and reduce it to a tight spec\nbefore writing out.\n"},{"id":"213187","messageId":"515DE263.2000105@web.de","threadId":"33386","inReplyTo":"7va9pehzql.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-04T20:28:19Z","receivedAt":"2013-04-04T20:28:19Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 04.04.2013 21:17, schrieb Junio C Hamano:\n> Linus Torvalds writes:\n> \n>> ... The features you seem to be after (ie that whole\n>> floating/refname thing) don't seem fundamentally antithetical to the\n>> current model (a \"commit\" SHA1 of all zeroes for floating, with a new\n>> refname field in .submodules? I dunno)..\n> \n> Just on this part.\n> \n> I think Heiko and Jens's (by the way, why aren't they on the Cc:\n> list when this topic is clearly discussing submodules? Don't we\n> want to learn how the current submodule subsystem is used to solve\n> what real-world problems?) .gitmodules updates is exactly going in\n> that direction.\n> \n> - A submodule can be marked as floating in .gitmodules and be\n> specified how (typially, \"use the tip of this branch in the\n> submodule\");\n> \n> - Running \"submodule update\" a floating submodule does not detach\n> the submodule working tree to commit in the index of the\n> superproject; instead it will use the specified branch tip;\n> \n> - A floating submodule records a concrete commit object name in the\n> index of the superproject (no need to stuff an unusual SHA-1\n> there to signal that the submodule is floating---it is recorded\n> in the .gitmodules). Thanks to this, a release out of the\n> top-level can still describe the state of the entire tree;\n> \n> - It would be normal for the commit recorded in the index of the\n> superproject not to match what is checked out in the submodule\n> working tree (i.e. the tip of the branch in the submodule may\n> have advanced). A traditional non-floating submodule has many\n> mechanisms to be noisy about this situation to prevent users from\n> making an incomplete commits, but they may have to be toned down\n> or squelched for floating submodules.\n> \n> Anything I missed, Jens, Heiko?\n\nNope, that perfectly sums it up.\n"},{"id":"213205","messageId":"515DEE86.3020301@web.de","threadId":"33386","inReplyTo":"CA+55aFyQwJfiYo06y1bRNpKT6wOquhG9a9M_4YvLG_UT3b34-w@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-04T21:20:06Z","receivedAt":"2013-04-04T21:20:06Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 04.04.2013 21:04, schrieb Linus Torvalds:\n> My gut feel is that yes, .gitmodules was always a bit of a hack, but\n> it's a *working* hack, and it does have advantages exactly because\n> it's more fluid than an actual git object (which by definition has to\n> be set 100% in stone).\n\nExactly. The flexibility of the .gitmodules file will really help us\nwhen it comes to the next feature that submodules are going to learn\nafter recursive update: automatically initialize and then populate\ncertain submodules during the clone of the superproject. You have to\nbe able to configure that per submodule, which needs a new config\noption in .gitmodules. Others may follow for different use cases.\n\nWhile starting to grok submodules I was wondering myself if the data\nstored in .gitmodules would better be stored in an extended gitlink\nobject, but I learned soon that the scope of the data that has to be\nstored there was not clear at that time (and still isn't). So I'm\nnot opposed per se to adding a special object containing all that\ninformation, but I strongly believe we are not even close to\nconsidering such a step (and won't be for quite some time and maybe\nnever will).\n"},{"id":"213207","messageId":"CALkWK0mb_GzmzO86Jayry4AzKDn_JbK12K=OjxkYovcN4pLPCA@mail.gmail.com","threadId":"33386","inReplyTo":"515DEE86.3020301@web.de","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T21:35:20Z","receivedAt":"2013-04-04T21:35:20Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jens Lehmann wrote:\n> Exactly. The flexibility of the .gitmodules file will really help us\n> when it comes to the next feature that submodules are going to learn\n> after recursive update:\n\nThat's like saying that the flexibility of a blob is invaluable: let's\nthrow out all the other objects, and make do with blobs. Ofcourse we\nmake mistakes: we didn't put a generation number in the commit object,\nfor instance (I'm not arguing about whether it's right or wrong: just\nthat some people think it's a mistake).\n\n> While starting to grok submodules I was wondering myself if the data\n> stored in .gitmodules would better be stored in an extended gitlink\n> object, but I learned soon that the scope of the data that has to be\n> stored there was not clear at that time (and still isn't). So I'm\n> not opposed per se to adding a special object containing all that\n> information, but I strongly believe we are not even close to\n> considering such a step (and won't be for quite some time and maybe\n> never will).\n\nNonsense. We will think through it before freezing the format, like\nwe did with the other objects.\n"},{"id":"213210","messageId":"7vd2uagd10.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"515DEE86.3020301@web.de","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-04T22:13:31Z","receivedAt":"2013-04-04T22:13:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann writes:\n\n> While starting to grok submodules I was wondering myself if the data\n> stored in .gitmodules would better be stored in an extended gitlink\n> object, but I learned soon that the scope of the data that has to be\n> stored there was not clear at that time (and still isn't). So I'm\n> not opposed per se to adding a special object containing all that\n> information, but I strongly believe we are not even close to\n> considering such a step (and won't be for quite some time and maybe\n> never will).\n\nI actually think the storage is more or less an orthogonal issue.\n\nThe format must be defined to be extensible (nobody is perfect and\nif you wait for an exhaustive list of attributes that cover all use\ncases including the ones that haven't even been invented yet, you\nwill get nowhere), and designed carefully to reduce the chance of\nallowing the extended/optional bit to express the same thing in two\ndifferent ways to make sure the object name will not become\nunnecessarily unstable, but you can start small, keep adding\noptional fields, and be prepared to design an upgrade path when you\nneed to add new mandatory fields---that cannot be helped whether you\nrecord the information about submodules in .gitmodules or a new\nblob-ish object at the location where the submodule tree should\nreside in the index and the tree.\n\nHowever, the current .gitmodules design, even though it originally\nwas invented as a way to carry information other than what a single\ncommit object name from an otherwise unrelated project can express\nwithout having to change anything in-core, has a few practical\nmerits. The information _about_ submodules is stored separately\n(i.e. in the .gitmodules file) from submodules themselves, and it\nmay be a good thing.\n\nWhen you are changing information _about_ submodules (e.g. you may\nbe updating the recommended URL to fetch it from), you can use the\nusual tools like \"git diff\" to see how it changed, just like changes\nto any other file. If the information _about_ a submodule A is\nstored at path A, and at the same time you have a working tree that\ncorresponds to the root of the submodule A at that path, it gets\nunclear what \"git diff A\" should report. Should it report the\nchange in the submodule itself, or should it report the change in\nthe information _about_ the submodule? By separating these two\nconcepts to two different places, .gitmodules design solves the\nissue nicely.\n"},{"id":"213211","messageId":"CALkWK0kQ8qYXHKr4e93A1dh3Y1vL+HZvMR_1xtKkUr-_7bMS6Q@mail.gmail.com","threadId":"33386","inReplyTo":"7vd2uagd10.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T22:18:17Z","receivedAt":"2013-04-04T22:18:17Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> When you are changing information _about_ submodules (e.g. you may\n> be updating the recommended URL to fetch it from), you can use the\n> usual tools like \"git diff\" to see how it changed, just like changes\n> to any other file. If the information _about_ a submodule A is\n> stored at path A, and at the same time you have a working tree that\n> corresponds to the root of the submodule A at that path, it gets\n> unclear what \"git diff A\" should report. Should it report the\n> change in the submodule itself, or should it report the change in\n> the information _about_ the submodule? By separating these two\n> concepts to two different places, .gitmodules design solves the\n> issue nicely.\n\ngit diff-link. Just turn it into a buffer and diff as usual.\n"},{"id":"213213","messageId":"7vy5cyexuf.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0kQ8qYXHKr4e93A1dh3Y1vL+HZvMR_1xtKkUr-_7bMS6Q@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-04T22:26:48Z","receivedAt":"2013-04-04T22:26:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Junio C Hamano wrote:\n>> When you are changing information _about_ submodules (e.g. you may\n>> be updating the recommended URL to fetch it from), you can use the\n>> usual tools like \"git diff\" to see how it changed, just like changes\n>> to any other file. If the information _about_ a submodule A is\n>> stored at path A, and at the same time you have a working tree that\n>> corresponds to the root of the submodule A at that path, it gets\n>> unclear what \"git diff A\" should report. Should it report the\n>> change in the submodule itself, or should it report the change in\n>> the information _about_ the submodule? By separating these two\n>> concepts to two different places, .gitmodules design solves the\n>> issue nicely.\n>\n> git diff-link. Just turn it into a buffer and diff as usual.\n\nSounds like you are saying that you can pile a new command on top of\nnew command to solve what the existing tools people are familar with\ncan already solve in a consistent way without adding anything new.\nAre you going to dupliate various options to \"git diff\" and \"git\nlog\" in \"git diff-link\"? Will you then next need \"git log-link\"?\n"},{"id":"213215","messageId":"CALkWK0kpf+AAzrLuqKQx5iv3nNKJ48R5etcBrcdoG46Z1ipgbQ@mail.gmail.com","threadId":"33386","inReplyTo":"7vy5cyexuf.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T22:32:48Z","receivedAt":"2013-04-04T22:32:48Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> Sounds like you are saying that you can pile a new command on top of\n> new command to solve what the existing tools people are familar with\n> can already solve in a consistent way without adding anything new.\n> Are you going to dupliate various options to \"git diff\" and \"git\n> log\" in \"git diff-link\"? Will you then next need \"git log-link\"?\n\nWhat I'm saying is: As always, we start with plumbing and work our way\nup to porcelain. We do have git diff-files, diff-index, diff-tree, so\nI don't see what the problem with diff-link is. The point is that we\ncan get an initial scripted version out quickly.\n\nAnd no, I never suggested a git log-link.\n"},{"id":"213218","messageId":"7vli8xgahc.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0kpf+AAzrLuqKQx5iv3nNKJ48R5etcBrcdoG46Z1ipgbQ@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-04T23:08:31Z","receivedAt":"2013-04-04T23:08:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Junio C Hamano wrote:\n>> Sounds like you are saying that you can pile a new command on top of\n>> new command to solve what the existing tools people are familar with\n>> can already solve in a consistent way without adding anything new.\n>> Are you going to dupliate various options to \"git diff\" and \"git\n>> log\" in \"git diff-link\"? Will you then next need \"git log-link\"?\n>\n> What I'm saying is: As always, we start with plumbing and work our way\n> up to porcelain. We do have git diff-files, diff-index, diff-tree, so\n> I don't see what the problem with diff-link is. The point is that we\n> can get an initial scripted version out quickly.\n>\n> And no, I never suggested a git log-link.\n\n\"git log -p .gitmodules\" would be a way to review what changed in\nthe information about submodules. Don't you need \"git log-link\" for\nexactly the same reason why you need \"git diff-link\" in the first\nplace?\n\nSo you may not have suggested it, but I suspect that was only\nbecause you haven't had enough time to think things through.\n"},{"id":"213220","messageId":"CALkWK0mcii_YGqiQTxhSOfgzn7MOHO0TEO3Rx3cRQLOi1ij5Tg@mail.gmail.com","threadId":"33386","inReplyTo":"7vli8xgahc.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-04T23:14:45Z","receivedAt":"2013-04-04T23:14:45Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> \"git log -p .gitmodules\" would be a way to review what changed in\n> the information about submodules. Don't you need \"git log-link\" for\n> exactly the same reason why you need \"git diff-link\" in the first\n> place?\n>\n> So you may not have suggested it, but I suspect that was only\n> because you haven't had enough time to think things through.\n\nWhat is this git log -p .gitmodules doing? It's walking down the\ncommit history, and picking out the commits in which that blob\nchanged. Then it's diffing the blobs in those commits with each\nother. Why is git log -p any different? We already know how\nto diff blobs, and we just need a way to diff links.\n"},{"id":"213246","messageId":"CALkWK0mpVbHtEPS5d4FA8wjsF6L0bncUeJnOJccPSYwgM_v-rg@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0nNjvV5VGvT_eaubFoOhMnJ-N8FECAayd5A2K3BzeRh6Q@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-05T06:53:07Z","receivedAt":"2013-04-05T06:53:07Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Ramkumar Ramachandra wrote:\n> After some discussion, I hope to be able to finalize a list of fields\n> that will suffice for (nearly) everything.\n\nThe task is actually much easier than this. All we have to do is\nfinalize the list of fields that will mandatorily be written to the\nlink object. As I might have indicated in my series, this is:\nupstream_url, checkout_rev, and ref_name. Really, the user only needs\nto supply a valid upstream_url: after a clone, everything else can be\ninferred (with the exception of a ref_name conflict; I don't like\nauto-mangling).\n\nOther fields are like .git/config fields. We can add new key/value\npairs in the future, without worrying about migration. A problem\narises only if we want to add a new mandatory field, change the\ndefault value of a key, or deprecate an existing key/value pair.\n"},{"id":"213247","messageId":"CALkWK0m5q7gW0LPdJVWbZTkTnHrFBYyfRxFMnRMtyuSePtLL9Q@mail.gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-8-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH 7/7] sha1_file: write ref_name to link object","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-05T07:03:21Z","receivedAt":"2013-04-05T07:03:21Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Ramkumar Ramachandra wrote:\n> Great. Now, we just have to write refs/modules//* at\n> commit-time.\n\nActually, we have to update things in refs/modules/ everytime we\nupdate things in refs/heads/. In the case of a 'git branch -M' for\nexample, refs/heads/ is rewritten to refs/heads/:\nsimilarly, refs/modules// needs to be moved to\nrefs/modules//.\n\nThere is one special case worth mentioning. Let's say I'm committing\nchanges to link objects to a detached HEAD, b8bb3f. Then, I write\nrefs/modules/b8bb3f/ at commit-time. A subsequent 'checkout -b' that\nupdates refs/heads/ will just have to move\nrefs/modules/b8bb3f/ to refs/modules//.\n\nThe caveat is that I might commit to the detached HEAD and leave it\nhanging around until the next gc. So, gc will need to remove\nrefs/modules/b8bb3f/ too, but that's not a pressing issue at the\nmoment.\n"},{"id":"213248","messageId":"CALkWK0n721iPE6=Lr2EZuq6OV++CW0U19wJDQ2BNfwo1ABKcYg@mail.gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-7-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH 6/7] clone: introduce clone.submodulegitdir","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-05T07:07:07Z","receivedAt":"2013-04-05T07:07:07Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Ramkumar Ramachandra wrote:\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index e0aaf13..1b798e6 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -658,11 +659,22 @@ static void write_refspec_config(const char* src_ref_prefix,\n> strbuf_release(&value);\n> }\n>\n> +static int git_clone_config(const char *var, const char *value, void *cb)\n> +{\n> + if (!strcmp(var, \"clone.submodulegitdir\")) {\n> + git_config_string(&submodule_gitdir, var, value);\n> + return 0;\n> + }\n> + return git_default_config(var, value, cb);\n> +}\n\nsubmodule_gitdir can be a human-path, and we will need real_path() to\nturn it into a concrete path. Why doesn't real_path() expand ~ yet?\n"},{"id":"213249","messageId":"CALkWK0k=XYPJXEEx8CMYRx_k0d9Z=b_W=wGGvbZsXJEe8K99EA@mail.gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-3-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH 2/7] sha1_file, link: write link objects to the database","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-05T07:11:21Z","receivedAt":"2013-04-05T07:11:21Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Ramkumar Ramachandra wrote:\n> diff --git a/link.c b/link.c\n> index bb20a51..349646d 100644\n> --- a/link.c\n> +++ b/link.c\n> @@ -20,8 +20,30 @@ struct link *lookup_link(const unsigned char *sha1)\n>\n> int parse_link_buffer(struct link *item, void *buffer, unsigned long size)\n> {\n> + char *bufptr = buffer;\n> + char *tail = buffer + size;\n> + char *eol;\n> +\n> if (item->object.parsed)\n> return 0;\n> item->object.parsed = 1;\n> + while (bufptr < tail) {\n> + eol = strchr(bufptr, '\\n');\n> + *eol = '\\0';\n> + if (!prefixcmp(bufptr, \"upstream_url = \"))\n> + item->upstream_url = xstrdup(bufptr + 15);\n> + else if (!prefixcmp(bufptr, \"checkout_rev = \"))\n> + item->checkout_rev = xstrdup(bufptr + 15);\n> + else if (!prefixcmp(bufptr, \"ref_name = \"))\n> + item->ref_name = xstrdup(bufptr + 11);\n> + else if (!prefixcmp(bufptr, \"floating = \"))\n> + item->floating = atoi(bufptr + 11);\n> + else if (!prefixcmp(bufptr, \"statthrough = \"))\n> + item->statthrough = atoi(bufptr + 14);\n> + else\n> + return error(\"Parse error in link buffer\");\n> +\n> + bufptr = eol + 1;\n> + }\n> return 0;\n> }\n\nThis needs to be replaced by a .git/config parser. However, I can't\nuse the parser from config.c as-it-is, because it expects a section\nlike [core] to be present. So, we have to refactor it to optionally\nparse section-less configs.\n"},{"id":"213253","messageId":"CALkWK0=VOPsPg7mC46aP_VZiUv9FD5vZZFkGYKUTjLbWXc0DDA@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0k=XYPJXEEx8CMYRx_k0d9Z=b_W=wGGvbZsXJEe8K99EA@mail.gmail.com","subject":"Re: [PATCH 2/7] sha1_file, link: write link objects to the database","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-05T07:59:35Z","receivedAt":"2013-04-05T07:59:35Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Ramkumar Ramachandra wrote:\n> This needs to be replaced by a .git/config parser. However, I can't\n> use the parser from config.c as-it-is, because it expects a section\n> like [core] to be present. So, we have to refactor it to optionally\n> parse section-less configs.\n\nEr, sorry about the thinko: I meant that edit-link should use the\n.git/config parser. This one is just fine.\n"},{"id":"213302","messageId":"7vfvz4dhy9.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0mcii_YGqiQTxhSOfgzn7MOHO0TEO3Rx3cRQLOi1ij5Tg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-05T17:07:42Z","receivedAt":"2013-04-05T17:07:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Junio C Hamano wrote:\n>> \"git log -p .gitmodules\" would be a way to review what changed in\n>> the information about submodules. Don't you need \"git log-link\" for\n>> exactly the same reason why you need \"git diff-link\" in the first\n>> place?\n>>\n>> So you may not have suggested it, but I suspect that was only\n>> because you haven't had enough time to think things through.\n>\n> What is this git log -p .gitmodules doing? It's walking down the\n> commit history, and picking out the commits in which that blob\n> changed. Then it's diffing the blobs in those commits with each\n> other. Why is git log -p any different? We already know how\n> to diff blobs, and we just need a way to diff links.\n\nYou already forget what you invented \"git diff-link\" as a \"solution\"\nfor, perhaps?\n\nBy recording the submodules themselve and information _about_ the\nsubmodules separately (the latter is in .gitmodules), \"git diff A\"\ncan show the difference in submodule A, while \"git diff .gitmodules\"\ncan show a change, which is a possibly in-working-tree-only proposed\nchange, in information about submoudules.\n\nOnce you start recording the latter also at path \"A\", it becomes\nunclear what \"git diff A\" should show.\n\nThat is what I said in the message, to which you invented \"diff-link\"\nas a solution to the \"unclear\"-ness.\n\nAm I misremembering the flow of discussion in this thread?\n"},{"id":"213266","messageId":"CALkWK0nVbn7hTEU=decDDdwKXP1-sZakxQpTw4ce6nx3uHO3ag@mail.gmail.com","threadId":"33386","inReplyTo":"7vfvz4dhy9.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-05T17:23:40Z","receivedAt":"2013-04-05T17:23:40Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> Once you start recording the latter also at path \"A\", it becomes\n> unclear what \"git diff A\" should show.\n>\n> That is what I said in the message, to which you invented \"diff-link\"\n> as a solution to the \"unclear\"-ness.\n\nI just thought it would be a stopgap until we get diff to support\nlinks natively. Obviously, when we get native diff support, 'log -p'\nwill be able to show differences as well. As it turns out from my\nlittle experiment with cat-link, it's really easy to get native diff\nsupport, and I'm targeting that directly instead of a scripted\nsolution.\n\nAs for the unclearness issue, it's a little more complicated than\nthat: a non-floating submodule could've previously been a floating\none, or vice-versa. As of this moment, I'm only planning to show\ndifferences between link buffers. In the case when two consecutive\ncommits change a link checkout_rev (where floating is set to 0), we\ncan come up with something like the current diff.submodule = log. I\nsee no cause to worry about the interface of that now.\n"},{"id":"213367","messageId":"CALkWK0mM9HBUvHGHPEw4Vdosp_qfu_1L49TaZrzdC5n3soWkYg@mail.gmail.com","threadId":"33386","inReplyTo":"1365100243-13676-1-git-send-email-artagnon@gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-06T20:10:32Z","receivedAt":"2013-04-06T20:10:32Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi again,\n\nSo we've thought about it for some time, and I really need you to\nstart reviewing the code now.\nI'll just summarize what we've discussed so far:\n\n1. The malleability argument doesn't hold, because we're proposing a\nlink object with optional fields.\n\n2. The local-fork argument doesn't hold, because users will be\nrebasing changes to the link object in exactly the same way as they\ncurrently do with the blob object .gitmodules.\n\n3. The worktree argument doesn't hold, because we're proposing to\ntreat the link object as nothing more than a blob object that can be\nparsed by git-core. It will stage and unstage just like a blob.\nSure, it's not accessible directly by the filesystem: so what? What\nis the difference does `emacsclient .gitmodules` versus `git edit-link\nclayoven` make to the end-user?\n\n4. The diff-confusion argument is just another by-the-way, but it\ndoesn't really hold either. Currently, we see:\n\n - Subproject commit b83492\n + Subproject commit 39ab2f\n\n(with diff.submodule set to log, we can actually see the log of the\nsubmodule between these two commits. With links, we will see:\n\n - checkout_rev = b83492\n + checkout_rev = 39ab2f\n\nThere's nothing that prevents us from respecting diff.submodule (some\nminor glue code will have to be written; that's all).\n\n*. There is actually one thing that .gitmodules does better than\nlinks. foreach. It's trivial to implement with .gitmodules and hard\nto implement with links: with .gitmodules, the paths of all the\nsubmodules are in one place. But with links, we'll have to\nunpack_trees() every tree in the entire repository, and dig through it\nto find all the link objects to initialize. Basically, inefficient\nand inelegant. However, I don't think this is a big problem in\npractice, since this is not exactly a common operation: I'd probably\nwant to recurse-submodules once at clone time.\n"},{"id":"213371","messageId":"7v61zz9fu3.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0mM9HBUvHGHPEw4Vdosp_qfu_1L49TaZrzdC5n3soWkYg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-07T03:31:32Z","receivedAt":"2013-04-07T03:31:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> So we've thought about it for some time, and I really need you to\n> start reviewing the code now.\n>\n> I'll just summarize what we've discussed so far:\n> ...\n\nI do not think we have heard anything concrete and usable about what\nyou are trying to achieve yet.\n\nYou may be proposing to discard baby with bathwater. We haven't\nseen an evidence that the change is really worth having. We do not\neven know what you are trying to change, other than \"I want to add a\nnew object type to largely replicate what is recorded in .gitmodules\nfile\". What are you trying to solve?\n\n\tI want to have a project for an appliance, that binds two\n\tprojects, the kernel and the appliance's userspace. The\n\tusual suspects to use to implement such a project would be\n\tGit submodule, repo, or Gitslave.\n\n\tI want to be able to do X and Y and Z in managing such a\n\tproject.\n\n If I try to use submodule, I cannot see how I could make it\n\tdo X for _this thing_, and it is not a bug in the\n\timplementation but is fundamental because of _this and\n\tthat_. If I try to use repo, ...... the same, and the same\n\tfor Gitslave. ......\n\n\tI propose to add a new \"gitlink\" object recorded in the tree\n\tand in the index, and the said cases X, Y and Z can be\n\tsolved in _such and such way_. We cannot solve it without\n\thaving a new \"gitlink\" object recorded in the tree object\n\tbecause of _this and that reason_.\n\nI think it is too premature to discuss _your_ code. The patches do\nnot even tell us anything about how much more work is needed to\nmerely make Git with your patches work properly again. For one\nthing, I suspect that you won't even be able to repack a repository\nthat has OBJ_LINK only with the patches you posted.\n\nAt this point the only thing that we can gain from reading your\npatch is that you can write C to do _something_, but that something\nis so fuzzily explained that we do not know what to make of that\nknowledge that you write good (or bad, we don't know) C.\n\nIt would be much more productive to learn what these specific issues\nX, Y and Z are, and if the problems you are having with existing\nsolutions are really fundamental that need changes to object layer\nto solve.\n"},{"id":"213384","messageId":"CALkWK0n64BW8zQXweR38T6DjT+iYBbsZDPCLZ-4igqGJHoFwKw@mail.gmail.com","threadId":"33386","inReplyTo":"7v61zz9fu3.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T07:27:14Z","receivedAt":"2013-04-07T07:27:14Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> I think it is too premature to discuss _your_ code. The patches do\n> not even tell us anything about how much more work is needed to\n> merely make Git with your patches work properly again. For one\n> thing, I suspect that you won't even be able to repack a repository\n> that has OBJ_LINK only with the patches you posted.\n\nLet me try to rephrase my original request: I'm an inexperienced\ncontributor trying to do something very ambitious. Having authored a\nhuge part of it, Linus and you understand git-core much better than I\ncan ever hope to understand. These are things that you need to tell\nme after reading the patches. I only have a rough idea to make Git\nwork properly with my patches again: I can't know for sure until I\nwrite all the code. What's more? Your guess will probably be better\nthan mine after you read the code.\n\nYou're asking me to submit a perfect 40 or 50 part series that's a\npotential candidate for merging. I'm sorry to say this, but I'm\nincapable of doing that without posting intermediate work and getting\nhelp. Frankly, it's a very unreasonable expectation; I don't think\nanyone except you or Linus can even get close after making such a\nfundamental change.\n\nI might end up writing all the code (which I'm perfectly okay with),\nand all I'm asking from you is to constantly keep picking my brain (by\nreviewing my code and posting good critiques). Am I being\nunreasonable?\n\n> At this point the only thing that we can gain from reading your\n> patch is that you can write C to do _something_, but that something\n> is so fuzzily explained that we do not know what to make of that\n> knowledge that you write good (or bad, we don't know) C.\n\nC is irrelevant here: I'm not asking you for style/ structuring tips;\nI'm asking you for a critique on the implementation of this idea.\nLinus raised some good points after reading [0/7] that I countered,\nand there isn't anything else either anyone can raise without reading\nmore. Code speaks more clearly than the English in [0/7]: you should\nbe able to deduce a lot more intent and direction.\n\n> It would be much more productive to learn what these specific issues\n> X, Y and Z are, and if the problems you are having with existing\n> solutions are really fundamental that need changes to object layer\n> to solve.\n\nTo reiterate: link does not make possible something that is not\nfundamentally _possible_ with a .githack and a 100k-line Perl script.\nAt its core, every variant of submodules does this. What I'm\nessentially proposing: break up the information in .githack into\nsmaller bits and create a new object type so it can be parsed by\ngit-core easily. If you agree that my proposal doesn't make\nimpossible what was possible earlier, and that it makes life easier\nfor everyone, we should be good to go.\n\nWhen the series matures, we can investigate the other implementations\nin greater detail so we can pick out more optional fields to add to\nthe link object before getting it merged. This is not the right time\nto do that: we're currently trying to get git-core working with the\nmandatory fields.\n\n> I do not think we have heard anything concrete and usable about what\n> you are trying to achieve yet.\n\nI'll try to rephrase your concerns here:\n\n0. We don't know if this approach will yield a mergeable series at\nall, because it breaks so many things and is so difficult to complete.\n\n1. We don't know how much work is needed to bring the series to a\npoint where it is in a mergeable state. There is no timeline\nspecified.\n\n2. We can't build an exhaustive list of the problems that this new\napproach will solve (ie. we haven't finalized the optional fields).\n\n3. We don't have anything useable yet.\n\nAnd your non-concerns should be:\n\n1. We know that this approach won't fundamentally limit us in not\nbeing able to solve some specific problems that the .githack approach\nsolves.\n\n2. We know that this approach makes life easier for everyone, and\nthere are significant concrete benefits to teaching git-core about\nlinks.\n\nI agree with all this fully. I don't have a concrete roadmap; we'll\njust have to dive in based on what we've seen so far, and hope that\nwe're able to finish what we started. So, my final question is: are\nyou still not convinced that this approach shows a lot of potential,\nand is worth exploring now?\n"},{"id":"213395","messageId":"7vr4im7m1y.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0n64BW8zQXweR38T6DjT+iYBbsZDPCLZ-4igqGJHoFwKw@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-07T09:00:09Z","receivedAt":"2013-04-07T09:00:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Junio C Hamano wrote:\n>> I think it is too premature to discuss _your_ code. The patches do\n>> not even tell us anything about how much more work is needed to\n>> merely make Git with your patches work properly again. For one\n>> thing, I suspect that you won't even be able to repack a repository\n>> that has OBJ_LINK only with the patches you posted.\n>\n> ...\n> You're asking me to submit a perfect 40 or 50 part series...\n\nNot at all. And please do not start _coding_.\n\nWhen the design is not clear enough that a 7-patch series is not\nready to be reviewed, certainly 50-patch series will not be. Not\nuntil you can explain what you are trying to solve and convince\nothers why other less disruptive approaches are fundamentally\nunworkable, and why we need to change the object layer.\n\n> To reiterate: link does not make possible something that is not\n> fundamentally _possible_ with a .githack and a 100k-line Perl script.\n>\n> At its core, every variant of submodules does this. What I'm\n> essentially proposing: break up the information in .githack into\n> smaller bits and create a new object type so it can be parsed by\n> git-core easily.\n\nThe .gitmodules file is designed to be easily parsable by the config\ninfrastructure and implemented as such already, thank-you-very-much.\n\nWhy do you keep calling an already working solution with derogatory\nmisspelling? That only gives others an impression that you do not\nunderstand how the current system works, and pursuade them not to\nwaste time responding to you. Stop it.\n\n> When the series matures, we can investigate the other implementations\n> in greater detail so we can pick out more optional fields to add to\n> the link object before getting it merged.\n\nSorry, but that is not how open source works in general, and\ncertainly not how this project works.\n\nWe do not add disruptive change just for the sake of changing it to\nbreak a working system, make an extra work to clean up the fallout\nfor ourselves (i.e. your \"40 to 50 patch series\", but honestly\nspeaking I expect it would be more like a 4 months work for a full\ntime engineer or two), for unproven design (that has not yet to be\nillustrated) to solve problems (that has not yet to be explained),\nwithout knowing that\n\n (1) the problems are worth solving;\n\n (2) the design will solve the problems; and\n\n (3) solving the same problems without such a disruptive change is\n impossible, or so cumbersome that it will be far larger than\n the work needed to clean-up the fallout of the disruptive\n change.\n\nSo what are your X, Y, Z? You still haven't answered that question.\n\nFor that matter, you didn't answer the same question that was more\ntersely phrased by Linus in the very first response in the thread:\n\n> Linus Torvalds wrote:\n>> I don't dispute that a new link object might be a good idea, but\n>> there's no explanation of the actual format of this thing anywhere,\n>> and what the real advantages would be. A clearer \"this is the design,\n>> this is the format of the link object, and this is what it buys us\"\n>> would be a good idea.\n>\n> Yeah, I need help with that. I've just stuffed in whatever fields\n> popped into my mind first. The current ones are:\n\n\nAnd what you listed were your back-then-current thinking on \"actual\nformat\".\n\nWhat are the real advantages? How are they used? What do they\nallow us to do what we cannot do with .gitmodules (or repo or\ngitslave for that matter)? What do they buy us?\n\nWhat problem are you trying to solve?\n\nI have this suspicion that you do not have to change anything in the\nobject layer to make Git behave very differently from the current\nsubmodule implementation. For example, if your gripe were (I am\njust speculating without any input from you in this thread) that\neach submodule working tree has \".git\" at their top and there is no\nunified view from the top-level [*1*], we certainly can solve it\nwithout any change to the object layer.\n\nWe currently add a cache entry that has the commit object name to\nthe index from the tree object when we check out the superproject,\nand create a separate repository with a working tree when we\ninstantiate a submodule.\n\nThis arrangement does not have to be fundamental. It is a design\nchoice of one particular working tree layout, which is totally local\nto individual superproject working tree.\n\nYou could arrange a single index (the one in the superproject) to\nhold the tree contents from the commit in the submodule, while\nnoting the original commit object name in a new mandatory extension\nsection in the index. The index will have a unified view of the\nwhole tree, and we do not have to have a .git at the root of each\nsubmodule working tree (be it a directory or a gitfile).\n\nI think the message where I talked about the \"bind\" idea in the list\narchive URLs I gave you earlier would give you such a layout, and\nyou should go read it again to understand how the flow from object\ndatabase to index to working tree back to index back to object\ndatabase was envisioned to work. I think the only thing we need to\ndo differently from that \"bind\" proposal in the current world order\nis not to record the new submodule state in the commit object of the\nsuperproject, but actually create a new commit for the submodule\npart and store it in the tree object for the commit in the\nsuperproject (the \"bind lines in the superproject commit\" was a\nhack, only because we didn't have a way to write the submodule\ncommit object name in the index and in the trees).\n\n\n[Footnote]\n\n*1* I am not saying that keeping everything down to the leaves of\nsubmodules is necessarily a good idea. This is only meant as an\nillustration of what kind of a system that looks drastically\ndifferent from the end-user's point of view you could build, without\nbreaking the object layer and having to redo everything.\n"},{"id":"213403","messageId":"CALkWK0=cCu-dH7Cstuj5WJ6WW2xSWO9FuBWK0zm4=LK7+J6mzg@mail.gmail.com","threadId":"33386","inReplyTo":"7vr4im7m1y.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T10:58:41Z","receivedAt":"2013-04-07T10:58:41Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Your overall hostility is unappreciated. The burden of proof is on\nme, while you calmly sit back and criticize anything that breaks the\ncurrent working state, and refuse to look at the implementation.\nAnyway, here we go again.\n\nJunio C Hamano wrote:\n> Not at all. And please do not start _coding_.\n\nYou've successfully killed all my enthusiasm. Congratulations.\n\n> When the design is not clear enough that a 7-patch series is not\n> ready to be reviewed, certainly 50-patch series will not be. Not\n> until you can explain what you are trying to solve and convince\n> others why other less disruptive approaches are fundamentally\n> unworkable, and why we need to change the object layer.\n\n\"So, my final question is: are you still not convinced that this\napproach shows a lot of potential, and is worth exploring now?\"\n\"No.\"\n\nI don't know how many times to repeat this: No, Junio. A less\ndisruptive approach is _not_ fundamentally unworkable. You can spend\nthe next five years fixing submodule.c/ git-submodule.sh, or can take\na step back and think about why it's in such pathetic shape right now.\n\n>> To reiterate: link does not make possible something that is not\n>> fundamentally _possible_ with a .githack and a 100k-line Perl script.\n>>\n>> At its core, every variant of submodules does this. What I'm\n>> essentially proposing: break up the information in .githack into\n>> smaller bits and create a new object type so it can be parsed by\n>> git-core easily.\n>\n> The .gitmodules file is designed to be easily parsable by the config\n> infrastructure and implemented as such already, thank-you-very-much.\n\nYou're missing the point. Who parses .gitmodules? submodule.c and\ngit-submodule.sh, as opposed to a link being parsed by git-core. How\nis it any different? That's what my series is trying to answer.\n\n> Why do you keep calling an already working solution with derogatory\n> misspelling? That only gives others an impression that you do not\n> understand how the current system works, and pursuade them not to\n> waste time responding to you. Stop it.\n\nI don't see why you have to get offended by my deliberate misspelling:\nwe're not emotionally attached to software, and I'm merely criticizing\nwhat I think is a bad hack. I'm not pointing out the concrete\nlimitations of git-submodule precisely because they can be fixed\nwithout any changes to the object layer: this thread will become a\ndiscussion about how to fix submodule.c/ git-submodule.sh. You want\nfloating submodules? Fine, we'll write a helper script that\nauto-commits to superproject everytime the SHA-1 changes. Everything\n_can_ be done.\n\nWhat exactly don't I \"understand\" about the current system, apart from\nthe fact that everybody is super-rigid and defensive about what\nalready works?\n\nLet us take a moment to look at the current state of git-submodule\n(note that this is after many years of hard work). This is just off\nthe top of my head:\n\n1. To add a submodule, you can't git add. You need to git submodule\nadd. And only from the toplevel directory. You can't first clone and\nthen add either: a git submodule add clones, adds lines to\n.gitmodules, AND stages everything.\n\n2. There is currently no way to remove a submodule. You have to git\nrm it, remove the lines in .gitmodules, and remove the GITDIR from\n.git/modules.\n\n3. It is currently impossible to git mv a submodule, because of the\namount of gymnastics required to relocate the object store, rewrite\nthe .gitmodules and stage the correct changes.\n\n4. It is currently impossible to do true floating submodules, because\nwe're using a commit object in-tree.\n\n5. You have to execute all submodule commands from the toplevel of the worktree.\n\n6. It is currently impossible to initialize a nested submodule without\ninitializing the container submodule. If I really want this, I have\nto trade-off composability and use repo.\n\nWhat is going on? Either the people working on git-submodule are\nhorribly incompetent, or there's some fundamental problem. I believe\nthe problem is the latter and have tried to show that the above quirks\ncan be fixed in a much simpler way with two days of work. What part\nof this didn't you understand?\n\n> Sorry, but that is not how open source works in general, and\n> certainly not how this project works.\n>\n> We do not add disruptive change just for the sake of changing it to\n> break a working system, make an extra work to clean up the fallout\n> for ourselves (i.e. your \"40 to 50 patch series\", but honestly\n> speaking I expect it would be more like a 4 months work for a full\n> time engineer or two), for unproven design (that has not yet to be\n> illustrated) to solve problems (that has not yet to be explained),\n> without knowing that\n>\n> (1) the problems are worth solving;\n>\n> (2) the design will solve the problems; and\n>\n> (3) solving the same problems without such a disruptive change is\n> impossible, or so cumbersome that it will be far larger than\n> the work needed to clean-up the fallout of the disruptive\n> change.\n>\n> So what are your X, Y, Z? You still haven't answered that question.\n\n(1) The ones that are currently solved by various existing\nimplementations. repo, mr, gitslave, git-subtree and git-submodule.\n\n(2) Currently, I'm targeting making the life of git-submodule.sh\nsimpler, fixing the UI/UX, and adding a few new features: floating\nsubmodules, refs for submodules, and blocking statthrough. Isn't this\na definite improvement over the current design? Why are you asking me\nto investigate and solve every problem exhaustively now?\n\n(3) Nothing is impossible. It's cumbersome, and that's what I'm\ntrying to answer with my series: a little bit of code written in two\ndays can simplify a lot of things. How do I give a definite answer to\nthis question without submitting two different series: one fixing\nsubmodule.c/ git-submodule.sh to do everything I want, and another to\nfix everything using my approach?\n\n> What are the real advantages? How are they used? What do they\n> allow us to do what we cannot do with .gitmodules (or repo or\n> gitslave for that matter)? What do they buy us?\n\nFor the 100th time, there's nothing you _cannot_ do with .gitmodules.\nI'm not solving any new problems. They're all solved by using a\ncombination of existing tools: each come with a specific set of\nbenefits and trade-offs. I'm trying to engineer an simple and elegant\nsolution that will solve many of those problems natively in git. They\nbuy us simplicity and elegance (I know you especially hate the word\n\"elegant\", but I have no other way to put it).\n\n> I have this suspicion that you do not have to change anything in the\n> object layer to make Git behave very differently from the current\n> submodule implementation.\n\nYes, you can! For the 101th time.\n\n> For example, if your gripe were (I am\n> just speculating without any input from you in this thread) that\n> each submodule working tree has \".git\" at their top and there is no\n> unified view from the top-level [*1*], we certainly can solve it\n> without any change to the object layer.\n\nI don't know where you got that idea from (certainly not from reading\nmy series): that is not my gripe at all. As I've already stated, my\ngripe is with how unnecessarily complicated, inelegant, and\nfeatureless submodule.c/ git-submodule.sh is.\n\n> We currently add a cache entry that has the commit object name to\n> the index from the tree object when we check out the superproject,\n> and create a separate repository with a working tree when we\n> instantiate a submodule.\n\nYes, I'm aware.\n\n> You could arrange a single index (the one in the superproject) to\n> hold the tree contents from the commit in the submodule, while\n> noting the original commit object name in a new mandatory extension\n> section in the index. The index will have a unified view of the\n> whole tree, and we do not have to have a .git at the root of each\n> submodule working tree (be it a directory or a gitfile).\n\nI think this is a very bad idea, because the toplevel index (and\ncombined object store) will blow up when we have lots of big\nsubmodules. One of my goals for the new submodule design is to answer\nthe scaling problem with ultra-large repositories: the answer is to\nbreak them up into smaller ones and compose them using this beautiful\nand powerful mechanism.\n\n> I think the message where I talked about the \"bind\" idea in the list\n> archive URLs I gave you earlier would give you such a layout, and\n> you should go read it again to understand how the flow from object\n> database to index to working tree back to index back to object\n> database was envisioned to work. I think the only thing we need to\n> do differently from that \"bind\" proposal in the current world order\n> is not to record the new submodule state in the commit object of the\n> superproject, but actually create a new commit for the submodule\n> part and store it in the tree object for the commit in the\n> superproject (the \"bind lines in the superproject commit\" was a\n> hack, only because we didn't have a way to write the submodule\n> commit object name in the index and in the trees).\n\nI don't see how this is relevant to our discussion, but anyway:\n\nYes, I read about your bind idea back from 2006. TL;DR version for\neveryone else reading: Junio proposed that the commit object be\nextended in the following way in January 2006.\n\ntree 04803b09c300c8325258ccf2744115acc4c57067\nbind 5b2bcc7b2d546c636f79490655b3347acc91d17f linux-2.6/\nbind 0bdd79af62e8621359af08f0afca0ce977348ac7 appliance/\nauthor Junio C Hamano 1137965565 -0800\ncommitter Junio C Hamano 1137965565 -0800\n\nThe bind lines are referring to tree objects. There's a reason the\nlink infrastructure written in 2007 by Linus made no reference to\nthis: it's a bad idea. It's a much better idea to compose using\ncommits. Even better to compose using a specialized link object. Why\nare you taking one step back from the current implementation?\n\nSo, my final question is: what do I have to do to convince you that\nthis approach shows promise? Haven't I answered the questions you\nkeep repeating: \"what problem does it solve?\" and \"why are existing\nimplementations fundamentally unworkable?\". I really don't know what\nmore to say, so can you give me a list of concrete actionable items\ninstead of repeating the same questions?\n"},{"id":"213416","messageId":"CALkWK0=jrO9MQotB+anAn-5YtG1RBTLWEnGL5=K4wy=yi7E-CQ@mail.gmail.com","threadId":"33386","inReplyTo":"7vr4im7m1y.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T15:51:44Z","receivedAt":"2013-04-07T15:51:44Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"I suspect you're overtly worried about the fallout of such a\ndisruptive change. If so, you could've just said: \"Ram, I like the\nidea. But what breakages do you estimate we'll have to deal with?\"\ninstead of attacking the idea and repeatedly questioning its purpose.\nSo, I'll make a rough guess based on the first iteration I intend to\nget merged:\n\n- Not all the git submodule subcommands will work. add/ status/ init/\ndeinit are easy to rewrite, but stuff like --recursive and foreach\nmight be slightly problematic as I already pointed out earlier. We'll\nhave to code depending on how far you think the first iteration should\ngo. After a few iterations, we can make 'git submodule' just print\n\"This command is deprecated. Please read `man gitsubmodules`.\"\n\n- All existing repositories with submodules will not be supported. My\nplan to deal with this: Have git-core code detect commit objects\nin-tree and disable things like diff. As soon as the user executes\nthe first 'git submodule' command, remove all existing submodules,\nalong with .gitmodules and re-add them as link objects. Then print a\nmessage saying: \"We've just migrated your submodules to the new\nformat. Please commit this.\"\n\nThat's really it. It's certainly not earth-shattering breakage; and I\nthink the inconvenience it causes is more than compensated by its\nbeautiful design and UI/UX.\n"},{"id":"213418","messageId":"20130407161209.GG2222@serenity.lan","threadId":"33386","inReplyTo":"CALkWK0=jrO9MQotB+anAn-5YtG1RBTLWEnGL5=K4wy=yi7E-CQ@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-04-07T16:12:09Z","receivedAt":"2013-04-07T16:12:09Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Apr 07, 2013 at 09:21:44PM +0530, Ramkumar Ramachandra wrote:\n> I suspect you're overtly worried about the fallout of such a\n> disruptive change. If so, you could've just said: \"Ram, I like the\n> idea. But what breakages do you estimate we'll have to deal with?\"\n> instead of attacking the idea and repeatedly questioning its purpose.\n> So, I'll make a rough guess based on the first iteration I intend to\n> get merged:\n> \n> - Not all the git submodule subcommands will work. add/ status/ init/\n> deinit are easy to rewrite, but stuff like --recursive and foreach\n> might be slightly problematic as I already pointed out earlier. We'll\n> have to code depending on how far you think the first iteration should\n> go. After a few iterations, we can make 'git submodule' just print\n> \"This command is deprecated. Please read `man gitsubmodules`.\"\n> \n> - All existing repositories with submodules will not be supported. My\n> plan to deal with this: Have git-core code detect commit objects\n> in-tree and disable things like diff. As soon as the user executes\n> the first 'git submodule' command, remove all existing submodules,\n> along with .gitmodules and re-add them as link objects. Then print a\n> message saying: \"We've just migrated your submodules to the new\n> format. Please commit this.\"\n\nMeaning that every repository using submodules need to have a flag day\nwhen all of the people using it switch to the new Git version at once?\n\nHow happy do you expect users to be if they have to remember to use\ndifferent Git version to work on different repositories because some\nhave switched and some haven't?\n\n>From a user's point of view, the current submodule support mostly works\nvery well. Yes there are some annoyances (\"you are not at the top\nlevel\") and some more advanced features require a bit too much work\n(moving a submodule) but in normal usage it works very well in my\nexperience.\n\nI think you need a much better argument than \"it makes the\nimplementation more beautiful\" to convince users that a flag day is\nnecessary.\n\n> That's really it. It's certainly not earth-shattering breakage;\n\nFor most users, the migration you've outlined above is exactly that.\n\nEven looking just at commits sent to this list, I've seen users on\nversions of Git from 1.7.10 to builds from next/pu in just the last\nweek. Coordinating a flag day for even a slightly popular repository is\ngoing to cause a lot of pain.\n"},{"id":"213420","messageId":"CALkWK0kSF_q0o1V6BhO6X2jKAJQxNQ0c6MCi5o=jZdMwrba48g@mail.gmail.com","threadId":"33386","inReplyTo":"20130407161209.GG2222@serenity.lan","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T16:42:28Z","receivedAt":"2013-04-07T16:42:28Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"John Keeping wrote:\n> Meaning that every repository using submodules need to have a flag day\n> when all of the people using it switch to the new Git version at once?\n\nNo, I would be totally against a migration that involves a flag-day.\nWhat I meant is that having old-style submodule side-by-side with\nnew-style submodules is confusing (think about people using an older\nversion and getting confused), and that we should disallow it. Users\nwill still be able to use existing repositories with new versions of\ngit with a few caveats:\n1. They won't be able to add new new submodules without migrating all\nexisting submodules.\n2. git ls-tree will show the in-tree object incorrectly as a link (ie.\nnot commit).\n\nThat's about it, I think. Obviously, everyone working on the\nrepository has to upgrade to a new version of git before they can use\nnew-style submodules.\n\n> I think you need a much better argument than \"it makes the\n> implementation more beautiful\" to convince users that a flag day is\n> necessary.\n\nThere is no flag day necessary, and that is not my argument at all:\nnew-style submodules brings lots of new functionality to the table.\n"},{"id":"213423","messageId":"20130407170201.GH2222@serenity.lan","threadId":"33386","inReplyTo":"CALkWK0kSF_q0o1V6BhO6X2jKAJQxNQ0c6MCi5o=jZdMwrba48g@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-04-07T17:02:01Z","receivedAt":"2013-04-07T17:02:01Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Apr 07, 2013 at 10:12:28PM +0530, Ramkumar Ramachandra wrote:\n> John Keeping wrote:\n> > Meaning that every repository using submodules need to have a flag day\n> > when all of the people using it switch to the new Git version at once?\n> \n> No, I would be totally against a migration that involves a flag-day.\n> What I meant is that having old-style submodule side-by-side with\n> new-style submodules is confusing (think about people using an older\n> version and getting confused), and that we should disallow it. Users\n> will still be able to use existing repositories with new versions of\n> git with a few caveats:\n> 1. They won't be able to add new new submodules without migrating all\n> existing submodules.\n> 2. git ls-tree will show the in-tree object incorrectly as a link (ie.\n> not commit).\n> \n> That's about it, I think. Obviously, everyone working on the\n> repository has to upgrade to a new version of git before they can use\n> new-style submodules.\n\nSo not a flag day, but still some point at which the repository\ntransitions to \"will not work with Git older than version X\". And if\nyou need to add a new submodule then you cannot delay that transition\nany longer.\n\n> > I think you need a much better argument than \"it makes the\n> > implementation more beautiful\" to convince users that a flag day is\n> > necessary.\n> \n> There is no flag day necessary, and that is not my argument at all:\n> new-style submodules brings lots of new functionality to the table.\n\nI haven't seen anywhere a concise list of what functionality this is.\nDo you have a simple bulleted list of what new features this would\nallow?\n"},{"id":"213427","messageId":"CALkWK0nSxfEzP7KHZxGjmBYD7pX5aa3CbMt1qAGrz4tonrtHhA@mail.gmail.com","threadId":"33386","inReplyTo":"20130407170201.GH2222@serenity.lan","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T17:22:50Z","receivedAt":"2013-04-07T17:22:50Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"John Keeping wrote:\n> So not a flag day, but still some point at which the repository\n> transitions to \"will not work with Git older than version X\". And if\n> you need to add a new submodule then you cannot delay that transition\n> any longer.\n\nYes, that is true. I don't see any way out of this.\n\n> I haven't seen anywhere a concise list of what functionality this is.\n> Do you have a simple bulleted list of what new features this would\n> allow?\n\nSure, I'll write it out for you from an end-user perspective:\n\n0. Great UI/UX. No more cd-to-toplevel, and a beautiful set of native\ncommands that are consistent with the overall design of git-core.\nWhich means: clone (to put something in an unstaged place), add (to\nstage), and commit (to commit the change). There's now exactly one\nplace in your worktree (which is represented as one file in git; think\nof it a sort of symlink) to look in for all the information. git\ncat-link to figure out its parameters, git edit-link to edit\nits parameters: no more \"find the matching pwd in .gitmodules in\ntoplevel\". To remove a submodule, just git rm. And git mv works!\n\n1. True floating submodules. You can have a submodule checked out at\n`master` or `v3.1`: no more detached HEADs in submodules unless you\nwant fixed submodules. No additional cruft required to do the\nfloating: the information is native, in a link object.\n\n2. Initializing a nested submodule without having to initialize the\nouter one: no more repo XML nonsense. And it's composable: you don't\nneed to put the information about all submodules in one central place.\n\n3. Ability to have very many large submodule repositories without the\nperformance hit. It makes sense to block stat() from going through\nwhen you have floating submodules. This means that many levels of\nnesting are very easily possible.\n\n4. It's suddenly much easier to add new features to this\nimplementation. You don't need to do the kind of gymnastics you'd\nhave to do if you were hacking on submodule.c/ git-submodule.sh.\n\nThis is basically how \"great design\" plays out.\n"},{"id":"213442","messageId":"20130407175210.GI2222@serenity.lan","threadId":"33386","inReplyTo":"CALkWK0nSxfEzP7KHZxGjmBYD7pX5aa3CbMt1qAGrz4tonrtHhA@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-04-07T17:52:10Z","receivedAt":"2013-04-07T17:52:10Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Apr 07, 2013 at 10:52:50PM +0530, Ramkumar Ramachandra wrote:\n> Sure, I'll write it out for you from an end-user perspective:\n\nTo play Devil's Advocate for a bit...\n\n> 0. Great UI/UX. No more cd-to-toplevel, and a beautiful set of native\n> commands that are consistent with the overall design of git-core.\n> Which means: clone (to put something in an unstaged place), add (to\n> stage), and commit (to commit the change). There's now exactly one\n> place in your worktree (which is represented as one file in git; think\n> of it a sort of symlink) to look in for all the information. git\n> cat-link to figure out its parameters, git edit-link to edit\n> its parameters: no more \"find the matching pwd in .gitmodules in\n> toplevel\". To remove a submodule, just git rm. And git mv works!\n\nPresumably now without .git/config support, so I can't override the\nchecked-in settings without my own custom branch. Even carrying a dirty\nworking tree seems problematic here since a checked-out link object is a\ndirectory, which can't have information like the remote URL in it.\n\n> 1. True floating submodules. You can have a submodule checked out at\n> `master` or `v3.1`: no more detached HEADs in submodules unless you\n> want fixed submodules. No additional cruft required to do the\n> floating: the information is native, in a link object.\n\nCan't I do that now with \"submodule..branch\" and \"git submodule\nupdate --remote --rebase\" and friends?\n\n> 2. Initializing a nested submodule without having to initialize the\n> outer one: no more repo XML nonsense. And it's composable: you don't\n> need to put the information about all submodules in one central place.\n\nHow does this interact when there is the following structure:\n\n super\n `-- sub\n `-- subsub (specified by sub)\n\nand subsub is specified as a submodule in *both* super and sub but with\ndifferent settings. Do I get different behaviour depending on $PWD?\n\n> 3. Ability to have very many large submodule repositories without the\n> performance hit. It makes sense to block stat() from going through\n> when you have floating submodules. This means that many levels of\n> nesting are very easily possible.\n\nCan't I already control this to some degree? Certainly the following\ncommands take different amounts of time to run:\n\n git status\n git -c status.submodulesummary=true status\n\n> 4. It's suddenly much easier to add new features to this\n> implementation. You don't need to do the kind of gymnastics you'd\n> have to do if you were hacking on submodule.c/ git-submodule.sh.\n> \n> This is basically how \"great design\" plays out.\n"},{"id":"213445","messageId":"CALkWK0n=vtPT7aFn9+T+bRxUpfXG+mYvV29YKC=_OAampQXJSA@mail.gmail.com","threadId":"33386","inReplyTo":"20130407175210.GI2222@serenity.lan","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T18:07:02Z","receivedAt":"2013-04-07T18:07:02Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"John Keeping wrote:\n> On Sun, Apr 07, 2013 at 10:52:50PM +0530, Ramkumar Ramachandra wrote:\n>> Sure, I'll write it out for you from an end-user perspective:\n>\n> To play Devil's Advocate for a bit...\n\nYes!\n\n>> 0. Great UI/UX. No more cd-to-toplevel, and a beautiful set of native\n>> commands that are consistent with the overall design of git-core.\n>> Which means: clone (to put something in an unstaged place), add (to\n>> stage), and commit (to commit the change). There's now exactly one\n>> place in your worktree (which is represented as one file in git; think\n>> of it a sort of symlink) to look in for all the information. git\n>> cat-link to figure out its parameters, git edit-link to edit\n>> its parameters: no more \"find the matching pwd in .gitmodules in\n>> toplevel\". To remove a submodule, just git rm. And git mv works!\n>\n> Presumably now without .git/config support, so I can't override the\n> checked-in settings without my own custom branch. Even carrying a dirty\n> working tree seems problematic here since a checked-out link object is a\n> directory, which can't have information like the remote URL in it.\n\nSure you can have a dirty worktree. It's just like .gitmodules:\nthere's zero difference but for the fact that .gitmodules is\naccessible directly via your filesystem, while links are not.\n\n>> 1. True floating submodules. You can have a submodule checked out at\n>> `master` or `v3.1`: no more detached HEADs in submodules unless you\n>> want fixed submodules. No additional cruft required to do the\n>> floating: the information is native, in a link object.\n>\n> Can't I do that now with \"submodule..branch\" and \"git submodule\n> update --remote --rebase\" and friends?\n\nYes, but that is not true floating: you shouldn't have to be sorry and\nrebase. In new-style submodules, they're first class citizens (ie.\ntrue): you can just replace the SHA-1 with a ref in the link.\n\n>> 2. Initializing a nested submodule without having to initialize the\n>> outer one: no more repo XML nonsense. And it's composable: you don't\n>> need to put the information about all submodules in one central place.\n>\n> How does this interact when there is the following structure:\n>\n> super\n> `-- sub\n> `-- subsub (specified by sub)\n>\n> and subsub is specified as a submodule in *both* super and sub but with\n> different settings. Do I get different behaviour depending on $PWD?\n\nThis is a very fringe case that I haven't thought about. I don't know\nhow it will behave: I haven't built it yet (and don't have the entire\nimplementation in my head yet).\n\n>> 3. Ability to have very many large submodule repositories without the\n>> performance hit. It makes sense to block stat() from going through\n>> when you have floating submodules. This means that many levels of\n>> nesting are very easily possible.\n>\n> Can't I already control this to some degree? Certainly the following\n> commands take different amounts of time to run:\n>\n> git status\n> git -c status.submodulesummary=true status\n\nYou can't control the most fundamental thing, stat(): this is the\nprimary killer of performance on a large worktree. There is currently\nno way to block stat(): new-style submodules offers a way to configure\nwhich submodules to block the stat() on.\n"},{"id":"213446","messageId":"20130407182112.GJ2222@serenity.lan","threadId":"33386","inReplyTo":"CALkWK0n=vtPT7aFn9+T+bRxUpfXG+mYvV29YKC=_OAampQXJSA@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-04-07T18:21:12Z","receivedAt":"2013-04-07T18:21:12Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Apr 07, 2013 at 11:37:02PM +0530, Ramkumar Ramachandra wrote:\n> John Keeping wrote:\n> > On Sun, Apr 07, 2013 at 10:52:50PM +0530, Ramkumar Ramachandra wrote:\n> >> Sure, I'll write it out for you from an end-user perspective:\n> >\n> > To play Devil's Advocate for a bit...\n> \n> Yes!\n> \n> >> 0. Great UI/UX. No more cd-to-toplevel, and a beautiful set of native\n> >> commands that are consistent with the overall design of git-core.\n> >> Which means: clone (to put something in an unstaged place), add (to\n> >> stage), and commit (to commit the change). There's now exactly one\n> >> place in your worktree (which is represented as one file in git; think\n> >> of it a sort of symlink) to look in for all the information. git\n> >> cat-link to figure out its parameters, git edit-link to edit\n> >> its parameters: no more \"find the matching pwd in .gitmodules in\n> >> toplevel\". To remove a submodule, just git rm. And git mv works!\n> >\n> > Presumably now without .git/config support, so I can't override the\n> > checked-in settings without my own custom branch. Even carrying a dirty\n> > working tree seems problematic here since a checked-out link object is a\n> > directory, which can't have information like the remote URL in it.\n> \n> Sure you can have a dirty worktree. It's just like .gitmodules:\n> there's zero difference but for the fact that .gitmodules is\n> accessible directly via your filesystem, while links are not.\n\nI can't see how this gets me a dirty working tree. Since the link needs\nto be stored somewhere, I assume it's in the index; so I can have staged\nchanges, but not unstaged changes.\n\n> >> 1. True floating submodules. You can have a submodule checked out at\n> >> `master` or `v3.1`: no more detached HEADs in submodules unless you\n> >> want fixed submodules. No additional cruft required to do the\n> >> floating: the information is native, in a link object.\n> >\n> > Can't I do that now with \"submodule..branch\" and \"git submodule\n> > update --remote --rebase\" and friends?\n> \n> Yes, but that is not true floating: you shouldn't have to be sorry and\n> rebase. In new-style submodules, they're first class citizens (ie.\n> true): you can just replace the SHA-1 with a ref in the link.\n\nBut what happens if I make any changes on top? With --rebase and\n--merge I can specify exactly what I want to happen (and obviously if I\ndon't have any changes then whichever I choose simply sets my branch to\nthe upstream ref).\n\n> >> 2. Initializing a nested submodule without having to initialize the\n> >> outer one: no more repo XML nonsense. And it's composable: you don't\n> >> need to put the information about all submodules in one central place.\n> >\n> > How does this interact when there is the following structure:\n> >\n> > super\n> > `-- sub\n> > `-- subsub (specified by sub)\n> >\n> > and subsub is specified as a submodule in *both* super and sub but with\n> > different settings. Do I get different behaviour depending on $PWD?\n> \n> This is a very fringe case that I haven't thought about. I don't know\n> how it will behave: I haven't built it yet (and don't have the entire\n> implementation in my head yet).\n> \n> >> 3. Ability to have very many large submodule repositories without the\n> >> performance hit. It makes sense to block stat() from going through\n> >> when you have floating submodules. This means that many levels of\n> >> nesting are very easily possible.\n> >\n> > Can't I already control this to some degree? Certainly the following\n> > commands take different amounts of time to run:\n> >\n> > git status\n> > git -c status.submodulesummary=true status\n> \n> You can't control the most fundamental thing, stat(): this is the\n> primary killer of performance on a large worktree. There is currently\n> no way to block stat(): new-style submodules offers a way to configure\n> which submodules to block the stat() on.\n\nSo it would be something like per-submodule --untracked-files and\n--ignore-submodules settings? I can see that being useful.\n"},{"id":"213447","messageId":"CALkWK0nj2+PH47OhHmq89Tzi4tdwaJTKaEneAt_k2rwpWbCnuA@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0n=vtPT7aFn9+T+bRxUpfXG+mYvV29YKC=_OAampQXJSA@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T18:22:02Z","receivedAt":"2013-04-07T18:22:02Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Ramkumar Ramachandra wrote:\n> You can't control the most fundamental thing, stat(): this is the\n> primary killer of performance on a large worktree. There is currently\n> no way to block stat(): new-style submodules offers a way to configure\n> which submodules to block the stat() on.\n\nLet me try to put this in simpler language. When you run 'git status'\non your toplevel repository, old-style submodules runs git status on\neach of the submodule repositories also: this is because submodules\nwere traditionally fixed; therefore, if you forget to commit some\nchanges in submodules before you make the superproject commit, you\nmight break the build. In new-style submodules, you can prevent this\nfrom happening on a link-specific basis.\n"},{"id":"213448","messageId":"5161BC33.8060707@web.de","threadId":"33386","inReplyTo":"20130407182112.GJ2222@serenity.lan","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-07T18:34:27Z","receivedAt":"2013-04-07T18:34:27Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 07.04.2013 20:21, schrieb John Keeping:\n> On Sun, Apr 07, 2013 at 11:37:02PM +0530, Ramkumar Ramachandra wrote:\n>> John Keeping wrote:\n>>> On Sun, Apr 07, 2013 at 10:52:50PM +0530, Ramkumar Ramachandra wrote:\n>>>> 3. Ability to have very many large submodule repositories without the\n>>>> performance hit. It makes sense to block stat() from going through\n>>>> when you have floating submodules. This means that many levels of\n>>>> nesting are very easily possible.\n>>>\n>>> Can't I already control this to some degree? Certainly the following\n>>> commands take different amounts of time to run:\n>>>\n>>> git status\n>>> git -c status.submodulesummary=true status\n>>\n>> You can't control the most fundamental thing, stat(): this is the\n>> primary killer of performance on a large worktree. There is currently\n>> no way to block stat(): new-style submodules offers a way to configure\n>> which submodules to block the stat() on.\n> \n> So it would be something like per-submodule --untracked-files and\n> --ignore-submodules settings? I can see that being useful.\n\nRam is plain wrong here (just like he is on \"git rm\" and \"git mv\",\neven though the latter is currently still in pu). This use case is\nhandled by submodules for years now. Take a look at the \"ignore\"\nsetting in .gitmodules which give you full control of the stat()s\nin submodules, in addition you have the repo-wide option\ndiff.ignoreSubmodules.\n\nThe whole feature list is full of red herrings like this which\nhave nothing to do with the advantages of a new object, but talk\nabout UI issues which are easy to solve in both worlds.\n"},{"id":"213449","messageId":"CALkWK0nSChFmYZJbaWVYsn5gdaPMrRQ7zOGbcsHvAKUmYvxtLQ@mail.gmail.com","threadId":"33386","inReplyTo":"20130407182112.GJ2222@serenity.lan","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T18:37:05Z","receivedAt":"2013-04-07T18:37:05Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"John Keeping wrote:\n> I can't see how this gets me a dirty working tree. Since the link needs\n> to be stored somewhere, I assume it's in the index; so I can have staged\n> changes, but not unstaged changes.\n\nIf you have changes in the index, your worktree is classified as\n\"dirty\". See git-sh-setup.sh:require_clean_work_tree(). Yes, you\ncan't have unstaged changes, only untracked changes. Linus: I made a\nmistake here.\n\n> But what happens if I make any changes on top? With --rebase and\n> --merge I can specify exactly what I want to happen (and obviously if I\n> don't have any changes then whichever I choose simply sets my branch to\n> the upstream ref).\n\nHm, that is a useful thing to support: we'll have to record both the\nSHA-1 and the ref.\n\n> So it would be something like per-submodule --untracked-files and\n> --ignore-submodules settings? I can see that being useful.\n\nYes. Never mind; Jens just pointed out that I'm wrong about this.\n"},{"id":"213450","messageId":"CALkWK0mBW63P0i6OhuujmAYO99pxLsS=ffFeqw8gBcBDgUpOPg@mail.gmail.com","threadId":"33386","inReplyTo":"5161BC33.8060707@web.de","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T18:44:54Z","receivedAt":"2013-04-07T18:44:54Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jens Lehmann wrote:\n> Ram is plain wrong here (just like he is on \"git rm\" and \"git mv\",\n> even though the latter is currently still in pu). This use case is\n> handled by submodules for years now. Take a look at the \"ignore\"\n> setting in .gitmodules which give you full control of the stat()s\n> in submodules, in addition you have the repo-wide option\n> diff.ignoreSubmodules.\n\nOh, I didn't know about ignore in .gitmodules. Sorry about that.\n\n> The whole feature list is full of red herrings like this which\n> have nothing to do with the advantages of a new object, but talk\n> about UI issues which are easy to solve in both worlds.\n\nReally? git-submodule.sh was written in 2007, and does not have git\nmv or cd-to-toplevel restriction removed to date. What does that say\nabout git-submodule?\n\nI specifically said end-user's perspective. Why exactly would I be\ntalking about the advantages of the link object?\n"},{"id":"213452","messageId":"20130407185905.GK2222@serenity.lan","threadId":"33386","inReplyTo":"5161BC33.8060707@web.de","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-04-07T18:59:05Z","receivedAt":"2013-04-07T18:59:05Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Apr 07, 2013 at 08:34:27PM +0200, Jens Lehmann wrote:\n> The whole feature list is full of red herrings like this which\n> have nothing to do with the advantages of a new object, but talk\n> about UI issues which are easy to solve in both worlds.\n\nWith the clarifications Ram's provided in this thread, I think there are\nalso some important regressions in functionality in his proposal (at\nleast as it currently stands), particularly losing the .gitconfig\noverrides.\n\nThe only proposed change that seems to me to be impossible with the\ncurrent .gitmodules approach is the \"submodule in a non-initialized\nsubmodule\" feature, but I've never seen anyone ask for that and it seems\nlikely to open a whole can of worms where the behaviour is likely to\nvary with $PWD. The current hierarchical approach provides sensible\nencapsulation of repositories and is simple to understand: once you're\nin a repository nothing above its root directory affects you.\n\nIt doesn't seem to me that \"it's harder than I'd like to add a feature I\nwant\" is a good reason to subject all users of submodules to a lot of\npain migrating to some new implementation that doesn't work the way\nthey're used to and which will mean they have to deal with complaints\nwhen people using an older version of Git can't clone their repository\n(and I doubt we want this mailing list to be flooded with such\ncomplaints either).\n"},{"id":"213453","messageId":"CALkWK0=v6SD_zEw=taauYWvYD5pHzdrED0vrhuFHKeS2zfgxSw@mail.gmail.com","threadId":"33386","inReplyTo":"20130407185905.GK2222@serenity.lan","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T19:06:59Z","receivedAt":"2013-04-07T19:06:59Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"John Keeping wrote:\n> With the clarifications Ram's provided in this thread, I think there are\n> also some important regressions in functionality in his proposal (at\n> least as it currently stands), particularly losing the .gitconfig\n> overrides.\n\nIf we want the entire feature list in the very first iteration, it's\ngoing to be huge.\n\n> The only proposed change that seems to me to be impossible with the\n> current .gitmodules approach is the \"submodule in a non-initialized\n> submodule\" feature, but I've never seen anyone ask for that and it seems\n> likely to open a whole can of worms where the behaviour is likely to\n> vary with $PWD. The current hierarchical approach provides sensible\n> encapsulation of repositories and is simple to understand: once you're\n> in a repository nothing above its root directory affects you.\n\nThat can be implemented in the current submodule system too, fwiw.\n\n> It doesn't seem to me that \"it's harder than I'd like to add a feature I\n> want\" is a good reason to subject all users of submodules to a lot of\n> pain migrating to some new implementation that doesn't work the way\n> they're used to and which will mean they have to deal with complaints\n> when people using an older version of Git can't clone their repository\n> (and I doubt we want this mailing list to be flooded with such\n> complaints either).\n\nLike I've said before: there is nothing that _cannot_ be done with the\ncurrent submodule system. To see the real advantages of this new\nsubmodule system, you have to think like a developer, not an end-user.\n Focusing just on end-user happiness is a very myopic way to develop\nsoftware, and I think the git community is better than that.\n"},{"id":"213455","messageId":"CALkWK0kqE_s3QT7NPRFqTtmhWQDZj4YxsTw55eVTaVbT4H04tA@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0=v6SD_zEw=taauYWvYD5pHzdrED0vrhuFHKeS2zfgxSw@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T19:17:27Z","receivedAt":"2013-04-07T19:17:27Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Ramkumar Ramachandra wrote:\n>> The only proposed change that seems to me to be impossible with the\n>> current .gitmodules approach is the \"submodule in a non-initialized\n>> submodule\" feature, but I've never seen anyone ask for that and it seems\n>> likely to open a whole can of worms where the behaviour is likely to\n>> vary with $PWD. The current hierarchical approach provides sensible\n>> encapsulation of repositories and is simple to understand: once you're\n>> in a repository nothing above its root directory affects you.\n>\n> That can be implemented in the current submodule system too, fwiw.\n\nAnd yes, it's going to be very ugly: I imagine special refs pointing\nto blobs, or something like that. Then again, what do end-users care\nabout?\n"},{"id":"213457","messageId":"CALkWK0kA86YgonuXv=h2hA_hV8RfWPsBjtrk6OnHXiP=qDBjwg@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0=jrO9MQotB+anAn-5YtG1RBTLWEnGL5=K4wy=yi7E-CQ@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T19:26:56Z","receivedAt":"2013-04-07T19:26:56Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"This reminds me of the commit generation numbers thread.\n\n\"But how can we determine ancestry?\"\n\"Use the commit timestamp.\"\n\"But what if there are clock skews?\"\n\"Put in a slop.\"\n\nIt breaks existing stuff, and it's hard to show any end-user benefit.\nI fear this proposal will meet with the same fate.\n"},{"id":"213463","messageId":"5161D3C5.9060804@web.de","threadId":"33386","inReplyTo":"CALkWK0mBW63P0i6OhuujmAYO99pxLsS=ffFeqw8gBcBDgUpOPg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-07T20:15:01Z","receivedAt":"2013-04-07T20:15:01Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 07.04.2013 20:44, schrieb Ramkumar Ramachandra:\n> Jens Lehmann wrote:\n>> The whole feature list is full of red herrings like this which\n>> have nothing to do with the advantages of a new object, but talk\n>> about UI issues which are easy to solve in both worlds.\n> \n> Really? git-submodule.sh was written in 2007, and does not have git\n> mv or cd-to-toplevel restriction removed to date. What does that say\n> about git-submodule?\n\nThat there is still some work to do, which I never denied and am\nactively working on (see \"git mv\" support in pu, which tackles one\nof the UI issues you mentioned).\n\n> I specifically said end-user's perspective. Why exactly would I be\n> talking about the advantages of the link object?\n\nBecause they are all that matters when it comes to decide if a link\nobject should be introduced to replace the current model. We should\ndiscuss the differences in the UI that result from introducing such\nan object, not the stuff that is still missing from our current\nimplementation (as that has to be coded either way and can not be\ntaken in favor of either solution). And we can additionally also\ntalk about the differences in hacking on git, where I concede that\nputting everything into a single object could lead to shorter code\nthan having to consult a .gitmodules file for that (even though I\nbelieve these arguments are much less important than UI changes).\n\nJust to be sure: I think we agree that both approaches are capable\nof allowing all relevant use cases, because they store the same\ninformation?\n\nDisclaimer: I am not opposed to the link object per se, but after\nall we are talking about severely changing user visible behavior.\nSo I want to see striking evidence that we gain something from it,\ndiscussed separately from UI deficiencies of the current code (no\ncd-to-toplevel please ;-).\n\nSo I started putting together a list of advantages and one of\ndisadvantages of the new link object compared to the current model.\nWe can extend and refine that to see what your proposal would mean\nfor us. After all we are talking about severely changing user\nvisible behavior, so we need convincing reasons to do that.\n\n\nAdvantages:\n\n* Information is stored in one place, no need to lookup stuff in\n another file/blob.\n\n* Easier coding, as we find all information in a single object.\n\n(I did not forget to add the point that you currently need a\nchecked out work tree to access the .gitmodules file, as there is\nongoing work to read the configuration directly from the database)\n\n(Another advantage would be that it is easier to merge the link\nobject, but a - still to be coded - .gitmodules aware merge driver\nwould work just as well)\n\n\nDisadvantages:\n\n* Changes in user visible behavior, possible compatibility\n problems when Git versions are mixed.\n\n* Special tools are needed to edit submodule information where\n currently a plain editor is sufficient.\n\n* merge conflicts are harder to resolve and require special git\n commands, solving them in .gitmodules is way more intuitive\n as users are already used to conflict markers.\n\n* A link object has no unstaged counterpart that a file easily\n has. What would that mean for adding a submodule and then\n unstaging it (or how could we add a submodule unstaged, like\n you proposed in another email)?\n\n(I think when we also put the submodule name in the object we\ncould also retain the ability to repopulated moved submodules\nfrom their old repo, which is found by that name)\n\n\nI'm not saying that this list is complete, I just wrote down\nwhat came to mind. When we e.g. find workable solutions to the\nDisadvantages we can remove them from the list and append them\nin parentheses for later reference like I did here. Does that\nsound like a plan?\n"},{"id":"213470","messageId":"CALkWK0mNQesytoDwaVHb1cdY+EC=33WRCefpE03hwp_9za1Vbw@mail.gmail.com","threadId":"33386","inReplyTo":"5161D3C5.9060804@web.de","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T20:49:10Z","receivedAt":"2013-04-07T20:49:10Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jens Lehmann wrote:\n> Just to be sure: I think we agree that both approaches are capable\n> of allowing all relevant use cases, because they store the same\n> information?\n\nYes.\n\n> Disclaimer: I am not opposed to the link object per se, but after\n> all we are talking about severely changing user visible behavior.\n> So I want to see striking evidence that we gain something from it,\n> discussed separately from UI deficiencies of the current code (no\n> cd-to-toplevel please ;-).\n\nThe only mandatory user-visible behavior change is the absence of\n.gitmodules. The git submodule subcommand will be have to be present\nand made to work, whether we like it or not.\n\n> (I did not forget to add the point that you currently need a\n> checked out work tree to access the .gitmodules file, as there is\n> ongoing work to read the configuration directly from the database)\n\nRead the configuration from the database? How?\nAlso, I want refs quite badly: I really can't stand repo.\n\n> (Another advantage would be that it is easier to merge the link\n> object, but a - still to be coded - .gitmodules aware merge driver\n> would work just as well)\n\nIt's very simple to implement: if you turn it into a blob, you can\ndiff and merge as usual.\n\n> Disadvantages:\n>\n> * Changes in user visible behavior, possible compatibility\n> problems when Git versions are mixed.\n\nAgreed.\n\n> * Special tools are needed to edit submodule information where\n> currently a plain editor is sufficient.\n\nUm, I actually really like this. I don't want to cd-to-toplevel, open\nup my .gitmodules and look for the relevant section. And it's a very\nsimple tool: see the git cat-file that I posted earlier.\n\n> * merge conflicts are harder to resolve and require special git\n> commands, solving them in .gitmodules is way more intuitive\n> as users are already used to conflict markers.\n\nThere shouldn't be that many merge conflicts to begin with! It\nhappens because you've stuffed all the information into one gigantic\n.gitmodules. With links, life is *much* easier: you already have a\ntight buffer format and a predefined order in which the key/value\npairs will appear. But yes, we will require to grow git-core to merge\nlinks seamlessly.\n\n> * A link object has no unstaged counterpart that a file easily\n> has. What would that mean for adding a submodule and then\n> unstaging it (or how could we add a submodule unstaged, like\n> you proposed in another email)?\n\nAdding a submodule untracked (not unstaged) is possible, and is\ndefault: git clone gets the submodules, and you have to use git add to\nstage it. I agree that you can't edit-link and have an unstaged\nchange, but I really don't care about that.\n\n> (I think when we also put the submodule name in the object we\n> could also retain the ability to repopulated moved submodules\n> from their old repo, which is found by that name)\n\nHm, considering that the information is not present anywhere\n(certainly not in the tree), this is probably a good idea. We'd have\nthe history of the submodule's name too.\n\n> I'm not saying that this list is complete, I just wrote down\n> what came to mind. When we e.g. find workable solutions to the\n> Disadvantages we can remove them from the list and append them\n> in parentheses for later reference like I did here. Does that\n> sound like a plan?\n\nYes, good plan.\n"},{"id":"213474","messageId":"CALkWK0k_vmXZr-x8=ZctouWbuVgv-1sptC0WX2aJ+yYD-T8cxA@mail.gmail.com","threadId":"33386","inReplyTo":"5161D3C5.9060804@web.de","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T20:57:16Z","receivedAt":"2013-04-07T20:57:16Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jens Lehmann wrote:\n> * Easier coding, as we find all information in a single object.\n\nIt's not just the difference between a single location versus multiple\nlocations. It's about the core object code of git parsing links, as\nopposed to a fringe submodule.c/ submodule.sh parsing .gitmodules.\nWhen you push git-submodule.sh into core, you'll have to constantly\ncall functions to parse .gitmodules and get the information. With\nlinks, all that information is free, provided you've parsed the\nobject.\n"},{"id":"213468","messageId":"20130407210225.GL2222@serenity.lan","threadId":"33386","inReplyTo":"CALkWK0mNQesytoDwaVHb1cdY+EC=33WRCefpE03hwp_9za1Vbw@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-04-07T21:02:25Z","receivedAt":"2013-04-07T21:02:25Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Mon, Apr 08, 2013 at 02:19:10AM +0530, Ramkumar Ramachandra wrote:\n> Jens Lehmann wrote:\n> > * A link object has no unstaged counterpart that a file easily\n> > has. What would that mean for adding a submodule and then\n> > unstaging it (or how could we add a submodule unstaged, like\n> > you proposed in another email)?\n> \n> Adding a submodule untracked (not unstaged) is possible, and is\n> default: git clone gets the submodules, and you have to use git add to\n> stage it. I agree that you can't edit-link and have an unstaged\n> change, but I really don't care about that.\n\nI do. I quite often use \"git add -p\" to sort things out and submodules\ncurrently fit into that seamlessly: I can add the submodule and then\nwait until later to commit it, without needing to either clone and\nremember to \"submodule add\" later or commit and play with rebase.\n\nLosing the ability to do that is a major usability regression as far as\nI'm concerned.\n"},{"id":"213475","messageId":"CALkWK0=GtOae9sizcftgCoqnoAOuQdGVQrcr2tKf7OCLDgMVSg@mail.gmail.com","threadId":"33386","inReplyTo":"20130407210225.GL2222@serenity.lan","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T21:11:38Z","receivedAt":"2013-04-07T21:11:38Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"John Keeping wrote:\n> I do. I quite often use \"git add -p\" to sort things out and submodules\n> currently fit into that seamlessly: I can add the submodule and then\n> wait until later to commit it, without needing to either clone and\n> remember to \"submodule add\" later or commit and play with rebase.\n>\n> Losing the ability to do that is a major usability regression as far as\n> I'm concerned.\n\nI didn't realize people cared so deeply about this. Sure, we can\nemulate it: keeping the information in .git/link-specs/ doesn't sound\nlike a bad idea. While it may feel like a hack, I think it's the\nright approach if we want to support non-fs-backed objects in our\ntree.\n"},{"id":"213476","messageId":"20130407212342.GA19857@elie.Belkin","threadId":"33386","inReplyTo":"CALkWK0k_vmXZr-x8=ZctouWbuVgv-1sptC0WX2aJ+yYD-T8cxA@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-04-07T21:23:42Z","receivedAt":"2013-04-07T21:23:42Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n\n> It's about the core object code of git parsing links, as\n> opposed to a fringe submodule.c/ submodule.sh parsing .gitmodules.\n\nWhat's stopping the core object code of git parsing .gitmodules? What\nis the core object code? How does this compare to other metadata\nfiles like .gitattributes and .gitignore?\n"},{"id":"213471","messageId":"CALkWK0=nGP2J_EQM83Pv--5orAMdAPmk-3t5uF2pkJYScpM2KQ@mail.gmail.com","threadId":"33386","inReplyTo":"CAP8UFD3i2vc3OSAHRERpiPY7cRjqhkqcBN9hVW0QmMksnCPccw@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T21:24:00Z","receivedAt":"2013-04-07T21:24:00Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Christian Couder wrote:\n> About generation numbers, please have a look at the thread leading to this\n> message:\n>\n> http://thread.gmane.org/gmane.comp.version-control.git/177146/focus=177586\n>\n> In short, generation numbers were not such a good idea because there were\n> already existing ways to get around the problem and because there was no\n> simple way to implement them without breaking other things.\n\nThanks for the interesting read, Christian. I didn't follow the\ndiscussion closely, and only have a passing understanding/ interest in\nthe issue.\n\n> My opinion is that your proposal can only be accepted if it is also a\n> solution, or a big step toward a solution, to other difficult problems, like\n> for example narrow/subtree clones.\n\nHm, a link object referring to a tree object, as opposed to a\nrevision. I'll think about this for some time.\n\n> So you should try to improve it by looking for other important features it\n> could provide in a simple way.\n> This would prove, or at least be a good sign, that it is a fundamental\n> improvement to add a link object the way you describe it.\n\nI'll look for more submodule-like features to strengthen my case.\n"},{"id":"213472","messageId":"CALkWK0=Q-P-fGLmkoiV3_CJ43MNmFzpfkvFjNUwxB+zOsqTxmg@mail.gmail.com","threadId":"33386","inReplyTo":"20130407212342.GA19857@elie.Belkin","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-07T21:30:53Z","receivedAt":"2013-04-07T21:30:53Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jonathan Nieder wrote:\n> What's stopping the core object code of git parsing .gitmodules?\n\nNothing, except that it's perversely unnatural for object parsing code\nto parse something outside the object store.\n\n> What\n> is the core object code?\n\nparse_link_buffer(): the conventions have already been set by\nparse_blob_buffer(), parse_tree_buffer() etc.\n\n> How does this compare to other metadata\n> files like .gitattributes and .gitignore?\n\n.gitignore and .gitattributes are parsed in dir.c, where git \"treats\"\nworktree paths. It's quite nicely integrated.\n"},{"id":"213497","messageId":"5162763D.5030708@web.de","threadId":"33386","inReplyTo":"CALkWK0=Q-P-fGLmkoiV3_CJ43MNmFzpfkvFjNUwxB+zOsqTxmg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-08T07:48:13Z","receivedAt":"2013-04-08T07:48:13Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 07.04.2013 23:30, schrieb Ramkumar Ramachandra:\n> Jonathan Nieder wrote:\n>> What's stopping the core object code of git parsing .gitmodules?\n\nJust to clarify that: git core already does that. A \"git grep\ngitmodules_config\" shows it is parsed by some git core commands:\ncheckout, commit, the diff family and fetch. Others will follow\nin the recursive update series. And \"git mv\" support will teach\nthat command to manipulate the .gitmodules file (and I hope that\na patch teaching \"git rm\" to remove the section from .gitmodules\nwill be accepted in the near future).\n\n> Nothing, except that it's perversely unnatural for object parsing code\n> to parse something outside the object store.\n\nHmm, at least the unstaged .gitmodules file has to be parsed from\nthe file system. And Heiko's current work on parsing .gitmodules\ndirectly from the object store will help here too, right?\n\n>> How does this compare to other metadata\n>> files like .gitattributes and .gitignore?\n> \n> .gitignore and .gitattributes are parsed in dir.c, where git \"treats\"\n> worktree paths. It's quite nicely integrated.\n\nAnd .gitmodules is parsed in submodule.c where Git treats\n.gitmodules entries. So I don't see a problem here, what am I\nmissing?\n"},{"id":"213498","messageId":"CALkWK0m5iN8h0J1EkbMTJoXy6QHHzyTjRfPnhL3XMgi=NAi2eg@mail.gmail.com","threadId":"33386","inReplyTo":"5162763D.5030708@web.de","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T08:07:26Z","receivedAt":"2013-04-08T08:07:26Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jens Lehmann wrote:\n> Hmm, at least the unstaged .gitmodules file has to be parsed from\n> the file system.\n\nYou seem to be touting it as a distinct advantage. In my opinion,\n.gitmodules is a wart that needs to be done away with: it should _not_\nbe on the filesystem, just like a commit object isn't on the\nfilesystem. Getting links to unstage is two hours of work, tops. And\nI'm the one writing the whole thing, so I don't see what everyone else\nis complaining about.\n\n> And Heiko's current work on parsing .gitmodules\n> directly from the object store will help here too, right?\n\nOfcourse, you _can_ parse a blob into a struct. It's just extremely\ngross to treat a blob located in a certain tree path differently from\nother blobs. It's a perverse violation of git's fundamental design,\nand I'm strongly against such a change.\n\nWhat I still fail to understand is why you keep mentioning\nwork-in-progress. You've had five years in which you haven't been\nable to do things that I did in two days. Yes, you _can_ keep\n.gitmodules and hack around everything, but why do you _want_ to do\nthat? Preserving backward compatibility is not *that* important, in\nmy opinion.\n"},{"id":"213501","messageId":"20130408081959.GD20159@elie.Belkin","threadId":"33386","inReplyTo":"CALkWK0m5iN8h0J1EkbMTJoXy6QHHzyTjRfPnhL3XMgi=NAi2eg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-04-08T08:19:59Z","receivedAt":"2013-04-08T08:19:59Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Ram,\n\nRamkumar Ramachandra wrote:\n\n> In my opinion,\n> .gitmodules is a wart that needs to be done away with: it should _not_\n> be on the filesystem, just like a commit object isn't on the\n> filesystem.\n\nWhat do you think of .gitignore and .gitattributes? Should they be\nsomewhere other than the filesystem as well?\n\n[...]\n> What I still fail to understand is why you keep mentioning\n> work-in-progress. You've had five years in which you haven't been\n> able to do things that I did in two days.\n\nI don't think Jens had any obligation to work on submodules and\nnothing else for the last five years. ;-)\n\nIf you end up convincing others that your tools are worth working\non and those tools pleasantly take care of the same workflows that\nsubmodules do, then I imagine people will be happy to migrate.\n\nSpeaking only for myself, I actually prefer the submodule UI, despite\nnot being thrilled with the\nsingle-.gitmodules-file-at-the-root-of-the-worktree feature. So I\nwill not be working on your proposed redesign, unless it evolves\nenough to be as pleasant a UI as (the long proposed UI of) submodules.\n\nHope that helps,\nJonathan\n"},{"id":"213499","messageId":"20130408083712.GE20159@elie.Belkin","threadId":"33386","inReplyTo":"CALkWK0m5iN8h0J1EkbMTJoXy6QHHzyTjRfPnhL3XMgi=NAi2eg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-04-08T08:37:12Z","receivedAt":"2013-04-08T08:37:12Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n> Jens Lehmann wrote:\n\n>> Hmm, at least the unstaged .gitmodules file has to be parsed from\n>> the file system.\n>\n> You seem to be touting it as a distinct advantage.\n\nTo clarify what I said in a side thread: yes, as long as the submodule\nmetadata includes the hostname I am downloading a library from, having\nit in an ordinary file is an advantage.\n\nThe problem with URLs (and especially hostnames) is that they change.\nWhen my project's previous domain name is lost because the hosting\ncompany lost interest, I want to be able to grep for all instances of\nthat domain name in my project's documentation and metadata and change\nthem all at once with a simple command like the following:\n\n\tgit grep -l -F -e oldhost.example.com |\n\txargs sed -i -e s/oldhost.example.com/newhost.example.com/g\n\nWhen I clone a project with --no-recurse-submodules, I want to be able\nto see what other servers will be contacted when I run \"git checkout\n--recurse-submodules\". The current .gitmodules file lets me find that\nout with a simple, intuitive command:\n\n\tcat .gitmodules\n\nI might change some URLs locally, because I know that some project's\nupstream has moved.\n\n\tgit submodule init\n\tgit config --edit\n\nOn the other hand, the single .gitmodules file will be a pain to merge\nif multiple branches modify it. So I do look forward to a merge\nstrategy that deals more intelligently with its content, and wouldn't\nhave minded a design that split this information into multiple files\nif we were starting over.\n\nJonathan\n"},{"id":"213502","messageId":"CALkWK0nz878Xh9BFSCrtBGT=g3fys3uBy7QQs9cdWaFtoS8Rsg@mail.gmail.com","threadId":"33386","inReplyTo":"20130408081959.GD20159@elie.Belkin","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T09:08:09Z","receivedAt":"2013-04-08T09:08:09Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jonathan Nieder wrote:\n> What do you think of .gitignore and .gitattributes? Should they be\n> somewhere other than the filesystem as well?\n\nI would argue that .gitignore and .gitattributes are done right. They\nare integrated into a very mature part of git-core very well, and\ntheir nature is fundamentally different from that of .gitmodules.\n\n.gitignore and .gitattributes specify extended globs (see: wildmatch)\nrules to apply on the worktree, and can be in multiple places in the\nworktree. They apply strictly on the current worktree; they have\nnothing to do with the index, and have no interaction with other\nobjects in the repository. Now, you might argue that they should be\npart of the tree object, but I will disagree because they don't\noperate on concrete entries in the tree but rather extended globs that\nmatch worktree paths. .gitmodules, on the other hand, specifies\nfundamental repository composition: it should be a special object in\nthe tree precisely because it changes the fundamental meaning of one\nconcrete tree entry. It has nothing to do with path treatment in the\nworktree, and hence has nothing to do with .gitattributes\nor.gitignore.\n\n> I don't think Jens had any obligation to work on submodules and\n> nothing else for the last five years. ;-)\n\nI know. What I'm saying is that his current approach is just filled\nwith tons of unnecessary complexity, inelegance, and pain. This is\nevidenced by the fact that the current submodule system is pathetic\nafter five years of work (and I don't think the developers working on\nit were particularly incompetent or lazy).\n\n> If you end up convincing others that your tools are worth working\n> on and those tools pleasantly take care of the same workflows that\n> submodules do, then I imagine people will be happy to migrate.\n\nYes, I'm planning a strict superset of the current submodule system\nfeatures. After some thought, I've decided not to have any feature\nregressions in my first version for merge (although that means a lot\nof work for me).\n\n> Speaking only for myself, I actually prefer the submodule UI, despite\n> not being thrilled with the\n> single-.gitmodules-file-at-the-root-of-the-worktree feature. So I\n> will not be working on your proposed redesign, unless it evolves\n> enough to be as pleasant a UI as (the long proposed UI of) submodules.\n\nI'm very interested in building a pleasant UI. I've always been a\nperson who cares deeply about UI: this is evidenced by my recent\nremote.pushdefault patch, and my pull.autostash WIP.\n"},{"id":"213503","messageId":"CALkWK0nO-K1BOOxNG0--Gx8AP-HOfLEjrn_UrkkccwuL6A449Q@mail.gmail.com","threadId":"33386","inReplyTo":"20130408083712.GE20159@elie.Belkin","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T09:14:09Z","receivedAt":"2013-04-08T09:14:09Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jonathan Nieder wrote:\n> git grep -l -F -e oldhost.example.com |\n> xargs sed -i -e s/oldhost.example.com/newhost.example.com/g\n\nYes, I've had to do this too: in a proxied environment I had to\ns/git:\\/\\//https:\\/\\//. So yes, we will have features to operate on\nmultiple links at the same time. I'm thinking something fine-grained\nthat allows you to pick which links to operate on. It's currently a\nvague thought, and I'm not sure what the implementation will look\nlike.\n"},{"id":"213505","messageId":"CACsJy8BAQUms9sgNh5_VFOx0_9LkobytjDk33VB+U_J-S44o_A@mail.gmail.com","threadId":"33386","inReplyTo":"20130404185522.GM30308@google.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-04-08T10:10:34Z","receivedAt":"2013-04-08T10:10:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Apr 5, 2013 at 5:55 AM, Jonathan Nieder wrote:\n> Ramkumar Ramachandra wrote:\n>\n>> 1. 'git add' should not go past submodule boundaries. I should not be\n>> able to 'git add clayoven/' or 'git add clayoven/LICENSE'. In\n>> addition, the shell completion also needs to be fixed.\n>\n> Yep. This is a bug.\n\nI notice that this case is handled by git-add, but there is probably a\nbug somewhere. Ram, can you make a test case for this?\n"},{"id":"213506","messageId":"1365416809-4396-1-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"CACsJy8BAQUms9sgNh5_VFOx0_9LkobytjDk33VB+U_J-S44o_A@mail.gmail.com","subject":"[PATCH] t3700 (add): add failing test for add with submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T10:26:49Z","receivedAt":"2013-04-08T10:26:49Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"git add currently goes past submodule boundaries. Document this bug.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n t/t3700-add.sh | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/t/t3700-add.sh b/t/t3700-add.sh\nindex 874b3a6..a1ea050 100755\n--- a/t/t3700-add.sh\n+++ b/t/t3700-add.sh\n@@ -310,4 +310,18 @@ test_expect_success 'git add --dry-run --ignore-missing of non-existing file out\n \ttest_i18ncmp expect.err actual.err\n '\n \n+test_expect_failure 'git add should not go past submodule boundaries' '\n+\tmkdir submodule_dir &&\n+\t(\n+\t\tcd submodule_dir &&\n+\t\tgit init &&\n+\t\tcat >foo <<-\\EOF &&\n+\t\tSome content\n+\t\tEOF\n+\t\tgit add foo &&\n+\t\tgit commit -a -m \"Add foo\"\n+\t) &&\n+\tgit add submodule_dir/foo\n+'\n+\n test_done\n-- \n1.8.2.373.g961c512\n"},{"id":"213507","messageId":"CACsJy8CAW1h4rAOK_2+EAFEL28CKeqf4vqROsLByjZaMU3D-ow@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0nz878Xh9BFSCrtBGT=g3fys3uBy7QQs9cdWaFtoS8Rsg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-04-08T10:29:06Z","receivedAt":"2013-04-08T10:29:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Apr 8, 2013 at 7:08 PM, Ramkumar Ramachandra wrote:\n> Jonathan Nieder wrote:\n>> What do you think of .gitignore and .gitattributes? Should they be\n>> somewhere other than the filesystem as well?\n>\n> I would argue that .gitignore and .gitattributes are done right. They\n> are integrated into a very mature part of git-core very well, and\n> their nature is fundamentally different from that of .gitmodules.\n\nProbably off-topic, but I'm starting to find \".gitignore can be found\nin every directory\" a burden to day-to-day git operations. So imo it's\nnot done right entirely ;-)\n\n> .gitignore and .gitattributes specify extended globs (see: wildmatch)\n> rules to apply on the worktree, and can be in multiple places in the\n> worktree. They apply strictly on the current worktree; they have\n> nothing to do with the index, and have no interaction with other\n> objects in the repository.\n\nIndex operations sometimes read these .git{ignore,attributes}. I\nbelieve git-archive reads worktree's .gitattributes, so it's not\nreally just about worktree.\n\n>> I don't think Jens had any obligation to work on submodules and\n>> nothing else for the last five years. ;-)\n>\n> I know. What I'm saying is that his current approach is just filled\n> with tons of unnecessary complexity, inelegance, and pain. This is\n> evidenced by the fact that the current submodule system is pathetic\n> after five years of work (and I don't think the developers working on\n> it were particularly incompetent or lazy).\n\nI don't follow this thread closely, but I think there's a common\nground where improvements can benefit both approaches. There are a lot\nof problems for deep integration and erasing submodule's boundaries\nfrom UI perspective. I think maybe you can work on that first, gain\nexperience along the way, and maintain the link-object changes\nseparately. Maybe someday you will manage to switch .gitmodules with\nit. Or maybe I'm wrong (partly because I did not read the whole\nthread)\n-- \nDuy\n"},{"id":"213509","messageId":"CACsJy8CCepx5njpMo40608vHf5mQ_VfeZA_e5KDaH98V8_o0-w@mail.gmail.com","threadId":"33386","inReplyTo":"1365416809-4396-1-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-04-08T11:04:13Z","receivedAt":"2013-04-08T11:04:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Apr 8, 2013 at 8:26 PM, Ramkumar Ramachandra wrote:\n> +test_expect_failure 'git add should not go past submodule boundaries' '\n> + mkdir submodule_dir &&\n> + (\n> + cd submodule_dir &&\n> + git init &&\n> + cat >foo <<-\\EOF &&\n> + Some content\n> + EOF\n> + git add foo &&\n> + git commit -a -m \"Add foo\"\n> + ) &&\n> + git add submodule_dir/foo\n\nThanks. I yhink the last line above should be \"test_must_fail git add\n...\". I'm half way of fixing it (I think treat_leading_directory is a\nbit loose). Will continue tomorrow (or this weekend, depending on\n$DAYJOB)\n--\nDuy\n"},{"id":"213511","messageId":"CALkWK0mLG78anjbcVRF7Zpj-hMmts1pshF=XhLh--MCYV66Gng@mail.gmail.com","threadId":"33386","inReplyTo":"CACsJy8CAW1h4rAOK_2+EAFEL28CKeqf4vqROsLByjZaMU3D-ow@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T11:06:09Z","receivedAt":"2013-04-08T11:06:09Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Duy Nguyen wrote:\n> Probably off-topic, but I'm starting to find \".gitignore can be found\n> in every directory\" a burden to day-to-day git operations. So imo it's\n> not done right entirely ;-)\n\nWhy is it a burden? I would argue that the tooling support is not yet\nthere, but git check-ignore is a step in the right direction. What\nalternate design would you propose, just out of curiosity?\n\n> Index operations sometimes read these .git{ignore,attributes}. I\n> believe git-archive reads worktree's .gitattributes, so it's not\n> really just about worktree.\n\nI should've said \"largely, only affects the current worktree\".\n\n> I don't follow this thread closely, but I think there's a common\n> ground where improvements can benefit both approaches. There are a lot\n> of problems for deep integration and erasing submodule's boundaries\n> from UI perspective. I think maybe you can work on that first, gain\n> experience along the way, and maintain the link-object changes\n> separately. Maybe someday you will manage to switch .gitmodules with\n> it. Or maybe I'm wrong (partly because I did not read the whole\n> thread)\n\nYes, there is some common ground. But:\n\n1. The inspiration for fixing fundamental design problems comes from\nmy redesign. For instance, I would've never discovered the git add\nbug if I'd not attempted to git add (as opposed to the unnatural\nabstraction that git submodule add presents).\n\n2. I think it is absolutely imperative that we do the redesign now,\nbefore we've descended too far into the madness that the current\ndesign is. I think I'm capable of doing the redesign now, with some\nhelp and support from the list. My attitude doesn't align with the\n\"I'm feeling lazy; why don't we postpone it?\" argument. Let's finish\nwhat I started now: I'm more than willing to dedicate the next few\nmonths full-time towards finishing this and getting it merged.\n"},{"id":"213510","messageId":"CALkWK0=LW1850irkcJDkdMwKRf8-QrDotRY-ciuxy7KiY1kkyg@mail.gmail.com","threadId":"33386","inReplyTo":"CACsJy8CAW1h4rAOK_2+EAFEL28CKeqf4vqROsLByjZaMU3D-ow@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T11:10:13Z","receivedAt":"2013-04-08T11:10:13Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Duy Nguyen wrote:\n> Probably off-topic, but I'm starting to find \".gitignore can be found\n> in every directory\" a burden to day-to-day git operations. So imo it's\n> not done right entirely ;-)\n\nOr are you saying it's hard to implement elegantly and efficiently in\ngit-core? If so, I agree wholeheartedly. I'm not yet sure what to do\nabout the situation.\n"},{"id":"213512","messageId":"CACsJy8D-M3xMkO7CPmOzeMxRV0Bv6MtCjH5eCr_vguxPF2N0vg@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0mLG78anjbcVRF7Zpj-hMmts1pshF=XhLh--MCYV66Gng@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-04-08T11:29:46Z","receivedAt":"2013-04-08T11:29:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Apr 8, 2013 at 9:06 PM, Ramkumar Ramachandra wrote:\n> Duy Nguyen wrote:\n>> Probably off-topic, but I'm starting to find \".gitignore can be found\n>> in every directory\" a burden to day-to-day git operations. So imo it's\n>> not done right entirely ;-)\n>\n> Why is it a burden? I would argue that the tooling support is not yet\n> there, but git check-ignore is a step in the right direction. What\n> alternate design would you propose, just out of curiosity?\n\nYou don't know if .gitignore is there, so you need to check for it in\nevery directory. If we fixed its location (e.g. worktree's top) we\nwould not have to look in every directory. Then again it may be a bit\ninconvenient that way. If you remove a directory, you also remove\n.gitignore rules inside when you distribute .gitignore files.\nOtherwise you need to clean up top .gitignore once in a while.\n\n> 1. The inspiration for fixing fundamental design problems comes from\n> my redesign. For instance, I would've never discovered the git add\n> bug if I'd not attempted to git add (as opposed to the unnatural\n> abstraction that git submodule add presents).\n\nI actually spotted a similar use of git-add in the test suite [1]. You\nsee, it's a bug that should be fixed but in that particular case, it's\nvalid to add something \"inside a submodule\". I wanted to fix that with\nmy read_directory rewrite (part of the pathspec stuff) but never got\naround to finish it and eventually gave up, which leads to your next\npoint..\n\n> 2. I think it is absolutely imperative that we do the redesign now,\n> before we've descended too far into the madness that the current\n> design is. I think I'm capable of doing the redesign now, with some\n> help and support from the list. My attitude doesn't align with the\n> \"I'm feeling lazy; why don't we postpone it?\" argument. Let's finish\n> what I started now: I'm more than willing to dedicate the next few\n> months full-time towards finishing this and getting it merged.\n\nGood luck. Bug such a big work usually requires more than one\nvolunteer. If you haven't convinced (*) the community it's right,\nmaybe you should take a few days thinking about it again before\nimplementing.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/177454\n(*) just a feeling after a quick glance, I may be terribly wrong again\n--\nDuy\n"},{"id":"213513","messageId":"CALkWK0m93ZzWVJVUxgVQGUTK8t2pTEaDAPs7vW-tcgiewjOLwA@mail.gmail.com","threadId":"33386","inReplyTo":"CACsJy8D-M3xMkO7CPmOzeMxRV0Bv6MtCjH5eCr_vguxPF2N0vg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T11:53:14Z","receivedAt":"2013-04-08T11:53:14Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Duy Nguyen wrote:\n> Good luck. Bug such a big work usually requires more than one\n> volunteer. If you haven't convinced (*) the community it's right,\n> maybe you should take a few days thinking about it again before\n> implementing.\n\nYes, I'm thinking about it before rushing in to implement it.\n\nThere will always be resistance to change, especially when it involves\na change that breaks a working implementation. If anything, the\nresistance is only going to get worse with time, as people pile more\nand more hacks on top of the current submodule implementation. I say:\ndo it now, before we lose steam.\n\nAs far as I can tell, I'm completely unbiased: I have no vested\ninterests in either implementation, and I just want to see the best\nimplementation win. My conviction in the new approach has only\nstrengthened after discussions on this thread: there must be some\nreason for that, no?\n\nFrankly, I was hoping that atleast one or two people on the thread\nwould take my side of the argument (or atleast tell me that I'm not\nderanged), but that hasn't happened. Nevertheless, I hope to convince\nmore people by doing more work and posting a beautifully working\nimplementation.\n\nI'm already prepared for the worst case: I'll be forced to dump all my\nwork and be disappointed with the git community.\n"},{"id":"213519","messageId":"7v61zx13nv.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"20130408083712.GE20159@elie.Belkin","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-08T14:46:12Z","receivedAt":"2013-04-08T14:46:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n[snipped everything I agree with...]\n\n> On the other hand, the single .gitmodules file will be a pain to merge\n> if multiple branches modify it. So I do look forward to a merge\n> strategy that deals more intelligently with its content, and wouldn't\n> have minded a design that split this information into multiple files\n> if we were starting over.\n\nI find it a sensible suggestion to have a content-aware merge\ndriver. Such a custom merge driver to help merging a structured\ndatafile in the config format will have other uses when we need to\ndo more than the current system (outside submodules there will be\nother things \"frotz\" that need \"information about frotz\" in the\nfuture, and a .gitfrotz file would be one possible way to do so).\n\nI do not think it needs to be split per-submodule.\n\nWhen a submodule in the common ancestor was at path dirA/, and you\nare merging with another branch that moved it to path dirB/, the\ncontents of .gitmodules file for that module (that is identified by\nits ) will need a three-way merge of its .path element:\n\n common ancestor: submodule..path = dirA/\n ours: submodule..path = dirA/\n theirs: submodule..path = dirB/\n\nAnd your content-aware merge driver should be able to do the\nresolving by following the usual three-way merge rules. We started\nfrom the same dirA/ and only they changed, so the result is dirB/.\n\nBy the way, that's a \"merge driver\" (which deals with per-path\ncontent merge), not a strategy (which deals with the entire tree\nlevel merge).\n"},{"id":"213521","messageId":"7v1ual12pj.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0m93ZzWVJVUxgVQGUTK8t2pTEaDAPs7vW-tcgiewjOLwA@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-08T15:06:48Z","receivedAt":"2013-04-08T15:06:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> As far as I can tell, I'm completely unbiased: I have no vested\n> interests in either implementation,...\n> ...\n> Frankly, I was hoping that atleast one or two people on the thread\n> would take my side of the argument (or atleast tell me that I'm not\n> deranged), but that hasn't happened.\n\nAren't these two quite contradicting?\n\nAfter listening to what others tell one with an unbiased mind and\nfinding that nobody agrees with what one initially proposed, an\nunbiased person would step back, take a deep breath and think again,\nbefore insulting the intelligence of others with a \"dissapointed\",\nlike this:\n\n> I'm already prepared for the worst case: I'll be forced to dump all my\n> work and be disappointed with the git community.\n\nWould it be possible that (at least some part of, or possibly all\nof) your ideas had some merit, but with all your hostility against\nthe current system and the work that went behind it, you did not\ncommunicate well enough to make others understand you?\n\nWhat I found very hard to read in this thread was that your messages\nall went like this:\n\n 1. In the current system, I have to be at the top level of a\n submodule to work in it (or some other problems).\n\n 2. I will fix it in a more \"elegant\" way.\n\n 3. I have to have a new object at the submodule path, not the\n current \"submodule is a commit bound at the submodule path, and\n information about the submodule is in .gitmodules\".\n\nThere was very little concrete explanation on how #3 leads to #2,\ni.e. the overall design of your new system and how it will work,\nother than you would read what we currently write in .gitmodules\nfrom a new kind of object.\n\nWhen an alternative solution was suggested, all your responses were\nfull of subjective \"inelegant\" and \"ugly\", and at least I couldn't\nread much substance in it (here, the usual me might add \"maybe\nothers differ\", but after reading this thread, I strongly suspect\nthat others share this problem).\n"},{"id":"213524","messageId":"7vtxnhysb8.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CACsJy8CCepx5njpMo40608vHf5mQ_VfeZA_e5KDaH98V8_o0-w@mail.gmail.com","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-08T15:07:23Z","receivedAt":"2013-04-08T15:07:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On Mon, Apr 8, 2013 at 8:26 PM, Ramkumar Ramachandra wrote:\n>> +test_expect_failure 'git add should not go past submodule boundaries' '\n>> + mkdir submodule_dir &&\n>> + (\n>> + cd submodule_dir &&\n>> + git init &&\n>> + cat >foo <<-\\EOF &&\n>> + Some content\n>> + EOF\n>> + git add foo &&\n>> + git commit -a -m \"Add foo\"\n>> + ) &&\n>> + git add submodule_dir/foo\n>\n> Thanks. I yhink the last line above should be \"test_must_fail git add\n> ...\". I'm half way of fixing it (I think treat_leading_directory is a\n> bit loose). Will continue tomorrow (or this weekend, depending on\n> $DAYJOB)\n\nThis is a good thing to test (to make sure \"add\" fails, as you\npointed out). Can you include a fixed one in your series?\n\nThanks.\n"},{"id":"213527","messageId":"CALkWK0=2T5BA8+dro8U7bcp3ODsuQAj78mEPnxs=ERqf9FhSBQ@mail.gmail.com","threadId":"33386","inReplyTo":"7v1ual12pj.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T16:08:12Z","receivedAt":"2013-04-08T16:08:12Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> Would it be possible that (at least some part of, or possibly all\n> of) your ideas had some merit, but with all your hostility against\n> the current system and the work that went behind it, you did not\n> communicate well enough to make others understand you?\n\nAgreed. My annoyance with the current system did go a little\noverboard, and I've been having a splitting headache for the last few\ndays.\n\n> What I found very hard to read in this thread was that your messages\n> all went like this:\n>\n> 1. In the current system, I have to be at the top level of a\n> submodule to work in it (or some other problems).\n>\n> 2. I will fix it in a more \"elegant\" way.\n>\n> 3. I have to have a new object at the submodule path, not the\n> current \"submodule is a commit bound at the submodule path, and\n> information about the submodule is in .gitmodules\".\n>\n> There was very little concrete explanation on how #3 leads to #2,\n> i.e. the overall design of your new system and how it will work,\n> other than you would read what we currently write in .gitmodules\n> from a new kind of object.\n\nI had no way of expressing what I wanted to do except by writing code\nwhen I started off this thread, but am in much better shape now. Let\nme try to explain my fundamental assumptions and code in a concise way\nnow.\n\n1. Having a toplevel .gitmodules means that any git-core command like\nadd/ rm/ mv will be burdened with looking for the .gitmodules at the\ntoplevel of the worktree and editing it appropriately along with\nwhatever it was built to do (ie. writing to the index and committing\nit). This is highly unnatural. Putting the information in link\nobjects means that we get a more natural UI + warts like\ncd-to-toplevel disappear with no extra code.\n\n2. If we want to make git-submodule a part of git-core (which I think\neveryone agrees with), we will need to make the information in\n.gitmodules available more easily to the rest of git-core. One way to\ndo it without breaking anything is to unpack the root tree, look for\nan entry with the path .gitmodules and handle it different from other\nblobs: ie. parse it into structured data that the rest of git-core can\nconsume. However, I think it is very gross as the blob is not\ninherently special in any way: it's just incidentally stored at a\nspecific tree path. The alternative is to have an inherently special\nkind of blob (ie. link object). In the git-core code, I can simply\nmatch for a link object and operate on it accordingly. As opposed to\nmatching a blob object, and its tree path. Moreover, this means that\nthe user can simply git edit-link from anywhere in the worktree\ninstead of having to refer to the appropriate section in the toplevel\n.gitmodules.\n\n3. Currently diffing/ merging one huge .gitmodules file is a mess, as\nit doesn't have to conform to a strict format. This means that I can\nget conflicts between these two:\n\n url = gh:artagnon/clayoven\n url =gh:artagnon/clayoven\n\nMoreover, since the fields are not ordered, a simple reordering of the\nfields will cause a merge conflict. The correct way to fix this is to\nsplit up .gitmodules into many logical files, have a git\nedit-gitmodules which reduces user input to a strict format, and then\nwrite custom diff/merge drivers. My proposal involves having a git\nedit-link, and teaching git-core to diff/merge appropriately. The\ninformation is already in logical bits.\n\n4. The only seeming disadvantage of not having a file accessible via\nthe filesystem is that it doesn't behave like a full blob. But it\ndoes; the code to unstage a link object (emulation) is actually very\nsimple: I'm currently writing it.\n\n5. Having a first-class link object comes with functional advantages.\nIt means that I can have a ref pointing to link objects and easily\ninitialize a nested repository without having to initialize the\ncontaining repository (ie. essentially replacing repo). We can have\ntrue floating submodules, which is really nice in my opinion: you can\nfix a library at v3.1 and switch it to v3.2 at some point in the\nfuture without using ugly SHA-1 hexes anywhere.\n\n6. While it is possible to work top-down from the current system, that\napproach is clearly taking too long and is too painful. This explains\nwhy submodules haven't come a long way in the last five years. With\nmy approach, I'm trying to make life simpler for everyone: it will\nsuddenly become much easier to hack on submodules, and it can improve\nmore rapidly over the next five years. I'm not thinking about\nshort-term fixes precisely for this reason: the long-term goal is\nworth a little bit of short-term inconvenience.\n\n7. I estimate that replacing the current submodule system without\nfeature regressions will not take a lot of effort and can be done with\nminimal breakages. It's not a lot of code or anything very complex.\nWe just have to follow along the lines of how git-core handles blobs,\nand write a little bit of code to make links behave like blobs (I'm\nhalfway done with this already).\n"},{"id":"213532","messageId":"CALkWK0kJ=r6Lg1Z9kUexMtVnori5zNj9d2M=u5EAcdHOHNOCOw@mail.gmail.com","threadId":"33386","inReplyTo":"CAP8UFD16gwWjE7T75D7kUM-VOXhtZaSRGtEg8fW5kmuKDLTQHQ@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T17:04:26Z","receivedAt":"2013-04-08T17:04:26Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Christian Couder wrote:\n> What if instead of a git submodule I want to have an hg, or, God/Linus/deity\n> forbid, an SVN submodule, inside my git worktree?\n> What if I just want a very big movie or .tgz downloaded from somewhere else?\n\nSince the link object is rooted to the tree, it's impossible to have\nanything but a working copy in the link directory. How can I have a\nnon-git-worktree link directory without breaking checkout? I think\nthat making it too generic will make the entire submodule experience\nsuffer, because the implementation must be coded according to the\nlowest-common-denominator. This is the mistake that the tool mr\nmakes: since it's so generic, it can't provide very powerful\nfunctionality specifically for git repositories.\n\nI'll try to think of something else.\n"},{"id":"213533","messageId":"7vy5ctx7xj.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"7v61zx13nv.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-08T17:12:56Z","receivedAt":"2013-04-08T17:12:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Jonathan Nieder writes:\n>\n> [snipped everything I agree with...]\n>\n>> On the other hand, the single .gitmodules file will be a pain to merge\n>> if multiple branches modify it. So I do look forward to a merge\n>> strategy that deals more intelligently with its content, and wouldn't\n>> have minded a design that split this information into multiple files\n>> if we were starting over.\n>\n> I find it a sensible suggestion to have a content-aware merge\n> driver. Such a custom merge driver to help merging a structured\n> datafile in the config format will have other uses when we need to\n> do more than the current system (outside submodules there will be\n> other things \"frotz\" that need \"information about frotz\" in the\n> future, and a .gitfrotz file would be one possible way to do so).\n>\n> I do not think it needs to be split per-submodule.\n\nAnother thing to think about is what to do when/if we want to\nexpress \"this is the default that applies to all submodules\". For\nexample, a superproject that binds multiple submodules may want to\nsay \"When on this branch, make all submodules also on 'next'\".\n\nWith a unified single place that holds information about all\nsubmodules, it is trivial to add a \"default\" section, perhaps like\nthis:\n\n\t[default]\n\t\tbranch = next\n\t[submodule \"framework\"]\n\t\turl = ...\n path = framework\n\t[submodule \"common\"]\n\t\turl = ...\n path = common\n branch = master ;# regardless of other modules...\n\non top of the \"submodule..branch\" mechanism for floating\ncheckout (the \"default\" is of course not limited to \"branch\" but\napplies in general).\n\nIt is not obvious where such a \"default\" piece should go once you\nstart splitting these into per-submodule files, be it a separate but\nstill in-tree file that is different from the submodule it desribes,\nor a blob-like object that sits at the path for the submodule in the\ntree and in the index as Ram wants to do (as I kept saying, the\nstorage mechanism is not fundamental).\n\nThis is similar to why .gitattributes is easy to work with, I think.\nYou can describe the information about paths in that file (which\nlives at a place different from the paths that are described), and\nyou can have a catch-all rule in it.\n\nThis is a tangent, but you could build a system that attaches\nattributes to individual paths and hide the attributes from the\nworking tree filesystem (think: svn:blah) and have a set of special\ncommands (think: svn propset, proplist, etc.) to work with them, and\nthat is an equally valid way to implement attributes (it does not\nmake .gitattributes less valid way to do so, though).\n"},{"id":"213568","messageId":"7va9p8yjuf.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0=2T5BA8+dro8U7bcp3ODsuQAj78mEPnxs=ERqf9FhSBQ@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-08T18:10:16Z","receivedAt":"2013-04-08T18:10:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> 1. Having a toplevel .gitmodules means that any git-core command like\n> add/ rm/ mv will be burdened with looking for the .gitmodules at the\n> toplevel of the worktree and editing it appropriately along with\n> whatever it was built to do (ie. writing to the index and committing\n> it).\n\nBurdened is a subjective word. What's bad about having a single place\nyou know you can read and find out information about things? You\nhave to learn about them to do anything specific to them anyway.\n\n> This is highly unnatural.\n\nUnnatural is a subjective word, and there is no justification I see\nhere in your message.\n\n> Putting the information in link\n> objects means that we get a more natural UI + warts like\n> cd-to-toplevel disappear with no extra code.\n\nI do not see how \"link objects\" _means_ \"natural UI\", yet, without\nan explanation how one leads to the other.\n\nWhat does cd-to-toplevel have anything to do with it? In case you\ndid not notice, all the core commands internally cd-to-toplevel and\ncarry the \"prefix\" information while doing so, and prepend the\nprefix to user-supplied paths to find which path the user is talking\nabout. So \"cd to toplevel before starting to carry the operation out\"\nis a natural pattern inside Git. As many people already told you,\n\"the user has to run 'git submodule' from the top-level of the\nsubmodule working tree\" is a simple oversight of the implementation.\n\n> 2. If we want to make git-submodule a part of git-core (which I think\n> everyone agrees with), we will need to make the information in\n> .gitmodules available more easily to the rest of git-core.\n\nCare to define \"more easily\" which is another subjective word? The\n.gitmodules file uses the bog-standard configuration format that can\nbe easily read with the config.c infrastructure. It is a separate\nmatter that git_config() API is cumbersome to use, but improving it\nwould help not just .gitmodules but also the regular non-submodule\nusers of Git. There is a topic in the works to read data in that\nformat from core Heiko is working on.\n"},{"id":"213586","messageId":"CALkWK0m4o=F=bh_mt9NGDfXKExgVtLhaZ7uxA_XFhpF5AN5-Fg@mail.gmail.com","threadId":"33386","inReplyTo":"7va9p8yjuf.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T19:03:23Z","receivedAt":"2013-04-08T19:03:23Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> Ramkumar Ramachandra writes:\n>\n>> 1. Having a toplevel .gitmodules means that any git-core command like\n>> add/ rm/ mv will be burdened with looking for the .gitmodules at the\n>> toplevel of the worktree and editing it appropriately along with\n>> whatever it was built to do (ie. writing to the index and committing\n>> it).\n>\n> Burdened is a subjective word. What's bad about having a single place\n> you know you can read and find out information about things? You\n> have to learn about them to do anything specific to them anyway.\n\n\"Burden\" refers to the extra work of looking for a file in the\nworktree, when this is completely unnecessary if you use a link\nobject.\n\n>> This is highly unnatural.\n>\n> Unnatural is a subjective word, and there is no justification I see\n> here in your message.\n\ngit's design follows along the lines of the UNIX philosophy: do one\nthing, do it well. git add/ rm/ mv have a very sharply defined task:\nthey first lock the index file, read the_index using read_cache(),\nbuild a cache_entry struct using user-supplied data (this might\ninvolve worktree code from dir.c to recurse subdirectories, for\nexample), write that cache_entry to the_index (removing existing\nentries with cache_tree_invalidate_path() if necessary), and finally\nwrite the_index to the index file, releasing the lock. Would you\nagree that any operation that doesn't follow along this line is\nunnatural? What then, does writing a special file in the worktree\n(aka .gitmodules) have to do with this entire process?\n\nDoes git diff/ commit/ add/ rm or any other command you can think of\nrely on a special file in the worktree (aka .gitmodules) to be checked\nout? Then why does git submodule require it? Isn't this a\nrequirement that is inconsistent with the rest of git-core?\n\n>> Putting the information in link\n>> objects means that we get a more natural UI + warts like\n>> cd-to-toplevel disappear with no extra code.\n>\n> I do not see how \"link objects\" _means_ \"natural UI\", yet, without\n> an explanation how one leads to the other.\n\nI should've said \"means an easy route to get the existing UI to work\nwith little or no additional code\". Making the submodule information\navailable to git-core is precisely what leads to this. In\nindex_path(), you can inject a case for S_IFDIR to write a link object\nto the database, writing the sha1 to the supplied argument. This is\nnot unnatural in any way, because we're just following along the lines\nof the S_IFLNK codepath, which writes a blob object to the database.\nNow index_path() is called by add_to_index(), which is the master\nfunction for adding anything to the index. Therefore, git add just\nworks. git rm is much simpler: it calls remove_file_from_index()\nwhich in turn calls cache_tree_invalidate() and\nremove_index_entry_at(). Once the entry is removed from the cache,\nour job is done. The link object will be cleaned up at gc-time. git\nmv is just a combination of git rm and git add: it invalidates an\nexisting entry and adds a new one with a different name.\n\nThere is no special .gitmodules to take care of.\n\n> What does cd-to-toplevel have anything to do with it? In case you\n> did not notice, all the core commands internally cd-to-toplevel and\n> carry the \"prefix\" information while doing so, and prepend the\n> prefix to user-supplied paths to find which path the user is talking\n> about. So \"cd to toplevel before starting to carry the operation out\"\n> is a natural pattern inside Git. As many people already told you,\n> \"the user has to run 'git submodule' from the top-level of the\n> submodule working tree\" is a simple oversight of the implementation.\n\nYes, I am aware. I'm piggy-banking on the mature parts of git-core to\nget functionality that I would otherwise have to write by hand. The\ncurrent implementation needs to hand-code this, and hasn't done it yet\n(presumably because it's non-trivial).\n\n>> 2. If we want to make git-submodule a part of git-core (which I think\n>> everyone agrees with), we will need to make the information in\n>> .gitmodules available more easily to the rest of git-core.\n>\n> Care to define \"more easily\" which is another subjective word? The\n> .gitmodules file uses the bog-standard configuration format that can\n> be easily read with the config.c infrastructure. It is a separate\n> matter that git_config() API is cumbersome to use, but improving it\n> would help not just .gitmodules but also the regular non-submodule\n> users of Git. There is a topic in the works to read data in that\n> format from core Heiko is working on.\n\nThis goes both ways: the information is both easier to read and write.\n I can easily create a link object from anywhere: index_path() or\ncmd_edit_link(). To do this, I just have to call write_sha1_file()\nwith the buffer filled out and with the parameter link_type (which is\nalready defined). To access the data in a link, I have to fill out a\ntree_desc with \"HEAD\", an unpack_tree_opts with a custom callback, and\npass it to unpack_trees(). An example of a custom callback:\ncmd_cat_link() which simply calls get_stat_data() to fill in the\nSHA-1, and read_sha1_file() to read that object into a buffer.\n\nI don't have to rely on a worktree with toplevel .gitmodules checked\nout. The information is easily readable/ writeable by default when\nI'm working with git, irrespective of my the state of my worktree.\n\nWhy must we use the git_config() API when it was never designed to do\nthis? Why not leverage the mature part of git that was intended for\nthis, where our new object fits in snugly?\n\nCan you now present a counter-argument about why .gitmodules is\n_better_ suited for the task of managing submodules? (Hint: No, you\ncan't; because it isn't). Can you give me an example of one thing\nthat will become more complex if we were to follow my approach (I've\nalready acknowledged foreach, so that's out)? Do you have any\nconcrete objections to the new design, apart from the fact that it\nbreaks what already \"works\"? Do you acknowledge that the new design\nwill remove a lot of existing complexity (if \"complex\" is too\nsubjective for you: result in a significant negative overall\ndiffstat)?\n"},{"id":"213594","messageId":"7vk3ocx0qq.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0m4o=F=bh_mt9NGDfXKExgVtLhaZ7uxA_XFhpF5AN5-Fg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-08T19:48:13Z","receivedAt":"2013-04-08T19:48:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Does git diff/ commit/ add/ rm or any other command you can think of\n> rely on a special file in the worktree (aka .gitmodules) to be checked\n> out?\n\nTry \"git add foo~\" with usual suspect in .gitignore ;-)\n"},{"id":"213596","messageId":"CALkWK0npHgsf_TyjrMRJdcT-twg_jRXOHtoNpmX+2XNeLq=ZAQ@mail.gmail.com","threadId":"33386","inReplyTo":"7vk3ocx0qq.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T19:54:18Z","receivedAt":"2013-04-08T19:54:18Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> Ramkumar Ramachandra writes:\n>\n>> Does git diff/ commit/ add/ rm or any other command you can think of\n>> rely on a special file in the worktree (aka .gitmodules) to be checked\n>> out?\n>\n> Try \"git add foo~\" with usual suspect in .gitignore ;-)\n\nFirst, it's not a hard requirement: in the worst case, git add will\nadd the file even without a -f. Second, I've already argued about how\nI think this is the right design: What part of that do you disagree\nwith? What alternate design do you propose?\n"},{"id":"213602","messageId":"7v8v4swysv.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0npHgsf_TyjrMRJdcT-twg_jRXOHtoNpmX+2XNeLq=ZAQ@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-08T20:30:08Z","receivedAt":"2013-04-08T20:30:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Junio C Hamano wrote:\n>> Ramkumar Ramachandra writes:\n>>\n>>> Does git diff/ commit/ add/ rm or any other command you can think of\n>>> rely on a special file in the worktree (aka .gitmodules) to be checked\n>>> out?\n>>\n>> Try \"git add foo~\" with usual suspect in .gitignore ;-)\n>\n> First, it's not a hard requirement: in the worst case, git add will\n> add the file even without a -f.\n\nIn the same sense .gitmodules is not a hard requirement, either. I\nuse a submodule without .gitmodules in one of my repositories (the\ntop-level houses the source to generete my dotfiles and is cloned to\nmy environment at work, but the submodule houses my private files\nthat live only at home). The gitlink entry in the index and the\ntree and presence of the .git repository in the submodule checkout\n(where it exists) is sufficient to make the layout work.\n\nIf your complaints were \"I cannot make X work with the current\nsystem, even with changes to git-submodule and some core part of the\nsystem, and I think the reason is because the way module information\nis stored is in a separate file .gitmodules\", with a concrete X,\npeople who are more versed with the submodule subsystem may be able\nto help you come up with a cleaner solution without throwing the\nbaby with the bathwater, but I do not think we saw any concrete X\nmentioned.\n\nThe same sentence followed by \"... and with an object of a new type\nstored at the path of the submodule, I can make it work by doing A,\nB and C\", with concrete A, B and C, some people may be interested in\npursuing that avenue with you, but I do not think we saw such\ncombinations of either.\n\nIf all of your argument starts from \"I think .gitmodules is ugly\nbecause it is not an object of a separate type stored at the path of\nthe submodule, and here are the reasons why I think it is ugly\", I\nhave nothing more to say to you. That \"ugly\" is at best skewed\naesthetics, and each and every example that comes up in this\ndiscussion, like this \"'git add' works with .gitignore\", and the one\nI sent on \".gitattributes vs .gitmodules on the default\" in the\nnearby subthread to Jonathan, makes me realize that .gitmodules is\n_more_ in line with the rest of the system, not less.\n"},{"id":"213603","messageId":"51632B95.3040103@web.de","threadId":"33386","inReplyTo":"5161D3C5.9060804@web.de","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-08T20:41:57Z","receivedAt":"2013-04-08T20:41:57Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Ok, here comes an updated version of our comparison list which\nI updated with what I read in recent discussions. As I said\nearlier, please speak up if I missed anything (or forgot to add\nanyone to the CC).\n\nI picked up one advantage (\"no need to cd-to-toplevel to edit\n.gitmodules) two new disadvantages (\"foreach\" and \"default\nsubmodule config\") and retired one Ram showed a solution for\n(the \"unstaged gitlink\").\n\n\nAdvantages:\n\n* Information is stored in one place, no need to lookup stuff in\n another file/blob.\n\n* Easier coding, as we find all information in a single object.\n\n* No need to cd-to-toplevel to change configuration in the\n .gitmodules file, the special tools to edit link information\n will work in any subdirectory.\n\n(We currently need a checked out work tree to access the\n.gitmodules file, but there is ongoing work to read the\nconfiguration directly from the database)\n\n(While it is easier to merge the link object, a .gitmodules\naware merge driver would work just as well)\n\n\nDisadvantages:\n\n* Changes in user visible behavior, possible compatibility\n problems when Git versions are mixed.\n\n* Special tools are needed to edit submodule information where\n currently a plain editor is sufficient and a standard format\n is used.\n\n* merge conflicts are harder to resolve and require special git\n commands, solving them in .gitmodules is way more intuitive\n as users are already used to conflict markers.\n\n* \"git submodule foreach\" becomes harder to implement\n\n* With .gitmodules we lose a central spot where configuration\n concerning many submodules can be stored\n\n(I think when we also put the submodule name in the object we\ncould also retain the ability to repopulated moved submodules\nfrom their old repo, which is found by that name)\n\n(That a link object can have no unstaged counterpart that a file\neasily has can be fixed by special casing this, e.g. in using a\nfile in .git/link-specs/)\n\n\nHmm, while it is still too early to close the polls, it looks\nto me as most advantages are about easier coding while most\ndisadvantages hit the user. That makes it more understandable\nfor me why Ram is so convinced of his approach and why on the\nother hand submodule users like myself are rather sceptical. I\nthink we need some more advantages that users will directly\nprofit from, the cd-to-toplevel for .gitmodules is definitely\nnot enough to support the change Ram is proposing. What other\nadvantages are missing here?\n"},{"id":"213605","messageId":"CALkWK0mG6FFd_K1Ei-r--58i0j94or5240k1hEVbed8kJ+ZURw@mail.gmail.com","threadId":"33386","inReplyTo":"7v8v4swysv.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T21:03:11Z","receivedAt":"2013-04-08T21:03:11Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"This is going nowhere. You're stuck at making the current submodule\nsystem work, not answering my questions, diverting conversation,\nrepeatedly asking the same stupid questions, labelling everything that\nI say \"subjective\", and refusing to look at the objective counterpart\n(aka, the code). It's clear to me that no matter how many more emails\nI write, you're not going to concede.\n\nI'm not interested in wasting any more of my time with this nonsense.\n\nI give up.\n"},{"id":"213613","messageId":"20130408213006.GD9649@sigill.intra.peff.net","threadId":"33386","inReplyTo":"1365416809-4396-1-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-04-08T21:30:06Z","receivedAt":"2013-04-08T21:30:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 08, 2013 at 03:56:49PM +0530, Ramkumar Ramachandra wrote:\n\n> git add currently goes past submodule boundaries. Document this bug.\n\nIt's not just submodules, but we should not recurse into any\nsub-repository. If I have an unrelated Meta/ repository, I should not be\nable to \"git add Meta/foo\", whether I have used \"git submodule\" or not.\n\nThis topic came about 2 years ago, and I had a quick-and-dirty patch:\n\n http://thread.gmane.org/gmane.comp.version-control.git/170937/focus=171040\n\nI do not recall anything about the patch at this point (i.e., whether it\nwas the right thing), but maybe it is a good starting point for somebody\nto look into it.\n\n> diff --git a/t/t3700-add.sh b/t/t3700-add.sh\n> index 874b3a6..a1ea050 100755\n> --- a/t/t3700-add.sh\n> +++ b/t/t3700-add.sh\n> @@ -310,4 +310,18 @@ test_expect_success 'git add --dry-run --ignore-missing of non-existing file out\n> \ttest_i18ncmp expect.err actual.err\n> '\n> \n> +test_expect_failure 'git add should not go past submodule boundaries' '\n> +\tmkdir submodule_dir &&\n> +\t(\n> +\t\tcd submodule_dir &&\n> +\t\tgit init &&\n> +\t\tcat >foo <<-\\EOF &&\n> +\t\tSome content\n> +\t\tEOF\n> +\t\tgit add foo &&\n> +\t\tgit commit -a -m \"Add foo\"\n> +\t) &&\n> +\tgit add submodule_dir/foo\n> +'\n\nThat is not actually a submodule, but rather just a repo that happens to\nbe inside our working tree. I know the distinction is subtle, but\naccording to the thread I linked to above, we may actually treat paths\nwith gitlinked index entries separately already (I did not try it,\nthough).\n\n-Peff\n"},{"id":"213614","messageId":"20130408213652.GE9649@sigill.intra.peff.net","threadId":"33386","inReplyTo":"51632B95.3040103@web.de","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-04-08T21:36:52Z","receivedAt":"2013-04-08T21:36:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 08, 2013 at 10:41:57PM +0200, Jens Lehmann wrote:\n\n> (While it is easier to merge the link object, a .gitmodules\n> aware merge driver would work just as well)\n\nI have not been following this thread that closely, so apologies if I\nmissed it, but one thing I have not seen mention of is how the extra\ninformation inside the gitlink object will require extra merge effort.\n\nImagine I have two branches; one updates the submodule's commit pointer,\nand the other updates some meta-information about the submodule (e.g.,\nit points the URL to a new host). In the current system, one change goes\ninto .gitmodules, and the other goes into the gitlink path. In a new\ncombined object, there is a conflict and we must do content-level\nmerging on it (which presumably would be done with a specialized merge\ndriver).\n\nSo I think that in some cases .gitmodules creates more conflicts\n(submodule A and submodule B are touched and have a textual conflict),\nand sometimes the combined object would create more objects (you touch\ntwo parts of the of the combined object). The solution in both cases is\na smarter merge driver that understands which parts semantically\nconflict and which parts do not.\n\n-Peff\n"},{"id":"213621","messageId":"CALkWK0njP+fpVVvzs4BH4nR=ddQg-9bvQd-nry4gZnpqt2wQrA@mail.gmail.com","threadId":"33386","inReplyTo":"7v8v4swysv.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-08T21:59:09Z","receivedAt":"2013-04-08T21:59:09Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> If all of your argument starts from \"I think .gitmodules is ugly\n> because it is not an object of a separate type stored at the path of\n> the submodule, and here are the reasons why I think it is ugly\", I\n> have nothing more to say to you.\n\n_This_ is how you summarize the seven points and the follow-up emails\nI wrote out for you? Seriously?\n\nI enjoy good debate, and would've loved to be beaten in argument. But\nthat's not what happened here. I was just too frustrated with your\nstupidity to continue.\n"},{"id":"213622","messageId":"7v8v4svfwi.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"20130408213006.GD9649@sigill.intra.peff.net","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-08T22:03:41Z","receivedAt":"2013-04-08T22:03:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Mon, Apr 08, 2013 at 03:56:49PM +0530, Ramkumar Ramachandra wrote:\n>\n>> git add currently goes past submodule boundaries. Document this bug.\n>\n> It's not just submodules, but we should not recurse into any\n> sub-repository. If I have an unrelated Meta/ repository, I should not be\n> able to \"git add Meta/foo\", whether I have used \"git submodule\" or not.\n>\n> This topic came about 2 years ago, and I had a quick-and-dirty patch:\n>\n> http://thread.gmane.org/gmane.comp.version-control.git/170937/focus=171040\n>\n> I do not recall anything about the patch at this point (i.e., whether it\n> was the right thing), but maybe it is a good starting point for somebody\n> to look into it.\n>\n>> diff --git a/t/t3700-add.sh b/t/t3700-add.sh\n>> index 874b3a6..a1ea050 100755\n>> --- a/t/t3700-add.sh\n>> +++ b/t/t3700-add.sh\n>> @@ -310,4 +310,18 @@ test_expect_success 'git add --dry-run --ignore-missing of non-existing file out\n>> \ttest_i18ncmp expect.err actual.err\n>> '\n>> \n>> +test_expect_failure 'git add should not go past submodule boundaries' '\n>> +\tmkdir submodule_dir &&\n>> +\t(\n>> +\t\tcd submodule_dir &&\n>> +\t\tgit init &&\n>> +\t\tcat >foo <<-\\EOF &&\n>> +\t\tSome content\n>> +\t\tEOF\n>> +\t\tgit add foo &&\n>> +\t\tgit commit -a -m \"Add foo\"\n>> +\t) &&\n>> +\tgit add submodule_dir/foo\n>> +'\n>\n> That is not actually a submodule, but rather just a repo that happens to\n> be inside our working tree. \n\nI think we should treat it as a submodule-to-be, waiting for the\nuser to run \"git add submodule_dir\".\n\nIf it is a file in the working tree of an unrelated and separate\nrepository, it still is wrong to allow it to be added to our\nrepository, no?\n\nIf we had \"git add submodule_dir\" before the last \"git add\", as you\npointed out, we should already error out.\n\n> I know the distinction is subtle, but according to the thread I\n> linked to above, we may actually treat paths with gitlinked index\n> entries separately already (I did not try it, though).\n"},{"id":"213624","messageId":"20130408220730.GA11873@sigill.intra.peff.net","threadId":"33386","inReplyTo":"7v8v4svfwi.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-04-08T22:07:30Z","receivedAt":"2013-04-08T22:07:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 08, 2013 at 03:03:41PM -0700, Junio C Hamano wrote:\n\n> >> +test_expect_failure 'git add should not go past submodule boundaries' '\n> >> +\tmkdir submodule_dir &&\n> >> +\t(\n> >> +\t\tcd submodule_dir &&\n> >> +\t\tgit init &&\n> >> +\t\tcat >foo <<-\\EOF &&\n> >> +\t\tSome content\n> >> +\t\tEOF\n> >> +\t\tgit add foo &&\n> >> +\t\tgit commit -a -m \"Add foo\"\n> >> +\t) &&\n> >> +\tgit add submodule_dir/foo\n> >> +'\n> >\n> > That is not actually a submodule, but rather just a repo that happens to\n> > be inside our working tree. \n> \n> I think we should treat it as a submodule-to-be, waiting for the\n> user to run \"git add submodule_dir\".\n> \n> If it is a file in the working tree of an unrelated and separate\n> repository, it still is wrong to allow it to be added to our\n> repository, no?\n\nSorry if I wasn't clear; I absolutely think this test is checking\nsomething reasonable, and we should fix it to pass. I was only referring\nto the wording, which is misleading (and hoped that pointing it out\nwould help whoever works on it in the right direction of where the\nproblem is).\n\n-Peff\n"},{"id":"213645","messageId":"CALkWK0m_82ThYKxcr=YHGBnOaHr9yBpYFHxx2KrnQkWD_zjp1A@mail.gmail.com","threadId":"33386","inReplyTo":"20130408213006.GD9649@sigill.intra.peff.net","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-09T09:19:24Z","receivedAt":"2013-04-09T09:19:24Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jeff King wrote:\n> That is not actually a submodule, but rather just a repo that happens to\n> be inside our working tree. I know the distinction is subtle, but\n> according to the thread I linked to above, we may actually treat paths\n> with gitlinked index entries separately already (I did not try it,\n> though).\n\nAgreed. treat_gitlink() dies if there is a gitlink cache_entry\nmatching any of the pathspecs; it does one thing well, and promises\nwhat it does: however, its core logic in check_path_for_gitlink() can\neasily be moved into lstat_cache_matchlen() as that is more generic\n(checks index and worktree). die_if_path_beyond_symlink() is the\nperfect example to replicate. Today, there is one more caller of\ndie_if_path_beyond_symlink(): check-ignore, so we must patch that too.\n\nOn a slightly related note treat_path() also contains the logic for\nchecking for a git repository in the worktree. Unfortunately, the\ncode cannot be reused because it checks for a '.git' in a dirent.\n\nOn the wording issue, a submodule is a submodule whether in-index or\notherwise. I would write two different tests: one for in-worktree\nsubmodule and another for in-index submodule, and name them\nappropriately. Does that make sense?\n"},{"id":"213646","messageId":"1365499297-8667-1-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"CALkWK0m_82ThYKxcr=YHGBnOaHr9yBpYFHxx2KrnQkWD_zjp1A@mail.gmail.com","subject":"[PATCH 0/2] Fix git add with submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-09T09:21:35Z","receivedAt":"2013-04-09T09:21:35Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\ngit add has a bug when operating on submodules. The following test\nfails:\n\n\tmkdir submodule_dir &&\n\t(\n\t\tcd submodule_dir &&\n\t\tgit init &&\n\t\tcat >foo <<-\\EOF &&\n\t\tSome content\n\t\tEOF\n\t\tgit add foo &&\n\t\tgit commit -a -m \"Add foo\"\n\t) &&\n\ttest_must_fail git add submodule_dir/foo\n\n[1/2] adds this failing test along with a passing related test.\n[2/2] fixes the failing test.\n\nRamkumar Ramachandra (2):\n t3700 (add): add two tests for testing add with submodules\n add: refuse to add paths beyond repository boundaries\n\n builtin/add.c | 5 +++--\n cache.h | 2 ++\n pathspec.c | 12 ++++++++++++\n pathspec.h | 1 +\n symlinks.c | 43 +++++++++++++++++++++++++++++++++++++------\n t/t3700-add.sh | 32 ++++++++++++++++++++++++++++++++\n 6 files changed, 87 insertions(+), 8 deletions(-)\n\n-- \n1.8.2.1.347.gdd82260.dirty\n"},{"id":"213647","messageId":"1365499297-8667-2-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"1365499297-8667-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 1/2] t3700 (add): add two tests for testing add with submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-09T09:21:36Z","receivedAt":"2013-04-09T09:21:36Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"The first test \"git add should not go past gitlink boundaries\" checks\nthat paths within a submodule added with 'git submodule add' cannot be\nadded to the index. It passes because of treat_gitlink() in\nbuiltin/add.c.\n\nThe second test \"git add should not go past git repository boundaries\"\nchecks that paths within a git repository in the worktree (not yet\nadded with 'git submodule add') cannot be added to the index. It\nfails because there is no existing code to check this.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n t/t3700-add.sh | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/t/t3700-add.sh b/t/t3700-add.sh\nindex 874b3a6..1ad2331 100755\n--- a/t/t3700-add.sh\n+++ b/t/t3700-add.sh\n@@ -310,4 +310,36 @@ test_expect_success 'git add --dry-run --ignore-missing of non-existing file out\n \ttest_i18ncmp expect.err actual.err\n '\n \n+test_expect_success 'git add should not go past gitlink boundaries' '\n+\trm -rf submodule_dir &&\n+\tmkdir submodule_dir &&\n+\t(\n+\t\tcd submodule_dir &&\n+\t\tgit init &&\n+\t\tgit config remote.origin.url \"quux\" &&\n+\t\tcat >foo <<-\\EOF &&\n+\t\tSome content\n+\t\tEOF\n+\t\tgit add foo &&\n+\t\tgit commit -a -m \"Add foo\"\n+\t) &&\n+\tgit submodule add ./submodule_dir &&\n+\ttest_must_fail git add submodule_dir/foo\n+'\n+\n+test_expect_failure 'git add should not go past git repository boundaries' '\n+\trm -rf submodule_dir &&\n+\tmkdir submodule_dir &&\n+\t(\n+\t\tcd submodule_dir &&\n+\t\tgit init &&\n+\t\tcat >foo <<-\\EOF &&\n+\t\tSome content\n+\t\tEOF\n+\t\tgit add foo &&\n+\t\tgit commit -a -m \"Add foo\"\n+\t) &&\n+\ttest_must_fail git add submodule_dir/foo\n+'\n+\n test_done\n-- \n1.8.2.1.347.gdd82260.dirty\n"},{"id":"213648","messageId":"1365499297-8667-3-git-send-email-artagnon@gmail.com","threadId":"33386","inReplyTo":"1365499297-8667-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-09T09:21:37Z","receivedAt":"2013-04-09T09:21:37Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Currently, git add has the logic for refusing to add gitlinks using\ntreat_path(), which in turn calls check_path_for_gitlink(). However,\nthis only checks for an in-index submodule (or gitlink cache_entry).\nA path inside a git repository in the worktree still adds fine, and\nthis is a bug. The logic for denying it is very similar to denying\nadding paths beyond symbolic links: die_if_path_beyond_symlink().\nFollow its example and write a die_if_path_beyond_gitrepo() to fix\nthis bug.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n builtin/add.c | 5 +++--\n cache.h | 2 ++\n pathspec.c | 12 ++++++++++++\n pathspec.h | 1 +\n symlinks.c | 43 +++++++++++++++++++++++++++++++++++++------\n t/t3700-add.sh | 2 +-\n 6 files changed, 56 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex ab1c9e8..1538129 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -155,8 +155,8 @@ static void refresh(int verbose, const char **pathspec)\n \n /*\n * Normalizes argv relative to prefix, via get_pathspec(), and then\n- * runs die_if_path_beyond_symlink() on each path in the normalized\n- * list.\n+ * runs die_if_path_beyond_symlink() and die_if_path_beyond_repository()\n+ * on each path in the normalized list.\n */\n static const char **validate_pathspec(const char **argv, const char *prefix)\n {\n@@ -166,6 +166,7 @@ static const char **validate_pathspec(const char **argv, const char *prefix)\n \t\tconst char **p;\n \t\tfor (p = pathspec; *p; p++) {\n \t\t\tdie_if_path_beyond_symlink(*p, prefix);\n+\t\t\tdie_if_path_beyond_gitrepo(*p, prefix);\n \t\t}\n \t}\n \ndiff --git a/cache.h b/cache.h\nindex e1e8ce8..987d7f3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -962,6 +962,8 @@ struct cache_def {\n \n extern int has_symlink_leading_path(const char *name, int len);\n extern int threaded_has_symlink_leading_path(struct cache_def *, const char *, int);\n+extern int has_gitrepo_leading_path(const char *name, int len);\n+extern int threaded_has_gitrepo_leading_path(struct cache_def *, const char *, int);\n extern int check_leading_path(const char *name, int len);\n extern int has_dirs_only_path(const char *name, int len, int prefix_len);\n extern void schedule_dir_for_removal(const char *name, int len);\ndiff --git a/pathspec.c b/pathspec.c\nindex 284f397..142631d 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -99,3 +99,15 @@ void die_if_path_beyond_symlink(const char *path, const char *prefix)\n \t\tdie(_(\"'%s' is beyond a symbolic link\"), path + len);\n \t}\n }\n+\n+/*\n+ * Dies if the given path refers to a file inside a directory with a\n+ * git repository in it.\n+ */\n+void die_if_path_beyond_gitrepo(const char *path, const char *prefix)\n+{\n+\tif (has_gitrepo_leading_path(path, strlen(path))) {\n+\t\tint len = prefix ? strlen(prefix) : 0;\n+\t\tdie(_(\"'%s' is beyond a git repository\"), path + len);\n+\t}\n+}\ndiff --git a/pathspec.h b/pathspec.h\nindex db0184a..c201c7b 100644\n--- a/pathspec.h\n+++ b/pathspec.h\n@@ -5,5 +5,6 @@ extern char *find_pathspecs_matching_against_index(const char **pathspec);\n extern void add_pathspec_matches_against_index(const char **pathspec, char *seen, int specs);\n extern const char *check_path_for_gitlink(const char *path);\n extern void die_if_path_beyond_symlink(const char *path, const char *prefix);\n+extern void die_if_path_beyond_gitrepo(const char *path, const char *prefix);\n \n #endif /* PATHSPEC_H */\ndiff --git a/symlinks.c b/symlinks.c\nindex c2b41a8..e551dae 100644\n--- a/symlinks.c\n+++ b/symlinks.c\n@@ -54,6 +54,7 @@ static inline void reset_lstat_cache(struct cache_def *cache)\n #define FL_LSTATERR (1 << 3)\n #define FL_ERR (1 << 4)\n #define FL_FULLPATH (1 << 5)\n+#define FL_GITREPO (1 << 6)\n \n /*\n * Check if name 'name' of length 'len' has a symlink leading\n@@ -142,8 +143,22 @@ static int lstat_cache_matchlen(struct cache_def *cache,\n \t\t\tif (errno == ENOENT)\n \t\t\t\t*ret_flags |= FL_NOENT;\n \t\t} else if (S_ISDIR(st.st_mode)) {\n-\t\t\tlast_slash_dir = last_slash;\n-\t\t\tcontinue;\n+\t\t\t/* Check to see if the directory contains a\n+\t\t\t git repository */\n+\t\t\tstruct stat st;\n+\t\t\tstruct strbuf dotgitentry = STRBUF_INIT;\n+\t\t\tstrbuf_addf(&dotgitentry, \"%s/.git\", cache->path);\n+\t\t\tif (lstat(dotgitentry.buf, &st) < 0) {\n+\t\t\t\tif (errno == ENOENT) {\n+\t\t\t\t\tstrbuf_release(&dotgitentry);\n+\t\t\t\t\tlast_slash_dir = last_slash;\n+\t\t\t\t\tcontinue;\n+\t\t\t\t}\n+\t\t\t\t*ret_flags = FL_LSTATERR;\n+\t\t\t}\n+\t\t\telse\n+\t\t\t\t*ret_flags = FL_GITREPO;\n+\t\t\tstrbuf_release(&dotgitentry);\n \t\t} else if (S_ISLNK(st.st_mode)) {\n \t\t\t*ret_flags = FL_SYMLINK;\n \t\t} else {\n@@ -153,11 +168,11 @@ static int lstat_cache_matchlen(struct cache_def *cache,\n \t}\n \n \t/*\n-\t * At the end update the cache. Note that max 3 different\n-\t * path types, FL_NOENT, FL_SYMLINK and FL_DIR, can be cached\n-\t * for the moment!\n+\t * At the end update the cache. Note that max 4 different\n+\t * path types: FL_NOENT, FL_SYMLINK, FL_GITREPO, and\n+\t * FL_DIR.\n \t */\n-\tsave_flags = *ret_flags & track_flags & (FL_NOENT|FL_SYMLINK);\n+\tsave_flags = *ret_flags & track_flags & (FL_NOENT|FL_SYMLINK|FL_GITREPO);\n \tif (save_flags && last_slash > 0 && last_slash <= PATH_MAX) {\n \t\tcache->path[last_slash] = '\\0';\n \t\tcache->len = last_slash;\n@@ -204,6 +219,14 @@ int threaded_has_symlink_leading_path(struct cache_def *cache, const char *name,\n }\n \n /*\n+ * Return non-zero if path 'name' has a leading gitrepo component\n+ */\n+int threaded_has_gitrepo_leading_path(struct cache_def *cache, const char *name, int len)\n+{\n+\treturn lstat_cache(cache, name, len, FL_GITREPO|FL_DIR, USE_ONLY_LSTAT) & FL_GITREPO;\n+}\n+\n+/*\n * Return non-zero if path 'name' has a leading symlink component\n */\n int has_symlink_leading_path(const char *name, int len)\n@@ -212,6 +235,14 @@ int has_symlink_leading_path(const char *name, int len)\n }\n \n /*\n+ * Return non-zero if path 'name' has a leading gitrepo component\n+ */\n+int has_gitrepo_leading_path(const char *name, int len)\n+{\n+\treturn threaded_has_gitrepo_leading_path(&default_cache, name, len);\n+}\n+\n+/*\n * Return zero if path 'name' has a leading symlink component or\n * if some leading path component does not exists.\n *\ndiff --git a/t/t3700-add.sh b/t/t3700-add.sh\nindex 1ad2331..4714734 100755\n--- a/t/t3700-add.sh\n+++ b/t/t3700-add.sh\n@@ -327,7 +327,7 @@ test_expect_success 'git add should not go past gitlink boundaries' '\n \ttest_must_fail git add submodule_dir/foo\n '\n \n-test_expect_failure 'git add should not go past git repository boundaries' '\n+test_expect_success 'git add should not go past git repository boundaries' '\n \trm -rf submodule_dir &&\n \tmkdir submodule_dir &&\n \t(\n-- \n1.8.2.1.347.gdd82260.dirty\n"},{"id":"213650","messageId":"5163FED4.8070403@gmail.com","threadId":"33386","inReplyTo":"20130408213006.GD9649@sigill.intra.peff.net","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2013-04-09T11:43:16Z","receivedAt":"2013-04-09T11:43:16Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 08.04.2013 23:30, Jeff King pisze:\n> On Mon, Apr 08, 2013 at 03:56:49PM +0530, Ramkumar Ramachandra wrote:\n> \n>> git add currently goes past submodule boundaries. Document this bug.\n> \n> It's not just submodules, but we should not recurse into any\n> sub-repository. If I have an unrelated Meta/ repository, I should not be\n> able to \"git add Meta/foo\", whether I have used \"git submodule\" or not.\n> \n> This topic came about 2 years ago, and I had a quick-and-dirty patch:\n> \n> http://thread.gmane.org/gmane.comp.version-control.git/170937/focus=171040\n> \n> I do not recall anything about the patch at this point (i.e., whether it\n> was the right thing), but maybe it is a good starting point for somebody\n> to look into it.\n\nHmmm... I used to do (and still do) such not-recommended thing,\ni.e. keeping git/gitweb/TODO etc. in git/gitweb/.git repository,\nwhile having git/gitweb/gitweb.perl in git/.git repository.\n\nSo my (admittedly strange) setup will stop working?\n\n-- \nJakub Narębski\n"},{"id":"213652","messageId":"516400DE.4080809@gmail.com","threadId":"33386","inReplyTo":"7va9p8yjuf.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2013-04-09T11:51:58Z","receivedAt":"2013-04-09T11:51:58Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n> Ramkumar Ramachandra wrote:\n\n>> 2. If we want to make git-submodule a part of git-core (which I think\n>> everyone agrees with), we will need to make the information in\n>> .gitmodules available more easily to the rest of git-core.\n\n> Care to define \"more easily\" which is another subjective word? The\n> .gitmodules file uses the bog-standard configuration format that can\n> be easily read with the config.c infrastructure. It is a separate\n> matter that git_config() API is cumbersome to use, but improving it\n> would help not just .gitmodules but also the regular non-submodule\n> users of Git. There is a topic in the works to read data in that\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> format from core Heiko is working on.\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nBTW. this is something that I was missing to implement better\nsubmodule support in gitweb (and thus git-instaweb) than just\nmarking it as submodule in 'tree' view.\n\n-- \nJakub Narębski\n"},{"id":"213653","messageId":"CALkWK0nMQbk3JQ1yX+uv3Eot3qbEvsWGSqVpg8UnoVXP=g_M7A@mail.gmail.com","threadId":"33386","inReplyTo":"5163FED4.8070403@gmail.com","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-09T11:54:43Z","receivedAt":"2013-04-09T11:54:43Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jakub Narębski wrote:\n> Hmmm... I used to do (and still do) such not-recommended thing,\n> i.e. keeping git/gitweb/TODO etc. in git/gitweb/.git repository,\n> while having git/gitweb/gitweb.perl in git/.git repository.\n\nWhy don't you put the gitweb/TODO in a different branch in the git.git\nrepository? Why do you feel the need to have two different\nrepositories tracking different files in the same path?\n\nJust out of curiosity, how does stuff work with your setup? Does the\nworktree gitweb/ belong to your gitweb.git repository or git.git\nrepository? How do half the git commands work? For example, won't\ngit clean -dfx remove the files tracked by your other repository?\nWill a conflicting checkout not stomp files tracked by the other\nrepository? How are worktree-rules like .gitignore applied?\n\n> So my (admittedly strange) setup will stop working?\n\nYes. I would persuade you not to use such a gross setup; this is not\nwhat git was intended for at all.\n"},{"id":"213657","messageId":"51641C73.9080006@gmail.com","threadId":"33386","inReplyTo":"CALkWK0nMQbk3JQ1yX+uv3Eot3qbEvsWGSqVpg8UnoVXP=g_M7A@mail.gmail.com","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2013-04-09T13:49:39Z","receivedAt":"2013-04-09T13:49:39Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On 09.04.2013, Ramkumar Ramachandra wrote:\n> Jakub Narębski wrote:\n\n>> Hmmm... I used to do (and still do) such not-recommended thing,\n>> i.e. keeping git/gitweb/TODO etc. in git/gitweb/.git repository,\n>> while having git/gitweb/gitweb.perl in git/.git repository.\n> \n> Why don't you put the gitweb/TODO in a different branch in the git.git\n> repository? Why do you feel the need to have two different\n> repositories tracking different files in the same path?\n\nIt is not only gitweb/TODO. If it was only that file, I could have\nused 'gitweb/todo' branch for it, or something. Though I would be\nmissing having it beside gitweb.perl, and having easy access to it\nduring work on gitweb.perl\n\nI want to have various files that I use when working on gitweb.perl\n(but should not and would not be in gitweb subsystem in git.git\nrepository) to be under version control, and be side-by-side near\ngitweb.perl.\n\nThese are:\n* gitweb/TODO - TODO file for gitweb (personal, not often updated),\n and similar gitweb/gitwebs-whats-cooking.txt\n\n Might be put into 'gitweb/todo/TODO' file and 'gitweb/todo/.git'\n private repository (and perhaps 'gitweb/todo' branch of my clone\n of git.git repository).\n\n* various *_test.perl files, where I test features to be possibly\n put into gitweb, like e.g. chop_str_test.perl or test_find_forks.perl\n (or similar benchmark_find_forks.perl)\n\n* private personal configuration files for testing its output, like\n gitweb/gitweb_config.perl and gitweb/magic.txt !!!\n\n Those are very much required to reside beside gitweb/gitweb.perl\n because of default GITWEB_CONFIG value. With those I can simply\n run current gitweb/gitweb.perl (sic!) from its directory while\n I am working on it.\n\n> Just out of curiosity, how does stuff work with your setup? Does the\n> worktree gitweb/ belong to your gitweb.git repository or git.git\n> repository?\n\nThe 'git/gitweb/' worktree belong to both repositories (it is 'gitweb/'\nin git.git clone i.e. git/.git, and it is top dir of git/gitweb/.git\nrepository).\n\n> How do half the git commands work? For example, won't\n> git clean -dfx remove the files tracked by your other repository?\n\nThey work, somewhat and with some care. I don't use \"git clean\"\nfor example.\n\n> Will a conflicting checkout not stomp files tracked by the other\n> repository? How are worktree-rules like .gitignore applied?\n\n'git/gitweb/.gitignore' belong to 'git/gitweb/.git' repository\nand is used to ignore 'git/.git' files (with the intent of marking\nthem untracked *precious*). I could have used 'info/exclude'\nhere because this repository is for the time being private.\n\nAs gitweb subsystem in git.git is quite stable, untracked but\nexisting 'gitweb/.gitignore' doesn't usually matter, as all files\nin 'gitweb/' are already tracked. Besides I can always use\n\"git add -f\" for adding to git.git if necessary (e.g. when splitting\ngitweb.js etc.).\n\n>> So my (admittedly strange) setup will stop working?\n> \n> Yes. I would persuade you not to use such a gross setup; this is not\n> what git was intended for at all.\n\nWhy not?\n\n-- \nJakub Narębski\n"},{"id":"213668","messageId":"20130409162735.GA21972@sigill.intra.peff.net","threadId":"33386","inReplyTo":"CALkWK0m_82ThYKxcr=YHGBnOaHr9yBpYFHxx2KrnQkWD_zjp1A@mail.gmail.com","subject":"Re: [PATCH] t3700 (add): add failing test for add with submodules","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-04-09T16:27:36Z","receivedAt":"2013-04-09T16:27:36Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 09, 2013 at 02:49:24PM +0530, Ramkumar Ramachandra wrote:\n\n> On the wording issue, a submodule is a submodule whether in-index or\n> otherwise. I would write two different tests: one for in-worktree\n> submodule and another for in-index submodule, and name them\n> appropriately. Does that make sense?\n\nYeah, that makes perfect sense to me.\n\n-Peff\n"},{"id":"213672","messageId":"20130409165032.GB21972@sigill.intra.peff.net","threadId":"33386","inReplyTo":"1365499297-8667-3-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-04-09T16:50:32Z","receivedAt":"2013-04-09T16:50:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 09, 2013 at 02:51:37PM +0530, Ramkumar Ramachandra wrote:\n\n> Currently, git add has the logic for refusing to add gitlinks using\n> treat_path(), which in turn calls check_path_for_gitlink(). However,\n> this only checks for an in-index submodule (or gitlink cache_entry).\n> A path inside a git repository in the worktree still adds fine, and\n> this is a bug. The logic for denying it is very similar to denying\n> adding paths beyond symbolic links: die_if_path_beyond_symlink().\n> Follow its example and write a die_if_path_beyond_gitrepo() to fix\n> this bug.\n\nThanks for working on this.\n\nI think the direction is a good one. It does disallow Jakub's crazy\nshared-directory setup. I am not too sad to see that disallowed, at\nleast by default, because there are so many ways to screw yourself while\nusing it if you are not careful (I tried something similar once, and\ngave up because I kept running into problematic cases).\n\nI am not opposed to having an escape hatch to operate in that mode, but\nit should be triggered explicitly so it doesn't catch users unaware.\n\n> Signed-off-by: Ramkumar Ramachandra \n> ---\n> builtin/add.c | 5 +++--\n> cache.h | 2 ++\n> pathspec.c | 12 ++++++++++++\n> pathspec.h | 1 +\n> symlinks.c | 43 +++++++++++++++++++++++++++++++++++++------\n> t/t3700-add.sh | 2 +-\n> 6 files changed, 56 insertions(+), 9 deletions(-)\n\nI am not super-familiar with this part of the code, but having worked on\nit once two years ago for the same problem, your solution looks like the\nright thing.\n\n> @@ -142,8 +143,22 @@ static int lstat_cache_matchlen(struct cache_def *cache,\n> \t\t\tif (errno == ENOENT)\n> \t\t\t\t*ret_flags |= FL_NOENT;\n> \t\t} else if (S_ISDIR(st.st_mode)) {\n> -\t\t\tlast_slash_dir = last_slash;\n> -\t\t\tcontinue;\n> +\t\t\t/* Check to see if the directory contains a\n> +\t\t\t git repository */\n> +\t\t\tstruct stat st;\n> +\t\t\tstruct strbuf dotgitentry = STRBUF_INIT;\n> +\t\t\tstrbuf_addf(&dotgitentry, \"%s/.git\", cache->path);\n\nCan we use mkpath here to avoid an allocation? Or even better,\ncache->path is PATH_MAX+1 bytes, and we munge it earlier in the\nfunction. Can we just check the length and stick \"/.git\" on the end?\n\n> +\t\t\tif (lstat(dotgitentry.buf, &st) < 0) {\n> +\t\t\t\tif (errno == ENOENT) {\n> +\t\t\t\t\tstrbuf_release(&dotgitentry);\n> +\t\t\t\t\tlast_slash_dir = last_slash;\n> +\t\t\t\t\tcontinue;\n> +\t\t\t\t}\n> +\t\t\t\t*ret_flags = FL_LSTATERR;\n> +\t\t\t}\n> +\t\t\telse\n> +\t\t\t\t*ret_flags = FL_GITREPO;\n> +\t\t\tstrbuf_release(&dotgitentry);\n\nIn my original patch long ago, Junio asked if we should be checking\nis_git_directory() when we find a \".git\" entry, to make sure it is not a\nfalse positive. I don't have a strong opinion either way, but if we do\nthat, we would possibly want to update treat_path to do the same thing\nfor consistency.\n\n-Peff\n"},{"id":"213675","messageId":"7v4nfftyu3.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"1365499297-8667-3-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T17:09:56Z","receivedAt":"2013-04-09T17:09:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Currently, git add has the logic for refusing to add gitlinks using\n> treat_path(), which in turn calls check_path_for_gitlink(). However,\n> this only checks for an in-index submodule (or gitlink cache_entry).\n> A path inside a git repository in the worktree still adds fine, and\n> this is a bug. The logic for denying it is very similar to denying\n> adding paths beyond symbolic links: die_if_path_beyond_symlink().\n> Follow its example and write a die_if_path_beyond_gitrepo() to fix\n> this bug.\n>\n> Signed-off-by: Ramkumar Ramachandra \n> ---\n\n> @@ -166,6 +166,7 @@ static const char **validate_pathspec(const char **argv, const char *prefix)\n> \t\tconst char **p;\n> \t\tfor (p = pathspec; *p; p++) {\n> \t\t\tdie_if_path_beyond_symlink(*p, prefix);\n> +\t\t\tdie_if_path_beyond_gitrepo(*p, prefix);\n> \t\t}\n> \t}\n> diff --git a/cache.h b/cache.h\n> index e1e8ce8..987d7f3 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -962,6 +962,8 @@ struct cache_def {\n> \n> extern int has_symlink_leading_path(const char *name, int len);\n> +extern int has_gitrepo_leading_path(const char *name, int len);\n\nI looked at the output from \"grep has_symlink_leading_path\" and also\nfor \"die_if_path_beyond\"; all of these places are checking \"I have\nthis multi-level path; I want to know if the path does not (should\nnot) be part of the current project\", I think. Certainly the one in\nthe \"update-index\" is about the same operation as \"git add\" you are\npatching.\n\nIsn't it a better approach to _rename_ the existing function not to\nsingle out \"symlink\"-ness of the path first ? A symlink in the\nmiddle of such a multi-level path that leads to a place outside the\nproject is _not_ the only way to step out of our project boundary. A\ndirectory in the middle of a multi-level path that is the top-level\nof the working tree of a foreign project is another way to step out\nof our project boundary. Perhaps\n\n\tdie_if_path_outside_our_project()\n path_outside_our_project()\n\nAnd then update the implementation of path_outside_our_project(),\nwhich only took \"symlink in the middle\" into account so far, and\nteach it that such a \"top-level of the working tree of a foreign\nproject\" is also stepping out of our project?\n\nThat way, you do not have to settle on fixing the bug only in \"git\nadd\" and keep the bug in \"git update-index\", I think.\n\nI think the hit in builtin/apply.c deals with the same \"beyond\nsymlink is outside our project\" check and can be updated like so. I\ndidn't look at the ones in diff-lib.c and dir.c so you may want to\ndouble check on what they use it for.\n"},{"id":"213680","messageId":"7vvc7vsj4p.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"7v4nfftyu3.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T17:34:30Z","receivedAt":"2013-04-09T17:34:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Ramkumar Ramachandra writes:\n>\n>> Currently, git add has the logic for refusing to add gitlinks using\n>> treat_path(), which in turn calls check_path_for_gitlink(). However,\n>> this only checks for an in-index submodule (or gitlink cache_entry).\n>> A path inside a git repository in the worktree still adds fine, and\n>> this is a bug. The logic for denying it is very similar to denying\n>> adding paths beyond symbolic links: die_if_path_beyond_symlink().\n>> Follow its example and write a die_if_path_beyond_gitrepo() to fix\n>> this bug.\n>>\n>> Signed-off-by: Ramkumar Ramachandra \n>> ---\n>\n>> @@ -166,6 +166,7 @@ static const char **validate_pathspec(const char **argv, const char *prefix)\n>> \t\tconst char **p;\n>> \t\tfor (p = pathspec; *p; p++) {\n>> \t\t\tdie_if_path_beyond_symlink(*p, prefix);\n>> +\t\t\tdie_if_path_beyond_gitrepo(*p, prefix);\n>> \t\t}\n>> \t}\n>> diff --git a/cache.h b/cache.h\n>> index e1e8ce8..987d7f3 100644\n>> --- a/cache.h\n>> +++ b/cache.h\n>> @@ -962,6 +962,8 @@ struct cache_def {\n>> \n>> extern int has_symlink_leading_path(const char *name, int len);\n>> +extern int has_gitrepo_leading_path(const char *name, int len);\n>\n> I looked at the output from \"grep has_symlink_leading_path\" and also\n> for \"die_if_path_beyond\"; all of these places are checking \"I have\n> this multi-level path; I want to know if the path does not (should\n> not) be part of the current project\", I think. Certainly the one in\n> the \"update-index\" is about the same operation as \"git add\" you are\n> patching.\n>\n> Isn't it a better approach to _rename_ the existing function not to\n> single out \"symlink\"-ness of the path first ? A symlink in the\n> middle of such a multi-level path that leads to a place outside the\n> project is _not_ the only way to step out of our project boundary. A\n> directory in the middle of a multi-level path that is the top-level\n> of the working tree of a foreign project is another way to step out\n> of our project boundary. Perhaps\n>\n> \tdie_if_path_outside_our_project()\n> path_outside_our_project()\n>\n> And then update the implementation of path_outside_our_project(),\n> which only took \"symlink in the middle\" into account so far, and\n> teach it that such a \"top-level of the working tree of a foreign\n> project\" is also stepping out of our project?\n>\n> That way, you do not have to settle on fixing the bug only in \"git\n> add\" and keep the bug in \"git update-index\", I think.\n>\n> I think the hit in builtin/apply.c deals with the same \"beyond\n> symlink is outside our project\" check and can be updated like so. I\n> didn't look at the ones in diff-lib.c and dir.c so you may want to\n> double check on what they use it for.\n\nThe first step (renaming and adjusting comments) would look like\nthis.\n\n\n builtin/add.c | 6 +++---\n builtin/apply.c | 8 ++++++--\n builtin/check-ignore.c | 2 +-\n builtin/update-index.c | 4 ++--\n cache.h | 4 ++--\n diff-lib.c | 2 +-\n dir.c | 2 +-\n pathspec.c | 6 +++---\n pathspec.h | 2 +-\n preload-index.c | 2 +-\n symlinks.c | 10 +++++-----\n t/t0008-ignores.sh | 2 +-\n 12 files changed, 27 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex ab1c9e8..7cb80ef 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -155,8 +155,8 @@ static void refresh(int verbose, const char **pathspec)\n \n /*\n * Normalizes argv relative to prefix, via get_pathspec(), and then\n- * runs die_if_path_beyond_symlink() on each path in the normalized\n- * list.\n+ * runs die_if_path_outside_our_project() on each path in the\n+ * normalized list.\n */\n static const char **validate_pathspec(const char **argv, const char *prefix)\n {\n@@ -165,7 +165,7 @@ static const char **validate_pathspec(const char **argv, const char *prefix)\n \tif (pathspec) {\n \t\tconst char **p;\n \t\tfor (p = pathspec; *p; p++) {\n-\t\t\tdie_if_path_beyond_symlink(*p, prefix);\n+\t\t\tdie_if_path_outside_our_project(*p, prefix);\n \t\t}\n \t}\n \ndiff --git a/builtin/apply.c b/builtin/apply.c\nindex 5b882d0..d0b408e 100644\n--- a/builtin/apply.c\n+++ b/builtin/apply.c\n@@ -3469,10 +3469,14 @@ static int check_to_create(const char *new_name, int ok_if_exists)\n \t\t * A leading component of new_name might be a symlink\n \t\t * that is going to be removed with this patch, but\n \t\t * still pointing at somewhere that has the path.\n-\t\t * In such a case, path \"new_name\" does not exist as\n+\t\t * Or it could be the top-level of a working tree of\n+\t\t * a different project that is embedded in our working\n+\t\t * tree.\n+\t\t *\n+\t\t * In such cases, path \"new_name\" does not exist as\n \t\t * far as git is concerned.\n \t\t */\n-\t\tif (has_symlink_leading_path(new_name, strlen(new_name)))\n+\t\tif (path_outside_our_project(new_name, strlen(new_name)))\n \t\t\treturn 0;\n \n \t\treturn EXISTS_IN_WORKTREE;\ndiff --git a/builtin/check-ignore.c b/builtin/check-ignore.c\nindex 0240f99..bce378d 100644\n--- a/builtin/check-ignore.c\n+++ b/builtin/check-ignore.c\n@@ -88,7 +88,7 @@ static int check_ignore(const char *prefix, const char **pathspec)\n \t\tfull_path = prefix_path(prefix, prefix\n \t\t\t\t\t? strlen(prefix) : 0, path);\n \t\tfull_path = check_path_for_gitlink(full_path);\n-\t\tdie_if_path_beyond_symlink(full_path, prefix);\n+\t\tdie_if_path_outside_our_project(full_path, prefix);\n \t\tif (!seen[i]) {\n \t\t\texclude = last_exclude_matching_path(&check, full_path,\n \t\t\t\t\t\t\t -1, &dtype);\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 5c7762e..7c47fa2 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -186,8 +186,8 @@ static int process_path(const char *path)\n \tstruct cache_entry *ce;\n \n \tlen = strlen(path);\n-\tif (has_symlink_leading_path(path, len))\n-\t\treturn error(\"'%s' is beyond a symbolic link\", path);\n+\tif (path_outside_our_project(path, len))\n+\t\treturn error(\"'%s' is outside our working tree\", path);\n \n \tpos = cache_name_pos(path, len);\n \tce = pos < 0 ? NULL : active_cache[pos];\ndiff --git a/cache.h b/cache.h\nindex e1e8ce8..f6359b5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -960,8 +960,8 @@ struct cache_def {\n \tint prefix_len_stat_func;\n };\n \n-extern int has_symlink_leading_path(const char *name, int len);\n-extern int threaded_has_symlink_leading_path(struct cache_def *, const char *, int);\n+extern int path_outside_our_project(const char *name, int len);\n+extern int threaded_path_outside_our_project(struct cache_def *, const char *, int);\n extern int check_leading_path(const char *name, int len);\n extern int has_dirs_only_path(const char *name, int len, int prefix_len);\n extern void schedule_dir_for_removal(const char *name, int len);\ndiff --git a/diff-lib.c b/diff-lib.c\nindex f35de0f..8aff906 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -32,7 +32,7 @@ static int check_removed(const struct cache_entry *ce, struct stat *st)\n \t\t\treturn -1;\n \t\treturn 1;\n \t}\n-\tif (has_symlink_leading_path(ce->name, ce_namelen(ce)))\n+\tif (path_outside_our_project(ce->name, ce_namelen(ce)))\n \t\treturn 1;\n \tif (S_ISDIR(st->st_mode)) {\n \t\tunsigned char sub[20];\ndiff --git a/dir.c b/dir.c\nindex 91cfd99..b90b57b 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1479,7 +1479,7 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const char\n {\n \tstruct path_simplify *simplify;\n \n-\tif (has_symlink_leading_path(path, len))\n+\tif (path_outside_our_project(path, len))\n \t\treturn dir->nr;\n \n \tsimplify = create_simplify(pathspec);\ndiff --git a/pathspec.c b/pathspec.c\nindex 284f397..336149f 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -92,10 +92,10 @@ const char *check_path_for_gitlink(const char *path)\n * Dies if the given path refers to a file inside a symlinked\n * directory in the index.\n */\n-void die_if_path_beyond_symlink(const char *path, const char *prefix)\n+void die_if_path_outside_our_project(const char *path, const char *prefix)\n {\n-\tif (has_symlink_leading_path(path, strlen(path))) {\n+\tif (path_outside_our_project(path, strlen(path))) {\n \t\tint len = prefix ? strlen(prefix) : 0;\n-\t\tdie(_(\"'%s' is beyond a symbolic link\"), path + len);\n+\t\tdie(_(\"'%s' is outside the working tree\"), path + len);\n \t}\n }\ndiff --git a/pathspec.h b/pathspec.h\nindex db0184a..ef816a8 100644\n--- a/pathspec.h\n+++ b/pathspec.h\n@@ -4,6 +4,6 @@\n extern char *find_pathspecs_matching_against_index(const char **pathspec);\n extern void add_pathspec_matches_against_index(const char **pathspec, char *seen, int specs);\n extern const char *check_path_for_gitlink(const char *path);\n-extern void die_if_path_beyond_symlink(const char *path, const char *prefix);\n+extern void die_if_path_outside_our_project(const char *path, const char *prefix);\n \n #endif /* PATHSPEC_H */\ndiff --git a/preload-index.c b/preload-index.c\nindex 49cb08d..b3e57d4 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -55,7 +55,7 @@ static void *preload_thread(void *_data)\n \t\t\tcontinue;\n \t\tif (!ce_path_match(ce, &pathspec))\n \t\t\tcontinue;\n-\t\tif (threaded_has_symlink_leading_path(&cache, ce->name, ce_namelen(ce)))\n+\t\tif (threaded_path_outside_our_project(&cache, ce->name, ce_namelen(ce)))\n \t\t\tcontinue;\n \t\tif (lstat(ce->name, &st))\n \t\t\tcontinue;\ndiff --git a/symlinks.c b/symlinks.c\nindex c2b41a8..baed93f 100644\n--- a/symlinks.c\n+++ b/symlinks.c\n@@ -196,19 +196,19 @@ static int lstat_cache(struct cache_def *cache, const char *name, int len,\n #define USE_ONLY_LSTAT 0\n \n /*\n- * Return non-zero if path 'name' has a leading symlink component\n+ * Return non-zero if path 'name' points outside the working tree\n */\n-int threaded_has_symlink_leading_path(struct cache_def *cache, const char *name, int len)\n+int threaded_path_outside_our_project(struct cache_def *cache, const char *name, int len)\n {\n \treturn lstat_cache(cache, name, len, FL_SYMLINK|FL_DIR, USE_ONLY_LSTAT) & FL_SYMLINK;\n }\n \n /*\n- * Return non-zero if path 'name' has a leading symlink component\n+ * Return non-zero if path 'name' points outside the working tree\n */\n-int has_symlink_leading_path(const char *name, int len)\n+int path_outside_our_project(const char *name, int len)\n {\n-\treturn threaded_has_symlink_leading_path(&default_cache, name, len);\n+\treturn threaded_path_outside_our_project(&default_cache, name, len);\n }\n \n /*\ndiff --git a/t/t0008-ignores.sh b/t/t0008-ignores.sh\nindex 9c1bde1..3881e7d 100755\n--- a/t/t0008-ignores.sh\n+++ b/t/t0008-ignores.sh\n@@ -397,7 +397,7 @@ test_expect_success_multi SYMLINKS 'symlink' '' '\n \n test_expect_success_multi SYMLINKS 'beyond a symlink' '' '\n \ttest_check_ignore \"a/symlink/foo\" 128 &&\n-\ttest_stderr \"fatal: '\\''a/symlink/foo'\\'' is beyond a symbolic link\"\n+\ttest_stderr \"fatal: '\\''a/symlink/foo'\\'' is outside the working tree\"\n '\n \n test_expect_success_multi SYMLINKS 'beyond a symlink from subdirectory' '' '\n"},{"id":"213684","messageId":"CALkWK0k0YMWjhJQFWLv6b7kAHqRm8-Dq23gMVHYfatq1SLUDhQ@mail.gmail.com","threadId":"33386","inReplyTo":"7vvc7vsj4p.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-09T17:41:28Z","receivedAt":"2013-04-09T17:41:28Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> The first step (renaming and adjusting comments) would look like\n> this.\n\nThanks for this! I like the name die_if_path_outside_our_project().\nI'll take care of the rest.`\n"},{"id":"213683","messageId":"7vr4ijsish.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"7vvc7vsj4p.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T17:41:50Z","receivedAt":"2013-04-09T17:41:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> I looked at the output from \"grep has_symlink_leading_path\" and also\n>> for \"die_if_path_beyond\"; all of these places are checking \"I have\n>> this multi-level path; I want to know if the path does not (should\n>> not) be part of the current project\", I think. Certainly the one in\n>> the \"update-index\" is about the same operation as \"git add\" you are\n>> patching.\n>>\n>> Isn't it a better approach to _rename_ the existing function not to\n>> single out \"symlink\"-ness of the path first ? A symlink in the\n>> middle of such a multi-level path that leads to a place outside the\n>> project is _not_ the only way to step out of our project boundary. A\n>> directory in the middle of a multi-level path that is the top-level\n>> of the working tree of a foreign project is another way to step out\n>> of our project boundary. Perhaps\n>>\n>> \tdie_if_path_outside_our_project()\n>> path_outside_our_project()\n>>\n>> And then update the implementation of path_outside_our_project(),\n>> which only took \"symlink in the middle\" into account so far, and\n>> teach it that such a \"top-level of the working tree of a foreign\n>> project\" is also stepping out of our project?\n>>\n>> That way, you do not have to settle on fixing the bug only in \"git\n>> add\" and keep the bug in \"git update-index\", I think.\n>>\n>> I think the hit in builtin/apply.c deals with the same \"beyond\n>> symlink is outside our project\" check and can be updated like so. I\n>> didn't look at the ones in diff-lib.c and dir.c so you may want to\n>> double check on what they use it for.\n>\n> The first step (renaming and adjusting comments) would look like\n> this.\n\nActually, there is another function \"check_leading_path()\" you may\nwant also adjust.\n\n /*\n * Return zero if path 'name' has a leading symlink component or\n * if some leading path component does not exists.\n *\n * Return -1 if leading path exists and is a directory.\n *\n * Return path length if leading path exists and is neither a\n * directory nor a symlink.\n */\n int check_leading_path(const char *name, int len)\n {\n return threaded_check_leading_path(&default_cache, name, len);\n }\n\nI think what the callers of this function care about is if the name\nis a path that should not be added to our index (i.e. points\n\"outside the repository\"). If you had a symlink d that points at e\nwhen our project does have a subdirectory e with file f,\n\n\tcheck_leading_path(\"d/f\")\n\nwants to say \"bad\", even though the real file pointed at, i.e. \"e/f\"\nis inside our working tree, so \"outside our working tree\" is not\nquite correct in the strict sense (this applies equally to\nhas_symlink_leading_path), but I think we should treat the case\nwhere \"d\" (and \"d/f\") belongs to the working tree of a repository\nfor a separate project, that is embedded in our working tree the\nsame way.\n"},{"id":"213687","messageId":"7vmwt7si6z.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0k0YMWjhJQFWLv6b7kAHqRm8-Dq23gMVHYfatq1SLUDhQ@mail.gmail.com","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T17:54:44Z","receivedAt":"2013-04-09T17:54:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sorry for repeated rerolls. I had missed another instance in t0008\nand also the explanation was lacking.\n\n-- >8 --\nSubject: [PATCH] symlinks: rename has_symlink_leading_path() to\n path_outside_our_project()\n\nThe purpose of the function is to prevent a path from getting added\nto our project when its path component steps outside our working\ntree, like this:\n\n\tln -s /etc myetc\n\tgit add myetc/passwd\n\nWe do not want to end up with \"myetc/passwd\" in our index. To make\nsure an attempt to add such a path is caught, the implementation\nchecks if there is any leading symbolic link in the given path\n(adding \"myetc\" itself as a symbolic link to our project is\naccepted).\n\nBut there are other cases to attempt to add a path that do not\nbelong to our project, which do not have to involve a symbolic link\nin the leading path.\n\nRename the function, and die_if_path_beyond_symlink() function, to\nclarify what they are really checking, not how they are checking.\n\nSigned-off-by: Junio C Hamano \n---\n builtin/add.c | 6 +++---\n builtin/apply.c | 8 ++++++--\n builtin/check-ignore.c | 2 +-\n builtin/update-index.c | 4 ++--\n cache.h | 4 ++--\n diff-lib.c | 2 +-\n dir.c | 2 +-\n pathspec.c | 6 +++---\n pathspec.h | 2 +-\n preload-index.c | 2 +-\n symlinks.c | 10 +++++-----\n t/t0008-ignores.sh | 4 ++--\n 12 files changed, 28 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex ab1c9e8..7cb80ef 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -155,8 +155,8 @@ static void refresh(int verbose, const char **pathspec)\n \n /*\n * Normalizes argv relative to prefix, via get_pathspec(), and then\n- * runs die_if_path_beyond_symlink() on each path in the normalized\n- * list.\n+ * runs die_if_path_outside_our_project() on each path in the\n+ * normalized list.\n */\n static const char **validate_pathspec(const char **argv, const char *prefix)\n {\n@@ -165,7 +165,7 @@ static const char **validate_pathspec(const char **argv, const char *prefix)\n \tif (pathspec) {\n \t\tconst char **p;\n \t\tfor (p = pathspec; *p; p++) {\n-\t\t\tdie_if_path_beyond_symlink(*p, prefix);\n+\t\t\tdie_if_path_outside_our_project(*p, prefix);\n \t\t}\n \t}\n \ndiff --git a/builtin/apply.c b/builtin/apply.c\nindex 5b882d0..d0b408e 100644\n--- a/builtin/apply.c\n+++ b/builtin/apply.c\n@@ -3469,10 +3469,14 @@ static int check_to_create(const char *new_name, int ok_if_exists)\n \t\t * A leading component of new_name might be a symlink\n \t\t * that is going to be removed with this patch, but\n \t\t * still pointing at somewhere that has the path.\n-\t\t * In such a case, path \"new_name\" does not exist as\n+\t\t * Or it could be the top-level of a working tree of\n+\t\t * a different project that is embedded in our working\n+\t\t * tree.\n+\t\t *\n+\t\t * In such cases, path \"new_name\" does not exist as\n \t\t * far as git is concerned.\n \t\t */\n-\t\tif (has_symlink_leading_path(new_name, strlen(new_name)))\n+\t\tif (path_outside_our_project(new_name, strlen(new_name)))\n \t\t\treturn 0;\n \n \t\treturn EXISTS_IN_WORKTREE;\ndiff --git a/builtin/check-ignore.c b/builtin/check-ignore.c\nindex 0240f99..bce378d 100644\n--- a/builtin/check-ignore.c\n+++ b/builtin/check-ignore.c\n@@ -88,7 +88,7 @@ static int check_ignore(const char *prefix, const char **pathspec)\n \t\tfull_path = prefix_path(prefix, prefix\n \t\t\t\t\t? strlen(prefix) : 0, path);\n \t\tfull_path = check_path_for_gitlink(full_path);\n-\t\tdie_if_path_beyond_symlink(full_path, prefix);\n+\t\tdie_if_path_outside_our_project(full_path, prefix);\n \t\tif (!seen[i]) {\n \t\t\texclude = last_exclude_matching_path(&check, full_path,\n \t\t\t\t\t\t\t -1, &dtype);\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 5c7762e..7c47fa2 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -186,8 +186,8 @@ static int process_path(const char *path)\n \tstruct cache_entry *ce;\n \n \tlen = strlen(path);\n-\tif (has_symlink_leading_path(path, len))\n-\t\treturn error(\"'%s' is beyond a symbolic link\", path);\n+\tif (path_outside_our_project(path, len))\n+\t\treturn error(\"'%s' is outside our working tree\", path);\n \n \tpos = cache_name_pos(path, len);\n \tce = pos < 0 ? NULL : active_cache[pos];\ndiff --git a/cache.h b/cache.h\nindex e1e8ce8..f6359b5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -960,8 +960,8 @@ struct cache_def {\n \tint prefix_len_stat_func;\n };\n \n-extern int has_symlink_leading_path(const char *name, int len);\n-extern int threaded_has_symlink_leading_path(struct cache_def *, const char *, int);\n+extern int path_outside_our_project(const char *name, int len);\n+extern int threaded_path_outside_our_project(struct cache_def *, const char *, int);\n extern int check_leading_path(const char *name, int len);\n extern int has_dirs_only_path(const char *name, int len, int prefix_len);\n extern void schedule_dir_for_removal(const char *name, int len);\ndiff --git a/diff-lib.c b/diff-lib.c\nindex f35de0f..8aff906 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -32,7 +32,7 @@ static int check_removed(const struct cache_entry *ce, struct stat *st)\n \t\t\treturn -1;\n \t\treturn 1;\n \t}\n-\tif (has_symlink_leading_path(ce->name, ce_namelen(ce)))\n+\tif (path_outside_our_project(ce->name, ce_namelen(ce)))\n \t\treturn 1;\n \tif (S_ISDIR(st->st_mode)) {\n \t\tunsigned char sub[20];\ndiff --git a/dir.c b/dir.c\nindex 91cfd99..b90b57b 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1479,7 +1479,7 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const char\n {\n \tstruct path_simplify *simplify;\n \n-\tif (has_symlink_leading_path(path, len))\n+\tif (path_outside_our_project(path, len))\n \t\treturn dir->nr;\n \n \tsimplify = create_simplify(pathspec);\ndiff --git a/pathspec.c b/pathspec.c\nindex 284f397..336149f 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -92,10 +92,10 @@ const char *check_path_for_gitlink(const char *path)\n * Dies if the given path refers to a file inside a symlinked\n * directory in the index.\n */\n-void die_if_path_beyond_symlink(const char *path, const char *prefix)\n+void die_if_path_outside_our_project(const char *path, const char *prefix)\n {\n-\tif (has_symlink_leading_path(path, strlen(path))) {\n+\tif (path_outside_our_project(path, strlen(path))) {\n \t\tint len = prefix ? strlen(prefix) : 0;\n-\t\tdie(_(\"'%s' is beyond a symbolic link\"), path + len);\n+\t\tdie(_(\"'%s' is outside the working tree\"), path + len);\n \t}\n }\ndiff --git a/pathspec.h b/pathspec.h\nindex db0184a..ef816a8 100644\n--- a/pathspec.h\n+++ b/pathspec.h\n@@ -4,6 +4,6 @@\n extern char *find_pathspecs_matching_against_index(const char **pathspec);\n extern void add_pathspec_matches_against_index(const char **pathspec, char *seen, int specs);\n extern const char *check_path_for_gitlink(const char *path);\n-extern void die_if_path_beyond_symlink(const char *path, const char *prefix);\n+extern void die_if_path_outside_our_project(const char *path, const char *prefix);\n \n #endif /* PATHSPEC_H */\ndiff --git a/preload-index.c b/preload-index.c\nindex 49cb08d..b3e57d4 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -55,7 +55,7 @@ static void *preload_thread(void *_data)\n \t\t\tcontinue;\n \t\tif (!ce_path_match(ce, &pathspec))\n \t\t\tcontinue;\n-\t\tif (threaded_has_symlink_leading_path(&cache, ce->name, ce_namelen(ce)))\n+\t\tif (threaded_path_outside_our_project(&cache, ce->name, ce_namelen(ce)))\n \t\t\tcontinue;\n \t\tif (lstat(ce->name, &st))\n \t\t\tcontinue;\ndiff --git a/symlinks.c b/symlinks.c\nindex c2b41a8..baed93f 100644\n--- a/symlinks.c\n+++ b/symlinks.c\n@@ -196,19 +196,19 @@ static int lstat_cache(struct cache_def *cache, const char *name, int len,\n #define USE_ONLY_LSTAT 0\n \n /*\n- * Return non-zero if path 'name' has a leading symlink component\n+ * Return non-zero if path 'name' points outside the working tree\n */\n-int threaded_has_symlink_leading_path(struct cache_def *cache, const char *name, int len)\n+int threaded_path_outside_our_project(struct cache_def *cache, const char *name, int len)\n {\n \treturn lstat_cache(cache, name, len, FL_SYMLINK|FL_DIR, USE_ONLY_LSTAT) & FL_SYMLINK;\n }\n \n /*\n- * Return non-zero if path 'name' has a leading symlink component\n+ * Return non-zero if path 'name' points outside the working tree\n */\n-int has_symlink_leading_path(const char *name, int len)\n+int path_outside_our_project(const char *name, int len)\n {\n-\treturn threaded_has_symlink_leading_path(&default_cache, name, len);\n+\treturn threaded_path_outside_our_project(&default_cache, name, len);\n }\n \n /*\ndiff --git a/t/t0008-ignores.sh b/t/t0008-ignores.sh\nindex 9c1bde1..a00ee75 100755\n--- a/t/t0008-ignores.sh\n+++ b/t/t0008-ignores.sh\n@@ -397,7 +397,7 @@ test_expect_success_multi SYMLINKS 'symlink' '' '\n \n test_expect_success_multi SYMLINKS 'beyond a symlink' '' '\n \ttest_check_ignore \"a/symlink/foo\" 128 &&\n-\ttest_stderr \"fatal: '\\''a/symlink/foo'\\'' is beyond a symbolic link\"\n+\ttest_stderr \"fatal: '\\''a/symlink/foo'\\'' is outside the working tree\"\n '\n \n test_expect_success_multi SYMLINKS 'beyond a symlink from subdirectory' '' '\n@@ -405,7 +405,7 @@ test_expect_success_multi SYMLINKS 'beyond a symlink from subdirectory' '' '\n \t\tcd a &&\n \t\ttest_check_ignore \"symlink/foo\" 128\n \t) &&\n-\ttest_stderr \"fatal: '\\''symlink/foo'\\'' is beyond a symbolic link\"\n+\ttest_stderr \"fatal: '\\''symlink/foo'\\'' is outside the working tree\"\n '\n \n ############################################################################\n-- \n1.8.2.1-465-gf55e5b3\n"},{"id":"213688","messageId":"CALkWK0=Py3_LiL=88GOiQkHsYWERnNPvvU1M5wDhu+WioN56Uw@mail.gmail.com","threadId":"33386","inReplyTo":"7vr4ijsish.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-09T17:56:39Z","receivedAt":"2013-04-09T17:56:39Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> I think what the callers of this function care about is if the name\n> is a path that should not be added to our index (i.e. points\n> \"outside the repository\"). If you had a symlink d that points at e\n> when our project does have a subdirectory e with file f,\n>\n> check_leading_path(\"d/f\")\n>\n> wants to say \"bad\", even though the real file pointed at, i.e. \"e/f\"\n> is inside our working tree, so \"outside our working tree\" is not\n> quite correct in the strict sense (this applies equally to\n> has_symlink_leading_path), but\n\nActually, you introduced one naming regression:\nhas_symlink_leading_path() is a good name for what the function does,\nas opposed to die_if_path_outside_our_tree(), which is misleading.\nWhat about die_if_path_contains_links() to encapsulate gitlinks and\nsymlinks?\n\n> I think we should treat the case\n> where \"d\" (and \"d/f\") belongs to the working tree of a repository\n> for a separate project, that is embedded in our working tree the\n> same way.\n\nI'm not too sure about this. It means that I can have symlinks to\nfiles in various parts of my worktree, but not to directories. Isn't\nthis an absurd limitation to impose? I'm not saying that it's\nparticularly useful to have a symlink at / pointing to a directory\ndeeply nested in your repository, but that limitations must have some\nconcrete rationale.\n\nAnyway, since we're not introducing any regressions (as\nhas_symlink_leading_path imposes the same absurd limitation), we don't\nhave to fix this now. But it's certainly something worth fixing in\nthe future, I think.\n"},{"id":"213694","messageId":"CALkWK0nF0aTLDZyPXfa=4kp=BnDPm3+_ZfMvP_6ZcimeJtzxvw@mail.gmail.com","threadId":"33386","inReplyTo":"7vmwt7si6z.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-09T18:17:40Z","receivedAt":"2013-04-09T18:17:40Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> But there are other cases to attempt to add a path that do not\n> belong to our project, which do not have to involve a symbolic link\n> in the leading path.\n\nThe reader is now wondering what this could possibly be, and why you\ndidn't send this patch earlier. Perhaps clarify with: s/there are\ncases/there may be cases/ and append \"One such case that we currently\ndon't handle yet is a path inside another git repository in our\nworktree, as demonstrated by test tXXXX.X.\"\n"},{"id":"213696","messageId":"51645EC6.407@gmail.com","threadId":"33386","inReplyTo":"7vvc7vsj4p.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2013-04-09T18:32:38Z","receivedAt":"2013-04-09T18:32:38Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 09.04.2013 19:34, Junio C Hamano pisze:\n\n> -\tif (has_symlink_leading_path(path, len))\n> -\t\treturn error(\"'%s' is beyond a symbolic link\", path);\n> +\tif (path_outside_our_project(path, len))\n> +\t\treturn error(\"'%s' is outside our working tree\", path);\n> \n\nDon't we lose important information here? Or we shouldn't care?\n\n-- \nJakub Narębski\n"},{"id":"213698","messageId":"7vehejsfpe.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0=Py3_LiL=88GOiQkHsYWERnNPvvU1M5wDhu+WioN56Uw@mail.gmail.com","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T18:48:29Z","receivedAt":"2013-04-09T18:48:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Junio C Hamano wrote:\n>> I think what the callers of this function care about is if the name\n>> is a path that should not be added to our index (i.e. points\n>> \"outside the repository\"). If you had a symlink d that points at e\n>> when our project does have a subdirectory e with file f,\n>>\n>> check_leading_path(\"d/f\")\n>>\n>> wants to say \"bad\", even though the real file pointed at, i.e. \"e/f\"\n>> is inside our working tree, so \"outside our working tree\" is not\n>> quite correct in the strict sense (this applies equally to\n>> has_symlink_leading_path), but\n>\n> Actually, you introduced one naming regression:\n> has_symlink_leading_path() is a good name for what the function does,\n> as opposed to die_if_path_outside_our_tree(), which is misleading.\n> What about die_if_path_contains_links() to encapsulate gitlinks and\n> symlinks?\n\nThe cases we know that \"$d/f\" (where $d is a path that is one or\nmore levels, e.g. \"dir\", \"d/i\", or \"d/i/r\") is bad are when\n\n - \"$d\" is a symlink, because what you could add to the index is \"$d\"\n and nothing underneath it; or\n\n - \"$d\" is a directory that is the top level of the working tree\n that is controled by \"$d/.git\", because what you could add to the\n index is \"$d\" and nothing underneath it.\n\nIf \"$d\" were added to our index, the former will make 120000 entry\nand the latter will make 160000 entry. But the user may not want to\nadd $d ever to our project, so in that case, neither will give us a\nsymlink or a gitlink.\n\nWe should find a word that makes it clear that \"this path is beyond\nsomething we _could_ add\". I do not think \"link\" is a good word for\nit. It shares the same mistake that led to the original misnomer,\ni.e. \"the case we happened to notice was when we have symlink so\nlet's name it with 'symlink' somewhere in it.\"\n\n>> I think we should treat the case\n>> where \"d\" (and \"d/f\") belongs to the working tree of a repository\n>> for a separate project, that is embedded in our working tree the\n>> same way.\n>\n> I'm not too sure about this. It means that I can have symlinks to\n> files in various parts of my worktree, but not to directories.\n\nIt does not mean that. It is valid to do\n\n\tln -s myetc /etc\n git add myetc\n\nIt is NOT valid to do\n\n\tgit add myetc/passwd\n\nOne can have symlinks to anywhere all one wants. We track symlinks.\n\nIt is the same for the top-level of the working tree of a separate\nproject, be it a submodule or not. It is valid to do\n\n\tmkdir foo && (cd foo && git init && >file)\n git add foo\n\nIt is NOT valid to do\n\n\tgit add foo/file\n"},{"id":"213699","messageId":"7va9p7sfl8.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0nF0aTLDZyPXfa=4kp=BnDPm3+_ZfMvP_6ZcimeJtzxvw@mail.gmail.com","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T18:50:59Z","receivedAt":"2013-04-09T18:50:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> The reader is now wondering what this could possibly be, and why you\n> didn't send this patch earlier. \n\nBecause it wasn't written back then?\n\n> Perhaps clarify with: s/there are\n> cases/there may be cases/ and append \"One such case that we currently\n> don't handle yet is a path inside another git repository in our\n> worktree, as demonstrated by test tXXXX.X.\"\n\nI think \"we currently don't handle\" is a misstatement. It is not a\nbug that we don't.\n"},{"id":"213700","messageId":"7v61zvsfkj.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"51645EC6.407@gmail.com","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T18:51:24Z","receivedAt":"2013-04-09T18:51:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narębski writes:\n\n> W dniu 09.04.2013 19:34, Junio C Hamano pisze:\n>\n>> -\tif (has_symlink_leading_path(path, len))\n>> -\t\treturn error(\"'%s' is beyond a symbolic link\", path);\n>> +\tif (path_outside_our_project(path, len))\n>> +\t\treturn error(\"'%s' is outside our working tree\", path);\n>> \n>\n> Don't we lose important information here? Or we shouldn't care?\n\nWhat important information is it?\n"},{"id":"213702","messageId":"516464E4.10503@gmail.com","threadId":"33386","inReplyTo":"7v61zvsfkj.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2013-04-09T18:58:44Z","receivedAt":"2013-04-09T18:58:44Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n> Jakub Narębski writes:\n> \n>> W dniu 09.04.2013 19:34, Junio C Hamano pisze:\n>>\n>>> -\tif (has_symlink_leading_path(path, len))\n>>> -\t\treturn error(\"'%s' is beyond a symbolic link\", path);\n>>> +\tif (path_outside_our_project(path, len))\n>>> +\t\treturn error(\"'%s' is outside our working tree\", path);\n>>> \n>>\n>> Don't we lose important information here? Or we shouldn't care?\n> \n> What important information is it?\n\nThat the cause is symbolic link (or other git repository, in the future).\n\n-- \nJakub Narębski\n"},{"id":"213703","messageId":"7vwqsbr05n.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"7va9p7sfl8.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T19:09:40Z","receivedAt":"2013-04-09T19:09:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Ramkumar Ramachandra writes:\n>\n>> The reader is now wondering what this could possibly be, and why you\n>> didn't send this patch earlier. \n>\n> Because it wasn't written back then?\n>\n>> Perhaps clarify with: s/there are\n>> cases/there may be cases/ and append \"One such case that we currently\n>> don't handle yet is a path inside another git repository in our\n>> worktree, as demonstrated by test tXXXX.X.\"\n>\n> I think \"we currently don't handle\" is a misstatement. It is not a\n> bug that we don't.\n\nWe can think of it this way.\n\nIn your working tree, there is an upper bound for the paths you can\ninclude in your commit. When you are at the top-level of your\nworking tree, you do not say \"git add ../f\" or \"git add ../d/f\".\nThe root-level of your working tree is an upper bound and you do not\ncross that boundary.\n\nIt turns out that there are lower bounds for the paths as well.\nWhen we say \"Git tracks symbolic links\", anything that appears\nbeyond a symbolic link is beyond that boundary. If we track a\nsymbolic link \"l\", we can of course add \"l\". When \"l\" leads to\na directory somewhere else, the filesystem gives you an illusion\nthat there are things under \"l\" (e.g. \"l\" points at \"/etc\" and there\nis \"l/passwd\" there), but that is beyond the boundary. You do not\nadd \"l/passwd\". Otherwise \"git add l\" would become meaningless.\nDoes it add the symbolic link itself, or all the files in there,\npretending \"l\" is actually a directory? We have chosen to say it is\nthe former, and apply that rule consistently.\n\nIt is the same for \"Git tracks submodules\", which defines that the\ntop-level of the submodule working tree as such a lower boundary.\n"},{"id":"213704","messageId":"7vsj2zr04z.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"516464E4.10503@gmail.com","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T19:10:04Z","receivedAt":"2013-04-09T19:10:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narębski writes:\n\n> Junio C Hamano wrote:\n>> Jakub Narębski writes:\n>> \n>>> W dniu 09.04.2013 19:34, Junio C Hamano pisze:\n>>>\n>>>> -\tif (has_symlink_leading_path(path, len))\n>>>> -\t\treturn error(\"'%s' is beyond a symbolic link\", path);\n>>>> +\tif (path_outside_our_project(path, len))\n>>>> +\t\treturn error(\"'%s' is outside our working tree\", path);\n>>>> \n>>>\n>>> Don't we lose important information here? Or we shouldn't care?\n>> \n>> What important information is it?\n>\n> That the cause is symbolic link (or other git repository, in the future).\n\nAnd in what way is it important?\n"},{"id":"213716","messageId":"7vzjx7phti.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0nF0aTLDZyPXfa=4kp=BnDPm3+_ZfMvP_6ZcimeJtzxvw@mail.gmail.com","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-09T20:31:05Z","receivedAt":"2013-04-09T20:31:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Junio C Hamano wrote:\n>> But there are other cases to attempt to add a path that do not\n>> belong to our project, which do not have to involve a symbolic link\n>> in the leading path.\n>\n> The reader is now wondering what this could possibly be, and why you\n> didn't send this patch earlier. Perhaps clarify with: s/there are\n> cases/there may be cases/ and append \"One such case that we currently\n> don't handle yet is a path inside another git repository in our\n> worktree, as demonstrated by test tXXXX.X.\"\n\nI _think_ I misread what you meant to say in the above.\n\nWe can go either way between \"are cases\" or \"may be cases\". I meant\nit as an immediate predecessor ([PATCH 1/n]) to the patch you were\nworking on ([PATCH 2/n] and later), so in that context, it does not\nmatter. [PATCH 2/n] will start as \"Now the naming is saner, let's\nstart noticing when the user gives a path is beyond our project\nboundary because it is under control of another repository by adding\nnecessary logic to that function.\"\n\nAnd I also misread \"we currently don't handle\" above as \"but we\nreally should allow adding d/f when d is at the top of the working\ntree of another project\", but that was not what you meant to say.\nInstead, \"We do not notice such a bad case in today's code yet\" was\nwhat you meant. But if we are to use \"there are cases\" in [1/n] and\nstart [2/n] with \"Now we have renamed, let's do this\", then we do\nnot have to bother saying anything in [1/n] about the upcoming\nchange in [2/n], especially the patches come back-to-back in a\nsingle series.\n"},{"id":"213755","messageId":"2B3B803A74A34749B8174B7C901A076A@PhilipOakley","threadId":"33386","inReplyTo":"CALkWK0mG6FFd_K1Ei-r--58i0j94or5240k1hEVbed8kJ+ZURw@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2013-04-10T07:23:46Z","receivedAt":"2013-04-10T07:23:46Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Ramkumar Ramachandra\" \nSent: Monday, April 08, 2013 10:03 PM\n> This is going nowhere. You're stuck at making the current submodule\n> system work, not answering my questions, diverting conversation,\n> repeatedly asking the same stupid questions, labelling everything that\n> I say \"subjective\", and refusing to look at the objective counterpart\n> (aka, the code). It's clear to me that no matter how many more emails\n> I write, you're not going to concede.\n>\n> I'm not interested in wasting any more of my time with this nonsense.\n>\n> I give up.\n> --\nPlease don't \"give up\". It is a bit of a 'wicked' problem [1].\n\nYes to taking a rest, stepping back and trying to summarise/review what \nwas discussed.\n\nI couldn't keep up with all the discussion, and I doubt many others kept \nup, especially those who have been frustrated in their (mis-) use of \nsubmodules. Do remember that Junio has multiple roles which belie the \nsoftness of the word 'maintainer'. It includes \"Defender of the \nHeritage\" in the same way that keepers of ancient monuments will want \nvisitors to enjoy the site, but rail against a garish new stainless \nsteel and glass entrance to the Colosseum (choose you local heritage \nsite) (see [1] again).\n\nI get confused (about sub-modules) with msysgit where git.git is a \nsub-module, and is the fastest moving (an inversion of control issue), \nand when hacking at (just) the msys level when the git sub-module isn't \nin sync.\n\nIn many ways sub-module tracking is like file renames and empty \ndirectories (both of which come up a lot). The submodule meta \ninformation issue has great similarity to the empty directories issue. \nIt's about meta information, not about content (which is certified \nverified by sha1), and about how users know what is going on and get a \n(natural) feeling of control (without upsetting other users/controllers) \n.\n\nregards\n\nPhilip\n[now to schedule some time to do the catch up reading. $dayjob beckons]\n\n[1] www.poppendieck.com/wicked.htm \n"},{"id":"213772","messageId":"CALkWK0k9nLhr_JTDvJq=7YXX8DoqXy850zC__Ux1kJxkY0BJhA@mail.gmail.com","threadId":"33386","inReplyTo":"7vzjx7phti.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-10T13:25:44Z","receivedAt":"2013-04-10T13:25:44Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> And I also misread \"we currently don't handle\" above as \"but we\n> really should allow adding d/f when d is at the top of the working\n> tree of another project\", but that was not what you meant to say.\n> Instead, \"We do not notice such a bad case in today's code yet\" was\n> what you meant. But if we are to use \"there are cases\" in [1/n] and\n> start [2/n] with \"Now we have renamed, let's do this\", then we do\n> not have to bother saying anything in [1/n] about the upcoming\n> change in [2/n], especially the patches come back-to-back in a\n> single series.\n\nExactly. Yeah, I don't think you patch makes sense as a standalone\nanyway: I'll use appropriate wording when I roll the series, so it\nfollows nicely.\n"},{"id":"213773","messageId":"CALkWK0=_NAfOdVKet9k33ZuwAtcaZ42ok_r+Ln1rswcaJi6eTA@mail.gmail.com","threadId":"33386","inReplyTo":"7vehejsfpe.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-10T13:38:28Z","receivedAt":"2013-04-10T13:38:28Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> One can have symlinks to anywhere all one wants. We track symlinks.\n> [...]\n\nYes, I know. We store symlinks as blobs containing one line, the path\nto the file, without a trailing newline. And we have a mode for it to\ndistinguish it from regular files.\n\nWhat I meant is:\n\n echo \"baz\" >newfile\n cd foo/bar/quux\n ln -s ../../../newfile\n cd ../../.. # Back to toplevel\n git add foo/bar/quux/newfile\n\nThis is allowed. While:\n\n cd foo/bar/quux\n echo \"baz\" >newfile\n cd ../../.. # Back to toplevel\n ln -s foo/bar/quux\n git add quux/newfile\n\nis disallowed. Then again, if we were to replace the last line with:\n\n cd quux\n git add newfile\n\nand it works.\n\nNotice that both symlinks are pointing to paths inside out repository,\nand the only difference is that the second example attempts to add a\npath with a symlink as the non-final component. The path is not\npointing \"outside\" our repository, as the function name would\nindicate.\n\nAnyway, it's just a minor detail that would be nice to fix in the\nfuture. Nothing urgent.\n"},{"id":"213783","messageId":"7v8v4qnyi6.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CALkWK0k9nLhr_JTDvJq=7YXX8DoqXy850zC__Ux1kJxkY0BJhA@mail.gmail.com","subject":"Re: [PATCH 2/2] add: refuse to add paths beyond repository boundaries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-10T16:25:53Z","receivedAt":"2013-04-10T16:25:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Exactly. Yeah, I don't think you patch makes sense as a standalone\n> anyway.\n\nYes, it was purely a preparatory step.\n"},{"id":"214599","messageId":"CACsJy8BoWfng7p=kHbiF9s6XYH1mPMtAGW6BRz54uYM5454O0w@mail.gmail.com","threadId":"33386","inReplyTo":"20130407212342.GA19857@elie.Belkin","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-04-17T10:37:46Z","receivedAt":"2013-04-17T10:37:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Apr 8, 2013 at 7:23 AM, Jonathan Nieder wrote:\n> Ramkumar Ramachandra wrote:\n>\n>> It's about the core object code of git parsing links, as\n>> opposed to a fringe submodule.c/ submodule.sh parsing .gitmodules.\n>\n> What's stopping the core object code of git parsing .gitmodules? What\n> is the core object code? How does this compare to other metadata\n> files like .gitattributes and .gitignore?\n\nSomewhat related to the topic. Why can't .gitattributes be used for\nstoring what's currently in .gitmodules?\n--\nDuy\n"},{"id":"214602","messageId":"CALkWK0mcnA8Qss3uxRXhfHst65RLkv43wje9xdFxmFKi7MtZvA@mail.gmail.com","threadId":"33386","inReplyTo":"CACsJy8BoWfng7p=kHbiF9s6XYH1mPMtAGW6BRz54uYM5454O0w@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-17T11:06:39Z","receivedAt":"2013-04-17T11:06:39Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Duy Nguyen wrote:\n> Somewhat related to the topic. Why can't .gitattributes be used for\n> storing what's currently in .gitmodules?\n\nIt can. It's just a small syntax change from \"key = value\" attributes\ninside a toplevel [submodule ] section separated by newlines, to\na path marked with multiple \"key=value\" attributes separated by\nwhitespace. However, we don't want to make this change because these\nsubmodule attributes are somewhat \"different\" from .gitattributes\nattributes.\n\nRoughly speaking, the current .gitmodules design treats submodule\ndirectories as \"directories with special attributes\", with two\ndifferences: these directories have a special mode in the index, and a\ncommit object is created in the database to represent the \"partial\nstate\" of this submodule. If you think about it, the information\nstored in the commit object is no less/ no more important than the\npath-attribute mapping in .gitmodules. I was arguing for using a\nspecial OBJ_LINK to represent the full state of the submodule, and\ndoing away with the attributes altogether, but not everyone agrees.\n"},{"id":"214604","messageId":"CACsJy8Bz+yNefiiwEivaaUgRymHTmUUKEFs8_uuonhmRfT3UGg@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0mcnA8Qss3uxRXhfHst65RLkv43wje9xdFxmFKi7MtZvA@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-04-17T11:27:40Z","receivedAt":"2013-04-17T11:27:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Apr 17, 2013 at 9:06 PM, Ramkumar Ramachandra\n wrote:\n> Duy Nguyen wrote:\n>> Somewhat related to the topic. Why can't .gitattributes be used for\n>> storing what's currently in .gitmodules?\n>\n> It can. It's just a small syntax change from \"key = value\" attributes\n> inside a toplevel [submodule ] section separated by newlines, to\n> a path marked with multiple \"key=value\" attributes separated by\n> whitespace. However, we don't want to make this change because these\n> submodule attributes are somewhat \"different\" from .gitattributes\n> attributes.\n>\n> Roughly speaking, the current .gitmodules design treats submodule\n> directories as \"directories with special attributes\", with two\n> differences: these directories have a special mode in the index, and a\n> commit object is created in the database to represent the \"partial\n> state\" of this submodule.\n\nThat was my thinking. .gitmodules would break if a user moves the\nsubmodule manually (or even if .gitattributes is used)\n\n> If you think about it, the information\n> stored in the commit object is no less/ no more important than the\n> path-attribute mapping in .gitmodules. I was arguing for using a\n> special OBJ_LINK to represent the full state of the submodule, and\n> doing away with the attributes altogether, but not everyone agrees.\n\nInclude me to those everyone. url feels like a local thing that should\nnot stay in object database (another way of looking at it is like an\nemail address: the primary one fixed in stone in commits with .mailmap\nfor future substitution). Other attributes like .update,\n.fetchRecursiveSubmodules... definitely should not be stored in object\ndatabase. I think if they are stored in the submodule's config file,\nthen the manual move problem above will go away.\n\nAnd if you're dead set on storing some submodule state in object\ndatabase, why not reuse tag object with some nea header lines?\n--\nDuy\n"},{"id":"214606","messageId":"CALkWK0m9QmZaSDruY=+2F-Kkw+fd6E1TYCTBpVQHRJrzq2VjCQ@mail.gmail.com","threadId":"33386","inReplyTo":"CACsJy8Bz+yNefiiwEivaaUgRymHTmUUKEFs8_uuonhmRfT3UGg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-17T11:56:41Z","receivedAt":"2013-04-17T11:56:41Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Duy Nguyen wrote:\n> Include me to those everyone. url feels like a local thing that should\n> not stay in object database (another way of looking at it is like an\n> email address: the primary one fixed in stone in commits with .mailmap\n> for future substitution).\n\nWe've been over this several times in earlier emails. That's like\nsaying that a blob should not be stored in the object database,\nbecause it is not \"fixed in stone\" (my OBJ_LINK is just a special kind\nof blob, as I've repeated many times already). I don't rely on what I\n\"feel\", which is why I started out by posting an implementation: the\nimplementation seems to indicate that getting an OBJ_LINK will\nsimplify a lot of things. And that is my primary criterion for\ndeciding: if the implementation is simple and elegant, it must clearly\nbe doing something right.\n\nAgain, I'm not saying that my approach is Correct and Final. What I'm\nsaying is: \"Here's what I've done. Something interesting is going on.\n It's probably worth a look?\"\n\n> Other attributes like .update,\n> .fetchRecursiveSubmodules... definitely should not be stored in object\n> database.\n\n\"Coffee and other beverages definitely should be served cold.\"\nAll very nice to say, but I don't see any rationale.\n\n> I think if they are stored in the submodule's config file,\n> then the manual move problem above will go away.\n\nWhat? The submodule's .git/config? Why should a submodule repository\nknow that it is being used as a submodule? What inherent properties\nof a git repository change if it is being used as a submodule?\n\n> And if you're dead set on storing some submodule state in object\n> database,\n\nI'm not. I'm just saying that it seems to be an interesting\nalternative approach. Considering that nobody else brought up a real\nalternative approach, and chose to just keep defending .gitmodules to\nthe death, it's the only other approach we have.\n\n> why not reuse tag object with some nea header lines?\n\nOr a unified blob, which is currently what we have. The point is to\nhave structured parseable information that the object-parsing code of\ngit code and easily slurp and give to the rest of git-core.\n\nPlease clear your reading backlog to avoid bringing up the same points\nover and over again.\n"},{"id":"214607","messageId":"CACsJy8Bw6x=WiQUi0C4ZtEQiA9ia8ZFgz5uH61K6kf-JPy1xyg@mail.gmail.com","threadId":"33386","inReplyTo":"CALkWK0m9QmZaSDruY=+2F-Kkw+fd6E1TYCTBpVQHRJrzq2VjCQ@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-04-17T12:06:58Z","receivedAt":"2013-04-17T12:06:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Apr 17, 2013 at 9:56 PM, Ramkumar Ramachandra\n wrote:\n>> why not reuse tag object with some nea header lines?\n>\n> Or a unified blob, which is currently what we have. The point is to\n> have structured parseable information that the object-parsing code of\n> git code and easily slurp and give to the rest of git-core.\n\nI think you misunderstood. I meant instead of introducing new object\ntype OBJ_LINK, you can reuse tag object and add new header lines for\nyour purposes.\n\n> Please clear your reading backlog to avoid bringing up the same points\n> over and over again.\n\nYep. I'll shut up until it's cleared.\n--\nDuy\n"},{"id":"214608","messageId":"CALkWK0mp5H2hXXBX6YDMSto+DHjXsAojQDb3vcJksUwoOV=2jg@mail.gmail.com","threadId":"33386","inReplyTo":"CACsJy8Bw6x=WiQUi0C4ZtEQiA9ia8ZFgz5uH61K6kf-JPy1xyg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-17T12:14:17Z","receivedAt":"2013-04-17T12:14:17Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Duy Nguyen wrote:\n> On Wed, Apr 17, 2013 at 9:56 PM, Ramkumar Ramachandra\n> wrote:\n>>> why not reuse tag object with some nea header lines?\n>>\n>> Or a unified blob, which is currently what we have. The point is to\n>> have structured parseable information that the object-parsing code of\n>> git code and easily slurp and give to the rest of git-core.\n>\n> I think you misunderstood. I meant instead of introducing new object\n> type OBJ_LINK, you can reuse tag object and add new header lines for\n> your purposes.\n\nOh, I interpreted your typo \"nea\" as \"neat\", when you meant \"new\".\nYeah, it's worth exploring: I don't know what backward compatibility\nbenefits it will yield yet.\n"},{"id":"214619","messageId":"7vsj2pf8ni.fsf@alter.siamese.dyndns.org","threadId":"33386","inReplyTo":"CACsJy8BoWfng7p=kHbiF9s6XYH1mPMtAGW6BRz54uYM5454O0w@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-17T16:01:53Z","receivedAt":"2013-04-17T16:01:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> Somewhat related to the topic. Why can't .gitattributes be used for\n> storing what's currently in .gitmodules?\n\nYou _could_ use gitattributes to encode, but it goes against what a\ngitattributes file does or is for. It is a mechanism to associate\ngroups of paths (that may not even exist) to a set of attributes.\nYou could list a single pattern that happens to match a single path\nand at the implementation level you may be able to make it work, but\nat the design/philosophical level, it is wrong.\n\nWe need info on each submodule and we need to key it with the name\nof the submodule, not with its path. At any given time, a single\nsubmodule lives at (at most) one path, so you could still use path\nas a key in the .gitattributes, but when you need to move the\nsubmodule path, you would need to update the entry for the submodule\nin .gitattributes file by finding a pattern that match the old path\nand making it a pattern that match the new path.\n\nWe have a much more suitable file format that we use to associate\nvarious values to keys: the config format. Also having a file that\nis only about submodules and nothing else means we could write a\ncontent-aware smart ll-merge driver that can take advantage of the\nknowledge that it is written in the config format and it talks about\nsubmodules.\n\nThe answer to \"why can't\" question is \"no\". No, there is no reason\nwhy you can't use it. We don't do it, because it just does not make\nsense.\n"},{"id":"214653","messageId":"BCD944518C304D7D9809346DD899B68F@PhilipOakley","threadId":"33386","inReplyTo":"CACsJy8Bz+yNefiiwEivaaUgRymHTmUUKEFs8_uuonhmRfT3UGg@mail.gmail.com","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2013-04-17T23:17:21Z","receivedAt":"2013-04-17T23:17:21Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Ramkumar Ramachandra\" \nSent: Wednesday, April 17, 2013 12:56 PM\n>\n> We've been over this several times in earlier emails. [...]\n\n> Again, I'm not saying that my approach is Correct and Final. What I'm\n> saying is: \"Here's what I've done. Something interesting is going on.\n> It's probably worth a look?\"\n>\n[...]\n> The point is to\n> have structured parseable information that the object-parsing code of\n> git code and easily slurp and give to the rest of git-core.\n>\n> Please clear your reading backlog to avoid bringing up the same points\n> over and over again.\n> --\n\nRam,\nThe email thread is pretty long with a lot of too and fro, that would be \ndifficult to catch up on (too much $dayjob+$family vs $sparetime).\n\nWould it be possible to summarise the key points and proposals of where \nthe subject is now?\n\nThe submodules does need 'fixing', as does agreeing the problem and \nabuse cases.\n\nPhilip \n"},{"id":"214690","messageId":"CALkWK0nb0Z2Uf7XJHgEiQ5rM7EVVA7-ZMhMOjMewmKKGYziw2w@mail.gmail.com","threadId":"33386","inReplyTo":"BCD944518C304D7D9809346DD899B68F@PhilipOakley","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-18T07:50:31Z","receivedAt":"2013-04-18T07:50:31Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Philip Oakley wrote:\n> Would it be possible to summarise the key points and proposals of where the\n> subject is now?\n\nSure.\n\nIf you want an update from the current approach, wait for a v2; I'm\ncooking it for some time, and getting some resulting ideas merged into\nupstream early (look for clone.submoduleGitDir on the list, for\ninstance). When upstream is in better shape to ease in a better\nfundamental design, I'll post my v2 to the list. I'll refrain from\nposting any updates now, because I don't think the resulting\ndiscussion will generate any value.\n\nIf you want to know what this thread was about, I think [1] and [2]\nsummarize my arguments quite well.\n\n[1]: http://thread.gmane.org/gmane.comp.version-control.git/220047/focus=220436\n[2]: http://thread.gmane.org/gmane.comp.version-control.git/220047/focus=220495\n"},{"id":"214847","messageId":"51717A28.5040106@web.de","threadId":"33386","inReplyTo":"BCD944518C304D7D9809346DD899B68F@PhilipOakley","subject":"Re: [RFC/PATCH 0/7] Rework git core for native submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-19T17:08:56Z","receivedAt":"2013-04-19T17:08:56Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 18.04.2013 01:17, schrieb Philip Oakley:\n> Would it be possible to summarise the key points and proposals of where the subject is now?\n\nHere you go, time to post our third iteration of the comparison\nlist, containing two updates:\n\n- \"easier coding\" was removed from the advantages\n\n- \"git submodule foreach\" was retired from the disadvantages\n\nAs in the two first versions, the issues in parentheses had been\nbrought up but were dismissed and are only kept for reference\ntogether with the reason why they aren't relevant anymore. Only\nthose preceded by a '*' are still considered valid.\n\n\nAdvantages:\n\n* Information is stored in one place, no need to lookup stuff in\n another file/blob.\n\n* No need to cd-to-toplevel to change configuration in the\n .gitmodules file, the special tools to edit link information\n will work in any subdirectory.\n\n(It is all but clear that this approach will lead to \"easier\ncoding\", some parts of the code - like rm and mv - will profit\nfrom that while others won't, e.g. we have to implement the link\nobject manipulation tools that are not needed for .gitmodules\nand we get another indirection retrieving the submodule commit\nfrom the link object. And then there is the fact that the new\ncode would have to catch up with functionality already coded\nusing .gitmodules, like the status/diff ignore and the fetch\nflags).\n\n(We currently need a checked out work tree to access the\n.gitmodules file, but there is ongoing work to read the\nconfiguration directly from the database)\n\n(While it is easier to merge the link object, a .gitmodules\naware merge driver would work just as well)\n\n\nDisadvantages:\n\n* Changes in user visible behavior, compatibility problems when\n Git versions are mixed.\n\n* Special tools are needed to edit submodule information where\n currently a plain editor is sufficient and a standard format\n is used.\n\n* merge conflicts are harder to resolve and require special git\n commands, solving them in .gitmodules is way more intuitive\n as users are already used to conflict markers.\n\n* With .gitmodules we lose a central spot where configuration\n concerning many submodules can be stored\n\n(\"git submodule foreach\" becomes harder to implement\" is not the\ncase, as that command currently also walks all tree objects and\ndoes not read the list of submodules from the .gitmodules file)\n\n(When we also put the submodule name in the link object we could\nalso retain the ability to repopulated moved submodules from\ntheir old repo, which is found by that name)\n\n(That a link object can have no unstaged counterpart that a file\neasily has can be fixed by special casing this, e.g. in using a\nfile in .git/link-specs/)\n\n\nAs no new arguments have been brought up, it all boils down to a\nchange that'll hurt users badly and won't fix any issue relevant\nto them. It'll bring them a flag day after which the .gitmodules\nis gone and they'll have to learn new tools to update and merge\nthe submodule metadata (and not only the users, GUIs have to\nfollow and implement support for something which currently is a\nperfectly normal merge conflict in a file). You'd have to smoke\nreally weird stuff to even consider such a change under these\ncircumstances (or you don't care one bit about your users).\n\n> The submodules does need 'fixing', as does agreeing the problem and abuse cases.\n\nSure, but almost all problems I know about are work tree related,\nso changing the internal representation buys us nothing here. It\nwill not magically do a bisect over submodules or will recursively\nupdate submodule work trees, and all that stuff won't be easier to\ncode either just because we have to get the information from a new\nobject instead of a gitlink/.gitmodules combo.\n\nLet's just close this case and get back to working on things that\nusers will actually profit from.\n"}]}