{"thread":{"id":"15217","subject":"Relative submodule URLs vs. clone URL DWIMming","startedAt":"2008-08-27T12:00:54Z","lastAt":"2008-09-02T07:23:28Z","messageCount":8,"participants":["Johan Herland","Mark Levedahl","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"88718","messageId":"200808271400.54302.johan@herland.net","threadId":"15217","inReplyTo":null,"subject":"Relative submodule URLs vs. clone URL DWIMming","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-08-27T12:00:54Z","receivedAt":"2008-08-27T12:00:54Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Hi,\n\nI'm observing a couple of bugs/issues when using relative submodule \nURLs:\n\nSay, I have a repo \"foo\" with a couple of submodules \"bar\", \"baz\", and \nthe following .gitmodules:\n\n\t[submodule \"bar\"]\n\t    path = bar\n\t    url = ./bar\n\t[submodule \"baz\"]\n\t    path = baz\n\t    url = ./baz\n\nNow, what I want to do is to clone this repo and initialize the \nsubmodules using the following commands:\n\n\tgit clone <URL> <DIR>\n\tcd <DIR>\n\tgit submodule update --init\n\nI will demonstrate that whether or not the \"git submodule update --init\" \nsucceeds depends on the value of <URL>:\n\n\nScenarios:\n\n#1: <URL> is a non-bare local repo: \"/home/user/foo\"\n\nbuiltin-clone.c:get_repo_path() will DWIM this to \"/home/user/foo/.git\", \nand store this as the origin URL. Relative submodule URLs will be \nresolved against this, and end up looking \nlike \"/home/user/foo/.git/bar\".\n\nResult: FAIL\n\n\n#2: <URL> is a non-bare remote repo: \"file:///home/user/foo\"\n\n(For the purposes of this discussion, a file:// URL is equivalent to a \nreal remote URL) builtin-clone.c:get_repo_path() fails to DWIM it, and \nwe fall back to using it verbatim as the origin URL. Relative submodule \nURLs will be resolved against this, and end up \nlike \"file:///home/user/foo/bar\".\n\nResult: Success\n\n\nNote: When using bare repos below, I assume that the repos are laid out \nin the following manner:\n\n\t/home/user/foo.git\n\t/home/user/foo.git/bar.git\n\t/home/user/foo.git/baz.git\n\n\n#3: <URL> is a bare local repo (indirectly): \"/home/user/foo\"\n\nbuiltin-clone.c:get_repo_path() will DWIM it to \"/home/user/foo.git\", \nand store this as the origin URL. Relative submodule URLs will be \nresolved against this, and end up looking \nlike \"/home/user/foo.git/bar\", which will work (because the same \nDWIMming is applied when cloning submodules).\n\nResult: Success\n\n\n#4: <URL> is a bare local repo (directly): \"/home/user/foo.git\"\n\nThis ends up looking exactly like scenario #3.\n\nResult: Success\n\n\n#5: <URL> is a bare remote repo (indirectly): \"file:///home/user/foo\"\n\nAs in scenario #2, builtin-clone.c:get_repo_path() fails to DWIM it, and \nthe origin URL ends up as a verbatim copy, but the initial clone still \nworks because of similar DWIMing at the remote/transport level. \nRelative submodule URLs will be resolved against the origin URL, \nhowever, and end up looking like \"file:///home/user/foo/bar\", which \nwill not work, since - on the submodule clone - remote/transport level \nwill only try to DWIM the last path component \n(i.e. \"file:///home/user/foo/bar.git\", and \nnot \"file:///home/user/foo.git/bar.git\").\n\nResult: FAIL\n\n\n#6: <URL> is a bare remote repo (directly): \"file:///home/user/foo.git\"\n\nThe initial clone goes off without a hitch, and the submodule URLs end \nup looking like \"file:///home/user/foo.git/bar\", which is subsequently \nDWIMmed \"correctly\" to \"file:///home/user/foo.git/bar\".\n\nResult: Success\n\n\nI'd like to fix this, but I'm not sure whether the fix belongs in \nbuiltin-clone.c (i.e. making sure the origin URL is always \"correct\" \nwrt. resolving relative submodule URLs), or in git-submodule.sh (i.e. \nadding smarts when resolving relative submodule URLs against the \nsuper-repo's origin URL).\n\n\n...Johan\n"},{"id":"88896","messageId":"48B6BB49.3000703@gmail.com","threadId":"15217","inReplyTo":"200808271400.54302.johan@herland.net","subject":"Re: Relative submodule URLs vs. clone URL DWIMming","fromName":"Mark Levedahl","fromEmail":"mlevedahl@gmail.com","sentAt":"2008-08-28T14:50:49Z","receivedAt":"2008-08-28T14:50:49Z","isPatch":false,"sender":{"key":"mdl123@verizon.net","avatar":"https://avatars.githubusercontent.com/u/5302462?v=4"},"body":"Johan Herland wrote:\n>\n> I'd like to fix this, but I'm not sure whether the fix belongs in \n> builtin-clone.c (i.e. making sure the origin URL is always \"correct\" \n> wrt. resolving relative submodule URLs), or in git-submodule.sh (i.e. \n> adding smarts when resolving relative submodule URLs against the \n> super-repo's origin URL).\n>\n>\n> ...Johan\n>\n>   \nI think the right approach is to start with clone and make it record the \nreal url it is using, regardless of what was input. The problem with \ndoing this in submodule is that in effect this replicates the search \nlogic clone would use, and furthermore could lead to nasty surprises by \ngrabbing the wrong submodule in an extreme case of having two \nidentically named repositories in different locations on a server.\n\nI was about to create a patch for submodule to always remove trailing \n\"/.git\" before resolving, but in fact a user could put the submodule \n.git into the superproject's .git, in a non-bare repository, and then \ngitlink that in the checked out submodule. So, it may also be good to \ndefine and enforce rules on how relative url naming can be used for this \npurpose. So far, I have only used it for bare repositories using the \n\"../<path> form keeping the submodules out of the superproject.\n\n\nMark\n"},{"id":"89029","messageId":"200808290101.20048.johan@herland.net","threadId":"15217","inReplyTo":"48B6BB49.3000703@gmail.com","subject":"Re: Relative submodule URLs vs. clone URL DWIMming","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-08-28T23:01:19Z","receivedAt":"2008-08-28T23:01:19Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Thursday 28 August 2008, Mark Levedahl wrote:\n> Johan Herland wrote:\n> > I'd like to fix this, but I'm not sure whether the fix belongs in\n> > builtin-clone.c (i.e. making sure the origin URL is always \"correct\"\n> > wrt. resolving relative submodule URLs), or in git-submodule.sh (i.e.\n> > adding smarts when resolving relative submodule URLs against the\n> > super-repo's origin URL).\n>\n> I think the right approach is to start with clone and make it record the\n> real url it is using, regardless of what was input. The problem with\n> doing this in submodule is that in effect this replicates the search\n> logic clone would use, and furthermore could lead to nasty surprises by\n> grabbing the wrong submodule in an extreme case of having two\n> identically named repositories in different locations on a server.\n\nThanks for your input. I looked into builtin-clone, and how to make it write\nthe real origin URL for remote repos (it already does this today for local\nrepos). But AFAICS, there's no way to get this information from the\ntransport layer. I assume that the actual repo location is resolved on the\nremote side, and simply not communicated back to the local side.\n\nAlso, after thinking some more about this, I'm unsure of the practical\nimplications of this. Let's assume we can actually make \"git clone\" write\nthe real URL in all cases (i.e. typically \"repo.git\" for bare repos\nand \"repo/.git\" for non-bare repos):\n\nIf we simply resolve submodule URLs against the _repo_ (i.e. the real origin\nURL) and not the work tree (if any), we get results that are coupled to\nwhether we use bare or non-bare repos: Take, for example, your use\nof \"../<path>\" to make submodules live outside the (bare) superproject. If I\nnow create a non-bare clone of this, I must move the submodule repos _into_\nmy work tree, so that the submodule repos are available, if someone tries to\nclone from me.\n\nAdditionally, when making super/sub repos available for others, I must make\na mental effort to interpret the relative submodule URLs in .gitmodules\nagainst - NOT the work tree in which .gitmodules is located - but rather\nagainst the .git directory which is (most often) one directory level further\ndown from the .gitmodules file.\n\nMaybe the easiest solution is to add a new config directive called something\nlike \"core.submodules.baseUrl\". When set, relative submodule URLs would be\nresolved against this instead of the origin URL. When unset, the origin URL\nis used as a fallback. Hmm?\n\nStill, it would be nice to have intuitive and consistent behaviour by\ndefault, though...\n\n> I was about to create a patch for submodule to always remove trailing\n> \"/.git\" before resolving, but in fact a user could put the submodule\n> .git into the superproject's .git, in a non-bare repository, and then\n> gitlink that in the checked out submodule. So, it may also be good to\n> define and enforce rules on how relative url naming can be used for this\n> purpose. So far, I have only used it for bare repositories using the\n> \"../<path> form keeping the submodules out of the superproject.\n\nYes, duplication of such logic is probably a bad idea. Would it be better to\nconsolidate all this URL DWIMming in one place which is #included from\nbuiltin-clone, git-submodule, git-fetch, and other commands that need this\nfunctionality?\n\nIn any case, defining the rules for how relative submodule URLs are supposed\nto work would certainly make a good step in the right direction.\n\n\nHave fun! :)\n\n...Johan\n\n\nPS: Here are the beginning of a test case for codifying the rules of\nrelative submodule URL behaviour. Needless to say, the last testcase\ncurrently fails:\n\ndiff --git a/t/t7403-submodule-relative.sh b/t/t7403-submodule-relative.sh\nnew file mode 100755\nindex 0000000..9c12248\n--- /dev/null\n+++ b/t/t7403-submodule-relative.sh\n@@ -0,0 +1,60 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2008 Johan Herland\n+#\n+\n+test_description='Testing repos with relative submodule URLs\n+\n+This test tries to verify the sanity of working with relative submodule URLs.\n+'\n+\n+. ./test-lib.sh\n+\n+TRASH_DIR=$(pwd)\n+\n+#\n+# Test setup:\n+#  - create the following super-/sub-repository hierarchy:\n+#     /\n+#     /file\n+#     /a submodule/\n+#     /a submodule/file\n+#\n+test_expect_success 'setup' '\n+\n+\techo file > file &&\n+\tgit add file &&\n+\ttest_tick &&\n+\tgit commit -m initial &&\n+\ttest_tick &&\n+\tgit clone . \"a submodule\" &&\n+\tcat >.gitmodules <<EOF &&\n+[submodule \"a submodule\"]\n+        path = \"a submodule\"\n+        url = \"./a submodule\"\n+EOF\n+\tgit add \"a submodule\" .gitmodules &&\n+\tgit commit -m \"a submodule\"\n+\n+'\n+\n+test_expect_success 'Cloning repo and updating submodules using file:// URL' '\n+\n+\tgit clone \"file://$TRASH_DIR\" clone &&\n+\tcd clone &&\n+\tgit submodule update --init &&\n+\ttest -d \"a submodule/.git\"\n+\n+'\n+\n+test_expect_success 'Cloning repo and updating submodules using local path' '\n+\n+\trm -rf clone &&\n+\tgit clone \"$TRASH_DIR\" clone &&\n+\tcd clone &&\n+\tgit submodule update --init &&\n+\ttest -d \"a submodule/.git\"\n+\n+'\n+\n+test_done\n\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"89273","messageId":"7vhc92f8ay.fsf@gitster.siamese.dyndns.org","threadId":"15217","inReplyTo":"200808290101.20048.johan@herland.net","subject":"Re: Relative submodule URLs vs. clone URL DWIMming","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-08-30T22:27:01Z","receivedAt":"2008-08-30T22:27:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland <johan@herland.net> writes:\n\n> ... But AFAICS, there's no way to get this information from the\n> transport layer. I assume that the actual repo location is resolved on the\n> remote side, and simply not communicated back to the local side.\n\nCorrect.\n\n> If we simply resolve submodule URLs against the _repo_ (i.e. the real origin\n> URL) and not the work tree (if any), we get results that are coupled to\n> whether we use bare or non-bare repos: Take, for example, your use\n> of \"../<path>\" to make submodules live outside the (bare) superproject. If I\n> now create a non-bare clone of this, I must move the submodule repos _into_\n> my work tree, so that the submodule repos are available, if someone tries to\n> clone from me.\n\nI personally feel that cases that involve cloning from non-bare\nrepositories (and in addition, DWIMmed repositories), with or without\nnested submodules, are not worth supporting.\n"},{"id":"89276","messageId":"200808310123.48018.johan@herland.net","threadId":"15217","inReplyTo":"7vhc92f8ay.fsf@gitster.siamese.dyndns.org","subject":"Re: Relative submodule URLs vs. clone URL DWIMming","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-08-30T23:23:47Z","receivedAt":"2008-08-30T23:23:47Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Sunday 31 August 2008, Junio C Hamano wrote:\n> Johan Herland <johan@herland.net> writes:\n> I personally feel that cases that involve cloning from non-bare\n> repositories (and in addition, DWIMmed repositories), with or without\n> nested submodules, are not worth supporting.\n\nOk. I guess that's fair. Such a decision should definitely be documented \nappropriately in the submodule documentation, though.\n\nI guess this also means you don't plan to do anything about the difference \nin origin URLs produced by \"git clone\" for the following two cases:\n\n$ git clone /repo/foo bar\n$ grep -B1 url bar/.git/config\n[remote \"origin\"]\n        url = /repo/foo/.git\n\nvs.\n\n$ git clone file:///repo/foo bar\n$ grep -B1 url bar/.git/config\n[remote \"origin\"]\n        file:///home/johan/git/foo\n\n\nHmm?\n\n...Johan\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"89277","messageId":"7vabeuf4oj.fsf@gitster.siamese.dyndns.org","threadId":"15217","inReplyTo":"200808310123.48018.johan@herland.net","subject":"Re: Relative submodule URLs vs. clone URL DWIMming","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-08-30T23:45:16Z","receivedAt":"2008-08-30T23:45:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland <johan@herland.net> writes:\n\n> $ git clone /repo/foo bar\n> $ grep -B1 url bar/.git/config\n> [remote \"origin\"]\n>         url = /repo/foo/.git\n>\n> vs.\n>\n> $ git clone file:///repo/foo bar\n> $ grep -B1 url bar/.git/config\n> [remote \"origin\"]\n>         file:///home/johan/git/foo\n>\n> Hmm?\n\nIf you mean \"the latter lacks 'url =' and is broken\", and if that is what\nhappens, then it needs to be fixed.  But otherwise, I do not personally\nfind these differences interesting.\n\nBut that does not mean I'd veto if somebody else cares deeply enough and\ncomes up with a clean solution.\n"},{"id":"89432","messageId":"200809012107.34020.johan@herland.net","threadId":"15217","inReplyTo":"7vabeuf4oj.fsf@gitster.siamese.dyndns.org","subject":"[PATCH] Bring local clone's origin URL in line with that of a remote clone","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-09-01T19:07:33Z","receivedAt":"2008-09-01T19:07:33Z","isPatch":true,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On a local clone, \"git clone\" would use the fully DWIMmed path as the origin\nURL in the resulting repo. This was slightly inconsistent with the case of a\nremote clone where the _given_ URL was used as the origin URL (because the\nDWIMming was done remotely, and was therefore not available to \"git clone\").\n\nThis behaviour caused problems when cloning a local non-bare repo with\nrelative submodule URLs, because these submodule URLs would then be resolved\nagainst the DWIMmed URL (e.g. \"/repo/.git\") instead of the given URL (e.g.\n\"/repo\").\n\nThis patch teaches \"git clone\" to use the _given_ URL - instead of the\nDWIMmed path - as the origin URL. This causes relative submodule URLs to be\nresolved correctly, as long the _given_ URL indicates the correct directory\nagainst which the submodule URLs should be resolved.\n\nThe patch also updates a testcase that contained the old-style origin URLs.\n\nSigned-off-by: Johan Herland <johan@herland.net>\n---\n\nOn Sunday 31 August 2008, Junio C Hamano wrote:\n> Johan Herland <johan@herland.net> writes:\n> > $ git clone /repo/foo bar\n> > $ grep -B1 url bar/.git/config\n> > [remote \"origin\"]\n> >         url = /repo/foo/.git\n> >\n> > vs.\n> >\n> > $ git clone file:///repo/foo bar\n> > $ grep -B1 url bar/.git/config\n> > [remote \"origin\"]\n> >         file:///home/johan/git/foo\n> >\n> > Hmm?\n>\n> If you mean \"the latter lacks 'url =' and is broken\", and if that is what\n> happens, then it needs to be fixed.  But otherwise, I do not personally\n> find these differences interesting.\n\nSorry, my fault (cut-n-paste error)\n\n> But that does not mean I'd veto if somebody else cares deeply enough and\n> comes up with a clean solution.\n\nIs this clean enough? It passes selftests on my box, and although it\ndoesn't resolve all the issues I raised in this thread, it _does_ resolve\nthe one issue that I first bumped into, and that triggered this thread in\nthe first place. I expect this one issue is also the first that most other\npeople would bump into when working locally with relative submodule URLs.\n\n\nHave fun! :)\n\n...Johan\n\n\n builtin-clone.c   |    2 +-\n t/t5505-remote.sh |    6 +++---\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin-clone.c b/builtin-clone.c\nindex c0e3086..f44ecea 100644\n--- a/builtin-clone.c\n+++ b/builtin-clone.c\n@@ -387,7 +387,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \tpath = get_repo_path(repo_name, &is_bundle);\n \tif (path)\n-\t\trepo = path;\n+\t\trepo = xstrdup(make_nonrelative_path(repo_name));\n \telse if (!strchr(repo_name, ':'))\n \t\trepo = xstrdup(make_absolute_path(repo_name));\n \telse\ndiff --git a/t/t5505-remote.sh b/t/t5505-remote.sh\nindex be9ee93..c449663 100755\n--- a/t/t5505-remote.sh\n+++ b/t/t5505-remote.sh\n@@ -109,7 +109,7 @@ test_expect_success 'remove remote' '\n \n cat > test/expect << EOF\n * remote origin\n-  URL: $(pwd)/one/.git\n+  URL: $(pwd)/one\n   Remote branch merged with 'git pull' while on branch master\n     master\n   New remote branch (next fetch will store in remotes/origin)\n@@ -140,7 +140,7 @@ test_expect_success 'show' '\n \n cat > test/expect << EOF\n * remote origin\n-  URL: $(pwd)/one/.git\n+  URL: $(pwd)/one\n   Remote branch merged with 'git pull' while on branch master\n     master\n   Tracked remote branches\n@@ -169,7 +169,7 @@ test_expect_success 'prune' '\n \n cat > test/expect << EOF\n Pruning origin\n-URL: $(pwd)/one/.git\n+URL: $(pwd)/one\n  * [would prune] origin/side2\n EOF\n \n-- \n1.6.0.96.g2fad1\n"},{"id":"89488","messageId":"7vod37m2of.fsf@gitster.siamese.dyndns.org","threadId":"15217","inReplyTo":"200809012107.34020.johan@herland.net","subject":"Re: [PATCH] Bring local clone's origin URL in line with that of a remote clone","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-09-02T07:23:28Z","receivedAt":"2008-09-02T07:23:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland <johan@herland.net> writes:\n\n> ... I expect this one issue is also the first that most other\n> people would bump into when working locally with relative submodule URLs.\n\nBy the way, I find your suggestion to set the canonical URL for the\ntoplevel repository that relative ones should base on quite reasonable.\n\nThe patch looked good.  Thanks.\n"}]}