{"thread":{"id":"42089","subject":"problems serving non-bare repos with submodules over http","startedAt":"2016-04-20T15:22:09Z","lastAt":"2016-04-21T22:42:50Z","messageCount":12,"participants":["Yaroslav Halchenko","Stefan Beller","Junio C Hamano","Jacob Keller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"283955","messageId":"20160420152209.GH23764@onerussian.com","threadId":"42089","inReplyTo":null,"subject":"problems serving non-bare repos with submodules over http","fromName":"Yaroslav Halchenko","fromEmail":"yoh@onerussian.com","sentAt":"2016-04-20T15:22:09Z","receivedAt":"2016-04-20T15:22:09Z","isPatch":false,"sender":{"key":"yoh@onerussian.com","avatar":"https://gravatar.com/avatar/8901b82415ae451a83aea49409708912726e53620e3ac92320bf1f86548d97e9?d=mp&s=160"},"body":"Dear Git Folks,\n\nI do realize that the situation is quite uncommon, partially I guess due\nto git submodules mechanism flexibility and power on one hand and\nunder-use (imho) on the other, which leads to discovery of regressions\n[e.g. 1] and corner cases as mine.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/288064\n[2] http://www.onerussian.com/tmp/git-web-submodules.sh\n\nMy use case:  We are trying to serve a git repository with submodules\nspecified with relative paths over http from a simple web server.  With a demo\ncase and submodule specification [complete script to reproduce including the\nwebserver using python is at 2] such as\n\n(git)hopa:/tmp/gitxxmsxYFO[master]git\n$> tree\n.\n├── f1\n└── sub1\n    └── f2\n\n$> cat .gitmodules\n[submodule \"sub1\"]\n    path = sub1\n    url = ./sub1\n\n\n1. After cloning \n\n    git clone http://localhost:8080/.git\n\n   I cannot 'submodule update' the sub1 in the clone since its url after\n   'submodule init' would be  http://localhost:8080/.git/sub1 .  If I manually fix\n   it up -- it seems to proceed normally since in original repository I have\n   sub1/.git/ directory and not the \"gitlink\" for that submodule.\n\n2. If I serve the clone [2 demos that too] itself, there is no easy remedy at\n   all since sub1/.git is not a directory but a gitlink.\n\nN.B. I haven't approached nested submodules case yet in [2]\n\nI wondered\n\na. could 'git clone' (probably actually some relevant helper used by fetch\n   etc) acquire ability to sense for URL/.git if URL itself doesn't point to a\n   usable git repository?\n\n    I think this could provide complete remedy for 1 since then relative urls\n    would be properly assembled, with similar 'sensing' for /.git for the final urls\n\n    I guess we could do it with rewrites/forwards on the \"server side\",\n    but it wouldn't be generally acceptable solution.\n\nb. is there a better or already existing way to remedy my situation?\n\nc. shouldn't \"git clone\" (or the relevant helper) be aware of remote\n   /.git possibly being a gitlink file within submodule?\n\n\nThank you in advance for your thoughts and feedback on this case.\n\nP.S. Please maintain the CC list in replies.\n-- \nYaroslav O. Halchenko\nCenter for Open Neuroscience     http://centerforopenneuroscience.org\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419\nWWW:   http://www.linkedin.com/in/yarik        \n"},{"id":"283960","messageId":"CAGZ79kYS-F1yKpNP7jmhTiZT1R_pucUBBTCbmHKZz6Xd6dy8EA@mail.gmail.com","threadId":"42089","inReplyTo":"20160420152209.GH23764@onerussian.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-04-20T16:14:08Z","receivedAt":"2016-04-20T16:14:08Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Apr 20, 2016 at 8:22 AM, Yaroslav Halchenko <yoh@onerussian.com> wrote:\n> Dear Git Folks,\n>\n> I do realize that the situation is quite uncommon, partially I guess due\n> to git submodules mechanism flexibility and power on one hand and\n> under-use (imho) on the other, which leads to discovery of regressions\n> [e.g. 1] and corner cases as mine.\n\nThanks for fixing the under-use and reporting bugs. :)\n\n>\n> [1] http://thread.gmane.org/gmane.comp.version-control.git/288064\n> [2] http://www.onerussian.com/tmp/git-web-submodules.sh\n>\n> My use case:  We are trying to serve a git repository with submodules\n> specified with relative paths over http from a simple web server.  With a demo\n> case and submodule specification [complete script to reproduce including the\n> webserver using python is at 2] such as\n>\n> (git)hopa:/tmp/gitxxmsxYFO[master]git\n> $> tree\n> .\n> ├── f1\n> └── sub1\n>     └── f2\n>\n> $> cat .gitmodules\n> [submodule \"sub1\"]\n>     path = sub1\n>     url = ./sub1\n>\n>\n> 1. After cloning\n>\n>     git clone http://localhost:8080/.git\n>\n>    I cannot 'submodule update' the sub1 in the clone since its url after\n>    'submodule init' would be  http://localhost:8080/.git/sub1 .  If I manually fix\n>    it up -- it seems to proceed normally since in original repository I have\n>    sub1/.git/ directory and not the \"gitlink\" for that submodule.\n\nSo the expected URL would be  http://localhost:8080/sub1/.git ?\n\nI thought you could leave out the .git prefix, i.e. you can type\n\n     git clone http://localhost:8080\n\nand Git will recognize the missing .git and try that as well. The relative URL\nwould then be constructed as http://localhost:8080/sub1, which will use the\nsame mechanism to find the missing .git ending.\n\n>\n> 2. If I serve the clone [2 demos that too] itself, there is no easy remedy at\n>    all since sub1/.git is not a directory but a gitlink.\n\nNot sure I understand the second question.\n\n>\n> N.B. I haven't approached nested submodules case yet in [2]\n>\n> I wondered\n>\n> a. could 'git clone' (probably actually some relevant helper used by fetch\n>    etc) acquire ability to sense for URL/.git if URL itself doesn't point to a\n>    usable git repository?\n\nSo you mean in case of relative submodules, we need to take the parent\nurl, and remove the \".git\" at the end and try again if we cannot find\nthe submodule?\n\n>\n>     I think this could provide complete remedy for 1 since then relative urls\n>     would be properly assembled, with similar 'sensing' for /.git for the final urls\n>\n>     I guess we could do it with rewrites/forwards on the \"server side\",\n>     but it wouldn't be generally acceptable solution.\n>\n> b. is there a better or already existing way to remedy my situation?\n>\n> c. shouldn't \"git clone\" (or the relevant helper) be aware of remote\n>    /.git possibly being a gitlink file within submodule?\n\nOh. I think that non-bare repositories including submodules are not designed\nto be cloned, because they are for use in the file system. Even a\nlocal clone fails:\n\n    # gerrit is a project I know which also has submodules:\n    git clone --recurse-submodules https://gerrit.googlesource.com/gerrit g1\n    git clone --recurse-submodules g1 g2\n    ...\nfatal: clone of '...' into submodule path '...' failed\n\nSo I think for cloning repositories you want to have each repository\nas its own thing (bare or non bare).\n\nThe submodule mechanism is just a way to express a relation between\nthe reositories, it's like composing them together, but by that composition\nit breaks the properties of each repository to be easily clonable.\n\nI think we should fix that.\n\nI guess the local clone case is 'easy' as you only need\nto handle the link instead of directory thing correctly.\n\nFor the case you describe (cloning from a remote, whether it is http or ssh),\nwe would need to discuss security implications I would assume? It sounds\nscary at first to follow a random git link to the outer space of the repository.\n(A similar thing is that you cannot have symlinks in a git repository pointing\noutside of it, IIRC? At least that was fishy.)\n\nThanks,\nStefan\n\n>\n>\n> Thank you in advance for your thoughts and feedback on this case.\n>\n> P.S. Please maintain the CC list in replies.\n> --\n> Yaroslav O. Halchenko\n> Center for Open Neuroscience     http://centerforopenneuroscience.org\n> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\n> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419\n> WWW:   http://www.linkedin.com/in/yarik\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"283987","messageId":"20160420194533.GO23764@onerussian.com","threadId":"42089","inReplyTo":"CAGZ79kYS-F1yKpNP7jmhTiZT1R_pucUBBTCbmHKZz6Xd6dy8EA@mail.gmail.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Yaroslav Halchenko","fromEmail":"yoh@onerussian.com","sentAt":"2016-04-20T19:45:33Z","receivedAt":"2016-04-20T19:45:33Z","isPatch":false,"sender":{"key":"yoh@onerussian.com","avatar":"https://gravatar.com/avatar/8901b82415ae451a83aea49409708912726e53620e3ac92320bf1f86548d97e9?d=mp&s=160"},"body":"\nOn Wed, 20 Apr 2016, Stefan Beller wrote:\n> > I do realize that the situation is quite uncommon, partially I guess due\n> > to git submodules mechanism flexibility and power on one hand and\n> > under-use (imho) on the other, which leads to discovery of regressions\n> > [e.g. 1] and corner cases as mine.\n\n> Thanks for fixing the under-use and reporting bugs. :)\n\nI am thrilled to help ;)\n\n> > [1] http://thread.gmane.org/gmane.comp.version-control.git/288064\n> > [2] http://www.onerussian.com/tmp/git-web-submodules.sh\n\n> > My use case:  We are trying to serve a git repository with submodules\n> > specified with relative paths over http from a simple web server.  With a demo\n> > case and submodule specification [complete script to reproduce including the\n> > webserver using python is at 2] such as\n\n> > (git)hopa:/tmp/gitxxmsxYFO[master]git\n> > $> tree\n> > .\n> > ├── f1\n> > └── sub1\n> >     └── f2\n\n> > $> cat .gitmodules\n> > [submodule \"sub1\"]\n> >     path = sub1\n> >     url = ./sub1\n\n\n> > 1. After cloning\n\n> >     git clone http://localhost:8080/.git\n\n> >    I cannot 'submodule update' the sub1 in the clone since its url after\n> >    'submodule init' would be  http://localhost:8080/.git/sub1 .  If I manually fix\n> >    it up -- it seems to proceed normally since in original repository I have\n> >    sub1/.git/ directory and not the \"gitlink\" for that submodule.\n\n> So the expected URL would be  http://localhost:8080/sub1/.git ?\n\nATM, yes\n\n> I thought you could leave out the .git prefix, i.e. you can type\n\n>      git clone http://localhost:8080\n\n> and Git will recognize the missing .git and try that as well. The relative URL\n> would then be constructed as http://localhost:8080/sub1, which will use the\n> same mechanism to find the missing .git ending.\n\n[note1] Unfortunately it is not the case ATM (git version\n2.8.1.369.geae769a, output is interspersed with log from the python's simple\nhttp server):\n\n$> git clone http://localhost:8080 xxx                   \nCloning into 'xxx'...             \n127.0.0.1 - - [20/Apr/2016 15:01:25] code 404, message File not found\n127.0.0.1 - - [20/Apr/2016 15:01:25] \"GET /info/refs?service=git-upload-pack HTTP/1.1\" 404 -\nfatal: repository 'http://localhost:8080/' not found\n\n\n> > 2. If I serve the clone [2 demos that too] itself, there is no easy remedy at\n> >    all since sub1/.git is not a directory but a gitlink.\n\n> Not sure I understand the second question.\n\nIf I serve via http a repository where sub1/.git is a \"gitlink\":\n\n    (git)hopa:/tmp/gitxxmsxYFO_[master]\n    $> cat sub1/.git \n    gitdir: ../.git/modules/sub1\n\nSuch repository cannot be cloned:\n\n    (git)hopa:/tmp/gitxxmsxYFO_[master]git\n    $> git clone http://localhost:8080/sub1 /tmp/xxx\n    Cloning into '/tmp/xxx'...                      \n    127.0.0.1 - - [20/Apr/2016 15:04:01] code 404, message File not found\n    127.0.0.1 - - [20/Apr/2016 15:04:01] \"GET /sub1/info/refs?service=git-upload-pack HTTP/1.1\" 404 -\n    fatal: repository 'http://localhost:8080/sub1/' not found\n\n    $> git clone http://localhost:8080/sub1/.git /tmp/xxx \n    Cloning into '/tmp/xxx'...\n    127.0.0.1 - - [20/Apr/2016 15:04:06] code 404, message File not found\n    127.0.0.1 - - [20/Apr/2016 15:04:06] \"GET /sub1/.git/info/refs?service=git-upload-pack HTTP/1.1\" 404 -\n    fatal: repository 'http://localhost:8080/sub1/.git/' not found\n\n\n> > N.B. I haven't approached nested submodules case yet in [2]\n\n> > I wondered\n\n> > a. could 'git clone' (probably actually some relevant helper used by fetch\n> >    etc) acquire ability to sense for URL/.git if URL itself doesn't point to a\n> >    usable git repository?\n\n> So you mean in case of relative submodules, we need to take the parent\n> url, and remove the \".git\" at the end and try again if we cannot find\n> the submodule?\n\nthat would be the a.2 which I have forgotten to outline ;)\n\nin a.  I was suggesting what you have assumed [note 1 above] would be\nhappening (but doesn't) ATM: that /.git would be automagically sensed.\n\n> >     I think this could provide complete remedy for 1 since then relative urls\n> >     would be properly assembled, with similar 'sensing' for /.git for the final urls\n\n> >     I guess we could do it with rewrites/forwards on the \"server side\",\n> >     but it wouldn't be generally acceptable solution.\n\n> > b. is there a better or already existing way to remedy my situation?\n\n> > c. shouldn't \"git clone\" (or the relevant helper) be aware of remote\n> >    /.git possibly being a gitlink file within submodule?\n\n> Oh. I think that non-bare repositories including submodules are not designed\n> to be cloned, because they are for use in the file system.\n\nWell -- that is the beauty of git being a distributed VCS, that non-bare repos\nseems to be as nicely cloneable as bare ones. And in general it seems to work\nwith submodules as well, since they should be the \"consistent\"\nphilosophically... \n\n>  Even a local clone fails:\n\n>     # gerrit is a project I know which also has submodules:\n>     git clone --recurse-submodules https://gerrit.googlesource.com/gerrit g1\n>     git clone --recurse-submodules g1 g2\n>     ...\n> fatal: clone of '...' into submodule path '...' failed\n\nI guess that is just yet another bug with relative paths in the\nsubmodules.\n\n> So I think for cloning repositories you want to have each repository\n> as its own thing (bare or non bare).\n\nin your first line in the example above you somewhat have shown the\ncounter-argument to the statement.  Indeed each repository should be its own\nthing, just possibly registered as a submodule to another one.\n\n> The submodule mechanism is just a way to express a relation between\n> the reositories, it's like composing them together, but by that composition\n> it breaks the properties of each repository to be easily clonable.\n\nIt doesn't really (unless in the cases we both pointed out).  E.g. I can as\neasily clone original sub1 repository which was  registered as a submodule of\nanother one.  Either treatment of them by git during cloning (and placing under\nroot repo's .git/modules, etc) undermines that feature -- that is the\nquestion we could also discuss here somewhat I guess ;)\n\n> I think we should fix that.\n\nwould be awesome! Thanks in advance ;)\n\n> I guess the local clone case is 'easy' as you only need\n> to handle the link instead of directory thing correctly.\n\n> For the case you describe (cloning from a remote, whether it is http or ssh),\n> we would need to discuss security implications I would assume? It sounds\n> scary at first to follow a random git link to the outer space of the repository.\n\nmore like \"into the inner space\".  git already (as  above example shown)\ndescends right away into  \"/info/refs?\", so how sensing \"/.git/\" would be any\ndifferent?\n\n> (A similar thing is that you cannot have symlinks in a git repository pointing\n> outside of it, IIRC? At least that was fishy.)\n\nthat might indeed be dangerous.  but once again, per above argument similarly\nup to the \"provider\" I guess to guarantee protection, e.g. forbidding following\nsymlink on the webserver for that served directory, if content is not under his\ncontrol.\n\nCheers and thanks for your quick reply Stefan!\n-- \nYaroslav O. Halchenko\nCenter for Open Neuroscience     http://centerforopenneuroscience.org\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419\nWWW:   http://www.linkedin.com/in/yarik        \n"},{"id":"283990","messageId":"xmqqh9ewukhw.fsf@gitster.mtv.corp.google.com","threadId":"42089","inReplyTo":"CAGZ79kYS-F1yKpNP7jmhTiZT1R_pucUBBTCbmHKZz6Xd6dy8EA@mail.gmail.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-04-20T19:51:23Z","receivedAt":"2016-04-20T19:51:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n>> 1. After cloning\n>>\n>>     git clone http://localhost:8080/.git\n>>\n>>    I cannot 'submodule update' the sub1 in the clone since its url after\n>>    'submodule init' would be  http://localhost:8080/.git/sub1 .  If I manually fix\n>>    it up -- it seems to proceed normally since in original repository I have\n>>    sub1/.git/ directory and not the \"gitlink\" for that submodule.\n>\n> So the expected URL would be  http://localhost:8080/sub1/.git ?\n>\n> I thought you could leave out the .git prefix, i.e. you can type\n>\n>      git clone http://localhost:8080\n>\n> and Git will recognize the missing .git and try that as well. The relative URL\n> would then be constructed as http://localhost:8080/sub1, which will use the\n> same mechanism to find the missing .git ending.\n\nI may be missing the subtleties, but if you are serving others from\na non-bare repository with submodules, I do not think you would want\nto expose the in-tree version of the submodule in the first place.\n\nThese $submodule/.git files point via \"gitdir:\" to their real\nrepository location, don't they?  And I would think that they are\nwhat you would want to expose to the outside world.  Your in-tree\nsubmodules may come and go as you checkout different branches in\nyour working tree, but these copies at their real locations will\nstay.\n"},{"id":"284001","messageId":"CAGZ79kZMOv0r9fRFbP1WV8qFJBm+s=V8=ueFbYvnyFtgV8j9iQ@mail.gmail.com","threadId":"42089","inReplyTo":"xmqqh9ewukhw.fsf@gitster.mtv.corp.google.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-04-20T21:05:43Z","receivedAt":"2016-04-20T21:05:43Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Apr 20, 2016 at 12:51 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Stefan Beller <sbeller@google.com> writes:\n>\n>>> 1. After cloning\n>>>\n>>>     git clone http://localhost:8080/.git\n>>>\n>>>    I cannot 'submodule update' the sub1 in the clone since its url after\n>>>    'submodule init' would be  http://localhost:8080/.git/sub1 .  If I manually fix\n>>>    it up -- it seems to proceed normally since in original repository I have\n>>>    sub1/.git/ directory and not the \"gitlink\" for that submodule.\n>>\n>> So the expected URL would be  http://localhost:8080/sub1/.git ?\n>>\n>> I thought you could leave out the .git prefix, i.e. you can type\n>>\n>>      git clone http://localhost:8080\n>>\n>> and Git will recognize the missing .git and try that as well. The relative URL\n>> would then be constructed as http://localhost:8080/sub1, which will use the\n>> same mechanism to find the missing .git ending.\n>\n> I may be missing the subtleties, but if you are serving others from\n> a non-bare repository with submodules, I do not think you would want\n> to expose the in-tree version of the submodule in the first place.\n\nWell I would imagine that is the exact point.\nIf I was not trying to expose my state, I could ask you to\nobtain your copy from $(git remote get-url origin) just as I did.\n\nI would imagine, if I have a problem with some repo I can tell my\ncoworker or others to get my copy to took into that exact state.\n(Or I want to transfer state from workstation to laptop to\ncontinue working)\n\nWithout submodules this workflow works. So I'd expect it\nto work with submodules as well eventually. Also we probably don't\nwant to mix cloning the superproject from this non bare repo and\nthe generic submodule locations as the superproject may have\nadvanced submodule pointers to commits which are not present\nin the generic submodule remotes.\n\nSo for the non-bare case I would really expect to be able to \"copy\"\nthe remote including submodules from that remote?\n\nWe could reason about only providing this for the superproject though\nand not for submodules, i.e. cloning from the non bare submodule\ncould be not supported. (If you really want that non bare submodule,\nyou can still clone it manually from\n\n    $GIT_DIR_SUPER_PROJECT/modules/$MODULE_NAME\n\n\n\n>\n> These $submodule/.git files point via \"gitdir:\" to their real\n> repository location, don't they?\n\nYes they do.\n\n> And I would think that they are\n> what you would want to expose to the outside world.  Your in-tree\n> submodules may come and go as you checkout different branches in\n> your working tree, but these copies at their real locations will\n> stay.\n\nRight instead of cloning $WORKTREE/sub/.git you rather want\n$GITDIR/module/sub\n\n(GITDIR and WORKTREE from the superprojects point of view)\n\nThe problem with a copy of a superproject including submodules is\nthe way cloning submodules work.\n\n  1) clone the superproject\n  2) for each gitlink in the tree, consult the .gitmodules file\n  3) if we have a match in the .gitmodules file, clone from there\n\nSo currently the protocol doesn't allow to even specify the submodules\ndirectories. In case the remote superproject is non bare in 1) the remote\nwould need to advertise the submodule repository URLS separately,\nsuch that the cloning can be performed from those direct copies.\n\n\nThanks,\nStefan\n\n>\n>\n"},{"id":"284007","messageId":"xmqqshygt1gs.fsf@gitster.mtv.corp.google.com","threadId":"42089","inReplyTo":"CAGZ79kZMOv0r9fRFbP1WV8qFJBm+s=V8=ueFbYvnyFtgV8j9iQ@mail.gmail.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-04-20T21:27:47Z","receivedAt":"2016-04-20T21:27:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n>> I may be missing the subtleties, but if you are serving others from\n>> a non-bare repository with submodules, I do not think you would want\n>> to expose the in-tree version of the submodule in the first place.\n>\n> Well I would imagine that is the exact point.\n> If I was not trying to expose my state, I could ask you to\n> obtain your copy from $(git remote get-url origin) just as I did.\n\nThat wasn't what I had in mind, but if the cloner cloned from your\nrepository with a working tree, the cloner would discover submodules\nyou use from your .gitmodules file, which would record the location\nyou cloned them from, so something like that may come into the\npicture.  What I had in mind was more like this one you mentioned\nbelow:\n\n>     $GIT_DIR_SUPER_PROJECT/modules/$MODULE_NAME\n> ...\n> Right instead of cloning $WORKTREE/sub/.git you rather want\n> $GITDIR/module/sub\n\n> So currently the protocol doesn't allow to even specify the submodules\n> directories.\n\nDepends on what you exactly mean by \"the protocol\", but the\nnetworking protocol is about accessing a single repository.  It is\nup to you to decide where to go next after learning what you can\nlearn from the result, typically by following what appears in\nthe .gitmodules file.\n\nThe only special case is when .gitmodules file records the URL in a\nrelative form, I would think.  Traditionally (i.e. when it was\nconsidered sane to clone only from bare repositories) I think people\nexpected a layout like this:\n\n\ttop.git/\n\ttop.git/refs/{heads,tags,...}/...\n        top.git/objects/...\n        top.git/sub.git/\n\ttop.git/sub.git/refs/{heads,tags,...}/...\n        top.git/sub.git/objects/...\n\nand refer to ./sub.git from .gitmodules recorded in top.git.  It\nstill would be norm for common distribution sites (i.e. the original\nplace Yaroslav likely has cloned things from) to be bare, and with\nor without $GIT_DIR/modules/, the relative path of submodule seen\nby its superproject would (have to) be different between a bare and\na non-bare repository.\n\nI'd imagine that people could agree on a common layout like this\neven for a forest of bare repositories:\n\n\ttop.git/\n\ttop.git/refs/{heads,tags,...}/...\n        top.git/objects/...\n        top.git/modules/sub.git/\n\ttop.git/modules/sub.git/refs/{heads,tags,...}/...\n        top.git/modules/sub.git/objects/...\n\nwhich would probably make the \"relative\" relationship between the\nsupermodule and its submodules the same between bare and non-bare\nrepositories, but I didn't think it too deeply.\n"},{"id":"284012","messageId":"CAGZ79kbyDuu8hUQXOUNmDLyO88GOi3kqSzsc0UfgzwAL9peZHg@mail.gmail.com","threadId":"42089","inReplyTo":"xmqqshygt1gs.fsf@gitster.mtv.corp.google.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-04-20T23:05:33Z","receivedAt":"2016-04-20T23:05:33Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Apr 20, 2016 at 2:27 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Stefan Beller <sbeller@google.com> writes:\n>\n>>> I may be missing the subtleties, but if you are serving others from\n>>> a non-bare repository with submodules, I do not think you would want\n>>> to expose the in-tree version of the submodule in the first place.\n>>\n>> Well I would imagine that is the exact point.\n>> If I was not trying to expose my state, I could ask you to\n>> obtain your copy from $(git remote get-url origin) just as I did.\n>\n> That wasn't what I had in mind, but if the cloner cloned from your\n> repository with a working tree, the cloner would discover submodules\n> you use from your .gitmodules file, which would record the location\n> you cloned them from, so something like that may come into the\n> picture.  What I had in mind was more like this one you mentioned\n> below:\n>\n>>     $GIT_DIR_SUPER_PROJECT/modules/$MODULE_NAME\n>> ...\n>> Right instead of cloning $WORKTREE/sub/.git you rather want\n>> $GITDIR/module/sub\n>\n>> So currently the protocol doesn't allow to even specify the submodules\n>> directories.\n>\n> Depends on what you exactly mean by \"the protocol\", but the\n> networking protocol is about accessing a single repository.  It is\n> up to you to decide where to go next after learning what you can\n> learn from the result, typically by following what appears in\n> the .gitmodules file.\n\nRight. But the .gitmodules file is not sufficient.\n\nIf I clone from a bare hosting location, the .gitmodules file\nis the best we can do and the .gitmodules file works as intended.\nBut in the non bare I case I think we would want to get the submodule\nfrom that location as well.\n\nSo in git clone (which calls out to git submodule update, which uses\nsubmodule--helper update_clone for cloning submodules) we'd want to see\n\n    if remote was bare:\n        do as usual (obtain URL from .gitmodules file)\n    else\n        take URL=$NON_BARE_REMOTE/module/submodule\n\n\n\n>\n> The only special case is when .gitmodules file records the URL in a\n> relative form, I would think.  Traditionally (i.e. when it was\n> considered sane to clone only from bare repositories) I think people\n> expected a layout like this:\n>\n>         top.git/\n>         top.git/refs/{heads,tags,...}/...\n>         top.git/objects/...\n>         top.git/sub.git/\n>         top.git/sub.git/refs/{heads,tags,...}/...\n>         top.git/sub.git/objects/...\n\nwhich could also be referred to as\n\n      top\n\nwithout the .git suffix as someone thought this was an optimization?\n\nRelative paths for submodules I have seen so far (on github,\ngooglesource, eclipse)\nstart with a ../ such that we'd have\n\n>         top.git/\n>         top.git/refs/{heads,tags,...}/...\n>         top.git/objects/...\n>         sub.git/\n>         sub.git/refs/{heads,tags,...}/...\n>         sub.git/objects/...\n\nand the .git suffix omission works as we only need to check for the last\nfor characters and not somewhere in between. The sub.git is a standalone\nrepository, and you cannot tell it is a submodule (except by its contents)\n\n>\n> and refer to ./sub.git from .gitmodules recorded in top.git.  It\n> still would be norm for common distribution sites (i.e. the original\n> place Yaroslav likely has cloned things from) to be bare, and with\n> or without $GIT_DIR/modules/, the relative path of submodule seen\n> by its superproject would (have to) be different between a bare and\n> a non-bare repository.\n\nI think on a hosting site they could even coexist when having the\nlayout as above.\n\n         top.git/\n         top.git/refs/{heads,tags,...}/...\n         top.git/objects/...\n         sub.git/\n         sub.git/refs/{heads,tags,...}/...\n         sub.git/objects/...\n\n         # the following only exist in non bare:\n         top.git/modules/sub.git/\n         top.git/modules/sub.git/refs/{heads,tags,...}/...\n         top.git/modules/sub.git/objects/...\n\nThe later files would be more reflective of what you *really*\nwant if you clone from top.git.\n\nTraditionally (when cloning was done from bare repos only),\nthe .gitmodules file provides a very good way to indicate what\nthe intent of the superproject is as the recorded sha1 in the tree\ndoesn't tell you anything and tracking the remote for the submodule\nout of tree is cumbersome, so an in tree solution makes perfect sense.\n\nIf we have a non bare repo, it is safe to assume that the cloner actually\nmeant to get the whole state from the remote (including submodules)?\n\nI am trying to think of reasons why you would not want to get that copy\nfrom the remote. One (weak) reason is that the submodule may be a\nwell known library, which you can obtain faster from a well known git\nhosting site rather than $remote.\n\n>\n> I'd imagine that people could agree on a common layout like this\n> even for a forest of bare repositories:\n>\n>         top.git/\n>         top.git/refs/{heads,tags,...}/...\n>         top.git/objects/...\n>         top.git/modules/sub.git/\n>         top.git/modules/sub.git/refs/{heads,tags,...}/...\n>         top.git/modules/sub.git/objects/...\n>\n> which would probably make the \"relative\" relationship between the\n> supermodule and its submodules the same between bare and non-bare\n> repositories, but I didn't think it too deeply.\n\nForrests as of now are handled as a flat level thing, e.g.\n\n    git clone git://git.eclipse.org/gitroot/platform/eclipse.platform.releng.aggregator.git\n\nwill produce a superproject with 25 submodules, all of them\nare either at ../ or at ../../ such that it would follow\n\n         projects/top.git/\n         projects/top.git/refs/{heads,tags,...}/...\n         projects/top.git/objects/...\n         projects/sub.git/\n         projects/sub.git/refs/{heads,tags,...}/...\n         projects/sub.git/objects/...\n         libs/sub2.git\n         libs/sub2.git/refs/{heads,tags,...}/...\n         libs/sub2.git/objects/...\n\nLooking at our internal code search there is no .gitmodules file\nwhose url starts with \"./\", they all start with ../ or are absolute.\n"},{"id":"284015","messageId":"20160421031426.GY23764@onerussian.com","threadId":"42089","inReplyTo":"CAGZ79kbyDuu8hUQXOUNmDLyO88GOi3kqSzsc0UfgzwAL9peZHg@mail.gmail.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Yaroslav Halchenko","fromEmail":"yoh@onerussian.com","sentAt":"2016-04-21T03:14:26Z","receivedAt":"2016-04-21T03:14:26Z","isPatch":false,"sender":{"key":"yoh@onerussian.com","avatar":"https://gravatar.com/avatar/8901b82415ae451a83aea49409708912726e53620e3ac92320bf1f86548d97e9?d=mp&s=160"},"body":"NB Thank you for the lively discussion!\n\nOn Wed, 20 Apr 2016, Stefan Beller wrote:\n\n> >> So currently the protocol doesn't allow to even specify the submodules\n> >> directories.\n\n> > Depends on what you exactly mean by \"the protocol\", but the\n> > networking protocol is about accessing a single repository.  It is\n> > up to you to decide where to go next after learning what you can\n> > learn from the result, typically by following what appears in\n> > the .gitmodules file.\n\n> Right. But the .gitmodules file is not sufficient.\n\nwhy?\n\n> >...<\n\n> I think on a hosting site they could even coexist when having the\n> layout as above.\n\n>          top.git/\n>          top.git/refs/{heads,tags,...}/...\n>          top.git/objects/...\n>          sub.git/\n>          sub.git/refs/{heads,tags,...}/...\n>          sub.git/objects/...\n\n>          # the following only exist in non bare:\n>          top.git/modules/sub.git/\n>          top.git/modules/sub.git/refs/{heads,tags,...}/...\n>          top.git/modules/sub.git/objects/...\n\n> The later files would be more reflective of what you *really*\n> want if you clone from top.git.\n\nmay be there is no need for assumptions and .gitmodules should be\nsufficient?\n\n- absolute url in .gitmodules provides absolute URL/path to the\n  submodule of interest, regardless either submodule is present in\n  originating repository as updated submodule.  Either cloning it\n  instead of original repository would be more efficient is already a\n  heuristic which might fail miserably (may be I have a faster\n  connection to the original repository pointed by the absolute\n  url than to this particular repository)\n\n- relative url in .gitmodules provides relative location to the location\n  of the \"top\" repository, and that is only when that submodule \"absolute\"\n  url should be resolved relative to the one of the \"top\" repository \n\nNB I will consider it a separate issue either relative paths\nwithout '../' prefix are having any sense in bare repositories.\n\nor have I missed the point?\n-- \nYaroslav O. Halchenko\nCenter for Open Neuroscience     http://centerforopenneuroscience.org\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419\nWWW:   http://www.linkedin.com/in/yarik        \n"},{"id":"284052","messageId":"CAGZ79kYmobvZgAeJv7MNXeFXd+Ahg5mtyagt5m+f7Gpp=Ps4iQ@mail.gmail.com","threadId":"42089","inReplyTo":"20160421031426.GY23764@onerussian.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-04-21T17:11:02Z","receivedAt":"2016-04-21T17:11:02Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Apr 20, 2016 at 8:14 PM, Yaroslav Halchenko <yoh@onerussian.com> wrote:\n> NB Thank you for the lively discussion!\n>\n> On Wed, 20 Apr 2016, Stefan Beller wrote:\n>\n>> >> So currently the protocol doesn't allow to even specify the submodules\n>> >> directories.\n>\n>> > Depends on what you exactly mean by \"the protocol\", but the\n>> > networking protocol is about accessing a single repository.  It is\n>> > up to you to decide where to go next after learning what you can\n>> > learn from the result, typically by following what appears in\n>> > the .gitmodules file.\n>\n>> Right. But the .gitmodules file is not sufficient.\n>\n> why?\n\nWhat do you expect from cloning a repo with submodules?\n\nIn case of a bare repo:\n\n    Get the repo from the specified remote and get the submodules\n    from \"somewhere\" (and .gitmodules helps you guessing where\n    \"somewhere\" is).\n\nThis has been the traditional way, and the .gitmodules file\nis just a helper for a best guess where to get a submodule sha1\nfrom. (The repo pointed at from the .gitmodules file may not exist\nany more; or it may have forgot the wanted commit)\n\nIn case of non bare:\n\n    Get the repo and all its submodules from the specified remote.\n    (As the submodule is right there, that's the best guess to get it from,\n    no need to get it from somewhere else. The submodule at the remote\n    is the closest match you can get for replicating the superproject with\n    its submodules.)\n\nThis way is heavy underutilized as it wasn't exercised as often I would\nguess, so the \"wrong\" default (to obtain the submodule information from\n.gitmodules instead of from the remote directly) was not pointed out before.\n\nNow that the client wants to make a decision where to get the\nsubmodules from, based on the bare-ness of the remote, it may\nrequire changes in the wire protocol, such that the remote simply\nadvertises it is a (non-)bare repository when you clone the superproject\nfrom it. Then the client can make a better decision where to get the\nsubmodules from.\n\n\n\n\n>\n>> >...<\n>\n>> I think on a hosting site they could even coexist when having the\n>> layout as above.\n>\n>>          top.git/\n>>          top.git/refs/{heads,tags,...}/...\n>>          top.git/objects/...\n>>          sub.git/\n>>          sub.git/refs/{heads,tags,...}/...\n>>          sub.git/objects/...\n>\n>>          # the following only exist in non bare:\n>>          top.git/modules/sub.git/\n>>          top.git/modules/sub.git/refs/{heads,tags,...}/...\n>>          top.git/modules/sub.git/objects/...\n>\n>> The later files would be more reflective of what you *really*\n>> want if you clone from top.git.\n>\n> may be there is no need for assumptions and .gitmodules should be\n> sufficient?\n>\n> - absolute url in .gitmodules provides absolute URL/path to the\n>   submodule of interest, regardless either submodule is present in\n>   originating repository as updated submodule.  Either cloning it\n>   instead of original repository would be more efficient is already a\n>   heuristic which might fail miserably (may be I have a faster\n>   connection to the original repository pointed by the absolute\n>   url than to this particular repository)\n>\n> - relative url in .gitmodules provides relative location to the location\n>   of the \"top\" repository, and that is only when that submodule \"absolute\"\n>   url should be resolved relative to the one of the \"top\" repository\n\nI think the .gitmodules file is not sufficient for the following reason:\n\n* As a \"downstream\" user you cannot change remote locations without\naltering the history. Maybe you just want to have a mirror of some cool\nopen source project without the hassle to always merge and maintain changes\nin your local submodules configuration. (c.f. git config url.<base>.insteadOf\nfor repos, just for submodule specific)\n>\n> NB I will consider it a separate issue either relative paths\n> without '../' prefix are having any sense in bare repositories.\n\nI guess it is a separate issue.\n\n>\n> or have I missed the point?\n> --\n> Yaroslav O. Halchenko\n> Center for Open Neuroscience     http://centerforopenneuroscience.org\n> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\n> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419\n> WWW:   http://www.linkedin.com/in/yarik\n"},{"id":"284058","messageId":"xmqqr3dysvne.fsf@gitster.mtv.corp.google.com","threadId":"42089","inReplyTo":"CAGZ79kYmobvZgAeJv7MNXeFXd+Ahg5mtyagt5m+f7Gpp=Ps4iQ@mail.gmail.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-04-21T17:45:41Z","receivedAt":"2016-04-21T17:45:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n> In case of non bare:\n>\n>     Get the repo and all its submodules from the specified remote.\n>     (As the submodule is right there, that's the best guess to get it from,\n>     no need to get it from somewhere else. The submodule at the remote\n>     is the closest match you can get for replicating the superproject with\n>     its submodules.)\n>\n> This way is heavy underutilized as it wasn't exercised as often I would\n> guess, \n\nMy guess is somewhat different. It is not used because the right\nsemantics for such a use case hasn't been defined yet (in other\nwords, what you suggested is _wrong_ as is).  You need to remember\nthat a particular clone may not be interested in all submodules, and\nit is far from \"the closest match\".\n"},{"id":"284059","messageId":"CAGZ79kYTo+CHPvhV6i0U3xqezhGBSXQs9h+5JL6MRh9oAWXvFg@mail.gmail.com","threadId":"42089","inReplyTo":"xmqqr3dysvne.fsf@gitster.mtv.corp.google.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-04-21T17:48:25Z","receivedAt":"2016-04-21T17:48:25Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Apr 21, 2016 at 10:45 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> Stefan Beller <sbeller@google.com> writes:\n>\n>> In case of non bare:\n>>\n>>     Get the repo and all its submodules from the specified remote.\n>>     (As the submodule is right there, that's the best guess to get it from,\n>>     no need to get it from somewhere else. The submodule at the remote\n>>     is the closest match you can get for replicating the superproject with\n>>     its submodules.)\n>>\n>> This way is heavy underutilized as it wasn't exercised as often I would\n>> guess,\n>\n> My guess is somewhat different. It is not used because the right\n> semantics for such a use case hasn't been defined yet (in other\n> words, what you suggested is _wrong_ as is).  You need to remember\n> that a particular clone may not be interested in all submodules, and\n> it is far from \"the closest match\".\n\nYes, when that clone doesn't have some submodules, we can still fall back\non the .gitmodules file.\n\nIf you have a submodule chances are, you are interested in it and modified it.\nSo the highest chance to get your changes is from your remote, no?\n"},{"id":"284082","messageId":"CA+P7+xrHpjaTNfu-spfdVGM9OcksYpATpp9yeO10-YXU9GTePQ@mail.gmail.com","threadId":"42089","inReplyTo":"CAGZ79kYTo+CHPvhV6i0U3xqezhGBSXQs9h+5JL6MRh9oAWXvFg@mail.gmail.com","subject":"Re: problems serving non-bare repos with submodules over http","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2016-04-21T22:42:50Z","receivedAt":"2016-04-21T22:42:50Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Thu, Apr 21, 2016 at 10:48 AM, Stefan Beller <sbeller@google.com> wrote:\n> On Thu, Apr 21, 2016 at 10:45 AM, Junio C Hamano <gitster@pobox.com> wrote:\n>> Stefan Beller <sbeller@google.com> writes:\n>>\n>>> In case of non bare:\n>>>\n>>>     Get the repo and all its submodules from the specified remote.\n>>>     (As the submodule is right there, that's the best guess to get it from,\n>>>     no need to get it from somewhere else. The submodule at the remote\n>>>     is the closest match you can get for replicating the superproject with\n>>>     its submodules.)\n>>>\n>>> This way is heavy underutilized as it wasn't exercised as often I would\n>>> guess,\n>>\n>> My guess is somewhat different. It is not used because the right\n>> semantics for such a use case hasn't been defined yet (in other\n>> words, what you suggested is _wrong_ as is).  You need to remember\n>> that a particular clone may not be interested in all submodules, and\n>> it is far from \"the closest match\".\n>\n> Yes, when that clone doesn't have some submodules, we can still fall back\n> on the .gitmodules file.\n>\n> If you have a submodule chances are, you are interested in it and modified it.\n> So the highest chance to get your changes is from your remote, no?\n> --\n\nI agree with Stefan. I think that if I clone from my local non-bare\nrepository that may have work done inside the submodule it would be\nbest if the clone could grab the submodules directly from here and get\nthis work which might not yet be in the \"real\" remote yet.\n\nThe case could be made that you don't want to do this, I suppose..\nGenerally I think since we're already connected to this remote we know\nwe can access it, and getting submodules from here means we know it\nwill work, and give us the actual sha1 that the clone is using.\n\nIf we use .gitmodules, we'll possibly get a module that doesn't have\nthe commit, and the current gitmodules url might not even work\nanymore.\n\nThat is, I don't really understand any downside to Stefan's\nproposal,and I see a bunch of upside.\n\nThanks,\nJake\n"}]}