{"thread":{"id":"30349","subject":"Location-agnostic submodules","startedAt":"2012-04-27T14:37:10Z","lastAt":"2012-05-02T16:55:43Z","messageCount":8,"participants":["Pierre Thierry","Phil Hord","Philip Oakley","Junio C Hamano","Heiko Voigt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"190213","messageId":"20120427143710.GA13953@pape.arcanes.fr.eu.org","threadId":"30349","inReplyTo":null,"subject":"Location-agnostic submodules","fromName":"Pierre Thierry","fromEmail":"pierre@nothos.net","sentAt":"2012-04-27T14:37:10Z","receivedAt":"2012-04-27T14:37:10Z","isPatch":false,"sender":{"key":"pierre@nothos.net","avatar":null},"body":"I just discovered the workings of the submodule command, and as I have\ngrown to like the fact that a repository is not unique with Git, and\nspecifically that it has no unique or central location, I'm bothered\nby how submodule works.\n\nWould there be any major issue in having (1) submodule to be able to\nclone the submodules from the super repository when they are available\nthere and (2) having zero, one or many addresses for each submodule,\nused as hints (obviously not used when (1) is satisfied)?\n\nWhen the repository is not bare, the submodules would be found at\ntheir place in the tree, nothing difficult here. When the repository\nis bare, there could be a tree with the bare repositories of the\nsubmodules.\n\nThis could be done by a new subcommand, that would take a remote as an\noptional argument, its default being origin, as usual:\n\n$ git submodule clone origin\n\n\nAs I see it, adding this 'clone' subcommand for the case where the\nrepository is not bare couldn't add any compatibility issue, so if I'm\nright on this point, I'd like to try and implement this soon.\n\nCuriously,\nPierre\n-- \npierre@nothos.net\nOpenPGP 0xD9D50D8A\n"},{"id":"190343","messageId":"CABURp0qA9hK1fBd3rmLG61ErpbbzrcxO=AAmFOsoxjt=ozu09Q@mail.gmail.com","threadId":"30349","inReplyTo":"20120427143710.GA13953@pape.arcanes.fr.eu.org","subject":"Re: Location-agnostic submodules","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2012-04-30T20:39:49Z","receivedAt":"2012-04-30T20:39:49Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On Fri, Apr 27, 2012 at 10:37 AM, Pierre Thierry wrote:\n> I just discovered the workings of the submodule command, and as I have\n> grown to like the fact that a repository is not unique with Git, and\n> specifically that it has no unique or central location, I'm bothered\n> by how submodule works.\n>\n> Would there be any major issue in having (1) submodule to be able to\n> clone the submodules from the super repository when they are available\n> there and (2) having zero, one or many addresses for each submodule,\n> used as hints (obviously not used when (1) is satisfied)?\n\nMaybe something like this:\n [submodule \"foo\"]\n path = foo-mod\n url = ../foo ../foo-alternate\nhttps://someplace.com/git/foo.git https://kernel.org/git/foo\n\n\nI think the problem now will be that you have an indeterminate source\nURL for your submodule. So far as all of your alternate locations are\nthe same it is probably not a problem. But if one of them lags behind\nthe others by a day or even an hour, then you may have gitlinks in\nyour superproject which have not made it into the lagging mirror yet.\nAnd this will cause problems.\n\nMoreover, each time you clone the repository you may get different\nresults. This would be confusing.\n\nBut aside from these administrative issues, I think this could work.\n\n> When the repository is not bare, the submodules would be found at\n> their place in the tree, nothing difficult here. When the repository\n> is bare, there could be a tree with the bare repositories of the\n> submodules.\n\nI think this could work even if the repository were bare; these days,\nsubmodule repositories are stored in .git/modules/* anyway. So if it\nwere possible to craft a bare repository with this structure in place,\nthen even \"bare\" repos could support embedded submodules like this.\n\nI think this amplifies the relative URL problem. That problem exists\nanyway, but this maybe gives it more ways to fail.\n\n> This could be done by a new subcommand, that would take a remote as an\n> optional argument, its default being origin, as usual:\n>\n> $ git submodule clone origin\n>\n> As I see it, adding this 'clone' subcommand for the case where the\n> repository is not bare couldn't add any compatibility issue, so if I'm\n> right on this point, I'd like to try and implement this soon.\n\nI don't think there is any need for a new 'clone' command since the\nclone porcelain already understands submodules. Maybe a new switch is\nneeded to control the remote to use, but this switch is needed for\nmore cases than just clone.\n\n git remote add shared http://elsewhere.com/project.git\n git submodule init --remote=shared foo\n\nThis could also solve some existing ambiguities with relative paths\nand with 'git-submodule sync', for example.\n\nAt first you worried me, but now I am starting to like this idea more and more.\n\nPhil\n"},{"id":"190351","messageId":"20120430220244.GL22827@pape.arcanes.fr.eu.org","threadId":"30349","inReplyTo":"CABURp0qA9hK1fBd3rmLG61ErpbbzrcxO=AAmFOsoxjt=ozu09Q@mail.gmail.com","subject":"Re: Location-agnostic submodules","fromName":"Pierre Thierry","fromEmail":"pierre@nothos.net","sentAt":"2012-04-30T22:02:44Z","receivedAt":"2012-04-30T22:02:44Z","isPatch":false,"sender":{"key":"pierre@nothos.net","avatar":null},"body":"Scribit Phil Hord dies 30/04/2012 hora 16:39:\n> Maybe something like this:\n> [submodule \"foo\"]\n> path = foo-mod\n> url = ../foo ../foo-alternate\n> https://someplace.com/git/foo.git https://kernel.org/git/foo\n\nThat is typically the kind of occasion when I wish every config\nfile were sexprs...\n\n> But if one of them lags behind the others by a day or even an hour,\n> then you may have gitlinks in your superproject which have not made\n> it into the lagging mirror yet. And this will cause problems.\n\nI see your point, but what if your only repository is lagging behind?\nI.e. in what way is it worse than now?\n\n> Moreover, each time you clone the repository you may get different\n> results. This would be confusing.\n\nAgain, I fail to see the difference with the current state. If the\ncommit is specified, you will always get the same results, now or with\nmy suggested addition.\n\n> I don't think there is any need for a new 'clone' command since the\n> clone porcelain already understands submodules.\n\nWhat do you mean? When I clone a repo with submodules, they are not\ncloned as well.\n\n-- \npierre@nothos.net\nOpenPGP 0xD9D50D8A\n"},{"id":"190422","messageId":"CABURp0rUKubfWXxX2ABG2E4dRTEQh4zA0bZFOeXGwv2m4b0YaA@mail.gmail.com","threadId":"30349","inReplyTo":"20120430220244.GL22827@pape.arcanes.fr.eu.org","subject":"Re: Location-agnostic submodules","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2012-05-01T15:16:35Z","receivedAt":"2012-05-01T15:16:35Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"Adding Jens Lehmann, in case he hasn't noticed this thread yet.\n\nOn Mon, Apr 30, 2012 at 6:02 PM, Pierre Thierry wrote:\n> Scribit Phil Hord dies 30/04/2012 hora 16:39:\n>> Maybe something like this:\n>>     [submodule \"foo\"]\n>>         path = foo-mod\n>>         url = ../foo ../foo-alternate\n>> https://someplace.com/git/foo.git  https://kernel.org/git/foo\n>\n> That is typically the kind of occasion when I wish every config\n> file were sexprs...\n\nInteresting. But at least it's not yaml. :-)\n\n>> But if one of them lags behind the others by a day or even an hour,\n>> then you may have gitlinks in your superproject which have not made\n>> it into the lagging mirror yet.  And this will cause problems.\n>\n> I see your point, but what if your only repository is lagging behind?\n> I.e. in what way is it worse than now?\n\nI actually do not think it is very much worse than now. But the\nspecific way it fails in this case is as follows:\n\nSuppose I have mirrors A and B, each containing a superproject and its\nsubmodule.\n\n A:super:master => A:sub:master\n B:super:master => B:sub:master\n\nA and B are coherent, meaning their superproject gitlinks point to\ncommits which do exist in the submodule repositories.\n\nNow, I push new commits to A:super and A:sub, giving this:\n A:super:new' => A:sub:new\n B:super:master => B:sub:master\n\nNow, A and B are both internally coherent, but I have a problem if I\ntry to do this:\n A:super:master' => B:sub:new\n\nThis is because the sub:new commit has not made it to B yet, perhaps\nintentionally or perhaps because of latency.\n\nThis problem still can occur without your change, so I do not think it\nis a fatal flaw. It is just a scenario to consider.\n\n>> Moreover, each time you clone the repository you may get different\n>> results.  This would be confusing.\n>\n> Again, I fail to see the difference with the current state. If the\n> commit is specified, you will always get the same results, now or with\n> my suggested addition.\n\nThe existing implementation has some flaws, and think the multiple\nURLs option may expose the flaws further. Again, not a fatal flaw to\nyour idea; just something to keep in mind.\n\nSomething else to keep in mind: What you are proposing amounts to a\nfeature which identifies mirror repositories to use for submodules\nwhen the primary remote repo cannot be reached. Superprojects have no\nsuch feature. Why not?\n\nMeanwhile, I really like the other feature you started off with, where\nthe \"embedded\" submodule repos could be used as the primary origin.\n\n>> I don't think there is any need for a new 'clone' command since the\n>> clone porcelain already understands submodules.\n>\n> What do you mean? When I clone a repo with submodules, they are not\n> cloned as well.\n\nSince git v1.6.5 or so, clone has known the --recursive option.\n\n git clone --recursive superproject\n\nSince about v1.7.3, fetch and pull also know how to recurse and can do\nso by default.\n\nPhil\n"},{"id":"190445","messageId":"732A8D2DB3374CD6BC12A5E7C4384313@PhilipOakley","threadId":"30349","inReplyTo":"CABURp0rUKubfWXxX2ABG2E4dRTEQh4zA0bZFOeXGwv2m4b0YaA@mail.gmail.com","subject":"Re: Location-agnostic submodules","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2012-05-01T17:19:38Z","receivedAt":"2012-05-01T17:19:38Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Phil Hord\" Sent: Tuesday, May 01, 2012 4:16 PM\n> Adding Jens Lehmann, in case he hasn't noticed this thread yet.\n>\n> On Mon, Apr 30, 2012 at 6:02 PM, Pierre Thierry wrote:\n>> Scribit Phil Hord dies 30/04/2012 hora 16:39:\n>>> Maybe something like this:\n>>> [submodule \"foo\"]\n>>> path = foo-mod\n>>> url = ../foo ../foo-alternate\n>>> https://someplace.com/git/foo.git https://kernel.org/git/foo\n>>\n>> That is typically the kind of occasion when I wish every config\n>> file were sexprs...\n>\n> Interesting. But at least it's not yaml. :-)\n>\n>>> But if one of them lags behind the others by a day or even an hour,\n>>> then you may have gitlinks in your superproject which have not made\n>>> it into the lagging mirror yet. And this will cause problems.\n>>\n>> I see your point, but what if your only repository is lagging behind?\n>> I.e. in what way is it worse than now?\n>\n> I actually do not think it is very much worse than now. But the\n> specific way it fails in this case is as follows:\n>\n> Suppose I have mirrors A and B, each containing a superproject and its\n> submodule.\n>\n> A:super:master => A:sub:master\n> B:super:master => B:sub:master\n>\n> A and B are coherent, meaning their superproject gitlinks point to\n> commits which do exist in the submodule repositories.\n>\n> Now, I push new commits to A:super and A:sub, giving this:\n> A:super:new' => A:sub:new\n> B:super:master => B:sub:master\n>\n> Now, A and B are both internally coherent, but I have a problem if I\n> try to do this:\n> A:super:master' => B:sub:new\n>\n> This is because the sub:new commit has not made it to B yet, perhaps\n> intentionally or perhaps because of latency.\n>\n> This problem still can occur without your change, so I do not think it\n> is a fatal flaw. It is just a scenario to consider.\n>\n>>> Moreover, each time you clone the repository you may get different\n>>> results. This would be confusing.\n>>\n>> Again, I fail to see the difference with the current state. If the\n>> commit is specified, you will always get the same results, now or with\n>> my suggested addition.\n>\n> The existing implementation has some flaws, and think the multiple\n> URLs option may expose the flaws further. Again, not a fatal flaw to\n> your idea; just something to keep in mind.\n\nWould an alternative be something like:\n git submodule update --from \n\nso that the user can state which of the current submodule's remotes should \nbe used for fetching the desired update.\n\nFor compatibility, the update may need to use the '--reference' or something \nsimilar, or perhaps a different command word\n `git submodule fetch --from `\n\nI was just stung on msysgit (submodule `git`) because its NetInstall clones \nfrom the main repos, but I had forked my own copies, so the submodule URLs \nweren't right for me (doh). Luckily I have a patch of my silenty overwritten \nchanges...\n\n>\n> Something else to keep in mind: What you are proposing amounts to a\n> feature which identifies mirror repositories to use for submodules\n> when the primary remote repo cannot be reached. Superprojects have no\n> such feature. Why not?\n>\n> Meanwhile, I really like the other feature you started off with, where\n> the \"embedded\" submodule repos could be used as the primary origin.\n>\n>>> I don't think there is any need for a new 'clone' command since the\n>>> clone porcelain already understands submodules.\n>>\n>> What do you mean? When I clone a repo with submodules, they are not\n>> cloned as well.\n>\n> Since git v1.6.5 or so, clone has known the --recursive option.\n>\n> git clone --recursive superproject\n>\n> Since about v1.7.3, fetch and pull also know how to recurse and can do\n> so by default.\n>\n> Phil\n\nPhilip \n"},{"id":"190452","messageId":"7v4nrzaioc.fsf@alter.siamese.dyndns.org","threadId":"30349","inReplyTo":"732A8D2DB3374CD6BC12A5E7C4384313@PhilipOakley","subject":"Re: Location-agnostic submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-05-01T17:57:07Z","receivedAt":"2012-05-01T17:57:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Philip Oakley\" writes:\n\n> Would an alternative be something like:\n> git submodule update --from \n>\n> so that the user can state which of the current submodule's remotes\n> should be used for fetching the desired update.\n\nAre you assuming that the in the above example will be different\nper invocation for a single user? I would imagine not---it would be more\nlike \"the upstream has this URL in .gitmodules, but this other mirror is\ncloser to my network environment\", i.e.\n\n\tcd && git config remote.origin.url $there\n\nno?\n"},{"id":"190484","messageId":"A688673CF88C4F469D3DF0D7E3AEFFA4@PhilipOakley","threadId":"30349","inReplyTo":"7v4nrzaioc.fsf@alter.siamese.dyndns.org","subject":"Re: Location-agnostic submodules","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2012-05-01T19:58:25Z","receivedAt":"2012-05-01T19:58:25Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Junio C Hamano\" Sent: Tuesday, May 01, 2012 6:57\nPM\n> \"Philip Oakley\" writes:\n>\n>> Would an alternative be something like:\n>> git submodule update --from \n>>\n>> so that the user can state which of the current submodule's remotes\n>> should be used for fetching the desired update.\n>\n> Are you assuming that the in the above example will be different\n> per invocation for a single user?\n\npossibly, but more likely the user would have identified which is the best\nremote to use to find her missing sha1.\n\n> I would imagine not---it would be more\n> like \"the upstream has this URL in .gitmodules, but this other mirror is\n> closer to my network environment\", i.e.\n>\n> cd && git config remote.origin.url $there\n>\nI was presuming a reverse time sequence, where the user had already set up\nthe desired remote, but hadn't managed to change the URL in .gitmodules; but\neither way, the user then let's 'git submodule' do the hard work of fetching\nthe correct sha1 to checkout.\n\nI didn't think that there was a command yet to do the URL update, which\nwould most likely match one of the sub-module's URLs.\n\n> no?\n>\n>\nPhilip\n"},{"id":"190576","messageId":"20120502165543.GB49462@book.hvoigt.net","threadId":"30349","inReplyTo":"7v4nrzaioc.fsf@alter.siamese.dyndns.org","subject":"Re: Location-agnostic submodules","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2012-05-02T16:55:43Z","receivedAt":"2012-05-02T16:55:43Z","isPatch":false,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"Hi,\n\nOn Tue, May 01, 2012 at 10:57:07AM -0700, Junio C Hamano wrote:\n> \"Philip Oakley\" writes:\n> \n> > Would an alternative be something like:\n> > git submodule update --from \n> >\n> > so that the user can state which of the current submodule's remotes\n> > should be used for fetching the desired update.\n> \n> Are you assuming that the in the above example will be different\n> per invocation for a single user? I would imagine not---it would be more\n> like \"the upstream has this URL in .gitmodules, but this other mirror is\n> closer to my network environment\", i.e.\n> \n> \tcd && git config remote.origin.url $there\n> \n> no?\n\nYes I think this is an important point. If we start working on this I\nwould like to emphasize the fork use case Philip brought up. When\ncloning a forked repository with submodules you always have the problem\nof changing/adding the forked submodules remotes afterwards.\n\nFor me it would be more like an additional lookup mechanism of the\n\"official\" urls / names. Since I like to stay as close as possible to\nthe upstream repository I usually refrain from changing the .gitmodules\nfile. A changed .gitmodules file with additional urls (possibly some\nprivate ones) is not something you can propagate upstream.\n\nWhat I would like is a mechanism that, given a wanted sha1, would lookup\nthe correct remote to clone/fetch from. But, I have to admit that even\nthough thinking about this for some time now, I have not got to a\nsatisfying answer for myself yet.\n\nCheers Heiko\n"}]}