# Location-agnostic submodules 8 messages from 2012-04-27 to 2012-05-02. Participants: Pierre Thierry, Phil Hord, Philip Oakley, Junio C Hamano, Heiko Voigt. Thread: https://gitlist.dev/t/30349 ## Pierre Thierry, 2012-04-27 14:37 Subject: Location-agnostic submodules Message-ID: <20120427143710.GA13953@pape.arcanes.fr.eu.org> URL: https://gitlist.dev/e/20120427143710.GA13953%40pape.arcanes.fr.eu.org ``` I just discovered the workings of the submodule command, and as I have grown to like the fact that a repository is not unique with Git, and specifically that it has no unique or central location, I'm bothered by how submodule works. Would there be any major issue in having (1) submodule to be able to clone the submodules from the super repository when they are available there and (2) having zero, one or many addresses for each submodule, used as hints (obviously not used when (1) is satisfied)? When the repository is not bare, the submodules would be found at their place in the tree, nothing difficult here. When the repository is bare, there could be a tree with the bare repositories of the submodules. This could be done by a new subcommand, that would take a remote as an optional argument, its default being origin, as usual: $ git submodule clone origin As I see it, adding this 'clone' subcommand for the case where the repository is not bare couldn't add any compatibility issue, so if I'm right on this point, I'd like to try and implement this soon. Curiously, Pierre -- pierre@nothos.net OpenPGP 0xD9D50D8A ``` ## Phil Hord, 2012-04-30 20:39 Subject: Re: Location-agnostic submodules Message-ID: URL: https://gitlist.dev/e/CABURp0qA9hK1fBd3rmLG61ErpbbzrcxO%3DAAmFOsoxjt%3Dozu09Q%40mail.gmail.com In-Reply-To: <20120427143710.GA13953@pape.arcanes.fr.eu.org> ``` On Fri, Apr 27, 2012 at 10:37 AM, Pierre Thierry wrote: > I just discovered the workings of the submodule command, and as I have > grown to like the fact that a repository is not unique with Git, and > specifically that it has no unique or central location, I'm bothered > by how submodule works. > > Would there be any major issue in having (1) submodule to be able to > clone the submodules from the super repository when they are available > there and (2) having zero, one or many addresses for each submodule, > used as hints (obviously not used when (1) is satisfied)? Maybe something like this: [submodule "foo"] path = foo-mod url = ../foo ../foo-alternate https://someplace.com/git/foo.git https://kernel.org/git/foo I think the problem now will be that you have an indeterminate source URL for your submodule. So far as all of your alternate locations are the same it is probably not a problem. But if one of them lags behind the others by a day or even an hour, then you may have gitlinks in your superproject which have not made it into the lagging mirror yet. And this will cause problems. Moreover, each time you clone the repository you may get different results. This would be confusing. But aside from these administrative issues, I think this could work. > When the repository is not bare, the submodules would be found at > their place in the tree, nothing difficult here. When the repository > is bare, there could be a tree with the bare repositories of the > submodules. I think this could work even if the repository were bare; these days, submodule repositories are stored in .git/modules/* anyway. So if it were possible to craft a bare repository with this structure in place, then even "bare" repos could support embedded submodules like this. I think this amplifies the relative URL problem. That problem exists anyway, but this maybe gives it more ways to fail. > This could be done by a new subcommand, that would take a remote as an > optional argument, its default being origin, as usual: > > $ git submodule clone origin > > As I see it, adding this 'clone' subcommand for the case where the > repository is not bare couldn't add any compatibility issue, so if I'm > right on this point, I'd like to try and implement this soon. I don't think there is any need for a new 'clone' command since the clone porcelain already understands submodules. Maybe a new switch is needed to control the remote to use, but this switch is needed for more cases than just clone. git remote add shared http://elsewhere.com/project.git git submodule init --remote=shared foo This could also solve some existing ambiguities with relative paths and with 'git-submodule sync', for example. At first you worried me, but now I am starting to like this idea more and more. Phil ``` ## Pierre Thierry, 2012-04-30 22:02 Subject: Re: Location-agnostic submodules Message-ID: <20120430220244.GL22827@pape.arcanes.fr.eu.org> URL: https://gitlist.dev/e/20120430220244.GL22827%40pape.arcanes.fr.eu.org In-Reply-To: ``` Scribit Phil Hord dies 30/04/2012 hora 16:39: > Maybe something like this: > [submodule "foo"] > path = foo-mod > url = ../foo ../foo-alternate > https://someplace.com/git/foo.git https://kernel.org/git/foo That is typically the kind of occasion when I wish every config file were sexprs... > But if one of them lags behind the others by a day or even an hour, > then you may have gitlinks in your superproject which have not made > it into the lagging mirror yet. And this will cause problems. I see your point, but what if your only repository is lagging behind? I.e. in what way is it worse than now? > Moreover, each time you clone the repository you may get different > results. This would be confusing. Again, I fail to see the difference with the current state. If the commit is specified, you will always get the same results, now or with my suggested addition. > I don't think there is any need for a new 'clone' command since the > clone porcelain already understands submodules. What do you mean? When I clone a repo with submodules, they are not cloned as well. -- pierre@nothos.net OpenPGP 0xD9D50D8A ``` ## Phil Hord, 2012-05-01 15:16 Subject: Re: Location-agnostic submodules Message-ID: URL: https://gitlist.dev/e/CABURp0rUKubfWXxX2ABG2E4dRTEQh4zA0bZFOeXGwv2m4b0YaA%40mail.gmail.com In-Reply-To: <20120430220244.GL22827@pape.arcanes.fr.eu.org> ``` Adding Jens Lehmann, in case he hasn't noticed this thread yet. On Mon, Apr 30, 2012 at 6:02 PM, Pierre Thierry wrote: > Scribit Phil Hord dies 30/04/2012 hora 16:39: >> Maybe something like this: >>     [submodule "foo"] >>         path = foo-mod >>         url = ../foo ../foo-alternate >> https://someplace.com/git/foo.git  https://kernel.org/git/foo > > That is typically the kind of occasion when I wish every config > file were sexprs... Interesting. But at least it's not yaml. :-) >> But if one of them lags behind the others by a day or even an hour, >> then you may have gitlinks in your superproject which have not made >> it into the lagging mirror yet.  And this will cause problems. > > I see your point, but what if your only repository is lagging behind? > I.e. in what way is it worse than now? I actually do not think it is very much worse than now. But the specific way it fails in this case is as follows: Suppose I have mirrors A and B, each containing a superproject and its submodule. A:super:master => A:sub:master B:super:master => B:sub:master A and B are coherent, meaning their superproject gitlinks point to commits which do exist in the submodule repositories. Now, I push new commits to A:super and A:sub, giving this: A:super:new' => A:sub:new B:super:master => B:sub:master Now, A and B are both internally coherent, but I have a problem if I try to do this: A:super:master' => B:sub:new This is because the sub:new commit has not made it to B yet, perhaps intentionally or perhaps because of latency. This problem still can occur without your change, so I do not think it is a fatal flaw. It is just a scenario to consider. >> Moreover, each time you clone the repository you may get different >> results.  This would be confusing. > > Again, I fail to see the difference with the current state. If the > commit is specified, you will always get the same results, now or with > my suggested addition. The existing implementation has some flaws, and think the multiple URLs option may expose the flaws further. Again, not a fatal flaw to your idea; just something to keep in mind. Something else to keep in mind: What you are proposing amounts to a feature which identifies mirror repositories to use for submodules when the primary remote repo cannot be reached. Superprojects have no such feature. Why not? Meanwhile, I really like the other feature you started off with, where the "embedded" submodule repos could be used as the primary origin. >> I don't think there is any need for a new 'clone' command since the >> clone porcelain already understands submodules. > > What do you mean? When I clone a repo with submodules, they are not > cloned as well. Since git v1.6.5 or so, clone has known the --recursive option. git clone --recursive superproject Since about v1.7.3, fetch and pull also know how to recurse and can do so by default. Phil ``` ## Philip Oakley, 2012-05-01 17:19 Subject: Re: Location-agnostic submodules Message-ID: <732A8D2DB3374CD6BC12A5E7C4384313@PhilipOakley> URL: https://gitlist.dev/e/732A8D2DB3374CD6BC12A5E7C4384313%40PhilipOakley In-Reply-To: ``` From: "Phil Hord" Sent: Tuesday, May 01, 2012 4:16 PM > Adding Jens Lehmann, in case he hasn't noticed this thread yet. > > On Mon, Apr 30, 2012 at 6:02 PM, Pierre Thierry wrote: >> Scribit Phil Hord dies 30/04/2012 hora 16:39: >>> Maybe something like this: >>> [submodule "foo"] >>> path = foo-mod >>> url = ../foo ../foo-alternate >>> https://someplace.com/git/foo.git https://kernel.org/git/foo >> >> That is typically the kind of occasion when I wish every config >> file were sexprs... > > Interesting. But at least it's not yaml. :-) > >>> But if one of them lags behind the others by a day or even an hour, >>> then you may have gitlinks in your superproject which have not made >>> it into the lagging mirror yet. And this will cause problems. >> >> I see your point, but what if your only repository is lagging behind? >> I.e. in what way is it worse than now? > > I actually do not think it is very much worse than now. But the > specific way it fails in this case is as follows: > > Suppose I have mirrors A and B, each containing a superproject and its > submodule. > > A:super:master => A:sub:master > B:super:master => B:sub:master > > A and B are coherent, meaning their superproject gitlinks point to > commits which do exist in the submodule repositories. > > Now, I push new commits to A:super and A:sub, giving this: > A:super:new' => A:sub:new > B:super:master => B:sub:master > > Now, A and B are both internally coherent, but I have a problem if I > try to do this: > A:super:master' => B:sub:new > > This is because the sub:new commit has not made it to B yet, perhaps > intentionally or perhaps because of latency. > > This problem still can occur without your change, so I do not think it > is a fatal flaw. It is just a scenario to consider. > >>> Moreover, each time you clone the repository you may get different >>> results. This would be confusing. >> >> Again, I fail to see the difference with the current state. If the >> commit is specified, you will always get the same results, now or with >> my suggested addition. > > The existing implementation has some flaws, and think the multiple > URLs option may expose the flaws further. Again, not a fatal flaw to > your idea; just something to keep in mind. Would an alternative be something like: git submodule update --from so that the user can state which of the current submodule's remotes should be used for fetching the desired update. For compatibility, the update may need to use the '--reference' or something similar, or perhaps a different command word `git submodule fetch --from ` I was just stung on msysgit (submodule `git`) because its NetInstall clones from the main repos, but I had forked my own copies, so the submodule URLs weren't right for me (doh). Luckily I have a patch of my silenty overwritten changes... > > Something else to keep in mind: What you are proposing amounts to a > feature which identifies mirror repositories to use for submodules > when the primary remote repo cannot be reached. Superprojects have no > such feature. Why not? > > Meanwhile, I really like the other feature you started off with, where > the "embedded" submodule repos could be used as the primary origin. > >>> I don't think there is any need for a new 'clone' command since the >>> clone porcelain already understands submodules. >> >> What do you mean? When I clone a repo with submodules, they are not >> cloned as well. > > Since git v1.6.5 or so, clone has known the --recursive option. > > git clone --recursive superproject > > Since about v1.7.3, fetch and pull also know how to recurse and can do > so by default. > > Phil Philip ``` ## Junio C Hamano, 2012-05-01 17:57 Subject: Re: Location-agnostic submodules Message-ID: <7v4nrzaioc.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v4nrzaioc.fsf%40alter.siamese.dyndns.org In-Reply-To: <732A8D2DB3374CD6BC12A5E7C4384313@PhilipOakley> ``` "Philip Oakley" writes: > Would an alternative be something like: > git submodule update --from > > so that the user can state which of the current submodule's remotes > should be used for fetching the desired update. Are you assuming that the in the above example will be different per invocation for a single user? I would imagine not---it would be more like "the upstream has this URL in .gitmodules, but this other mirror is closer to my network environment", i.e. cd && git config remote.origin.url $there no? ``` ## Philip Oakley, 2012-05-01 19:58 Subject: Re: Location-agnostic submodules Message-ID: URL: https://gitlist.dev/e/A688673CF88C4F469D3DF0D7E3AEFFA4%40PhilipOakley In-Reply-To: <7v4nrzaioc.fsf@alter.siamese.dyndns.org> ``` From: "Junio C Hamano" Sent: Tuesday, May 01, 2012 6:57 PM > "Philip Oakley" writes: > >> Would an alternative be something like: >> git submodule update --from >> >> so that the user can state which of the current submodule's remotes >> should be used for fetching the desired update. > > Are you assuming that the in the above example will be different > per invocation for a single user? possibly, but more likely the user would have identified which is the best remote to use to find her missing sha1. > I would imagine not---it would be more > like "the upstream has this URL in .gitmodules, but this other mirror is > closer to my network environment", i.e. > > cd && git config remote.origin.url $there > I was presuming a reverse time sequence, where the user had already set up the desired remote, but hadn't managed to change the URL in .gitmodules; but either way, the user then let's 'git submodule' do the hard work of fetching the correct sha1 to checkout. I didn't think that there was a command yet to do the URL update, which would most likely match one of the sub-module's URLs. > no? > > Philip ``` ## Heiko Voigt, 2012-05-02 16:55 Subject: Re: Location-agnostic submodules Message-ID: <20120502165543.GB49462@book.hvoigt.net> URL: https://gitlist.dev/e/20120502165543.GB49462%40book.hvoigt.net In-Reply-To: <7v4nrzaioc.fsf@alter.siamese.dyndns.org> ``` Hi, On Tue, May 01, 2012 at 10:57:07AM -0700, Junio C Hamano wrote: > "Philip Oakley" writes: > > > Would an alternative be something like: > > git submodule update --from > > > > so that the user can state which of the current submodule's remotes > > should be used for fetching the desired update. > > Are you assuming that the in the above example will be different > per invocation for a single user? I would imagine not---it would be more > like "the upstream has this URL in .gitmodules, but this other mirror is > closer to my network environment", i.e. > > cd && git config remote.origin.url $there > > no? Yes I think this is an important point. If we start working on this I would like to emphasize the fork use case Philip brought up. When cloning a forked repository with submodules you always have the problem of changing/adding the forked submodules remotes afterwards. For me it would be more like an additional lookup mechanism of the "official" urls / names. Since I like to stay as close as possible to the upstream repository I usually refrain from changing the .gitmodules file. A changed .gitmodules file with additional urls (possibly some private ones) is not something you can propagate upstream. What I would like is a mechanism that, given a wanted sha1, would lookup the correct remote to clone/fetch from. But, I have to admit that even though thinking about this for some time now, I have not got to a satisfying answer for myself yet. Cheers Heiko ```