Location-agnostic submodules

I just discovered the workings of the submodule command, and as I have grown to like the fact that a repository is not unique with Git, and specifically that it has no unique or central location, I'm bothered by how submodule works.

Would there be any major issue in having (1) submodule to be able to clone the submodules from the super repository when they are available there and (2) having zero, one or many addresses for each submodule, used as hints (obviously not used when (1) is satisfied)?

When the repository is not bare, the submodules would be found at their place in the tree, nothing difficult here. When the repository is bare, there could be a tree with the bare repositories of the submodules.

This could be done by a new subcommand, that would take a remote as an optional argument, its default being origin, as usual:

$ git submodule clone origin

As I see it, adding this 'clone' subcommand for the case where the repository is not bare couldn't add any compatibility issue, so if I'm right on this point, I'd like to try and implement this soon.

Curiously, Pierre

On Fri, Apr 27, 2012 at 10:37 AM, Pierre Thierry <pierre@nothos.net> wrote:

Maybe something like this:
    [submodule "foo"]
        path = foo-mod
        url = ../foo ../foo-alternate
https://someplace.com/git/foo.git  https://kernel.org/git/foo

I think the problem now will be that you have an indeterminate source URL for your submodule. So far as all of your alternate locations are the same it is probably not a problem. But if one of them lags behind the others by a day or even an hour, then you may have gitlinks in your superproject which have not made it into the lagging mirror yet. And this will cause problems.

Moreover, each time you clone the repository you may get different results. This would be confusing.

But aside from these administrative issues, I think this could work.

I think this could work even if the repository were bare; these days, submodule repositories are stored in .git/modules/* anyway. So if it were possible to craft a bare repository with this structure in place, then even "bare" repos could support embedded submodules like this.

I think this amplifies the relative URL problem. That problem exists anyway, but this maybe gives it more ways to fail.

I don't think there is any need for a new 'clone' command since the clone porcelain already understands submodules. Maybe a new switch is needed to control the remote to use, but this switch is needed for more cases than just clone.

   git remote add shared http://elsewhere.com/project.git
   git submodule init --remote=shared foo

This could also solve some existing ambiguities with relative paths and with 'git-submodule sync', for example.

At first you worried me, but now I am starting to like this idea more and more.

Phil

Scribit Phil Hord dies 30/04/2012 hora 16:39:

<rant>That is typically the kind of occasion when I wish every config file were sexprs...</rant>

I see your point, but what if your only repository is lagging behind? I.e. in what way is it worse than now?

Again, I fail to see the difference with the current state. If the commit is specified, you will always get the same results, now or with my suggested addition.

What do you mean? When I clone a repo with submodules, they are not cloned as well.

Adding Jens Lehmann, in case he hasn't noticed this thread yet.

On Mon, Apr 30, 2012 at 6:02 PM, Pierre Thierry <pierre@nothos.net> wrote:

Interesting.  But at least it's not yaml.  :-)

I actually do not think it is very much worse than now. But the specific way it fails in this case is as follows:

Suppose I have mirrors A and B, each containing a superproject and its submodule.

  A:super:master => A:sub:master
  B:super:master => B:sub:master

A and B are coherent, meaning their superproject gitlinks point to commits which do exist in the submodule repositories.

Now, I push new commits to A:super and A:sub, giving this:
  A:super:new' => A:sub:new
  B:super:master => B:sub:master

Now, A and B are both internally coherent, but I have a problem if I
try to do this:
  A:super:master' => B:sub:new

This is because the sub:new commit has not made it to B yet, perhaps intentionally or perhaps because of latency.

This problem still can occur without your change, so I do not think it is a fatal flaw. It is just a scenario to consider.

The existing implementation has some flaws, and think the multiple URLs option may expose the flaws further. Again, not a fatal flaw to your idea; just something to keep in mind.

Something else to keep in mind: What you are proposing amounts to a feature which identifies mirror repositories to use for submodules when the primary remote repo cannot be reached. Superprojects have no such feature. Why not?

Meanwhile, I really like the other feature you started off with, where the "embedded" submodule repos could be used as the primary origin.

Since git v1.6.5 or so, clone has known the --recursive option.

  git clone --recursive superproject

Since about v1.7.3, fetch and pull also know how to recurse and can do so by default.

Phil

From: "Phil Hord" <phil.hord@gmail.com> Sent: Tuesday, May 01, 2012 4:16 PM

Would an alternative be something like:
    git submodule update <module> --from <remote>

so that the user can state which of the current submodule's remotes should be used for fetching the desired update.

For compatibility, the update may need to use the '--reference' or something 
similar, or perhaps a different command word
    `git submodule fetch <module> --from <remote>`

I was just stung on msysgit (submodule `git`) because its NetInstall clones from the main repos, but I had forked my own copies, so the submodule URLs weren't right for me (doh). Luckily I have a patch of my silenty overwritten changes...

Philip 

"Philip Oakley" <philipoakley@iee.org> writes:

Are you assuming that the <remote> in the above example will be different per invocation for a single user? I would imagine not---it would be more like "the upstream has this URL in .gitmodules, but this other mirror is closer to my network environment", i.e.

	cd <module's directory> && git config remote.origin.url $there

no?

From: "Junio C Hamano" <gitster@pobox.com> Sent: Tuesday, May 01, 2012 6:57
PM

possibly, but more likely the user would have identified which is the best remote to use to find her missing sha1.

I was presuming a reverse time sequence, where the user had already set up the desired remote, but hadn't managed to change the URL in .gitmodules; but either way, the user then let's 'git submodule' do the hard work of fetching the correct sha1 to checkout.

I didn't think that there was a command yet to do the URL update, which would most likely match one of the sub-module's URLs.

Philip

Hi,

On Tue, May 01, 2012 at 10:57:07AM -0700, Junio C Hamano wrote:

Yes I think this is an important point. If we start working on this I would like to emphasize the fork use case Philip brought up. When cloning a forked repository with submodules you always have the problem of changing/adding the forked submodules remotes afterwards.

For me it would be more like an additional lookup mechanism of the "official" urls / names. Since I like to stay as close as possible to the upstream repository I usually refrain from changing the .gitmodules file. A changed .gitmodules file with additional urls (possibly some private ones) is not something you can propagate upstream.

What I would like is a mechanism that, given a wanted sha1, would lookup the correct remote to clone/fetch from. But, I have to admit that even though thinking about this for some time now, I have not got to a satisfying answer for myself yet.

Cheers Heiko