A design for distributed submodules

I think I finally agree that it's best to develop submodules further rather than introduce a new tool for the functionality I require. Here are some explicit proposals for submodules so we can at least establish agreement on what should be done. These are in order of decreasing importance (to me).

* Upstreamless submodules

If there is no 'url' key defined for a submodule in .gitconfig, there is no "authoritative upstream" for it. When a recursive fetch/pull/clone/push is performed on a remote superproject, its upstreamless submodules are also fetched/pulled/cloned/pushed directly from/to the submodule repositories under the superproject .git/modules. If this is the first time that remote's submodules are accessed, that remote is initialized for the local submodules: the submodule of the remote superproject becomes a remote of the local submodule, and is given the same name as the remote of the superproject.

So, suppose we have a superproject with .gitmodules:

[submodule "sub"]
	path = sub

which is hosted at repositories at URL1 and URL2. Then we do:

git clone --recursive URL1 super cd super git remote add other URL2 git fetch --recursive URL2

Now .git/modules/sub/config has:

[remote "origin"]
	url = URL1/.git/modules/sub
[remote "other"]
	url = URL2/.git/modules/sub

So the effect is similar to just setting the submodule's url as ".git/modules/sub", except that:

  - it hides the implementation detail of the exact location of the
    submodule repository from the publicly visible configuration file

  - it also works with bare remotes (where the actual remote submodule
    location would be URL/modules/sub)

  - it allows multiple simultaneous superproject remotes (where
    git-submodule currently always resolves relative urls against
    branch.$branch.remote with no option to fetch from a different
    remote).

* Submodule discovery across all refs

This is what Jens already mentioned. If we fetch multiple refs of a remote superproject, we also need to fetch _all_ of the submodules referenced by _any_ of the refs, not just the ones in the currently active branch. Finding the complete list of submodules probably has to be implemented by reading .gitmodules in all of the (updated) refs, which is a bit ugly, but not too bad.

* Recording the active branch of a submodule

When a submodule is added, its active branch should be stored in .gitmodules as submodule.$sub.branch. Then, when the submodule is checked out, and the head of that branch is the same as the commit in the gitlink (i.e. the superproject tree is "current"), then that branch is set as the active branch in the checked-out submodule working tree. Otherwise, a detached head is used.

* Multiple working trees for a submodule

A superproject may have multiple paths for the same submodule, presumably for different commits. This is for cases where the superproject is a snapshot of a developer's directory hierarchy, and the developer is simultaneously working on multiple branches of a submodule and it is convenient to have separate working trees for each of them.

This is a bit hard to express with the current .gitconfig format, since paths are attributes of repository ids instead of vice versa. I'd introduce an alternative section format where you can say:

[mount "path1"]
   module = sub
   branch = master

[mount "path2"]
   module = sub
   branch = topic

Implementing this is a bit intricate, since we need to use the git-new-workdir method to create multiple working directories that share the same refs, config, and object store, but have separate HEAD and index. I think this is a problem with the repository layout: the non-workdir-specific bits should all be in a single directory so that a single symlink would be enough.

Obviously, I'm willing to implement the above functionalities since I need them. However, I think I'm going to work in Dulwich (which doesn't currently have any submodule support): a Python API is currently more important to me than a command-line tool, and the git.git codebase doesn't look like a very attractive place to contribute anyway. No offense, it's just not to my tastes.

So the main reason I'd like to reach some tentative agreement about the details of the proposal is to ensure that _once_ someone finally implements this kind of functionality in git.git, it will use the same configuration format and same conventions, so that it will be compatible with my code. The compatibility between different tools is after all the main reason for doing this stuff as an extension to submodules instead of something completely different.

Lauri