[RFC] git-submodule: multiple-level modules definition

1. .gitmodules supports multiple submodule level definition

This follows suggestions from Linus and and Sven Verdoolaege etc. See "Let .git/config specify the url for submodules" (http://article.gmane.org/gmane.comp.version-control.git/48939).

.gitmodules with with multiple level of indirection
------------------------------------------------------
[submodule "service"]
    submodule = crawler
    submodule = search
[submodule "crawler"]
    submodule = util
    submodule = imcrawter
[submodule "search"]
    submodule = util
    submodule = imsearch
[submodule "util"]                      (1)
    url = git://xyzzy/util.git
[submodule "imsearch"]
    path = search/imsearch
    url = git://xyzzy/imsearch.git
[submodule "imcrawler"]
    path = crawler/imcrawter
    url = git://xyzzy/imcrawter.git

(1): path default to submodule name ------------------------------------------------------

To simplify the case, submodule sections with submodule attribute should have neither path attribute nor url attribute (if have, should be ignored).

2. Designate modules by logical names instead of real path

"--name|-n" options is proposed with which modules are designated by logical names instead of real paths as following example shows.

Identical commands forms with/without "--name" --------------------------------------------------- $ git submodule XXX util imcrawler (1) $ git submodule XXX -n crawler (2) $ git submodule XXX util imcrawler imsearch (3) $ git submodule XXX -n service (4) $ git submodule XXX -n crawler search (5)

* XXX represent list, update or init, but not add
* (1) and (2) are idetical conditionally (explained below)
* (3), (4) and (5) are idetical conditionally
---------------------------------------------------

There are still minor difference between these two forms.

In the no "--name" form, the path parameter may be not the real submodule path, and it just acts as the filter for real submodule paths.

In the "--name" form, name parameter must be the logical name, and the real paths corresponding to the logical name may be neither a submodule path nor even existent. How to handle such a path depends on the subcommand.

  1) status: Output 0{40} as the sha1. Doing this can remind the user to
     add the path as submodule or delete the path from .gitmodules.
  2) update: Skip that path and issue a warning
  3) init: Also init for that path

So in the example above, commands (1) and (2) are identical only when util and imcrawler are already submodules.

3. Add submodules in batch with "git submodule add --name"

The former workflow to add submodules is adding one by one with "git submodule add url path" which then modifies .gitmodules. However, sometimes it may be more convenient to work in the reverse way: edit .gitmodules first and then add submodules in batch.

Now "git submodule add --name modulename" can help us to do that. It will find all submodules corresponding to the logical name and add them in batch by using the path and url from .gitmodules. Of course, it will skip those paths which have already been submodules.

4. Do we really need 'init' subcommand?

I think .gitmodules should have been bound more tightly with "git submodule". Now it seems that 'update' or 'status' can still be performed without .gitmodules after "git submodule init".

So why not to read submodules info from .gitmodules directly just like what .gitignore nor .gitattributes have been dealt with? Just for editing them more free?

which just records some paths special to the local repository. But .git/info/exclude can't replace .gitignore. But why can .git/config replace .gitmodules which is a naive attribute of the repository?

After introducing multi-level module definition, .gitmodules becomes more naive to the repository. And it becomes more inappropriate to put all .gitmodules contents into .git/config since multi-level module definition shouldn't be edited outside .gitmodules.

On Sun, Mar 2, 2008 at 6:11 PM, Ping Yin <pkufranky@gmail.com> wrote:

Yes: .git/config tells git which submodule(s) the user is interested
in and where to fetch updates from for these submodules, while
.gitmodules is used to map a submodule _name_ to a submodule _path_
and also to provide a url where submodule updates _can_ be fetched.

So the actual use of submodules is strictly a local configuration issue and as such it needs to be kept in .git/config. And to help the user populate .git/config with submodule information we have 'git submodule init'.

-- larsh

On Mon, Mar 3, 2008 at 2:18 AM, Lars Hjemli <hjemli@gmail.com> wrote:

Is it always a local configuration issue? In a project with hundreds of submodule, will anybody modify the submodule url? I think .gitmodules can be the default repository configuration, and when we have local configuration issue, we can override the configuration in .git/config with the help of git init. But why always use git init even when we don't have the local configuration requirement?

On Mon, Mar 3, 2008 at 2:47 AM, Ping Yin <pkufranky@gmail.com> wrote:

We need a way to tell git which of those submodules we are interested in, i.e. which submodules to clone/fetch during 'git submodule update'.

-- larsh

On Mon, Mar 3, 2008 at 2:57 PM, Lars Hjemli <hjemli@gmail.com> wrote:

Now that's easy with the multiple level module definition. We can now group our modules by hierachy and designate modules we are interested by logical names such as crawler or search as my proposal has mentioned.

And if there are not a logical module name satisfying our requirement, we can define one by ourself in .git/config (if it's a special or temporary requirement) or in .gitmodules (if it's a common requirement).

Now .git/config and .gitmodules will play the same as .git/info/exclude and .gitignore.

On Mon, Mar 3, 2008 at 12:02 PM, Ping Yin <pkufranky@gmail.com> wrote:

Sorry, you've lost me here. How is 'git submodule update' going to avoid cloning/fetching all submodules with your proposal?

PS: the supposed work-flow is an initial 'git submodule init foo bar
baz' (i.e. register the submodules you're interested in checking out)
followed by an occasional 'git submodule update' (i.e. update all the
submodules you are interested in).

-- lh

On Mon, Mar 3, 2008 at 7:51 PM, Lars Hjemli <hjemli@gmail.com> wrote:

See my first email. In following .gitmodules example, we can use "git submodule update service" to update all sub-submodules of submodule "service" (util, imsearch, imcrawler). We don't have to do "git init" first.

Without the multiple level module definition, "git init" is good idea, since every time we want to 'git update', we needn't give so many module paths. However, if we introduce multiple level module definition, we can define the module groups we are interested such as crawler, search and the toppest group service. So we just need to type one or few module names which makes "git init" not so useful.

And it's more flexible by using module name: at one time we can update "cralwer" module group, and at another time we can update "search" module group.

.gitmodules example
------------------------------------------------------
[submodule "service"]
   submodule = crawler
   submodule = search
[submodule "crawler"]
   submodule = util
   submodule = imcrawter
[submodule "search"]
   submodule = util
   submodule = imsearch
[submodule "util"]                      (1)
   url = git://xyzzy/util.git
[submodule "imsearch"]
   path = search/imsearch
   url = git://xyzzy/imsearch.git
[submodule "imcrawler"]
   path = crawler/imcrawter
   url = git://xyzzy/imcrawter.git
------------------------------------------------------

On Mon, Mar 3, 2008 at 1:11 AM, Ping Yin <pkufranky@gmail.com> wrote:

Maybe i should introduce more about the background.

In my company we try to keep all the codes in a unique big environment which consists of many sub projects (more than 30). These subprojects are hierachily organized. There are different roles which need touch different part of these sub projects. For example, crawler deveoper should touch crawler-related projects, and search developer need touch search-related projects. There are also some common modules which are used by most of these roles. There are also many other roles.

So for easy use, we need define one logical name for each kind of role and this name should correspond to all the role-related sub projects. Filtering submodules by path is not so useful for us because role-related projects may live in many different directories.

same big environment. And we don't want to use different branch for different roles. So all roles will get the same .gitmodules file. They are not much familiar with git and can't afford to type git init path1 ... pathn each time when they want to build their develop environment. It's more friendly to them if they only need to remember a logical name such as crawler or search.

And some people have multiple roles. Sometime they want to work on search and sometimes on crawler. So logical name is very convenient to them.

As a conclusion, my main point is
1. Multiple level module definition is very useful since it can help
us build hierachy of modules
2. Designating modules by logical name which corresponds to a group of
role-related modules is usefull since it can help role-based
development much.