Re: Making submodules easier to work with

From: Finn Arne Gangstad <finnag@pvv.org>
Date: May 1, 2008, 18:38 UTC
Message-ID: <20080501183837.GA4772@pvv.org>
In-Reply-To: <32541b130804301528k70ae2f7eq5229c0b4bb1d3788@mail.gmail.com>

On Wed, Apr 30, 2008 at 06:28:00PM -0400, Avery Pennarun wrote:

Show 16 quoted lines

> On 4/30/08, Roman Shaposhnik <rvs@sun.com> wrote:
> > On Wed, 2008-04-30 at 17:48 -0400, Avery Pennarun wrote:
> >  > It would be awesome if you could turn the fancy behaviour of this
> >  > bundle into patches to git-submodule, for example, and then have your
> >  > textmate macros call the modified git-submodule.  It might be a bit of
> >  > an uphill battle to get the patches accepted into the release, but I
> >  > think it's worth the effort, as git-submodule in its current state is
> >  > just a non-starter for my group at least.
> >
> > Doesn't the fact that the workflows around submodules tend to differ so
> >  much call for different incarnations of git-submodule? IOW, wouldn't
> >  it be ok to have an alternative to git-submodule somewhere in contrib?
> 
> This would be okay with me.  git-submodule itself doesn't seem to do
> anything very complex.  I would be happy to help contribute to such a
> thing, as my group needs it rather desperately.

Where do we want to go with submodules?

Today, submodules seem to be a "read-only" implementation of the supermodule. By that I mean that it is (only?) suited for creating a supermodule that consists of independently released submodules, where all development happens in the submodules, and you sometimes update the supermodule to refer to a new version of a submodule.

What I've tried to achieve with submodules is a bit different: I want most development to happen in the supermodule _as if_ the submodules were part of the supermodule. There are two reasons for not doing it with one big module: Total size can be a bit too big, but most importantly, some submodules are shared between different super modules and there is a certain level of synchronizing. Does this match your scenarios in any way?

Simplified but realistic scenario [*]:

Supermodule "crawler" has some native code, and uses submodule "os-lib". Supermodule "indexer" has some native code, and uses submodule "os-lib".

One group works on "crawler" and releases a new crawler every 3 months. Another group works on "indexer" and releases a new indexer every 2 weeks.

The guy who wrote "os-lib" quit 2 years ago, and no one really maintains it. Once a year or so someone might try to unify the "os-lib" releases (so it should be POSSIBLE, just not the norm).

What I would like: People should be able to work on "crawler" as if it was a single git repository, in particular this means:

o Branching "crawler" means branching "os-lib"
o You can send a patch that contains changes both to "crawler" and "os-lib"
  and get it applied in a resonable way as ONE modification (and git-am
  would do the right thing)
o Merging branch a and branch b in "crawler" also merges the matching
  branches a and b in "os-lib".
o Pushing the supermodule also pushes the submodules

The branch names in "os-lib" can (and probably should) be mangled, so they are a combination of the supermodule and the branch name. E.g. something like crawler::branch, crawler/branch, or even {crawler}.branch. (Is there a character that is currently not allowed in branch names that can be used as an escape?)

Some lose implementation ideas: (code would be better, I know):

- Enable new behaviour with "git subdirectory" instead of "git submodule",
  and let "git submodule" keep the old behaviour.
- subdirectories get the following magic additions:
  - branching in parent creates {parent}.<branch> in subdirectories
  - merging branches in parent merges the corresponding {parent}.branches
    in subdirectories
  - pushing a parent also pushes the subdirectories
  - changing the parent to something that does not contain the subdirectory
    is only allowed if the subdirectory is "clean" (no uncommitted changes).

[*] In reality: there are 300+ submodules, in all states from "actively maintained" to "last person who understood the code left 10 years ago". There are maybe 5 or 10 supermodules.

- Finn Arne