Composing git repositories

From: Ramkumar Ramachandra <artagnon@gmail.com>
Date: Mar 26, 2013, 07:56 UTC
Message-ID: <CALkWK0=CsuAWQwk5Guf0pbC4_ZEoZiwQpamcRvBGz5LJ0QGKHg@mail.gmail.com>

(Changed subject) (+CC: Peff; I suspect he'll be interested in the repository composition discussion)

Jens Lehmann wrote:

Show 7 quoted lines

> Am 25.03.2013 20:57, schrieb Ramkumar Ramachandra:
>> Okay, I'll do it step-by-step now, with a live example:
>> [...]
>>  What is going on, seriously?
>
> Pilot error, mostly omitting the --recursive option and some
> - fixable - usability issues. Patches welcome.

As a user inexperienced with recursive submodules (I've only used them in this repository), I found it highly confusing. Thanks for clearing them up.

It was highly exaggerated and dramatized to make the following points clear:

1. With the current limitation of cd-to-toplevel, it's extremely
unpleasant to work with many levels of nesting.

2. I thought no operation could be performed without cd-to-toplevel,
as even 'git submodule foo' complains about this (instead of saying
"command not found").  Having to go to each submodule subdirectory to
perform operations is just horrible.

3. To change a submodule, I'll have to propagate the change upwards,
creating new commits in each of the outer submodules, and the
superproject.

Apart from the implementation glitches, I don't like the design; submodules don't compose well:

1. There's an inherent asymmetry between the superproject and each of
the subprojects, because the superproject owns all the object stores.
Why is it absolutely necessary to relocate the object stores?

2. The metadata is tracked as a file (.gitmodules) in the git
repository.  While it makes it possible for different branches to have
different submodules, it's impossible to say 'git submodule sync
a/b/c/d' (see: 'repo sync'), where b, c haven't been initialized.

3. The current implementation only allows me to compose with commit
objects, but what if I want to compose with refs?  ie. What if I want
to track the tip of the 'master' of a submodule in a superproject?  I
don't want to commit-cascade everytime I make a minor change in a
deeply nested submodule.

Other solutions such as repo (see: depot_tools) solve different problems but are huge failures when it comes to composability; repo uses a central manifest repository, for example. Is it impossible to build a tool that can truly compose git repositories? I think submodules gets a lot of things Right, and we'd have to start from there.

Show 10 quoted lines

>> This is just two levels of nesting: with more levels of nesting,
>> things only get worse.
>
> Yes, you cannot have the cake and eat it. Either you incorporate
> everything into a single repo (e.g. using subtree) and loose the
> strong distinction which content belongs to which upstream repo
> (which AFAIK is a valid choice unless you want to contribute back
> to the submodule's upstream) or you'll have to cope with the
> submodule borders showing up from time to time, reminding you
> which part of the work tree has another upstream.

I don't think it's impossible. We just haven't thought hard enough about composition.