Re: How to manage multiple repos using submodules?

From: Phil Hord <hordp@cisco.com>
Date: Apr 19, 2011, 13:18 UTC
Message-ID: <4DAD8B93.1040707@cisco.com>
In-Reply-To: <20110416182053.GA11017@elie>

Thanks, Jonathan.

On 04/16/2011 02:20 PM, Jonathan Nieder wrote:

Show 14 quoted lines

> Hi Andrew,
>
> Andrew Wong wrote:
>
>> It seems like submodule isn't meant for this, but many people seems
>> to use submodule to link many smaller repos together. With this
>> setup, I imagine whenever someone pushed a small repo, they're
>> /supposed/ to push the big repo as well. This way, if I simply
>> update the big repo and do a "git status", git will tell me that
>> which of the smaller repos are out of date.
> Yep, if you want to keep track of the state of a bunch of repos over
> time, submodules are not so bad[*].  In practice, often one instead
> wants to keep a bunch of repos up-to-date, and all this meta-history
> tracking is overkill.

I had been thinking about this a lot lately. We are using submodules somewhat extensively to manage a project with lots of common code feeding onto 12 different platforms (various OSes, compilers and SOCs). It all feels quiet unwieldy but, at the same time, necessary.

I looked briefly at the android repo tool, but it's not just where I want it to be either. Also, it loses history (by design, though that seems to be a mistake they intend to correct someday).

But all this thinking got me closer to understanding that there are two
competing needs here, as you both pointed out:
  1. Point-in-time history using SHA-1s (the git model)
  2. Branch following (the 'repo', mr, gitslave or svn model)

When I am looking for a bug introduced in the past, I want to have the first model. When I am eager to pull in bug fixes, new features, or a different meta-branch from other developers (and possibly other projects), I want the second model.

Submodules make it difficult to manage both at the same time. It requires extra steps to do what it should be doing by default (acting like git, tracking project point-in-time history). I understand why it is like this, but empathy doesn't ease the pain.

But I really do want both models. I want point-in-time history, and I want semi-automatic branch following. Maybe something like this:

1. Easily record point-in-time history
   # Create and switch to a branch on superproject and all submodules
   git checkout -b topic-foo --recursive

   submodule A/topic-foo:  hack, hack, hack
   submodule B/topic-foo:  hack, hack, hack

   # Commit and push changes on each submodule
   git commit --recursive
   git push origin HEAD:ph/foo --recursive

2. Ignore the gitlinks and just checkout master everywhere
   git checkout master --recursive

I think I can do both with just a few tweaks of the submodule machinery. I haven't had time to spelunk there much, but I see others are working to advance submodules in somewhat similar directions. (Thanks for that; it is much appreciated.)

Have these dueling modes been discussed into the dirt somewhere already? I find lots of discussions about other tools to perform model 2 instead of model 1, but I don't see much discussion about the simultaneous need for both. Maybe I am overthinking it. Or maybe my particular code model is unusual.

Show 5 quoted lines

> I'd suggest using the mr tool.  Some projects (e.g., the
> debian-installer project) are using it to help people keep all their
> repos up to date.
>
> http://kitenet.net/~joey/code/mr/

Thanks.  I hadn't seen that one before.  I'll take a look.

Phil