How to manage multiple repos using submodules?

It seems like submodule isn't meant for this, but many people seems to use submodule to link many smaller repos together. With this setup, I imagine whenever someone pushed a small repo, they're /supposed/ to push the big repo as well. This way, if I simply update the big repo and do a "git status", git will tell me that which of the smaller repos are out of date. However, this is only reliable if everyone remembers to push the big repo. If someone pushed a smaller repo, but forgot to push the big repo, then I won't be aware that some of the smaller repos are out-of-date until I push.

I suppose one way is to do a check/auto-update with pre/post commit script to enforce that the big repo is always up-to-date. Another way is to use the "submodule foreach" to do a fetch and status on smaller ones, but this doesn't seem very elegant.

So, I'm wondering how do people who use submodules this way manage all their repos? Or how to (reliably) get a good sense of the state of the small repos? Or maybe I shouldn't be using submodules this way at all?

Hi Andrew,

Andrew Wong wrote:

Yep, if you want to keep track of the state of a bunch of repos over time, submodules are not so bad[*]. In practice, often one instead wants to keep a bunch of repos up-to-date, and all this meta-history tracking is overkill.

I'd suggest using the mr tool. Some projects (e.g., the debian-installer project) are using it to help people keep all their repos up to date.

http://kitenet.net/~joey/code/mr/

Hope that helps, Jonathan

Hi,

Jonathan Nieder wrote:

A kind person pointed out that I left out a footnote. I think all I had been planning to say is that, roughly speaking, submodules are about[1] saying that a specific commit is known to work well with the rest of the code. A supermodule like the one discussed in [2] is only likely to be useful if you are interested in what historical combinations of repositories were published and meant to work well together.

Ciao, Jonathan

[1] e.g., http://thread.gmane.org/gmane.comp.version-control.git/27803/focus=27830 [2] http://lists.x.org/archives/xorg-devel/2009-September/001966.html

Thanks for the mail archives! They were a very good read.

The smaller projects I have are fairly independent, so there isn't a situation where a commit works well with the other commits. I just wanted some ways to split each project out to its own repo. So that when I want to do some git operations on one project, I don't have to worry about the other projects. While submodules isn't an ideal solution, it seems to be the closest. Maybe what I need from submodules is a way for the super-repo to not record the commit of the sub-repos. i.e. Just use the head of a branch. But if that's the case, maybe it's out of the scope of a SCM, since I'm not really tracking a history anymore.

I haven't tried it yet, but the mr tool you mentioned seems interesting too. I'll check it out.

Thanks! Andrew

On 11-04-17 2:48 AM, Jonathan Nieder wrote:

Thanks, Jonathan.

On 04/16/2011 02:20 PM, Jonathan Nieder wrote:

I had been thinking about this a lot lately. We are using submodules somewhat extensively to manage a project with lots of common code feeding onto 12 different platforms (various OSes, compilers and SOCs). It all feels quiet unwieldy but, at the same time, necessary.

I looked briefly at the android repo tool, but it's not just where I want it to be either. Also, it loses history (by design, though that seems to be a mistake they intend to correct someday).

But all this thinking got me closer to understanding that there are two
competing needs here, as you both pointed out:
  1. Point-in-time history using SHA-1s (the git model)
  2. Branch following (the 'repo', mr, gitslave or svn model)

When I am looking for a bug introduced in the past, I want to have the first model. When I am eager to pull in bug fixes, new features, or a different meta-branch from other developers (and possibly other projects), I want the second model.

Submodules make it difficult to manage both at the same time. It requires extra steps to do what it should be doing by default (acting like git, tracking project point-in-time history). I understand why it is like this, but empathy doesn't ease the pain.

But I really do want both models. I want point-in-time history, and I want semi-automatic branch following. Maybe something like this:

1. Easily record point-in-time history
   # Create and switch to a branch on superproject and all submodules
   git checkout -b topic-foo --recursive

   submodule A/topic-foo:  hack, hack, hack
   submodule B/topic-foo:  hack, hack, hack

   # Commit and push changes on each submodule
   git commit --recursive
   git push origin HEAD:ph/foo --recursive

2. Ignore the gitlinks and just checkout master everywhere
   git checkout master --recursive
  

I think I can do both with just a few tweaks of the submodule machinery. I haven't had time to spelunk there much, but I see others are working to advance submodules in somewhat similar directions. (Thanks for that; it is much appreciated.)

Have these dueling modes been discussed into the dirt somewhere already? I find lots of discussions about other tools to perform model 2 instead of model 1, but I don't see much discussion about the simultaneous need for both. Maybe I am overthinking it. Or maybe my particular code model is unusual.

Thanks.  I hadn't seen that one before.  I'll take a look.

Phil