Managing submodules on large multi-user projects

> As some of you may recall from my last swath of emails to the list
> regarding memory usage and repository size, we have quite a large
> repository. About a month ago, I added a submodule to the primary repo
> in an effort to start to segment where possible, particularly around
> third party modules.
>
> I've noticed that keeping submodules updated is an absolute pain,
> particularly with a large multiuser setup with *lots* of branches.

> On Fri, May 29, 2009 at 2:41 PM, R. Tyler Ballance <tyler@slide.com> wrote:
> > As some of you may recall from my last swath of emails to the list
> > regarding memory usage and repository size, we have quite a large
> > repository. About a month ago, I added a submodule to the primary repo
> > in an effort to start to segment where possible, particularly around
> > third party modules.
> >
> > I've noticed that keeping submodules updated is an absolute pain,
> > particularly with a large multiuser setup with *lots* of branches.
> 
> Just so I understand, is the reason you're splitting into submodules
> *just* to avoid memory usage / repository size issues?  I can sort of
> understand the memory usage issues - sort of - but how does it reduce
> repository size if you need to need to check out all the submodule
> repositories along with the main project anyway?

> On Fri, May 29, 2009 at 03:53:26PM -0400, Avery Pennarun wrote:
>> Just so I understand, is the reason you're splitting into submodules
>> *just* to avoid memory usage / repository size issues?  I can sort of
>> understand the memory usage issues - sort of - but how does it reduce
>> repository size if you need to need to check out all the submodule
>> repositories along with the main project anyway?
>
> I've got an eye on submodules as a way of avoiding the need to require a
> whole tree clone to just work on parts of it, but that's not really
> relevant to my query, just explaining our environment and setting the stage ;)
>
> We're using submodules right now similar to how we used svn externals in
> the past (except better, clearly), to incorporate outside components
> (like open source projects) that our stack depends on.

>> Just looking to clarify for myself.  (I'm continuing my work on
>> git-subtree, which is getting more and more positive feedback.  It
>> solves all the *other* problems that you listed vs. submodules, but it
>> certainly doesn't resolve any repository size issues.)
>
> Good to know, we're still on Git 1.6.1, are there any benefits or
> additional features in more recent releases of Git that help alleviate
> the submodules issues I outlined at the top of the thread?

> As some of you may recall from my last swath of emails to the list
> regarding memory usage and repository size, we have quite a large
> repository. About a month ago, I added a submodule to the primary repo
> in an effort to start to segment where possible, particularly around
> third party modules.
>
> I've noticed that keeping submodules updated is an absolute pain,
> particularly with a large multiuser setup with *lots* of branches.
>
>
> What will tend to happen is that the submodule reference will be updated
> in the master branch (we use a centralized model) and then committed
> (imagine the commit reference was incremented from A-B).
>
> Other developers with other branches will then periodically merge master
> into their project/topic branches but will either neglect to run
> `git submodule update` or our bootstrap script (which also executes the
> submodule update command). At this point they'll have outstanding
> changes of their own, and the submodule will be marked as "modified" as
> well. Usually what will then happen is they'll `git commit -a` without
> thinking and the submodule's reference will be changed (typically from
> B->A, undoing the previous change).
>
>
> Are there any saner ways of managing this? I've been trying to get the
> `git submodule update` command to run with as many hooks as possible
> (pre-commit, post-update) to make sure that developers aren't
> inadvertantly breaking things, but nothing seems to ensure that
> *everybody* is up to date and that *everybody* doesn't inadvertantly
> commit changes to the submodule?

> On Fri, May 29, 2009 at 9:41 PM, R. Tyler Ballance <tyler@slide.com> wrote:
> > I've noticed that keeping submodules updated is an absolute pain,
> > particularly with a large multiuser setup with *lots* of branches.
> >
> >
> > What will tend to happen is that the submodule reference will be updated
> > in the master branch (we use a centralized model) and then committed
> > (imagine the commit reference was incremented from A-B).
> >
> > Other developers with other branches will then periodically merge master
> > into their project/topic branches but will either neglect to run
> > `git submodule update` or our bootstrap script (which also executes the
> > submodule update command). At this point they'll have outstanding
> > changes of their own, and the submodule will be marked as "modified" as
> > well. Usually what will then happen is they'll `git commit -a` without
> > thinking and the submodule's reference will be changed (typically from
> > B->A, undoing the previous change).
> >
> >
> > Are there any saner ways of managing this? I've been trying to get the
> > `git submodule update` command to run with as many hooks as possible
> > (pre-commit, post-update) to make sure that developers aren't
> > inadvertantly breaking things, but nothing seems to ensure that
> > *everybody* is up to date and that *everybody* doesn't inadvertantly
> > commit changes to the submodule?
> 
> Have you tried repo?
> http://source.android.com/download/using-repo

> Other developers with other branches will then periodically merge master
> into their project/topic branches but will either neglect to run
> `git submodule update` or our bootstrap script (which also executes the
> submodule update command). At this point they'll have outstanding
> changes of their own, and the submodule will be marked as "modified" as
> well. Usually what will then happen is they'll `git commit -a` without
> thinking and the submodule's reference will be changed (typically from
> B->A, undoing the previous change).