Re: [RFC] Submodules in GIT

> 
> On Fri, 1 Dec 2006, sf wrote:
>> Linus Torvalds wrote:
>> ...
>>> Think of it this way: one common use for submodules is really to just
>>> (occasionally) track somebody elses code. The submodule should be a
>>> totally pristine copy from somebody else (ie it might be the "intel driver
>>> for X.org" submodule, maintained within intel), and the supermodule just
>>> refers to it indirectly (ie the supermodule might be the "Fedora Core X
>>> group" which contains all the different drivers from different people).
>> Could you please be a little bit more specific about how you would store the
>> "pristine copy".
> 
> Note that it's not necessarily "pristine", since the submodule clearly is 
> a local git repository in its own right. So like _any_ git repository, you 
> can (and may well end up) having your own local branches in the submodule, 
> with your own local modifications.
> 
> So I'm not claiming that a submodule must always match some external git 
> tree 100%, and that it must be read-only or anything like that. I'm just 
> saying that I suspect that quite often, one of the MOST IMPORTANT parts is 
> that the submodule is really something that somebody else technically 
> maintains, and that this is actually one of the _reasons_ why it is a 
> submodule in the first place. 
> 
> For example, a lot of projects end up having some kind of "library 
> component" as a submodule. Take something like a video player project, 
> which would have something like ffmpeg as a submodule, not because you'd 
> maintain ffmpeg yourself, but simply because (let's say) the library 
> interface changes enough, or you need a specific version with some of your 
> own fixes that haven't been released widely yet, so you want to carry all 
> the libraries you need _with_ you, even though you don't really maintain 
> that submodule. You at most have some small extensions of your own.
> 
> Now, in this situation, it's relaly really _important_ that the submodule 
> really is totally independent of the supermodule, for several reasons.
> 
> For example, since you don't "really" own that project, carrying around 
> your own fixes is really really painful. We know it happens all the time, 
> and a lot of projects end up needing their own version, but the _last_ 
> thing you want is to be in merge hell all the time. So as a supermodule 
> maintainer, the best possible thing for you is to be able to push back 
> those local changes to the original project maintainer, so that you 
> _don't_ have to maintain your own changes.

> But you need to realize that the real maintainer of the submodule is 
> TOTALLY UNINTERESTED in your supermodule. He's not going to maintain it, 
> and in fact, if you have anything in the submodule that ends up talking 
> about your supermodule, that's just going to make it a lot less likely 
> that the upstream maintainer will ever pull your changes. He might take a 
> diff from you, but in a perfect world, you'd actually be able to tell him: 
> 
>  "Hey, I've got a git repository with a few fixes to your ffmpeg git tree, 
>   please pull from git://myhost.com/submodule.git to get these fixes:
> 
> 	... explanation of fixes and commits that are relevant to
> 	ffmpeg, and have nothing to do with the supermodule, except
> 	that you need those bug-fixes because you _use_ ffmpeg ...
> 
>   Thanks"
> 
> See?

>> There seems to be some agreement to store the commit id of
>> the submodule instead of a plain tree id in the supermodules tree object, and
>> that all objects that are reachable from this commit are made part of the
>> supermodule repository (either fetched or via alternates). Do you agree?
> 
> Well, I would actually argue that you may often want to have a supermodule 
> and then at least have the _option_ to decide to not fetch all the 
> submodules.

>> If I understand you correctly you cannot make any changes to the submodules
>> code _in the supermodule's repository_, no bugfixes, no extensions, no
>> adaptions, nothing. Do you mean that?
> 
> Yes. I think you should make all changes _within_ the submodule, because 
> the submodule should still be an independent git tree in its own right.

> And the importance of keeping the submodule independent is partly just 
> stability and sanity, but partly also scalability. For example, the 
> "index" in a supermodule should NOT include the indexes of all the 
> submodules. That's really important, because the index doesn't really 
> scale. Things do slow down with large indexes. 
> 
> For example, git can handle tens of thousands of files easily. I suspect 
> it scales well to hundreds of thousands of filenames. But with 
> supermodules, you really can end up in the situation where you have _tens_ 
> of these submodules, maybe even hundreds. And if you try to maintain one 
> unified index for the _whole_ thing, I guarantee you that you'll start 
> feeling the pain. Indexing millions of files is just not going to be 
> pretty.

> So just from a git stability and scalability point, it's important to keep 
> subprojects _separate_. There is obviously integration stuff, but they 
> should still be seen as truly independent projects. Even the supermodule 
> should have clearly its own life even _regardless_ of submodules, because 
> (as I said) quite often you may want the supermodule, but you don't want 
> to have _all_ of the submodules.
> 
> But it's more than that stability and scalability thing too - keeping them 
> separate is what allows you to do pulls and pushes on an individual 
> subproject basis, and have people really work at that level. For example, 
> if you're the compiler guy at a company, you really do want to work with 
> other compiler people _outside_ the company, but you sure as hell may not 
> be able to give them access to your supermodule. But you may want to work 
> on _just_ the compiler parts (or at least share some branches in public), 
> which means that the subproject really has to be able to work 
> _independently_ of the supermodule.

> So "independent" here is really key, for several reasons. And that all 
> means, for example, that here must NEVER be any "backpointers". A 
> subproject really can _never_ have backpointers to the superproject, 
> because that fundamentally means that the above kind of "compiler guy 
> works on the compiler subproject in public" cannot work, if your 
> supermodule isn't public.