threads / discuss / 27794

Nesting a submodule inside of another...

Subject: Nesting a submodule inside of another...

## tl;dr

11 messages between Jul 12, 2011 and Jul 15, 2011.

replies: 10people: 5as markdown or json

John Szakmeister· Jul 12, 2011, 12:01 UTC · lore
Hi all,
I've got a project where we have several frameworks involved, and
external modules we want to pull into the framework tree.  We'd like
to make use of submodules and have something like this:
    top-level/            <-- .gitmodules lives here
        src/
        framework1/   <-- a submodule
            module/     <-- another submodule
        framework2/   <-- a submodule
            module2/    <-- another submodule

Currently, git fails trying to do this. It's not happy about .gitmodules living at the top-level and nesting a submodule inside of another[1]. Is there a technical reason that this is not allowed? Limiting the traversal up the tree, etc.? I've worked around the lack of support to do this for now, but it would be really nice if we could do such a thing.

Thanks!
-John
[1]  I may generate a patch for the error message, as the current
implementation obscures the real error message.  When I try this, git
says:
    The following path is ignored by one of your .gitignore files:
    framework1/module
    Use -f if you really want to add it.
    After some careful debugging, I found out the real error message
was about this sort of nesting being disallowed.
Jens Lehmann· Jul 12, 2011, 19:23 UTC · re: John Szakmeister · lore

Re: Nesting a submodule inside of another...

Am 12.07.2011 14:01, schrieb John Szakmeister:
Show 13 quoted lines
> I've got a project where we have several frameworks involved, and
> external modules we want to pull into the framework tree.  We'd like
> to make use of submodules and have something like this:
>     top-level/            <-- .gitmodules lives here
>         src/
>         framework1/   <-- a submodule
>             module/     <-- another submodule
>         framework2/   <-- a submodule
>             module2/    <-- another submodule
> 
> Currently, git fails trying to do this.  It's not happy about
> .gitmodules living at the top-level and nesting a submodule inside of
> another[1].  Is there a technical reason that this is not allowed?

Submodules are repositories of their own, so their work tree is outside the superproject's work tree. You're attempting to have a submodule of the superproject inside another submodule of the superproject, but this won't work because it does not live in the superproject's work tree (as that doesn't include submodules and their subdirectories).

To have nested submodules you must add "module" as a submodule of "framework1" and "module2" as a submodule of "framework2". Then git will handle them just fine (assuming you use the --recursive option for "git submodule update" and friends). Of course that means that if you want to record a new commit of a sub-submodule in the superpoject, you'll have to commit that in it's parent submodule first and then can record that commit in the superproject. (But if "framework1" wouldn't depend on a specific version of "module", there would be no reason to put it inside it in the first place, right?)

> Limiting the traversal up the tree, etc.?  I've worked around the lack
> of support to do this for now, but it would be really nice if we could
> do such a thing.
I can't think of a sane way to make that work.
Show 9 quoted lines
> [1]  I may generate a patch for the error message, as the current
> implementation obscures the real error message.  When I try this, git
> says:
>     The following path is ignored by one of your .gitignore files:
>     framework1/module
>     Use -f if you really want to add it.
> 
>     After some careful debugging, I found out the real error message
> was about this sort of nesting being disallowed.

But at least since 1.6.2 git should issue a meaningful error message. With current master I get:

$ git add sub/file fatal: Path 'sub/file' is in submodule 'sub'

What version are you using?
John Szakmeister· Jul 13, 2011, 01:21 UTC · re: Jens Lehmann · lore

Re: Nesting a submodule inside of another...

On Tue, Jul 12, 2011 at 3:23 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
Show 20 quoted lines
> Am 12.07.2011 14:01, schrieb John Szakmeister:
>> I've got a project where we have several frameworks involved, and
>> external modules we want to pull into the framework tree.  We'd like
>> to make use of submodules and have something like this:
>>     top-level/            <-- .gitmodules lives here
>>         src/
>>         framework1/   <-- a submodule
>>             module/     <-- another submodule
>>         framework2/   <-- a submodule
>>             module2/    <-- another submodule
>>
>> Currently, git fails trying to do this.  It's not happy about
>> .gitmodules living at the top-level and nesting a submodule inside of
>> another[1].  Is there a technical reason that this is not allowed?
>
> Submodules are repositories of their own, so their work tree is outside
> the superproject's work tree. You're attempting to have a submodule of
> the superproject inside another submodule of the superproject, but this
> won't work because it does not live in the superproject's work tree (as
> that doesn't include submodules and their subdirectories).
I guess I still don't see how that's a problem.  It seems to me that
for the superproject the ignore rule needs to be:
  ignore framework1, except framework1/module

And then it could check its status. Of course, framework1 would somehow need to know to ignore it... and I'm not sure how that can be communicated.

Show 9 quoted lines
> To have nested submodules you must add "module" as a submodule of
> "framework1" and "module2" as a submodule of "framework2". Then git
> will handle them just fine (assuming you use the --recursive option for
> "git submodule update" and friends). Of course that means that if you
> want to record a new commit of a sub-submodule in the superpoject, you'll
> have to commit that in it's parent submodule first and then can record
> that commit in the superproject. (But if "framework1" wouldn't depend on
> a specific version of "module", there would be no reason to put it inside
> it in the first place, right?)

We can't do that. The base framework should not include the module in its repository. We drop in different modules that get shared across some projects, but not others. They get mixed and matched. This was pretty straight-forward to do with Subversion. Admittedly, I don't like the layout of it... but unfortunately, I'm not really in control of that either.

Show 5 quoted lines
>> Limiting the traversal up the tree, etc.?  I've worked around the lack
>> of support to do this for now, but it would be really nice if we could
>> do such a thing.
>
> I can't think of a sane way to make that work.
:-(
[snip]
Show 5 quoted lines
> But at least since 1.6.2 git should issue a meaningful error message.
> With current master I get:
>
> $ git add sub/file
> fatal: Path 'sub/file' is in submodule 'sub'

Sorry, I wasn't trying to do a 'git add'. I was trying to do a 'git submodule add'.

> What version are you using?
I'm using 1.7.6rc3.  I updated to the latest trunk and still get the same error:

:: git submodule add $PWD/../c b/c The following path is ignored by one of your .gitignore files: b/c Use -f if you really want to add it.

HTH.
-John
Jens Lehmann· Jul 13, 2011, 21:20 UTC · re: John Szakmeister · lore

Re: Nesting a submodule inside of another...

Am 13.07.2011 03:21, schrieb John Szakmeister:
Show 10 quoted lines
> On Tue, Jul 12, 2011 at 3:23 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Submodules are repositories of their own, so their work tree is outside
>> the superproject's work tree. You're attempting to have a submodule of
>> the superproject inside another submodule of the superproject, but this
>> won't work because it does not live in the superproject's work tree (as
>> that doesn't include submodules and their subdirectories).
> 
> I guess I still don't see how that's a problem.  It seems to me that
> for the superproject the ignore rule needs to be:
>   ignore framework1, except framework1/module

But that would mean the superproject would have to scan framework1's work tree for changes too, which is conflicting with the idea that each submodule is a repo of it's own, so that won't fly.

> And then it could check its status.  Of course, framework1 would
> somehow need to know to ignore it... and I'm not sure how that can be
> communicated.

It can't be communicated, as submodules are unaware of their superproject. And adding module manually to framework1's .gitignore is just a problem waiting to happen. Having a file or submodule in the work tree of one git repository being referenced by another is just asking for trouble ...

Show 8 quoted lines
>> But at least since 1.6.2 git should issue a meaningful error message.
>> With current master I get:
>>
>> $ git add sub/file
>> fatal: Path 'sub/file' is in submodule 'sub'
> 
> Sorry, I wasn't trying to do a 'git add'.  I was trying to do a 'git
> submodule add'.
Ok.
Show 8 quoted lines
>> What version are you using?
> 
> I'm using 1.7.6rc3.  I updated to the latest trunk and still get the same error:
> 
> :: git submodule add $PWD/../c b/c
> The following path is ignored by one of your .gitignore files:
> b/c
> Use -f if you really want to add it.

You're right, the error message of "git submodule add" should be better here.

Junio C Hamano· Jul 13, 2011, 21:27 UTC · re: Jens Lehmann · lore

Re: Nesting a submodule inside of another...

Jens Lehmann <Jens.Lehmann@web.de> writes:
> It can't be communicated, as submodules are unaware of their superproject.
It's more like s/can't/shouldn't/ and s/are/shouldn't be/, no?
Jens Lehmann· Jul 13, 2011, 22:04 UTC · re: Junio C Hamano · lore

Re: Nesting a submodule inside of another...

Am 13.07.2011 23:27, schrieb Junio C Hamano:
Show 5 quoted lines
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> It can't be communicated, as submodules are unaware of their superproject.
> 
> It's more like s/can't/shouldn't/ and s/are/shouldn't be/, no?

Right. Submodules /could/ be taught to check if they are part of a superproject, but unless I miss something obvious they are currently totally unaware of that (at least when it concerns their work tree, the gitfile effort is trying to introduce some superproject awareness). And as far as I can see that is a sound design decision as far as it concerns the work tree. Imagine the files and/or directories a submodule has to ignore would depend on the settings of their superproject too (which is what this thread is about). That would lead to very interesting problems when the same submodule would be used by different superprojects which ignore different files and/or directories inside the submodule, which - to make things even more interesting - might be recorded in the submodule itself ... not good.

So what about: s/can't/isn't and shouldn't/ and s/are/are and shouldn't/?
Junio C Hamano· Jul 13, 2011, 22:44 UTC · re: Jens Lehmann · lore

Re: Nesting a submodule inside of another...

Jens Lehmann <Jens.Lehmann@web.de> writes:
> ... (at least when it concerns their work tree, the
> gitfile effort is trying to introduce some superproject awareness).
I don't think so.

The "gitfile effort" just makes it more convenient and potentially safer for superproject to remove submodule working tree by running an equivalent of "rm -fr submodule". The submodule shouldn't even care where its .git metadirectory is by looking at the "gitfile: " line, nor should it assume that location its .git metadirectory moved out of its working tree has some fixed relationship with its supermodule's repository. E.g. the superproject could choose to store it under $HOME/cache or /var/state as long as it knows where to point it back to after it runs "rm -rf submodule" and then it has to re-instantiate the submodule.

IOW, the superproject must be aware of that location, but submodule shouldn't depend on what convention the superproject chose to use. After all, there may not even be a superproject when gitfile is in use, so running "test -f .git" and assuming that it is used as a submodule of somebody else is already a bogus test.

John Szakmeister· Jul 14, 2011, 09:36 UTC · re: Jens Lehmann · lore

Re: Nesting a submodule inside of another...

On Wed, Jul 13, 2011 at 5:20 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 13.07.2011 03:21, schrieb John Szakmeister:
[snip]
Show 7 quoted lines
>> I guess I still don't see how that's a problem.  It seems to me that
>> for the superproject the ignore rule needs to be:
>>   ignore framework1, except framework1/module
>
> But that would mean the superproject would have to scan framework1's
> work tree for changes too, which is conflicting with the idea that each
> submodule is a repo of it's own, so that won't fly.

I guess I just don't see that. Even though the path is inside of a submodules, it would be part of the superproject's work tree. But, I can see how it's a violation of a submodule being a repo of it's own. And I certainly understand the desire to maintain that quality.

Show 8 quoted lines
>> And then it could check its status.  Of course, framework1 would
>> somehow need to know to ignore it... and I'm not sure how that can be
>> communicated.
>
> It can't be communicated, as submodules are unaware of their superproject.
> And adding module manually to framework1's .gitignore is just a problem
> waiting to happen. Having a file or submodule in the work tree of one
> git repository being referenced by another is just asking for trouble ...

Again, I don't see how the submodule needs to be aware of the superproject. In this case, it'd be the responsibility of the superproject to setup whatever is necessary at 'git submodule init/add'. I don't see how the submodule *must* know about the superproject for it to succeed. I see the opposite, the superproject needs to communicate some information down to the submodule, but I don't see the reverse.

What I'm hearing is that while it may be possible, the idea of violating the concept that the "subrepo is standalone" is unacceptable. Which means, unfortunately, git isn't a solution for us in this case.

[snip]
Show 9 quoted lines
>> I'm using 1.7.6rc3.  I updated to the latest trunk and still get the same error:
>>
>> :: git submodule add $PWD/../c b/c
>> The following path is ignored by one of your .gitignore files:
>> b/c
>> Use -f if you really want to add it.
>
> You're right, the error message of "git submodule add" should be better
> here.
I'll write a patch for this.  Thanks.

And thanks for the input. I was just looking to see if there were any technical/morale reasons that submodules weren't allowed in this way. I appreciate you all taking the time to answer.

-John
Seth Robertson· Jul 14, 2011, 14:33 UTC · re: John Szakmeister · lore

Re: Nesting a submodule inside of another...

In message <CAEBDL5VUPE9YCX1C4pqkjb+EODkAWo9h774B=Jv5eUNbocMuZQ@mail.gmail.com>, John Szakmeister writes:
    > I've got a project where we have several frameworks involved, and
    > external modules we want to pull into the framework tree.  We'd like
    > to make use of submodules and have something like this:
    >      top-level/<-- .gitmodules lives here
    >          src/
    >          framework1/<-- a submodule
    >              module/<-- another submodule
    >          framework2/<-- a submodule
    >              module2/<-- another submodule
    Again, I don't see how the submodule needs to be aware of the
    superproject.  In this case, it'd be the responsibility of the
    superproject to setup whatever is necessary at 'git submodule
    init/add'.  I don't see how the submodule *must* know about the
    superproject for it to succeed.  I see the opposite, the superproject
    needs to communicate some information down to the submodule, but I
    don't see the reverse.
    What I'm hearing is that while it may be possible, the idea of
    violating the concept that the "subrepo is standalone" is
    unacceptable.  Which means, unfortunately, git isn't a solution for us
    in this case.

You might find that gitslave (http://gitslave.sf.net) might be a better solution for you than git-submodules in this case. It works better for many workflows (and worse for others) but is much simpler to understand since with gitslave you have JBOR (just a bunch of repositories) with a program which can be thought of as running the requested git command over each repository in turn. Gitslave thus has a loose binding between the repositories, and you can only guarantee the relationship between repositories at tagged locations, though in practice this isn't a major concern.

gitslave supports nested repositories (and recursive gitslave repositories, but those are different). With gitslave nested repositories it is also true that you would have to have a supplemental gitignore entry in framework1 (which gitslave will create).

If you have any questions, please let me know.
					-Seth Robertson
John Szakmeister· Jul 15, 2011, 08:34 UTC · re: Seth Robertson · lore

Re: Nesting a submodule inside of another...

On Thu, Jul 14, 2011 at 10:33 AM, Seth Robertson <in-gitvger@baka.org> wrote: [snip]

Show 22 quoted lines
>    Again, I don't see how the submodule needs to be aware of the
>    superproject.  In this case, it'd be the responsibility of the
>    superproject to setup whatever is necessary at 'git submodule
>    init/add'.  I don't see how the submodule *must* know about the
>    superproject for it to succeed.  I see the opposite, the superproject
>    needs to communicate some information down to the submodule, but I
>    don't see the reverse.
>
>    What I'm hearing is that while it may be possible, the idea of
>    violating the concept that the "subrepo is standalone" is
>    unacceptable.  Which means, unfortunately, git isn't a solution for us
>    in this case.
>
> You might find that gitslave (http://gitslave.sf.net) might be a
> better solution for you than git-submodules in this case.  It works
> better for many workflows (and worse for others) but is much simpler
> to understand since with gitslave you have JBOR (just a bunch of
> repositories) with a program which can be thought of as running the
> requested git command over each repository in turn.  Gitslave thus has
> a loose binding between the repositories, and you can only guarantee
> the relationship between repositories at tagged locations, though in
> practice this isn't a major concern.
Interesting.  I'll take a look!
Show 7 quoted lines
> gitslave supports nested repositories (and recursive gitslave
> repositories, but those are different).  With gitslave nested
> repositories it is also true that you would have to have a
> supplemental gitignore entry in framework1 (which gitslave will
> create).
>
> If you have any questions, please let me know.
I appreciate it.  Thanks Seth!
-John
Andreas Ericsson· Jul 13, 2011, 08:43 UTC · re: John Szakmeister · lore

Re: Nesting a submodule inside of another...

On 07/12/2011 02:01 PM, John Szakmeister wrote:
Show 15 quoted lines
> Hi all,
> 
> I've got a project where we have several frameworks involved, and
> external modules we want to pull into the framework tree.  We'd like
> to make use of submodules and have something like this:
>      top-level/<-- .gitmodules lives here
>          src/
>          framework1/<-- a submodule
>              module/<-- another submodule
>          framework2/<-- a submodule
>              module2/<-- another submodule
> 
> Currently, git fails trying to do this.  It's not happy about
> .gitmodules living at the top-level and nesting a submodule inside of
> another[1].  Is there a technical reason that this is not allowed?

Yes. Everything inside a submodule is owned by that submodule, so the master repo can't know anything about it. You can have a submodule which in turn has submodules though. If you couldn't, it wouldn't be possible to checkout only the framework1 repository and get all its dependencies.

I have no idea what problems you run into with more than 2 tiers of submodules though, but I guess that's for you to find out and report about.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

← back to recent threads