{"thread":{"id":"27794","subject":"Nesting a submodule inside of another...","startedAt":"2011-07-12T12:01:09Z","lastAt":"2011-07-15T08:34:46Z","messageCount":11,"participants":["John Szakmeister","Jens Lehmann","Andreas Ericsson","Junio C Hamano","Seth Robertson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"171147","messageId":"CAEBDL5XQDehUyqKoazxy+YVHh_2iAyt9aE-77H1jZg4oujMJgw@mail.gmail.com","threadId":"27794","inReplyTo":null,"subject":"Nesting a submodule inside of another...","fromName":"John Szakmeister","fromEmail":"john@szakmeister.net","sentAt":"2011-07-12T12:01:09Z","receivedAt":"2011-07-12T12:01:09Z","isPatch":false,"sender":{"key":"john@szakmeister.net","avatar":"https://avatars.githubusercontent.com/u/448087?v=4"},"body":"Hi all,\n\nI've got a project where we have several frameworks involved, and\nexternal modules we want to pull into the framework tree.  We'd like\nto make use of submodules and have something like this:\n    top-level/            <-- .gitmodules lives here\n        src/\n        framework1/   <-- a submodule\n            module/     <-- another submodule\n        framework2/   <-- a submodule\n            module2/    <-- another submodule\n\nCurrently, git fails trying to do this.  It's not happy about\n.gitmodules living at the top-level and nesting a submodule inside of\nanother[1].  Is there a technical reason that this is not allowed?\nLimiting the traversal up the tree, etc.?  I've worked around the lack\nof support to do this for now, but it would be really nice if we could\ndo such a thing.\n\nThanks!\n\n-John\n\n[1]  I may generate a patch for the error message, as the current\nimplementation obscures the real error message.  When I try this, git\nsays:\n    The following path is ignored by one of your .gitignore files:\n    framework1/module\n    Use -f if you really want to add it.\n\n    After some careful debugging, I found out the real error message\nwas about this sort of nesting being disallowed.\n"},{"id":"171177","messageId":"4E1C9F21.6070300@web.de","threadId":"27794","inReplyTo":"CAEBDL5XQDehUyqKoazxy+YVHh_2iAyt9aE-77H1jZg4oujMJgw@mail.gmail.com","subject":"Re: Nesting a submodule inside of another...","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2011-07-12T19:23:13Z","receivedAt":"2011-07-12T19:23:13Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 12.07.2011 14:01, schrieb John Szakmeister:\n> I've got a project where we have several frameworks involved, and\n> external modules we want to pull into the framework tree.  We'd like\n> to make use of submodules and have something like this:\n>     top-level/            <-- .gitmodules lives here\n>         src/\n>         framework1/   <-- a submodule\n>             module/     <-- another submodule\n>         framework2/   <-- a submodule\n>             module2/    <-- another submodule\n> \n> Currently, git fails trying to do this.  It's not happy about\n> .gitmodules living at the top-level and nesting a submodule inside of\n> another[1].  Is there a technical reason that this is not allowed?\n\nSubmodules are repositories of their own, so their work tree is outside\nthe superproject's work tree. You're attempting to have a submodule of\nthe superproject inside another submodule of the superproject, but this\nwon't work because it does not live in the superproject's work tree (as\nthat doesn't include submodules and their subdirectories).\n\nTo have nested submodules you must add \"module\" as a submodule of\n\"framework1\" and \"module2\" as a submodule of \"framework2\". Then git\nwill handle them just fine (assuming you use the --recursive option for\n\"git submodule update\" and friends). Of course that means that if you\nwant to record a new commit of a sub-submodule in the superpoject, you'll\nhave to commit that in it's parent submodule first and then can record\nthat commit in the superproject. (But if \"framework1\" wouldn't depend on\na specific version of \"module\", there would be no reason to put it inside\nit in the first place, right?)\n\n> Limiting the traversal up the tree, etc.?  I've worked around the lack\n> of support to do this for now, but it would be really nice if we could\n> do such a thing.\n\nI can't think of a sane way to make that work.\n\n> [1]  I may generate a patch for the error message, as the current\n> implementation obscures the real error message.  When I try this, git\n> says:\n>     The following path is ignored by one of your .gitignore files:\n>     framework1/module\n>     Use -f if you really want to add it.\n> \n>     After some careful debugging, I found out the real error message\n> was about this sort of nesting being disallowed.\n\nBut at least since 1.6.2 git should issue a meaningful error message.\nWith current master I get:\n\n$ git add sub/file\nfatal: Path 'sub/file' is in submodule 'sub'\n\nWhat version are you using?\n"},{"id":"171197","messageId":"CAEBDL5U0F+QaqhW92i-s82-C9fj2knp6JPNtNvgdJY68kRYwWQ@mail.gmail.com","threadId":"27794","inReplyTo":"4E1C9F21.6070300@web.de","subject":"Re: Nesting a submodule inside of another...","fromName":"John Szakmeister","fromEmail":"john@szakmeister.net","sentAt":"2011-07-13T01:21:04Z","receivedAt":"2011-07-13T01:21:04Z","isPatch":false,"sender":{"key":"john@szakmeister.net","avatar":"https://avatars.githubusercontent.com/u/448087?v=4"},"body":"On Tue, Jul 12, 2011 at 3:23 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:\n> Am 12.07.2011 14:01, schrieb John Szakmeister:\n>> I've got a project where we have several frameworks involved, and\n>> external modules we want to pull into the framework tree.  We'd like\n>> to make use of submodules and have something like this:\n>>     top-level/            <-- .gitmodules lives here\n>>         src/\n>>         framework1/   <-- a submodule\n>>             module/     <-- another submodule\n>>         framework2/   <-- a submodule\n>>             module2/    <-- another submodule\n>>\n>> Currently, git fails trying to do this.  It's not happy about\n>> .gitmodules living at the top-level and nesting a submodule inside of\n>> another[1].  Is there a technical reason that this is not allowed?\n>\n> Submodules are repositories of their own, so their work tree is outside\n> the superproject's work tree. You're attempting to have a submodule of\n> the superproject inside another submodule of the superproject, but this\n> won't work because it does not live in the superproject's work tree (as\n> that doesn't include submodules and their subdirectories).\n\nI guess I still don't see how that's a problem.  It seems to me that\nfor the superproject the ignore rule needs to be:\n  ignore framework1, except framework1/module\n\nAnd then it could check its status.  Of course, framework1 would\nsomehow need to know to ignore it... and I'm not sure how that can be\ncommunicated.\n\n> To have nested submodules you must add \"module\" as a submodule of\n> \"framework1\" and \"module2\" as a submodule of \"framework2\". Then git\n> will handle them just fine (assuming you use the --recursive option for\n> \"git submodule update\" and friends). Of course that means that if you\n> want to record a new commit of a sub-submodule in the superpoject, you'll\n> have to commit that in it's parent submodule first and then can record\n> that commit in the superproject. (But if \"framework1\" wouldn't depend on\n> a specific version of \"module\", there would be no reason to put it inside\n> it in the first place, right?)\n\nWe can't do that.  The base framework should not include the module in\nits repository.  We drop in different modules that get shared across\nsome projects, but not others.  They get mixed and matched.  This was\npretty straight-forward to do with Subversion.  Admittedly, I don't\nlike the layout of it... but unfortunately, I'm not really in control\nof that either.\n\n>> Limiting the traversal up the tree, etc.?  I've worked around the lack\n>> of support to do this for now, but it would be really nice if we could\n>> do such a thing.\n>\n> I can't think of a sane way to make that work.\n\n:-(\n\n[snip]\n> But at least since 1.6.2 git should issue a meaningful error message.\n> With current master I get:\n>\n> $ git add sub/file\n> fatal: Path 'sub/file' is in submodule 'sub'\n\nSorry, I wasn't trying to do a 'git add'.  I was trying to do a 'git\nsubmodule add'.\n\n> What version are you using?\n\nI'm using 1.7.6rc3.  I updated to the latest trunk and still get the same error:\n\n:: git submodule add $PWD/../c b/c\nThe following path is ignored by one of your .gitignore files:\nb/c\nUse -f if you really want to add it.\n\nHTH.\n\n-John\n"},{"id":"171223","messageId":"4E1D5AC3.5050409@op5.se","threadId":"27794","inReplyTo":"CAEBDL5XQDehUyqKoazxy+YVHh_2iAyt9aE-77H1jZg4oujMJgw@mail.gmail.com","subject":"Re: Nesting a submodule inside of another...","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2011-07-13T08:43:47Z","receivedAt":"2011-07-13T08:43:47Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"On 07/12/2011 02:01 PM, John Szakmeister wrote:\n> Hi all,\n> \n> I've got a project where we have several frameworks involved, and\n> external modules we want to pull into the framework tree.  We'd like\n> to make use of submodules and have something like this:\n>      top-level/<-- .gitmodules lives here\n>          src/\n>          framework1/<-- a submodule\n>              module/<-- another submodule\n>          framework2/<-- a submodule\n>              module2/<-- another submodule\n> \n> Currently, git fails trying to do this.  It's not happy about\n> .gitmodules living at the top-level and nesting a submodule inside of\n> another[1].  Is there a technical reason that this is not allowed?\n\nYes. Everything inside a submodule is owned by that submodule, so the\nmaster repo can't know anything about it. You can have a submodule\nwhich in turn has submodules though. If you couldn't, it wouldn't be\npossible to checkout only the framework1 repository and get all its\ndependencies.\n\nI have no idea what problems you run into with more than 2 tiers of\nsubmodules though, but I guess that's for you to find out and report\nabout.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"171292","messageId":"4E1E0C27.60903@web.de","threadId":"27794","inReplyTo":"CAEBDL5U0F+QaqhW92i-s82-C9fj2knp6JPNtNvgdJY68kRYwWQ@mail.gmail.com","subject":"Re: Nesting a submodule inside of another...","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2011-07-13T21:20:39Z","receivedAt":"2011-07-13T21:20:39Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 13.07.2011 03:21, schrieb John Szakmeister:\n> On Tue, Jul 12, 2011 at 3:23 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:\n>> Submodules are repositories of their own, so their work tree is outside\n>> the superproject's work tree. You're attempting to have a submodule of\n>> the superproject inside another submodule of the superproject, but this\n>> won't work because it does not live in the superproject's work tree (as\n>> that doesn't include submodules and their subdirectories).\n> \n> I guess I still don't see how that's a problem.  It seems to me that\n> for the superproject the ignore rule needs to be:\n>   ignore framework1, except framework1/module\n\nBut that would mean the superproject would have to scan framework1's\nwork tree for changes too, which is conflicting with the idea that each\nsubmodule is a repo of it's own, so that won't fly.\n\n> And then it could check its status.  Of course, framework1 would\n> somehow need to know to ignore it... and I'm not sure how that can be\n> communicated.\n\nIt can't be communicated, as submodules are unaware of their superproject.\nAnd adding module manually to framework1's .gitignore is just a problem\nwaiting to happen. Having a file or submodule in the work tree of one\ngit repository being referenced by another is just asking for trouble ...\n\n>> But at least since 1.6.2 git should issue a meaningful error message.\n>> With current master I get:\n>>\n>> $ git add sub/file\n>> fatal: Path 'sub/file' is in submodule 'sub'\n> \n> Sorry, I wasn't trying to do a 'git add'.  I was trying to do a 'git\n> submodule add'.\n\nOk.\n\n>> What version are you using?\n> \n> I'm using 1.7.6rc3.  I updated to the latest trunk and still get the same error:\n> \n> :: git submodule add $PWD/../c b/c\n> The following path is ignored by one of your .gitignore files:\n> b/c\n> Use -f if you really want to add it.\n\nYou're right, the error message of \"git submodule add\" should be better\nhere.\n"},{"id":"171293","messageId":"7vhb6p6fcl.fsf@alter.siamese.dyndns.org","threadId":"27794","inReplyTo":"4E1E0C27.60903@web.de","subject":"Re: Nesting a submodule inside of another...","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-07-13T21:27:38Z","receivedAt":"2011-07-13T21:27:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> It can't be communicated, as submodules are unaware of their superproject.\n\nIt's more like s/can't/shouldn't/ and s/are/shouldn't be/, no?\n"},{"id":"171294","messageId":"4E1E1672.7040503@web.de","threadId":"27794","inReplyTo":"7vhb6p6fcl.fsf@alter.siamese.dyndns.org","subject":"Re: Nesting a submodule inside of another...","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2011-07-13T22:04:34Z","receivedAt":"2011-07-13T22:04:34Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 13.07.2011 23:27, schrieb Junio C Hamano:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n> \n>> It can't be communicated, as submodules are unaware of their superproject.\n> \n> It's more like s/can't/shouldn't/ and s/are/shouldn't be/, no?\n\nRight. Submodules /could/ be taught to check if they are part of a\nsuperproject, but unless I miss something obvious they are currently\ntotally unaware of that (at least when it concerns their work tree, the\ngitfile effort is trying to introduce some superproject awareness). And as\nfar as I can see that is a sound design decision as far as it concerns the\nwork tree. Imagine the files and/or directories a submodule has to ignore\nwould depend on the settings of their superproject too (which is what this\nthread is about). That would lead to very interesting problems when the same\nsubmodule would be used by different superprojects which ignore different\nfiles and/or directories inside the submodule, which - to make things even\nmore interesting - might be recorded in the submodule itself ... not good.\n\nSo what about: s/can't/isn't and shouldn't/ and s/are/are and shouldn't/?\n"},{"id":"171298","messageId":"7v8vs16bst.fsf@alter.siamese.dyndns.org","threadId":"27794","inReplyTo":"4E1E1672.7040503@web.de","subject":"Re: Nesting a submodule inside of another...","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-07-13T22:44:18Z","receivedAt":"2011-07-13T22:44:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> ... (at least when it concerns their work tree, the\n> gitfile effort is trying to introduce some superproject awareness).\n\nI don't think so.\n\nThe \"gitfile effort\" just makes it more convenient and potentially safer\nfor superproject to remove submodule working tree by running an equivalent\nof \"rm -fr submodule\". The submodule shouldn't even care where its .git\nmetadirectory is by looking at the \"gitfile: \" line, nor should it assume\nthat location its .git metadirectory moved out of its working tree has\nsome fixed relationship with its supermodule's repository.  E.g. the\nsuperproject could choose to store it under $HOME/cache or /var/state as\nlong as it knows where to point it back to after it runs \"rm -rf submodule\"\nand then it has to re-instantiate the submodule.\n\nIOW, the superproject must be aware of that location, but submodule\nshouldn't depend on what convention the superproject chose to use.\nAfter all, there may not even be a superproject when gitfile is in use,\nso running \"test -f .git\" and assuming that it is used as a submodule of\nsomebody else is already a bogus test.\n"},{"id":"171312","messageId":"CAEBDL5VUPE9YCX1C4pqkjb+EODkAWo9h774B=Jv5eUNbocMuZQ@mail.gmail.com","threadId":"27794","inReplyTo":"4E1E0C27.60903@web.de","subject":"Re: Nesting a submodule inside of another...","fromName":"John Szakmeister","fromEmail":"john@szakmeister.net","sentAt":"2011-07-14T09:36:46Z","receivedAt":"2011-07-14T09:36:46Z","isPatch":false,"sender":{"key":"john@szakmeister.net","avatar":"https://avatars.githubusercontent.com/u/448087?v=4"},"body":"On Wed, Jul 13, 2011 at 5:20 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:\n> Am 13.07.2011 03:21, schrieb John Szakmeister:\n[snip]\n>> I guess I still don't see how that's a problem.  It seems to me that\n>> for the superproject the ignore rule needs to be:\n>>   ignore framework1, except framework1/module\n>\n> But that would mean the superproject would have to scan framework1's\n> work tree for changes too, which is conflicting with the idea that each\n> submodule is a repo of it's own, so that won't fly.\n\nI guess I just don't see that.  Even though the path is inside of a\nsubmodules, it would be part of the superproject's work tree.  But, I\ncan see how it's a violation of a submodule being a repo of it's own.\nAnd I certainly understand the desire to maintain that quality.\n\n>> And then it could check its status.  Of course, framework1 would\n>> somehow need to know to ignore it... and I'm not sure how that can be\n>> communicated.\n>\n> It can't be communicated, as submodules are unaware of their superproject.\n> And adding module manually to framework1's .gitignore is just a problem\n> waiting to happen. Having a file or submodule in the work tree of one\n> git repository being referenced by another is just asking for trouble ...\n\nAgain, I don't see how the submodule needs to be aware of the\nsuperproject.  In this case, it'd be the responsibility of the\nsuperproject to setup whatever is necessary at 'git submodule\ninit/add'.  I don't see how the submodule *must* know about the\nsuperproject for it to succeed.  I see the opposite, the superproject\nneeds to communicate some information down to the submodule, but I\ndon't see the reverse.\n\nWhat I'm hearing is that while it may be possible, the idea of\nviolating the concept that the \"subrepo is standalone\" is\nunacceptable.  Which means, unfortunately, git isn't a solution for us\nin this case.\n\n[snip]\n>> I'm using 1.7.6rc3.  I updated to the latest trunk and still get the same error:\n>>\n>> :: git submodule add $PWD/../c b/c\n>> The following path is ignored by one of your .gitignore files:\n>> b/c\n>> Use -f if you really want to add it.\n>\n> You're right, the error message of \"git submodule add\" should be better\n> here.\n\nI'll write a patch for this.  Thanks.\n\nAnd thanks for the input.  I was just looking to see if there were any\ntechnical/morale reasons that submodules weren't allowed in this way.\nI appreciate you all taking the time to answer.\n\n-John\n"},{"id":"171338","messageId":"201107141433.p6EEXbb9023826@no.baka.org","threadId":"27794","inReplyTo":"CAEBDL5VUPE9YCX1C4pqkjb+EODkAWo9h774B=Jv5eUNbocMuZQ@mail.gmail.com","subject":"Re: Nesting a submodule inside of another...","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2011-07-14T14:33:37Z","receivedAt":"2011-07-14T14:33:37Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message <CAEBDL5VUPE9YCX1C4pqkjb+EODkAWo9h774B=Jv5eUNbocMuZQ@mail.gmail.com>, John Szakmeister writes:\n\n    > I've got a project where we have several frameworks involved, and\n    > external modules we want to pull into the framework tree.  We'd like\n    > to make use of submodules and have something like this:\n    >      top-level/<-- .gitmodules lives here\n    >          src/\n    >          framework1/<-- a submodule\n    >              module/<-- another submodule\n    >          framework2/<-- a submodule\n    >              module2/<-- another submodule\n\n    Again, I don't see how the submodule needs to be aware of the\n    superproject.  In this case, it'd be the responsibility of the\n    superproject to setup whatever is necessary at 'git submodule\n    init/add'.  I don't see how the submodule *must* know about the\n    superproject for it to succeed.  I see the opposite, the superproject\n    needs to communicate some information down to the submodule, but I\n    don't see the reverse.\n\n    What I'm hearing is that while it may be possible, the idea of\n    violating the concept that the \"subrepo is standalone\" is\n    unacceptable.  Which means, unfortunately, git isn't a solution for us\n    in this case.\n\nYou might find that gitslave (http://gitslave.sf.net) might be a\nbetter solution for you than git-submodules in this case.  It works\nbetter for many workflows (and worse for others) but is much simpler\nto understand since with gitslave you have JBOR (just a bunch of\nrepositories) with a program which can be thought of as running the\nrequested git command over each repository in turn.  Gitslave thus has\na loose binding between the repositories, and you can only guarantee\nthe relationship between repositories at tagged locations, though in\npractice this isn't a major concern.\n\ngitslave supports nested repositories (and recursive gitslave\nrepositories, but those are different).  With gitslave nested\nrepositories it is also true that you would have to have a\nsupplemental gitignore entry in framework1 (which gitslave will\ncreate).\n\nIf you have any questions, please let me know.\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"171398","messageId":"CAEBDL5XzhuSQSzEMQ8yRupcufyQ=shtiTWejez0Mt75e0sJ-Vg@mail.gmail.com","threadId":"27794","inReplyTo":"201107141433.p6EEXbb9023826@no.baka.org","subject":"Re: Nesting a submodule inside of another...","fromName":"John Szakmeister","fromEmail":"john@szakmeister.net","sentAt":"2011-07-15T08:34:46Z","receivedAt":"2011-07-15T08:34:46Z","isPatch":false,"sender":{"key":"john@szakmeister.net","avatar":"https://avatars.githubusercontent.com/u/448087?v=4"},"body":"On Thu, Jul 14, 2011 at 10:33 AM, Seth Robertson <in-gitvger@baka.org> wrote:\n[snip]\n>    Again, I don't see how the submodule needs to be aware of the\n>    superproject.  In this case, it'd be the responsibility of the\n>    superproject to setup whatever is necessary at 'git submodule\n>    init/add'.  I don't see how the submodule *must* know about the\n>    superproject for it to succeed.  I see the opposite, the superproject\n>    needs to communicate some information down to the submodule, but I\n>    don't see the reverse.\n>\n>    What I'm hearing is that while it may be possible, the idea of\n>    violating the concept that the \"subrepo is standalone\" is\n>    unacceptable.  Which means, unfortunately, git isn't a solution for us\n>    in this case.\n>\n> You might find that gitslave (http://gitslave.sf.net) might be a\n> better solution for you than git-submodules in this case.  It works\n> better for many workflows (and worse for others) but is much simpler\n> to understand since with gitslave you have JBOR (just a bunch of\n> repositories) with a program which can be thought of as running the\n> requested git command over each repository in turn.  Gitslave thus has\n> a loose binding between the repositories, and you can only guarantee\n> the relationship between repositories at tagged locations, though in\n> practice this isn't a major concern.\n\nInteresting.  I'll take a look!\n\n> gitslave supports nested repositories (and recursive gitslave\n> repositories, but those are different).  With gitslave nested\n> repositories it is also true that you would have to have a\n> supplemental gitignore entry in framework1 (which gitslave will\n> create).\n>\n> If you have any questions, please let me know.\n\nI appreciate it.  Thanks Seth!\n\n-John\n"}]}