{"thread":{"id":"12915","subject":"git-submodule getting submodules from the parent repository","startedAt":"2008-03-29T22:35:59Z","lastAt":"2008-04-02T21:32:29Z","messageCount":20,"participants":["Avery Pennarun","Sam Vilain","Eyvind Bernhardsen","Johannes Sixt","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"73341","messageId":"32541b130803291535m317e84e6p321ebccd5dedaab3@mail.gmail.com","threadId":"12915","inReplyTo":null,"subject":"git-submodule getting submodules from the parent repository","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-03-29T22:35:59Z","receivedAt":"2008-03-29T22:35:59Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"Hi all,\n\nI have a fairly pressing need for git-submodule-like behaviour, but\nhaving tried git-submodule, it doesn't really work the way I'd like.\n\nA super-simplified example of what I do:\n\n- Project A (app) includes project B (build environment), which\nincludes project C (tool library)\n\n- The projects are all open source, but B includes some binary\npackages so it's a big download.  If you don't need the binary\npackages, people want to just download C (hence the separation). But\neveryone using A wants B and C, because they're lazy and bandwidth\nisn't a problem.\n\n- We have a local repository at work with mirrors of A, B, and C,\nwhich are also available publicly (but there's no reason for everyone\nin our office to be uploading/downloading the same big blobs all the\ntime).\n\n- We frequently change B and C as part of building A (as well as other\nA-like applications).\n\nHere are the main problems, all in a jumble:\n\nIt's a  pain to check out / mirror / check in / push.  git-submodule\ndoesn't even init automatically when you check out A, so you have to\nrun it yourself.  The relative paths of A, B, and C on your mirror\nhave to be the same as upstream.  You can't make a local mirror of A\nwithout mirroring B and C.  B and C start out with a disconnected\nHEAD, so if you check in, it goes nowhere, and then when you push,\nnothing happens, and if you're unlucky enough to pull someone else's\nupdate to A and then \"git-submodule update\", it forgets your changes\nentirely.  When you check in to C, you then have to check in to B, and\nthen to A, all by hand; and when you git-pull, you'd better to C, then\nB, then A, or risk having A try to check out a revision from B that\nyou haven't pulled, etc.\n\n...phew.\n\nIt would probably be possible to fix each of these problems\nindividually, but it would be a whole series of different fixes.  I'd\nlike to propose a rather different way of doing things that I think\nwould solve most of these problems, and get some feedback:\n\nWhat if *all* the objects for A, B, and C were always in the *same*\nrepository?  Almost all the problems would go away.  Imagine if it\nworked like this:\n\n1. git-clone had a way to *not* clone every single object from every\nbranch in the parent repository; only the ones you were interested in.\nSo if you git-clone only project C, you don't end up with A and B.  (I\npersonally would never need to clone A without B and C, but maybe\nsomeone else would.  It's actually a different implementation issue,\nsince A refers to B, but B doesn't refer to A.)\n\n2. You still check into C, then B, then A, but it doesn't actually\nmatter if you put B and C on a branch first or not, because 'git push'\nwill work properly, because it auto-pushes B and C revisions based on\nthe fact that A refers to them (ie. implicit branches via the\nsubmodule mechanism).\n\n3. You can still upload B and C to their own separate upstream\nrepository, which is obviously a critical feature.  But you can do\nthat simply by making a branch in your copy of C and pushing just that\nbranch.  The objects come from your one local A repository, but you\nsimply avoid accidentally pushing the wrong branch.\n\n4. You can 'git clone' a local copy of A, and B/C will be cloned\nautomatically along with it.\n\n5. B and C, when git-submodule checks them out, should have their own\n.git directories, but use A as an 'alternatives' entry.\n\n6. git-pull should be modified to auto-download objects referred to by\n'submodule' references in trees.\n\nThis would really help my workflow a lot.  Am I missing something?\n\nThoughts?\n\nThanks,\n\nAvery\n"},{"id":"73342","messageId":"47EECF1F.60908@vilain.net","threadId":"12915","inReplyTo":"32541b130803291535m317e84e6p321ebccd5dedaab3@mail.gmail.com","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-03-29T23:22:07Z","receivedAt":"2008-03-29T23:22:07Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Avery Pennarun wrote:\n> It would probably be possible to fix each of these problems\n> individually, but it would be a whole series of different fixes.  I'd\n> like to propose a rather different way of doing things that I think\n> would solve most of these problems, and get some feedback:\n> \n> What if *all* the objects for A, B, and C were always in the *same*\n> repository?  Almost all the problems would go away.  Imagine if it\n> worked like this:\n\nWell, that would create a lot of unnecessary work when cloning.\nPartitioning by project is a natural way to divide the projects up.\nIt's worth noting that the early implementations of submodules were\nbased on this design, of keeping everything together.\n\nHowever, what you are suggesting should IMHO be allowed to work.  In\nparticular, if the submodule path is \".\", then I think there's a good\ncase that they should come from within the same project.  If it's a\nrelative URL, it should initialize based on the remote URL that was used\nfor the original fetch (or, rather, the remote URL for the current branch).\n\nAnd, if it happens that after a checkout, that the commit of a submodule\nis already in the object directory (ie, there's another branch), then\nmaybe that should automatically check out.\n\n> 1. git-clone had a way to *not* clone every single object from every\n> branch in the parent repository; only the ones you were interested in.\n\nIt could easily, if someone would allow clone to have a --track option\nlike git remote does:\n\n  git init\n  git remote add -t branch -f URL\n\n> 2. You still check into C, then B, then A, but it doesn't actually\n> matter if you put B and C on a branch first or not, because 'git push'\n> will work properly, because it auto-pushes B and C revisions based on\n> the fact that A refers to them (ie. implicit branches via the\n> submodule mechanism).\n\nThis push failure thing is regrettable; however it's not clear which\nbranch name the submodules should get.  A given commit might exist on\nseveral branches, which one do you choose to name it?\n\n> 4. You can 'git clone' a local copy of A, and B/C will be cloned\n> automatically along with it.\n> 6. git-pull should be modified to auto-download objects referred to by\n> 'submodule' references in trees.\n\nI think this could be a switch to git clone/pull, configurable to be the\ndefault action.\n\n> 5. B and C, when git-submodule checks them out, should have their own\n> .git directories, but use A as an 'alternatives' entry.\n\nThere is also a Google Summer of Code project for this - see\nhttp://git.or.cz/gitwiki/SoC2008Ideas#head-9215572f23513542a23d3555aa72775bc4b91038\n\n> This would really help my workflow a lot.  Am I missing something?\n\nWell, no, it's true that the current workflow has interface niggles;\nhowever it's important to understand why the current implementation is\nthe way it is, and make sure that new designs build on top of the parts\nwhich are already designed well, where they can.\n\nSam\n"},{"id":"73354","messageId":"D0F821FA-AF53-4F1F-B9CC-58346828FA15@orakel.ntnu.no","threadId":"12915","inReplyTo":"47EECF1F.60908@vilain.net","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Eyvind Bernhardsen","fromEmail":"eyvind-git@orakel.ntnu.no","sentAt":"2008-03-30T13:32:21Z","receivedAt":"2008-03-30T13:32:21Z","isPatch":false,"sender":{"key":"eyvind.bernhardsen@gmail.com","avatar":"https://avatars.githubusercontent.com/u/106762?v=4"},"body":"On 30. mars. 2008, at 00.22, Sam Vilain <sam@vilain.net> wrote:\n\n> Avery Pennarun wrote:\n>\n>> 2. You still check into C, then B, then A, but it doesn't actually\n>> matter if you put B and C on a branch first or not, because 'git  \n>> push'\n>> will work properly, because it auto-pushes B and C revisions based on\n>> the fact that A refers to them (ie. implicit branches via the\n>> submodule mechanism).\n>\n> This push failure thing is regrettable; however it's not clear which\n> branch name the submodules should get.  A given commit might exist on\n> several branches, which one do you choose to name it?\n\nI solved that by adding a \"submodule push\" that pushes the detached  \nhead of each submodule to its own ref (\"refs/submodule-update/commit- \n$sha1\", imaginatively).  I also made \"submodule update\" try to fetch  \nthat ref when looking for a sha1.\n\nI ran into trouble trying to avoid pushing every submodule for each  \n\"submodule push\", and then more or less decided not to use submodules,  \nso it's not quite fit for public consumption.  I still think it's a  \nsound idea in principle, so I'll clean it up and send it to the list  \nif there's any interest.\n-- \nEyvind Bernhardsen\n"},{"id":"73360","messageId":"47EFD253.6020105@vilain.net","threadId":"12915","inReplyTo":"D0F821FA-AF53-4F1F-B9CC-58346828FA15@orakel.ntnu.no","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-03-30T17:48:03Z","receivedAt":"2008-03-30T17:48:03Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Eyvind Bernhardsen wrote:\n> I solved that by adding a \"submodule push\" that pushes the detached head\n> of each submodule to its own ref (\"refs/submodule-update/commit-$sha1\",\n> imaginatively).  I also made \"submodule update\" try to fetch that ref\n> when looking for a sha1.\n\nHmm, a reasonable decision, but I think it would be better to force the\nuser to choose which branch they want to push to.  Leaving breadcrumbs\nin the refs/ space like that is best avoided.\n\n> I ran into trouble trying to avoid pushing every submodule for each\n> \"submodule push\", and then more or less decided not to use submodules,\n> so it's not quite fit for public consumption.  I still think it's a\n> sound idea in principle, so I'll clean it up and send it to the list if\n> there's any interest.\n\nIndeed - it can only become \"fit for public consumption\" if people\nsubmit their usability enhancements!\n\nSam.\n"},{"id":"73361","messageId":"C38585A9-F09C-4A5B-8E72-6F3127DB2BB9@orakel.ntnu.no","threadId":"12915","inReplyTo":"47EFD253.6020105@vilain.net","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Eyvind Bernhardsen","fromEmail":"eyvind-git@orakel.ntnu.no","sentAt":"2008-03-30T19:50:24Z","receivedAt":"2008-03-30T19:50:24Z","isPatch":false,"sender":{"key":"eyvind.bernhardsen@gmail.com","avatar":"https://avatars.githubusercontent.com/u/106762?v=4"},"body":"On 30. mars. 2008, at 19.48, Sam Vilain wrote:\n> Eyvind Bernhardsen wrote:\n>> I solved that by adding a \"submodule push\" that pushes the detached  \n>> head\n>> of each submodule to its own ref (\"refs/submodule-update/commit- \n>> $sha1\",\n>> imaginatively).  I also made \"submodule update\" try to fetch that ref\n>> when looking for a sha1.\n>\n> Hmm, a reasonable decision, but I think it would be better to force  \n> the\n> user to choose which branch they want to push to.  Leaving breadcrumbs\n> in the refs/ space like that is best avoided.\n\nWell, the point of \"submodule push\" was to avoid having to push in  \neach submodule manually; not enforcing the requirement that commits in  \nsubmodules must be publicly available before pushing from the main  \nmodule is a recipe for disaster, or at least annoyance.  And nobody  \nlikes an annoying git.\n\nPushing to a branch works except that I couldn't figure out what to do  \nif the push doesn't succeed, ie, the branch has advanced on the remote  \nend.  That's a problem if more than one module references the  \nsubmodule or there are multiple branches in the main module.\n\nOne solution that occurred to me was to have a branch in each  \nsubmodule for every main module and branch.  A branch name would be  \nprovided for each submodule in .gitmodules, used by \"submodule push\"  \nbut not \"submodule update\".  In this case, if the push to the branch  \nfails, the main module branch is probably behind too.\n\nThis seemed like a good idea, but it's racy.  If two simultaneous  \n\"submodule push\"es try to push to the same branch on a submodule, one  \nof them will be rejected, but it might already have updated branches  \non other submodules.  Ick.\n\nI briefly toyed with creating tags named after the main module and its  \nbranch, with the submodule sha1 included for good measure, but that  \nleaves a _real_ mess in refs/tags.  Figuring out that I could use refs/ \nsubmodule-push instead seemed like an epiphany at the time.\n\n\nAs an aside, my mental model of what the submodule needs is a  \nfetchable reflog for every main module and branch that uses it,  \ncontaining the history of commits used by that module/branch.\n\nIt's a reflog, not a branch, because a submodule can be changed to a  \ndifferent branch, rewound, etc between commits in the main module;  \nthere's no requirement that the old commit is in the new commit's  \nhistory.  You actually don't want to fetch the whole thing, but you  \nhave to be able to fetch every sha1 contained in it, by sha1.\n\n...so that's what refs/submodule-push is supposed to look like.  You  \nmight need to squint a bit.\n\n> Indeed - it can only become \"fit for public consumption\" if people\n> submit their usability enhancements!\n\nTrue :)  I still have no idea how to figure out which submodules are  \ndirty, though.  Solving that will enable a safe \"submodule update\",  \nwhich I think is more important than \"submodule push\".\n-- \nEyvind Bernhardsen\n"},{"id":"73362","messageId":"47EFF5BE.7030502@vilain.net","threadId":"12915","inReplyTo":"C38585A9-F09C-4A5B-8E72-6F3127DB2BB9@orakel.ntnu.no","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-03-30T20:19:10Z","receivedAt":"2008-03-30T20:19:10Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Eyvind Bernhardsen wrote:\n> Well, the point of \"submodule push\" was to avoid having to push in  \n> each submodule manually; not enforcing the requirement that commits in  \n> submodules must be publicly available before pushing from the main  \n> module is a recipe for disaster, or at least annoyance.  And nobody  \n> likes an annoying git.\n\nok.  so, refuse to push without forcing, don't do something dumb.\n\n> Pushing to a branch works except that I couldn't figure out what to do  \n> if the push doesn't succeed, ie, the branch has advanced on the remote  \n> end.\n\nIt's simple.  You just fail and tell the user what happened, and let\nthem decide what to do.\n\n> It's a reflog, not a branch, because a submodule can be changed to a  \n> different branch, rewound, etc between commits in the main module;  \n> there's no requirement that the old commit is in the new commit's  \n> history.\n\nIf it is a rewind there is no issue, because you don't even need to push.\n\nBut again it comes back to - let the user sort it out, don't try to be\ntoo clever.\n\nSam.\n"},{"id":"73373","messageId":"32541b130803301600g5005876enf0fbcfe03e660fc8@mail.gmail.com","threadId":"12915","inReplyTo":"47EECF1F.60908@vilain.net","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-03-30T23:00:55Z","receivedAt":"2008-03-30T23:00:55Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Sat, Mar 29, 2008 at 7:22 PM, Sam Vilain <sam@vilain.net> wrote:\n> Avery Pennarun wrote:\n>  > What if *all* the objects for A, B, and C were always in the *same*\n>  > repository?  Almost all the problems would go away.  Imagine if it\n>  > worked like this:\n>\n>  Well, that would create a lot of unnecessary work when cloning.\n>  Partitioning by project is a natural way to divide the projects up.\n\nWhat unnecessary work do you mean?  Certainly fetching only a\nparticular set of refs from a remote repository is possible, as that's\nwhat 'git pull' does.\n\nI agree that partitioning by project makes sense... but it also seems\nto me that throwing extra objects into a repository that requires them\nanyhow shouldn't have any major negative results.  After all, if you\ncan't build A without B, then downloading A might as well download the\nobjects from B too.  Which is not to say that B shouldn't *also* have\nits own repository.\n\n>  It's worth noting that the early implementations of submodules were\n>  based on this design, of keeping everything together.\n\nI'd like to read about the rationale behind this change.  Is there a\nthread you can point to?\n\n>  However, what you are suggesting should IMHO be allowed to work.  In\n>  particular, if the submodule path is \".\", then I think there's a good\n>  case that they should come from within the same project.  If it's a\n>  relative URL, it should initialize based on the remote URL that was used\n>  for the original fetch (or, rather, the remote URL for the current branch).\n\nI agree, there's no reason to take away the existing functionality of\nallowing split repos.  I was more suggesting a new functionality so\nthat splitting isn't *required*.\n\n>  > 2. You still check into C, then B, then A, but it doesn't actually\n>  > matter if you put B and C on a branch first or not, because 'git push'\n>  > will work properly, because it auto-pushes B and C revisions based on\n>  > the fact that A refers to them (ie. implicit branches via the\n>  > submodule mechanism).\n>\n>  This push failure thing is regrettable; however it's not clear which\n>  branch name the submodules should get.  A given commit might exist on\n>  several branches, which one do you choose to name it?\n\nOne option is to make a simple \"git push origin\" operation fail if\nyou're not on any branch; iirc, if you try that now, it just silently\n*succeeds* without uploading anything at all, which is one reason I so\nfrequently screw it up.  Alternatively, is there a reason I can't\nupload an object *without* giving it a branch name?  I guess that\nwould cause problems with garbage collection.\n\nNow, the fail-on-branchless-push option still isn't really perfect,\nbecause then I'll screw up like this:\n- make change\n- check in\n- try to push: fails\n- switch to branch\n- realize I've lost my checkin(s) and have to go scrounge in the\nreflog to try to find it\n\nIf we could disallow checkins to disconnected heads, then I'd get an\nerror at step 1, before I had a chance to screw up.  I think that\nwould be a usability improvement to git in general.  For example, if I\nscrew up a git-rebase and forget to abort, my HEAD ends up\ndisconnected and I occasionally check things in by accident and then\nlose them (only to be saved by the reflog).  Perhaps an extra option\nto git-commit that must be used if you want to check into a\nnon-branch?  Is that too harsh?\n\nAnother option would be to simply *always* create/update a branch tag\nwhen doing \"git submodule update\".  But then the question is which\nbranch tag.  One thing that would give pretty useful semantics would\nbe to create a local branch tag with the same name as it had when the\nsubmodule ref was checked in in the first place.\n\nThat is, if I make a change to B on branch \"master,\" and then check\nthat into A, then the next time someone checks out A, it would be\ngreat if it retrieved B with the same commitid, then named that\ncommitid \"master\".  This is *despite* the fact that the\n\"remotes/master\" branch might be *newer* than the newly created local\n\"master\".  Why?  Because it would give standard git semantics to my\nnew checkout: I can \"git pull\" to pull in the latest changes from\nremote/master, or I can checkin and try to \"git push\" and it'll fail,\njust like it should, which would encourage me to either \"git pull\" or\ncreate a *new* topic branch, just as it should.\n\nI think that solution would be great for me, but it would require\nchanges to the tree format in order to store the branch name, which is\nunfortunately since trees currently don't know *anything* about branch\nnames afaict.  Is there a better way to achieve the same result?\n(Note that in this case, \"correct\" operation doesn't *require* the new\nbranch name information; we just use it as a hint.)\n\n>  > 4. You can 'git clone' a local copy of A, and B/C will be cloned\n>  > automatically along with it.\n>\n> > 6. git-pull should be modified to auto-download objects referred to by\n>  > 'submodule' references in trees.\n>\n>  I think this could be a switch to git clone/pull, configurable to be the\n>  default action.\n\nSure.  Or it could try by default, but not error out if it turned out\nnot to be able to find them.\n\n>  > 5. B and C, when git-submodule checks them out, should have their own\n>  > .git directories, but use A as an 'alternatives' entry.\n>\n>  There is also a Google Summer of Code project for this - see\n>  http://git.or.cz/gitwiki/SoC2008Ideas#head-9215572f23513542a23d3555aa72775bc4b91038\n\nok.  I was hoping it wouldn't be so hard as to require an entire SoC\nproject, since using --alternate when checking out the child repo\nshouldn't be too hard.\n\nHave fun,\n\nAvery\n"},{"id":"73374","messageId":"32541b130803301603u65d1b6b6ladac0f6200433e5f@mail.gmail.com","threadId":"12915","inReplyTo":"C38585A9-F09C-4A5B-8E72-6F3127DB2BB9@orakel.ntnu.no","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-03-30T23:03:27Z","receivedAt":"2008-03-30T23:03:27Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Sun, Mar 30, 2008 at 3:50 PM, Eyvind Bernhardsen\n<eyvind-git@orakel.ntnu.no> wrote:\n>  Pushing to a branch works except that I couldn't figure out what to do\n>  if the push doesn't succeed, ie, the branch has advanced on the remote\n>  end.  That's a problem if more than one module references the\n>  submodule or there are multiple branches in the main module.\n\nThat's easy: just error out in that case.  If the current system would\njust error out when I screwed up, I'd at least be able to deal with\nit.  Right now I silently create un-check-outable parent repositories\nbecause I failed silently to upload my latest checkins to the child\nrepository.\n\n>  True :)  I still have no idea how to figure out which submodules are\n>  dirty, though.  Solving that will enable a safe \"submodule update\",\n>  which I think is more important than \"submodule push\".\n\nWhat is unsafe about \"submodule update\"?\n\nThanks,\n\nAvery\n"},{"id":"73412","messageId":"47F08343.20209@viscovery.net","threadId":"12915","inReplyTo":"32541b130803291535m317e84e6p321ebccd5dedaab3@mail.gmail.com","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-03-31T06:22:59Z","receivedAt":"2008-03-31T06:22:59Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Avery Pennarun schrieb:\n> It's a  pain to check out / mirror / check in / push.  git-submodule\n> doesn't even init automatically when you check out A, so you have to\n> run it yourself.  The relative paths of A, B, and C on your mirror\n> have to be the same as upstream.  You can't make a local mirror of A\n> without mirroring B and C.  B and C start out with a disconnected\n> HEAD, so if you check in, it goes nowhere, and then when you push,\n> nothing happens, and if you're unlucky enough to pull someone else's\n> update to A and then \"git-submodule update\", it forgets your changes\n> entirely.  When you check in to C, you then have to check in to B, and\n> then to A, all by hand; and when you git-pull, you'd better to C, then\n> B, then A, or risk having A try to check out a revision from B that\n> you haven't pulled, etc.\n\nWould a \"recurse\" sub-command help your workflow?\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/69834\n\n-- Hannes\n"},{"id":"73425","messageId":"834174D1-82F4-4438-9854-762F416BB5EF@orakel.ntnu.no","threadId":"12915","inReplyTo":"32541b130803301603u65d1b6b6ladac0f6200433e5f@mail.gmail.com","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Eyvind Bernhardsen","fromEmail":"eyvind-git@orakel.ntnu.no","sentAt":"2008-03-31T09:29:56Z","receivedAt":"2008-03-31T09:29:56Z","isPatch":false,"sender":{"key":"eyvind.bernhardsen@gmail.com","avatar":"https://avatars.githubusercontent.com/u/106762?v=4"},"body":"On 31. mars. 2008, at 01.03, Avery Pennarun wrote:\n\n> On Sun, Mar 30, 2008 at 3:50 PM, Eyvind Bernhardsen\n> <eyvind-git@orakel.ntnu.no> wrote:\n>> Pushing to a branch works except that I couldn't figure out what to  \n>> do\n>> if the push doesn't succeed, ie, the branch has advanced on the  \n>> remote\n>> end.  That's a problem if more than one module references the\n>> submodule or there are multiple branches in the main module.\n>\n> That's easy: just error out in that case.  If the current system would\n> just error out when I screwed up, I'd at least be able to deal with\n> it.  Right now I silently create un-check-outable parent repositories\n> because I failed silently to upload my latest checkins to the child\n> repository.\n\nAs I tried to explain, all the automatic push solutions I could come  \nup with were flawed, so I decided not to use submodules at all and  \njust have the build tool check out every module (that's what we  \ncurrently do with CVS, so it's the easy way out anyway).\n\nIf I understand you correctly, you want to be forced to create a  \nbranch and push to that?  I don't think that works well with many  \ndevelopers pushing to a shared repository (my situation), and is in  \nany case not the \"automagical push\" solution that I want.  I agree  \nthat it would be an improvement, but it doesn't scratch my itch :)\n\n>> True :)  I still have no idea how to figure out which submodules are\n>> dirty, though.  Solving that will enable a safe \"submodule update\",\n>> which I think is more important than \"submodule push\".\n>\n> What is unsafe about \"submodule update\"?\n\nIf you have local changes committed in a submodule that is updated by  \na pull in the main module, \"submodule update\" will silently overwrite  \nthem.  I was wrong, though, because you can fix that just by making  \n\"submodule update\" error out when a submodule doesn't have its HEAD  \nwhere the main module thinks it should be.\n-- \nEyvind Bernhardsen\n"},{"id":"73431","messageId":"62D616ED-83C8-4AB7-9113-98688732DE09@orakel.ntnu.no","threadId":"12915","inReplyTo":"47EFF5BE.7030502@vilain.net","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Eyvind Bernhardsen","fromEmail":"eyvind-git@orakel.ntnu.no","sentAt":"2008-03-31T10:05:20Z","receivedAt":"2008-03-31T10:05:20Z","isPatch":false,"sender":{"key":"eyvind.bernhardsen@gmail.com","avatar":"https://avatars.githubusercontent.com/u/106762?v=4"},"body":"On 30. mars. 2008, at 22.19, Sam Vilain wrote:\n\n> Eyvind Bernhardsen wrote:\n>> Well, the point of \"submodule push\" was to avoid having to push in\n>> each submodule manually; not enforcing the requirement that commits  \n>> in\n>> submodules must be publicly available before pushing from the main\n>> module is a recipe for disaster, or at least annoyance.  And nobody\n>> likes an annoying git.\n>\n> ok.  so, refuse to push without forcing, don't do something dumb.\n\nI think you misunderstood: what I'm saying is that submodules'  \n_current_ behaviour is annoying, since you're guaranteed to forget to  \npush a submodule before pushing the main module at least once.  My  \nattempt to solve that became too complicated, so I dropped it, and  \nsince the current behaviour is annoying, I gave up on submodules  \nentirely.\n\n>> Pushing to a branch works except that I couldn't figure out what to  \n>> do\n>> if the push doesn't succeed, ie, the branch has advanced on the  \n>> remote\n>> end.\n>\n> It's simple.  You just fail and tell the user what happened, and let\n> them decide what to do.\n\nSure, that solves the annoyance problem, but I wanted something more  \nautomatic.\n\n>> It's a reflog, not a branch, because a submodule can be changed to a\n>> different branch, rewound, etc between commits in the main module;\n>> there's no requirement that the old commit is in the new commit's\n>> history.\n>\n> If it is a rewind there is no issue, because you don't even need to  \n> push.\n>\n> But again it comes back to - let the user sort it out, don't try to be\n> too clever.\n\nYep, my problem was wanting to be cleverer than my limited git skills  \nwill allow.\n-- \nEyvind Bernhardsen\n"},{"id":"73456","messageId":"32541b130803311424h339a30e6g4328ea27045c681d@mail.gmail.com","threadId":"12915","inReplyTo":"47F08343.20209@viscovery.net","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-03-31T21:24:13Z","receivedAt":"2008-03-31T21:24:13Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Mon, Mar 31, 2008 at 2:22 AM, Johannes Sixt <j.sixt@viscovery.net> wrote:\n> Avery Pennarun schrieb:\n> > It's a  pain to check out / mirror / check in / push.  git-submodule\n>  > doesn't even init automatically when you check out A, so you have to\n>  > run it yourself.  The relative paths of A, B, and C on your mirror\n>  > have to be the same as upstream.  You can't make a local mirror of A\n>  > without mirroring B and C.  B and C start out with a disconnected\n>  > HEAD, so if you check in, it goes nowhere, and then when you push,\n>  > nothing happens, and if you're unlucky enough to pull someone else's\n>  > update to A and then \"git-submodule update\", it forgets your changes\n>  > entirely.  When you check in to C, you then have to check in to B, and\n>  > then to A, all by hand; and when you git-pull, you'd better to C, then\n>  > B, then A, or risk having A try to check out a revision from B that\n>  > you haven't pulled, etc.\n>\n>  Would a \"recurse\" sub-command help your workflow?\n>\n>  http://thread.gmane.org/gmane.comp.version-control.git/69834\n\nWell, typing \"git submodule recurse push\" or something would allow me\nto lose the same data without typing quite as much, so strictly\nspeaking I guess it would be an improvement :)\n\nI'd like it even more if \"git push\" actually somehow refused to push\nat all if I forgot to push in the submodules.\n\nHave fun,\n\nAvery\n"},{"id":"73458","messageId":"32541b130803311436t7b5041a4pabface15aad8ce63@mail.gmail.com","threadId":"12915","inReplyTo":"834174D1-82F4-4438-9854-762F416BB5EF@orakel.ntnu.no","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-03-31T21:36:35Z","receivedAt":"2008-03-31T21:36:35Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Mon, Mar 31, 2008 at 5:29 AM, Eyvind Bernhardsen\n<eyvind-git@orakel.ntnu.no> wrote:\n> On 31. mars. 2008, at 01.03, Avery Pennarun wrote:\n>  As I tried to explain, all the automatic push solutions I could come\n>  up with were flawed, so I decided not to use submodules at all and\n>  just have the build tool check out every module (that's what we\n>  currently do with CVS, so it's the easy way out anyway).\n\nI even *use* git-submodule and had to modify my build scripts because\n\"git submodule init\" and \"git submodule update\" don't seem to kick in\nautomatically for some reason.  The ideal situation would be to have\ngit just manage the version control without having to babysit it, of\ncourse.  That's hard to do in the general case, but should be quite\npossible in the limited situation that I'm proposing in this thread.\n\n>  If I understand you correctly, you want to be forced to create a\n>  branch and push to that?  I don't think that works well with many\n>  developers pushing to a shared repository (my situation),\n\nHmm, this is curious.  If you're *not* using submodules, then I don't\nthink you can push successfully without being on a branch, can you?\nSo the suggestion merely extends this behaviour to submodules.\n\n(To be more precise, 'git push' seems only to be able to push branch\nheads.  When you're not using git-submodule, commits are by default\nattached to branch heads, so this doesn't cause a problem.  If you\ndisconnect your HEAD, trying to push will silently do nothing, because\nit'll push some other branch head that hasn't changed, or maybe no\nbranch at all.  But with git-submodule, the *default* is a\ndisconnected HEAD, which is too dangerous.  I propose to simply have\nit fail out in this case.)\n\nIf you 'git checkout -b branchname' inside a submodule, then 'git\npush' will do the right thing, so I'm not sure what you'd want to be\nmore automagical than that.\n\n>  If you have local changes committed in a submodule that is updated by\n>  a pull in the main module, \"submodule update\" will silently overwrite\n>  them.  I was wrong, though, because you can fix that just by making\n>  \"submodule update\" error out when a submodule doesn't have its HEAD\n>  where the main module thinks it should be.\n\nShouldn't \"git merge\" get a merge conflict if you've made a checkin\nthat changed the submodule pointer, then try to pull someone else's\nchecking that changes the submodules pointer to something else?  It\nwould seem there's no better option than that.\n\nWhile we're here, it's inconvenient to have to call \"git submodule\nupdate\" at all when there *isn't* a conflict.  It should always be\nsafe for git checkout or git merge to do that for you, no?\n\nThanks,\n\nAvery\n"},{"id":"73518","messageId":"47F2BFCD.5070202@vilain.net","threadId":"12915","inReplyTo":"32541b130803311436t7b5041a4pabface15aad8ce63@mail.gmail.com","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-04-01T23:05:49Z","receivedAt":"2008-04-01T23:05:49Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Avery Pennarun wrote:\n> I even *use* git-submodule and had to modify my build scripts because\n> \"git submodule init\" and \"git submodule update\" don't seem to kick in\n> automatically for some reason.\n\nThe reason is that not everyone wants that by default.  Perhaps it is a\ngood idea for it to be default behaviour; but all in good time.  It can\nbe a user-selected option to clone first.\n\n> The ideal situation would be to have\n> git just manage the version control without having to babysit it, of\n> course.\n\nI can understand the motivation to write such disparaging remarks;\nhowever it may be more productive to come up with good ideas about how\nit can be made to work better for you, without getting in the way of\nother users.  patches are even better!\n\n>>  If I understand you correctly, you want to be forced to create a\n>>  branch and push to that?  I don't think that works well with many\n>>  developers pushing to a shared repository (my situation),\n> \n> Hmm, this is curious.  If you're *not* using submodules, then I don't\n> think you can push successfully without being on a branch, can you?\n\nSure, you could;\n\n  git push origin HEAD:branchname\n\nHowever I think the right solution to this is to name the branch\nappropriately somehow, so that the default push operation works.\n\n> If you 'git checkout -b branchname' inside a submodule, then 'git\n> push' will do the right thing, so I'm not sure what you'd want to be\n> more automagical than that.\n\nWell, where did you get the branch name from?  That's the part that\nrequires user intervention.  You could make an educated guess, such as\nwith git name-rev, but it would not necessarily be the right guess - so\nuser confirmation of the choice would be desirable.\n\nSam.\n"},{"id":"73519","messageId":"47F2C0E4.10506@vilain.net","threadId":"12915","inReplyTo":"32541b130803301600g5005876enf0fbcfe03e660fc8@mail.gmail.com","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-04-01T23:10:28Z","receivedAt":"2008-04-01T23:10:28Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Avery Pennarun wrote:\n>>  Well, that would create a lot of unnecessary work when cloning.\n>>  Partitioning by project is a natural way to divide the projects up.\n> \n> What unnecessary work do you mean?\n\nA full clone takes a few shortcuts, especially over dumb transports like\nHTTP.  I think there might be shortcuts in the git-daemon code as well.\n Forcing these to be partial might make these full fetches involve more\ntime.\n\n>>  However, what you are suggesting should IMHO be allowed to work.  In\n>>  particular, if the submodule path is \".\", then I think there's a good\n>>  case that they should come from within the same project.  If it's a\n>>  relative URL, it should initialize based on the remote URL that was used\n>>  for the original fetch (or, rather, the remote URL for the current branch).\n> I agree, there's no reason to take away the existing functionality of\n> allowing split repos.  I was more suggesting a new functionality so\n> that splitting isn't *required*.\n\nYes - I look forward to a patch.\n\n>>  This push failure thing is regrettable; however it's not clear which\n>>  branch name the submodules should get.  A given commit might exist on\n>>  several branches, which one do you choose to name it?\n> One option is to make a simple \"git push origin\" operation fail if\n> you're not on any branch; iirc, if you try that now, it just silently\n> *succeeds* without uploading anything at all, which is one reason I so\n> frequently screw it up.\n\nSounds workable.\n\n> Alternatively, is there a reason I can't\n> upload an object *without* giving it a branch name?  I guess that\n> would cause problems with garbage collection.\n\nYou've answered your own question there.\n\n>>  There is also a Google Summer of Code project for this - see\n>>  http://git.or.cz/gitwiki/SoC2008Ideas#head-9215572f23513542a23d3555aa72775bc4b91038\n> \n> ok.  I was hoping it wouldn't be so hard as to require an entire SoC\n> project, since using --alternate when checking out the child repo\n> shouldn't be too hard.\n\nIf you think it is simpler, then I'm sure that submodules users would\nappreciate you sharing your ideas as a patch.  Sorry if I am starting to\nsound like a parrot ;-).\n\nSam.\n"},{"id":"73521","messageId":"32541b130804011656l2e907895i98e5260b49743bbe@mail.gmail.com","threadId":"12915","inReplyTo":"47F2BFCD.5070202@vilain.net","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-04-01T23:56:10Z","receivedAt":"2008-04-01T23:56:10Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 4/1/08, Sam Vilain <sam@vilain.net> wrote:\n>  > The ideal situation would be to have\n>  > git just manage the version control without having to babysit it, of\n>  > course.\n>\n> I can understand the motivation to write such disparaging remarks;\n>  however it may be more productive to come up with good ideas about how\n>  it can be made to work better for you, without getting in the way of\n>  other users.  patches are even better!\n\nI didn't mean anything disparaging.  I have nothing against babysitters :)\n\nI'll be happy to work on patches once we have some sort of consensus\non what would be desirable.  I think we're slowly getting there.\n\n>  >>  If I understand you correctly, you want to be forced to create a\n>  >>  branch and push to that?  I don't think that works well with many\n>  >>  developers pushing to a shared repository (my situation),\n>  >\n>  > Hmm, this is curious.  If you're *not* using submodules, then I don't\n>  > think you can push successfully without being on a branch, can you?\n>\n> Sure, you could;\n>\n>   git push origin HEAD:branchname\n\nOkay, yes.  But that's just arbitrarily avoiding a local branch and\ncreating a remote one instead.  I can't imagine a situation where\nyou'd really want the local branch to be anonymous while the remote\none is not.\n\nWhen doing a normal \"git clone\" without submodules, git automatically\ncreates you a local branch with the same name as the remote's\n.git/HEAD - which is rather arbitrary, but even an arbitrary local\nname is better than no name, and when checking out a brand new\nsubmodule, there are *no* local branches, so a name conflict is\nimpossible.\n\n>  > If you 'git checkout -b branchname' inside a submodule, then 'git\n>  > push' will do the right thing, so I'm not sure what you'd want to be\n>  > more automagical than that.\n>\n> Well, where did you get the branch name from?  That's the part that\n>  requires user intervention.  You could make an educated guess, such as\n>  with git name-rev, but it would not necessarily be the right guess - so\n>  user confirmation of the choice would be desirable.\n\nHere's a paraphrase of what I suggested earlier.  I don't think it got\na response:\n\nInstead of storing only the commitid of each submodule in the parent\ntree, store the current branch name as well.  Use this as a hint to\n'submodule update' so that when it checks out commitid, it names the\nlocal branch with the same name as it used to have.  (This is rather\nuser-friendly since if I check in, push, and clone, my new submodule\ncheckout will have the same branchname as it used to have.)\n\nNote that the newly checked-out submodule branch will probably have\nthe same name as as remote branch.  However, the remote branch may\nrefer to a different commitid (for example, if someone has pushed to\nthat branch after the parent repo was last updated).  This is exactly\nright; it means that if I cd into the submodule and \"git push\", it'll\nfail because I'm not up to date (I can always switch to a new branch\nif I want), and if I \"git pull\", it'll pull from the place where it\nshould.\n\nThis way, cloning a project with submodules will work much like\ncloning the parent project; pushing and pulling the parent and the\nsubmodules will do as you expect.\n\nThe bad news is that this would require a change to the tree format\nfor submodules (to contain the branch name).  Is that a problem?  Can\nit be done in a backwards-compatible way?\n\nAlso, I think this is the only time I've seen a branch name in the\ncommit/tree structure, which normally refers only to sha-1 hashes.  Is\nthat a problem?  Is there a better way?\n\nThanks,\n\nAvery\n"},{"id":"73522","messageId":"7vd4p9ccw8.fsf@gitster.siamese.dyndns.org","threadId":"12915","inReplyTo":"32541b130804011656l2e907895i98e5260b49743bbe@mail.gmail.com","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-04-02T00:35:03Z","receivedAt":"2008-04-02T00:35:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Avery Pennarun\" <apenwarr@gmail.com> writes:\n\n> Instead of storing only the commitid of each submodule in the parent\n> tree, store the current branch name as well.\n> ...\n> This way, cloning a project with submodules will work much like\n> cloning the parent project; pushing and pulling the parent and the\n> submodules will do as you expect.\n\nThat goes quite against the fundamental design of git submodules in that\nthe submodules are by themselves independent entities.  An often-cited\nexample is an appliance project, where superproject bundles a clone of\nLinux kernel and a clone of busybox repositories as its submodules.\n\nEach submodule is an independent project, and as such, must not know\nanything about the containing superproject (iow, the superproject can know\nwhat the submodules are doing, but submodules should not know they are\ncontained within a particular superproject).\n\nIf your superproject (i.e. the appliance product) uses two branches to\nmanage two product lines, named \"v1\" and \"v2\", these names are local to\nthe superproject.  It should not force the projects you borrow your\nsubmodules from to have branches with corresponding name.\n\nAlso, the submodules and the superproject are meant to be loosely coupled.\nA single branch in superproject (say \"v1\") may have many branches in a\nsubmodule (\"add frotz to v1 product\", \"improve nitfol in v1 product\") that\ncan potentially be merged and bound to.\n\nThe work flow for updating a tree would look like:\n\n - People \"git fetch\" in superproject and in its submodules. They\n   obviously prime their tree with \"git clone\" of superproject, and the\n   submodules they are interested in, and a single fetch will update all\n   the remote tracking branches, so it does not really matter which branch\n   is checked out.  However, if you employ a central repository model to\n   keep them, an invariant must hold: all the necessary commits in\n   submodules must be _reachable_ from some branch in them.\n\n - When not working in a particular submodule, but using it as a component\n   to build the superproject, it would be better to leave its HEAD\n   detached to the version the superproject points at.  IOW, usually you\n   won't have to be on any branch in submodules unless you are working in\n   them.\n\n - Sometimes you need to work in a submodule; e.g. you would want to add\n   'frotz' tool to your copy of busybox.  You chdir to the submodule\n   directory, and develop as if there is no superproject.\n\n   - Most likely, because the superproject and the submodules are meant to\n     be tied loosely, the (detached) HEAD you see in the submodule\n     together with the superproject may be _behind_ the tip of any\n     branches in the submodule repository.  This is really up to the\n     project's policy, but the superproject is meant to point at a commit\n     from submodules that is proven to be robust, and it may even be the\n     responsibility of people above you to decide when and to what commit\n     to advance the commit the superproject points at, and developers are\n     given a single \"development\" branch to work on.  The project policy\n     might be \"a new feature should be on a new topic branch forked at the\n     commit the superproject points at, and let the project leader to know\n     what the topic is about so that it can be merged to the integration\n     branch\".  Follow the project policy and check out the branch to work\n     on.\n\n   - Then work on adding that 'frotz' tool.  Make commits, test it in\n     isolation and test with superproject.  Push it out as whichever\n     remote branch the project policy asks you to.\n\n - If you are the person who decides which commit in a submodule should be\n   pointed at by the tree of the superproject, then check out the\n   superproject, and check out the submodule at that commit.  Update the\n   gitlink for the submodule and make a commit in the superproject.  This\n   step may happen at the same time as the previous step if you worked in\n   the submodule to advance its development branch tip and if you are the\n   person who decides which submodule commit to point at from the\n   superproject.\n\n - At any step above, a new clone (or a previously cloned trees, updated\n   with \"git fetch\" in appropriate places) will have all necessary\n   objects.\n"},{"id":"73526","messageId":"32541b130804011903u3b5d0794w984e26c4570c4688@mail.gmail.com","threadId":"12915","inReplyTo":"7vd4p9ccw8.fsf@gitster.siamese.dyndns.org","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-04-02T02:03:50Z","receivedAt":"2008-04-02T02:03:50Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 4/1/08, Junio C Hamano <gitster@pobox.com> wrote:\n> \"Avery Pennarun\" <apenwarr@gmail.com> writes:\n>  > Instead of storing only the commitid of each submodule in the parent\n>  > tree, store the current branch name as well.\n> > ...\n> > This way, cloning a project with submodules will work much like\n>  > cloning the parent project; pushing and pulling the parent and the\n>  > submodules will do as you expect.\n>\n> That goes quite against the fundamental design of git submodules in that\n>  the submodules are by themselves independent entities.\n\nNot sure what you mean here; the supermodule already stores the\ncommitid of the submodule.  All I'm proposing is that it also store\nthe default branchname (ie. the branchname that the submodule was\nusing when its gitlink was checked into the supermodule) along with\nthat commitid.  The submodule never knows anything about the\nsupermodule.\n\n>  An often-cited\n>  example is an appliance project, where superproject bundles a clone of\n>  Linux kernel and a clone of busybox repositories as its submodules.\n\nWhat a coincidence!  This is almost exactly like my situation :)\n\n>  If your superproject (i.e. the appliance product) uses two branches to\n>  manage two product lines, named \"v1\" and \"v2\", these names are local to\n>  the superproject.  It should not force the projects you borrow your\n>  submodules from to have branches with corresponding name.\n\nI meant that we should store the submodule's branch name when\ncommitting the superproject, and put it back when checking out the\nsubmodule fresh from the superproject.\n\n>   - When not working in a particular submodule, but using it as a component\n>    to build the superproject, it would be better to leave its HEAD\n>    detached to the version the superproject points at.  IOW, usually you\n>    won't have to be on any branch in submodules unless you are working in\n>    them.\n\nI agree that the submodule should have its HEAD pointing at exactly\nthe superproject-specified commit.  However, I believe this commit\nshould have a local branch name (in the subproject) attached to it, or\nelse (as I and my co-workers have frequently experienced) people will\naccidentally check in to a nameless branch, causing 'git push' to\nsilently not upload anything, and thus lose track of their commits.  I\nhave lost work this way.\n\nThe idea of naming the local-subproject-branch with the same name as\nit had on checking is that then \"git pull\" in the subproject will work\nexactly as expected: it'll get you the latest version of the branch\nthe superproject developer was on.  But if you *don't* explicitly \"git\npull\" in the subproject, I'd expect (of course) the checkout to stick\nto the commit specified by the superproject - and also to leave its\nlocal branch name pointing at exactly that commit.\n\n>   - Sometimes you need to work in a submodule; e.g. you would want to add\n>    'frotz' tool to your copy of busybox.  You chdir to the submodule\n>    directory, and develop as if there is no superproject.\n\nThis is where my workflow is a bit different.  One of my subprojects\nis a library that gets used by several application superprojects.  I\noften add features to my library in the process of editing a\nparticular superproject.  I also expect my co-developers to want to do\nthe same.  Thus, the difference from your example is that I want to\nstreamline the process of working in a subproject as well as a\nsuperproject, and minimize the chances of losing data in this case.\n\nWith the current system the way it is, it's too easy to make mistakes,\nand it requires too many steps to fetch/merge/rebranch each submodule.\n\n>    - Then work on adding that 'frotz' tool.  Make commits, test it in\n>      isolation and test with superproject.  Push it out as whichever\n>      remote branch the project policy asks you to.\n\nAs an orthogonal secondary wish, I'd like to have the subproject and\nsuperproject hosted in the same remote repository.  This appears to be\npossible (albeit inefficiently right now) by using \".\" as the remote\nrepo name in .gitmodules.  It would be more efficient if\ngit-submodule-update would use the superproject's checkout as a\n--alternate when cloning the submodule... I think that would be easy\nand harmless, right?\n\nThe super-summary of all that is I think I'd like to make three git\nchanges here:\n\n1) When checking out a submodule from scratch, use the local\nsupermodule as a --alternate.  That way if both super and submodule\nare hosted in the same remote repo, I don't have to clone them twice.\n(And cloning my local repo to another copy doesn't stop git-submodule\nfrom working.)\n\n2) When checking out a submodule, give the submodule's current commit\na useful branch name (ideally, the name it had when the gitlink was\nchecked into the supermodule).  When updating a submodule with\ngit-submodule-update, quietly fixup the submodule's local branch ref\nif it hasn't been changed; else produce a conflict of some sort.\n\n3) Bonus: make \"git push\" operate recursively on submodules, and \"git\npull\" automatically run git-submodule-update.\n\nDoes that make sense?\n\nThanks,\n\nAvery\n"},{"id":"73562","messageId":"47F3E74C.5010804@vilain.net","threadId":"12915","inReplyTo":"32541b130804011903u3b5d0794w984e26c4570c4688@mail.gmail.com","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2008-04-02T20:06:36Z","receivedAt":"2008-04-02T20:06:36Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Avery Pennarun wrote:\n>>  > Instead of storing only the commitid of each submodule in the parent\n>>  > tree, store the current branch name as well.\n>> That goes quite against the fundamental design of git submodules in that\n>>  the submodules are by themselves independent entities.\n>>     \n> Not sure what you mean here; the supermodule already stores the\n> commitid of the submodule.  All I'm proposing is that it also store\n> the default branchname (ie. the branchname that the submodule was\n> using when its gitlink was checked into the supermodule) along with\n> that commitid.  The submodule never knows anything about the\n> supermodule.\n>   \n\nHow about this.\n\nThis could be an optional disambiguator in .gitmodules in the \nsuperproject, to allow you to \"store the branch it was made from\".  Glue \nto make this automatic/easy optional.\n\nWhen updating a submodule, with an option set (or configured; which \nmight even later become a default if people like it enough), it will try \nto figure out a reasonable branch for that commit, using git-name-rev, \nand check out the branch with that name.  It first uses the hint above \nas an argument to git-name-rev --refs=XX, and if that doesn't provide a \nreasonable answer then look for any branch.\n\nI think this approach would not get in the way of people who don't want \nto use it, and provide for making things automatic for those who want that.\n\n> I agree that the submodule should have its HEAD pointing at exactly\n> the superproject-specified commit.  However, I believe this commit\n> should have a local branch name (in the subproject) attached to it, or\n> else (as I and my co-workers have frequently experienced) people will\n> accidentally check in to a nameless branch, causing 'git push' to\n> silently not upload anything, and thus lose track of their commits.  I\n> have lost work this way.\n>   \n\nI think this is a separate argument against git-push, the default \nbehaviour of which also causes me to tell people not to use the \nargument-less form of git-push until they understand how to use the \ntwo-argument form.\n\nIn the context of git-submodule, adding features to it to avoid this if \nthat is the way that the user wants to work would be good IMHO.\n\n> The idea of naming the local-subproject-branch with the same name as\n> it had on checking is that then \"git pull\" in the subproject will work\n> exactly as expected:\n\nIt's not really the local branch name anyway, it's how the default push \ngets configured; perhaps it's worth distinguishing which part you are \ninterested in.\n\n> The super-summary of all that is I think I'd like to make three git\n> changes here:\n>\n> 1) When checking out a submodule from scratch, use the local\n> supermodule as a --alternate.  That way if both super and submodule\n> are hosted in the same remote repo, I don't have to clone them twice.\n> (And cloning my local repo to another copy doesn't stop git-submodule\n> from working.)\n>   \n\nYes - you've already seen the SoC plan for that, although I believe no \nstudents applied for that one, and if you think it's minor enough to do, \ngreat.\n\n> 2) When checking out a submodule, give the submodule's current commit\n> a useful branch name (ideally, the name it had when the gitlink was\n> checked into the supermodule).  When updating a submodule with\n> git-submodule-update, quietly fixup the submodule's local branch ref\n> if it hasn't been changed; else produce a conflict of some sort.\n>   \n\nSee above for how I think that should work.\n\n> 3) Bonus: make \"git push\" operate recursively on submodules, and \"git\n> pull\" automatically run git-submodule-update.\n>   \n\nI'd appreciate that feature - though I'm more interested in making sure \nthat I don't push anything where the submodule commit is not available \nvia the URL listed in .gitmodules.\n\nPresuming such a check, would that check happen at push time, or do you \ncheck at a different time, such as when committing, or when adding the \nsubmodule to the index?\n\nI think checking that referential integrity is something perhaps easier \nto bite off and get people to agree on.  I think it would solve the \noverall process problem, by forcing people to push the submodule before \nthe commit of the superproject can succeed without forcing.\n\nThoughts/comments?\nSam\n"},{"id":"73568","messageId":"7v7ifg6iz6.fsf@gitster.siamese.dyndns.org","threadId":"12915","inReplyTo":"C38585A9-F09C-4A5B-8E72-6F3127DB2BB9@orakel.ntnu.no","subject":"Re: git-submodule getting submodules from the parent repository","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-04-02T21:32:29Z","receivedAt":"2008-04-02T21:32:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eyvind Bernhardsen <eyvind-git@orakel.ntnu.no> writes:\n\n> One solution that occurred to me was to have a branch in each\n> submodule for every main module and branch.  A branch name would be\n> provided for each submodule in .gitmodules, used by \"submodule push\"\n> but not \"submodule update\".  In this case, if the push to the branch\n> fails, the main module branch is probably behind too.\n>\n> This seemed like a good idea, but it's racy.\n\nIt's not just racy, but I think it's wrong to limit to _one_ branch in\neach submodule..\n\nA submodule is an independent project on its own.\n\nSuppose the commit DAG in the submodule looked like this:\n\n                 o---o\n                /     \\\n     --o---o---o---o---o---X---o---Z\n            \\                 /\n             o---o---o---o---o---o\n                  \\     /\n                   o---o\n\nand the superproject points at commit X. You may need to tweak the\nsubmodule to make it work better with the change you are making to the\nsuperproject.\n\nYou have two choices:\n\n (1) update to some \"stable\" branch head that is descendant of X first,\n     and make sure it works with the superproject.  Then develop on top of\n     it, and bind the tip of suc development trail to the superproject:\n\n                 o---o\n                /     \\\n     --o---o---o---o---o---X---o---Z---o---o---Y (your changes are Z..Y)\n            \\                 /\n             o---o---o---o---o---o\n                  \\     /\n                   o---o\n\nI think this is what you are suggesting.  But the superproject may not be\nready to use the submodule with the history from the lower side branch\nmerged in.  You would\n\n (2) fork off of X and develop; bind the tip of such development trail to\n     the superproject.  IOW, you make the submodule DAG like this, and\n     then \"git add\" commit Y in superproject.\n\n                 o---o       o---o---Y (your changes)\n                /     \\     /\n     --o---o---o---o---o---X---o---Z\n            \\                 /\n             o---o---o---o---o---o\n                  \\     /\n                   o---o\n\nSometimes forked branches need to be maintained until it proves stable\n(and then your \"tip\" Y may be merged back to the tip of a public branch\nZ).  So you would at least need to allow a set of topic branches in\nsubmodules that corresponds to a single lineage of superproject history.\n\nThen when both Z (with the changes from the lower side branch) and Y (your\nchanges) prove stable, the submodule project may decide to make a merge\nbetween Y and Z.\n\n                 o---o       o---o---Y (your changes)\n                /     \\     /         \\\n     --o---o---o---o---o---X---o---Z---W\n            \\                 /\n             o---o---o---o---o---o\n                  \\     /\n                   o---o\n\nThe superproject may decide to \"git add\" the result of such a merge, but\nthat decision is done separately (and obviously after such a merge is\nmade).\n\nIf everybody involved in the superproject forked from whatever happened to\nbe bound to the superproject, however, the submodule will have\nuncontrolled number of unmerged \"tips\".  For the submodule to stay viable\nas an independent project, some management has to be done to clean up this\nmess.  To manage the forked development inside submodule properly, I do\nnot think you can autocruise from the toplevel superproject and leave the\nrandom branches or unnamed commits unmerged.\n\nThat is why I suggested to:\n\n * Leave the HEAD in submodule detached, if you are not working in it;\n\n * Have a project policy regarding the use of branches in the submodule.\n   When you need to work on submodule, first switch to the branch (the\n   policy may allow/encourage you to create a new topic branch here), and\n   commit to it.\n\n * The policy should also say when these forked branches should be merged;\n   keep them tidy by following that policy.\n\nAnd by pushing from submodule and then in toplevel, you will never have\n\"superproject names a commit unreachable from any of the branch tips of\nsubmodules\" problem.  Nor there is any raciness issue -- only after you\npush out the submodule successfully, you push out the toplevel (and if the\nformer fails, you may need to redo the toplevel commit, but that happens\nbefore you push it out so you can afford to rebase or amend).\n"}]}