{"thread":{"id":"19597","subject":"Managing submodules on large multi-user projects","startedAt":"2009-05-29T18:41:25Z","lastAt":"2009-06-23T22:58:28Z","messageCount":7,"participants":["R. Tyler Ballance","Avery Pennarun","Felipe Contreras","Alex Riesen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"115039","messageId":"20090529184125.GE11222@starfruit.corp.slide.com","threadId":"19597","inReplyTo":null,"subject":"Managing submodules on large multi-user projects","fromName":"R. Tyler Ballance","fromEmail":"tyler@slide.com","sentAt":"2009-05-29T18:41:25Z","receivedAt":"2009-05-29T18:41:25Z","isPatch":false,"sender":{"key":"tyler@slide.com","avatar":null},"body":"As some of you may recall from my last swath of emails to the list\nregarding memory usage and repository size, we have quite a large\nrepository. About a month ago, I added a submodule to the primary repo\nin an effort to start to segment where possible, particularly around\nthird party modules.\n\nI've noticed that keeping submodules updated is an absolute pain,\nparticularly with a large multiuser setup with *lots* of branches. \n\n\nWhat will tend to happen is that the submodule reference will be updated\nin the master branch (we use a centralized model) and then committed \n(imagine the commit reference was incremented from A-B).\n\nOther developers with other branches will then periodically merge master \ninto their project/topic branches but will either neglect to run \n`git submodule update` or our bootstrap script (which also executes the\nsubmodule update command). At this point they'll have outstanding\nchanges of their own, and the submodule will be marked as \"modified\" as\nwell. Usually what will then happen is they'll `git commit -a` without\nthinking and the submodule's reference will be changed (typically from\nB->A, undoing the previous change).\n\n\nAre there any saner ways of managing this? I've been trying to get the \n`git submodule update` command to run with as many hooks as possible\n(pre-commit, post-update) to make sure that developers aren't\ninadvertantly breaking things, but nothing seems to ensure that\n*everybody* is up to date and that *everybody* doesn't inadvertantly\ncommit changes to the submodule?\n\n\nFeeling trapped in a box of PEBKAC.\n\nCheers\n-- \n-R. Tyler Ballance\nSlide, Inc.\n"},{"id":"115040","messageId":"32541b130905291253k3fa1d675yde1dddb5e8090ef9@mail.gmail.com","threadId":"19597","inReplyTo":"20090529184125.GE11222@starfruit.corp.slide.com","subject":"Re: Managing submodules on large multi-user projects","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-05-29T19:53:26Z","receivedAt":"2009-05-29T19:53:26Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Fri, May 29, 2009 at 2:41 PM, R. Tyler Ballance <tyler@slide.com> wrote:\n> As some of you may recall from my last swath of emails to the list\n> regarding memory usage and repository size, we have quite a large\n> repository. About a month ago, I added a submodule to the primary repo\n> in an effort to start to segment where possible, particularly around\n> third party modules.\n>\n> I've noticed that keeping submodules updated is an absolute pain,\n> particularly with a large multiuser setup with *lots* of branches.\n\nJust so I understand, is the reason you're splitting into submodules\n*just* to avoid memory usage / repository size issues?  I can sort of\nunderstand the memory usage issues - sort of - but how does it reduce\nrepository size if you need to need to check out all the submodule\nrepositories along with the main project anyway?\n\nJust looking to clarify for myself.  (I'm continuing my work on\ngit-subtree, which is getting more and more positive feedback.  It\nsolves all the *other* problems that you listed vs. submodules, but it\ncertainly doesn't resolve any repository size issues.)\n\nHave fun,\n\nAvery\n"},{"id":"115042","messageId":"20090529200928.GH11222@starfruit.corp.slide.com","threadId":"19597","inReplyTo":"32541b130905291253k3fa1d675yde1dddb5e8090ef9@mail.gmail.com","subject":"Re: Managing submodules on large multi-user projects","fromName":"R. Tyler Ballance","fromEmail":"tyler@slide.com","sentAt":"2009-05-29T20:09:29Z","receivedAt":"2009-05-29T20:09:29Z","isPatch":false,"sender":{"key":"tyler@slide.com","avatar":null},"body":"On Fri, May 29, 2009 at 03:53:26PM -0400, Avery Pennarun wrote:\n> On Fri, May 29, 2009 at 2:41 PM, R. Tyler Ballance <tyler@slide.com> wrote:\n> > As some of you may recall from my last swath of emails to the list\n> > regarding memory usage and repository size, we have quite a large\n> > repository. About a month ago, I added a submodule to the primary repo\n> > in an effort to start to segment where possible, particularly around\n> > third party modules.\n> >\n> > I've noticed that keeping submodules updated is an absolute pain,\n> > particularly with a large multiuser setup with *lots* of branches.\n> \n> Just so I understand, is the reason you're splitting into submodules\n> *just* to avoid memory usage / repository size issues?  I can sort of\n> understand the memory usage issues - sort of - but how does it reduce\n> repository size if you need to need to check out all the submodule\n> repositories along with the main project anyway?\n\nI've got an eye on submodules as a way of avoiding the need to require a\nwhole tree clone to just work on parts of it, but that's not really\nrelevant to my query, just explaining our environment and setting the stage ;)\n\nWe're using submodules right now similar to how we used svn externals in\nthe past (except better, clearly), to incorporate outside components\n(like open source projects) that our stack depends on.\n\n> Just looking to clarify for myself.  (I'm continuing my work on\n> git-subtree, which is getting more and more positive feedback.  It\n> solves all the *other* problems that you listed vs. submodules, but it\n> certainly doesn't resolve any repository size issues.)\n\nGood to know, we're still on Git 1.6.1, are there any benefits or\nadditional features in more recent releases of Git that help alleviate\nthe submodules issues I outlined at the top of the thread?\n\n\nCheers\n-- \n-R. Tyler Ballance\nSlide, Inc.\n"},{"id":"115043","messageId":"32541b130905291318j2fb5188fk889a6dc9bfca37b3@mail.gmail.com","threadId":"19597","inReplyTo":"20090529200928.GH11222@starfruit.corp.slide.com","subject":"Re: Managing submodules on large multi-user projects","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-05-29T20:18:21Z","receivedAt":"2009-05-29T20:18:21Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Fri, May 29, 2009 at 4:09 PM, R. Tyler Ballance <tyler@slide.com> wrote:\n> On Fri, May 29, 2009 at 03:53:26PM -0400, Avery Pennarun wrote:\n>> Just so I understand, is the reason you're splitting into submodules\n>> *just* to avoid memory usage / repository size issues?  I can sort of\n>> understand the memory usage issues - sort of - but how does it reduce\n>> repository size if you need to need to check out all the submodule\n>> repositories along with the main project anyway?\n>\n> I've got an eye on submodules as a way of avoiding the need to require a\n> whole tree clone to just work on parts of it, but that's not really\n> relevant to my query, just explaining our environment and setting the stage ;)\n>\n> We're using submodules right now similar to how we used svn externals in\n> the past (except better, clearly), to incorporate outside components\n> (like open source projects) that our stack depends on.\n\nThat makes sense.\n\n>> Just looking to clarify for myself.  (I'm continuing my work on\n>> git-subtree, which is getting more and more positive feedback.  It\n>> solves all the *other* problems that you listed vs. submodules, but it\n>> certainly doesn't resolve any repository size issues.)\n>\n> Good to know, we're still on Git 1.6.1, are there any benefits or\n> additional features in more recent releases of Git that help alleviate\n> the submodules issues I outlined at the top of the thread?\n\ngit-subtree is my own little project that hasn't been accepted into\ngit proper yet.  It does work with git 1.6.1 (and also git 1.5.4, at\nleast) just by dropping the script into your PATH.  Google \"git\nsubtree\" for more.\n\nAFAIK, the particular issues you outlined with submodules continue to\nexist in latest git.  They are certainly fixable (they aren't\n*fundamental* problems), but nobody has fixed them yet.  I looked at\nthe issues for a long time and failed miserably to find a good general\nsolution, but that's just me.\n\nHave fun,\n\nAvery\n"},{"id":"115056","messageId":"94a0d4530905291558u564b4648ted79900b405f91b4@mail.gmail.com","threadId":"19597","inReplyTo":"20090529184125.GE11222@starfruit.corp.slide.com","subject":"Re: Managing submodules on large multi-user projects","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2009-05-29T22:58:03Z","receivedAt":"2009-05-29T22:58:03Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Fri, May 29, 2009 at 9:41 PM, R. Tyler Ballance <tyler@slide.com> wrote:\n> As some of you may recall from my last swath of emails to the list\n> regarding memory usage and repository size, we have quite a large\n> repository. About a month ago, I added a submodule to the primary repo\n> in an effort to start to segment where possible, particularly around\n> third party modules.\n>\n> I've noticed that keeping submodules updated is an absolute pain,\n> particularly with a large multiuser setup with *lots* of branches.\n>\n>\n> What will tend to happen is that the submodule reference will be updated\n> in the master branch (we use a centralized model) and then committed\n> (imagine the commit reference was incremented from A-B).\n>\n> Other developers with other branches will then periodically merge master\n> into their project/topic branches but will either neglect to run\n> `git submodule update` or our bootstrap script (which also executes the\n> submodule update command). At this point they'll have outstanding\n> changes of their own, and the submodule will be marked as \"modified\" as\n> well. Usually what will then happen is they'll `git commit -a` without\n> thinking and the submodule's reference will be changed (typically from\n> B->A, undoing the previous change).\n>\n>\n> Are there any saner ways of managing this? I've been trying to get the\n> `git submodule update` command to run with as many hooks as possible\n> (pre-commit, post-update) to make sure that developers aren't\n> inadvertantly breaking things, but nothing seems to ensure that\n> *everybody* is up to date and that *everybody* doesn't inadvertantly\n> commit changes to the submodule?\n\nHave you tried repo?\nhttp://source.android.com/download/using-repo\n\n-- \nFelipe Contreras\n"},{"id":"115132","messageId":"81b0412b0905310639i12440ae4i9331d57a752a6b96@mail.gmail.com","threadId":"19597","inReplyTo":"20090529184125.GE11222@starfruit.corp.slide.com","subject":"Re: Managing submodules on large multi-user projects","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2009-05-31T13:39:28Z","receivedAt":"2009-05-31T13:39:28Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"2009/5/29 R. Tyler Ballance <tyler@slide.com>:\n> Other developers with other branches will then periodically merge master\n> into their project/topic branches but will either neglect to run\n> `git submodule update` or our bootstrap script (which also executes the\n> submodule update command). At this point they'll have outstanding\n> changes of their own, and the submodule will be marked as \"modified\" as\n> well. Usually what will then happen is they'll `git commit -a` without\n> thinking and the submodule's reference will be changed (typically from\n> B->A, undoing the previous change).\n\nThis (the fact that \"git commit -a\" updates submodules in the index after merge)\nis probably our bug (or at least an unfinished feature).\n"},{"id":"116839","messageId":"20090623225828.GD734@starfruit.corp.slide.com","threadId":"19597","inReplyTo":"94a0d4530905291558u564b4648ted79900b405f91b4@mail.gmail.com","subject":"Re: Managing submodules on large multi-user projects","fromName":"R. Tyler Ballance","fromEmail":"tyler@slide.com","sentAt":"2009-06-23T22:58:28Z","receivedAt":"2009-06-23T22:58:28Z","isPatch":false,"sender":{"key":"tyler@slide.com","avatar":null},"body":"Hey Felipe, reply inline..\n\nOn Sat, 30 May 2009, Felipe Contreras wrote:\n\n> On Fri, May 29, 2009 at 9:41 PM, R. Tyler Ballance <tyler@slide.com> wrote:\n> > I've noticed that keeping submodules updated is an absolute pain,\n> > particularly with a large multiuser setup with *lots* of branches.\n> >\n> >\n> > What will tend to happen is that the submodule reference will be updated\n> > in the master branch (we use a centralized model) and then committed\n> > (imagine the commit reference was incremented from A-B).\n> >\n> > Other developers with other branches will then periodically merge master\n> > into their project/topic branches but will either neglect to run\n> > `git submodule update` or our bootstrap script (which also executes the\n> > submodule update command). At this point they'll have outstanding\n> > changes of their own, and the submodule will be marked as \"modified\" as\n> > well. Usually what will then happen is they'll `git commit -a` without\n> > thinking and the submodule's reference will be changed (typically from\n> > B->A, undoing the previous change).\n> >\n> >\n> > Are there any saner ways of managing this? I've been trying to get the\n> > `git submodule update` command to run with as many hooks as possible\n> > (pre-commit, post-update) to make sure that developers aren't\n> > inadvertantly breaking things, but nothing seems to ensure that\n> > *everybody* is up to date and that *everybody* doesn't inadvertantly\n> > commit changes to the submodule?\n> \n> Have you tried repo?\n> http://source.android.com/download/using-repo\n\nNo I've not tried repo, and the likelihood of getting our now 100+ user\norganization to switch over is highly unlikely.\n\nSince I originally posted to this thread, I've had to entirely *remove*\nthe submodule from the super-project and just dump the code in (boo,\nhiss) since it just caused too much damn trouble.\n\nI'm going to give a newer version of Git a try and hope that everythin\nis better now, since the need has arisen for a git submodule again and \nthings will get gnarly if I have to do another source dump.\n\n:(\n\n-R. Tyler Ballance\nSlide, Inc.\n"}]}