{"thread":{"id":"27116","subject":"How to manage multiple repos using submodules?","startedAt":"2011-04-16T16:45:27Z","lastAt":"2011-04-19T13:18:11Z","messageCount":5,"participants":["Andrew Wong","Jonathan Nieder","Phil Hord"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"165969","messageId":"4DA9C7A7.4010503@sohovfx.com","threadId":"27116","inReplyTo":null,"subject":"How to manage multiple repos using submodules?","fromName":"Andrew Wong","fromEmail":"andrew.w@sohovfx.com","sentAt":"2011-04-16T16:45:27Z","receivedAt":"2011-04-16T16:45:27Z","isPatch":false,"sender":{"key":"andrew.w@sohovfx.com","avatar":null},"body":"It seems like submodule isn't meant for this, but many people seems to \nuse submodule to link many smaller repos together. With this setup, I \nimagine whenever someone pushed a small repo, they're /supposed/ to push \nthe big repo as well. This way, if I simply update the big repo and do a \n\"git status\", git will tell me that which of the smaller repos are out \nof date. However, this is only reliable if everyone remembers to push \nthe big repo. If someone pushed a smaller repo, but forgot to push the \nbig repo, then I won't be aware that some of the smaller repos are \nout-of-date until I push.\n\nI suppose one way is to do a check/auto-update with pre/post commit \nscript to enforce that the big repo is always up-to-date. Another way is \nto use the \"submodule foreach\" to do a fetch and status on smaller ones, \nbut this doesn't seem very elegant.\n\nSo, I'm wondering how do people who use submodules this way manage all \ntheir repos? Or how to (reliably) get a good sense of the state of the \nsmall repos? Or maybe I shouldn't be using submodules this way at all?\n"},{"id":"165979","messageId":"20110416182053.GA11017@elie","threadId":"27116","inReplyTo":"4DA9C7A7.4010503@sohovfx.com","subject":"Re: How to manage multiple repos using submodules?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-04-16T18:20:54Z","receivedAt":"2011-04-16T18:20:54Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Andrew,\n\nAndrew Wong wrote:\n\n> It seems like submodule isn't meant for this, but many people seems\n> to use submodule to link many smaller repos together. With this\n> setup, I imagine whenever someone pushed a small repo, they're\n> /supposed/ to push the big repo as well. This way, if I simply\n> update the big repo and do a \"git status\", git will tell me that\n> which of the smaller repos are out of date.\n\nYep, if you want to keep track of the state of a bunch of repos over\ntime, submodules are not so bad[*].  In practice, often one instead\nwants to keep a bunch of repos up-to-date, and all this meta-history\ntracking is overkill.\n\nI'd suggest using the mr tool.  Some projects (e.g., the\ndebian-installer project) are using it to help people keep all their\nrepos up to date.\n\nhttp://kitenet.net/~joey/code/mr/\n\nHope that helps,\nJonathan\n"},{"id":"166005","messageId":"20110417064818.GA25344@elie","threadId":"27116","inReplyTo":"20110416182053.GA11017@elie","subject":"Re: How to manage multiple repos using submodules?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-04-17T06:48:19Z","receivedAt":"2011-04-17T06:48:19Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJonathan Nieder wrote:\n\n> Yep, if you want to keep track of the state of a bunch of repos over\n> time, submodules are not so bad[*].\n\nA kind person pointed out that I left out a footnote.  I think all I\nhad been planning to say is that, roughly speaking, submodules are\nabout[1] saying that a specific commit is known to work well with the\nrest of the code.  A supermodule like the one discussed in [2] is only\nlikely to be useful if you are interested in what historical\ncombinations of repositories were published and meant to work well\ntogether.\n\nCiao,\nJonathan\n\n[1] e.g., http://thread.gmane.org/gmane.comp.version-control.git/27803/focus=27830\n[2] http://lists.x.org/archives/xorg-devel/2009-September/001966.html\n"},{"id":"166015","messageId":"4DAB3495.3090200@sohovfx.com","threadId":"27116","inReplyTo":"20110417064818.GA25344@elie","subject":"Re: How to manage multiple repos using submodules?","fromName":"Andrew Wong","fromEmail":"andrew.w@sohovfx.com","sentAt":"2011-04-17T18:42:29Z","receivedAt":"2011-04-17T18:42:29Z","isPatch":false,"sender":{"key":"andrew.w@sohovfx.com","avatar":null},"body":"Thanks for the mail archives! They were a very good read.\n\nThe smaller projects I have are fairly independent, so there isn't a \nsituation where a commit works well with the other commits. I just \nwanted some ways to split each project out to its own repo. So that when \nI want to do some git operations on one project, I don't have to worry \nabout the other projects. While submodules isn't an ideal solution, it \nseems to be the closest. Maybe what I need from submodules is a way for \nthe super-repo to not record the commit of the sub-repos. i.e. Just use \nthe head of a branch. But if that's the case, maybe it's out of the \nscope of a SCM, since I'm not really tracking a history anymore.\n\nI haven't tried it yet, but the mr tool you mentioned seems interesting \ntoo. I'll check it out.\n\nThanks!\nAndrew\n\n\nOn 11-04-17 2:48 AM, Jonathan Nieder wrote:\n> Hi,\n>\n> Jonathan Nieder wrote:\n>\n>> Yep, if you want to keep track of the state of a bunch of repos over\n>> time, submodules are not so bad[*].\n> A kind person pointed out that I left out a footnote.  I think all I\n> had been planning to say is that, roughly speaking, submodules are\n> about[1] saying that a specific commit is known to work well with the\n> rest of the code.  A supermodule like the one discussed in [2] is only\n> likely to be useful if you are interested in what historical\n> combinations of repositories were published and meant to work well\n> together.\n>\n> Ciao,\n> Jonathan\n>\n> [1] e.g., http://thread.gmane.org/gmane.comp.version-control.git/27803/focus=27830\n> [2] http://lists.x.org/archives/xorg-devel/2009-September/001966.html\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"166083","messageId":"4DAD8B93.1040707@cisco.com","threadId":"27116","inReplyTo":"20110416182053.GA11017@elie","subject":"Re: How to manage multiple repos using submodules?","fromName":"Phil Hord","fromEmail":"hordp@cisco.com","sentAt":"2011-04-19T13:18:11Z","receivedAt":"2011-04-19T13:18:11Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"Thanks, Jonathan.\n\nOn 04/16/2011 02:20 PM, Jonathan Nieder wrote:\n> Hi Andrew,\n>\n> Andrew Wong wrote:\n>\n>> It seems like submodule isn't meant for this, but many people seems\n>> to use submodule to link many smaller repos together. With this\n>> setup, I imagine whenever someone pushed a small repo, they're\n>> /supposed/ to push the big repo as well. This way, if I simply\n>> update the big repo and do a \"git status\", git will tell me that\n>> which of the smaller repos are out of date.\n> Yep, if you want to keep track of the state of a bunch of repos over\n> time, submodules are not so bad[*].  In practice, often one instead\n> wants to keep a bunch of repos up-to-date, and all this meta-history\n> tracking is overkill.\n\nI had been thinking about this a lot lately.  We are using submodules\nsomewhat extensively to manage a project with lots of common code\nfeeding onto 12 different platforms (various OSes, compilers and SOCs). \nIt all feels quiet unwieldy but, at the same time, necessary.\n\nI looked briefly at the android repo tool, but it's not just where I\nwant it to be either. Also, it loses history (by design, though that\nseems to be a mistake they intend to correct someday).\n\nBut all this thinking got me closer to understanding that there are two\ncompeting needs here, as you both pointed out:\n  1. Point-in-time history using SHA-1s (the git model)\n  2. Branch following (the 'repo', mr, gitslave or svn model)\n\nWhen I am looking for a bug introduced in the past, I want to have the\nfirst model.  When I am eager to pull in bug fixes, new features, or a\ndifferent meta-branch from other developers (and possibly other\nprojects), I want the second model.\n\nSubmodules make it difficult to manage both at the same time.   It\nrequires extra steps to do what it should be doing by default (acting\nlike git, tracking project point-in-time history).  I understand why it\nis like this, but empathy doesn't ease the pain.\n\nBut I really do want both models.  I want point-in-time history, and I\nwant semi-automatic branch following.  Maybe something like this:\n\n1. Easily record point-in-time history\n   # Create and switch to a branch on superproject and all submodules\n   git checkout -b topic-foo --recursive\n\n   submodule A/topic-foo:  hack, hack, hack\n   submodule B/topic-foo:  hack, hack, hack\n\n   # Commit and push changes on each submodule\n   git commit --recursive\n   git push origin HEAD:ph/foo --recursive\n\n2. Ignore the gitlinks and just checkout master everywhere\n   git checkout master --recursive\n  \n\nI think I can do both with just a few tweaks of the submodule\nmachinery.  I haven't had time to spelunk there much, but I see others\nare working to advance submodules in somewhat similar directions. \n(Thanks for that; it is much appreciated.)\n\nHave these dueling modes been discussed into the dirt somewhere\nalready?  I find lots of discussions about other tools to perform model\n2 instead of model 1, but I don't see much discussion about the\nsimultaneous need for both.   Maybe I am overthinking it.  Or maybe my\nparticular code model is unusual.\n\n> I'd suggest using the mr tool.  Some projects (e.g., the\n> debian-installer project) are using it to help people keep all their\n> repos up to date.\n>\n> http://kitenet.net/~joey/code/mr/\n\nThanks.  I hadn't seen that one before.  I'll take a look.\n\nPhil\n"}]}