{"thread":{"id":"30257","subject":"organizing multiple repositories with dependencies","startedAt":"2012-04-16T09:27:12Z","lastAt":"2012-04-30T19:43:57Z","messageCount":34,"participants":["Namit Bhalla","Jakub Narebski","dag@cray.com","Hilco Wijbenga","Seth Robertson","PJ Weisberg","Jens Lehmann","Eugene Sajine","username localhost","Phil Hord"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"189374","messageId":"1334568432.53977.YahooMailNeo@web65906.mail.ac4.yahoo.com","threadId":"30257","inReplyTo":null,"subject":"organizing multiple repositories with dependencies","fromName":"Namit Bhalla","fromEmail":"namitbhalla@yahoo.com","sentAt":"2012-04-16T09:27:12Z","receivedAt":"2012-04-16T09:27:12Z","isPatch":false,"sender":{"key":"namitbhalla@yahoo.com","avatar":null},"body":"I am looking to track some projects using Git with each project as a \nseparate repository.\nEven after reading the documentation, I am still wondering if there is a \nway to organize things as described below.\n\nConsider 2 projects, Project-a and Project-b, which are housed in \nrepositories Repo-a and Repo-b respectively. \nProject-a develops reusable libraries which are needed by Project-b \n(otherwise Project-b will not compile).\nWhen a new stable version of Project-a libraries has to be delivered, they \nare \"checked into\" a path in Repo-a.\nNow, I would like to setup Repo-b so that when someone starts working on \nProject-b, he should be able to retrieve the code from Repo-b as well as the libraries from Repo-a. Is there any way to achieve that in \nGit?\n\nThanks for any pointers!\n"},{"id":"189406","messageId":"m3hawjagw9.fsf@localhost.localdomain","threadId":"30257","inReplyTo":"1334568432.53977.YahooMailNeo@web65906.mail.ac4.yahoo.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2012-04-16T14:30:17Z","receivedAt":"2012-04-16T14:30:17Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Namit Bhalla writes:\n\n> I am looking to track some projects using Git with each project as a \n> separate repository.\n> Even after reading the documentation, I am still wondering if there is a \n> way to organize things as described below.\n> \n> Consider 2 projects, Project-a and Project-b, which are housed in \n> repositories Repo-a and Repo-b respectively. \n> Project-a develops reusable libraries which are needed by Project-b \n> (otherwise Project-b will not compile).\n> When a new stable version of Project-a libraries has to be delivered, they \n> are \"checked into\" a path in Repo-a.\n> Now, I would like to setup Repo-b so that when someone starts working on \n> Project-b, he should be able to retrieve the code from Repo-b as well as the libraries from Repo-a. Is there any way to achieve that in \n> Git?\n\nPut reusable library in its own repository, and use submodules to link\nit up to project-a and project-b repositories.\n\nHTH\n-- \nJakub Narebski\n"},{"id":"189453","messageId":"nng3983phhc.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"m3hawjagw9.fsf@localhost.localdomain","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-16T20:08:31Z","receivedAt":"2012-04-16T20:08:31Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Jakub Narebski writes:\n\n> Put reusable library in its own repository, and use submodules to link\n> it up to project-a and project-b repositories.\n\ngit-subtree is another option. It was recently merged into contrib/.\nWhether to use submodules or subtrees largely depends on the work style\nof your group and how coupled the projects are to each other.\nsubmodules requires a bit more day-to-day maintenance by the user (in my\nexperience) while with subtrees it's a bit more involved to push changes\nfrom the combined repository back to the separate repositories.\n\n -Dave\n"},{"id":"189535","messageId":"CAE1pOi1KnvRk4yxK8OQHi9h_ueNnh5Ar3tbKFBKTA69=Aje0TQ@mail.gmail.com","threadId":"30257","inReplyTo":"nng3983phhc.fsf@transit.us.cray.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-04-17T17:29:56Z","receivedAt":"2012-04-17T17:29:56Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 16 April 2012 13:08, wrote:\n> Jakub Narebski writes:\n>\n>> Put reusable library in its own repository, and use submodules to link\n>> it up to project-a and project-b repositories.\n>\n> git-subtree is another option.  It was recently merged into contrib/.\n> Whether to use submodules or subtrees largely depends on the work style\n> of your group and how coupled the projects are to each other.\n> submodules requires a bit more day-to-day maintenance by the user (in my\n> experience) while with subtrees it's a bit more involved to push changes\n> from the combined repository back to the separate repositories.\n\n(My reply below is based on my experience with Git and submodules from\nabout a year ago. I would really like to see better support for\nincluding separate repos in Git. It does not appear to be an easy nut\nto crack, though.)\n\nIf you really have only one or two libraries then submodules will work\njust fine but if you have quite a few (we had around 50 when we moved\naway from submodules) you will find it pretty much unworkable. It's\nfragile and hard to keep the submodules in synch with each other and\nthe umbrella project. Another problem is branches. Branches are per\nsubmodule but you want them for all submodules. You might want to look\ninto git-slave if you want to go this route. I haven't used\ngit-subtree so I can't comment on that. (I do not get the impression\nthat it really is a big step forward, though. I would be *very* happy\nto be proven wrong, though.)\n\nIn general, I do not think the blanket statement \"one repo per\nproject\" is good advice. If projects depend on each other they should\nbe in the same repo. At least with the current support in GIt for\nincluding separate projects. Please note that I'm not disagreeing with\nthe notion \"one repo per project\" itself. It's just not supported well\nenough to be feasible if you have a fairly large group of projects\nthat depend on each other.\n"},{"id":"189539","messageId":"nnglilunt6h.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"CAE1pOi1KnvRk4yxK8OQHi9h_ueNnh5Ar3tbKFBKTA69=Aje0TQ@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-17T17:51:02Z","receivedAt":"2012-04-17T17:51:02Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Hilco Wijbenga writes:\n\n> Another problem is branches. Branches are per submodule but you want\n> them for all submodules.\n\nBranches have a similar problem in git-subtree. It is one of the things\nI would like to improve in git-subtree going forward. I don't see any\nfundamental reason that some git-slave-like operations can't be included\nin git-subtree, though with a slightly different model.\n\n -Dave\n"},{"id":"189552","messageId":"201204171837.q3HIbbcW013784@no.baka.org","threadId":"30257","inReplyTo":"CAE1pOi1KnvRk4yxK8OQHi9h_ueNnh5Ar3tbKFBKTA69=Aje0TQ@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2012-04-17T18:37:37Z","receivedAt":"2012-04-17T18:37:37Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message , Hilco Wijbenga writes:\n\n On 16 April 2012 13:08, wrote:\n > Jakub Narebski writes:\n >> Put reusable library in its own repository, and use submodules to link\n >> it up to project-a and project-b repositories.\n\n If you really have only one or two libraries then submodules will work\n just fine but if you have quite a few (we had around 50 when we moved\n away from submodules) you will find it pretty much unworkable. [...]\n Branches are per submodule but you want them for all\n submodules. You might want to look into git-slave if you want to\n go this route.\n\n In general, I do not think the blanket statement \"one repo per\n project\" is good advice. If projects depend on each other they should\n be in the same repo. At least with the current support in GIt for\n including separate projects. Please note that I'm not disagreeing with\n the notion \"one repo per project\" itself. It's just not supported well\n enough to be feasible if you have a fairly large group of projects\n that depend on each other.\n\nAs you mentioned, this is exactly the environment that gitslave was\ndesigned for. It provides the flexibility to work on the subprojects\nas if they were standalone independent git repositories (which of\ncourse they are) or treat the entire superproject as one giant git\nrepository (with only a few cracks showing through). All gitslave\ncommands are just git commands (s/^git\\s/gits /) so training to use it is\nrather easy.\n\nUnlike with git-submodules there is no strict binding between the\nparent repo's commits and the sub-project's commits except at tag\nboundaries. This gives you the flexibility of person A saying that A\nis master and B is underneath it while person B says that B is master\nand A is underneath it (or alternatively you can also say that A\ninclude B plus whatever B includes). However, I would in general\nrecommend that the common library be factored out and be a child of A\nand B. gitslave makes it trivial to work with federated git\nrepositories, if you can handle only having binding between\nrepositories at tag boundaries.\n\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"189559","messageId":"CAE1pOi29dKd2LHW7MJ+TTN4HzFkOPFEyf7Sf2emSsBYm93uYUA@mail.gmail.com","threadId":"30257","inReplyTo":"201204171837.q3HIbbcW013784@no.baka.org","subject":"Re: organizing multiple repositories with dependencies","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-04-17T19:55:19Z","receivedAt":"2012-04-17T19:55:19Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 17 April 2012 11:37, Seth Robertson wrote:\n> In message , Hilco Wijbenga writes:\n>\n>    On 16 April 2012 13:08,   wrote:\n>    > Jakub Narebski writes:\n>    >> Put reusable library in its own repository, and use submodules to link\n>    >> it up to project-a and project-b repositories.\n>\n>    If you really have only one or two libraries then submodules will work\n>    just fine but if you have quite a few (we had around 50 when we moved\n>    away from submodules) you will find it pretty much unworkable. [...]\n>    Branches are per submodule but you want them for all\n>    submodules. You might want to look into git-slave if you want to\n>    go this route.\n>\n>    In general, I do not think the blanket statement \"one repo per\n>    project\" is good advice. If projects depend on each other they should\n>    be in the same repo. At least with the current support in GIt for\n>    including separate projects. Please note that I'm not disagreeing with\n>    the notion \"one repo per project\" itself. It's just not supported well\n>    enough to be feasible if you have a fairly large group of projects\n>    that depend on each other.\n>\n> As you mentioned, this is exactly the environment that gitslave was\n> designed for.  It provides the flexibility to work on the subprojects\n> as if they were standalone independent git repositories (which of\n> course they are) or treat the entire superproject as one giant git\n> repository (with only a few cracks showing through).  All gitslave\n> commands are just git commands (s/^git\\s/gits /) so training to use it is\n> rather easy.\n>\n> Unlike with git-submodules there is no strict binding between the\n> parent repo's commits and the sub-project's commits except at tag\n> boundaries.  This gives you the flexibility of person A saying that A\n> is master and B is underneath it while person B says that B is master\n> and A is underneath it (or alternatively you can also say that A\n> include B plus whatever B includes).  However, I would in general\n> recommend that the common library be factored out and be a child of A\n> and B.  gitslave makes it trivial to work with federated git\n> repositories, if you can handle only having binding between\n> repositories at tag boundaries.\n\nWell, since I seem to have pretty much everyone who is involved with\n\"subprojects\" (be it submodules, subtree, or gitslave) in this thread,\nI wanted to clarify what I meant with \"not supported well in Git\". To\nme, subproject support is one of the most important pieces missing in\nGit. Git submodules, subtree and gitslave all provide a part of what\nI'm looking for: non-invasive subproject support. I would like to be\nable to use normal Git commands on the umbrella project that \"trickle\ndown\" to the subprojects. If you work on a subproject (in its own\nrepo) then a subsequent pull in the umbrella project should bring this\nnew code into the umbrella project (assuming that would make sense\ngiven the branches involved).\n\nAfter rereading my earlier reply I felt that it might be interpreted\nas being disparaging of submodules/subtree/gitslave. I just wanted to\nmake clear that that was not my intent. I'm hopeful that we can get\nsome sort of combination of submodules, subtree and gitslave to\nprovide subproject support in Git.\n"},{"id":"189568","messageId":"nng1unmnksx.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"CAE1pOi29dKd2LHW7MJ+TTN4HzFkOPFEyf7Sf2emSsBYm93uYUA@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-17T20:51:58Z","receivedAt":"2012-04-17T20:51:58Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Hilco Wijbenga writes:\n\n> If you work on a subproject (in its own repo) then a subsequent pull\n> in the umbrella project should bring this new code into the umbrella\n> project (assuming that would make sense given the branches involved).\n\nI don't necessarily think this is always what should happen. I can't\ncomment on git-submodule since I haven't used it in its more recent\nincarnation, but one thing I like about git-subtree is that it's\nexplicit. I have to do a \"git subtree pull\" on the umbrella project to\npull in the new changes from a subproject. That gives me some degree of\ncontrol over when to update sources. I suspect one can do the same by\nusing \"git pull\" in submodule directories.\n\nIf you want the behavior you describe, a post-receive hook on the\ncomponent repositories is easy to implement. I just did that a couple\nof weeks ago for a subtree-aggregated repository. When a component\nreceives something it immediately does a \"git subtree pull\" on a\nworkarea on the server and then does a push from that workarea to the\nsubtree-aggregated repository.\n\nOf course, this is entirely driven by git-subtree's model of actually\nincorporating subproject history into one big umbrella repository.\nThere is no separation between the subprojects and umbrella projects.\nIt's one giant history. Therefore, push/pull to/from subprojects are\nexplicit operations. That's probably not the best model for every\nsituation but I find it very nice.\n\n> After rereading my earlier reply I felt that it might be interpreted\n> as being disparaging of submodules/subtree/gitslave.\n\nI didn't interpret it that way at all. I agree with you that\nsubproject/superproject support could be much better. But I don't agree\nthat we'll be able to design one model that works for everyone. svn\nexternals are just one model to aggregate projects but it is not the\nonly one. It just happens that no one working on Subversion bothered to\nimplement anything else.\n\nPerhaps a good way to go would be to provide the basic operations (I\nthink we have most of that) and some hooks in contrib/ or elsewere to\nimplement various models. Just like git imposes no particular workflow\nmodel I don't think git should impose one particular aggregation model.\nWhat we do need is better documentation of what the various models and\ntools are. For example, I would find a subtree/submodule comparison\nhighly valuable. It would help people decide which model is best for\nthem.\n\n -Dave\n"},{"id":"189572","messageId":"CAE1pOi38krwXZuiYxtpLwm92N=NvWkP30V_=6cnHw=sdyk6QhA@mail.gmail.com","threadId":"30257","inReplyTo":"nng1unmnksx.fsf@transit.us.cray.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-04-17T21:43:37Z","receivedAt":"2012-04-17T21:43:37Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 17 April 2012 13:51, wrote:\n> Hilco Wijbenga writes:\n>\n>> If you work on a subproject (in its own repo) then a subsequent pull\n>> in the umbrella project should bring this new code into the umbrella\n>> project (assuming that would make sense given the branches involved).\n>\n> I don't necessarily think this is always what should happen.  I can't\n> comment on git-submodule since I haven't used it in its more recent\n> incarnation, but one thing I like about git-subtree is that it's\n> explicit.  I have to do a \"git subtree pull\" on the umbrella project to\n> pull in the new changes from a subproject.  That gives me some degree of\n> control over when to update sources.  I suspect one can do the same by\n> using \"git pull\" in submodule directories.\n\nI'm assuming that if you have subproject S in umbrella project U and a\nbranch \"topic\" in U then that same branch should exist in S. Any\nchanges in S's topic should show up in U's topic (probably after some\nsort of update command like git fetch/pull). This should be unusual,\nthough, you should be working in U, not S. If you want to work on\nsomething in S that you don't want to see in U, then you should not be\nworking in S's topic.\n\n> If you want the behavior you describe, a post-receive hook on the\n> component repositories is easy to implement.  I just did that a couple\n> of weeks ago for a subtree-aggregated repository.  When a component\n> receives something it immediately does a \"git subtree pull\" on a\n> workarea on the server and then does a push from that workarea to the\n> subtree-aggregated repository.\n\n[1] Would such a post-receive hook be something that the user has to\nset up? Or would that be automatically set up after git clone?\n\nThe main problem with the current submodule support is that there is\nso much manual work needed. It is too easy to forget a step. Moreover,\nit's not easy to determine *that* you forgot a step or which step you\nforgot.\n\n> Of course, this is entirely driven by git-subtree's model of actually\n> incorporating subproject history into one big umbrella repository.\n> There is no separation between the subprojects and umbrella projects.\n> It's one giant history.  Therefore, push/pull to/from subprojects are\n> explicit operations.  That's probably not the best model for every\n> situation but I find it very nice.\n\nI do not have enough (okay, any) experience with subtree to comment on\nthat. The first part seems just what I want. I'm not sure about the\nexplicit pushing/pulling part. That sounds too much like asking for\nthe sort of problems that scared us away from submodules. Hopefully,\nI'm dead wrong. :-)\n\n>> After rereading my earlier reply I felt that it might be interpreted\n>> as being disparaging of submodules/subtree/gitslave.\n>\n> I didn't interpret it that way at all.  I agree with you that\n> subproject/superproject support could be much better.\n\nGood. I just wanted to be extra clear because you (and others) are\nworking on something that is very important to me. The last thing I\nwant to do is discourage you. :-)\n\n> But I don't agree\n> that we'll be able to design one model that works for everyone.  svn\n> externals are just one model to aggregate projects but it is not the\n> only one.  It just happens that no one working on Subversion bothered to\n> implement anything else.\n\n:-) I think I made it pretty clear that I was listing what *I* want.\nWhat *I* am looking for is something that is as invisible and\nautomatic as possible.\n\n(I find working with Git really quite enjoyable but it has a very\nsteep learning curve. E.g., I have (literally) spent hours explaining\nrebase and merge to our new developers. Surprisingly, some come from\ncollege/university without having ever used an SCM tool but even for\nthose that have learning Git is quite a challenge. And Git's API isn't\nalways particularly helpful. The \"checkout\" command is a perfect (bad)\nexample in that regard. Even those that haven't used SVN/CVS before do\nnot associate \"checkout\" with switching branches. And using git\ncheckout to go back to the HEAD version of a file you've changed?\nSure, it can be explained and learned but it doesn't make automatic\nsense. What does switching branches have to do with undoing changes?\n[Yes, it makes sense given Git's implementation but *not* from the\nuser's point of view.]\n\nGiven that, I really do *not* want to pile on more just to accommodate\nsubprojects.)\n\n> Perhaps a good way to go would be to provide the basic operations (I\n> think we have most of that) and some hooks in contrib/ or elsewere to\n> implement various models.  Just like git imposes no particular workflow\n> model I don't think git should impose one particular aggregation model.\n> What we do need is better documentation of what the various models and\n> tools are.  For example, I would find a subtree/submodule comparison\n> highly valuable.  It would help people decide which model is best for\n> them.\n\nThat all sounds good. As long as the hooks are automatic (I'm hopeful\nyou said \"no\" and \"yes\" to [1] above). If so, then I can promise you\nI'll be taking a look at subtree. :-)\n"},{"id":"189584","messageId":"CAJsNXTmfRZvpO=ooB8yKUqGqbU4g5A78=dzt2vPPrs1q+J4ZrA@mail.gmail.com","threadId":"30257","inReplyTo":"CAE1pOi38krwXZuiYxtpLwm92N=NvWkP30V_=6cnHw=sdyk6QhA@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"PJ Weisberg","fromEmail":"pj@irregularexpressions.net","sentAt":"2012-04-17T22:25:30Z","receivedAt":"2012-04-17T22:25:30Z","isPatch":false,"sender":{"key":"pj@irregularexpressions.net","avatar":"https://gravatar.com/avatar/aa2c1edcc61b536cc5c9f37fbce084e655446e2309f9818d43f13d47304a602b?d=mp&s=160"},"body":"On Tue, Apr 17, 2012 at 2:43 PM, Hilco Wijbenga\n wrote:\n\n> I'm assuming that if you have subproject S in umbrella project U and a\n> branch \"topic\" in U then that same branch should exist in S. Any\n> changes in S's topic should show up in U's topic (probably after some\n> sort of update command like git fetch/pull). This should be unusual,\n> though, you should be working in U, not S. If you want to work on\n> something in S that you don't want to see in U, then you should not be\n> working in S's topic.\n\nThis paragraph makes me wonder why you want to use submodules at all.\nWouldn't a sparse checkout be a better fit for what you're trying to\naccomplish?\n\n-PJ\n\nGehm's Corollary to Clark's Law: Any technology distinguishable from\nmagic is insufficiently advanced.\n"},{"id":"189587","messageId":"CAE1pOi2FQtyahcJUJ+CZRLLYN7vxGaOmUiBJ-_AfC-T-ZEY_Sg@mail.gmail.com","threadId":"30257","inReplyTo":"CAJsNXTmfRZvpO=ooB8yKUqGqbU4g5A78=dzt2vPPrs1q+J4ZrA@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-04-17T22:49:59Z","receivedAt":"2012-04-17T22:49:59Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 17 April 2012 15:25, PJ Weisberg wrote:\n> On Tue, Apr 17, 2012 at 2:43 PM, Hilco Wijbenga\n> wrote:\n>\n>> I'm assuming that if you have subproject S in umbrella project U and a\n>> branch \"topic\" in U then that same branch should exist in S. Any\n>> changes in S's topic should show up in U's topic (probably after some\n>> sort of update command like git fetch/pull). This should be unusual,\n>> though, you should be working in U, not S. If you want to work on\n>> something in S that you don't want to see in U, then you should not be\n>> working in S's topic.\n>\n> This paragraph makes me wonder why you want to use submodules at all.\n> Wouldn't a sparse checkout be a better fit for what you're trying to\n> accomplish?\n\nNo, I don't think so but I could be wrong. I want to be able to easily\nbuild and release the individual projects separately (manually and on\nthe build server). I believe that with a sparse checkout I still get\nthe entire directory tree. This just doesn't work well. I can make it\nwork but then I lose other nice features (unrelated to Git).\n\nBasically, I want things separate for release management but together\nfor development.\n"},{"id":"189612","messageId":"loom.20120418T115454-63@post.gmane.org","threadId":"30257","inReplyTo":"CAE1pOi2FQtyahcJUJ+CZRLLYN7vxGaOmUiBJ-_AfC-T-ZEY_Sg@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Namit Bhalla","fromEmail":"namitbhalla@yahoo.com","sentAt":"2012-04-18T10:15:01Z","receivedAt":"2012-04-18T10:15:01Z","isPatch":false,"sender":{"key":"namitbhalla@yahoo.com","avatar":null},"body":"Hilco Wijbenga gmail.com> writes:\n> \n> Basically, I want things separate for release management but together\n> for development.\n> \n\nThank you all for this very informative discussion.\nI will give these different approaches a try in the next few days.\n"},{"id":"189623","messageId":"4F8EAEF4.8030706@web.de","threadId":"30257","inReplyTo":"CAE1pOi38krwXZuiYxtpLwm92N=NvWkP30V_=6cnHw=sdyk6QhA@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-04-18T12:09:24Z","receivedAt":"2012-04-18T12:09:24Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 17.04.2012 23:43, schrieb Hilco Wijbenga:\n> The main problem with the current submodule support is that there is\n> so much manual work needed. It is too easy to forget a step. Moreover,\n> it's not easy to determine *that* you forgot a step or which step you\n> forgot.\n\nLooks like you are talking about the submodule support how it was a\nfew years ago. Since 1.7.0 you cannot forget to commit changes in the\nsubmodule anymore, and since 1.7.5 all referenced submodule commits\nare fetched when you fetch the superproject. The only thing missing\n(with some work done towards that in last years GSoc) is supporting\nthe pushing of submodule changes and the transparent update of\nsubmodule content when the superproject is updated, both of which are\ncurrently being worked on.\n\nWhat else was bothering you so much you dumped submodules?\n\n>> Of course, this is entirely driven by git-subtree's model of actually\n>> incorporating subproject history into one big umbrella repository.\n>> There is no separation between the subprojects and umbrella projects.\n>> It's one giant history. Therefore, push/pull to/from subprojects are\n>> explicit operations. That's probably not the best model for every\n>> situation but I find it very nice.\n> \n> I do not have enough (okay, any) experience with subtree to comment on\n> that. The first part seems just what I want. I'm not sure about the\n> explicit pushing/pulling part. That sounds too much like asking for\n> the sort of problems that scared us away from submodules. Hopefully,\n> I'm dead wrong. :-)\n\nAs I understand subtree the pushing and pulling of the subprojects\nis needed pretty much at the same points in time it is needed when\nusing submodules (to share the subproject work between different\nsuperprojects via their upstream). The difference is you import all\nsubproject changes into a single repo when using subtree, while they\nstay separate when using submodules (and additionally you have to\nrecord the updated subprojects in the superproject in an extra\ncommit there). Submodules enforce the distinction between submodules\nand the superproject while subtree doesn't, which may or may not be\njust what you want ;-)\n"},{"id":"189624","messageId":"4F8EB157.5060707@web.de","threadId":"30257","inReplyTo":"nng1unmnksx.fsf@transit.us.cray.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-04-18T12:19:35Z","receivedAt":"2012-04-18T12:19:35Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 17.04.2012 22:51, schrieb dag@cray.com:\n> Hilco Wijbenga writes:\n> \n>> If you work on a subproject (in its own repo) then a subsequent pull\n>> in the umbrella project should bring this new code into the umbrella\n>> project (assuming that would make sense given the branches involved).\n> \n> I don't necessarily think this is always what should happen.\n\nI agree, the reason that we have three different implementations of\nsubproject support is that there is no model that fits all work flows.\n\n> I can't\n> comment on git-submodule since I haven't used it in its more recent\n> incarnation, but one thing I like about git-subtree is that it's\n> explicit. I have to do a \"git subtree pull\" on the umbrella project to\n> pull in the new changes from a subproject. That gives me some degree of\n> control over when to update sources. I suspect one can do the same by\n> using \"git pull\" in submodule directories.\n\nIt's explicit too when using submodules, you can update each submodule\nto the commit you want, review and test that and then decide if you want\nto commit that (or e.g. it's parent) in the superproject or just rewind\nthe submodule because the new changes don't work for you. For a lot of\nuse cases an automatic pull of changes you haven't even seen yet and\nthen automatically promote them to the superproject (which is how I\nunderstand \"git subtree pull\", but I might be wrong) is undesirable, for\nothers it might very well work.\n\n> Perhaps a good way to go would be to provide the basic operations (I\n> think we have most of that) and some hooks in contrib/ or elsewere to\n> implement various models. Just like git imposes no particular workflow\n> model I don't think git should impose one particular aggregation model.\n> What we do need is better documentation of what the various models and\n> tools are. For example, I would find a subtree/submodule comparison\n> highly valuable. It would help people decide which model is best for\n> them.\n\nI agree and am willing to provide information about submodule use cases,\nadvantages and problems, but I'm not a user of subtree so I can't really\ncomment on it. Now that subtree is in git core, what about putting such\na comparison under Documentation/subproject-support.txt?\n"},{"id":"189971","messageId":"nnghaw93v8n.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"CAE1pOi38krwXZuiYxtpLwm92N=NvWkP30V_=6cnHw=sdyk6QhA@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-24T17:17:12Z","receivedAt":"2012-04-24T17:17:12Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Hilco Wijbenga writes:\n\n> I'm assuming that if you have subproject S in umbrella project U and a\n> branch \"topic\" in U then that same branch should exist in S. \n\nNo, I think that is actually very rare. If topic branches really should\nbe mirrored then U and S should be one repository. They are too closely\ncoupled to be separated. But see the but about git-subtree and topic\nbranches below.\n\nFor release tags, etc. I agree that this kind of mirrored tag/branch\nbehavior is the common case.\n\n>> If you want the behavior you describe, a post-receive hook on the\n>> component repositories is easy to implement.\n>\n> [1] Would such a post-receive hook be something that the user has to\n> set up? Or would that be automatically set up after git clone?\n\nThe user/admin would have to set this up, at least for now.\n\n> The main problem with the current submodule support is that there is\n> so much manual work needed. It is too easy to forget a step. Moreover,\n> it's not easy to determine *that* you forgot a step or which step you\n> forgot.\n\nI agree. We can certainly make things more user-friendly.\n\n>> Of course, this is entirely driven by git-subtree's model of actually\n>> incorporating subproject history into one big umbrella repository.\n>> There is no separation between the subprojects and umbrella projects.\n>> It's one giant history.  Therefore, push/pull to/from subprojects are\n>> explicit operations.  That's probably not the best model for every\n>> situation but I find it very nice.\n>\n> I do not have enough (okay, any) experience with subtree to comment on\n> that. The first part seems just what I want. I'm not sure about the\n> explicit pushing/pulling part. That sounds too much like asking for\n> the sort of problems that scared us away from submodules. Hopefully,\n> I'm dead wrong. :-)\n\nWith subtrees, a topic branch in the umbrella project WILL be reflected\nin the subproject because it is really one big repository. It's a\nlittle inconvenient to subtree push a new tag at the moment. You have\nto do a subtree split to a new branch and then push the branch to the\noriginal component repository. That's one thing I want to improve in\nthe short term. I have found a need for then when creating release\ntags.\n\nBut still, it seems odd to me that you'd create a topic branch in U and\nthen want to push it to a separate S repository. Topic branches are by\nnature ephemeral and I have never had a need to do something like that.\nIt just seems to go against the grain of what a topic branch is. As I\nsaid above, release tags and such are in a different category and that\nis the main target of the subtree push enhancements I want to make.\n\n>> But I don't agree\n>> that we'll be able to design one model that works for everyone.  svn\n>> externals are just one model to aggregate projects but it is not the\n>> only one.  It just happens that no one working on Subversion bothered to\n>> implement anything else.\n>\n> :-) I think I made it pretty clear that I was listing what *I* want.\n> What *I* am looking for is something that is as invisible and\n> automatic as possible.\n\nAbsolutely.\n\n> That all sounds good. As long as the hooks are automatic (I'm hopeful\n> you said \"no\" and \"yes\" to [1] above). If so, then I can promise you\n> I'll be taking a look at subtree. :-)\n\nI think at the very least we can provide setup scripts in contrib. To\nbe honest I haven't thought deeply enough about this to determine if\nthere's a way to make it more convenient.\n\n -Dave\n"},{"id":"189972","messageId":"nngbomh3uz0.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"4F8EB157.5060707@web.de","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-24T17:22:59Z","receivedAt":"2012-04-24T17:22:59Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Jens Lehmann writes:\n\n> It's explicit too when using submodules, you can update each submodule\n> to the commit you want, review and test that and then decide if you want\n> to commit that (or e.g. it's parent) in the superproject or just rewind\n> the submodule because the new changes don't work for you.\n\nYes, that is very useful.\n\n> For a lot of use cases an automatic pull of changes you haven't even\n> seen yet and then automatically promote them to the superproject\n> (which is how I understand \"git subtree pull\", but I might be wrong)\n> is undesirable, for others it might very well work.\n\nSince subtrees are really just directories in a single-history\nrepository, a subtree pull does \"prommote\" the changes to the\nsuperproject because there is no superproject/subproject. That's one of\nthe reasons subtree can be used to create subprojects out of existing\nrepositories.\n\nSubtrees and submodules really are very different models. I see\nadvantages and dsadvantages to both depending on one's work flow.\n\n> I agree and am willing to provide information about submodule use cases,\n> advantages and problems, but I'm not a user of subtree so I can't really\n> comment on it. Now that subtree is in git core, what about putting such\n> a comparison under Documentation/subproject-support.txt?\n\nThat would be great. Do you want to start work on that? I can\ncontribute some text about git-subtree.\n\n -Dave\n"},{"id":"189978","messageId":"201204241759.q3OHxSbH017287@no.baka.org","threadId":"30257","inReplyTo":"nngbomh3uz0.fsf@transit.us.cray.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2012-04-24T17:59:28Z","receivedAt":"2012-04-24T17:59:28Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message , dag@cray.com writes:\n\n > I agree and am willing to provide information about submodule use cases,\n > advantages and problems, but I'm not a user of subtree so I can't really\n > comment on it. Now that subtree is in git core, what about putting such\n > a comparison under Documentation/subproject-support.txt?\n\n That would be great. Do you want to start work on that? I can\n contribute some text about git-subtree.\n\nI have a document I created for gitslave which I have cleaned up a bit\nand might be the start of such comparison.\n\n----------------------------------------------------------------------\ngit-submodules is the legacy solution for putting repositories inside\nof other repositories. With git submodules, the submodule is checked\nout to a semi-fixed commit, typically on a detached HEAD. To make a\nchange to the subproject, you need to check the submodule repository\nout onto the correct branch, make the desired change (possibly\ninvolving pull), commit, and then go into the superproject and commit\nthe commit (or at least record the new location of the submodule). It\nwas designed for third party projects which you typically do not doing\nactive development on. Many/most git commands performed on the\nsuperproject will not recurse down into the submodules. submodules\ngive you a tight mapping between subproject commits and superproject\ncommits (you always know which commit a subproject was in for any\ngiven superproject commit). git-submodules is considered difficult to\nuse for less experienced git developers who need to modify the\nsubproject.\n\nAnother option is to stick everything in one giant repository,\ntypically by using git-subtree. This might make your repository large\nand you have to manually run git-subtree commands to export your\nchanges back out to the individual non-aggregated repositories. All\ngit commands run as normal, though when examining pre-subtree history,\nyou can see the individual lines of development on different branches.\n\ngitslave creates a federation of git repositories—a superproject\nrepository and a number of slave repositories—all of which may be\nconcurrently developed on and on which all desired git operations will\ntouch. In a typical use case, you would branch all repositories (to\nthe same branch name) at the same time, and checkout or tag or get of\nthe status of all repositories at the same time. For essentially any\ngit command, you simply replace \"git\" with \"gits\" to get the specified\ngit command to run on all repositories. Of course, some commands do\nnot necessarily make a great deal of sense to run over all git\nrepositories (eg. `git add filename`), but you are allowed to run any\nnormal git command on any of the repositories at any time. gitslave\nprovides a loose binding so that it is not necessarily completely\nclear which revision one repositories was at when a commit was made in\na different repository, except after tag operations.\n\nAnother options include repo from Google, used with Android. Repo\nseems to work much like gitslave from a high level perspective, but\nthere isn't a lot of documentation on using it for other projects.\n\nStill another option is kitenet's mr which supports multiple\nrepository types (CVS, SVN, git, etc). It is absolutely the solution\nfor multi-SCM projects, but since it works on the lowest common\ndenominator you would lose much of the expressive power of git.\n\nThe final option is to just put git repositories inside of other\nrepositories. However, you must be sure to add the subproject into\nthe superproject's .gitignore to prevent `git add` from adding the\nsubproject as a broken submodule commit (broken because no .gitmodules\nor git-config entry will exist for it).\n----------------------------------------------------------------------\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"189980","messageId":"CAE1pOi2KgeLPg7UVRP7dbqLFJErsKUx22Mi5aSkphy7KMJhoUQ@mail.gmail.com","threadId":"30257","inReplyTo":"nnghaw93v8n.fsf@transit.us.cray.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-04-24T18:54:08Z","receivedAt":"2012-04-24T18:54:08Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 24 April 2012 10:17, wrote:\n> Hilco Wijbenga writes:\n>\n>> I'm assuming that if you have subproject S in umbrella project U and a\n>> branch \"topic\" in U then that same branch should exist in S.\n>\n> No, I think that is actually very rare.  If topic branches really should\n> be mirrored then U and S should be one repository.  They are too closely\n> coupled to be separated.  But see the but about git-subtree and topic\n> branches below.\n\nToo closely coupled? I do not think breaking up a project into a set\nof libraries makes everything tightly coupled. I would argue the\nopposite. :-) Anyway, you answer my concern below.\n\n>>> Of course, this is entirely driven by git-subtree's model of actually\n>>> incorporating subproject history into one big umbrella repository.\n>>> There is no separation between the subprojects and umbrella projects.\n>>> It's one giant history.  Therefore, push/pull to/from subprojects are\n>>> explicit operations.  That's probably not the best model for every\n>>> situation but I find it very nice.\n>>\n>> I do not have enough (okay, any) experience with subtree to comment on\n>> that. The first part seems just what I want. I'm not sure about the\n>> explicit pushing/pulling part. That sounds too much like asking for\n>> the sort of problems that scared us away from submodules. Hopefully,\n>> I'm dead wrong. :-)\n>\n> With subtrees, a topic branch in the umbrella project WILL be reflected\n> in the subproject because it is really one big repository.  It's a\n> little inconvenient to subtree push a new tag at the moment.  You have\n> to do a subtree split to a new branch and then push the branch to the\n> original component repository.  That's one thing I want to improve in\n> the short term.  I have found a need for then when creating release\n> tags.\n\nOkay, that would work fine. I have no problem with a bit of extra work\nhere. (In fact, this is probably where I would *want* a bit of extra\ncontrol.)\n\nWhat would happen if you had a bunch of commits in the umbrella\nproject and then did a push? Would that error out? Are there\nprotections in place to prevent developers from making silly mistakes\nlike that?\n\n> But still, it seems odd to me that you'd create a topic branch in U and\n> then want to push it to a separate S repository.  Topic branches are by\n> nature ephemeral and I have never had a need to do something like that.\n> It just seems to go against the grain of what a topic branch is.  As I\n> said above, release tags and such are in a different category and that\n> is the main target of the subtree push enhancements I want to make.\n\nI had not realized all changes would simply be part of the umbrella\nproject. Given that, a topic branch in each subproject's repo is\nunnecessary.\n\nCheers,\nHilco\n"},{"id":"189987","messageId":"CAPZPVFbHseYHdPOXmbyGxncZmmzSHwY_fJkNRRQAMVtGZBA0CQ@mail.gmail.com","threadId":"30257","inReplyTo":"1334568432.53977.YahooMailNeo@web65906.mail.ac4.yahoo.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Eugene Sajine","fromEmail":"euguess@gmail.com","sentAt":"2012-04-24T19:48:54Z","receivedAt":"2012-04-24T19:48:54Z","isPatch":false,"sender":{"key":"euguess@gmail.com","avatar":null},"body":"On Mon, Apr 16, 2012 at 5:27 AM, Namit Bhalla wrote:\n> I am looking to track some projects using Git with each project as a\n> separate repository.\n> Even after reading the documentation, I am still wondering if there is a\n> way to organize things as described below.\n>\n> Consider 2 projects, Project-a and Project-b, which are housed in\n> repositories Repo-a and Repo-b respectively.\n> Project-a develops reusable libraries which are needed by Project-b\n> (otherwise Project-b will not compile).\n> When a new stable version of Project-a libraries has to be delivered, they\n> are \"checked into\" a path in Repo-a.\n> Now, I would like to setup Repo-b so that when someone starts working on\n> Project-b, he should be able to retrieve the code from Repo-b as well as the libraries from Repo-a. Is there any way to achieve that in\n> Git?\n>\n> Thanks for any pointers!\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n\nWe are working in the environment where we have hundreds (700 + and\ncounting) or projects with many of them reused by others.\nWe are following strictly one project = one repo rule without any\nsubtrees or submodules.\nWhat you are asking about is \"integration\" and IMHO has nothing to do\nwith git - i.e. should be VCS independent.\nWe are using integration on the artifact level and it works amazingly well.\nBut we also use pretty strict naming and location convention that\nallows us to script around the whole setup very easily.\n\nIn order to track dependencies between projects we use Ivy.\nThe project can be compiled locally using local copies of the upstream\nproject artifacts built by developer on the same machine or if not\npresent, the current production artifacts are used.\nWe also have Jenkins CI server that helps with integration. It is very\nsimple and straight forward set up without any unnecessary\ncomplications IMHO.\nFeel free to contact me if you need more info about such set up.\n\nJust my 2 cents.\n\nThanks,\nEugene\n"},{"id":"189993","messageId":"4F970C92.3030704@web.de","threadId":"30257","inReplyTo":"201204241759.q3OHxSbH017287@no.baka.org","subject":"Re: organizing multiple repositories with dependencies","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-04-24T20:26:58Z","receivedAt":"2012-04-24T20:26:58Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 24.04.2012 19:59, schrieb Seth Robertson:\n> \n> In message , dag@cray.com writes:\n> \n> > I agree and am willing to provide information about submodule use cases,\n> > advantages and problems, but I'm not a user of subtree so I can't really\n> > comment on it. Now that subtree is in git core, what about putting such\n> > a comparison under Documentation/subproject-support.txt?\n> \n> That would be great. Do you want to start work on that? I can\n> contribute some text about git-subtree.\n> \n> I have a document I created for gitslave which I have cleaned up a bit\n> and might be the start of such comparison.\n\nThanks for providing the input. Unfortunately I'll be pretty occupied for\nthe next three weeks, so I won't be able to put much work into that before\nmid-May. But maybe we can get the ball rolling ...\n\nIn the end I'd like to see a document people can use to decide what\nsubproject support suits their needs best. Maybe it should start with\nthe basic concept behind each of them:\n\nsubmodules: A submodule is a full fledged repository of which a certain\ncommit is recorded in a gitlink entry in each of the the superproject's\ncommits.\nThe emphasis lies on tightly coupling versions of both while keeping the\nboundaries between superproject and submodules visible.\nThis leads to some extra cost when doing changes in a submodule but makes\nit easy to evaluate and select new changes from upstream and push back\nlocal changes to their respective upstream.\n\nsubtree: All subprojects become an integral part of the history of the\nsuperproject.\nThe emphasis lies on incorporating the subtree and its history into the\nsuperproject.\nThat adds some extra cost when it comes to pushing subtree changes back\nto their upstream (starting with the need for careful commit planning for\nlocal commits intended to be pushed out again) and less fine grained\ncontrol over importing changes from the subtrees upstream.\n\ngitslave: This creates a federation of full fledged git repositories which\nare operated on by the gits commands together (where a git command would\nonly operate on the superproject).\nThe emphasis lies on the simultaneous operation of gits commands on all\ngit repositories.\nIt does not provide any coupling of the commits in the superproject and the\nslave repositories (but you can use tags to have that at some points in the\nhistory).\n\n\nWhat do you think? (Please point out anything I misrepresented in the last\ntwo paragraphs, they are based solely on what I picked up on this list and\nare not based on any actual experience ;-)\n\nThen we could describe in a table what to do when to fetch new subproject\ncommits, how to \"select\" them in the superproject and how to push them\nback to their respective upstream. Another interesting question could be\nhow a bug in a subproject that affects the superproject is handled in each\nof the scenarios.\nDoes that sound like a start?\n"},{"id":"189996","messageId":"201204242052.q3OKqrYT020792@no.baka.org","threadId":"30257","inReplyTo":"4F970C92.3030704@web.de","subject":"Re: organizing multiple repositories with dependencies","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2012-04-24T20:52:53Z","receivedAt":"2012-04-24T20:52:53Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message <4F970C92.3030704@web.de>, Jens Lehmann writes:\n\n gitslave: This creates a federation of full fledged git repositories which\n are operated on by the gits commands together (where a git command would\n only operate on the superproject).\n The emphasis lies on the simultaneous operation of gits commands on all\n git repositories.\n It does not provide any coupling of the commits in the superproject and the\n slave repositories (but you can use tags to have that at some points in the\n history).\n\nWell, gitslave is essentially a loop to run the listed git command in\neach repository, so there are no atomic operations and you can get\npartial success and partial failure, thus \"simultaneous operation\"\nisn't a very good description.\n\nPerhaps a better sentence would be, \"The emphasis lies in the\nsimplicity and convenience of having gits commands run the same git\noperation on all linked repositories, with output summarizing.\"\n\nJust a FYI: partial success and partial failure in different\nrepositories isn't a major problem when using git. However, in the\ninterest of full disclosure, two users racing to push could in theory\ncause a broken project given specific combinations of some users\nmodifying different repositories than others with mutual dependencies\nbetween them. But in all of the years of using gitslave no-one has\never had/reported such a problem. If you can assuming any sort of\nsane QA and deployment practice, this should be able to cause an\noperational problem.\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"189997","messageId":"CAJsNXTmk_A1NxR3ZcuEkEcX-+LyWPq2yY0mdDwAfVLX=Eg9coA@mail.gmail.com","threadId":"30257","inReplyTo":"CAE1pOi2KgeLPg7UVRP7dbqLFJErsKUx22Mi5aSkphy7KMJhoUQ@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"PJ Weisberg","fromEmail":"pj@irregularexpressions.net","sentAt":"2012-04-24T21:09:25Z","receivedAt":"2012-04-24T21:09:25Z","isPatch":false,"sender":{"key":"pj@irregularexpressions.net","avatar":"https://gravatar.com/avatar/aa2c1edcc61b536cc5c9f37fbce084e655446e2309f9818d43f13d47304a602b?d=mp&s=160"},"body":"On Tue, Apr 24, 2012 at 11:54 AM, Hilco Wijbenga\n wrote:\n> On 24 April 2012 10:17,   wrote:\n>> Hilco Wijbenga writes:\n>>\n>>> I'm assuming that if you have subproject S in umbrella project U and a\n>>> branch \"topic\" in U then that same branch should exist in S.\n>>\n>> No, I think that is actually very rare.  If topic branches really should\n>> be mirrored then U and S should be one repository.  They are too closely\n>> coupled to be separated.  But see the but about git-subtree and topic\n>> branches below.\n>\n> Too closely coupled? I do not think breaking up a project into a set\n> of libraries makes everything tightly coupled. I would argue the\n> opposite. :-) Anyway, you answer my concern below.\n\nIndeed. But when you make a branch in your main project, wouldn't you\nusually still want to use the master branch of the libraries? Or if\nthere's an experimental branch in a library and you want to use that\nbranched version, wouldn't you still use the master version of all the\nother libraries? What if you have two projects that both use a\nlibrary, but are otherwise unrelated? If you create a branch called\n'hotfix' in one project, do you automatically find your library\nversion switching to an unrelated 'hotfix' from another project?\n\n-PJ\n\nGehm's Corollary to Clark's Law: Any technology distinguishable from\nmagic is insufficiently advanced.\n"},{"id":"189999","messageId":"CAE1pOi3=wu_9kmVimvAQjq+O6xUpEbEdnQ4qgeM2SAO7TwD0Mw@mail.gmail.com","threadId":"30257","inReplyTo":"CAJsNXTmk_A1NxR3ZcuEkEcX-+LyWPq2yY0mdDwAfVLX=Eg9coA@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-04-24T22:04:51Z","receivedAt":"2012-04-24T22:04:51Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 24 April 2012 14:09, PJ Weisberg wrote:\n> On Tue, Apr 24, 2012 at 11:54 AM, Hilco Wijbenga\n> wrote:\n>> On 24 April 2012 10:17,   wrote:\n>>> Hilco Wijbenga writes:\n>>>\n>>>> I'm assuming that if you have subproject S in umbrella project U and a\n>>>> branch \"topic\" in U then that same branch should exist in S.\n>>>\n>>> No, I think that is actually very rare.  If topic branches really should\n>>> be mirrored then U and S should be one repository.  They are too closely\n>>> coupled to be separated.  But see the but about git-subtree and topic\n>>> branches below.\n>>\n>> Too closely coupled? I do not think breaking up a project into a set\n>> of libraries makes everything tightly coupled. I would argue the\n>> opposite. :-) Anyway, you answer my concern below.\n>\n> Indeed.  But when you make a branch in your main project, wouldn't you\n> usually still want to use the master branch of the libraries?\n\nFor those that haven't changed? Sure. I just (incorrectly) assumed\nthat if I had a topic branch in the umbrella project I would need\ntopic branches in all repos as well, otherwise I would end up with\nchanges in both various masters and my topic branch. Too much exposure\nto submodules, I suppose. :-) But subtree works around that quite\nnicely so it really doesn't matter. I simply made one too many\nassumptions. :-)\n\nSimilar answers to everything below.\n\n> Or if\n> there's an experimental branch in a library and you want to use that\n> branched version, wouldn't you still use the master version of all the\n> other libraries?  What if you have two projects that both use a\n> library, but are otherwise unrelated?  If you create a branch called\n> 'hotfix' in one project, do you automatically find your library\n> version switching to an unrelated 'hotfix' from another project?\n"},{"id":"190000","messageId":"CAE1pOi3XuiDA1f-NaBGeGYKcWqnCxNer4Ce-MsfjPD2hXU_mgg@mail.gmail.com","threadId":"30257","inReplyTo":"CAPZPVFbHseYHdPOXmbyGxncZmmzSHwY_fJkNRRQAMVtGZBA0CQ@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Hilco Wijbenga","fromEmail":"hilco.wijbenga@gmail.com","sentAt":"2012-04-24T22:11:54Z","receivedAt":"2012-04-24T22:11:54Z","isPatch":false,"sender":{"key":"hilco.wijbenga@gmail.com","avatar":null},"body":"On 24 April 2012 12:48, Eugene Sajine wrote:\n> We are working in the environment where we have hundreds (700 + and\n> counting) or projects with many of them reused by others.\n> We are following strictly one project = one repo rule without any\n> subtrees or submodules.\n> What you are asking about is \"integration\" and IMHO has nothing to do\n> with git - i.e. should be VCS independent.\n> We are using integration on the artifact level and it works amazingly well.\n> But we also use pretty strict naming and location convention that\n> allows us to script around the whole setup very easily.\n>\n> In order to track dependencies between projects we use Ivy.\n> The project can be compiled locally using local copies of the upstream\n> project artifacts built by developer on the same machine or if not\n> present, the current production artifacts are used.\n> We also have Jenkins CI server that helps with integration. It is very\n> simple and straight forward set up without any unnecessary\n> complications IMHO.\n> Feel free to contact me if you need more info about such set up.\n\nSo how do you handle implementing features/changes that involve more\nthan one project? Surely, you do not manually create topic branches in\neach of the involved repos?\n"},{"id":"190033","messageId":"nngobqg1ztl.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"4F970C92.3030704@web.de","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-24T23:21:10Z","receivedAt":"2012-04-24T23:21:10Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Jens Lehmann writes:\n\n[Thanks for working this! I have a few comments inlined below to\nhopefully help make this even better.]\n\n> In the end I'd like to see a document people can use to decide what\n> subproject support suits their needs best. Maybe it should start with\n> the basic concept behind each of them:\n\nExactly.\n\n> submodules: A submodule is a full fledged repository of which a certain\n> commit is recorded in a gitlink entry in each of the the superproject's\n> commits.\n\nThat's far too technical. I don't even know what that means. :) I\nthink we want to go for the average user who just wants to make an\ninformed decision among the various models available.\n\n> The emphasis lies on tightly coupling versions of both while keeping the\n> boundaries between superproject and submodules visible.\n\nThe above is good but could use some expanding. What exactly does\n\"tightly coupling\" mean? It's kind of a generic phrase.\n\n> This leads to some extra cost when doing changes in a submodule but makes\n> it easy to evaluate and select new changes from upstream and push back\n> local changes to their respective upstream.\n\nThis, I think is a key differentiator for submodules and should be\nemphasized.\n\n> subtree: All subprojects become an integral part of the history of the\n> superproject.\n> The emphasis lies on incorporating the subtree and its history into the\n> superproject.\n> That adds some extra cost when it comes to pushing subtree changes back\n> to their upstream (starting with the need for careful commit planning for\n> local commits intended to be pushed out again) and less fine grained\n> control over importing changes from the subtrees upstream.\n\nThat's a good start. I'll add some text to this later as I think there\nare some advantages to the approach that should be called out.\n\n> gitslave: This creates a federation of full fledged git repositories which\n> are operated on by the gits commands together (where a git command would\n> only operate on the superproject).\n> The emphasis lies on the simultaneous operation of gits commands on all\n> git repositories.\n> It does not provide any coupling of the commits in the superproject and the\n> slave repositories (but you can use tags to have that at some points in the\n> history).\n\nShould gitslave be covered in this document if gitslave is not in the\nupstream git repository? I'm not knocking gitslave, in fact I think\nit's cool technology and probably _should_ be contributed upstream. I'm\njust asking the question about whether stuff in Documentation/ is or\nshould be limited to things in the upstream repository.\n\nThat said, the above is good but as a user I would want more\nclarification on how submodules and gitslave differ. The same is true\nfor subtrees but I'm assuming I'll handle that. :)\n\n> What do you think? (Please point out anything I misrepresented in the last\n> two paragraphs, they are based solely on what I picked up on this list and\n> are not based on any actual experience ;-)\n\nIt looks very good as a starting point. Thanks!\n\n> Then we could describe in a table what to do when to fetch new subproject\n> commits, how to \"select\" them in the superproject and how to push them\n> back to their respective upstream. Another interesting question could be\n> how a bug in a subproject that affects the superproject is handled in each\n> of the scenarios.\n\nYes, I was imagining exactly this sort of table.\n\nHow about creating a topic branch for this and publishing it so several\nof us can collaborate? I think that would make things a bit easier\nmoving forward.\n\n -Dave\n"},{"id":"190034","messageId":"nngipgo1zn3.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"201204241759.q3OHxSbH017287@no.baka.org","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-24T23:25:04Z","receivedAt":"2012-04-24T23:25:04Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Seth Robertson writes:\n\n> I have a document I created for gitslave which I have cleaned up a bit\n> and might be the start of such comparison.\n\nI'll look through this is more detail and I think some of the text\ncan be combined with other contributions. I just asked Jens if he wants\nto create a topic branch on which we may collaborate.\n\nMy inclination is to limit the documentation to what's in the upstream\nrepository, which would eliminate repo, mr and unfortunately, for the\ntime being, git slave, but that's just one opinion and it's a newbie\nopinion at that. :) Please don't take it as knocking git-slave because\nI think it's really cool. I would like to see it go upstream!\n\n -Dave\n"},{"id":"190035","messageId":"nngd36w1z9n.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"CAE1pOi2KgeLPg7UVRP7dbqLFJErsKUx22Mi5aSkphy7KMJhoUQ@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-24T23:33:08Z","receivedAt":"2012-04-24T23:33:08Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Hilco Wijbenga writes:\n\n>>> I'm assuming that if you have subproject S in umbrella project U and a\n>>> branch \"topic\" in U then that same branch should exist in S.\n>>\n>> No, I think that is actually very rare.  If topic branches really should\n>> be mirrored then U and S should be one repository.  They are too closely\n>> coupled to be separated.  But see the but about git-subtree and topic\n>> branches below.\n>\n> Too closely coupled? I do not think breaking up a project into a set\n> of libraries makes everything tightly coupled. I would argue the\n> opposite. :-) Anyway, you answer my concern below.\n\nIf you need the same topic branch for each component they would indeed\nseem to be very tightly coupled, even if the code is \"physically\"\nseparated. I can't think of a situation where I would need to implement\nthe same or similar features in multiple components where those\ncomponents are not tightly coupled in some way.\n\n> What would happen if you had a bunch of commits in the umbrella\n> project and then did a push? Would that error out? Are there\n> protections in place to prevent developers from making silly mistakes\n> like that?\n\nIt would push to the remote/origin of the umbrella project, maintaining\nthe same \"whole project\" history. It's an explicit operation to split\nthe commits on any subproject out and push them to the subproject's\norigin.\n\nSo let's say you want to branch each subproject for release. You could\ndo something like this (off the top of my head so don't copy/paste\nverbatim):\n\nbranch U release_X # Create the branch in the umbrella project\nwork, work, work\ngit subtree split S1 S1_release_X # Split commits to S1 made on\n # release_X branch\ngit subtree split S2 S2_release_X\ngit subtree split S3 S3_release_X\ngit checkout S1_release_X # Send commits to S1 to origin,\ngit push origin_S1 release_X # creating branch release_X\ngit checkout S2_release_X\ngit push origin_S2 release_X\ngit checkout S3_release_X\ngit push origin_S3 release_X\n\nIt's the split/checkout/push sequence that I'd like to optimize and make\nsimpler.\n\n -Dave\n"},{"id":"190036","messageId":"nng7gx41z3j.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"CAPZPVFbHseYHdPOXmbyGxncZmmzSHwY_fJkNRRQAMVtGZBA0CQ@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-24T23:36:48Z","receivedAt":"2012-04-24T23:36:48Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Eugene Sajine writes:\n\n> What you are asking about is \"integration\" and IMHO has nothing to do\n> with git - i.e. should be VCS independent.\n\nIndeed, integration is another good strategy. I mainly use submodules\nas a convenience because it is a frequent (though not regular)\noccurrence that we want to change multiple libraries at the same time.\nPlus it provides a convenient way for developers to check out \"the\nproject.\"\n\nAgain, it depends on the development practice and individual situations.\n:)\n\n -Dave\n"},{"id":"190037","messageId":"nng1unc1z09.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"CAE1pOi3XuiDA1f-NaBGeGYKcWqnCxNer4Ce-MsfjPD2hXU_mgg@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-24T23:38:46Z","receivedAt":"2012-04-24T23:38:46Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Hilco Wijbenga writes:\n\n> On 24 April 2012 12:48, Eugene Sajine wrote:\n\n> So how do you handle implementing features/changes that involve more\n> than one project? Surely, you do not manually create topic branches in\n> each of the involved repos?\n\nThat is indeed what integrators usually do for release branches. For\nfeatures, developers create topic branches for whatever component\nthey're working on and these never hit a shared repository anyway so it\ndoesn't really matter.\n\nAgain, it's just another model which works well in a lot of cases, but\nnot all.\n\n -Dave\n"},{"id":"190059","messageId":"201204251248.q3PCmN2F007496@no.baka.org","threadId":"30257","inReplyTo":"nngipgo1zn3.fsf@transit.us.cray.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2012-04-25T12:48:23Z","receivedAt":"2012-04-25T12:48:23Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message , dag@cray.com writes:\n\n My inclination is to limit the documentation to what's in the upstream\n repository, which would eliminate repo, mr and unfortunately, for the\n time being, git slave, but that's just one opinion and it's a newbie\n opinion at that. :) Please don't take it as knocking git-slave because\n I think it's really cool. I would like to see it go upstream!\n\nSo would I, and I'm also happy to do work to cause that to happen, but\nthe question is would it be accepted?\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"190212","messageId":"nngehr9xnhu.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"201204251248.q3PCmN2F007496@no.baka.org","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-27T14:23:09Z","receivedAt":"2012-04-27T14:23:09Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Seth Robertson writes:\n\n> So would I, and I'm also happy to do work to cause that to happen, but\n> the question is would it be accepted?\n\nStart by asking the question. Start a thread on the list and cc Junio.\nThat's how I started with git-subtree. Of course Avery did some prep\nwork to soften up the maintainers. :)\n\n -Dave\n"},{"id":"190238","messageId":"loom.20120428T191337-86@post.gmane.org","threadId":"30257","inReplyTo":"nngobqg1ztl.fsf@transit.us.cray.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"username localhost","fromEmail":"username.localhost@gmail.com","sentAt":"2012-04-28T17:31:54Z","receivedAt":"2012-04-28T17:31:54Z","isPatch":false,"sender":{"key":"username.localhost@gmail.com","avatar":null},"body":"dag writes:\n> \n> Should gitslave be covered in this document if gitslave is not in the\n> upstream git repository? I'm not knocking gitslave, in fact I think\n> it's cool technology and probably _should_ be contributed upstream. I'm\n> just asking the question about whether stuff in Documentation/ is or\n> should be limited to things in the upstream repository.\n\nI think that it would be better to be inclusive while initially writing such a\ndocument. The work of helping to distinguish gitslave from the others should\nalso help to flesh out the differences between submodules and git-subtree.\n\nOnce the document is in good enough shape that it can be merged, it would be\neasy enough to strip gitslave out if it were decided that it should not be\nincluded. \n\n\n> \n> How about creating a topic branch for this and publishing it so several\n> of us can collaborate? I think that would make things a bit easier\n> moving forward.\n\nI second that. As a git user, I would love to see a document that described\nthese systems, explaining how they differ, showing how various \"common\" actions\nare done in each, and describing the workflows each is best in, and what\nworkflows each does poorly with.\n\nIf somebody does not create a branch and post a starting point, I'm afraid this\nidea will simply die out.\n\n--\nusername@localhost\n"},{"id":"190336","messageId":"CABURp0pHcZfUw8p5F=7W3BipGHdc2Q0QQ7WuaPPVWOYdG1S=BQ@mail.gmail.com","threadId":"30257","inReplyTo":"nngd36w1z9n.fsf@transit.us.cray.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2012-04-30T19:25:54Z","receivedAt":"2012-04-30T19:25:54Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On Tue, Apr 24, 2012 at 7:33 PM, wrote:\n> Hilco Wijbenga writes:\n>>> No, I think that is actually very rare.  If topic branches really should\n>>> be mirrored then U and S should be one repository.  They are too closely\n>>> coupled to be separated.  But see the but about git-subtree and topic\n>>> branches below.\n>>\n>> Too closely coupled? I do not think breaking up a project into a set\n>> of libraries makes everything tightly coupled. I would argue the\n>> opposite. :-) Anyway, you answer my concern below.\n>\n> If you need the same topic branch for each component they would indeed\n> seem to be very tightly coupled, even if the code is \"physically\"\n> separated.  I can't think of a situation where I would need to implement\n> the same or similar features in multiple components where those\n> components are not tightly coupled in some way.\n\nI tend to agree. However, I have a use case that I suffer on a daily basis.\n\nWe have code that runs on multiple platforms (embedded SoCs). I have\na superproject that has a common library and some vendor-specific code\nfor each supported platform broken out into submodules.\n\n super-all\n +-- CommonAPI\n +-- VendorA\n +-- VendorB\n +-- VendorC\n\nThe code in the Vendor submodules contains the proprietary\nimplementations for specific vendor's systems of the CommonAPI\nlibrary. When the CommonAPI gets a new feature, it often gets\nimplemented in all the vendor submodules as well.\n\nWe could easily do this without submodules, of course. But this setup\nallows us to define alternative super-projects that we can then share\nwith subcontractors and original vendors without exposing proprietary\nthird-party code.\n\n super-B\n +-- CommonAPI\n +-- VendorB\n\n super-C\n +-- CommonAPI\n +-- VendorC\n\nWe could still handle this with git-subtree. But we don't.\n\nPhil\n"},{"id":"190338","messageId":"nngd36pxawy.fsf@transit.us.cray.com","threadId":"30257","inReplyTo":"CABURp0pHcZfUw8p5F=7W3BipGHdc2Q0QQ7WuaPPVWOYdG1S=BQ@mail.gmail.com","subject":"Re: organizing multiple repositories with dependencies","fromName":"","fromEmail":"dag@cray.com","sentAt":"2012-04-30T19:43:57Z","receivedAt":"2012-04-30T19:43:57Z","isPatch":false,"sender":{"key":"dag@cray.com","avatar":null},"body":"Phil Hord writes:\n\n>> I can't think of a situation where I would need to implement the same\n>> or similar features in multiple components where those components are\n>> not tightly coupled in some way.\n>\n> I tend to agree. However, I have a use case that I suffer on a daily basis.\n>\n> We have code that runs on multiple platforms (embedded SoCs). I have\n> a superproject that has a common library and some vendor-specific code\n> for each supported platform broken out into submodules.\n>\n> super-all\n> +-- CommonAPI\n> +-- VendorA\n> +-- VendorB\n> +-- VendorC\n>\n> The code in the Vendor submodules contains the proprietary\n> implementations for specific vendor's systems of the CommonAPI\n> library. When the CommonAPI gets a new feature, it often gets\n> implemented in all the vendor submodules as well.\n\nAh yes, that's a good example.\n\n> We could easily do this without submodules, of course. But this setup\n> allows us to define alternative super-projects that we can then share\n> with subcontractors and original vendors without exposing proprietary\n> third-party code.\n>\n> super-B\n> +-- CommonAPI\n> +-- VendorB\n>\n> super-C\n> +-- CommonAPI\n> +-- VendorC\n>\n> We could still handle this with git-subtree. But we don't.\n\nYes, I agree that this is a very important use case. This is the case\nwhere subprojects exist because of vendor barriers, not necessarily due\nto software engineering concerns.\n\n -Dave\n"}]}