{"thread":{"id":"40152","subject":"Submodule, subtree, or something else?","startedAt":"2015-08-21T22:47:42Z","lastAt":"2015-08-24T17:53:15Z","messageCount":6,"participants":["Jānis Rukšāns","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"268462","messageId":"1440197262.23145.191.camel@gmail.com","threadId":"40152","inReplyTo":null,"subject":"Submodule, subtree, or something else?","fromName":"Jānis Rukšāns","fromEmail":"janis.ruksans@gmail.com","sentAt":"2015-08-21T22:47:42Z","receivedAt":"2015-08-21T22:47:42Z","isPatch":false,"sender":{"key":"janis.ruksans@gmail.com","avatar":null},"body":"Hello,\n\n\nFirst of all, I apologise for the wall of text that follows; obviously I\nam bad at this.\n\nMy $DAYJOB is switching from Subversion to Git, primarily because of\nit's distributed nature (we are scattered all across the globe), and the\nease of branching and merging.  One issue that has popped up is how to\nmanage code shared between multiple projects.\n\nOur SVN setup used a shared repository for all projects, either using\nexternals for shared code, or, more often than not, simply merging the\ncode between projects as needed.  Ignoring the fact that merging with\nSVN is somewhat cumbersome, overall it has worked quite well for us,\nespecially when combined with git-svn.\n\nFor external libraries that rarely change, submodules appear to be the\nobvious choice when using Git.  On the other hand, I've found them\nsomewhat cumbersome to use, and subtree merging (either using git\nsubtree, or directly with git merge -s subtree) is closer to what we\nwere doing in SVN.  A major drawback of submodules in my opinion is the\ninability to make a full clone from an existing one without having\naccess to the central repository, which is something I have to do from\ntime to time.\n\nFor internal libraries, the situation is even less clear.  For many of\nthese libraries, most of the development happens within the context of a\nsingle project, with commits to main project being interleaved with\ncommits to the subproject(s), resulting in histories resembling:\n\n (using git submodule)\n\n   A---B---S1---S2---C---S3\n          ,´   ,´       ,´\n     N---O----P----Q---R\n\n (using git subtree with --rejoin)\n\n   A---B---N---O---M1---M2---Q---C---R---M3\n                  /    /                /\n             N'--O'---P--------Q'------R'\n\n (using merge -s subtree)\n\n   A---B---M1---M2---C---M3\n          /    /        /\n     N---O----P----Q---R\n\nwhere A, B and C are changes to the main project, N, O, P, Q and R are\nchanges to library code, and Sn and Mn are submodule updates and merge\ncommits, respectively.\n\nFrom what I have gathered, submodules have issues with branching and\nmerging, therefore, unless I'm mistaken, submodules are kinda out of\nquestion.  Of the remaining two options, merging directly results in a\nnicer history, but requires making all changes to the library repo first\n(although I am quite sure that a similar effect can be achieved with\nplumbing, similarly to how git subtree split works), and is harder to\nuse than git subtree.  Also, all three options can result in the main\nproject history being cluttered with extra commits.\n\nLastly, there is a particularly painful 3rd party library that has an\nenormous amount of local modifications that are never going to make it\nupstream, essentially making it a fork, project specific changes that\nare required for one project, but would break others, separate language\nbindings that access the internals (often requiring bug fixes to be made\nsimultaneously to both), and, if that wasn't enough, it *requires*\nseveral source files to be modified for each individual project that\nuses it.  It's a complete mess, but we're stuck with it for the existing\nprojects, as switching to an alternative would be too time consuming.\n\n\nTo sum up, I'm looking for something that would let us share code\nbetween multiple projects, allow for:\n\n1) separate histories with relatively easy branching and merging\n\n2) distributed workflow without having to set up a multiple repositories\neverywhere (eg. work <-> home <-> laptop)\n\n3) to work on the shared code within a project using it\n\n4) inspection of the complete history\n\n5) modifications that are not shared with other projects\n\nand would not result in lots of clutter in the history.\n\nRepository size is somewhat less of an issue, because each submodule has\nto be checked out anyway.\n\nSubmodules let you have #3, and #1, #2 and #5 to a point, after which it\nbecomes a pain.  git subtree allows #1, #2, #3 and #4, and #5 with some\npain (?), but results in duplicate commits.  Using subtree merge\nstrategy directly gives everything except #3, but is harder to use than\nsubmodules or subtree.\n\nAre there any other options beside these three for sharing (or in some\ncases, not sharing) common code between projects using Git, that would\naddress the above points better?  Or, alternatively, ways to work around\nthe drawbacks of the existing tools?\n\nLastly, I will be grateful for any suggestions about how to handle the\nmessy case described above better.\n\nThanks,\nJānis\n"},{"id":"268466","messageId":"CAGZ79kbUXwEYnpDWgKqnUab2xP4m9m7FMskaK2u8WcqnLSSoog@mail.gmail.com","threadId":"40152","inReplyTo":"1440197262.23145.191.camel@gmail.com","subject":"Re: Submodule, subtree, or something else?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-08-22T00:07:04Z","receivedAt":"2015-08-22T00:07:04Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Aug 21, 2015 at 3:47 PM, Jānis Rukšāns <janis.ruksans@gmail.com> wrote:\n> Hello,\n>\n>\n> First of all, I apologise for the wall of text that follows; obviously I\n> am bad at this.\n>\n> My $DAYJOB is switching from Subversion to Git, primarily because of\n> it's distributed nature (we are scattered all across the globe), and the\n> ease of branching and merging.  One issue that has popped up is how to\n> manage code shared between multiple projects.\n>\n> Our SVN setup used a shared repository for all projects, either using\n> externals for shared code, or, more often than not, simply merging the\n> code between projects as needed.  Ignoring the fact that merging with\n> SVN is somewhat cumbersome, overall it has worked quite well for us,\n> especially when combined with git-svn.\n>\n> For external libraries that rarely change, submodules appear to be the\n> obvious choice when using Git.  On the other hand, I've found them\n> somewhat cumbersome to use, and subtree merging (either using git\n> subtree, or directly with git merge -s subtree) is closer to what we\n> were doing in SVN.  A major drawback of submodules in my opinion is the\n> inability to make a full clone from an existing one without having\n> access to the central repository, which is something I have to do from\n> time to time.\n\nCan you elaborate on that a bit more?\ngit clone --recurse-submodules should do that no matter which remote\nyou contact?\n\n\n>\n> For internal libraries, the situation is even less clear.  For many of\n> these libraries, most of the development happens within the context of a\n> single project, with commits to main project being interleaved with\n> commits to the subproject(s), resulting in histories resembling:\n>\n>  (using git submodule)\n>\n>    A---B---S1---S2---C---S3\n>           ,´   ,´       ,´\n>      N---O----P----Q---R\n>\n>  (using git subtree with --rejoin)\n>\n>    A---B---N---O---M1---M2---Q---C---R---M3\n>                   /    /                /\n>              N'--O'---P--------Q'------R'\n>\n>  (using merge -s subtree)\n>\n>    A---B---M1---M2---C---M3\n>           /    /        /\n>      N---O----P----Q---R\n>\n> where A, B and C are changes to the main project, N, O, P, Q and R are\n> changes to library code, and Sn and Mn are submodule updates and merge\n> commits, respectively.\n>\n> From what I have gathered, submodules have issues with branching and\n> merging, therefore, unless I'm mistaken, submodules are kinda out of\n> question.  Of the remaining two options, merging directly results in a\n> nicer history, but requires making all changes to the library repo first\n> (although I am quite sure that a similar effect can be achieved with\n> plumbing, similarly to how git subtree split works), and is harder to\n> use than git subtree.  Also, all three options can result in the main\n> project history being cluttered with extra commits.\n>\n> Lastly, there is a particularly painful 3rd party library that has an\n> enormous amount of local modifications that are never going to make it\n> upstream, essentially making it a fork, project specific changes that\n> are required for one project, but would break others, separate language\n> bindings that access the internals (often requiring bug fixes to be made\n> simultaneously to both), and, if that wasn't enough, it *requires*\n> several source files to be modified for each individual project that\n> uses it.  It's a complete mess, but we're stuck with it for the existing\n> projects, as switching to an alternative would be too time consuming.\n>\n>\n> To sum up, I'm looking for something that would let us share code\n> between multiple projects, allow for:\n>\n> 1) separate histories with relatively easy branching and merging\n>\n> 2) distributed workflow without having to set up a multiple repositories\n> everywhere (eg. work <-> home <-> laptop)\n>\n> 3) to work on the shared code within a project using it\n>\n> 4) inspection of the complete history\n>\n> 5) modifications that are not shared with other projects\n>\n> and would not result in lots of clutter in the history.\n>\n> Repository size is somewhat less of an issue, because each submodule has\n> to be checked out anyway.\n>\n> Submodules let you have #3, and #1, #2 and #5 to a point, after which it\n> becomes a pain.  git subtree allows #1, #2, #3 and #4, and #5 with some\n> pain (?), but results in duplicate commits.  Using subtree merge\n> strategy directly gives everything except #3, but is harder to use than\n> submodules or subtree.\n>\n> Are there any other options beside these three for sharing (or in some\n> cases, not sharing) common code between projects using Git, that would\n> address the above points better?  Or, alternatively, ways to work around\n> the drawbacks of the existing tools?\n>\n> Lastly, I will be grateful for any suggestions about how to handle the\n> messy case described above better.\n>\n> Thanks,\n> Jānis\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"268530","messageId":"1440339066.32140.17.camel@gmail.com","threadId":"40152","inReplyTo":"CAGZ79kbUXwEYnpDWgKqnUab2xP4m9m7FMskaK2u8WcqnLSSoog@mail.gmail.com","subject":"Re: Submodule, subtree, or something else?","fromName":"Jānis Rukšāns","fromEmail":"janis.ruksans@gmail.com","sentAt":"2015-08-23T14:11:06Z","receivedAt":"2015-08-23T14:11:06Z","isPatch":false,"sender":{"key":"janis.ruksans@gmail.com","avatar":null},"body":"On Pk, 2015-08-21 at 17:07 -0700, Stefan Beller wrote:\n> On Fri, Aug 21, 2015 at 3:47 PM, Jānis Rukšāns <janis.ruksans@gmail.com> wrote:\n> > \n> > A major drawback of submodules in my opinion is the\n> > inability to make a full clone from an existing one without having\n> > access to the central repository, which is something I have to do from\n> > time to time.\n> \n> Can you elaborate on that a bit more?\n> git clone --recurse-submodules should do that no matter which remote\n> you contact?\n\nI mean that if I have cloned a repository with submodules, cloning that\nrepository with --recurse-submodules will either access the \"central\nserver\" if absolute URLs are used, or requires additional clones for\neach submodule.  For example\n\ngit clone --recursive http://somewhere/projectA.git\ngit clone --recursive file://$(pwd)/projectA projectA.tmp\n\nThe second command will cause the submodules to be downloaded again, or\nexpect them to be found in $(pwd).\n\nOr am I mistaken, or doing something wrong?\n"},{"id":"268579","messageId":"CAGZ79kY+HLzd4esdy+i82Eh5Vhc1NEDDfinD2iiBMvV41GSCiw@mail.gmail.com","threadId":"40152","inReplyTo":"1440339066.32140.17.camel@gmail.com","subject":"Re: Submodule, subtree, or something else?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-08-24T16:51:01Z","receivedAt":"2015-08-24T16:51:01Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, Aug 23, 2015 at 7:11 AM, Jānis Rukšāns <janis.ruksans@gmail.com> wrote:\n> On Pk, 2015-08-21 at 17:07 -0700, Stefan Beller wrote:\n>> On Fri, Aug 21, 2015 at 3:47 PM, Jānis Rukšāns <janis.ruksans@gmail.com> wrote:\n>> >\n>> > A major drawback of submodules in my opinion is the\n>> > inability to make a full clone from an existing one without having\n>> > access to the central repository, which is something I have to do from\n>> > time to time.\n>>\n>> Can you elaborate on that a bit more?\n>> git clone --recurse-submodules should do that no matter which remote\n>> you contact?\n>\n> I mean that if I have cloned a repository with submodules, cloning that\n> repository with --recurse-submodules will either access the \"central\n> server\" if absolute URLs are used, or requires additional clones for\n> each submodule.  For example\n>\n> git clone --recursive http://somewhere/projectA.git\n> git clone --recursive file://$(pwd)/projectA projectA.tmp\n>\n> The second command will cause the submodules to be downloaded again, or\n> expect them to be found in $(pwd).\n\nIIUC, the second command will lookup the submodules in $(pwd), but if they\nare not there they are skipped, so all of the existing submodules are cloned.\nWhy do you need more submodules in the tmp clone than in $(pwd)/projectA\nwould be my next question. But I see your point now.\n\n\n\n>\n> Or am I mistaken, or doing something wrong?\n>\n"},{"id":"268586","messageId":"1440436358.32140.72.camel@gmail.com","threadId":"40152","inReplyTo":"CAK6hiNiBD+DUdNq0c2DY9LWg2PCgE56SpbBip8BNNmHTsEttuQ@mail.gmail.com","subject":"Re: Submodule, subtree, or something else?","fromName":"Jānis Rukšāns","fromEmail":"janis.ruksans@gmail.com","sentAt":"2015-08-24T17:12:38Z","receivedAt":"2015-08-24T17:12:38Z","isPatch":false,"sender":{"key":"janis.ruksans@gmail.com","avatar":null},"body":"On Sv, 2015-08-23 at 17:13 -0600, Cox, Michael wrote:\n> You might want to take a look at how the Boost (boost.org) project\n> uses submodules.  They use submodules for each library.  I know they\n> use relative paths in their .gitmodules file to avoid the problem\n> you're referring to regarding \"git clone --recurse-submodules\".\n\nThanks!  I had a look at their setup, and they are using ../libx.git for\nsubmodules, which unfortunately breaks when cloning from another\n\"working copy\":\n\n$ git clone --recursive file:///tmp/gittest/repo.a/main.git main.work\nCloning into 'main.work'...\n<snip>\nSubmodule 'liba' (file:///tmp/gittest/repo.a/liba.git) registered for path 'liba'\nCloning into 'liba'...\n<snip>\nSubmodule path 'liba': checked out '6a0ef37c03a7068328956dcb8a08bc39f280edfc'\n\n$ git clone --recursive file://($pwd)/main.work main.home\nCloning into 'main.home'...\n<snip>\nSubmodule 'liba' (file:///tmp/gittest/work/liba.git) registered for path 'liba'\nCloning into 'liba'...\nfatal: '/tmp/gittest/work/liba.git' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\nClone of 'file:///tmp/gittest/work/liba.git' into submodule path 'liba' failed\n\n\nAfter some trial and error I managed to get what I wanted to achieve by\nusing ./liba as the submodule URL (no .git suffix!), and creating a file\nnamed liba in /tmp/gittest/repo.a/main.git (ie. the bare \"origin\" repo)\nwith a single line in it:\n\ngitdir: ../liba.git\n\nHowever, I'm not sure it is the right thing, or even advisable to do so.\n"},{"id":"268590","messageId":"1440438795.32140.98.camel@gmail.com","threadId":"40152","inReplyTo":"CAGZ79kY+HLzd4esdy+i82Eh5Vhc1NEDDfinD2iiBMvV41GSCiw@mail.gmail.com","subject":"Re: Submodule, subtree, or something else?","fromName":"Jānis Rukšāns","fromEmail":"janis.ruksans@gmail.com","sentAt":"2015-08-24T17:53:15Z","receivedAt":"2015-08-24T17:53:15Z","isPatch":false,"sender":{"key":"janis.ruksans@gmail.com","avatar":null},"body":"On P , 2015-08-24 at 09:51 -0700, Stefan Beller wrote:\n> IIUC, the second command will lookup the submodules in $(pwd), but if\n> they are not there they are skipped, so all of the existing submodules\n> are cloned.\n> Why do you need more submodules in the tmp clone than in\n> $(pwd)/projectA would be my next question. But I see your point now.\n\nThe $(pwd) was just an example to illustrate my point.  The actual use\ncase is that I would be hacking on something at work, notice that it is\nalready late and I have to catch the last bus home, yet I don't want to\npostpone whatever I was working on until the next day.  So I would do\ngit commit -a -m \"[WIP] Stuff, finish at home\" to save my work so far,\ngo home, and clone / fetch it over ssh.\n\nAnother important factor is that a lot of our code can be meaningfully\ntested only on the actual hardware, and is built in a VM.  Quite often\ngetting things right involve many iterations of hack hack hack, git\ncommit --amend, fetch && reset --hard in the VM, build, test, repeat.\nBeing able to clone / fetch directly from the copy I am working on makes\nit a lot easier.\n\nAs I wrote in the other e-mail, I managed to achieve the desired result\nby using ./<submodule> (without .git suffix) as the submodule URL, and\ncreating a file named <submodule> in the bare repo with\n'gitdir: ../<submodule.git>' as it's contents, but I'm not sure whether\nit is a good idea or not.\n\nJānis\n"}]}