{"thread":{"id":"42657","subject":"Managing sub-projects","startedAt":"2016-06-18T23:20:56Z","lastAt":"2016-06-21T23:44:36Z","messageCount":4,"participants":["Michael Eager","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"289593","messageId":"5765D752.4000404@eagerm.com","threadId":"42657","inReplyTo":null,"subject":"Managing sub-projects","fromName":"Michael Eager","fromEmail":"eager@eagerm.com","sentAt":"2016-06-18T23:20:50Z","receivedAt":"2016-06-18T23:20:56Z","isPatch":false,"sender":{"key":"eager@eagerm.com","avatar":null},"body":"I'm trying to create a git repository for a tool chain for a proprietary\nprocessor.  I'd like to create a private repo with documentation, build\nscripts, etc., which includes several sub-projects: binutils, gcc, newlib,\netc.  Each of the sub-projects will have a branch which has support for\nthe new processor.  These branches need to be maintained in my repo, not\nin the upstream repo.  I want to be able to periodically rebase these\nbranches from the upstream repo.\n\nI've looked at several schemes, but each one seems to do something other\nthan what I want.\n\nGit submodule:  Branches created in the sub-projects are pushed to the\nupstream repo, not to my repo.  I tried to change origin and created an\nupstream reference, but was not able to get changes pushed to my repo.\n\ngit subtree:  Does not maintain sub-project history or allow rebase.\n\ngit slave:  Requires multiple private repos.  Appears to require the\nsame branch names in each sub-project.\n\nrepo: Appears to work a bit like git submodules, where pushes on the\nsub-projects go to the upstream repo, not to the private repo.\n\nAny other ways to do what I want without creating a separate forked\nrepo for each of the sub-projects?  Or have I misunderstood one of\nthese schemes?\n\n-- \nMichael Eager\t eager@eagercon.com\n1960 Park Blvd., Palo Alto, CA 94306  650-325-8077\n"},{"id":"289650","messageId":"CAGZ79kYqtLGHjEirP=6ssiUf2fuHirtKDXr117bjwomho2uf3A@mail.gmail.com","threadId":"42657","inReplyTo":"5765D752.4000404@eagerm.com","subject":"Re: Managing sub-projects","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-06-20T02:01:02Z","receivedAt":"2016-06-20T02:09:33Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sat, Jun 18, 2016 at 4:20 PM, Michael Eager <eager@eagerm.com> wrote:\n>\n> Any other ways to do what I want without creating a separate forked\n> repo for each of the sub-projects?  Or have I misunderstood one of\n> these schemes?\n\nI think forking is the way to go here, as you want to have new code\nand maintain that.\n\nI assume you want to keep the history for each project separate, so\nI would recommend against using subtrees.\n\ngit slave looks interesting for your use case. (I looked at it once\nbefore, but have no deep knowledge about it as it is not part of core git)\n\nIIUC the repo tool is tailored to be a multi-repo manager optimised\nfor usage with Gerrit, not just plain Git. The repo tool tracks branches\nin the subproject instead of versions (as submodules do), so consistency\nis hard, specially when looking back in history. (Not sure if you care about\nthat, but if you want to use e.g. git bisect, having an easy reproducable\nhistory is a must)\n\nPersonally I would try out submodules.\n\n> Git submodule:  Branches created in the sub-projects are pushed to the\n> upstream repo, not to my repo.  I tried to change origin and created an\n> upstream reference, but was not able to get changes pushed to my repo.\n\nBeware that there are 2 areas you need to look at. First the submodule repo\nneeds to have a remote that points away from the projects origin (to your\nprivate fork).\n\nThen you have to look at the superproject that\n1) records the sha1 for the submodules internally\n2) all other information except the tracking sha1s must be user provided,\n    where the .gitmodules file contains recommendations (i.e. the url where to\n    obtain the submodule from, whether to clone it shallowly,\n    if we have a specific branch in mind). The contents of that file\n    are not binding, e.g. if the url provided in the .gitmodules file becomes\n    outdated later, it is still possible to setup the\nsubmodule/superproject correctly.\n\nHowever for your business purpose, you would put the url of the private forks\nin the recommended URL of the submodules.\n\nAs the superproject only tracks the sha1, and has this recommended pointer\nwhere to get the submodule repository from, you need to take special care\nin a rebase workflow, because the old rebased commits fall out of the\nreachability\nof the graph of objects, e.g.:\n\nSay you have a version `abc` in a submodule that is one commit on top of\ncanonical projects history, and `abc` is recorded as the sha1 in the\nsuperproject.\n\nThen you rebase the commit in the submodule to a newer version of the upstream,\nwhich then becomes a new commit `def` and `abc` is not referenced any more,\nso it can be garbage collected.\n\nThis is bad for the history of the superproject as it then points to\nan unreachable\ncommit in its history.\n\nTo preserve the historic non-rebased `abc` commit, you could have a\nset of branches\n(or tags) that maintain all the old non rebased versions.\n\nThis problem comes up with submodules with any workflow that requires\nnon fast forward changes (forced pushes), I think.\n\nSo maybe you need to have an alias in the submodule for rebasing, that\nis roughly:\n\nrebase:\n    if rebased history is published\n        create a tag, e.g.: \"$(date -I)-${sha1}\"\n        (and push that tag here or later?)\n    rebase as normal\n    carry on with life\n\nTo get back to your complaint:\n\n>  I tried to change origin and created an\n> upstream reference, but was not able to get changes pushed to my repo.\n\nI would imagine this to be\n\n     (cd submodule && git remote set-url origin <your fork> && git push origin)\n\nfor plain pushing in the submodule and then\n\n    $EDIT .gitmodules\n    # edit submodule.<name>.url to point at <your fork>\n\nto get the superproject correct.\n\nThanks,\nStefan\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n>\n> --\n> Michael Eager    eager@eagercon.com\n> 1960 Park Blvd., Palo Alto, CA 94306  650-325-8077\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"289820","messageId":"5769C883.3060103@eagerm.com","threadId":"42657","inReplyTo":"CAGZ79kYqtLGHjEirP=6ssiUf2fuHirtKDXr117bjwomho2uf3A@mail.gmail.com","subject":"Re: Managing sub-projects","fromName":"Michael Eager","fromEmail":"eager@eagerm.com","sentAt":"2016-06-21T23:06:43Z","receivedAt":"2016-06-21T23:07:06Z","isPatch":false,"sender":{"key":"eager@eagerm.com","avatar":null},"body":"Hi Stephan!\n\nOn 06/19/2016 07:01 PM, Stefan Beller wrote:\n> On Sat, Jun 18, 2016 at 4:20 PM, Michael Eager <eager@eagerm.com> wrote:\n>>\n>> Any other ways to do what I want without creating a separate forked\n>> repo for each of the sub-projects?  Or have I misunderstood one of\n>> these schemes?\n>\n> I think forking is the way to go here, as you want to have new code\n> and maintain that.\n\nThis was my conclusion.\n\nWhat I originally wanted was a repo with two origins, the upstream for\nthe master and public branches, and my repo for my branches.  Git may\nbe able to do all kinds of magic, but this two-origin scheme sounded\nstrange after I thought about it for a while.\n\n> Personally I would try out submodules.\n\nI've used submodules on another project.  There are some odd quirks,\nand lots of web pages which say to avoid submodules like the plague, but I\ndidn't have lots of trouble.  (After an initial bit of confusion while\ngetting familiar with submodules, which is what I can say about every\nfeature in Git.)\n\n>> Git submodule:  Branches created in the sub-projects are pushed to the\n>> upstream repo, not to my repo.  I tried to change origin and created an\n>> upstream reference, but was not able to get changes pushed to my repo.\n>\n> Beware that there are 2 areas you need to look at. First the submodule repo\n> needs to have a remote that points away from the projects origin (to your\n> private fork).\n\nI'll create an \"upstream\" remote to the project repo, so I can pull/rebase\nfrom the upstream into my forked repo.  The \"origin\" will point to my repo.\n\n> Then you have to look at the superproject that\n> 1) records the sha1 for the submodules internally\n> 2) all other information except the tracking sha1s must be user provided,\n>      where the .gitmodules file contains recommendations (i.e. the url where to\n>      obtain the submodule from, whether to clone it shallowly,\n>      if we have a specific branch in mind). The contents of that file\n>      are not binding, e.g. if the url provided in the .gitmodules file becomes\n>      outdated later, it is still possible to setup the\n> submodule/superproject correctly.\n>\n> However for your business purpose, you would put the url of the private forks\n> in the recommended URL of the submodules.\n>\n> As the superproject only tracks the sha1, and has this recommended pointer\n> where to get the submodule repository from, you need to take special care\n> in a rebase workflow, because the old rebased commits fall out of the\n> reachability\n> of the graph of objects, e.g.:\n>\n> Say you have a version `abc` in a submodule that is one commit on top of\n> canonical projects history, and `abc` is recorded as the sha1 in the\n> superproject.\n>\n> Then you rebase the commit in the submodule to a newer version of the upstream,\n> which then becomes a new commit `def` and `abc` is not referenced any more,\n> so it can be garbage collected.\n>\n> This is bad for the history of the superproject as it then points to\n> an unreachable\n> commit in its history.\n>\n> To preserve the historic non-rebased `abc` commit, you could have a\n> set of branches\n> (or tags) that maintain all the old non rebased versions.\n\nSounds like every time I rebase, I should tag the repo to annotate this,\nand (as a side effect) retain the history.\n\n> This problem comes up with submodules with any workflow that requires\n> non fast forward changes (forced pushes), I think.\n>\n> So maybe you need to have an alias in the submodule for rebasing, that\n> is roughly:\n>\n> rebase:\n>      if rebased history is published\n>          create a tag, e.g.: \"$(date -I)-${sha1}\"\n>          (and push that tag here or later?)\n>      rebase as normal\n>      carry on with life\n\nWhat do you mean \"if rebased history is published\".\n\nGenerally I'd apply a tag after the rebase was completed successfully,\nthen push both the updated branch and tags to my repo.\n\n> To get back to your complaint:\n>\n>>   I tried to change origin and created an\n>> upstream reference, but was not able to get changes pushed to my repo.\n>\n> I would imagine this to be\n>\n>       (cd submodule && git remote set-url origin <your fork> && git push origin)\n>\n> for plain pushing in the submodule and then\n>\n>      $EDIT .gitmodules\n>      # edit submodule.<name>.url to point at <your fork>\n>\n> to get the superproject correct.\n\nThanks.\n\n\n-- \nMichael Eager\t eager@eagercon.com\n1960 Park Blvd., Palo Alto, CA 94306  650-325-8077\n"},{"id":"289821","messageId":"CAGZ79kYaUH5gqiVqjVwG9QYyLvh2FAVdYMU+MiyWmbL2YabphQ@mail.gmail.com","threadId":"42657","inReplyTo":"5769C883.3060103@eagerm.com","subject":"Re: Managing sub-projects","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-06-21T23:36:55Z","receivedAt":"2016-06-21T23:44:36Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Hi Michael,\n\nOn Tue, Jun 21, 2016 at 4:06 PM, Michael Eager <eager@eagerm.com> wrote:\n> Hi Stephan!\n>\n> On 06/19/2016 07:01 PM, Stefan Beller wrote:\n>>\n>> On Sat, Jun 18, 2016 at 4:20 PM, Michael Eager <eager@eagerm.com> wrote:\n>>>\n>>>\n>>> Any other ways to do what I want without creating a separate forked\n>>> repo for each of the sub-projects?  Or have I misunderstood one of\n>>> these schemes?\n>>\n>>\n>> I think forking is the way to go here, as you want to have new code\n>> and maintain that.\n>\n>\n> This was my conclusion.\n>\n> What I originally wanted was a repo with two origins, the upstream for\n> the master and public branches, and my repo for my branches.  Git may\n> be able to do all kinds of magic, but this two-origin scheme sounded\n> strange after I thought about it for a while.\n\nWell, 2 origins sound strange indeed, but \"origin\" is just a name to point at\na remote place. You could have [\"origin\" , \"private\"].\n\nOnce upon a time, I used [\"mainline\", \"origin\", \"<other peoples name>\", ...],\nwhich I confused myself with, so now I am down to\n[\"origin\", \"private\", \"<other peoples name\">].\n\nThe difference for my work flow is that I have read permissions only\non all but one remote.\n\n>\n>> Personally I would try out submodules.\n>\n>\n> I've used submodules on another project.  There are some odd quirks,\n> and lots of web pages which say to avoid submodules like the plague, but I\n> didn't have lots of trouble.  (After an initial bit of confusion while\n> getting familiar with submodules, which is what I can say about every\n> feature in Git.)\n>\n>>> Git submodule:  Branches created in the sub-projects are pushed to the\n>>> upstream repo, not to my repo.  I tried to change origin and created an\n>>> upstream reference, but was not able to get changes pushed to my repo.\n>>\n>>\n>> Beware that there are 2 areas you need to look at. First the submodule\n>> repo\n>> needs to have a remote that points away from the projects origin (to your\n>> private fork).\n>\n>\n> I'll create an \"upstream\" remote to the project repo, so I can pull/rebase\n> from the upstream into my forked repo.  The \"origin\" will point to my repo.\n\nThat is similar to the \"mainline\" I mention above. :)\n\nIn your work flow is there such a thing of an upstream of the\nsuperpoject containing\nall these subprojects? I thought that was a collection you are\nultimately creating,\nsuch that the superproject has only one remote (your authoritative copy), while\neach submodule has 2 remotes \"upstream\" (that I assume to be read only for you)\nand an \"origin\" (your maintained version, which then contains stuff that is\nreferenced by your superproject).\n\n>\n>\n>> Then you have to look at the superproject that\n>> 1) records the sha1 for the submodules internally\n>> 2) all other information except the tracking sha1s must be user provided,\n>>      where the .gitmodules file contains recommendations (i.e. the url\n>> where to\n>>      obtain the submodule from, whether to clone it shallowly,\n>>      if we have a specific branch in mind). The contents of that file\n>>      are not binding, e.g. if the url provided in the .gitmodules file\n>> becomes\n>>      outdated later, it is still possible to setup the\n>> submodule/superproject correctly.\n>>\n>> However for your business purpose, you would put the url of the private\n>> forks\n>> in the recommended URL of the submodules.\n>>\n>> As the superproject only tracks the sha1, and has this recommended pointer\n>> where to get the submodule repository from, you need to take special care\n>> in a rebase workflow, because the old rebased commits fall out of the\n>> reachability\n>> of the graph of objects, e.g.:\n>>\n>> Say you have a version `abc` in a submodule that is one commit on top of\n>> canonical projects history, and `abc` is recorded as the sha1 in the\n>> superproject.\n>>\n>> Then you rebase the commit in the submodule to a newer version of the\n>> upstream,\n>> which then becomes a new commit `def` and `abc` is not referenced any\n>> more,\n>> so it can be garbage collected.\n>>\n>> This is bad for the history of the superproject as it then points to\n>> an unreachable\n>> commit in its history.\n>>\n>> To preserve the historic non-rebased `abc` commit, you could have a\n>> set of branches\n>> (or tags) that maintain all the old non rebased versions.\n>\n>\n> Sounds like every time I rebase, I should tag the repo to annotate this,\n> and (as a side effect) retain the history.\n>\n>> This problem comes up with submodules with any workflow that requires\n>> non fast forward changes (forced pushes), I think.\n>>\n>> So maybe you need to have an alias in the submodule for rebasing, that\n>> is roughly:\n>>\n>> rebase:\n>>      if rebased history is published\n>>          create a tag, e.g.: \"$(date -I)-${sha1}\"\n>>          (and push that tag here or later?)\n>>      rebase as normal\n>>      carry on with life\n>\n>\n> What do you mean \"if rebased history is published\".\n\nbad wording.\n\n    \"If the commits that you are going to rebase are published:\"\n\n\n\n>\n> Generally I'd apply a tag after the rebase was completed successfully,\n> then push both the updated branch and tags to my repo.\n\nSure, you can also tag after rebasing.\n\nYou only need to make sure that the history that is lost during rebase is not\npointed at from the superproject, which is the case in either version. When\nfirst rebasing and then tagging, the tag is preserving the history for the next\nrebase though, which is why I did not think of it first.\n\n>\n>> To get back to your complaint:\n>>\n>>>   I tried to change origin and created an\n>>> upstream reference, but was not able to get changes pushed to my repo.\n>>\n>>\n>> I would imagine this to be\n>>\n>>       (cd submodule && git remote set-url origin <your fork> && git push\n>> origin)\n>>\n>> for plain pushing in the submodule and then\n>>\n>>      $EDIT .gitmodules\n>>      # edit submodule.<name>.url to point at <your fork>\n>>\n>> to get the superproject correct.\n>\n>\n> Thanks.\n>\n\nThanks,\nStefan\n"}]}