{"thread":{"id":"36846","subject":"Submodules with feature branches","startedAt":"2014-06-05T14:03:25Z","lastAt":"2014-06-05T19:18:29Z","messageCount":7,"participants":["Robert Dailey","W. Trevor King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"243394","messageId":"CAHd499Bn7CCVy=vhFzpLYXCssxR0oGxm3Vdgou_Yk5zSt1gfmA@mail.gmail.com","threadId":"36846","inReplyTo":null,"subject":"Submodules with feature branches","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2014-06-05T14:03:25Z","receivedAt":"2014-06-05T14:03:25Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"I have a question regarding submodules and their applicability given\nour workflow at the place I work.\n\nWhen I work on a feature, I normally create a feature branch. If I\nhappen to make changes to the submodule that only work with the\nchanges introduced in my feature branch, that seems to complicate\nthings. For the purposes of the feature branch, do I need to create a\ncorresponding feature branch in the submodule and temporarily update\nthe submodule URL to point to it? When I merge my feature branch, I'd\nhave to swap it back?\n\nWhat is a recommended workflow for this? Thanks in advance.\n"},{"id":"243400","messageId":"20140605151549.GQ21803@odin.tremily.us","threadId":"36846","inReplyTo":"CAHd499Bn7CCVy=vhFzpLYXCssxR0oGxm3Vdgou_Yk5zSt1gfmA@mail.gmail.com","subject":"Re: Submodules with feature branches","fromName":"W. Trevor King","fromEmail":"wking@tremily.us","sentAt":"2014-06-05T15:15:49Z","receivedAt":"2014-06-05T15:15:49Z","isPatch":false,"sender":{"key":"wking@tremily.us","avatar":"https://avatars.githubusercontent.com/u/209920?v=4"},"body":"On Thu, Jun 05, 2014 at 09:03:25AM -0500, Robert Dailey wrote:\n> When I work on a feature, I normally create a feature branch. If I\n> happen to make changes to the submodule that only work with the\n> changes introduced in my feature branch, that seems to complicate\n> things. For the purposes of the feature branch, do I need to create\n> a corresponding feature branch in the submodule and temporarily\n> update the submodule URL to point to it? When I merge my feature\n> branch, I'd have to swap it back?\n\nSo you have:\n\n  On the trunk host:   On your public host:   Locally:\n  superproject         superproject           superproject\n  submodule            submodule              `-- submodule\n\nIn that case, a corresponding feature branch to the submodule, and an\nupdate to submodule.<name>.url (and possibly submodule.<name>.branch)\nwould be the way I'd go (at A in the figure below).  Once the trunk\nmaintainers were happy with things, they could merge the submodule\nbranch into trunk's submodule (at B in the figure below), and you\ncould add a capping commit to your superproject branch that reverted\nthe gitmodule changes (at C in the figure below):\n\n  -o---o---o---o-------o  trunk's superproject/master\n    \\                 /\n     A---o---o---o---C    your superproject/feature\n\n  -o---o-----------B  trunk's submodule/master\n    \\             /\n     o---o---o----    your submodule/feature\n\nAn alternative is to use relative URLs in the trunk:\n\n  superproject$ cat .gitmodules\n  [submodule \"bpl-subset\"]\n    path = submod\n    url = ../submodule\n\nwhich makes it easier for folks who mirror/fork both the superproject\nand submodule (no need to change submodule.<name>.url).  However, it\nmakes it harder for folks who just mirror/fork the superproject (and\ndon't need to tweak the submodule), because they have to mirror/fork\nthe submodule as well to support the relative URL (or edit\nsubmodule.<name>.url, which turns attempted mirrors into forks).\nPersonally, I prefer relative URLs [1,2], but both external projects\nI've approached on this front have ended up with absolute URLs [3,4]\n;).\n\nThis is less of an issue for loosely-coupled submodules, since you'll\ncan motivate your submodule changes to the submodule maintainers\nindependent of the superproject (i.e. you can just say things like\n“I'm extending the API so I can iterate over widgets.  This lets you\ndo things like frobbling whatsits in superproject” without having to\npresent the associated superproject code).  Once you land the\nsubmodule changes upstream, your superproject branch will work without\nthe need to tweak the URL (for absolute URLs) or publish a sibling\nmirror (for relative URLs).\n\nCheers,\nTrevor\n\n[1]: https://github.com/inducer/pycuda/pull/21\n[2]: http://thread.gmane.org/gmane.comp.python.ipython.devel/10287/focus=10299 \n[3]: https://github.com/wking/pycuda/commit/5218bd449d6aae0bce3a3d1bf54a91377445e2f9\n[4]: https://github.com/minrk/ipython/commit/4fe230e96e357b3612b6fadaeec9d8de71d6fca9\n\n-- \nThis email may be signed or encrypted with GnuPG (http://www.gnupg.org).\nFor more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy\n"},{"id":"243404","messageId":"CAHd499Dc7_fob2-X1KZ77sdx20r+erQ_9JbDc7y4G0RUxG65eg@mail.gmail.com","threadId":"36846","inReplyTo":"20140605151549.GQ21803@odin.tremily.us","subject":"Re: Submodules with feature branches","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2014-06-05T15:57:17Z","receivedAt":"2014-06-05T15:57:17Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Thu, Jun 5, 2014 at 10:15 AM, W. Trevor King <wking@tremily.us> wrote:\n> So you have:\n>\n>   On the trunk host:   On your public host:   Locally:\n>   superproject         superproject           superproject\n>   submodule            submodule              `-- submodule\n>\n> In that case, a corresponding feature branch to the submodule, and an\n> update to submodule.<name>.url (and possibly submodule.<name>.branch)\n> would be the way I'd go (at A in the figure below).  Once the trunk\n> maintainers were happy with things, they could merge the submodule\n> branch into trunk's submodule (at B in the figure below), and you\n> could add a capping commit to your superproject branch that reverted\n> the gitmodule changes (at C in the figure below):\n>\n>   -o---o---o---o-------o  trunk's superproject/master\n>     \\                 /\n>      A---o---o---o---C    your superproject/feature\n>\n>   -o---o-----------B  trunk's submodule/master\n>     \\             /\n>      o---o---o----    your submodule/feature\n>\n> An alternative is to use relative URLs in the trunk:\n>\n>   superproject$ cat .gitmodules\n>   [submodule \"bpl-subset\"]\n>     path = submod\n>     url = ../submodule\n>\n> which makes it easier for folks who mirror/fork both the superproject\n> and submodule (no need to change submodule.<name>.url).  However, it\n> makes it harder for folks who just mirror/fork the superproject (and\n> don't need to tweak the submodule), because they have to mirror/fork\n> the submodule as well to support the relative URL (or edit\n> submodule.<name>.url, which turns attempted mirrors into forks).\n> Personally, I prefer relative URLs [1,2], but both external projects\n> I've approached on this front have ended up with absolute URLs [3,4]\n> ;).\n>\n> This is less of an issue for loosely-coupled submodules, since you'll\n> can motivate your submodule changes to the submodule maintainers\n> independent of the superproject (i.e. you can just say things like\n> “I'm extending the API so I can iterate over widgets.  This lets you\n> do things like frobbling whatsits in superproject” without having to\n> present the associated superproject code).  Once you land the\n> submodule changes upstream, your superproject branch will work without\n> the need to tweak the URL (for absolute URLs) or publish a sibling\n> mirror (for relative URLs).\n\nThanks, this is excellent information. Perhaps I should provide a\nlittle more detail into what I'm doing. I know that having such\ndependencies between superproject & submodule is bad and creates\ncomplications like this, so maybe there is a different approach.\n\nRight now our build system does not download third party dependencies.\nWe build on Windows & Android, so we maintain our own package binaries\n& source code. Right now these are stored in 7zip files and checked\ninto the superproject. I was planning on creating a submodule for our\nthird party libs and store them extracted in there. That way, when we\nswitch branches, we don't have to delete & re-extract the third party\narchives again (since between branches, libraries may change, be added\nor removed). The submodule buys us an important thing, which is that\nwe won't have big binary blobs in our history. If we ever want to\nremove them, all we're removing is a weak link to the submodule. The\nbinaries live with us forever since they're in history.\n\nThe only other thing I can think to do is incorporate logic into our\nmakefiles to copy down the 7zips from a permanent server, and we'd\nadopt a naming format for the archives and download them based on that\ninformation. This way, when we go back to an earlier tag, it will\nalways pull the correct version of the dependencies. We'd have to make\nsure to never delete old libraries from the remote server or edit\nexisting ones.\n\nI was exploring submodules to see if they would solve this problem.\nHowever, because of the feature branch workflow, they do not seem\npractical. I'm open to any other suggestions. Thanks!!\n"},{"id":"243405","messageId":"20140605162333.GR21803@odin.tremily.us","threadId":"36846","inReplyTo":"CAHd499Dc7_fob2-X1KZ77sdx20r+erQ_9JbDc7y4G0RUxG65eg@mail.gmail.com","subject":"Re: Submodules with feature branches","fromName":"W. Trevor King","fromEmail":"wking@tremily.us","sentAt":"2014-06-05T16:23:33Z","receivedAt":"2014-06-05T16:23:33Z","isPatch":false,"sender":{"key":"wking@tremily.us","avatar":"https://avatars.githubusercontent.com/u/209920?v=4"},"body":"On Thu, Jun 05, 2014 at 10:57:17AM -0500, Robert Dailey wrote:\n> I was planning on creating a submodule for our third party libs and\n> store them extracted in there.\n\n3rd party libraries sound loosely-coupled to me ;).  In one of my more\nmature projects I did a similar thing, and just used relative URLs [1]\nand sibling mirrors/forks [2,3,4].\n\nCheers,\nTrevor\n\n[1]: https://github.com/wking/pygrader/blob/master/.gitmodules\n[2]: https://github.com/wking/pgp-mime\n[3]: https://github.com/wking/pyassuan\n[4]: https://github.com/wking/jinja2\n\n-- \nThis email may be signed or encrypted with GnuPG (http://www.gnupg.org).\nFor more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy\n"},{"id":"243410","messageId":"CAHd499CBAQHG4rdojb8pdjymUCaZNYSnKb-ksmsLesq73OWTyA@mail.gmail.com","threadId":"36846","inReplyTo":"20140605162333.GR21803@odin.tremily.us","subject":"Re: Submodules with feature branches","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2014-06-05T18:31:39Z","receivedAt":"2014-06-05T18:31:39Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Thu, Jun 5, 2014 at 11:23 AM, W. Trevor King <wking@tremily.us> wrote:\n> 3rd party libraries sound loosely-coupled to me ;).  In one of my more\n> mature projects I did a similar thing, and just used relative URLs [1]\n> and sibling mirrors/forks [2,3,4].\n>\n> Cheers,\n> Trevor\n>\n> [1]: https://github.com/wking/pygrader/blob/master/.gitmodules\n> [2]: https://github.com/wking/pgp-mime\n> [3]: https://github.com/wking/pyassuan\n> [4]: https://github.com/wking/jinja2\n\nI guess I'm still confused on how relative URLs help here. Won't the\ncapping commits (A and C in your first email) still be needed? Or is\nthere a way I can modify the local \"../third-party.git\" submodule repo\ninstead? Can you explain?\n\nUnfortunately, the reason why I feel third party in a submodule\ncreates tight coupling is because:\n\n* You can't make changes to third party libs for your feature branch\nwithout breaking the trunk\n* Merge conflicts are insane to resolve and involve two clones if\ntrunk maintainers modify third party binaries and you do as well.\n* Feature branching requires those capping / meta commits to simply\nsetup your branch to be a feature branch.\n\nInstead of just creating my branch and starting to make commits, I now\nhave to setup my submodule branch first. Also pull requests won't show\nthe changes to the third party libraries unless I do a second pull\nrequest for the third party repo.\n\nIt just seems like a mess :-(\n"},{"id":"243414","messageId":"20140605190033.GV21803@odin.tremily.us","threadId":"36846","inReplyTo":"CAHd499CBAQHG4rdojb8pdjymUCaZNYSnKb-ksmsLesq73OWTyA@mail.gmail.com","subject":"Re: Submodules with feature branches","fromName":"W. Trevor King","fromEmail":"wking@tremily.us","sentAt":"2014-06-05T19:00:33Z","receivedAt":"2014-06-05T19:00:33Z","isPatch":false,"sender":{"key":"wking@tremily.us","avatar":"https://avatars.githubusercontent.com/u/209920?v=4"},"body":"On Thu, Jun 05, 2014 at 01:31:39PM -0500, Robert Dailey wrote:\n> On Thu, Jun 5, 2014 at 11:23 AM, W. Trevor King wrote:\n> > 3rd party libraries sound loosely-coupled to me ;).  In one of my more\n> > mature projects I did a similar thing, and just used relative URLs [1]\n> > and sibling mirrors/forks [2,3,4].\n> >\n> > Cheers,\n> > Trevor\n> >\n> > [1]: https://github.com/wking/pygrader/blob/master/.gitmodules\n> > [2]: https://github.com/wking/pgp-mime\n> > [3]: https://github.com/wking/pyassuan\n> > [4]: https://github.com/wking/jinja2\n> \n> I guess I'm still confused on how relative URLs help here.\n\nIf you want to add a feature to pygrader that needs tweaks to\npgp-mime, you can put your public repositories somewhere as siblings,\nand:\n\n  $ git clone --recursive git://you.net/pygrader.git\n\nwill work fine (drawing from git://you.net/pgp-mime.git, etc.).\n\n> Won't the capping commits (A and C in your first email) still be\n> needed? Or is there a way I can modify the local\n> \"../third-party.git\" submodule repo instead? Can you explain?\n\nAnyone reviewing your changes locally will need a way to get your\nsubmodule commits as well as your superproject commits.  In both\ncases, they can use the usual:\n\n  $ git add remote you git://you.net/….git\n  $ git fetch\n\nor other tweaks like GitHub's refs/pull/*/head namespace [1].  Even a\nshared central repository, if that's how your team rolls.\n\n> Unfortunately, the reason why I feel third party in a submodule\n> creates tight coupling is because:\n> \n> * You can't make changes to third party libs for your feature branch\n> without breaking the trunk\n\nYou can in a branch.  Maybe I'm missing something here.  In any case,\nedits to third party libs are best upstreamed ;).\n\n> * Merge conflicts are insane to resolve and involve two clones if\n> trunk maintainers modify third party binaries and you do as well.\n\nYou resolve the merge conflicts in the submodule, and then amend the\nsuperproject merge commit to point to the resolved submodule commit.\nThat is one --amend away from what you're doing without submodules.\n\n> * Feature branching requires those capping / meta commits to simply\n> setup your branch to be a feature branch.\n\nWith relative URLs (or shared centralized repository, or a\nrefs/pull/*/head namespace) it's easy to share the commits themselves.\nUnless you're using 'git submodule update --remote …', you don't need\nto care where the gitlinked commits live, you just need to get them\ninto the submodule repository somehow.  That seems fairly orthogonal\nto feature branching to me.\n\n> Instead of just creating my branch and starting to make commits, I\n> now have to setup my submodule branch first. Also pull requests\n> won't show the changes to the third party libraries unless I do a\n> second pull request for the third party repo.\n\nThat I agree with ;).  However, if you're treating the third-party\nlibrary as a separate repo, I think it makes sense that you need to be\nmaking branches and pull requests in the submodule independently from\nyour branches and pull requests in the superproject.  If you feel that\nthe minimal (branch + PR for changes) overhead of managing the\nprojects independently is too high, you're probably better off with\nthe single repository or subtree approach.\n\nCheers,\nTrevor\n\n[1]: https://help.github.com/articles/checking-out-pull-requests-locally\n\n-- \nThis email may be signed or encrypted with GnuPG (http://www.gnupg.org).\nFor more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy\n"},{"id":"243413","messageId":"20140605191829.GA32192@odin.tremily.us","threadId":"36846","inReplyTo":"20140605190033.GV21803@odin.tremily.us","subject":"Re: Submodules with feature branches","fromName":"W. Trevor King","fromEmail":"wking@tremily.us","sentAt":"2014-06-05T19:18:29Z","receivedAt":"2014-06-05T19:18:29Z","isPatch":false,"sender":{"key":"wking@tremily.us","avatar":"https://avatars.githubusercontent.com/u/209920?v=4"},"body":"On Thu, Jun 05, 2014 at 12:00:33PM -0700, W. Trevor King wrote:\n> On Thu, Jun 05, 2014 at 01:31:39PM -0500, Robert Dailey wrote:\n> > Instead of just creating my branch and starting to make commits, I\n> > now have to setup my submodule branch first. Also pull requests\n> > won't show the changes to the third party libraries unless I do a\n> > second pull request for the third party repo.\n> \n> That I agree with ;).  However, if you're treating the third-party\n> library as a separate repo, I think it makes sense that you need to\n> be making branches and pull requests in the submodule independently\n> from your branches and pull requests in the superproject.\n\nTo make this more concrete, I think you'll rarely have tight\none-to-one binding between third-party library changes and your\nsuperproject.  More likely, you'll have some high-level feature branch\nin the superproject (“accept comments via email”) and an unrelated\nnumber of prerequisite feature branches for your libraries (“add\nsupport for MIME documents,” “parse RFC 2822 dates,” …).  You only\nhave synchronized branches when you mess with the API tying components\ntogether (updating the submodule API and updating the superproject to\nuse it).  With good library design, that type of API migration should\nhappen more and more rarely as the library stabilizes.\n\nCheers,\nTrevor\n\n-- \nThis email may be signed or encrypted with GnuPG (http://www.gnupg.org).\nFor more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy\n"}]}