Submodules with feature branches

I have a question regarding submodules and their applicability given our workflow at the place I work.

When I work on a feature, I normally create a feature branch. If I happen to make changes to the submodule that only work with the changes introduced in my feature branch, that seems to complicate things. For the purposes of the feature branch, do I need to create a corresponding feature branch in the submodule and temporarily update the submodule URL to point to it? When I merge my feature branch, I'd have to swap it back?

What is a recommended workflow for this? Thanks in advance.

On Thu, Jun 05, 2014 at 09:03:25AM -0500, Robert Dailey wrote:

So you have:

  On the trunk host:   On your public host:   Locally:
  superproject         superproject           superproject
  submodule            submodule              `-- submodule

In that case, a corresponding feature branch to the submodule, and an update to submodule.<name>.url (and possibly submodule.<name>.branch) would be the way I'd go (at A in the figure below). Once the trunk maintainers were happy with things, they could merge the submodule branch into trunk's submodule (at B in the figure below), and you could add a capping commit to your superproject branch that reverted the gitmodule changes (at C in the figure below):

  -o---o---o---o-------o  trunk's superproject/master
    \                 /
     A---o---o---o---C    your superproject/feature

  -o---o-----------B  trunk's submodule/master
    \             /
     o---o---o----    your submodule/feature

An alternative is to use relative URLs in the trunk:

  superproject$ cat .gitmodules
  [submodule "bpl-subset"]
    path = submod
    url = ../submodule

which makes it easier for folks who mirror/fork both the superproject and submodule (no need to change submodule.<name>.url). However, it makes it harder for folks who just mirror/fork the superproject (and don't need to tweak the submodule), because they have to mirror/fork the submodule as well to support the relative URL (or edit submodule.<name>.url, which turns attempted mirrors into forks). Personally, I prefer relative URLs [1,2], but both external projects I've approached on this front have ended up with absolute URLs [3,4] ;).

This is less of an issue for loosely-coupled submodules, since you'll can motivate your submodule changes to the submodule maintainers independent of the superproject (i.e. you can just say things like “I'm extending the API so I can iterate over widgets. This lets you do things like frobbling whatsits in superproject” without having to present the associated superproject code). Once you land the submodule changes upstream, your superproject branch will work without the need to tweak the URL (for absolute URLs) or publish a sibling mirror (for relative URLs).

Cheers, Trevor

[1]: https://github.com/inducer/pycuda/pull/21 [2]: http://thread.gmane.org/gmane.comp.python.ipython.devel/10287/focus=10299 [3]: https://github.com/wking/pycuda/commit/5218bd449d6aae0bce3a3d1bf54a91377445e2f9 [4]: https://github.com/minrk/ipython/commit/4fe230e96e357b3612b6fadaeec9d8de71d6fca9

On Thu, Jun 5, 2014 at 10:15 AM, W. Trevor King <wking@tremily.us> wrote:

Thanks, this is excellent information. Perhaps I should provide a little more detail into what I'm doing. I know that having such dependencies between superproject & submodule is bad and creates complications like this, so maybe there is a different approach.

Right now our build system does not download third party dependencies. We build on Windows & Android, so we maintain our own package binaries & source code. Right now these are stored in 7zip files and checked into the superproject. I was planning on creating a submodule for our third party libs and store them extracted in there. That way, when we switch branches, we don't have to delete & re-extract the third party archives again (since between branches, libraries may change, be added or removed). The submodule buys us an important thing, which is that we won't have big binary blobs in our history. If we ever want to remove them, all we're removing is a weak link to the submodule. The binaries live with us forever since they're in history.

The only other thing I can think to do is incorporate logic into our makefiles to copy down the 7zips from a permanent server, and we'd adopt a naming format for the archives and download them based on that information. This way, when we go back to an earlier tag, it will always pull the correct version of the dependencies. We'd have to make sure to never delete old libraries from the remote server or edit existing ones.

I was exploring submodules to see if they would solve this problem. However, because of the feature branch workflow, they do not seem practical. I'm open to any other suggestions. Thanks!!

On Thu, Jun 05, 2014 at 10:57:17AM -0500, Robert Dailey wrote:

3rd party libraries sound loosely-coupled to me ;). In one of my more mature projects I did a similar thing, and just used relative URLs [1] and sibling mirrors/forks [2,3,4].

Cheers, Trevor

[1]: https://github.com/wking/pygrader/blob/master/.gitmodules [2]: https://github.com/wking/pgp-mime [3]: https://github.com/wking/pyassuan [4]: https://github.com/wking/jinja2

On Thu, Jun 5, 2014 at 11:23 AM, W. Trevor King <wking@tremily.us> wrote:

I guess I'm still confused on how relative URLs help here. Won't the capping commits (A and C in your first email) still be needed? Or is there a way I can modify the local "../third-party.git" submodule repo instead? Can you explain?

Unfortunately, the reason why I feel third party in a submodule creates tight coupling is because:

* You can't make changes to third party libs for your feature branch
without breaking the trunk
* Merge conflicts are insane to resolve and involve two clones if
trunk maintainers modify third party binaries and you do as well.
* Feature branching requires those capping / meta commits to simply
setup your branch to be a feature branch.

Instead of just creating my branch and starting to make commits, I now have to setup my submodule branch first. Also pull requests won't show the changes to the third party libraries unless I do a second pull request for the third party repo.

It just seems like a mess :-(

On Thu, Jun 05, 2014 at 01:31:39PM -0500, Robert Dailey wrote:

If you want to add a feature to pygrader that needs tweaks to pgp-mime, you can put your public repositories somewhere as siblings, and:

  $ git clone --recursive git://you.net/pygrader.git

will work fine (drawing from git://you.net/pgp-mime.git, etc.).

Anyone reviewing your changes locally will need a way to get your submodule commits as well as your superproject commits. In both cases, they can use the usual:

  $ git add remote you git://you.net/….git
  $ git fetch

or other tweaks like GitHub's refs/pull/*/head namespace [1]. Even a shared central repository, if that's how your team rolls.

You can in a branch. Maybe I'm missing something here. In any case, edits to third party libs are best upstreamed ;).

You resolve the merge conflicts in the submodule, and then amend the superproject merge commit to point to the resolved submodule commit. That is one --amend away from what you're doing without submodules.

With relative URLs (or shared centralized repository, or a refs/pull/*/head namespace) it's easy to share the commits themselves. Unless you're using 'git submodule update --remote …', you don't need to care where the gitlinked commits live, you just need to get them into the submodule repository somehow. That seems fairly orthogonal to feature branching to me.

That I agree with ;). However, if you're treating the third-party library as a separate repo, I think it makes sense that you need to be making branches and pull requests in the submodule independently from your branches and pull requests in the superproject. If you feel that the minimal (branch + PR for changes) overhead of managing the projects independently is too high, you're probably better off with the single repository or subtree approach.

Cheers, Trevor

[1]: https://help.github.com/articles/checking-out-pull-requests-locally

On Thu, Jun 05, 2014 at 12:00:33PM -0700, W. Trevor King wrote:

To make this more concrete, I think you'll rarely have tight one-to-one binding between third-party library changes and your superproject. More likely, you'll have some high-level feature branch in the superproject (“accept comments via email”) and an unrelated number of prerequisite feature branches for your libraries (“add support for MIME documents,” “parse RFC 2822 dates,” …). You only have synchronized branches when you mess with the API tying components together (updating the submodule API and updating the superproject to use it). With good library design, that type of API migration should happen more and more rarely as the library stabilizes.

Cheers, Trevor