{"thread":{"id":"23950","subject":"Simplifying work across multiple projects (while tracking relationships among commit histories)","startedAt":"2010-05-31T06:41:28Z","lastAt":"2010-05-31T08:56:27Z","messageCount":2,"participants":["Yang Zhang","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"142602","messageId":"AANLkTimC0SoVAG-2xByZFFitl2KNlAhYlAw5fN_XqriT@mail.gmail.com","threadId":"23950","inReplyTo":null,"subject":"Simplifying work across multiple projects (while tracking relationships among commit histories)","fromName":"Yang Zhang","fromEmail":"yanghatespam@gmail.com","sentAt":"2010-05-31T06:41:28Z","receivedAt":"2010-05-31T06:41:28Z","isPatch":false,"sender":{"key":"yanghatespam@gmail.com","avatar":"https://gravatar.com/avatar/e0796ebc342b90612276b87755da4abc776ddfebbc23049a79bb8b192570161d?d=mp&s=160"},"body":"After looking at some of the tools/techniques out there for working\nwith multiple git projects (submodules, subtree merge, braid, repo),\nit seems that none are really well-suited for our use case. We're\ndeveloping a large system consisting of several components (libraries,\nservers, applications, etc.). None of these components will ever exist\nor be released as a stand-alone product. We're in \"rapid development\"\nmode, so we're not even close to dealing with e.g. manually\nmaintaining information on versions/dependencies, and we just want\nvery tight integration among all the components -- yet the components\ndo deserve their own disentangled histories and (eventually)\nindependent branches/tags/versions/etc.\n\nIf we were using svn, all the code would live in a single repository,\nand that would be all there was to think about this. However, it seems\nthat our use case (surprisingly) doesn't have a lot of good support in\nthe DVCS world.\n\nFor now, we'll probably just have some simple scripts that basically\ndo 'for i in $projects' loops for pulls, pushes, commits, etc.\nHowever, this loses a lot of information that should be tracked about\nthe version/dependency information among the projects -- information\nthat at the same time we're not interested in manually tracking. We're\ncurrently thinking of having a simple system that is initially set up\nwith a dependency graph among projects, e.g.:\n\n  a: no dependencies\n  b: depends on a\n\nand whenever a commit is made to a project with dependencies (b), the\ncommit (perhaps in the commit message) contains a reference to the\nparticular versions of the dependent project(s) (a) that were checked\nout.\n\nThe tool could simplify the use of such a scheme, e.g.:\n\n- automatically augmenting commit messages with this information\n- on commits/pushes, first commit/push the dependent projects\n- checking out consistent versions of all the projects (or subgraphs thereof)\n\nDoes this make sense to others? Are we overlooking a better/existing\napproach? Would it be worth building this? Suggestions on design\nimprovements to such a tool over what was described (e.g. better\napproach than augmenting commit messages)?\n--\nYang Zhang\nhttp://yz.mit.edu/\n"},{"id":"142603","messageId":"20100531085627.GA2094@progeny.tock","threadId":"23950","inReplyTo":"AANLkTimC0SoVAG-2xByZFFitl2KNlAhYlAw5fN_XqriT@mail.gmail.com","subject":"Re: Simplifying work across multiple projects (while tracking relationships among commit histories)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-05-31T08:56:27Z","receivedAt":"2010-05-31T08:56:27Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Yang,\n\nYang Zhang wrote:\n\n> We're\n> developing a large system consisting of several components (libraries,\n> servers, applications, etc.).\n[...]\n> For now, we'll probably just have some simple scripts that basically\n> do 'for i in $projects' loops for pulls, pushes, commits, etc.\n> However, this loses a lot of information that should be tracked about\n> the version/dependency information among the projects -- information\n> that at the same time we're not interested in manually tracking. We're\n> currently thinking of having a simple system that is initially set up\n> with a dependency graph among projects, e.g.:\n> \n>   a: no dependencies\n>   b: depends on a\n> \n> and whenever a commit is made to a project with dependencies (b), the\n> commit (perhaps in the commit message) contains a reference to the\n> particular versions of the dependent project(s) (a) that were checked\n> out.\n\nIt sounds to me like submodules would be a better approach.  Because\nit fits my great love of complaining (and I would like to hear what\nsolutions you come up with, if any), let me try to go through the\nproblems you would run into.\n\nI am not a submodule developer or heavy user, so please check anything\nI say before relying on it.\n\n1. Suppose you are working on the program frobber and you notice it\n   contains a usable sub-component veryfastregexp.  So you make a new\n   repository for it, make sure it builds on its own, and publish.\n   As always when starting a new project, there is a question of how\n   much early history to preserve.  Probably best to start with a\n   single commit, and provide a separate branch with\n   ‘filter-branch --subdirectory-filter’ output if you are feeling\n   generous.\n\n   In frobber, you remove the copy of veryfastregexp, add it back\n   with ‘git submodule add git://someserver/path/to/veryfastregexp’,\n   commit, and publish.\n\n    - New clones must use ‘git clone --recursive’.  How do you\n      advertise this?\n\n    - Existing clones must use ‘git submodule update --init’ after\n      they pull.  In fact, it seems to me it’s not a bad idea to\n      always use ‘git submodule update --init --recursive’ after each\n      pull.  How do you advertise this?\n\n    - Incoming patches that touch both veryfastregexp and frobber\n      have to be split into separate patches for the two projects.\n      How?\n\n    - Pull requests are even worse (or just as bad, depending on\n      how you solved the previous problem).\n\n2. Some people like to use the latest stable version of all components\n   they use, while other people like to avoid change wherever\n   possible.  I’ll consider the latter sort of person in a moment.\n\n   The developers of frobber want to use the latest version on the\n   master branch for all components.  So they try the following:\n\n\tgit submodule foreach '\n\t\tgit checkout master &&\n\t\tgit pull &&\n\t\tgit submodule update --init --recursive\n\t'\n\n   This checks out a branch tracking the upstream master branch\n   for each submodule.\n\n   Next they run ‘git add -u’ to mention all the updated submodule\n   versions, test to make sure everything’s okay, and commit.\n\n    - This does not bring sub-submodules to the latest version at the\n      same time.  If the frobber developers wanted to do that, they\n      might try\n\n\tgit submodule foreach --recursive '\n\t\tgit checkout master &&\n\t\tgit pull &&\n\t\tgit submodule update --init --recursive\n\t'\n\n      Then they run ‘git add -u’, test, and run ‘git commit’.  But\n      the editor informs them that submodules have unstaged changes.\n      What happened, what are the consequences for others using this\n      project, and can this be avoided?\n\n      I’ll return to this in a moment.\n\n3. Some people never upgrade until forced to.\n\n   The veryfastregexp library is a resounding success, picked up\n   by other people in the company, and rapidly developed.  After a\n   particularly painful upgrade, the developers of frobber have a turn\n   for the conservative.  From now on, their policy is “necessary\n   fixes only”.  So they would like to maintain their own branch and\n   cherry-pick from master.\n\n   They put in a request for privileges to commit to their own branch,\n   and wait.  In the meantime, an important fix comes up.  So they\n   do the only thing they can do: publish a fork of veryfastregexp,\n   update .gitmodules to point to it, run ‘git add -u’ to register\n   the version they are using, test, commit, and push.\n\n   New clones made with ‘git clone --recursive’ will use the\n   project-specific version of veryfastregexp.\n\n    - Users with existing clones must update the URL with\n      ‘git remote set-url origin <new url>’ from the submodule\n      or ‘git submodule sync’; otherwise, the next time they run\n      ‘git submodule update’ there will be an ‘unable to checkout’\n      error.  How do you advertise this?\n\n    - Suppose a frobber developer tries the following from the\n      frobber repository:\n\n\tcd veryfastregexp && git cherry-pick important-fix\n\n      runs ‘git add -u’ from the toplevel, tests, commits, and\n      pushes the result.  Of course, an important step is missing: he\n      forgot to push to the veryfastregexp-frobber repository!  \n      Anyone who tries to pull this change and run ‘git submodule\n      update’ will find the commit object missing and be unable to\n      check out the new revision.\n\n      An update hook could have prevented this, since from the\n      server side it is obvious which objects a new clone will have\n      access to.  Where can one find such a hook?\n\n   The frobber developers’ request for a branch in the veryfastregexp\n   repository is granted.  So they switch .gitmodules back again\n   and keep the submodule pointed to the for-frobber branch, updating\n   as needed.\n\n    - Now the old recipe\n\n\tgit submodule foreach --recursive '\n\t\tgit checkout master &&\n\t\tgit pull &&\n\t\tgit submodule update --init --recursive\n\t'\n\n     does not work for them anymore, since this would switch\n     the frobber branch back.  What should they do to adjust?\n     How can they make it easy for new people on their team to\n     get started, too?\n\n4. The developers who want all components to be aggressively updated\n   (see #2 above) need to do something similar.  They first switch all\n   components to point to repositories with branches they own, and\n   then run aggressively-update, where aggressively-update is a\n   script something like\n\n\t#!/bin/sh\n\tgit reset --keep upstream/master &&\n\tgit submodule foreach aggressively-update &&\n\tgit add -u &&\n\tmake test &&\n\tgit commit -v &&\n\tgit push -f origin master\n\n   - Can this be modified to pick up new submodules?\n\nThoughts welcome.\nJonathan\n"}]}