{"thread":{"id":"11440","subject":"How to structure a project distributed with varyingly interdependent feature branches?","startedAt":"2007-12-31T22:20:13Z","lastAt":"2008-01-15T19:02:08Z","messageCount":3,"participants":["Matt McCutchen","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"64280","messageId":"1199139613.2360.83.camel@localhost","threadId":"11440","inReplyTo":null,"subject":"How to structure a project distributed with varyingly interdependent feature branches?","fromName":"Matt McCutchen","fromEmail":"matt@mattmccutchen.net","sentAt":"2007-12-31T22:20:13Z","receivedAt":"2007-12-31T22:20:13Z","isPatch":false,"sender":{"key":"matt@mattmccutchen.net","avatar":"https://avatars.githubusercontent.com/u/8885753?v=4"},"body":"Dear git people,\n\nThe rsync source repository has recently been migrated from CVS to git\nat my encouragement ( http://git.samba.org/rsync.git/ ), and I am trying\nto figure out a better way to structure the repository (to propose to\nthe maintainer, Wayne Davison, or to use myself).\n\nRsync presents a unique challenge because it has a number of long-term\nfeature patches that are maintained and distributed with the main source\ncode.  Furthermore, some of the patches depend on other patches.  With\nCVS, the patch files were just stored in the repository; this worked,\nbut browsing and developing the branched versions represented by the\npatches required a lot of extra work.  My hope now is to represent these\nbranches and their dependencies properly in git so that the git tools\nhandle them with the same elegance with which they handle everything\nelse.\n\nThe feature branches seem to present two fundamental problems:\n\n1. How to properly represent the history of an individual branch and\nupdate it when the trunk (or the branch on which it depends) changes.\nRight now, Wayne updates the branch by rebasing; unfortunately, if the\ntrunk changes in such a way that one of the intermediate commits no\nlonger makes sense, it is impossible to update the branch while\npreserving a record that the intermediate commit once existed.  Merging\ngives the right history and is very well-supported by the git tools, but\nit has one fatal flaw: when a dependency on another branch is removed,\nthere is no way to undo the merge of the other branch.  One can revert\nthe tree changes, but the other branch is still present in the ancestry,\nwhich causes problems if the dependency is added again.  It seems to me\nthat overcoming this flaw would require significant changes to git.  Is\nthere some other workflow that does what is needed?  Ideally \"git blame\"\nwould work correctly.\n\n2. How to manage the large number of branches in a practical way.  The\nbranch dependency information should be pullable from the main\nrepository to minimize the need for each developer to configure his/her\nrepository manually on initial cloning and when dependencies change.\nAlso, for an rsync release, it should be possible to tag the trunk and\nall branches as a unit so that developers can ask questions like \"which\nbranches changed between these two pre-releases?\".  My feeling is that\nthe solution will involve a taggable, pullable master tree object that\ncontains the trunk and branches as submodules and also some\nmetainformation (including the dependencies).  However, special scripts\nwill then be needed to conveniently check out and update the submodules\nusing a single rsync source/build tree.\n\nI'm sure the rsync project could make do without a fully general\nsolution for the feature branches, but it would be awesome (and a\ntestament to git's flexibility) if one existed.  Also, Linux\ndistributions maintain packages with evolving and potentially\ninterdependent packages, and a good solution would make it possible for\nthem to do all their package maintenance and patching with git.  Ubuntu\n[1] and Fedora [2] are thinking about this.  From my perspective, such a\nsetup would make modifying distribution packages much less painful.\n\n[1] https://wiki.ubuntu.com/NoMoreSourcePackages/GitImpl\n[2] http://fcp.surfsite.org/modules/newbb/viewtopic.php?topic_id=38716&start=0#forumpost175743\n\nI would appreciate any ideas toward a solution.\n\nI am not subscribed to the list (I once was but the traffic overwhelmed\nme), so please be sure to CC me in replies.\n\nMatt\n"},{"id":"64356","messageId":"7vwsqrvpur.fsf@gitster.siamese.dyndns.org","threadId":"11440","inReplyTo":"1199139613.2360.83.camel@localhost","subject":"Re: How to structure a project distributed with varyingly interdependent feature branches?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-02T22:12:44Z","receivedAt":"2008-01-02T22:12:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matt McCutchen <matt@mattmccutchen.net> writes:\n\n> Rsync presents a unique challenge because it has a number of long-term\n> feature patches that are maintained and distributed with the main source\n> code.  Furthermore, some of the patches depend on other patches.\n> ...\n> 1. How to properly represent the history of an individual branch and\n> update it when the trunk (or the branch on which it depends) changes.\n> Right now, Wayne updates the branch by rebasing; unfortunately, if the\n> trunk changes in such a way that one of the intermediate commits no\n> longer makes sense, it is impossible to update the branch while\n> preserving a record that the intermediate commit once existed.\n\nI take this to mean a situation like this:\n\n * There is a series of patch X Y Z that implements some nicety\n   not present in the mainline yet.  This set applies to older\n   codebase at point A.\n\n * Newer codebase B does things differently from codebase A and\n   patch X is no longer needed --- IOW, what X achieves on top\n   of A has already been incorporated somewhere between A and B.\n   Applying Y and Z suffices to obtain that nice feature on top\n   of B.\n\n               .---X---Y---Z       .---Y'--Z' \n              /                   /\n      ---o---A---o---.....---o---B\n\nThis is natively expressed with git, if your history were like\nthis:\n\n               .----------X------Y-------Z\n              /            \\\n      ---o---A---o---....---M---....---B\n\nand you publish Z.  People who would want to apply the series to\ntheir codebase simply merge Z.  The patch dependency of X is\ncaptured in the mainline history because the effect of X is\nincorporated to the mainline as a merge at M.  If the point they\nwant to apply the series is older than point M (where patch X\nbecomes unnecessary because of the mainline change), X, Y and Z\nwill be brought in, while people basing on commits newer than M\nwill pull in only Y and Z (because they already have X).\n\nBut that works only in theory and with a developer with perfect\nvision.  Nobody would be able to forsee that X would become such\na crucial change when it is developed.  Often you would just\ncreate a regular commit M somewhere between A and B that has the\neffect of X, and later regret that you had made a separate X\nforked from an earlier A and recorded M as a merge.\n\nPeople use quilt (or guilt) to manage patch stacks for things\nlike this.\n\nHaving said that, I think you can maintain a single patch series\n(say the above X-Y-Z) and publish it as a few variants for\ndownloaders as a prefabricated set of branches.\n\nSay if you have this (the first picture), you do not create Y'-Z'\nhistory (again, M is the point patch X becomes unnecessary, but\nwe live in the real world and it was not recorded as a merge of\nX into the mainline):\n\n                 .---X---Y---Z\n                /\n               /\n              /\n      ---o---A---o---.....---M---...---o---B\n\nand X-Y-Z _is_ what you would maintain and keep tweaking.  You\ncan publish Z and people who would want to \"apply\" the series\ncan cleanly apply X-Y-Z (or merge Z if they are git users).\n\nBut people who would want to \"apply\" the series to codebase\nnewer than point M would individually need to resolve conflicts\nor know X is unneeded for them.  To alleviate this, you would\npublish _another_ commit:\n\n                 .---X---Y---Z\n                /             \\\n               /               Z'\n              /               /\n      ---o---A---o---.....---M---...---o---B\n\nThat is, you merge Z to newer mainline for them to create Z',\nand:\n\n * publish Z' for git users --- they can merge this to their\n   codebase that is newer than M and they do not have to resolve\n   conflicts you needed to resolve, arising from the overlap\n   between X and M, when you created Z'.\n\n * publish diff between M-Z', for non git users who would want\n   to apply it with \"patch\".\n\nYou suggested that you would keep rebasing X--Y--Z while\nimproving the long living patch series.  You would recreate Z'\nwhenever the series is updated (and rerere will hopefully help\nyou re-resolve the conflicts between X and M automatically).\n"},{"id":"65400","messageId":"1200423728.3865.69.camel@localhost","threadId":"11440","inReplyTo":"7vwsqrvpur.fsf@gitster.siamese.dyndns.org","subject":"Re: How to structure a project distributed with varyingly interdependent feature branches?","fromName":"Matt McCutchen","fromEmail":"matt@mattmccutchen.net","sentAt":"2008-01-15T19:02:08Z","receivedAt":"2008-01-15T19:02:08Z","isPatch":false,"sender":{"key":"matt@mattmccutchen.net","avatar":"https://avatars.githubusercontent.com/u/8885753?v=4"},"body":"Junio, thanks for the response.  I am finally getting around to\nfollowing up.\n\nOn Wed, 2008-01-02 at 14:12 -0800, Junio C Hamano wrote: \n> > 1. How to properly represent the history of an individual branch and\n> > update it when the trunk (or the branch on which it depends) changes.\n> > Right now, Wayne updates the branch by rebasing; unfortunately, if the\n> > trunk changes in such a way that one of the intermediate commits no\n> > longer makes sense, it is impossible to update the branch while\n> > preserving a record that the intermediate commit once existed.\n> \n> I take this to mean a situation like this:\n> \n>  * There is a series of patch X Y Z that implements some nicety\n>    not present in the mainline yet.  This set applies to older\n>    codebase at point A.\n> \n>  * Newer codebase B does things differently from codebase A and\n>    patch X is no longer needed --- IOW, what X achieves on top\n>    of A has already been incorporated somewhere between A and B.\n>    Applying Y and Z suffices to obtain that nice feature on top\n>    of B.\n\nActually, I was thinking of a change to the mainline that causes a\nconflict with the patch series.  For example, my repository of git was\nat point A when I made the first draft X of my \"gitweb: snapshot\ncleanups & support for offering multiple formats\" change.  Then I\nupdated my repository and got commit B, \"gitweb.perl - Optionally send\narchives as .zip files\", among others.  When I rebased X on top of the\nnew master C, there was a conflict, which I resolved to produce X':\n\n     X                   X'\n    /                   /\n---A---...---B---...---C\n\nBut now, with refs only to C and X', I have lost the information that\nthe previous incarnation of X' was X.\n\nEssentially, my objection to rebasing is that I want to keep a history\nfor the patch series containing all of the patched versions that I have\nreleased (here X and X'), especially when they differ in interesting\nways (i.e., conflict resolutions), and this history should be a\nfirst-class object that others can pull from me via the git remote\nsystem.\n\nI only want to use a separate patch management tool if it is integrated\nwith git.  I have some familiarity with StGIT, and I assume guilt is\nsimilar.  StGIT uses rebasing and keeps an additional \"patch changelog\"\nviewable by \"stg log\" which might seem to be what I want.  The trouble\nis that this changelog behaves more like a reflog than an orderly\nhistory, and \"stg refresh\" does not support storing a user-entered\nmessage describing *the change to the patch* in the patch changelog.\nOne option I am considering is to use StGIT and track some subset of the\nStGIT area itself (.git/patches) in git.\n\nThe other approach is to maintain the feature patches/branches by\nmerging instead of rebasing.  This has two significant advantages: patch\nhistory is naturally kept and the full power of git's distributed merge\nis available.  However, it also has two significant disadvantages: the\ncomplaint by Linus about \"useless merges\" mentioned in the git-rerere\nmanpage applies, and it's impossible to fully revert a merge (the\nancestry remains and will cause trouble if the merge is redone later).\n\nMatt\n"}]}