{"thread":{"id":"23108","subject":"rethinking patch management with GIT / topgit","startedAt":"2010-03-20T17:02:51Z","lastAt":"2010-03-23T08:09:16Z","messageCount":7,"participants":["Thomas Koch","Petr Baudis"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"137349","messageId":"201003201802.51129.thomas@koch.ro","threadId":"23108","inReplyTo":null,"subject":"rethinking patch management with GIT / topgit","fromName":"Thomas Koch","fromEmail":"thomas@koch.ro","sentAt":"2010-03-20T17:02:51Z","receivedAt":"2010-03-20T17:02:51Z","isPatch":false,"sender":{"key":"thomas@koch.ro","avatar":null},"body":"Hi,\n\nI'd like to argue, that topgit (tg) fails to fullfill it's task and propose a \ndifferent approach to the problem of patch management on top of git.\n\nIMHO tg fails due to the following reasons:\n\n- it's to complicate\n- AFAIK nobody has solved the problem of managing different patchsets with tg\n- it pollutes the patch branches with metadata (.topmsg, .topdeps)\n- even after 1 1/2 years topgit isn't feature complete and development seems \nto stagnate\n- there also aren't best practices documented\n- it quickly fills the list of branches\n- it's very easy to break something\n\nAlthough the above is quite a harsh judgement, I'd like to note, that tg has \nhad its merrit to promote one right idea: Patches should be managed in the \nform of branches by the means of the underlying VCS and not as simple \npatchfiles.\n\nThe new approach I'd like to propose is based on two core ideas:\n\n1) merge commits to save history\n\ngit allows the free creation of merge commits with an arbitrary content tree. \nSo we can create an octupus merge combining all patch-branches while the \ncontent of this merge can contain meta data about a patchset instead of the \ncontent of the merged commits.\nSuch a merge commit thus provides a pointer to all the history of all patches \nand can contain all metadata about the merged patch branches. Pushing only a \nbranch or tag with this commit to a central repository thus pushes all the \nhistory of all contained patches.\n\n2) collapse / expand branches\n\nManaging a Debian package in stable, unstable and experimental can quickly \ndoom you to manage at least three different patchsets with possibly three \ndifferent roots. The list of branches grows in the douzens. Which branches \nbelong to which patchset? Which branches are already pushed or pulled?\nIt may be an advantage to see only some main branches and the branches of one \npatchset I'm currently working on.\nThe tool I propose would manage each patchset in one branch per patchset. This \nbranch has two roles:\n- keep the metadata of the patchset as files in the content tree\n- keep pointers to the top of the patch-branches in the parent pointers of the \ncommit\nWith the help of such a patchset-metadata-branch I can:\n- delete the patch-branches of one patchset while the commits are kept save\n- recreate the patch-branches of one patchset\n\nFor Debian packaging one last function is needed: export a patchset as quilt \nseries.\n\nIs my explanation understandable? Could this approach work or did I miss \nsomething? Who has time to implement it (GSOC?)?\n\nBest regards,\n\nThomas Koch, http://www.koch.ro\n"},{"id":"137352","messageId":"20100320174740.GC4535@machine.or.cz","threadId":"23108","inReplyTo":"201003201802.51129.thomas@koch.ro","subject":"Re: rethinking patch management with GIT / topgit","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2010-03-20T17:47:41Z","receivedAt":"2010-03-20T17:47:41Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"  Hi!\n\nOn Sat, Mar 20, 2010 at 06:02:51PM +0100, Thomas Koch wrote:\n> I'd like to argue, that topgit (tg) fails to fullfill it's task and propose a \n> different approach to the problem of patch management on top of git.\n> \n> IMHO tg fails due to the following reasons:\n> \n> - it pollutes the patch branches with metadata (.topmsg, .topdeps)\n\nI'd like to single this out - this is a very arbitrary decision and has\nits distinct pros and cons compared to alternative approaches, and was\nnot taken lightly. It wouldn't be that difficult to go in a different\nway and it's not a fundamental limitation.\n\n> - AFAIK nobody has solved the problem of managing different patchsets with tg\n\nWhy not just manage them in separate repositories? If you want to pick\npatches between patchsets, that should be easily posible through remotes.\n\n> - it's to complicate\n> - even after 1 1/2 years topgit isn't feature complete and development seems \n> to stagnate\n> - there also aren't best practices documented\n> - it quickly fills the list of branches\n> - it's very easy to break something\n> \n> Although the above is quite a harsh judgement, I'd like to note, that tg has \n> had its merrit to promote one right idea: Patches should be managed in the \n> form of branches by the means of the underlying VCS and not as simple \n> patchfiles.\n\n  I agree that topgit really sucks in many areas. :) While it is usable\nin practice if you know what you are doing, in some aspects it never got\nout of the proof-of-concept stage.\n\n> The new approach I'd like to propose is based on two core ideas:\n> \n> 1) merge commits to save history\n> \n> git allows the free creation of merge commits with an arbitrary content tree. \n> So we can create an octupus merge combining all patch-branches while the \n> content of this merge can contain meta data about a patchset instead of the \n> content of the merged commits.\n> Such a merge commit thus provides a pointer to all the history of all patches \n> and can contain all metadata about the merged patch branches. Pushing only a \n> branch or tag with this commit to a central repository thus pushes all the \n> history of all contained patches.\n\n  But how is this different from TopGit's approach? TopGit doesn't use\noctopus merges for various reasons (I mainly guess since I was too lazy\nto implement that), but that's more of an implementation detail than\nanything conceptual.\n\n> 2) collapse / expand branches\n> \n> Managing a Debian package in stable, unstable and experimental can quickly \n> doom you to manage at least three different patchsets with possibly three \n> different roots. The list of branches grows in the douzens. Which branches \n> belong to which patchset? Which branches are already pushed or pulled?\n> It may be an advantage to see only some main branches and the branches of one \n> patchset I'm currently working on.\n> The tool I propose would manage each patchset in one branch per patchset. This \n> branch has two roles:\n> - keep the metadata of the patchset as files in the content tree\n> - keep pointers to the top of the patch-branches in the parent pointers of the \n> commit\n> With the help of such a patchset-metadata-branch I can:\n> - delete the patch-branches of one patchset while the commits are kept save\n> - recreate the patch-branches of one patchset\n\n  The disadvantage is that you will have to invent a lot of arbitrary\nmetadata and wide range of commands to manipulate and work with this\nmetadata, all to accomplish something that _is actually one of the main\nGit functions to do_!\n\n  Wouldn't it be better to do the collapsing/expanding instead, e.g.\nhave a convention for patchset/stage branch tying up all patchset/*\nbranches, and an alias that lists only */stage branches and another that\nlists only patchset/* minus patchset/stage branches.\n\n> For Debian packaging one last function is needed: export a patchset as quilt \n> series.\n> \n> Is my explanation understandable? Could this approach work or did I miss \n> something? Who has time to implement it (GSOC?)?\n\n  It's especially not clear at all whether you propose to start over or\njust make two improvements to TopGit, and if the former, how would your\napproach differ from TopGit in all the other aspects. It's all a bit\nambiguous.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nhttp://pasky.or.cz/ | \"Ars longa, vita brevis.\" -- Hippocrates\n"},{"id":"137357","messageId":"201003201953.34666.thomas@koch.ro","threadId":"23108","inReplyTo":"20100320174740.GC4535@machine.or.cz","subject":"Re: rethinking patch management with GIT / topgit","fromName":"Thomas Koch","fromEmail":"thomas@koch.ro","sentAt":"2010-03-20T18:53:34Z","receivedAt":"2010-03-20T18:53:34Z","isPatch":false,"sender":{"key":"thomas@koch.ro","avatar":null},"body":"Petr Baudis:\n> > - it pollutes the patch branches with metadata (.topmsg, .topdeps)\n> \n> I'd like to single this out - this is a very arbitrary decision and has\n> its distinct pros and cons compared to alternative approaches, and was\n> not taken lightly. It wouldn't be that difficult to go in a different\n> way and it's not a fundamental limitation.\nIt's also the most minor problem I see, rather a matter of taste.\n\n> > - AFAIK nobody has solved the problem of managing different patchsets\n> > with tg\n> \n> Why not just manage them in separate repositories? If you want to pick\n> patches between patchsets, that should be easily posible through remotes.\nYou mean having multiple local working directories per project? That's not \nnice. If you also work on the project with an IDE, it'd most probably mean \nsetting up the same project multiple times.\n\n>   I agree that topgit really sucks in many areas. :) While it is usable\n> in practice if you know what you are doing, in some aspects it never got\n> out of the proof-of-concept stage.\nThank you for not taking it personaly! :-)\n\n> > 1) merge commits to save history\n> >\n> > git allows the free creation of merge commits with an arbitrary content\n> > tree. So we can create an octupus merge combining all patch-branches\n> > while the content of this merge can contain meta data about a patchset\n> > instead of the content of the merged commits.\n> > Such a merge commit thus provides a pointer to all the history of all\n> > patches and can contain all metadata about the merged patch branches.\n> > Pushing only a branch or tag with this commit to a central repository\n> > thus pushes all the history of all contained patches.\n> \n>   But how is this different from TopGit's approach? TopGit doesn't use\n> octopus merges for various reasons (I mainly guess since I was too lazy\n> to implement that), but that's more of an implementation detail than\n> anything conceptual.\nMaybe I didn't make clear that the commits on the patchset branch should not \nnecessarily contain the merged content of all patches. The role of the merges \nis solely to keep pointers to the commits making up the patch branches.\nNow that you're asking there may however be a similarity to topgit in that you \ncan have one topgit controlled branch that combines all topgit branches \nbelonging to the same patchset.\nHowever there still remain some differences:\n\n- in tg you need to push all tg branches while in my approach you'd need to \npush only one branch per patchset\n- As far as I understand tg, all commands are repository global while for the \nDebian packaging use case you'd need patchset local commands:\n  - tg export <patchset>\n    export <patchset> as a quilt series\n\n  - tg summary <patchset>\n    gives summary information for the branches of <patchset>\n\n  - tg delete <patchset> <branch>\n    removes <branch> from <patchset>. However old history of <branch> is still \nrecorded in the history of the <patchset>-branch\n\n  - tg add <patchset> <branch>\n    adds <branch> to <patchset>\n    (The question is open, where to record the merge resolution of all \npatches)\n\n  - tg recreate <patchset> <newbase> <new patchset name>\n    Creates a new patchset with root <newbase> by creating new patch branches \nfor each patch branch in <patchset>\n    This command is useful if you need to keep the old patchset to maintain an \nolder version of your Debian package.\n    \n> > 2) collapse / expand branches\n> >\n> > Managing a Debian package in stable, unstable and experimental can\n> > quickly doom you to manage at least three different patchsets with\n> > possibly three different roots. The list of branches grows in the\n> > douzens. Which branches belong to which patchset? Which branches are\n> > already pushed or pulled? It may be an advantage to see only some main\n> > branches and the branches of one patchset I'm currently working on.\n> > The tool I propose would manage each patchset in one branch per patchset.\n> > This branch has two roles:\n> > - keep the metadata of the patchset as files in the content tree\n> > - keep pointers to the top of the patch-branches in the parent pointers\n> > of the commit\n> > With the help of such a patchset-metadata-branch I can:\n> > - delete the patch-branches of one patchset while the commits are kept\n> > save - recreate the patch-branches of one patchset\n> \n>   The disadvantage is that you will have to invent a lot of arbitrary\n> metadata and wide range of commands to manipulate and work with this\n> metadata, all to accomplish something that _is actually one of the main\n> Git functions to do_!\nI don't see this. What do I miss? All metadata I'd need to manage is:\n- one file with the name of each branch, it's last commit and the names of its \ndependencies (the root of the patchset, if empty)\n- one message file for each patch\n- the root of the patchset\n\nThe example commands given above would manipulate or read the patchset branch \nin the background much like pristine-tar does it with its metadata branch.\n \n>   Wouldn't it be better to do the collapsing/expanding instead, e.g.\n> have a convention for patchset/stage branch tying up all patchset/*\n> branches, and an alias that lists only */stage branches and another that\n> lists only patchset/* minus patchset/stage branches.\nSo you propose not to delete/recreate the patch branches but to provide extra \ncommands to list only the desired subset of branches? This would still mean \nthat I'd see douzens of patch branches in gitweb and that I't need to push \ndouzens of branches to my co-packagers. - That doesn't solve it for me.\nIt may also be of interest, that Guido Günther, the author of git-\nbuildpackage[1] wrote a command git-pq to \"maintain debian/patches on a patch \nqueue branch\". - There seems to be a need to have few branches and that it \nseems to be practical to create and destroy patch branches in your workflow.\n\n[1] https://honk.sigxcpu.org/piki/projects/git-buildpackage/\n\n> > Is my explanation understandable? Could this approach work or did I miss\n> > something? Who has time to implement it (GSOC?)?\n> \n>   It's especially not clear at all whether you propose to start over or\n> just make two improvements to TopGit, and if the former, how would your\n> approach differ from TopGit in all the other aspects. It's all a bit\n> ambiguous.\nI hope I managed to make it clearer this time. I believe my proposals are \nincompatible to topgit and thus would require a new project from scratch. \nHowever it may well be that some code from topgit could be reused.\n\nBest regards,\n\nThomas Koch, http://www.koch.ro\n"},{"id":"137478","messageId":"20100321203626.GE3533@machine.or.cz","threadId":"23108","inReplyTo":"201003201953.34666.thomas-5j3myg3OO4w@public.gmane.org","subject":"Re: rethinking patch management with GIT / topgit","fromName":"Petr Baudis","fromEmail":"pasky-alswssmvlrq@public.gmane.org","sentAt":"2010-03-21T20:36:26Z","receivedAt":"2010-03-21T20:36:26Z","isPatch":false,"sender":{"key":"pasky-alswssmvlrq@public.gmane.org","avatar":null},"body":"On Sat, Mar 20, 2010 at 07:53:34PM +0100, Thomas Koch wrote:\n> Petr Baudis:\n>   - tg recreate <patchset> <newbase> <new patchset name>\n>     Creates a new patchset with root <newbase> by creating new patch branches \n> for each patch branch in <patchset>\n>     This command is useful if you need to keep the old patchset to maintain an \n> older version of your Debian package.\n\nThis means wiping out history again; in TopGit, you would ideally\ncheckpoint all the branches within the patchset, then just tg update\nyour branches. It's another matter that the former is now difficult to\ndo easily.\n\n> I don't see this. What do I miss? All metadata I'd need to manage is:\n> - one file with the name of each branch, it's last commit and the names of its \n> dependencies (the root of the patchset, if empty)\n> - one message file for each patch\n> - the root of the patchset\n> \n> The example commands given above would manipulate or read the patchset branch \n> in the background much like pristine-tar does it with its metadata branch.\n\nHmm, to a degree I misunderstood your idea. You would still need quirky\ncommands to update the references when you make a new commit, to go to a\ncertain patch (at which point git will start acting a bit annoyed since\nit's not on a branch), etc. Other than that, I can offer only my gut\nfeeling.  ;-)\n\n> >   Wouldn't it be better to do the collapsing/expanding instead, e.g.\n> > have a convention for patchset/stage branch tying up all patchset/*\n> > branches, and an alias that lists only */stage branches and another that\n> > lists only patchset/* minus patchset/stage branches.\n> So you propose not to delete/recreate the patch branches but to provide extra \n> commands to list only the desired subset of branches? This would still mean \n> that I'd see douzens of patch branches in gitweb and that I't need to push \n> douzens of branches to my co-packagers. - That doesn't solve it for me.\n\nThere are already some patches in the wild to make gitweb topgit-aware;\nI don't see why is the latter a problem.\n\n> I hope I managed to make it clearer this time. I believe my proposals are \n> incompatible to topgit and thus would require a new project from scratch. \n\nYes, I finally understood what do you mean, sorry for being a bit dense.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nhttp://pasky.or.cz/ | \"Ars longa, vita brevis.\" -- Hippocrates\n"},{"id":"137527","messageId":"201003220859.41820.thomas@koch.ro","threadId":"23108","inReplyTo":"20100321203626.GE3533@machine.or.cz","subject":"Re: rethinking patch management with GIT / topgit","fromName":"Thomas Koch","fromEmail":"thomas@koch.ro","sentAt":"2010-03-22T07:59:40Z","receivedAt":"2010-03-22T07:59:40Z","isPatch":false,"sender":{"key":"thomas@koch.ro","avatar":null},"body":"Hi Petr,\n\nPetr Baudis:\n> On Sat, Mar 20, 2010 at 07:53:34PM +0100, Thomas Koch wrote:\n> > Petr Baudis:\n> >   - tg recreate <patchset> <newbase> <new patchset name>\n> >     Creates a new patchset with root <newbase> by creating new patch\n> > branches for each patch branch in <patchset>\n> >     This command is useful if you need to keep the old patchset to\n> > maintain an older version of your Debian package.\n> \n> This means wiping out history again; in TopGit, you would ideally\n> checkpoint all the branches within the patchset, then just tg update\n> your branches. It's another matter that the former is now difficult to\n> do easily.\nIt's not necessary to lose history. Creating a new patchset from a given \npatchset could be done in at least three different ways. Given that you've a \npatchset named patches-x.y.z/* and you'd like to create patches-x.y.z+1/* from \nthis, you could\n\n- checkout new patch branches from the top of the old patch branches and merge \nupstream into each of them\n\n- recreate (like rebase) the full history of the patch branches on top of the \nnew upstream\n\n- collapse the branch history and create one commit per patch branch on top of \nthe new upstream\n\nIn all three cases you're free to either keep or throw away the old patchset.\n\nBest regards,\n\nThomas Koch, http://www.koch.ro\n"},{"id":"137580","messageId":"20100322193035.GM3533@machine.or.cz","threadId":"23108","inReplyTo":"201003220859.41820.thomas@koch.ro","subject":"Re: rethinking patch management with GIT / topgit","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2010-03-22T19:30:35Z","receivedAt":"2010-03-22T19:30:35Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Mon, Mar 22, 2010 at 08:59:40AM +0100, Thomas Koch wrote:\n> In all three cases you're free to either keep or throw away the old patchset.\n\nYes, but to the same degree e.g. with StGIT I'm free to keep the head of\nthe old patch series. That does not mean the operation *preserves* the\nhistory, only that the history is still around somewhere in the\nrepository, however it won't be around in other incarnations of the\nrepository and it will not be connected in any way to the current\nversion of the patchset.\n\nYes, if you are lucky you can figure out the name of the previous\nversion, but it's like starting development of each new kernel version\nby a clean import of the sources.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nhttp://pasky.or.cz/ | \"Ars longa, vita brevis.\" -- Hippocrates\n"},{"id":"137614","messageId":"201003230909.16414.thomas@koch.ro","threadId":"23108","inReplyTo":"20100322193035.GM3533@machine.or.cz","subject":"Re: rethinking patch management with GIT / topgit","fromName":"Thomas Koch","fromEmail":"thomas@koch.ro","sentAt":"2010-03-23T08:09:16Z","receivedAt":"2010-03-23T08:09:16Z","isPatch":false,"sender":{"key":"thomas@koch.ro","avatar":null},"body":"Petr Baudis:\n> On Mon, Mar 22, 2010 at 08:59:40AM +0100, Thomas Koch wrote:\n> > In all three cases you're free to either keep or throw away the old\n> > patchset.\n> \n> Yes, but to the same degree e.g. with StGIT I'm free to keep the head of\n> the old patch series. That does not mean the operation *preserves* the\n> history, only that the history is still around somewhere in the\n> repository, however it won't be around in other incarnations of the\n> repository and it will not be connected in any way to the current\n> version of the patchset.\n> \n> Yes, if you are lucky you can figure out the name of the previous\n> version, but it's like starting development of each new kernel version\n> by a clean import of the sources.\n\nHi Petr,\n\nlet me copy the list of methods from my last mail:\n\n1) checkout new patch branches from the top of the old patch branches and \nmerge upstream into each of them\n\n2) recreate (like rebase) the full history of the patch branches on top of the \nnew upstream\n\n3) collapse the branch history and create one commit per patch branch on top \nof the new upstream\n\n>From these methods, 3) loses all history, 2) loses some history but preserves \nthe individual history of one patch branch on a new base and 1) preserves all \nhistory. Let me give an example for method 1). \nYou've got a patchset identified by the prefix debian/. Now you want to \npackage a new upstream but need to retain the old patchset in case of security \nupdates in Debian stable. Debian stable has version 0.1, new upstream is 0.2.\nSo you\n- rename the old patchset from debian/ to debian-0.1/\n- clone/copy/recreate (pick a name) a new patchset debian/ on top of \nupstream/0.2. This is done by merging upstream/0.2 into each debian/* branch.\n- Once you don't maintain version 0.1 anymore, you can delete the debian-0.1/* \nbranches\n\nAfter these steps, the debian/ branch still contains pointers to all commits \nfrom the debian-0.1/* branches.\nIt's an additional question, how to deal with commits that are done in \ndebian-0.1/* after the new upstream merge.\n\nBest regards,\n\nThomas Koch, http://www.koch.ro\n"}]}