threads / discuss / 23108

rethinking patch management with GIT / topgit

Subject: rethinking patch management with GIT / topgit

## tl;dr

7 messages between Mar 20, 2010 and Mar 23, 2010.

replies: 6people: 3as markdown or json

Thomas Koch· Mar 20, 2010, 17:02 UTC · lore
Hi,

I'd like to argue, that topgit (tg) fails to fullfill it's task and propose a different approach to the problem of patch management on top of git.

IMHO tg fails due to the following reasons:
- it's to complicate
- AFAIK nobody has solved the problem of managing different patchsets with tg
- it pollutes the patch branches with metadata (.topmsg, .topdeps)
- even after 1 1/2 years topgit isn't feature complete and development seems 
to stagnate
- there also aren't best practices documented
- it quickly fills the list of branches
- it's very easy to break something

Although the above is quite a harsh judgement, I'd like to note, that tg has had its merrit to promote one right idea: Patches should be managed in the form of branches by the means of the underlying VCS and not as simple patchfiles.

The new approach I'd like to propose is based on two core ideas:
1) merge commits to save history

git allows the free creation of merge commits with an arbitrary content tree. So we can create an octupus merge combining all patch-branches while the content of this merge can contain meta data about a patchset instead of the content of the merged commits. Such a merge commit thus provides a pointer to all the history of all patches and can contain all metadata about the merged patch branches. Pushing only a branch or tag with this commit to a central repository thus pushes all the history of all contained patches.

2) collapse / expand branches
Managing a Debian package in stable, unstable and experimental can quickly 
doom you to manage at least three different patchsets with possibly three 
different roots. The list of branches grows in the douzens. Which branches 
belong to which patchset? Which branches are already pushed or pulled?
It may be an advantage to see only some main branches and the branches of one 
patchset I'm currently working on.
The tool I propose would manage each patchset in one branch per patchset. This 
branch has two roles:
- keep the metadata of the patchset as files in the content tree
- keep pointers to the top of the patch-branches in the parent pointers of the 
commit
With the help of such a patchset-metadata-branch I can:
- delete the patch-branches of one patchset while the commits are kept save
- recreate the patch-branches of one patchset

For Debian packaging one last function is needed: export a patchset as quilt series.

Is my explanation understandable? Could this approach work or did I miss something? Who has time to implement it (GSOC?)?

Best regards,
Thomas Koch, http://www.koch.ro
Petr Baudis· Mar 20, 2010, 17:47 UTC · re: Thomas Koch · lore

Re: rethinking patch management with GIT / topgit

  Hi!
On Sat, Mar 20, 2010 at 06:02:51PM +0100, Thomas Koch wrote:
Show 6 quoted lines
> I'd like to argue, that topgit (tg) fails to fullfill it's task and propose a 
> different approach to the problem of patch management on top of git.
> 
> IMHO tg fails due to the following reasons:
> 
> - it pollutes the patch branches with metadata (.topmsg, .topdeps)

I'd like to single this out - this is a very arbitrary decision and has its distinct pros and cons compared to alternative approaches, and was not taken lightly. It wouldn't be that difficult to go in a different way and it's not a fundamental limitation.

> - AFAIK nobody has solved the problem of managing different patchsets with tg

Why not just manage them in separate repositories? If you want to pick patches between patchsets, that should be easily posible through remotes.

Show 11 quoted lines
> - it's to complicate
> - even after 1 1/2 years topgit isn't feature complete and development seems 
> to stagnate
> - there also aren't best practices documented
> - it quickly fills the list of branches
> - it's very easy to break something
> 
> Although the above is quite a harsh judgement, I'd like to note, that tg has 
> had its merrit to promote one right idea: Patches should be managed in the 
> form of branches by the means of the underlying VCS and not as simple 
> patchfiles.
  I agree that topgit really sucks in many areas. :) While it is usable
in practice if you know what you are doing, in some aspects it never got
out of the proof-of-concept stage.
Show 12 quoted lines
> The new approach I'd like to propose is based on two core ideas:
> 
> 1) merge commits to save history
> 
> git allows the free creation of merge commits with an arbitrary content tree. 
> So we can create an octupus merge combining all patch-branches while the 
> content of this merge can contain meta data about a patchset instead of the 
> content of the merged commits.
> Such a merge commit thus provides a pointer to all the history of all patches 
> and can contain all metadata about the merged patch branches. Pushing only a 
> branch or tag with this commit to a central repository thus pushes all the 
> history of all contained patches.
  But how is this different from TopGit's approach? TopGit doesn't use
octopus merges for various reasons (I mainly guess since I was too lazy
to implement that), but that's more of an implementation detail than
anything conceptual.
Show 16 quoted lines
> 2) collapse / expand branches
> 
> Managing a Debian package in stable, unstable and experimental can quickly 
> doom you to manage at least three different patchsets with possibly three 
> different roots. The list of branches grows in the douzens. Which branches 
> belong to which patchset? Which branches are already pushed or pulled?
> It may be an advantage to see only some main branches and the branches of one 
> patchset I'm currently working on.
> The tool I propose would manage each patchset in one branch per patchset. This 
> branch has two roles:
> - keep the metadata of the patchset as files in the content tree
> - keep pointers to the top of the patch-branches in the parent pointers of the 
> commit
> With the help of such a patchset-metadata-branch I can:
> - delete the patch-branches of one patchset while the commits are kept save
> - recreate the patch-branches of one patchset
  The disadvantage is that you will have to invent a lot of arbitrary
metadata and wide range of commands to manipulate and work with this
metadata, all to accomplish something that _is actually one of the main
Git functions to do_!
  Wouldn't it be better to do the collapsing/expanding instead, e.g.
have a convention for patchset/stage branch tying up all patchset/*
branches, and an alias that lists only */stage branches and another that
lists only patchset/* minus patchset/stage branches.
Show 5 quoted lines
> For Debian packaging one last function is needed: export a patchset as quilt 
> series.
> 
> Is my explanation understandable? Could this approach work or did I miss 
> something? Who has time to implement it (GSOC?)?
  It's especially not clear at all whether you propose to start over or
just make two improvements to TopGit, and if the former, how would your
approach differ from TopGit in all the other aspects. It's all a bit
ambiguous.
-- 
				Petr "Pasky" Baudis
http://pasky.or.cz/ | "Ars longa, vita brevis." -- Hippocrates
Thomas Koch· Mar 20, 2010, 18:53 UTC · re: Petr Baudis · lore

Re: rethinking patch management with GIT / topgit

Petr Baudis:
Show 6 quoted lines
> > - it pollutes the patch branches with metadata (.topmsg, .topdeps)
> 
> I'd like to single this out - this is a very arbitrary decision and has
> its distinct pros and cons compared to alternative approaches, and was
> not taken lightly. It wouldn't be that difficult to go in a different
> way and it's not a fundamental limitation.
It's also the most minor problem I see, rather a matter of taste.
Show 5 quoted lines
> > - AFAIK nobody has solved the problem of managing different patchsets
> > with tg
> 
> Why not just manage them in separate repositories? If you want to pick
> patches between patchsets, that should be easily posible through remotes.

You mean having multiple local working directories per project? That's not nice. If you also work on the project with an IDE, it'd most probably mean setting up the same project multiple times.

>   I agree that topgit really sucks in many areas. :) While it is usable
> in practice if you know what you are doing, in some aspects it never got
> out of the proof-of-concept stage.
Thank you for not taking it personaly! :-)
Show 15 quoted lines
> > 1) merge commits to save history
> >
> > git allows the free creation of merge commits with an arbitrary content
> > tree. So we can create an octupus merge combining all patch-branches
> > while the content of this merge can contain meta data about a patchset
> > instead of the content of the merged commits.
> > Such a merge commit thus provides a pointer to all the history of all
> > patches and can contain all metadata about the merged patch branches.
> > Pushing only a branch or tag with this commit to a central repository
> > thus pushes all the history of all contained patches.
> 
>   But how is this different from TopGit's approach? TopGit doesn't use
> octopus merges for various reasons (I mainly guess since I was too lazy
> to implement that), but that's more of an implementation detail than
> anything conceptual.

Maybe I didn't make clear that the commits on the patchset branch should not necessarily contain the merged content of all patches. The role of the merges is solely to keep pointers to the commits making up the patch branches. Now that you're asking there may however be a similarity to topgit in that you can have one topgit controlled branch that combines all topgit branches belonging to the same patchset. However there still remain some differences:

- in tg you need to push all tg branches while in my approach you'd need to 
push only one branch per patchset
- As far as I understand tg, all commands are repository global while for the 
Debian packaging use case you'd need patchset local commands:
  - tg export <patchset>
    export <patchset> as a quilt series
  - tg summary <patchset>
    gives summary information for the branches of <patchset>
  - tg delete <patchset> <branch>
    removes <branch> from <patchset>. However old history of <branch> is still 
recorded in the history of the <patchset>-branch
  - tg add <patchset> <branch>
    adds <branch> to <patchset>
    (The question is open, where to record the merge resolution of all 
patches)
  - tg recreate <patchset> <newbase> <new patchset name>
    Creates a new patchset with root <newbase> by creating new patch branches 
for each patch branch in <patchset>
    This command is useful if you need to keep the old patchset to maintain an 
older version of your Debian package.
    
Show 21 quoted lines
> > 2) collapse / expand branches
> >
> > Managing a Debian package in stable, unstable and experimental can
> > quickly doom you to manage at least three different patchsets with
> > possibly three different roots. The list of branches grows in the
> > douzens. Which branches belong to which patchset? Which branches are
> > already pushed or pulled? It may be an advantage to see only some main
> > branches and the branches of one patchset I'm currently working on.
> > The tool I propose would manage each patchset in one branch per patchset.
> > This branch has two roles:
> > - keep the metadata of the patchset as files in the content tree
> > - keep pointers to the top of the patch-branches in the parent pointers
> > of the commit
> > With the help of such a patchset-metadata-branch I can:
> > - delete the patch-branches of one patchset while the commits are kept
> > save - recreate the patch-branches of one patchset
> 
>   The disadvantage is that you will have to invent a lot of arbitrary
> metadata and wide range of commands to manipulate and work with this
> metadata, all to accomplish something that _is actually one of the main
> Git functions to do_!
I don't see this. What do I miss? All metadata I'd need to manage is:
- one file with the name of each branch, it's last commit and the names of its 
dependencies (the root of the patchset, if empty)
- one message file for each patch
- the root of the patchset
The example commands given above would manipulate or read the patchset branch 
in the background much like pristine-tar does it with its metadata branch.
 
>   Wouldn't it be better to do the collapsing/expanding instead, e.g.
> have a convention for patchset/stage branch tying up all patchset/*
> branches, and an alias that lists only */stage branches and another that
> lists only patchset/* minus patchset/stage branches.

So you propose not to delete/recreate the patch branches but to provide extra commands to list only the desired subset of branches? This would still mean that I'd see douzens of patch branches in gitweb and that I't need to push douzens of branches to my co-packagers. - That doesn't solve it for me. It may also be of interest, that Guido Günther, the author of git- buildpackage[1] wrote a command git-pq to "maintain debian/patches on a patch queue branch". - There seems to be a need to have few branches and that it seems to be practical to create and destroy patch branches in your workflow.

[1] https://honk.sigxcpu.org/piki/projects/git-buildpackage/
Show 7 quoted lines
> > Is my explanation understandable? Could this approach work or did I miss
> > something? Who has time to implement it (GSOC?)?
> 
>   It's especially not clear at all whether you propose to start over or
> just make two improvements to TopGit, and if the former, how would your
> approach differ from TopGit in all the other aspects. It's all a bit
> ambiguous.

I hope I managed to make it clearer this time. I believe my proposals are incompatible to topgit and thus would require a new project from scratch. However it may well be that some code from topgit could be reused.

Best regards,
Thomas Koch, http://www.koch.ro
Petr Baudis· Mar 21, 2010, 20:36 UTC · lore

Re: rethinking patch management with GIT / topgit

On Sat, Mar 20, 2010 at 07:53:34PM +0100, Thomas Koch wrote:
Show 6 quoted lines
> Petr Baudis:
>   - tg recreate <patchset> <newbase> <new patchset name>
>     Creates a new patchset with root <newbase> by creating new patch branches 
> for each patch branch in <patchset>
>     This command is useful if you need to keep the old patchset to maintain an 
> older version of your Debian package.

This means wiping out history again; in TopGit, you would ideally checkpoint all the branches within the patchset, then just tg update your branches. It's another matter that the former is now difficult to do easily.

Show 8 quoted lines
> I don't see this. What do I miss? All metadata I'd need to manage is:
> - one file with the name of each branch, it's last commit and the names of its 
> dependencies (the root of the patchset, if empty)
> - one message file for each patch
> - the root of the patchset
> 
> The example commands given above would manipulate or read the patchset branch 
> in the background much like pristine-tar does it with its metadata branch.

Hmm, to a degree I misunderstood your idea. You would still need quirky commands to update the references when you make a new commit, to go to a certain patch (at which point git will start acting a bit annoyed since it's not on a branch), etc. Other than that, I can offer only my gut feeling. ;-)

Show 8 quoted lines
> >   Wouldn't it be better to do the collapsing/expanding instead, e.g.
> > have a convention for patchset/stage branch tying up all patchset/*
> > branches, and an alias that lists only */stage branches and another that
> > lists only patchset/* minus patchset/stage branches.
> So you propose not to delete/recreate the patch branches but to provide extra 
> commands to list only the desired subset of branches? This would still mean 
> that I'd see douzens of patch branches in gitweb and that I't need to push 
> douzens of branches to my co-packagers. - That doesn't solve it for me.

There are already some patches in the wild to make gitweb topgit-aware; I don't see why is the latter a problem.

> I hope I managed to make it clearer this time. I believe my proposals are 
> incompatible to topgit and thus would require a new project from scratch. 
Yes, I finally understood what do you mean, sorry for being a bit dense.
-- 
				Petr "Pasky" Baudis
http://pasky.or.cz/ | "Ars longa, vita brevis." -- Hippocrates
Thomas Koch· Mar 22, 2010, 07:59 UTC · re: Petr Baudis · lore

Re: rethinking patch management with GIT / topgit

Hi Petr,
Petr Baudis:
Show 12 quoted lines
> On Sat, Mar 20, 2010 at 07:53:34PM +0100, Thomas Koch wrote:
> > Petr Baudis:
> >   - tg recreate <patchset> <newbase> <new patchset name>
> >     Creates a new patchset with root <newbase> by creating new patch
> > branches for each patch branch in <patchset>
> >     This command is useful if you need to keep the old patchset to
> > maintain an older version of your Debian package.
> 
> This means wiping out history again; in TopGit, you would ideally
> checkpoint all the branches within the patchset, then just tg update
> your branches. It's another matter that the former is now difficult to
> do easily.

It's not necessary to lose history. Creating a new patchset from a given patchset could be done in at least three different ways. Given that you've a patchset named patches-x.y.z/* and you'd like to create patches-x.y.z+1/* from this, you could

- checkout new patch branches from the top of the old patch branches and merge 
upstream into each of them
- recreate (like rebase) the full history of the patch branches on top of the 
new upstream
- collapse the branch history and create one commit per patch branch on top of 
the new upstream
In all three cases you're free to either keep or throw away the old patchset.
Best regards,
Thomas Koch, http://www.koch.ro
Petr Baudis· Mar 22, 2010, 19:30 UTC · re: Thomas Koch · lore

Re: rethinking patch management with GIT / topgit

On Mon, Mar 22, 2010 at 08:59:40AM +0100, Thomas Koch wrote:
> In all three cases you're free to either keep or throw away the old patchset.

Yes, but to the same degree e.g. with StGIT I'm free to keep the head of the old patch series. That does not mean the operation *preserves* the history, only that the history is still around somewhere in the repository, however it won't be around in other incarnations of the repository and it will not be connected in any way to the current version of the patchset.

Yes, if you are lucky you can figure out the name of the previous version, but it's like starting development of each new kernel version by a clean import of the sources.

-- 
				Petr "Pasky" Baudis
http://pasky.or.cz/ | "Ars longa, vita brevis." -- Hippocrates
Thomas Koch· Mar 23, 2010, 08:09 UTC · re: Petr Baudis · lore

Re: rethinking patch management with GIT / topgit

Petr Baudis:
Show 14 quoted lines
> On Mon, Mar 22, 2010 at 08:59:40AM +0100, Thomas Koch wrote:
> > In all three cases you're free to either keep or throw away the old
> > patchset.
> 
> Yes, but to the same degree e.g. with StGIT I'm free to keep the head of
> the old patch series. That does not mean the operation *preserves* the
> history, only that the history is still around somewhere in the
> repository, however it won't be around in other incarnations of the
> repository and it will not be connected in any way to the current
> version of the patchset.
> 
> Yes, if you are lucky you can figure out the name of the previous
> version, but it's like starting development of each new kernel version
> by a clean import of the sources.
Hi Petr,
let me copy the list of methods from my last mail:
1) checkout new patch branches from the top of the old patch branches and 
merge upstream into each of them
2) recreate (like rebase) the full history of the patch branches on top of the 
new upstream
3) collapse the branch history and create one commit per patch branch on top 
of the new upstream
>From these methods, 3) loses all history, 2) loses some history but preserves 
the individual history of one patch branch on a new base and 1) preserves all 
history. Let me give an example for method 1). 
You've got a patchset identified by the prefix debian/. Now you want to 
package a new upstream but need to retain the old patchset in case of security 
updates in Debian stable. Debian stable has version 0.1, new upstream is 0.2.
So you
- rename the old patchset from debian/ to debian-0.1/
- clone/copy/recreate (pick a name) a new patchset debian/ on top of 
upstream/0.2. This is done by merging upstream/0.2 into each debian/* branch.
- Once you don't maintain version 0.1 anymore, you can delete the debian-0.1/* 
branches

After these steps, the debian/ branch still contains pointers to all commits from the debian-0.1/* branches. It's an additional question, how to deal with commits that are done in debian-0.1/* after the new upstream merge.

Best regards,
Thomas Koch, http://www.koch.ro

← back to recent threads