threads / discuss / 1899

[Core GIT] Long-term cherrypicking

Subject: [Core GIT] Long-term cherrypicking

## tl;dr

8 messages between Sep 21, 2005 and Sep 22, 2005.

replies: 7people: 4as markdown or json

Petr Baudis· Sep 21, 2005, 16:40 UTC · lore
  Hello,
  I've been working out some workflow possibilities with Karsten Keil at
the SuSE Labs conference, and I'd like to see your opinions. His
position does not seem unique at all, so I'm a bit surprised that noone
solved this before, or at least I didn't find any documentation about
this in the tutorial and howtos (or perhaps I missed it?).
  His situation is that he has some patches in the ISDN subsystem, a
public repository, but sends the patches over e-mail to Linus. So he is
something between the subsystem maintainer and individual developer in
the categories listed out in the tutorial. What should be his merging
strategy?
  Since he has public repository, he cannot rebase, but he needs to
be cherrypicking. The trouble is, git-cherry really does not work
"through" merges now - it stops at his last merge with Linus. The
obvious solution would be something like this patch:
diff --git a/git-cherry.sh b/git-cherry.sh
--- a/git-cherry.sh
+++ b/git-cherry.sh
@@ -5,18 +5,22 @@
 
 . git-sh-setup || die "Not a git archive."
 
-usage="usage: $0 "'[-v] <upstream> [<head>]
+usage="usage: $0 "'[-v] <upstream> [<head>] [<base>]
 
              __*__*__*__*__> <upstream>
-            /
-  fork-point
+            /         \
+      <base>           \ fork-point
             \__+__+__+__+__+__+__+__> <head>
 
 Each commit between the fork-point and <head> is examined, and
-compared against the change each commit between the fork-point and
+compared against the change each commit between the <base> and
 <upstream> introduces.  If the change seems to be in the upstream,
 it is shown on the standard output with prefix "+".  Otherwise
 it is shown with prefix "-".
+
+If no <base> is specified, it is assumed to be the same as fork-point.
+This is the case if you run git-cherry only before you merge the
+upstream.
 '
 
 case "$1" in -v) verbose=t; shift ;; esac 
@@ -35,12 +39,16 @@ case "$#" in
 2) upstream=`git-rev-parse --verify "$1"` &&
    ours=`git-rev-parse --verify "$2"` || exit
    ;;
+3) upstream=`git-rev-parse --verify "$1"` &&
+   ours=`git-rev-parse --verify "$2"` &&
+   base=`git-rev-parse --verify "$3"` || exit
 *) echo >&2 "$usage"; exit 1 ;;
 esac
 
 # Note that these list commits in reverse order;
 # not that the order in inup matters...
-inup=`git-rev-list ^$ours $upstream` &&
+[ -z "$base" ] && base=$ours
+inup=`git-rev-list ^$base $upstream` &&
 ours=`git-rev-list $ours ^$upstream` || exit
 
 tmp=.cherry-tmp$$


  Then you would invoke it like

	git cherry origin master forkpoint

where forkpoint is a tag pointing at the last known point in Linus'
branch when he had all my (well, Karsten's) patches integrated in his
tree. I tested it briefly and it seems to work - I suppose that I will
leave more testing to him. :-) If GIT wants to be even cooler, it could
remember the last forkpoint and automatically advance it.

  Opinions?


PS: Perex hints over my shoulder that the ALSA guys are just manually
recording which patches were merged and which weren't. It's a shame that
GIT has no UI to do that for them if it can. Anyway, I'll be thinking
about if/how to implement the cherrypicking workflow support for Cogito.
-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Junio C Hamano· Sep 21, 2005, 20:57 UTC · re: Petr Baudis · lore

Re: [Core GIT] Long-term cherrypicking

Let me see if I understand what you are trying to solve.

S is subsystem line of development, L is Linus line of development, M is a merge from L into S.

A' and B' are resulting commits from e-mail patch, that contain changes introduced to S by A and B commits, respectively.

 S:     A - B - C - M - D - ...
      /           /      
 L:  O- X - A'- Y - Z - ...

After sending off patches A and B, L accepts only A and creates commit A'. After a while, S pulls from L and merges at M [*1*]. In the meantime, there are other changes on both ends (C and Y). Also note that I am not drawing commits in S that are not to be sent to L via e-mail nor pull request -- there may be other commits on lines between B and C, C and M, etc.

M contains changes B and C which are lacking from L. On top of that, a commit D is made and S wants to ask L to incorporate changes in S but not in L (B, C and D), but finding B, C and D is cumbersome [*2*]. You solved this by 'git cherry' to go beyond M, which is a valid workaround, but then the problem becomes how to find where to stop.

I am assuming the reason this is a cherry-pick problem is because S does not necessarily send everything out, and whenever S does 'format-patch L' to generate a list of patches, only a subset of them is actually sent out. If that is the case, how about maintaining another branch T on the S side, which is private to S?

T starts out as the same commit as the head of L. Whenever S has merge-worthy commits, run 'git-cherry-pick' to copy them over to T, and 'format-patch L' is run in T, not in S, to generate patches to be sent out.

T being a private tree allows you to rebase every once in a while to L [*3*], so that already applied patches are dropped from it.

[Footnote]

*1* I am assuming that merge M would not be made any more difficult by having A and A' on both sides. Is it a problem in practice?

*2* Finding B for resend may be debatable -- it is more of an inter developer communication issue.

*3* For the above workflow to work, I think the current 'git rebase' needs to be improved much in two aspects. (1) The current 'cherry' uses 'git-patch-id' which shares the same strictness with 'git-apply' and would not tolerate much fuzz between what you send out and what got actually applied. (2) 'rebase' keeps going when a merge is not cleanly done, noting them to sort them out by hand at the end. This loses the order of commits in the tree being rebased and very unwieldy. I have a plan to fix this in the TODO list for some time.

Petr Baudis· Sep 21, 2005, 22:19 UTC · re: Junio C Hamano · lore

Re: [Core GIT] Long-term cherrypicking

Dear diary, on Wed, Sep 21, 2005 at 10:57:04PM CEST, I got a letter where Junio C Hamano <junkio@cox.net> told me that...

> You solved this by 'git cherry' to go
> beyond M, which is a valid workaround, but then the problem
> becomes how to find where to stop.

When forking your branch, record that in a tag. When running git-cherry with the tag as base, take the earliest cherrypick found and find the LCA of the two corresponding commits. Reset the tag to that.

Show 6 quoted lines
> I am assuming the reason this is a cherry-pick problem is
> because S does not necessarily send everything out, and whenever
> S does 'format-patch L' to generate a list of patches, only a
> subset of them is actually sent out.  If that is the case, how
> about maintaining another branch T on the S side, which is
> private to S?

That is not the case, this is a cherry-pick problem since L dropped some of the patches (if the git's definition of "cherry-pick problem" isn't something different to what I'd expect)...

Show 8 quoted lines
> T starts out as the same commit as the head of L.  Whenever S
> has merge-worthy commits, run 'git-cherry-pick' to copy them
> over to T, and 'format-patch L' is run in T, not in S, to
> generate patches to be sent out.
> 
> T being a private tree allows you to rebase every once in a
> while to L [*3*], so that already applied patches are dropped
> from it.

...nevertheless your solution still seems to apply and sounds good to me, thanks. I wonder why this didn't occur to me. :-) Perhaps it could be noted in the tutorial?

The disadvantage is that this seems as a lot of work. You have to explicitly cherry-pick each commit, and the fact that you have a floating (continuously rebased) tree means that you can conveniently keep it only on just one machine. [*1*]

The advantage is that it is easy to "forget" commits which were dropped altogether or applied modified. You could put together some filter and grep -v, but that's ugly and doesn't scale.

Well, I guess that if you are going to do this all, it really gets visible that this rebasing stuff is really quite an ugly hack and that you are probably much better off with something like StGIT. ;-)

Show 5 quoted lines
> [Footnote]
> 
> *1* I am assuming that merge M would not be made any more
> difficult by having A and A' on both sides.  Is it a problem in
> practice?

That's something Karsten Keil would be more qualified to reply to, but unfortunately you removed him from the Cc list. ;-)

*1* I find the fast-forward restriction more and more annoying in this context, actually, and I'm seriously considering dropping it from Cogito altogether. GIT seems to care a lot about it, but Cogito doesn't so far, it would only have problems fast-forwarding your master to catch up with the branch you are merging, trying to do a tree merge instead. But

	if old_ref == master then master = new_ref
would solve that. Can it ever really hurt?
-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Junio C Hamano· Sep 21, 2005, 23:26 UTC · re: Petr Baudis · lore

Re: [Core GIT] Long-term cherrypicking

Petr Baudis <pasky@suse.cz> writes:
> Well, I guess that if you are going to do this all, it really gets
> visible that this rebasing stuff is really quite an ugly hack and that
> you are probably much better off with something like StGIT. ;-)
Yes.
> *1* I find the fast-forward restriction more and more annoying in this
> context, actually, and I'm seriously considering dropping it from Cogito
> altogether.

My "pu" is always floating and I have this in my .git/remotes/ko:

    URL: master.kernel.org:/pub/scm/git/git.git/
    Push: master maint rc +pu
Greg KH· Sep 22, 2005, 08:31 UTC · re: Petr Baudis · lore

Re: [Core GIT] Long-term cherrypicking

On Wed, Sep 21, 2005 at 06:40:15PM +0200, Petr Baudis wrote:
Show 13 quoted lines
>   Hello,
> 
>   I've been working out some workflow possibilities with Karsten Keil at
> the SuSE Labs conference, and I'd like to see your opinions. His
> position does not seem unique at all, so I'm a bit surprised that noone
> solved this before, or at least I didn't find any documentation about
> this in the tutorial and howtos (or perhaps I missed it?).
> 
>   His situation is that he has some patches in the ISDN subsystem, a
> public repository, but sends the patches over e-mail to Linus. So he is
> something between the subsystem maintainer and individual developer in
> the categories listed out in the tutorial. What should be his merging
> strategy?
Not to use git for this.

Seriously, that's what I have switched to doing, and it's so much easier. I use quilt to manage patches from the community, and Andrew pulls them into to the -mm releases, and all users can test them. I keep them up to date with the git snapshots, which handles the different merge and fuzz issues very well.

Then, when it's time to merge with Linus, I pick and choose the patches that I want to send off, create a git tree, add them to the tree, and send them off.

It's ended up saving me a lot of time (I used to do what Karsten is trying to do with bitkeeper, and it had the same issues that he is running into) and would recommend this situation for anyone who wants to keep patches from being merged immediatly (like he is trying to do.)

thanks,
greg k-h
Catalin Marinas· Sep 22, 2005, 09:53 UTC · re: Greg KH · lore

Re: [Core GIT] Long-term cherrypicking

Greg KH <greg@kroah.com> wrote:
Show 14 quoted lines
> On Wed, Sep 21, 2005 at 06:40:15PM +0200, Petr Baudis wrote:
>>   His situation is that he has some patches in the ISDN subsystem, a
>> public repository, but sends the patches over e-mail to Linus. So he is
>> something between the subsystem maintainer and individual developer in
>> the categories listed out in the tutorial. What should be his merging
>> strategy?
>
> Not to use git for this.
>
> Seriously, that's what I have switched to doing, and it's so much
> easier.  I use quilt to manage patches from the community, and Andrew
> pulls them into to the -mm releases, and all users can test them.  I
> keep them up to date with the git snapshots, which handles the different
> merge and fuzz issues very well.

You might want to try StGIT as well, unless you have a huge number of patches and rebasing them takes too much time. It follows the quilt ideas but it does a three-way merge instead of patch+fuzz. This allows it to detect patches accepted by Linus and also see whether they were not fully applied or were modified. You can run 'stg export' to generate a patch series to send to Andrew.

> Then, when it's time to merge with Linus, I pick and choose the patches
> that I want to send off, create a git tree, add them to the tree, and
> send them off.

With StGIT, you can either use 'stg mail <patches...>' or re-organise the stack with push/pop so that you only keep the patches to be merged by Linus and tell him the HEAD value to merge. Even if you modify the stack afterwards, the HEAD value still represents the stack state at that moment (unless you run 'git prune' and the value is not saved in a file under refs/heads/).

StGIT also support cherry-picking via 'stg import --commit=...' if you want to create a separate branch.

But what I understand from Petr's e-mail is that re-basing the patches is not acceptable for a repository which is public. One option would be to keep a private tree where the patches are re-based and a public one which periodically pulls from Linus' tree and your private one. Not sure how complicated the commit graph would look.

> It's ended up saving me a lot of time (I used to do what Karsten is
> trying to do with bitkeeper, and it had the same issues that he is
> running into) and would recommend this situation for anyone who wants to
> keep patches from being merged immediatly (like he is trying to do.)

I only maintain a small set of patches (< 20) and send some of them upstream via RMK. I would recommend StGIT :-) as an alternative to quilt (I don't say a better one but some people might like it).

-- 
Catalin
Petr Baudis· Sep 22, 2005, 09:58 UTC · re: Catalin Marinas · lore

Re: [Core GIT] Long-term cherrypicking

Dear diary, on Thu, Sep 22, 2005 at 11:53:27AM CEST, I got a letter where Catalin Marinas <catalin.marinas@gmail.com> told me that...

Show 5 quoted lines
> But what I understand from Petr's e-mail is that re-basing the patches
> is not acceptable for a repository which is public. One option would
> be to keep a private tree where the patches are re-based and a public
> one which periodically pulls from Linus' tree and your private
> one. Not sure how complicated the commit graph would look.

It would be pretty ugly since the same patches would re-appear in the commit graph in another incarnations many times.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Catalin Marinas· Sep 22, 2005, 10:17 UTC · re: Petr Baudis · lore

Re: [Core GIT] Long-term cherrypicking

Petr Baudis <pasky@suse.cz> wrote:
Show 10 quoted lines
> Dear diary, on Thu, Sep 22, 2005 at 11:53:27AM CEST, I got a letter
> where Catalin Marinas <catalin.marinas@gmail.com> told me that...
>> But what I understand from Petr's e-mail is that re-basing the patches
>> is not acceptable for a repository which is public. One option would
>> be to keep a private tree where the patches are re-based and a public
>> one which periodically pulls from Linus' tree and your private
>> one. Not sure how complicated the commit graph would look.
>
> It would be pretty ugly since the same patches would re-appear in the
> commit graph in another incarnations many times.

True. And re-basing patches in the public branch would generate the same complicated graph in people's repositories which were pulling from it. If this is not desirable, StGIT on the public branch wouldn't help much (and neither quilt since quilt patches are not visible via GIT anyway).

As Junio said, using 'git cherry-pick' (or 'stg import --commit') on a private branch would help organising the patches to be sent upstream. One problem with this approach is that the conflicts generated by merging with Linus in the public branch (and already fixed there) would need to be re-solved in the private one after cherry-picking.

-- 
Catalin

← back to recent threads