[BUG] rebase no longer omits local commits

Starting with git 1.9.0, rebase no longer omits local commits that appear in both the upstream and local branches.

I've bisected this down to commit bb3f458: "rebase: fix fork-point with zero arguments". The attached script reproduces the problem. Reverting the aforementioned commit fixes the problem.

A failed run of this script will result in conflicts. A successful run against master with bb3f458 reverted ends as follows:

 From /tmp/rebase-issue/maint
    fe401cd..955af04  master     -> origin/master
fatal: Not a valid object name: ''
First, rewinding head to replay your work on top of it...
Applying: Third change

(I'm not sure if that "fatal: Not a valid object name: ''" is of any concern. It started appearing for me at some point during the bisect.)

Let me know if there's more I can do to help.

#!/bin/bash

# git-rebase is supposed to drop commits that it finds in both the # local and upstream branches. As of 1.9.0, this isn't happening. # This script reproduces the problem.

# I've bisected the issue down to commit bb3f458. Reverting that commit # solves the problem.

# Run this in a directory where you have create privs. # At the end, if there are conflicts, then the test has failed.

# Create a repo. mkdir rebase-issue cd rebase-issue mkdir maint cd maint git init

# Create a README file and put some text in it echo "Hi there!" >README git add README git commit -a -m "Initial commit"

# Clone the repo for "dev" cd .. git clone maint dev

# Dev makes *two* changes to the *same* area. cd dev # edit something, make some typos echo "Freekwently Mispeled Werdz" >README git commit -a -m "First change" # edit same thing, fix those typos echo "Frequently Misspelled Words" >README git commit -a -m "Second change"

# Create patches to send to maintainer... git format-patch -M origin/master mv *.patch ../maint

# Add a third change that should make it through for completeness. echo "Frequently Misspelled Words version 2" >README git commit -a -m "Third change"

# We have to sleep (to make sure the times do not match?). # If we don't, this script will succeed on fast machines. # This can probably be reduced to 2 which should guarantee that # the seconds will turn over on the clock. echo echo "Waiting 5 seconds to make sure apply time is different from patch time..." sleep 5

echo echo "Maint applies patches..." cd ../maint git am -3 *.patch

echo echo "Dev does the fetch/rebase..." cd ../dev git fetch git rebase

echo git --version

On Thu, Jul 03, 2014 at 11:14:26AM -0400, Ted Felix wrote:

It is the problem that bb3f458 fixes. The change in behaviour is actually introduced by ad8261d (rebase: use reflog to find common base with upstream).

In your example, I think this is working as designed. You can restore the previous behaviour either with `git rebase --no-fork-point` or with `git rebase @{u}`.

The change is designed to help users recover from an upstream rebase, as described in the "DISCUSSION ON FORK-POINT MODE" section of git-merge-base(1) and makes `git rebase` match the behaviour of `git pull --rebase` so that:

	git fetch &&
	git rebase

really is equivalent to:

	git pull --rebase

On Thu, Jul 03, 2014 at 08:09:17PM +0100, John Keeping wrote:

Having thought about this a bit more, I think the case you've identified is an unexpected side effect of that commit.

Perhaps we shuld do something like this (which passes the test suite):

-- >8 --

John Keeping <john@keeping.me.uk> writes:

Probably; but the check to avoid giving worse fork-point should be in the implementation of "merge-base --fork-point" itself, so that we do not have to do the above to both "rebase" and "pull --rebase", no?

On Mon, Jul 07, 2014 at 10:56:23AM -0700, Junio C Hamano wrote:

I don't think so, since in that case we're not actually finding the fork point as defined in the documentation, we're finding the upstream rebase wants.

Having played with this a bit, I think we shouldn't be replacing the upstream with the fork point but should instead add the fork point as an additional negative ref:

	$upstream...$orig_head ^$fork_point

Here's a script that creates a repository showing this:

-- >8 -- #!/bin/sh git init rebase-test && cd rebase-test && echo one >file && git add file && git commit -m one && echo frist >file2 && git add file2 && git commit -m first && git branch --track dev && echo first >file2 && git commit -a --amend --no-edit && echo two >file && git commit -a -m two && echo three >file && git commit -a -m three && echo second >file2 && git commit -a -m second && git checkout dev && git cherry-pick -2 master && echo four >file && git commit -a -m four && printf '\nWithout fork point (old behaviour)\n' && git rev-list --oneline --cherry @{u}... && printf '\nFork point as upstream (current behaviour)\n' && git rev-list --oneline --cherry $(git merge-base --fork-point master HEAD)... && printf '\nWith fork point\n' && git rev-list --oneline --cherry @{u}... ^$(git merge-base --fork-point master HEAD) -- 8< --

In this case the rebase should be clean since the only applicable patch changes "three" to "four" in "file", but the current rebase code fails both with `--fork-point` and with `--no-fork-point`.

With `--fork-point` we try to apply "two" and "three" which have already been cherry-picked across (as Ted originally reported) and with `--no-fork-point`, we try to apply "first" which conflicts because we have the version prior to it being fixed up on master.

I hacked up git-rebase to test this and the change to use the fork point as in the last line of the script above does indeed make the rebase go through cleanly, but I have not yet looked at how to cleanly patch in that behaviour.

I haven't tested git-pull, but it looks like it has always (since 2009) behaved in the way `git rebase --fork-point` does now, failing to detect cherry-picked commits that are now in the upstream.

When using `git format-patch --ignore-if-in-upstream` we are only allowed to give a single revision range. In the next commit we will want to add an additional exclusion revision in order to handle fork points correctly, so convert `git-rebase--am` to use a symmetric difference with `--cherry-pick --right-only`.

This does not change the result of the format-patch invocation, just how we spell the arguments.

Signed-off-by: John Keeping <john@keeping.me.uk>
---
 git-rebase--am.sh | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

When the `--fork-point` argument was added to `git rebase`, we changed the value of $upstream to be the fork point instead of the point from which we want to rebase. When $orig_head..$upstream is empty this does not change the behaviour, but when there are new changes in the upstream we are no longer checking if any of them are patch-identical with changes in $upstream..$orig_head.

Fix this by introducing a new variable to hold the fork point and using this to restrict the range as an extra (negative) revision argument so that the set of desired revisions becomes (in fork-point mode):

	git rev-list --cherry-pick --right-only \
		$upstream...$orig_head ^$fork_point

This allows us to correctly handle the scenario where we have the following topology:

	    C --- D --- E  <- dev
	   /
	  B  <- master@{1}
	 /
	o --- B' --- C* --- D*  <- master

where:
- B' is a fixed-up version of B that is not patch-identical with B;
- C* and D* are patch-identical to C and D respectively and conflict
  textually if applied in the wrong order;
- E depends textually on D.

The correct result of `git rebase master dev` is that B is identified as the fork-point of dev and master, so that C, D, E are the commits that need to be replayed onto master; but C and D are patch-identical with C* and D* and so can be dropped, so that the end result is:

	o --- B' --- C* --- D* --- E  <- dev

If the fork-point is not identified, then picking B onto a branch containing B' results in a conflict and if the patch-identical commits are not correctly identified then picking C onto a branch containing D (or equivalently D*) results in a conflict.

This change allows us to handle both of these cases, where previously we either identified the fork-point (with `--fork-point`) but not the patch-identical commits *or* (with `--no-fork-point`) identified the patch-identical commits but not the fact that master had been rewritten.

Reported-by: Ted Felix <ted@tedfelix.com>
Signed-off-by: John Keeping <john@keeping.me.uk>
---
 git-rebase--am.sh          | 6 ++++--
 git-rebase--interactive.sh | 2 +-
 git-rebase.sh              | 7 ++++---
 3 files changed, 9 insertions(+), 6 deletions(-)

   Thanks, John.  My test script is working fine for me now with these 
patches.

John Keeping <john@keeping.me.uk> writes:

Can we have some tests, too?

The changes look correct from a cursory reading.

Thanks.

When using `git format-patch --ignore-if-in-upstream` we are only allowed to give a single revision range. In the next commit we will want to add an additional exclusion revision in order to handle fork points correctly, so convert `git-rebase--am` to use a symmetric difference with `--cherry-pick --right-only`.

This does not change the result of the format-patch invocation, just how we spell the arguments.

Signed-off-by: John Keeping <john@keeping.me.uk>
---

Unchanged from v1.

 git-rebase--am.sh | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

When the `--fork-point` argument was added to `git rebase`, we changed the value of $upstream to be the fork point instead of the point from which we want to rebase. When $orig_head..$upstream is empty this does not change the behaviour, but when there are new changes in the upstream we are no longer checking if any of them are patch-identical with changes in $upstream..$orig_head.

Fix this by introducing a new variable to hold the fork point and using this to restrict the range as an extra (negative) revision argument so that the set of desired revisions becomes (in fork-point mode):

	git rev-list --cherry-pick --right-only \
		$upstream...$orig_head ^$fork_point

This allows us to correctly handle the scenario where we have the following topology:

	    C --- D --- E  <- dev
	   /
	  B  <- master@{1}
	 /
	o --- B' --- C* --- D*  <- master

where:
- B' is a fixed-up version of B that is not patch-identical with B;
- C* and D* are patch-identical to C and D respectively and conflict
  textually if applied in the wrong order;
- E depends textually on D.

The correct result of `git rebase master dev` is that B is identified as the fork-point of dev and master, so that C, D, E are the commits that need to be replayed onto master; but C and D are patch-identical with C* and D* and so can be dropped, so that the end result is:

	o --- B' --- C* --- D* --- E  <- dev

If the fork-point is not identified, then picking B onto a branch containing B' results in a conflict and if the patch-identical commits are not correctly identified then picking C onto a branch containing D (or equivalently D*) results in a conflict.

This change allows us to handle both of these cases, where previously we either identified the fork-point (with `--fork-point`) but not the patch-identical commits *or* (with `--no-fork-point`) identified the patch-identical commits but not the fact that master had been rewritten.

Reported-by: Ted Felix <ted@tedfelix.com>
Signed-off-by: John Keeping <john@keeping.me.uk>
---

Change from v1:
    - add a test case

 git-rebase--am.sh          |  6 ++++--
 git-rebase--interactive.sh |  2 +-
 git-rebase.sh              |  7 ++++---
 t/t3400-rebase.sh          | 23 +++++++++++++++++++++++
 4 files changed, 32 insertions(+), 6 deletions(-)

John Keeping <john@keeping.me.uk> writes:

mental note: this rebases default (i.e. the current branch) on default-base; it depends on branch.default.{remote,merge} being set by the previous test piece.

Thanks. Do these labels on the commits have any relation to the topology illustrated in the log message?

It looks like the above produces this:

      a'---D---B'--new_B'---final_B (default)
     /
    o----a---B---new_B---C (default-base)
                          \
                           D''---final_B''

where 'a' is "Modify A." from the original set-up and B and new_B are the cherry-picks to be filtered out during the rebase. Am I reading the test correctly?

On Wed, Jul 16, 2014 at 01:26:32PM -0700, Junio C Hamano wrote:

No, I didn't consider the log message when adding the test; it simply uses the next letters that are unused in the test repository.

Yes, that's right. The idea is to make sure that we skip the cherry-picked commits, which requires two commits that will conflict if we try to apply the first on top of the second and to make sure that we don't lose the new commit on the upstream (C).

On 07/16/2014 03:23 PM, John Keeping wrote:

Test case is working fine for me. It passes with the patch and fails without. However, it does seem to cause all the rest of the test cases to fail if it fails. Is there some cleanup missing?

Ted.

On Wed, Jul 16, 2014 at 05:36:03PM -0400, Ted Felix wrote:

The individual test cases in the script don't run in isolation, so I don't think it's surprising the the remainder fail if this one ends up stuck in rebase-in-progress state. I think the same will happen with most of the other test cases in this script.