[BUG] git-cherry false-positive on upstream new-line change at end-of-file

>
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
>
> What did you do before the bug happened? (Steps to reproduce your issue)
>
> I was working on two branches "master" and "feature". On "master" I
> "git cherry-pick feature" in order to pick the latest commit from
> "feature".
>
> What did you expect to happen? (Expected behavior)
>
> When I run "git cherry master feature" the picked commit is
> marked with "-".
>
> What happened instead? (Actual behavior)
>
> When I run "git cherry master feature" the picked commit is
> marked with "+".
>
> What's different between what you expected and what actually happened?
>
> Because the commit has just been picked, it should be marked as "-".
> The actual behavior claims the commit had not been picked.
>
> Anything else you want to add:
>
> It is important to note that the only change, I did on master after
> branching "feature", was to add a missing newline at the end of the
> file that I changed on "feature".
>
> If I replay the same scenario with a file that has a newline at the end
> and I am only adding a newline at the beginning of it on master, cherry
> gives the expected "-" after cherry-picking.
>
> Please see this simple example:
>
> > git clone https://github.com/tvogel/cherry-broken.git
> > cd cherry-broken
>
> # see broken behaviour:
> > git cherry origin/master origin/feature
> + 26f77def8f1380cc3fd08ebc3f70652da79f1a2e
>
> # check that patch-id is actually the same:
> > git show --patch origin/master | git patch-id
> ca5d666a572ec32bff9cec17b97d611316f89236
> 0725b090dd82f2e6f9949ce853778391e60b08d1
> > git show --patch origin/feature | git patch-id
> ca5d666a572ec32bff9cec17b97d611316f89236
> 26f77def8f1380cc3fd08ebc3f70652da79f1a2e
>
> # see OK behaviour:
> > git cherry origin/master-ok origin/feature-ok
> - 7ce2c0ecd3024c04bcf7b78d332841b2e791e06f
>
> In the "*-ok" branches, the file has newline at the end from the beginning
> and "master" only inserts a newline at the beginning of the file. "feature-ok"
> has the same change as "feature" (content-wise).
>
>
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
>
>
> [System Info]
> git version:
> git version 2.27.0
> cpu: x86_64
> no commit associated with this build
> sizeof-long: 8
> sizeof-size_t: 8
> uname: Linux 5.7.5-1-default #1 SMP Tue Jun 23 06:00:46 UTC 2020 (a1775d0)
> x86_64
> compiler info: gnuc: 10.1
> libc info: glibc: 2.31
>
>
> [Enabled Hooks]
> not run from a git repository - no hooks to show
>
>

> Whitespace is ignored when calculating patch IDs.  This is done by
> removing all whitespace from diff lines before hashing them, including
> a newline at the end of a file.  If that newline is missing, however,
> diff reports that fact in a separate line containing "\ No newline at
> end of file\n", and this marker is hashed like a context line.

> This goes against our goal of making patch IDs independent of
> whitespace.  Use the same heuristic that 2485eab55cc (git-patch-id: do
> not trip over "no newline" markers, 2011-02-17) added to git patch-id
> instead and skip diff lines that start with a backslash and a space
> and are longer than twelve characters.

> Reported-by: Tilman Vogel <tilman.vogel@web.de>
> Initial-test-by: Tilman Vogel <tilman.vogel@web.de>
> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>  diff.c            |  2 ++
>  t/t3500-cherry.sh | 23 +++++++++++++++++++++++
>  2 files changed, 25 insertions(+)

> diff --git a/diff.c b/diff.c
> index f9709de7b45..f175019eb7a 100644
> --- a/diff.c
> +++ b/diff.c
> @@ -6044,6 +6044,8 @@ static void patch_id_consume(void *priv, char *line, unsigned long len)
>  	struct patch_id_t *data = priv;
>  	int new_len;
>
> +	if (len > 12 && starts_with(line, "\\ "))
> +		return;
>  	new_len = remove_space(line, len);
>
>  	the_hash_algo->update_fn(data->ctx, line, new_len);
> diff --git a/t/t3500-cherry.sh b/t/t3500-cherry.sh
> index f038f34b7c0..2b8d9cb38ed 100755
> --- a/t/t3500-cherry.sh
> +++ b/t/t3500-cherry.sh
> @@ -55,4 +55,27 @@ test_expect_success \
>       expr "$(echo $(git cherry master my-topic-branch) )" : "+ [^ ]* - .*"
>  '
>
> +test_expect_success 'cherry ignores whitespace' '
> +	git switch --orphan=upstream-with-space &&
> +	test_commit initial file &&
> +	>expect &&
> +	git switch --create=feature-without-space &&
> +
> +	# A spaceless file on the feature branch.  Expect a match upstream.
> +	printf space >file &&
> +	git add file &&
> +	git commit -m"file without space" &&
> +	git log --format="- %H" -1 >>expect &&
> +
> +	# A further change.  Should not match upstream.
> +	test_commit change file &&
> +	git log --format="+ %H" -1 >>expect &&
> +
> +	git switch upstream-with-space &&
> +	# Same as the spaceless file, just with spaces and on upstream.
> +	test_commit "file with space" file "s p a c e" file-with-space &&
> +	git cherry upstream-with-space feature-without-space >actual &&
> +	test_cmp expect actual
> +'
> +
>  test_done
> --
> 2.28.0

> René Scharfe <l.s.r@web.de> writes:
>
> > Whitespace is ignored when calculating patch IDs.  This is done by
> > removing all whitespace from diff lines before hashing them, including
> > a newline at the end of a file.  If that newline is missing, however,
> > diff reports that fact in a separate line containing "\ No newline at
> > end of file\n", and this marker is hashed like a context line.
>
> Ah, ouch.
>
> > This goes against our goal of making patch IDs independent of
> > whitespace.  Use the same heuristic that 2485eab55cc (git-patch-id: do
> > not trip over "no newline" markers, 2011-02-17) added to git patch-id
> > instead and skip diff lines that start with a backslash and a space
> > and are longer than twelve characters.
>
> Good find of previous example.  Excellent.

>
> > Reported-by: Tilman Vogel <tilman.vogel@web.de>
> > Initial-test-by: Tilman Vogel <tilman.vogel@web.de>
> > Signed-off-by: René Scharfe <l.s.r@web.de>
> > ---
> >  diff.c            |  2 ++
> >  t/t3500-cherry.sh | 23 +++++++++++++++++++++++
> >  2 files changed, 25 insertions(+)
>
> Thanks.
>
> > diff --git a/diff.c b/diff.c
> > index f9709de7b45..f175019eb7a 100644
> > --- a/diff.c
> > +++ b/diff.c
> > @@ -6044,6 +6044,8 @@ static void patch_id_consume(void *priv, char *line, unsigned long len)
> >  	struct patch_id_t *data = priv;
> >  	int new_len;
> >
> > +	if (len > 12 && starts_with(line, "\\ "))
> > +		return;
> >  	new_len = remove_space(line, len);
> >
> >  	the_hash_algo->update_fn(data->ctx, line, new_len);
> > diff --git a/t/t3500-cherry.sh b/t/t3500-cherry.sh
> > index f038f34b7c0..2b8d9cb38ed 100755
> > --- a/t/t3500-cherry.sh
> > +++ b/t/t3500-cherry.sh
> > @@ -55,4 +55,27 @@ test_expect_success \
> >       expr "$(echo $(git cherry master my-topic-branch) )" : "+ [^ ]* - .*"
> >  '
> >
> > +test_expect_success 'cherry ignores whitespace' '
> > +	git switch --orphan=upstream-with-space &&
> > +	test_commit initial file &&
> > +	>expect &&
> > +	git switch --create=feature-without-space &&
> > +
> > +	# A spaceless file on the feature branch.  Expect a match upstream.
> > +	printf space >file &&
> > +	git add file &&
> > +	git commit -m"file without space" &&
> > +	git log --format="- %H" -1 >>expect &&
> > +
> > +	# A further change.  Should not match upstream.
> > +	test_commit change file &&
> > +	git log --format="+ %H" -1 >>expect &&
> > +
> > +	git switch upstream-with-space &&
> > +	# Same as the spaceless file, just with spaces and on upstream.
> > +	test_commit "file with space" file "s p a c e" file-with-space &&
> > +	git cherry upstream-with-space feature-without-space >actual &&
> > +	test_cmp expect actual
> > +'
> > +
> >  test_done
> > --
> > 2.28.0
>

>
> Whitespace is ignored when calculating patch IDs.  This is done by
> removing all whitespace from diff lines before hashing them, including
> a newline at the end of a file.  If that newline is missing, however,
> diff reports that fact in a separate line containing "\ No newline at
> end of file\n", and this marker is hashed like a context line.
>
> This goes against our goal of making patch IDs independent of
> whitespace.  Use the same heuristic that 2485eab55cc (git-patch-id: do
> not trip over "no newline" markers, 2011-02-17) added to git patch-id
> instead and skip diff lines that start with a backslash and a space
> and are longer than twelve characters.
>
> Reported-by: Tilman Vogel <tilman.vogel@web.de>
> Initial-test-by: Tilman Vogel <tilman.vogel@web.de>
> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>  diff.c            |  2 ++
>  t/t3500-cherry.sh | 23 +++++++++++++++++++++++
>  2 files changed, 25 insertions(+)
>
> diff --git a/diff.c b/diff.c
> index f9709de7b45..f175019eb7a 100644
> --- a/diff.c
> +++ b/diff.c
> @@ -6044,6 +6044,8 @@ static void patch_id_consume(void *priv, char *line, unsigned long len)
>         struct patch_id_t *data = priv;
>         int new_len;
>
> +       if (len > 12 && starts_with(line, "\\ "))
> +               return;
>         new_len = remove_space(line, len);
>
>         the_hash_algo->update_fn(data->ctx, line, new_len);
> diff --git a/t/t3500-cherry.sh b/t/t3500-cherry.sh
> index f038f34b7c0..2b8d9cb38ed 100755
> --- a/t/t3500-cherry.sh
> +++ b/t/t3500-cherry.sh
> @@ -55,4 +55,27 @@ test_expect_success \
>       expr "$(echo $(git cherry master my-topic-branch) )" : "+ [^ ]* - .*"
>  '
>
> +test_expect_success 'cherry ignores whitespace' '
> +       git switch --orphan=upstream-with-space &&
> +       test_commit initial file &&
> +       >expect &&
> +       git switch --create=feature-without-space &&
> +
> +       # A spaceless file on the feature branch.  Expect a match upstream.
> +       printf space >file &&
> +       git add file &&
> +       git commit -m"file without space" &&
> +       git log --format="- %H" -1 >>expect &&
> +
> +       # A further change.  Should not match upstream.
> +       test_commit change file &&
> +       git log --format="+ %H" -1 >>expect &&
> +
> +       git switch upstream-with-space &&
> +       # Same as the spaceless file, just with spaces and on upstream.
> +       test_commit "file with space" file "s p a c e" file-with-space &&
> +       git cherry upstream-with-space feature-without-space >actual &&
> +       test_cmp expect actual
> +'
> +
>  test_done
> --
> 2.28.0