threads / discuss / 48293

Regression in patch add?

Subject: Regression in patch add?

## tl;dr

22 messages between Apr 15, 2018 and Jul 11, 2018.

replies: 21people: 8as markdown or json

mqudsi@neosmart.net· Apr 15, 2018, 12:21 UTC · lore

Hello all,

I'm currently running the latest version of git built from `master`, and I'm running into what appears to be a regression in the behavior of the piecewise `git add -p` when applying a manually edited chunk.

I first run `git add -p`, then manually edit a chunk (after hitting `s` once, if it matters). The chunk originally contains the following:

```diff # Manual hunk edit mode -- see bottom for a quick guide

@@ -20,7 +20,7 @@
 	"call dein#add('Shougo/dein.vim', {'rev': 'master'})

 	" Add or remove your plugins here:
-	" call dein#add('flazz/vim-colorschemes')
-	call dein#add('Haron-Prime/evening_vim')
+	call dein#add('flazz/vim-colorschemes')
+	call dein#add('danilo-augusto/vim-afterglow')

 	"core plugins that change the behavior of vim and how we use it globally
```

Under git 2.7.4, I can edit it to the following, which is accepted
without a problem:

```diff
# Manual hunk edit mode -- see bottom for a quick guide
@@ -20,7 +20,7 @@
	"call dein#add('Shougo/dein.vim', {'rev': 'master'})

	" Add or remove your plugins here:
-	" call dein#add('flazz/vim-colorschemes')
-	call dein#add('Haron-Prime/evening_vim')
+	call dein#add('flazz/vim-colorschemes')
+	call dein#add('Haron-Prime/evening_vim')

	"core plugins that change the behavior of vim and how we use it globally
```

All I did here was remove one `+` line and manually add another (which
is a variant of the second `-` line).

Under git 2.17.0.252.gfe0a9ea, the same piece is opened in $VISUAL for
editing (and if left unmodified applies OK), but when modified in the
to the same exact value, after exiting the editor I receive the
following error from git:

    error: patch fragment without header at line 15: @@ -25,7 +25,8 @@

I'm not sure what to make of this.

Thank you,

Mahmoud Al-Qudsi
NeoSmart Technologies

Martin Ågren· Apr 15, 2018, 13:59 UTC · re: mqudsi@neosmart.net · lore

Re: Regression in patch add?

Hi Mahmoud

On 15 April 2018 at 14:21,  <mqudsi@neosmart.net> wrote:

> I first run `git add -p`, then manually edit a chunk (after hitting `s`
> once, if it matters). The chunk originally contains the following:

[...]

Show 19 quoted lines

> Under git 2.7.4, I can edit it to the following, which is accepted
> without a problem:
>
> ```diff
> # Manual hunk edit mode -- see bottom for a quick guide
> @@ -20,7 +20,7 @@
>         "call dein#add('Shougo/dein.vim', {'rev': 'master'})
>
>         " Add or remove your plugins here:
> -       " call dein#add('flazz/vim-colorschemes')
> -       call dein#add('Haron-Prime/evening_vim')
> +       call dein#add('flazz/vim-colorschemes')
> +       call dein#add('Haron-Prime/evening_vim')
>
>         "core plugins that change the behavior of vim and how we use it globally
> ```
>
> All I did here was remove one `+` line and manually add another (which
> is a variant of the second `-` line).

So the line is identical (sans s/^-/+/). Interesting.

Show 6 quoted lines

> Under git 2.17.0.252.gfe0a9ea, the same piece is opened in $VISUAL for
> editing (and if left unmodified applies OK), but when modified in the
> to the same exact value, after exiting the editor I receive the
> following error from git:
>
>     error: patch fragment without header at line 15: @@ -25,7 +25,8 @@

I can't seem to reproduce this with some very simple testing. Are you able to share your files? Or even better, derive a minimal reproduction recipe?

What happens if you do not do a "remove this line, then add it again", but instead turn that unchanged line into context? That is, you edit the hunk into something like this (but without white-space damage):

...
-       " call dein#add('flazz/vim-colorschemes')
+       call dein#add('flazz/vim-colorschemes')
        call dein#add('Haron-Prime/evening_vim')
...

Adding Phillip to cc, since he was recently working in this area and might have an idea.

Martin

Phillip Wood· Apr 16, 2018, 10:01 UTC · re: Martin Ågren · lore

Re: Regression in patch add?

On 15/04/18 14:59, Martin Ågren wrote:

Show 50 quoted lines

> Hi Mahmoud
> 
> On 15 April 2018 at 14:21,  <mqudsi@neosmart.net> wrote:
>> I first run `git add -p`, then manually edit a chunk (after hitting `s`
>> once, if it matters). The chunk originally contains the following:
> 
> [...]
> 
>> Under git 2.7.4, I can edit it to the following, which is accepted
>> without a problem:
>>
>> ```diff
>> # Manual hunk edit mode -- see bottom for a quick guide
>> @@ -20,7 +20,7 @@
>>         "call dein#add('Shougo/dein.vim', {'rev': 'master'})
>>
>>         " Add or remove your plugins here:
>> -       " call dein#add('flazz/vim-colorschemes')
>> -       call dein#add('Haron-Prime/evening_vim')
>> +       call dein#add('flazz/vim-colorschemes')
>> +       call dein#add('Haron-Prime/evening_vim')
>>
>>         "core plugins that change the behavior of vim and how we use it globally
>> ```
>>
>> All I did here was remove one `+` line and manually add another (which
>> is a variant of the second `-` line).
> 
> So the line is identical (sans s/^-/+/). Interesting.
> 
>> Under git 2.17.0.252.gfe0a9ea, the same piece is opened in $VISUAL for
>> editing (and if left unmodified applies OK), but when modified in the
>> to the same exact value, after exiting the editor I receive the
>> following error from git:
>>
>>     error: patch fragment without header at line 15: @@ -25,7 +25,8 @@
> 
> I can't seem to reproduce this with some very simple testing. Are you
> able to share your files? Or even better, derive a minimal reproduction
> recipe?
> 
> What happens if you do not do a "remove this line, then add it again",
> but instead turn that unchanged line into context? That is, you edit the
> hunk into something like this (but without white-space damage):
> 
> ...
> -       " call dein#add('flazz/vim-colorschemes')
> +       call dein#add('flazz/vim-colorschemes')
>         call dein#add('Haron-Prime/evening_vim')
> ...

That's a good idea to try

> Adding Phillip to cc, since he was recently working in this area

Thanks for cc-ing me

> and might have an idea.

I wish I did!

Best Wishes

Phillip

> 
> Martin
>

Phillip Wood· Apr 16, 2018, 10:00 UTC · re: mqudsi@neosmart.net · lore

Re: Regression in patch add?

On 15/04/18 13:21, mqudsi wrote:

Show 9 quoted lines

> 
> Hello all,
> 
> I'm currently running the latest version of git built from `master`, and
> I'm running into what appears to be a regression in the behavior of the
> piecewise `git add -p` when applying a manually edited chunk.
> 
> I first run `git add -p`, then manually edit a chunk (after hitting `s`
> once, if it matters).

Thanks for mentioning that, it can matter as the code that stitches split hunks back together can't cope with edited hunks properly (though the code that checks the hunk immediately after it's been edited doesn't bother to try and stitch things back together).

Show 42 quoted lines

> The chunk originally contains the following:
> 
> ```diff
> # Manual hunk edit mode -- see bottom for a quick guide
> @@ -20,7 +20,7 @@
>  	"call dein#add('Shougo/dein.vim', {'rev': 'master'})
> 
>  	" Add or remove your plugins here:
> -	" call dein#add('flazz/vim-colorschemes')
> -	call dein#add('Haron-Prime/evening_vim')
> +	call dein#add('flazz/vim-colorschemes')
> +	call dein#add('danilo-augusto/vim-afterglow')
> 
>  	"core plugins that change the behavior of vim and how we use it globally
> ```
> 
> Under git 2.7.4, I can edit it to the following, which is accepted
> without a problem:
> 
> ```diff
> # Manual hunk edit mode -- see bottom for a quick guide
> @@ -20,7 +20,7 @@
> 	"call dein#add('Shougo/dein.vim', {'rev': 'master'})
> 
> 	" Add or remove your plugins here:
> -	" call dein#add('flazz/vim-colorschemes')
> -	call dein#add('Haron-Prime/evening_vim')
> +	call dein#add('flazz/vim-colorschemes')
> +	call dein#add('Haron-Prime/evening_vim')
> 
> 	"core plugins that change the behavior of vim and how we use it globally
> ```
> 
> All I did here was remove one `+` line and manually add another (which
> is a variant of the second `-` line).
> 
> Under git 2.17.0.252.gfe0a9ea, the same piece is opened in $VISUAL for
> editing (and if left unmodified applies OK), but when modified in the
> to the same exact value, after exiting the editor I receive the
> following error from git:
> 
>     error: patch fragment without header at line 15: @@ -25,7 +25,8 @@

I'm not quite sure what that error message is telling us, I need to spend some time understanding the code in apply.c that creates this error message.

I assume that the header is coming from the next hunk which was created when you split the original hunk. If you could post the original hunk before it was split and the hunk starting at line 25 after it was split that might help.

As Martin said if you could share the files or come up with a reproducible example that would really help in figuring out what is going wrong.

Thanks for reporting this

Phillip

Show 8 quoted lines

> I'm not sure what to make of this.
> 
> Thank you,
> 
> Mahmoud Al-Qudsi
> NeoSmart Technologies
> 
>

Oliver Joseph Ash· May 10, 2018, 10:41 UTC · re: mqudsi@neosmart.net · lore

Re: Regression in patch add?

I just ran into a similar problem: https://stackoverflow.com/questions/50258565/git-editing-hunks-fails-when-file-has-other-hunks

I can reproduce on 2.17.0. The issue doesn't occur on 2.16.2, however.

Is this a bug?

Martin Ågren· May 10, 2018, 12:17 UTC · re: Oliver Joseph Ash · lore

Re: Regression in patch add?

On 10 May 2018 at 12:41, Oliver Joseph Ash <oliverjash@gmail.com> wrote:

Show 5 quoted lines

> I just ran into a similar problem: https://stackoverflow.com/questions/50258565/git-editing-hunks-fails-when-file-has-other-hunks
>
> I can reproduce on 2.17.0. The issue doesn't occur on 2.16.2, however.
>
> Is this a bug?

I would think so. Thanks for finding this thread. To keep history around, it would be nice to have your reproduction recipe on the list, not just on stackoverflow. That said, I cannot reproduce on v2.17.0 using your recipe. I suspect there is something quite interesting going on here, considering how trivial your edit is.

As a shot in the dark, does your test involve unusual file systems, funny characters in filenames, ..? You are on some sort of Linux, right?

The first thing to try out might be something like

$ # create the initial file as before, with "bar" $ # git add, git commit ... $ # do the "change bar to bar1" everywhere $ git diff >test-patch $ git reset --hard $ # edit the *FIRST* hunk in test.patch like before (bar1 -> bar2) $ git apply --check test.patch && echo "ok..." $ git apply test.patch

Does that succeed at all?

$ git diff

should now show bar2 in the first hunk and bar1 in the second hunk, just like your edited test.patch.

If that works, it would seem that the problem is with `git add -p`, and how it is generating the patches for `git apply`. I have some ideas about how to debug from there, but ... How comfortable are you with building Git from the sources? Or with temporarily fiddling around with your Git installation? (git-add--interactive is a Perl script, so it would be possible to edit it in place to emit various debug information. That has potential for messing up royally, though.)

Martin

Oliver Joseph Ash· May 10, 2018, 13:16 UTC · re: Martin Ågren · lore

Re: Regression in patch add?

(Apologies, I accidentally sent this as a reply to the original post, instead of your email. I'm new to this!)

> does your test involve unusual file systems, funny characters in filenames, ..? You are on some sort of Linux, right?

I'm running macOS 10.13.4. I don't have any unusual file system setup, as far as I'm aware. The filename in my test case is simply `foo`.

I tried the steps you suggested: on git 2.17.0, saving the patch, editing it, and applying it, and it succeeded.

> should now show bar2 in the first hunk and bar1 in the second hunk, just like your edited test.patch.

That was the case, although I had to remove the `--check` flag from `git apply`.

> How comfortable are you with building Git from the sources?

I've never done it before, but I assume it's well documented, so I'm willing to give it a shot!

Happy to try any steps to debug this! Although I'm a bit surprised no-one else can reproduce it with the same version of Git, which makes it seem less likely this could be a bug, and more likely it's something in my setup.

Martin Ågren· May 10, 2018, 13:54 UTC · re: Oliver Joseph Ash · lore

Re: Regression in patch add?

On 10 May 2018 at 15:16, Oliver Joseph Ash <oliverjash@gmail.com> wrote:

> (Apologies, I accidentally sent this as a reply to the original post, instead of your email. I'm new to this!)

(No worries.) ;-)

>> does your test involve unusual file systems, funny characters in filenames, ..? You are on some sort of Linux, right?
>
> I'm running macOS 10.13.4. I don't have any unusual file system setup, as far as I'm aware. The filename in my test case is simply `foo`.

I'm not too familiar with Mac, unfortunately, but let's see..

Show 5 quoted lines

> I tried the steps you suggested: on git 2.17.0, saving the patch, editing it, and applying it, and it succeeded.
>
>> should now show bar2 in the first hunk and bar1 in the second hunk, just like your edited test.patch.
>
> That was the case, although I had to remove the `--check` flag from `git apply`.

Hmm, you mean that `git apply --check test.patch` failed? With error messages? Or, you had to remove the --check flag in order for the patch to actually be applied on disk? I would guess it's the latter, but just to be clear.

Show 5 quoted lines

>> How comfortable are you with building Git from the sources?
>
> I've never done it before, but I assume it's well documented, so I'm willing to give it a shot!
>
> Happy to try any steps to debug this! Although I'm a bit surprised no-one else can reproduce it with the same version of Git, which makes it seem less likely this could be a bug, and more likely it's something in my setup.

Where do the git 2.17.0 and 2.16.2 come from that you have been testing? Homebrew? Apple? (Ple

So you should be able to do `git clone https://github.com/git/git.git` and read INSTALL. It might be useful to start with `git checkout v2.17.0` to make sure you're testing roughly the same thing as before.

As for obtaining the dependencies, since I'm not familiar with Mac, I cannot give any good hints.

I see now that Phillip has replied with a good guess. Let's hope he has managed to circle in on what's causing your problem.

Martin

Phillip Wood· May 10, 2018, 13:49 UTC · re: Martin Ågren · lore

Re: Regression in patch add?

On 10/05/18 13:17, Martin Ågren wrote:

Show 13 quoted lines

> 
> On 10 May 2018 at 12:41, Oliver Joseph Ash <oliverjash@gmail.com> wrote:
>> I just ran into a similar problem: https://stackoverflow.com/questions/50258565/git-editing-hunks-fails-when-file-has-other-hunks
>>
>> I can reproduce on 2.17.0. The issue doesn't occur on 2.16.2, however.
>>
>> Is this a bug?
> 
> I would think so. Thanks for finding this thread. To keep history
> around, it would be nice to have your reproduction recipe on the list,
> not just on stackoverflow. That said, I cannot reproduce on v2.17.0
> using your recipe. I suspect there is something quite interesting going
> on here, considering how trivial your edit is.

Thanks Oliver for posting an example that we can test, that said I can't reproduce it on Linux if the hunk is edited correctly. However if I remove the leading space from the empty line between 'baz' and 'foo' then I get the same error as you. Perhaps your editor is stripping trailing white space? If so that will lead to problems when editing diffs as the leading space is needed for apply to know that it's an empty context line.

For the mailing list the hunk in question looks like

@@ -1,5 +1,5 @@
 foo
-bar
+bar1
 baz

 foo

I've tried using 'git apply --recount --cached' directly and was
surprised to see that it accepts the patch with the broken context line.
In 2.17.0 'add -p' no longer uses the --recount option, instead it
counts the patch it's self but stops counting when it runs out of lines
starting with [- +], this explains the difference from earlier versions.
It seems it's not uncommon for editors to strip the space from empty
context lines so maybe 'add -p' should take that into account when
recounting patches. I'm about to go off line for a couple of weeks so it
will probably be next month before I'm able to put a patch together
(assuming Junio agrees we should support broken hunks)

Best Wishes

Phillip


> As a shot in the dark, does your test involve unusual file systems,
> funny characters in filenames, ..? You are on some sort of Linux, right?
> 
> The first thing to try out might be something like
> 
> $ # create the initial file as before, with "bar"
> $ # git add, git commit ...
> $ # do the "change bar to bar1" everywhere
> $ git diff >test-patch
> $ git reset --hard
> $ # edit the *FIRST* hunk in test.patch like before (bar1 -> bar2)
> $ git apply --check test.patch && echo "ok..."
> $ git apply test.patch
> 
> Does that succeed at all?
> 
> $ git diff
> 
> should now show bar2 in the first hunk and bar1 in the second hunk,
> just like your edited test.patch.
> 
> If that works, it would seem that the problem is with `git add -p`, and
> how it is generating the patches for `git apply`. I have some ideas
> about how to debug from there, but ... How comfortable are you with
> building Git from the sources? Or with temporarily fiddling around with
> your Git installation? (git-add--interactive is a Perl script, so it
> would be possible to edit it in place to emit various debug
> information. That has potential for messing up royally, though.)
> 
> Martin
>

Oliver Joseph Ash· May 10, 2018, 14:11 UTC · re: Phillip Wood · lore

Re: Regression in patch add?

You found the problem Phillip! My editor was trimming trailing white space, which breaks the context line.

I had tried to use an alternative editor to account for any editor specific behaviour, but it turns out both the editors I tested in were doing this!

I suspect this change in behaviour will effect a lot of users? If so, it would be good if `git add -p` allowed for this behaviour, in the same way `git apply` does.

Meanwhile, I can easily configure my editor not to do this for `*.diff` files.

Thanks for your help, Phillip and Martin!

Mahmoud, does this also explain your problem as per your original post?

Phillip Wood· May 10, 2018, 17:58 UTC · re: Oliver Joseph Ash · lore

Re: Regression in patch add?

On 10/05/18 15:11, Oliver Joseph Ash wrote:

> You found the problem Phillip! My editor was trimming trailing white space, which breaks the context line.

I'm glad we found the source of the problem (and that it wasn't some obscure bug)

> I had tried to use an alternative editor to account for any editor specific behaviour, but it turns out both the editors I tested in were doing this!
> 
> I suspect this change in behaviour will effect a lot of users? If so, it would be good if `git add -p` allowed for this behaviour, in the same way `git apply` does.

Yes, I think it probably makes sense to do that. Originally I didn't count empty lines as context lines in case the user accidentally added some empty lines at the end of the hunk but if 'git apply' does then I think 'git add -p' should as well

> Meanwhile, I can easily configure my editor not to do this for `*.diff` files.
> 
> Thanks for your help, Phillip and Martin!

Thanks for posting an example so we could test it, it makes it much easier to track the problem down

Best Wishes

Phillip

> Mahmoud, does this also explain your problem as per your original post?
>

Junio C Hamano· May 11, 2018, 02:47 UTC · re: Phillip Wood · lore

Re: Regression in patch add?

Phillip Wood <phillip.wood@talktalk.net> writes:

> Yes, I think it probably makes sense to do that. Originally I didn't
> count empty lines as context lines in case the user accidentally added
> some empty lines at the end of the hunk but if 'git apply' does then I
> think 'git add -p' should as well

I am not sure if "adding to the tail" should be tolerated, but in any case, newer GNU diff can show an empty unaffected line as an empty line (unlike traditional unified context format in which such a line is expressed as a line with a lone SP on it), which is allowed as "implementation defined" by POSIX.1 [*1*]. Modern "git apply" knows about this.

If "add -p" parses a patch, it should learn to do so, too.

[Reference]

*1* http://pubs.opengroup.org/onlinepubs/9699919799/utilities/diff.html

Show 14 quoted lines

>
>> Meanwhile, I can easily configure my editor not to do this for `*.diff` files.
>> 
>> Thanks for your help, Phillip and Martin!
>
> Thanks for posting an example so we could test it, it makes it much
> easier to track the problem down
>
> Best Wishes
>
> Phillip
>
>> Mahmoud, does this also explain your problem as per your original post?
>>

Phillip Wood· May 11, 2018, 18:23 UTC · re: Junio C Hamano · lore

Re: Regression in patch add?

On 11/05/18 03:47, Junio C Hamano wrote:

Show 13 quoted lines

> Phillip Wood <phillip.wood@talktalk.net> writes:
> 
>> Yes, I think it probably makes sense to do that. Originally I didn't
>> count empty lines as context lines in case the user accidentally added
>> some empty lines at the end of the hunk but if 'git apply' does then I
>> think 'git add -p' should as well
> 
> I am not sure if "adding to the tail" should be tolerated, but in
> any case, newer GNU diff can show an empty unaffected line as an
> empty line (unlike traditional unified context format in which such
> a line is expressed as a line with a lone SP on it), which is
> allowed as "implementation defined" by POSIX.1 [*1*]. Modern "git
> apply" knows about this.

Thanks for the reference, I hadn't realized the space was optional.

> If "add -p" parses a patch, it should learn to do so, too.

I'm about to go off line for a while, I'll send a fix when I'm back up and running at next month (unfortunately the reroll of pw/add-p-select will have to wait until then as well)

Best Wishes

Phillip

Show 19 quoted lines

> 
> [Reference]
> 
> *1* http://pubs.opengroup.org/onlinepubs/9699919799/utilities/diff.html
> 
>>
>>> Meanwhile, I can easily configure my editor not to do this for `*.diff` files.
>>>
>>> Thanks for your help, Phillip and Martin!
>>
>> Thanks for posting an example so we could test it, it makes it much
>> easier to track the problem down
>>
>> Best Wishes
>>
>> Phillip
>>
>>> Mahmoud, does this also explain your problem as per your original post?
>>>

Oliver Joseph Ash· May 10, 2018, 13:15 UTC · re: mqudsi@neosmart.net · lore

Re: Regression in patch add?

> does your test involve unusual file systems, funny characters in filenames, ..? You are on some sort of Linux, right?

I'm running macOS 10.13.4. I don't have any unusual file system setup, as far as I'm aware. The filename in my test case is simply `foo`.

I tried the steps you suggested: on git 2.17.0, saving the patch, editing it, and applying it, and it succeeded.

> should now show bar2 in the first hunk and bar1 in the second hunk, just like your edited test.patch.

That was the case, although I had to remove the `--check` flag from `git apply`.

> How comfortable are you with building Git from the sources?

I've never done it before, but I assume it's well documented, so I'm willing to give it a shot!

Happy to try any steps to debug this! Although I'm a bit surprised no-one else can reproduce it with the same version of Git, which makes it seem less likely this could be a bug, and more likely it's something in my setup.

Phillip Wood· Jun 1, 2018, 17:46 UTC · re: mqudsi@neosmart.net · lore

[PATCH] add -p: fix counting empty context lines in edited patches

From: Phillip Wood <phillip.wood@dunelm.org.uk>

recount_edited_hunk() introduced in commit 2b8ea7f3c7 ("add -p: calculate offset delta for edited patches", 2018-03-05) required all context lines to start with a space, empty lines are not counted. This was intended to avoid any recounting problems if the user had introduced empty lines at the end when editing the patch. However this introduced a regression into 'git add -p' as it seems it is common for editors to strip the trailing whitespace from empty context lines when patches are edited thereby introducing empty lines that should be counted. 'git apply' knows how to deal with such empty lines and POSIX states that whether or not there is an space on an empty context line is implementation defined [1].

Fix the regression by counting lines consist solely of a newline as well as lines starting with a space as context lines and add a test to prevent future regressions.

[1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/diff.html

Reported-by: Mahmoud Al-Qudsi <mqudsi@neosmart.net>
Reported-by: Oliver Joseph Ash <oliverjash@gmail.com>
Reported-by: Jeff Felchner <jfelchner1@gmail.com>
Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
---
My apologies to everyone who was affected by this regression.

 git-add--interactive.perl  |  2 +-
 t/t3701-add-interactive.sh | 43 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 1 deletion(-)

diff --git a/git-add--interactive.perl b/git-add--interactive.perl
index ab022ec073..bb6f249f03 100755
--- a/git-add--interactive.perl
+++ b/git-add--interactive.perl
@@ -1047,7 +1047,7 @@ sub recount_edited_hunk {
 			$o_cnt++;
 		} elsif ($mode eq '+') {
 			$n_cnt++;
-		} elsif ($mode eq ' ') {
+		} elsif ($mode eq ' ' or $_ eq "\n") {
 			$o_cnt++;
 			$n_cnt++;
 		}
diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
index e5c66f7500..f1bb879ea4 100755
--- a/t/t3701-add-interactive.sh
+++ b/t/t3701-add-interactive.sh
@@ -175,6 +175,49 @@ test_expect_success 'real edit works' '
 	diff_cmp expected output
 '
 
+test_expect_success 'setup file' '
+	test_write_lines a "" b "" c >file &&
+	git add file &&
+	test_write_lines a "" d "" c >file
+'
+
+test_expect_success 'setup patch' '
+	SP=" " &&
+	NULL="" &&
+	cat >patch <<-EOF
+	@@ -1,4 +1,4 @@
+	 a
+	$NULL
+	-b
+	+f
+	$SP
+	c
+	EOF
+'
+
+test_expect_success 'setup expected' '
+	cat >expected <<-EOF
+	diff --git a/file b/file
+	index b5dd6c9..f910ae9 100644
+	--- a/file
+	+++ b/file
+	@@ -1,5 +1,5 @@
+	 a
+	$SP
+	-f
+	+d
+	$SP
+	 c
+	EOF
+'
+
+test_expect_success 'edit can strip spaces from empty context lines' '
+	test_write_lines e n q | git add -p 2>error &&
+	test_must_be_empty error &&
+	git diff >output &&
+	diff_cmp expected output
+'
+
 test_expect_success 'skip files similarly as commit -a' '
 	git reset &&
 	echo file >.gitignore &&

-- 
2.17.0

Jacob Keller· Jun 1, 2018, 19:07 UTC · re: Phillip Wood · lore

Re: [PATCH] add -p: fix counting empty context lines in edited patches

On Fri, Jun 1, 2018 at 10:46 AM, Phillip Wood <phillip.wood@talktalk.net> wrote:

Show 27 quoted lines

> From: Phillip Wood <phillip.wood@dunelm.org.uk>
>
> recount_edited_hunk() introduced in commit 2b8ea7f3c7 ("add -p:
> calculate offset delta for edited patches", 2018-03-05) required all
> context lines to start with a space, empty lines are not counted. This
> was intended to avoid any recounting problems if the user had
> introduced empty lines at the end when editing the patch. However this
> introduced a regression into 'git add -p' as it seems it is common for
> editors to strip the trailing whitespace from empty context lines when
> patches are edited thereby introducing empty lines that should be
> counted. 'git apply' knows how to deal with such empty lines and POSIX
> states that whether or not there is an space on an empty context line
> is implementation defined [1].
>
> Fix the regression by counting lines consist solely of a newline as
> well as lines starting with a space as context lines and add a test to
> prevent future regressions.
>
> [1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/diff.html
>
> Reported-by: Mahmoud Al-Qudsi <mqudsi@neosmart.net>
> Reported-by: Oliver Joseph Ash <oliverjash@gmail.com>
> Reported-by: Jeff Felchner <jfelchner1@gmail.com>
> Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
> ---
> My apologies to everyone who was affected by this regression.
>

Ahhh I suspect this is why my edited code in add -p was sometimes failing to apply!

Thanks, Jake

Show 74 quoted lines

>  git-add--interactive.perl  |  2 +-
>  t/t3701-add-interactive.sh | 43 ++++++++++++++++++++++++++++++++++++++
>  2 files changed, 44 insertions(+), 1 deletion(-)
>
> diff --git a/git-add--interactive.perl b/git-add--interactive.perl
> index ab022ec073..bb6f249f03 100755
> --- a/git-add--interactive.perl
> +++ b/git-add--interactive.perl
> @@ -1047,7 +1047,7 @@ sub recount_edited_hunk {
>                         $o_cnt++;
>                 } elsif ($mode eq '+') {
>                         $n_cnt++;
> -               } elsif ($mode eq ' ') {
> +               } elsif ($mode eq ' ' or $_ eq "\n") {
>                         $o_cnt++;
>                         $n_cnt++;
>                 }
> diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
> index e5c66f7500..f1bb879ea4 100755
> --- a/t/t3701-add-interactive.sh
> +++ b/t/t3701-add-interactive.sh
> @@ -175,6 +175,49 @@ test_expect_success 'real edit works' '
>         diff_cmp expected output
>  '
>
> +test_expect_success 'setup file' '
> +       test_write_lines a "" b "" c >file &&
> +       git add file &&
> +       test_write_lines a "" d "" c >file
> +'
> +
> +test_expect_success 'setup patch' '
> +       SP=" " &&
> +       NULL="" &&
> +       cat >patch <<-EOF
> +       @@ -1,4 +1,4 @@
> +        a
> +       $NULL
> +       -b
> +       +f
> +       $SP
> +       c
> +       EOF
> +'
> +
> +test_expect_success 'setup expected' '
> +       cat >expected <<-EOF
> +       diff --git a/file b/file
> +       index b5dd6c9..f910ae9 100644
> +       --- a/file
> +       +++ b/file
> +       @@ -1,5 +1,5 @@
> +        a
> +       $SP
> +       -f
> +       +d
> +       $SP
> +        c
> +       EOF
> +'
> +
> +test_expect_success 'edit can strip spaces from empty context lines' '
> +       test_write_lines e n q | git add -p 2>error &&
> +       test_must_be_empty error &&
> +       git diff >output &&
> +       diff_cmp expected output
> +'
> +
>  test_expect_success 'skip files similarly as commit -a' '
>         git reset &&
>         echo file >.gitignore &&
> --
> 2.17.0
>

Eric Sunshine· Jun 1, 2018, 20:03 UTC · re: Phillip Wood · lore

Re: [PATCH] add -p: fix counting empty context lines in edited patches

On Fri, Jun 1, 2018 at 1:46 PM, Phillip Wood <phillip.wood@talktalk.net> wrote:

Show 13 quoted lines

> recount_edited_hunk() introduced in commit 2b8ea7f3c7 ("add -p:
> calculate offset delta for edited patches", 2018-03-05) required all
> context lines to start with a space, empty lines are not counted. This
> was intended to avoid any recounting problems if the user had
> introduced empty lines at the end when editing the patch. However this
> introduced a regression into 'git add -p' as it seems it is common for
> editors to strip the trailing whitespace from empty context lines when
> patches are edited thereby introducing empty lines that should be
> counted. 'git apply' knows how to deal with such empty lines and POSIX
> states that whether or not there is an space on an empty context line
> is implementation defined [1].
>
> Fix the regression by counting lines consist solely of a newline as

s/consist/&ing/ --or-- s/consist/that &/

Show 10 quoted lines

> well as lines starting with a space as context lines and add a test to
> prevent future regressions.
>
> Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
> ---
>  git-add--interactive.perl  |  2 +-
> diff --git a/git-add--interactive.perl b/git-add--interactive.perl
> @@ -1047,7 +1047,7 @@ sub recount_edited_hunk {
> -               } elsif ($mode eq ' ') {
> +               } elsif ($mode eq ' ' or $_ eq "\n") {

Based upon a very cursory read of parts of git-add-interactive.perl, do I understand correctly that we don't have to worry about $_ ever being "\r\n" on Windows?

Show 28 quoted lines

> diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
> @@ -175,6 +175,49 @@ test_expect_success 'real edit works' '
> +test_expect_success 'setup file' '
> +       test_write_lines a "" b "" c >file &&
> +       git add file &&
> +       test_write_lines a "" d "" c >file
> +'
> +
> +test_expect_success 'setup patch' '
> +       SP=" " &&
> +       NULL="" &&
> +       cat >patch <<-EOF
> +       [...]
> +       EOF
> +'
> +
> +test_expect_success 'setup expected' '
> +       cat >expected <<-EOF
> +       [...]
> +       EOF
> +'
> +
> +test_expect_success 'edit can strip spaces from empty context lines' '
> +       test_write_lines e n q | git add -p 2>error &&
> +       test_must_be_empty error &&
> +       git diff >output &&
> +       diff_cmp expected output
> +'

I would have expected all the setup work to be contained directly in the sole test which needs it rather than spread over three tests (two of which are composed of a single command). Not a big deal, and not worth a re-roll.

Phillip Wood· Jun 4, 2018, 10:08 UTC · re: Eric Sunshine · lore

Re: [PATCH] add -p: fix counting empty context lines in edited patches

On 01/06/18 21:03, Eric Sunshine wrote:

Show 18 quoted lines

> On Fri, Jun 1, 2018 at 1:46 PM, Phillip Wood <phillip.wood@talktalk.net> wrote:
>> recount_edited_hunk() introduced in commit 2b8ea7f3c7 ("add -p:
>> calculate offset delta for edited patches", 2018-03-05) required all
>> context lines to start with a space, empty lines are not counted. This
>> was intended to avoid any recounting problems if the user had
>> introduced empty lines at the end when editing the patch. However this
>> introduced a regression into 'git add -p' as it seems it is common for
>> editors to strip the trailing whitespace from empty context lines when
>> patches are edited thereby introducing empty lines that should be
>> counted. 'git apply' knows how to deal with such empty lines and POSIX
>> states that whether or not there is an space on an empty context line
>> is implementation defined [1].
>>
>> Fix the regression by counting lines consist solely of a newline as
> 
> s/consist/&ing/
> --or--
> s/consist/that &/

Thanks, I'd intended to say 'that consist'

Show 15 quoted lines

>> well as lines starting with a space as context lines and add a test to
>> prevent future regressions.
>>
>> Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
>> ---
>>  git-add--interactive.perl  |  2 +-
>> diff --git a/git-add--interactive.perl b/git-add--interactive.perl
>> @@ -1047,7 +1047,7 @@ sub recount_edited_hunk {
>> -               } elsif ($mode eq ' ') {
>> +               } elsif ($mode eq ' ' or $_ eq "\n") {
> 
> Based upon a very cursory read of parts of git-add-interactive.perl,
> do I understand correctly that we don't have to worry about $_ ever
> being "\r\n" on Windows?
>

Good question, I think the short answer no. If my understanding of the newline section of perlport [1] is correct then on Windows "\n" eq "\012" and the io layer replaces "\015\012" with "\n" when reading in 'text' mode (which I think is the default if you don't specify one when opening the file/process or with binmode()). As "\n" is only one character it would perhaps be better to test '$mode' rather than '$_' above - what do you think.

[1] http://perldoc.perl.org/perlport.html#Newlines

Show 33 quoted lines

>> diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
>> @@ -175,6 +175,49 @@ test_expect_success 'real edit works' '
>> +test_expect_success 'setup file' '
>> +       test_write_lines a "" b "" c >file &&
>> +       git add file &&
>> +       test_write_lines a "" d "" c >file
>> +'
>> +
>> +test_expect_success 'setup patch' '
>> +       SP=" " &&
>> +       NULL="" &&
>> +       cat >patch <<-EOF
>> +       [...]
>> +       EOF
>> +'
>> +
>> +test_expect_success 'setup expected' '
>> +       cat >expected <<-EOF
>> +       [...]
>> +       EOF
>> +'
>> +
>> +test_expect_success 'edit can strip spaces from empty context lines' '
>> +       test_write_lines e n q | git add -p 2>error &&
>> +       test_must_be_empty error &&
>> +       git diff >output &&
>> +       diff_cmp expected output
>> +'
> 
> I would have expected all the setup work to be contained directly in
> the sole test which needs it rather than spread over three tests (two
> of which are composed of a single command). Not a big deal, and not
> worth a re-roll.

Good point I was torn between that and matching the existing style in that file seems to be to create a million ancillary tests to do the set-up.

Thanks

Phillip

Eric Sunshine· Jun 4, 2018, 17:21 UTC · re: Phillip Wood · lore

Re: [PATCH] add -p: fix counting empty context lines in edited patches

On Mon, Jun 4, 2018 at 6:08 AM, Phillip Wood <phillip.wood@talktalk.net> wrote:

Show 13 quoted lines

> On 01/06/18 21:03, Eric Sunshine wrote:
>> On Fri, Jun 1, 2018 at 1:46 PM, Phillip Wood <phillip.wood@talktalk.net> wrote:
>>> +               } elsif ($mode eq ' ' or $_ eq "\n") {
>>
>> Based upon a very cursory read of parts of git-add-interactive.perl,
>> do I understand correctly that we don't have to worry about $_ ever
>> being "\r\n" on Windows?
>
> Good question, I think the short answer no. If my understanding of the
> newline section of perlport [1] is correct then on Windows "\n" eq
> "\012" and the io layer replaces "\015\012" with "\n" when reading in
> 'text' mode (which I think is the default if you don't specify one when
> opening the file/process or with binmode()).

That was my interpretation, as well (though I didn't audit the code closely).

> As "\n" is only one
> character it would perhaps be better to test '$mode' rather than '$_'
> above - what do you think.

That could be clearer. As a reviewer, I had to spend extra brain cycles wondering why $mode was used everywhere else but $_ in just this one place. (Not that it was difficult to figure out.)

Show 25 quoted lines

>>> diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
>>> @@ -175,6 +175,49 @@ test_expect_success 'real edit works' '
>>> +test_expect_success 'setup file' '
>>> +       [...]
>>> +'
>>> +test_expect_success 'setup patch' '
>>> +       [...]
>>> +'
>>> +test_expect_success 'setup expected' '
>>> +       [...]
>>> +'
>>> +test_expect_success 'edit can strip spaces from empty context lines' '
>>> +       test_write_lines e n q | git add -p 2>error &&
>>> +       test_must_be_empty error &&
>>> +       git diff >output &&
>>> +       diff_cmp expected output
>>> +'
>>
>> I would have expected all the setup work to be contained directly in
>> the sole test which needs it rather than spread over three tests (two
>> of which are composed of a single command). Not a big deal, and not
>> worth a re-roll.
>
> Good point I was torn between that and matching the existing style in
> that file seems to be to create a million ancillary tests to do the set-up.

I see what you mean. Following existing practice in the file makes sense, though breaking from that practice by bundling all the setup into the single test which uses it wouldn't hurt either. It's a judgment call (and not worrying about too much).

Phillip Wood· Jun 11, 2018, 09:46 UTC · re: Phillip Wood · lore

[PATCH v2] add -p: fix counting empty context lines in edited patches

From: Phillip Wood <phillip.wood@dunelm.org.uk>

Fix the regression by counting lines that consist solely of a newline as well as lines starting with a space as context lines and add a test to prevent future regressions.

[1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/diff.html

Reported-by: Mahmoud Al-Qudsi <mqudsi@neosmart.net>
Reported-by: Oliver Joseph Ash <oliverjash@gmail.com>
Reported-by: Jeff Felchner <jfelchner1@gmail.com>
Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
---

Thanks for the feedback, the only changes since v1 are to fix the commit message to match what was in pu and to change '$_' to '$mode' in the comparison as I think that is clearer. In the end I decided to leave the tests as they are.

 git-add--interactive.perl  |  2 +-
 t/t3701-add-interactive.sh | 43 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 1 deletion(-)

diff --git a/git-add--interactive.perl b/git-add--interactive.perl
index ab022ec073..8361ef45e7 100755
--- a/git-add--interactive.perl
+++ b/git-add--interactive.perl
@@ -1047,7 +1047,7 @@ sub recount_edited_hunk {
 			$o_cnt++;
 		} elsif ($mode eq '+') {
 			$n_cnt++;
-		} elsif ($mode eq ' ') {
+		} elsif ($mode eq ' ' or $mode eq "\n") {
 			$o_cnt++;
 			$n_cnt++;
 		}
diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
index e5c66f7500..f1bb879ea4 100755
--- a/t/t3701-add-interactive.sh
+++ b/t/t3701-add-interactive.sh
@@ -175,6 +175,49 @@ test_expect_success 'real edit works' '
 	diff_cmp expected output
 '
 
+test_expect_success 'setup file' '
+	test_write_lines a "" b "" c >file &&
+	git add file &&
+	test_write_lines a "" d "" c >file
+'
+
+test_expect_success 'setup patch' '
+	SP=" " &&
+	NULL="" &&
+	cat >patch <<-EOF
+	@@ -1,4 +1,4 @@
+	 a
+	$NULL
+	-b
+	+f
+	$SP
+	c
+	EOF
+'
+
+test_expect_success 'setup expected' '
+	cat >expected <<-EOF
+	diff --git a/file b/file
+	index b5dd6c9..f910ae9 100644
+	--- a/file
+	+++ b/file
+	@@ -1,5 +1,5 @@
+	 a
+	$SP
+	-f
+	+d
+	$SP
+	 c
+	EOF
+'
+
+test_expect_success 'edit can strip spaces from empty context lines' '
+	test_write_lines e n q | git add -p 2>error &&
+	test_must_be_empty error &&
+	git diff >output &&
+	diff_cmp expected output
+'
+
 test_expect_success 'skip files similarly as commit -a' '
 	git reset &&
 	echo file >.gitignore &&

-- 
2.17.0

Jeff Felchner· Jul 11, 2018, 20:27 UTC · re: Phillip Wood · lore

Re: [PATCH v2] add -p: fix counting empty context lines in edited patches

Hey all, I assumed this was going to be in 2.18, but I'm still having the same issue.  What's the plan for release of this?

Show 107 quoted lines

> On 2018 Jun 11, at 4:46, Phillip Wood <phillip.wood@talktalk.net> wrote:
> 
> From: Phillip Wood <phillip.wood@dunelm.org.uk>
> 
> recount_edited_hunk() introduced in commit 2b8ea7f3c7 ("add -p:
> calculate offset delta for edited patches", 2018-03-05) required all
> context lines to start with a space, empty lines are not counted. This
> was intended to avoid any recounting problems if the user had
> introduced empty lines at the end when editing the patch. However this
> introduced a regression into 'git add -p' as it seems it is common for
> editors to strip the trailing whitespace from empty context lines when
> patches are edited thereby introducing empty lines that should be
> counted. 'git apply' knows how to deal with such empty lines and POSIX
> states that whether or not there is an space on an empty context line
> is implementation defined [1].
> 
> Fix the regression by counting lines that consist solely of a newline
> as well as lines starting with a space as context lines and add a test
> to prevent future regressions.
> 
> [1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/diff.html
> 
> Reported-by: Mahmoud Al-Qudsi <mqudsi@neosmart.net>
> Reported-by: Oliver Joseph Ash <oliverjash@gmail.com>
> Reported-by: Jeff Felchner <jfelchner1@gmail.com>
> Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
> ---
> 
> Thanks for the feedback, the only changes since v1 are to fix the
> commit message to match what was in pu and to change '$_' to '$mode'
> in the comparison as I think that is clearer. In the end I decided to
> leave the tests as they are.
> 
> git-add--interactive.perl  |  2 +-
> t/t3701-add-interactive.sh | 43 ++++++++++++++++++++++++++++++++++++++
> 2 files changed, 44 insertions(+), 1 deletion(-)
> 
> diff --git a/git-add--interactive.perl b/git-add--interactive.perl
> index ab022ec073..8361ef45e7 100755
> --- a/git-add--interactive.perl
> +++ b/git-add--interactive.perl
> @@ -1047,7 +1047,7 @@ sub recount_edited_hunk {
> 			$o_cnt++;
> 		} elsif ($mode eq '+') {
> 			$n_cnt++;
> -		} elsif ($mode eq ' ') {
> +		} elsif ($mode eq ' ' or $mode eq "\n") {
> 			$o_cnt++;
> 			$n_cnt++;
> 		}
> diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
> index e5c66f7500..f1bb879ea4 100755
> --- a/t/t3701-add-interactive.sh
> +++ b/t/t3701-add-interactive.sh
> @@ -175,6 +175,49 @@ test_expect_success 'real edit works' '
> 	diff_cmp expected output
> '
> 
> +test_expect_success 'setup file' '
> +	test_write_lines a "" b "" c >file &&
> +	git add file &&
> +	test_write_lines a "" d "" c >file
> +'
> +
> +test_expect_success 'setup patch' '
> +	SP=" " &&
> +	NULL="" &&
> +	cat >patch <<-EOF
> +	@@ -1,4 +1,4 @@
> +	 a
> +	$NULL
> +	-b
> +	+f
> +	$SP
> +	c
> +	EOF
> +'
> +
> +test_expect_success 'setup expected' '
> +	cat >expected <<-EOF
> +	diff --git a/file b/file
> +	index b5dd6c9..f910ae9 100644
> +	--- a/file
> +	+++ b/file
> +	@@ -1,5 +1,5 @@
> +	 a
> +	$SP
> +	-f
> +	+d
> +	$SP
> +	 c
> +	EOF
> +'
> +
> +test_expect_success 'edit can strip spaces from empty context lines' '
> +	test_write_lines e n q | git add -p 2>error &&
> +	test_must_be_empty error &&
> +	git diff >output &&
> +	diff_cmp expected output
> +'
> +
> test_expect_success 'skip files similarly as commit -a' '
> 	git reset &&
> 	echo file >.gitignore &&
> -- 
> 2.17.0
>

Junio C Hamano· Jul 11, 2018, 20:50 UTC · re: Jeff Felchner · lore

Re: [PATCH v2] add -p: fix counting empty context lines in edited patches

Jeff Felchner <jfelchner1@gmail.com> writes:

> Hey all, I assumed this was going to be in 2.18, but I'm still having the same issue.  What's the plan for release of this?

You assumed wrong ;-) A patch written on June 11th that is already deep into pre-release freeze, unless it is about fixing a regression during the same cycle, would never be in the release tagged on 21st.

It is already a part of the 'master' branch after v2.18, so v2.19 would be the first feature release that would see it (unless we discover problems in that change and need to revert it, that is).