threads / patch / 19367

patchnew test fails "add -p" for adds on the top line

Subject: [PATCH] new test fails "add -p" for adds on the top line

## tl;dr

8 messages between May 16, 2009 and May 16, 2009. Diffs are folded; open one to read it.

replies: 7people: 5as markdown or json

Matt Graham· May 16, 2009, 03:10 UTC · lore

add -p doesn't work for some diffs. diffs adding a new line at the top of the file with other adds later in the file are one way to trigger the problem.

during add -p, split the diff and then answer y for all segments. the file won't have been added to the index.

Signed-off-by: Matthew Graham <mdg149@gmail.com>
---
 t/t3701-add-interactive.sh |   32 ++++++++++++++++++++++++++++++++
 1 files changed, 32 insertions(+), 0 deletions(-)
Show changes to t/t3701-add-interactive.sh +30 −0
diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
index dfc6560..45da6c8 100755
--- a/t/t3701-add-interactive.sh
+++ b/t/t3701-add-interactive.sh
@@ -163,6 +163,38 @@ test_expect_success FILEMODE 'stage mode but not hunk' '
 	git diff          file | grep "+content"
 '

+# Write the patch file with a new line at the top and bottom
+cat >patch <<EOF
+index 180b47c..b6f2c08 100644
+--- a/file
++++ b/file
+@@ -1,2 +1,4 @@
++firstline
+ baseline
+ content
++lastline
+EOF
+# Expected output, similar to the patch but w/ diff at the top
+cat >expected <<EOF
+diff --git a/file b/file
+index b6f2c08..61b9053 100755
+--- a/file
++++ b/file
+@@ -1,2 +1,4 @@
++firstline
+ baseline
+ content
++lastline
+EOF
+# Test splitting the first patch, then adding both
+test_expect_failure 'add first line works' '
+	git commit -am "clear local changes" &&
+	git apply patch &&
+	(echo s; echo y; echo y) | git add -p file &&
+	git diff --cached > diff &&
+	test_cmp expected diff
+'
+
 # end of tests disabled when filemode is not usable

 test_done
-- 
1.6.3.9.g6345
Nanako Shiraishi· May 16, 2009, 10:25 UTC · re: Matt Graham · lore

Re: [PATCH] new test fails "add -p" for adds on the top line

Quoting Matt Graham <mdg149@gmail.com>:
Show 7 quoted lines
> add -p doesn't work for some diffs.  diffs adding a new line at the top of
> the file with other adds later in the file are one way to trigger the problem.
>
> during add -p, split the diff and then answer y for all segments.  the file
> won't have been added to the index.
>
> Signed-off-by: Matthew Graham <mdg149@gmail.com>
I tried "git-add -p" from different versions and I found out that versions before the commit 0beee4c6dec15292415e3d56075c16a76a22af54 doesn't have this problem.
commit 0beee4c6dec15292415e3d56075c16a76a22af54
Author: Thomas Rast <trast@student.ethz.ch>
Date:   Wed Jul 2 23:59:44 2008 +0200
    git-add--interactive: remove hunk coalescing
    
    Current git-apply has no trouble at all applying chunks that have
    overlapping context, as produced by the splitting feature. So we can
    drop the manual coalescing.
    
    Signed-off-by: Thomas Rast <trast@student.ethz.ch>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>
-- 
Nanako Shiraishi
http://ivory.ap.teacup.com/nanako3/
Thomas Rast· May 16, 2009, 14:12 UTC · re: Nanako Shiraishi · lore

Re: [PATCH] new test fails "add -p" for adds on the top line

Nanako Shiraishi wrote:
Show 24 quoted lines
> Quoting Matt Graham <mdg149@gmail.com>:
> 
> > add -p doesn't work for some diffs.  diffs adding a new line at the top of
> > the file with other adds later in the file are one way to trigger the problem.
> >
> > during add -p, split the diff and then answer y for all segments.  the file
> > won't have been added to the index.
> >
> > Signed-off-by: Matthew Graham <mdg149@gmail.com>
> 
> I tried "git-add -p" from different versions and I found out that versions before the commit 0beee4c6dec15292415e3d56075c16a76a22af54 doesn't have this problem.
> 
> commit 0beee4c6dec15292415e3d56075c16a76a22af54
> Author: Thomas Rast <trast@student.ethz.ch>
> Date:   Wed Jul 2 23:59:44 2008 +0200
> 
>     git-add--interactive: remove hunk coalescing
>     
>     Current git-apply has no trouble at all applying chunks that have
>     overlapping context, as produced by the splitting feature. So we can
>     drop the manual coalescing.
>     
>     Signed-off-by: Thomas Rast <trast@student.ethz.ch>
>     Signed-off-by: Junio C Hamano <gitster@pobox.com>

The above commit still reverts cleanly, but AFAICS merge_hunk blindly trusts the hunk headers, an assumption that is no longer valid due to the 'edit' feature. So either we need to recount the hunk headers prior to merging (which was rejected back in the 'edit' feature discussion due to code complexity) or find some other solution.

Passing either --unidiff-zero or -C1 with the failing patch fixes the problem, but oddly (to me at least) -C2 does not. The generated error looks like

  $ git apply --check -v -C2 < patch
  Checking patch file...
  error: while searching for:
  baseline
  content
  error: patch failed: file:1
  error: file: patch does not apply
The corresponding call (builtin-apply.c:2093) is
			error("while searching for:\n%.*s",
			      (int)(old - oldlines), oldlines);

so it does not seem to insert the extra newline. Is it actually looking for a blank line in the context? If so, wouldn't that be a git-apply bug?

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Junio C Hamano· May 16, 2009, 17:48 UTC · re: Thomas Rast · lore

Re: [PATCH] new test fails "add -p" for adds on the top line

Thomas Rast <trast@student.ethz.ch> writes:
Show 29 quoted lines
> Nanako Shiraishi wrote:
>> Quoting Matt Graham <mdg149@gmail.com>:
>> 
>> > add -p doesn't work for some diffs.  diffs adding a new line at the top of
>> > the file with other adds later in the file are one way to trigger the problem.
>> >
>> > during add -p, split the diff and then answer y for all segments.  the file
>> > won't have been added to the index.
>> >
>> > Signed-off-by: Matthew Graham <mdg149@gmail.com>
>> 
>> I tried "git-add -p" from different versions and I found out that versions before the commit 0beee4c6dec15292415e3d56075c16a76a22af54 doesn't have this problem.
>> 
>> commit 0beee4c6dec15292415e3d56075c16a76a22af54
>> Author: Thomas Rast <trast@student.ethz.ch>
>> Date:   Wed Jul 2 23:59:44 2008 +0200
>> 
>>     git-add--interactive: remove hunk coalescing
>>     
>>     Current git-apply has no trouble at all applying chunks that have
>>     overlapping context, as produced by the splitting feature. So we can
>>     drop the manual coalescing.
>>     
>>     Signed-off-by: Thomas Rast <trast@student.ethz.ch>
>>     Signed-off-by: Junio C Hamano <gitster@pobox.com>
>
> The above commit still reverts cleanly, but AFAICS merge_hunk blindly
> trusts the hunk headers, an assumption that is no longer valid due to
> the 'edit' feature.
Heh, here is my "I told you so" moment ;-).

We could also remove that "edit" thing; I do not use nor trust it (fundamentally you cannot trust it).

But how about doing it this way?
-- >8 --
Subject: Revert "git-add--interactive: remove hunk coalescing"
This reverts commit 0beee4c6dec15292415e3d56075c16a76a22af54 but with a
bit of twist, as we have added "edit hunk manually" hack and we cannot
rely on the original line numbers of the hunks that were manually edited.
---
 git-add--interactive.perl  |   96 +++++++++++++++++++++++++++++++++++++++++++-
 t/t3701-add-interactive.sh |    2 +-
 2 files changed, 96 insertions(+), 2 deletions(-)
Show changes to 2 files +96 −2

git-add--interactive.perl, t/t3701-add-interactive.sh

diff --git a/git-add--interactive.perl b/git-add--interactive.perl
index f6e536e..a06172c 100755
--- a/git-add--interactive.perl
+++ b/git-add--interactive.perl
@@ -767,6 +767,96 @@ sub split_hunk {
 	return @split;
 }
 
+sub find_last_o_ctx {
+	my ($it) = @_;
+	my $text = $it->{TEXT};
+	my ($o_ofs, $o_cnt) = parse_hunk_header($text->[0]);
+	my $i = @{$text};
+	my $last_o_ctx = $o_ofs + $o_cnt;
+	while (0 < --$i) {
+		my $line = $text->[$i];
+		if ($line =~ /^ /) {
+			$last_o_ctx--;
+			next;
+		}
+		last;
+	}
+	return $last_o_ctx;
+}
+
+sub merge_hunk {
+	my ($prev, $this) = @_;
+	my ($o0_ofs, $o0_cnt, $n0_ofs, $n0_cnt) =
+	    parse_hunk_header($prev->{TEXT}[0]);
+	my ($o1_ofs, $o1_cnt, $n1_ofs, $n1_cnt) =
+	    parse_hunk_header($this->{TEXT}[0]);
+
+	my (@line, $i, $ofs, $o_cnt, $n_cnt);
+	$ofs = $o0_ofs;
+	$o_cnt = $n_cnt = 0;
+	for ($i = 1; $i < @{$prev->{TEXT}}; $i++) {
+		my $line = $prev->{TEXT}[$i];
+		if ($line =~ /^\+/) {
+			$n_cnt++;
+			push @line, $line;
+			next;
+		}
+
+		last if ($o1_ofs <= $ofs);
+
+		$o_cnt++;
+		$ofs++;
+		if ($line =~ /^ /) {
+			$n_cnt++;
+		}
+		push @line, $line;
+	}
+
+	for ($i = 1; $i < @{$this->{TEXT}}; $i++) {
+		my $line = $this->{TEXT}[$i];
+		if ($line =~ /^\+/) {
+			$n_cnt++;
+			push @line, $line;
+			next;
+		}
+		$ofs++;
+		$o_cnt++;
+		if ($line =~ /^ /) {
+			$n_cnt++;
+		}
+		push @line, $line;
+	}
+	my $head = ("@@ -$o0_ofs" .
+		    (($o_cnt != 1) ? ",$o_cnt" : '') .
+		    " +$n0_ofs" .
+		    (($n_cnt != 1) ? ",$n_cnt" : '') .
+		    " @@\n");
+	@{$prev->{TEXT}} = ($head, @line);
+}
+
+sub coalesce_overlapping_hunks {
+	my (@in) = @_;
+	my @out = ();
+
+	my ($last_o_ctx, $last_was_dirty);
+
+	for (grep { $_->{USE} } @in) {
+		my $text = $_->{TEXT};
+		my ($o_ofs) = parse_hunk_header($text->[0]);
+		if (defined $last_o_ctx &&
+		    $o_ofs <= $last_o_ctx &&
+		    !$_->{DIRTY} &&
+		    !$last_was_dirty) {
+			merge_hunk($out[-1], $_);
+		}
+		else {
+			push @out, $_;
+		}
+		$last_o_ctx = find_last_o_ctx($out[-1]);
+		$last_was_dirty = $_->{DIRTY};
+	}
+	return @out;
+}
 
 sub color_diff {
 	return map {
@@ -878,7 +968,8 @@ sub edit_hunk_loop {
 		my $newhunk = {
 			TEXT => $text,
 			TYPE => $hunk->[$ix]->{TYPE},
-			USE => 1
+			USE => 1,
+			DIRTY => 1,
 		};
 		if (diff_applies($head,
 				 @{$hunk}[0..$ix-1],
@@ -1210,6 +1301,8 @@ sub patch_update_file {
 		}
 	}
 
+	@hunk = coalesce_overlapping_hunks(@hunk);
+
 	my $n_lofs = 0;
 	my @result = ();
 	for (@hunk) {
@@ -1224,6 +1317,7 @@ sub patch_update_file {
 		open $fh, '| git apply --cached --recount';
 		for (@{$head->{TEXT}}, @result) {
 			print $fh $_;
+			print STDERR $_;
 		}
 		if (!close $fh) {
 			for (@{$head->{TEXT}}, @result) {
diff --git a/t/t3701-add-interactive.sh b/t/t3701-add-interactive.sh
index 45da6c8..c5220a1 100755
--- a/t/t3701-add-interactive.sh
+++ b/t/t3701-add-interactive.sh
@@ -187,7 +187,7 @@ index b6f2c08..61b9053 100755
 +lastline
 EOF
 # Test splitting the first patch, then adding both
-test_expect_failure 'add first line works' '
+test_expect_success 'add first line works' '
 	git commit -am "clear local changes" &&
 	git apply patch &&
 	(echo s; echo y; echo y) | git add -p file &&
-- 
1.6.3.1.9.g95405b
Sverre Rabbelier· May 16, 2009, 17:55 UTC · re: Junio C Hamano · lore

Re: [PATCH] new test fails "add -p" for adds on the top line

Heya,
On Sat, May 16, 2009 at 19:48, Junio C Hamano <gitster@pobox.com> wrote:
> We could also remove that "edit" thing; I do not use nor trust it
> (fundamentally you cannot trust it).

But it is very useful to split up patches! Of course, I always verify the end result (usually by diffing against HEAD@{...}), but it is definitely a very useful tool that I would hate to see removed :(.

-- 
Cheers,

Sverre Rabbelier
Junio C Hamano· May 16, 2009, 19:13 UTC · re: Sverre Rabbelier · lore

Re: [PATCH] new test fails "add -p" for adds on the top line

Sverre Rabbelier <srabbelier@gmail.com> writes:
Show 9 quoted lines
> Heya,
>
> On Sat, May 16, 2009 at 19:48, Junio C Hamano <gitster@pobox.com> wrote:
>> We could also remove that "edit" thing; I do not use nor trust it
>> (fundamentally you cannot trust it).
>
> But it is very useful to split up patches! Of course, I always verify
> the end result (usually by diffing against HEAD@{...}), but it is
> definitely a very useful tool that I would hate to see removed :(.
Sorry, forgot a smiley ;-).
Sverre Rabbelier· May 16, 2009, 19:14 UTC · re: Junio C Hamano · lore

Re: [PATCH] new test fails "add -p" for adds on the top line

Heya,
On Sat, May 16, 2009 at 21:13, Junio C Hamano <gitster@pobox.com> wrote:
> Sorry, forgot a smiley ;-).
Phew, what would be without those :).
-- 
Cheers,

Sverre Rabbelier
Junio C Hamano· May 16, 2009, 19:51 UTC · re: Junio C Hamano · lore

Re: [PATCH] new test fails "add -p" for adds on the top line

Junio C Hamano <gitster@pobox.com> writes:
Show 7 quoted lines
> Thomas Rast <trast@student.ethz.ch> writes:
> ...
>> The above commit still reverts cleanly, but AFAICS merge_hunk blindly
>> trusts the hunk headers, an assumption that is no longer valid due to
>> the 'edit' feature.
>
> Heh, here is my "I told you so" moment ;-).

It never blindly trusted before the edit 'feature'; it counted carefully and it could do so because it had all the necessary information.

I told you that 'edit' could remember the line offset and line numbers before giving the buffer to the end user, and then recount and adjust the count after getting the edited results back, to update the offset and count with the same carefulness. You (and I think there was somebody else who was helping) didn't listen.

Fundamentally, after you remove some hunks (and worse yet, you modify some) from the patch and feed that to "git apply --recount", it can never do as thorough a job as you could do inside "add -p" itself. The latter has more information (the omitted hunks, and the hunks before/after the user edited) necessary to reconstruct the line numbers and hunk size. To keep the whole process more robust and trustworthy, you must do the necessary computation while you still have all the information about the hunks you are not feeding to the downstream.

That was what the "I told you so" was about in my message.

It is not too late to teach the 'edit hack' to do so. That would allow us to remove the "$_->{DIRTY}" bit my "how about this" patch adds, and I'll stop calling it the 'edit hack' and start calling it the 'edit feature' when that happens ;-).

But at least the "how about this" patch should restore the original behaviour as long as the user does not use the 'edit hack' for now.

← back to recent threads