word-diff-regex=. sometimes ignores newlines

高橋全 (Tamo)· Sep 3, 2024, 12:30 UTC · lore

Thank you for filling out a Git bug report! Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

mkdir test cd test git init

cat >a.txt <<EOF NRZ /NZRQ/NBRQ/ NRZ(C) /NZRCQ/ NRZ(M) /NZRMQ/ EOF

git add a.txt git commit -m 1

cat >a.txt <<EOF NRZ /NZRMQ/NZRCQ/NZRQ/NBRQ/ EOF

git diff --word-diff-regex=.

What did you expect to happen? (Expected behavior)

diff --git a/a.txt b/a.txt
index 278ea76..7e6f42f 100644
--- a/a.txt
+++ b/a.txt
@@ -1,3 +1 @@
NRZ /NZR{+M+}Q/N[-BRQ/-]{+ZRCQ/NZRQ/NBRQ/+}
[-NRZ(C) /NZRCQ/-]
[-NRZ(M) /NZRMQ/-]

or anything whose hunk has three lines


What happened instead? (Actual behavior)

diff --git a/a.txt b/a.txt
index 278ea76..7e6f42f 100644
--- a/a.txt
+++ b/a.txt
@@ -1,3 +1 @@
NRZ /NZR{+M+}Q/N[-BRQ/-]
[-NRZ(C) /N-]ZRCQ/N[-R-]Z[-(M) -]{+RQ+}/N[-Z-]{+B+}R[-M-]Q/



What's different between what you expected and what actually happened?

some newlines are ignored
and the length of the hunk is wrong;
git says "@@ -1,3 +1 @@" but the hunk has only 2 lines



Anything else you want to add:

this may be clearer with --word-diff=porcelain
because it doesn't emit enough number of "~"s


```
$ git diff --word-diff-regex=. --word-diff=porcelain
diff --git a/a.txt b/a.txt
index 278ea76..7e6f42f 100644
--- a/a.txt
+++ b/a.txt
@@ -1,3 +1 @@
 NRZ /NZR
+M
 Q/N
-BRQ/
~
-NRZ(C) /N
 ZRCQ/N
-R
 Z
-(M)
+RQ
 /N
-Z
+B
 R
-M
 Q/
~
```

this should emit 3 tildes




Please review the rest of the bug report below.
You can delete any lines you don't wish to share.


[System Info]
git version:
git version 2.46.0
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
libcurl: 7.68.0
zlib: 1.2.11
uname: Linux 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29
23:14:13 UTC 2024 x86_64
compiler info: gnuc: 9.4
libc info: glibc: 2.31
$SHELL (typically, interactive shell): /bin/bash


[Enabled Hooks]

Johannes Schindelin· Sep 6, 2024, 10:08 UTC · re: 高橋全 (Tamo) · lore

Re: word-diff-regex=. sometimes ignores newlines

Hi Tamo,

On Tue, 3 Sep 2024, 高橋全 (Tamo) wrote:

Show 53 quoted lines

> What did you do before the bug happened? (Steps to reproduce your issue)
>
> mkdir test
> cd test
> git init
>
> cat >a.txt <<EOF
> NRZ /NZRQ/NBRQ/
> NRZ(C) /NZRCQ/
> NRZ(M) /NZRMQ/
> EOF
>
> git add a.txt
> git commit -m 1
>
> cat >a.txt <<EOF
> NRZ /NZRMQ/NZRCQ/NZRQ/NBRQ/
> EOF
>
> git diff --word-diff-regex=.
>
>
> What did you expect to happen? (Expected behavior)
>
> diff --git a/a.txt b/a.txt
> index 278ea76..7e6f42f 100644
> --- a/a.txt
> +++ b/a.txt
> @@ -1,3 +1 @@
> NRZ /NZR{+M+}Q/N[-BRQ/-]{+ZRCQ/NZRQ/NBRQ/+}
> [-NRZ(C) /NZRCQ/-]
> [-NRZ(M) /NZRMQ/-]
>
> or anything whose hunk has three lines
>
>
> What happened instead? (Actual behavior)
>
> diff --git a/a.txt b/a.txt
> index 278ea76..7e6f42f 100644
> --- a/a.txt
> +++ b/a.txt
> @@ -1,3 +1 @@
> NRZ /NZR{+M+}Q/N[-BRQ/-]
> [-NRZ(C) /N-]ZRCQ/N[-R-]Z[-(M) -]{+RQ+}/N[-Z-]{+B+}R[-M-]Q/
>
>
>
> What's different between what you expected and what actually happened?
>
> some newlines are ignored
> and the length of the hunk is wrong;
> git says "@@ -1,3 +1 @@" but the hunk has only 2 lines

The reason is the regular expression, which does not match newlines. See https://github.com/git/git/blob/v2.46.0/diff.c#L2268-L2270, which shows how the regular expression is compiled:

		if (regcomp(ecbdata->diff_words->word_regex,
			    o->word_regex,
			    REG_EXTENDED | REG_NEWLINE))

Note the flag `REG_NEWLINE`, described in detail at https://pubs.opengroup.org/onlinepubs/9699919799/functions/regcomp.html:

	If REG_NEWLINE is set, then <newline> shall be treated as an
	ordinary character except as follows:

	1. A <newline> in string shall not be matched by a <period>
	   outside a bracket expression or by any form of a non-matching
	   list (see XBD Regular Expressions).

You will note that you can see three lines in the output when using `--word-diff-regex='[^ \t\n]+|[ \t\n]+'`:

	$ git diff --word-diff-regex='[^ \t\n]+|[ \t\n]+'
	diff --git a/a.txt b/a.txt
	index 278ea76..7e6f42f 100644
	--- a/a.txt
	+++ b/a.txt
	@@ -1,3 +1 @@
	NRZ [-/NZRQ/NBRQ/-]
	[-NRZ(C) /NZRCQ/-]
	[-NRZ(M) /NZRMQ/-]{+/NZRMQ/NZRCQ/NZRQ/NBRQ/+}

However, when including the slash in the boundary characters, the newlines are suppressed again:

	$ git diff --word-diff-regex='[^/ \t\n]+|[/ \t\n]+'
	diff --git a/a.txt b/a.txt
	index 278ea76..7e6f42f 100644
	--- a/a.txt
	+++ b/a.txt
	@@ -1,3 +1 @@
	NRZ /[-NZRQ/NBRQ-]{+NZRMQ+}/[-NRZ(C) /-]NZRCQ/[-NRZ(M) /NZRMQ-]{+NZRQ/NBRQ+}/

I am fairly convinced that the reason for this behavior is that the word diff machinery special-cases newlines and _never_ makes them part of the "words", see https://github.com/git/git/blob/v2.46.0/diff.c#L2072-L2074 for the code implementing that logic.

Now, is this a bug? I can't really say. From my perspective, it is not: When I implemented the original version of the word diff code, my use case was LaTeX-formatted scientific articles, which traditionally do not contain newline characters within paragraphs. I still have a hard time wrapping my head around use cases where any pattern that includes a newline would match a what is considered a word.

I do remember how I struggled (and punted) on the question how to display newlines in word diffs. There just is no good way to do it that would address all valid scenarios.

Ciao, Johannes