{"thread":{"id":"62044","subject":"word-diff-regex=. sometimes ignores newlines","startedAt":"2024-09-03T12:30:47Z","lastAt":"2024-09-06T10:08:17Z","messageCount":2,"participants":["高橋全 (Tamo)","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"502034","messageId":"CABiJAjbEpYkcrxj82uQ=O27tR9fKoUFH0=MOCobDfa9cWsbdAA@mail.gmail.com","threadId":"62044","inReplyTo":null,"subject":"word-diff-regex=. sometimes ignores newlines","fromName":"高橋全 (Tamo)","fromEmail":"ttakah@gmail.com","sentAt":"2024-09-03T12:30:35Z","receivedAt":"2024-09-03T12:30:47Z","isPatch":false,"sender":{"key":"ttakah@gmail.com","avatar":null},"body":"Thank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\nmkdir test\ncd test\ngit init\n\ncat >a.txt <<EOF\nNRZ /NZRQ/NBRQ/\nNRZ(C) /NZRCQ/\nNRZ(M) /NZRMQ/\nEOF\n\ngit add a.txt\ngit commit -m 1\n\ncat >a.txt <<EOF\nNRZ /NZRMQ/NZRCQ/NZRQ/NBRQ/\nEOF\n\ngit diff --word-diff-regex=.\n\n\nWhat did you expect to happen? (Expected behavior)\n\ndiff --git a/a.txt b/a.txt\nindex 278ea76..7e6f42f 100644\n--- a/a.txt\n+++ b/a.txt\n@@ -1,3 +1 @@\nNRZ /NZR{+M+}Q/N[-BRQ/-]{+ZRCQ/NZRQ/NBRQ/+}\n[-NRZ(C) /NZRCQ/-]\n[-NRZ(M) /NZRMQ/-]\n\nor anything whose hunk has three lines\n\n\nWhat happened instead? (Actual behavior)\n\ndiff --git a/a.txt b/a.txt\nindex 278ea76..7e6f42f 100644\n--- a/a.txt\n+++ b/a.txt\n@@ -1,3 +1 @@\nNRZ /NZR{+M+}Q/N[-BRQ/-]\n[-NRZ(C) /N-]ZRCQ/N[-R-]Z[-(M) -]{+RQ+}/N[-Z-]{+B+}R[-M-]Q/\n\n\n\nWhat's different between what you expected and what actually happened?\n\nsome newlines are ignored\nand the length of the hunk is wrong;\ngit says \"@@ -1,3 +1 @@\" but the hunk has only 2 lines\n\n\n\nAnything else you want to add:\n\nthis may be clearer with --word-diff=porcelain\nbecause it doesn't emit enough number of \"~\"s\n\n\n```\n$ git diff --word-diff-regex=. --word-diff=porcelain\ndiff --git a/a.txt b/a.txt\nindex 278ea76..7e6f42f 100644\n--- a/a.txt\n+++ b/a.txt\n@@ -1,3 +1 @@\n NRZ /NZR\n+M\n Q/N\n-BRQ/\n~\n-NRZ(C) /N\n ZRCQ/N\n-R\n Z\n-(M)\n+RQ\n /N\n-Z\n+B\n R\n-M\n Q/\n~\n```\n\nthis should emit 3 tildes\n\n\n\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n\n[System Info]\ngit version:\ngit version 2.46.0\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nlibcurl: 7.68.0\nzlib: 1.2.11\nuname: Linux 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29\n23:14:13 UTC 2024 x86_64\ncompiler info: gnuc: 9.4\nlibc info: glibc: 2.31\n$SHELL (typically, interactive shell): /bin/bash\n\n\n[Enabled Hooks]\n"},{"id":"502317","messageId":"d0686080-6f7c-8128-dc24-ecafc13e0120@gmx.de","threadId":"62044","inReplyTo":"CABiJAjbEpYkcrxj82uQ=O27tR9fKoUFH0=MOCobDfa9cWsbdAA@mail.gmail.com","subject":"Re: word-diff-regex=. sometimes ignores newlines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2024-09-06T10:08:14Z","receivedAt":"2024-09-06T10:08:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Tamo,\n\nOn Tue, 3 Sep 2024, 高橋全 (Tamo) wrote:\n\n> What did you do before the bug happened? (Steps to reproduce your issue)\n>\n> mkdir test\n> cd test\n> git init\n>\n> cat >a.txt <<EOF\n> NRZ /NZRQ/NBRQ/\n> NRZ(C) /NZRCQ/\n> NRZ(M) /NZRMQ/\n> EOF\n>\n> git add a.txt\n> git commit -m 1\n>\n> cat >a.txt <<EOF\n> NRZ /NZRMQ/NZRCQ/NZRQ/NBRQ/\n> EOF\n>\n> git diff --word-diff-regex=.\n>\n>\n> What did you expect to happen? (Expected behavior)\n>\n> diff --git a/a.txt b/a.txt\n> index 278ea76..7e6f42f 100644\n> --- a/a.txt\n> +++ b/a.txt\n> @@ -1,3 +1 @@\n> NRZ /NZR{+M+}Q/N[-BRQ/-]{+ZRCQ/NZRQ/NBRQ/+}\n> [-NRZ(C) /NZRCQ/-]\n> [-NRZ(M) /NZRMQ/-]\n>\n> or anything whose hunk has three lines\n>\n>\n> What happened instead? (Actual behavior)\n>\n> diff --git a/a.txt b/a.txt\n> index 278ea76..7e6f42f 100644\n> --- a/a.txt\n> +++ b/a.txt\n> @@ -1,3 +1 @@\n> NRZ /NZR{+M+}Q/N[-BRQ/-]\n> [-NRZ(C) /N-]ZRCQ/N[-R-]Z[-(M) -]{+RQ+}/N[-Z-]{+B+}R[-M-]Q/\n>\n>\n>\n> What's different between what you expected and what actually happened?\n>\n> some newlines are ignored\n> and the length of the hunk is wrong;\n> git says \"@@ -1,3 +1 @@\" but the hunk has only 2 lines\n\nThe reason is the regular expression, which does not match newlines. See\nhttps://github.com/git/git/blob/v2.46.0/diff.c#L2268-L2270, which shows\nhow the regular expression is compiled:\n\n\t\tif (regcomp(ecbdata->diff_words->word_regex,\n\t\t\t    o->word_regex,\n\t\t\t    REG_EXTENDED | REG_NEWLINE))\n\nNote the flag `REG_NEWLINE`, described in detail at\nhttps://pubs.opengroup.org/onlinepubs/9699919799/functions/regcomp.html:\n\n\tIf REG_NEWLINE is set, then <newline> shall be treated as an\n\tordinary character except as follows:\n\n\t1. A <newline> in string shall not be matched by a <period>\n\t   outside a bracket expression or by any form of a non-matching\n\t   list (see XBD Regular Expressions).\n\nYou will note that you can see three lines in the output when using\n`--word-diff-regex='[^ \\t\\n]+|[ \\t\\n]+'`:\n\n\t$ git diff --word-diff-regex='[^ \\t\\n]+|[ \\t\\n]+'\n\tdiff --git a/a.txt b/a.txt\n\tindex 278ea76..7e6f42f 100644\n\t--- a/a.txt\n\t+++ b/a.txt\n\t@@ -1,3 +1 @@\n\tNRZ [-/NZRQ/NBRQ/-]\n\t[-NRZ(C) /NZRCQ/-]\n\t[-NRZ(M) /NZRMQ/-]{+/NZRMQ/NZRCQ/NZRQ/NBRQ/+}\n\nHowever, when including the slash in the boundary characters, the newlines\nare suppressed again:\n\n\t$ git diff --word-diff-regex='[^/ \\t\\n]+|[/ \\t\\n]+'\n\tdiff --git a/a.txt b/a.txt\n\tindex 278ea76..7e6f42f 100644\n\t--- a/a.txt\n\t+++ b/a.txt\n\t@@ -1,3 +1 @@\n\tNRZ /[-NZRQ/NBRQ-]{+NZRMQ+}/[-NRZ(C) /-]NZRCQ/[-NRZ(M) /NZRMQ-]{+NZRQ/NBRQ+}/\n\nI am fairly convinced that the reason for this behavior is that the word\ndiff machinery special-cases newlines and _never_ makes them part of the\n\"words\", see https://github.com/git/git/blob/v2.46.0/diff.c#L2072-L2074\nfor the code implementing that logic.\n\nNow, is this a bug? I can't really say. From my perspective, it is not:\nWhen I implemented the original version of the word diff code, my use case\nwas LaTeX-formatted scientific articles, which traditionally do not\ncontain newline characters within paragraphs. I still have a hard time\nwrapping my head around use cases where any pattern that includes a\nnewline would match a what is considered a word.\n\nI do remember how I struggled (and punted) on the question how to display\nnewlines in word diffs. There just is no good way to do it that would\naddress all valid scenarios.\n\nCiao,\nJohannes\n"}]}