{"thread":{"id":"58619","subject":"Git diff misattributes the first word of a line to the previous line","startedAt":"2022-10-13T05:51:53Z","lastAt":"2022-10-13T12:16:14Z","messageCount":4,"participants":["Gurjeet Singh","Johannes Sixt","Philip Oakley","Bagas Sanjaya"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"464792","messageId":"CABwTF4U-KXHF7=8RWY7Ecbspz205Msa3syZFiWYDg3XmZsNGVw@mail.gmail.com","threadId":"58619","inReplyTo":null,"subject":"Git diff misattributes the first word of a line to the previous line","fromName":"Gurjeet Singh","fromEmail":"gurjeet@singh.im","sentAt":"2022-10-13T05:51:31Z","receivedAt":"2022-10-13T05:51:53Z","isPatch":false,"sender":{"key":"gurjeet@singh.im","avatar":null},"body":"Git diff seems to get confused about word boundaries, and includes the\nfirst word from the next line.\n\nIn the output below, you'd be forgiven to assume that the word 'ab'\nwas removed from first line, and was replaced with the word 'opt1'.\nBut as you can see in the contents of file '1.txt', that word was on\nthe _second_ line.\n\nIt seems that the first word of a line gets attributed to the previous\nline, ignoring the fact that there's an intervening newline before the\nword.\n\nThis confusion is exhibited by the --color-words option, too, where I\ndiscovered it first. But when trying to create a reproducible test\ncase, I discovered that this may be a problem with the word-boundary\nidentification, and the --word-diff=plain is a better way to\ndemonstrate the bug.\n\nI have also eliminated the possibility that this may be due to some\nmisconfiguration in my GIt config files, by setting some environment\nvariables, as seen in the last command.\n\n$ git diff --word-diff=plain /tmp/1.txt /tmp/2.txt\ndiff --git a/tmp/1.txt b/tmp/2.txt\nindex 8239f93..099fb80 100644\n--- a/tmp/1.txt\n+++ b/tmp/2.txt\n@@ -1,2 +1,2 @@\n    x = yz [-ab-]{+opt1+}\n{+    ac+} = [-cd ef-]{+pq opt2+}\n\n$ cat /tmp/1.txt\n    x = yz\n    ab = cd ef\n\n$ cat /tmp/2.txt\n    x = yz opt1\n    ac = pq opt2\n\n# Git installed on macOS, via Nixpkgs\n$ git --version\ngit version 2.35.1\n\n# Also tested on Git installed via Homebrew\n$ /usr/local/bin/git --version\ngit version 2.38.0\n\n# Try to run with a clean environment.\n$ export GIT_CONFIG_GLOBAL=/dev/null\n$ export GIT_CONFIG_SYSTEM=/dev/null\n$ export GIT_CONFIG_NOSYSTEM=yes\n$ git diff --word-diff=plain /tmp/1.txt /tmp/2.txt # Same buggy output\n\n# **** Expected **** output\n$ git diff --word-diff=plain /tmp/1.txt /tmp/2.txt\ndiff --git a/tmp/1.txt b/tmp/2.txt\nindex 8239f93..099fb80 100644\n--- a/tmp/1.txt\n+++ b/tmp/2.txt\n@@ -1,2 +1,2 @@\n    x = yz {+opt1+}\n    [-ab-]{+ac+} = [-cd ef-]{+pq opt2+}\n\nBest regards,\nGurjeet\nhttp://Gurje.et\n"},{"id":"464799","messageId":"f2453084-9f9a-74a1-6a43-0c1b91c95e93@kdbg.org","threadId":"58619","inReplyTo":"CABwTF4U-KXHF7=8RWY7Ecbspz205Msa3syZFiWYDg3XmZsNGVw@mail.gmail.com","subject":"Re: Git diff misattributes the first word of a line to the previous line","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2022-10-13T06:45:55Z","receivedAt":"2022-10-13T06:46:02Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 13.10.22 um 07:51 schrieb Gurjeet Singh:\n> Git diff seems to get confused about word boundaries, and includes the\n> first word from the next line.\n\nNo, that would misattribute the perceived malfunction.\n\n> It seems that the first word of a line gets attributed to the previous\n> line, ignoring the fact that there's an intervening newline before the\n> word.\n> [...]\n> $ git diff --word-diff=plain /tmp/1.txt /tmp/2.txt\n> diff --git a/tmp/1.txt b/tmp/2.txt\n> index 8239f93..099fb80 100644\n> --- a/tmp/1.txt\n> +++ b/tmp/2.txt\n> @@ -1,2 +1,2 @@\n>     x = yz [-ab-]{+opt1+}\n> {+    ac+} = [-cd ef-]{+pq opt2+}\n> \n> $ cat /tmp/1.txt\n>     x = yz\n>     ab = cd ef\n> \n> $ cat /tmp/2.txt\n>     x = yz opt1\n>     ac = pq opt2\n\nThe reason for this is that the implementation of word-diff does not\ntreat newline characters in any special way. They are treated as\n\"whitespace\" like any other character that is not captured by the\nword-diff patterns. Whitespace characters following each word are\nrecorded, but are disregarded when the word-diff is computed. When the\ntext is reconstructed in the output, these recorded space characters are\nprinted only for unchanged and added words, but are not printed for\nremoved words (IIRC). Combine this with the fact that when there is a\nchange, i.e., a combination of removal and addition, then the removal is\nprinted before the addition, and you get the observed output.\n\nI don't see an easy solution for this without completely rewriting the\nimplementation.\n\n-- Hannes\n\n"},{"id":"464820","messageId":"4cb7d3cb-36ff-0302-45c0-aa61644ef90b@iee.email","threadId":"58619","inReplyTo":"f2453084-9f9a-74a1-6a43-0c1b91c95e93@kdbg.org","subject":"Re: Git diff misattributes the first word of a line to the previous line","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2022-10-13T11:30:03Z","receivedAt":"2022-10-13T11:30:11Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Gurjeet,\n\nOn 13/10/2022 07:45, Johannes Sixt wrote:\n> Am 13.10.22 um 07:51 schrieb Gurjeet Singh:\n>> Git diff seems to get confused about word boundaries, and includes the\n>> first word from the next line.\n> No, that would misattribute the perceived malfunction.\n>\n>> It seems that the first word of a line gets attributed to the previous\n>> line, ignoring the fact that there's an intervening newline before the\n>> word.\n\nGiven that this effect is a part of the design (LF => whitespace), are\nthere any changes to the *documentation* that could be made to help\nclarify this? E.g. looking back (you did check the manual? ;-) did you\nmiss some aspect in the man pages that could have been more prominent,\nplaced earlier, or clarified?\n\nWhy is this way of reporting even expected (e.g. confusion between\nflowed text, and line oriented code, without a mode change), etc. ?\n\nAny other `retrospective` thoughts that could help?\n\n--\nPhilip\n\n>> [...]\n>> $ git diff --word-diff=plain /tmp/1.txt /tmp/2.txt\n>> diff --git a/tmp/1.txt b/tmp/2.txt\n>> index 8239f93..099fb80 100644\n>> --- a/tmp/1.txt\n>> +++ b/tmp/2.txt\n>> @@ -1,2 +1,2 @@\n>>     x = yz [-ab-]{+opt1+}\n>> {+    ac+} = [-cd ef-]{+pq opt2+}\n>>\n>> $ cat /tmp/1.txt\n>>     x = yz\n>>     ab = cd ef\n>>\n>> $ cat /tmp/2.txt\n>>     x = yz opt1\n>>     ac = pq opt2\n> The reason for this is that the implementation of word-diff does not\n> treat newline characters in any special way. They are treated as\n> \"whitespace\" like any other character that is not captured by the\n> word-diff patterns. Whitespace characters following each word are\n> recorded, but are disregarded when the word-diff is computed. When the\n> text is reconstructed in the output, these recorded space characters are\n> printed only for unchanged and added words, but are not printed for\n> removed words (IIRC). Combine this with the fact that when there is a\n> change, i.e., a combination of removal and addition, then the removal is\n> printed before the addition, and you get the observed output.\n>\n> I don't see an easy solution for this without completely rewriting the\n> implementation.\n>\n> -- Hannes\n>\n\n"},{"id":"464825","messageId":"1ebea198-fa2a-3f79-e698-64fa4799bcde@gmail.com","threadId":"58619","inReplyTo":"CABwTF4U-KXHF7=8RWY7Ecbspz205Msa3syZFiWYDg3XmZsNGVw@mail.gmail.com","subject":"Re: Git diff misattributes the first word of a line to the previous line","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2022-10-13T12:16:05Z","receivedAt":"2022-10-13T12:16:14Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 10/13/22 12:51, Gurjeet Singh wrote:\n> # **** Expected **** output\n> $ git diff --word-diff=plain /tmp/1.txt /tmp/2.txt\n> diff --git a/tmp/1.txt b/tmp/2.txt\n> index 8239f93..099fb80 100644\n> --- a/tmp/1.txt\n> +++ b/tmp/2.txt\n> @@ -1,2 +1,2 @@\n>     x = yz {+opt1+}\n>     [-ab-]{+ac+} = [-cd ef-]{+pq opt2+}\n> \n\nWhat Git version (and on what system) you made the expected diff above?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n\n"}]}