{"thread":{"id":"25674","subject":"Understanding and improving --word-diff","startedAt":"2010-11-08T15:16:01Z","lastAt":"2010-11-18T06:40:50Z","messageCount":10,"participants":["Matthijs Kooijman","Matthieu Moy","Thomas Rast","Wincent Colaiuta","Jeff King","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"155387","messageId":"20101108151601.GF22067@login.drsnuggles.stderr.nl","threadId":"25674","inReplyTo":null,"subject":"Understanding and improving --word-diff","fromName":"Matthijs Kooijman","fromEmail":"matthijs@stdin.nl","sentAt":"2010-11-08T15:16:01Z","receivedAt":"2010-11-08T15:16:01Z","isPatch":false,"sender":{"key":"matthijs@stdin.nl","avatar":"https://avatars.githubusercontent.com/u/194491?v=4"},"body":"Hi folks,\n\nI recently discovered --word-diff (or rather, --color-words and found\n--word-diff when I started to hack on the git master version) and I had\nhoped it would make the unified diffs generated by git-diff more\nreadable.\n\nMore specifically, I had expected to get a normal unified diff, with\ncolouring added to highlight the changes within the normal - and + lines\n(so you don't have to review the entire changed line to see that just a\nsingle word or character has changed).\n\nE.g., I would like to see:\n\n-a <r>b</r> c\n+a <g>x</g> c\n\nUnfortunately, all --word-diff types currently departs from line-based -\nand + lines and show the new version of the file with the changed words\n(both old and new versions) shown inline, marked with coloring or\n{- ...  -} kind of syntax. E.g., with --word-diff=color, the above would\nlook like:\n\na <r>b</r><g>x</g> c\n\nPersonally, I think that the first example above is easier to read than\nthe second one (at least for diffs of code).\n\nI was planning to let this mail be accompanied with a patch, so I've\nstarted hacking on this feature already. However, halfway through some\ncleanups and a prototype implementation of the above (breaking some of\nthe other --word-diff formats in the process), I found that the current\ngeneralization of the different styles as stored in diff_words_styles[]\ndoes not apply cleanly enough to my intended output format. While trying\nto extend this generalization to something that would fit, I found that\nI don't actually understand the rationale behind --word-diff and the\nformats well enough to find a proper implemenation (see the link at the\nbottom of this email for the unfinished code I hacked up until now).\n\nSo, here's some observations and questions about how --word-diff works\nor should work. Comments are welcome, both in general terms as well as\nin terms of the word-diff implementation (I know my way around there by\nnow).\n\nIntended use\n------------\nFirst of all, it seems that the main intended use of --word-diff is for\nLaTeX or HTML documents or similar, where blocks of running text might\nbe hard-wrapped (and thus rewrapped after a small change). In these\ncases, a small change in wording could cause a lot of whitespace to\nshift, resulting in a big normal diff. The current word-diff\nimplementation therefore simply does not show the whitespace (or rather,\nnon-word) changes, since they're usually not relevant to LaTeX anyway.\n\nIs this indeed the main usecase, or are there others I'm missing?\n\nInexact output\n--------------\nSecondly, the --word-diff output currently never displays any changes to\nthe non-word (whitespace) parts of a file. This makes sense for the\nLaTeX case, but sometimes you might want to get exact diff output\ninstead. At first glance this seems possible by specifiying a word-regex\nof \".\" or something similar (i.e., make sure that the word regex matches\neverything). But this is problematic for newlines. The documentation\nstates that stuff gets silently ignored if a newline ends up inside a\nword. For the --word-diff=color format, this is probably a fixed\nlimitation of the otput format: you can't give a color to a newline (or\na space, for that matter).\n\nIncluding a newline inside a {- ... -} block should not be a problem\nwith the --word-diff=plain format, and something similar can be argued\nfor the porcelain format.\n\nAn alternative approach would be to add a --word-diff-exact flag, which\nwould cause the whitespace between to matches of the word regex to be\ntreated as a word as well and have it included in the generate\nword-diff.\n\nThis still leaves an implementation problem: To generate the word-diff,\nthe current code looks at one patch hunk at a time, collecting all the\nplus and minus lines. It then splits those lines into words and\ngenerates two new \"files\" containing one word per line. It then applies\na diff to this new document to get the word-diff.\n\nWhen a word would contain a newline, this would effectively mean the\nword would be split into two words for the word-diff, which will\nprobably screw up the output. An obvious solution would be to use some\nescape sequence (e.g. \\n) for a newline, though that might get messy and\ninefficient. An alternative that seems feasable is to use the empty word\n(i.e., an empty line in the word-diff \"files\") to mean a newline.\n\nThis would mean that every newline always breaks a word into two,\nregardless of what the word regex is set to (but I guess that makes\nsense anyway?). I also think this would allow complete diff output wrt\nwhitespace and newlines, for output formats that support it: plain,\n(modified) porcelain and my proposed format.\n\nPorcelain format\n----------------\nLastly, the \"porcelain\" word-diff format seems a bit weird to me. Is\nthe format specified somewhere, or are there any programs that use it\ncurrently? I couldn't find any users inside the git.git tree itself?\n\nLooking at the format itself, it's a bit unclear to me what the ~ lines\nmean exactly. Commit 882749, which introduced the format says the mean\n\"newlines in the input\", but I'm not sure if this means the old file,\nnew file or both.\n\nIn fact, it seems that this uncertainty makes the porcelain-format\nambiguous wrt newlines. For example, these two diff hunks:\n\n@@ -1,3 +1,2 @@\n a\n-b\n c\n\n@@ -1,3 +1,3 @@\n a\n-b\n+\n c\n\nboth look the same in porcelain format, except for the hunk header.\n\n@@ -1,3 +1,2 @@\n a\n~\n-b\n~\n c\n~\n@@ -1,3 +1,3 @@\n a\n~\n-b\n~\n c\n~\n\nThis is somewhat expected, of course, since the --word-diff formats are\ndocumented to show only changes to words, not to non-words/whitespace.\nSo I guess it is expected that the output is ambigious wrt whitespace,\nbut if so, what is the use of this porcelain format? Wouldn't it be make\na lot more sense to make the format unambiguous and make it do\nword-based diff at the same time? I think this should be possible\nbecause of the explicit notation used for the newline.\n\nFor example, Specifying the ~ lines to mean a newline in the old, new or\nboth files depending on the previous +, - or space prefixed line is\nprobably enough for this. By generating empty +, - or space prefixed\nlines when needed, every occurence of ~ could be disambiguated.\n\nFor example, the above two diff hunks would then become the following.\nThe only difference is the near-empty line (just a space prefix) after\n-b in the second hunk.\n\n@@ -1,3 +1,2 @@\n a\n~\n-b\n~\n c\n~\n@@ -1,3 +1,3 @@\n a\n~\n-b\n \n~\n c\n~\n\n\n\n\nSo, these are some thoughts I've had while hacking on the code. As said,\nsuggestions are welcome. I'd like my hacking to result in some useful\npatches, but right now I'm unsure what direction(s) I should be\nthinking/working in.\n\nIn case you're interested in the hacking I've done so far, I've put it\nup here:\n\nhttp://git.stderr.nl/gitweb?p=matthijs/upstream/git.git;a=shortlog;h=refs/heads/word-diff\n\nMost of it is broken or not properly tested, but it gets an idea what\nkinds of cleanup I've been doing.\n\nGr.\n\nMatthijs\n"},{"id":"155388","messageId":"vpqlj53zuum.fsf@bauges.imag.fr","threadId":"25674","inReplyTo":"20101108151601.GF22067@login.drsnuggles.stderr.nl","subject":"Re: Understanding and improving --word-diff","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2010-11-08T15:41:05Z","receivedAt":"2010-11-08T15:41:05Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Matthijs Kooijman <matthijs@stdin.nl> writes:\n\n> Lastly, the \"porcelain\" word-diff format seems a bit weird to me. Is\n> the format specified somewhere, or are there any programs that use it\n> currently? I couldn't find any users inside the git.git tree itself?\n\nThere's a pending patch for \"gitk --color-words\". Actually, the\nporcelain format was precisely implemented to allow this patch to\nexist IIRC.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"155389","messageId":"20101108154903.GH22067@login.drsnuggles.stderr.nl","threadId":"25674","inReplyTo":"vpqlj53zuum.fsf@bauges.imag.fr","subject":"Re: Understanding and improving --word-diff","fromName":"Matthijs Kooijman","fromEmail":"matthijs@stdin.nl","sentAt":"2010-11-08T15:49:03Z","receivedAt":"2010-11-08T15:49:03Z","isPatch":false,"sender":{"key":"matthijs@stdin.nl","avatar":"https://avatars.githubusercontent.com/u/194491?v=4"},"body":"Hi Matthieu,\n\n> There's a pending patch for \"gitk --color-words\". Actually, the\n> porcelain format was precisely implemented to allow this patch to\n> exist IIRC.\nI think you mean this patch?\n\n[PATCH v4' 2/2] gitk: add the equivalent of diff --color-words\nhttps://kerneltrap.org/mailarchive/git/2010/4/19/28707\n\nI had seen it around, but hadn't realized it wasn't in master yet. I'll\nhave a closer look at it.\n\nThanks.\n\nMatthijs\n"},{"id":"155390","messageId":"201011081733.06502.trast@student.ethz.ch","threadId":"25674","inReplyTo":"vpqlj53zuum.fsf@bauges.imag.fr","subject":"Re: Understanding and improving --word-diff","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-11-08T16:33:06Z","receivedAt":"2010-11-08T16:33:06Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Matthieu Moy wrote:\n> Matthijs Kooijman <matthijs@stdin.nl> writes:\n> \n> > Lastly, the \"porcelain\" word-diff format seems a bit weird to me. Is\n> > the format specified somewhere, or are there any programs that use it\n> > currently? I couldn't find any users inside the git.git tree itself?\n> \n> There's a pending patch for \"gitk --color-words\". Actually, the\n> porcelain format was precisely implemented to allow this patch to\n> exist IIRC.\n\nAlso, my intent was to specify the format fully under its option\ndescription.  It's rather simple, but if you think the\nexplanation/definition is inadequate, please provide a patch.  We'd\nreally want the writers of consumers (GUIs and such) to be very clear\nabout how it is formatted.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"155393","messageId":"857393A7-596D-4848-A002-6AC06FF6B2EA@wincent.com","threadId":"25674","inReplyTo":"20101108151601.GF22067@login.drsnuggles.stderr.nl","subject":"Re: Understanding and improving --word-diff","fromName":"Wincent Colaiuta","fromEmail":"win@wincent.com","sentAt":"2010-11-08T17:35:48Z","receivedAt":"2010-11-08T17:35:48Z","isPatch":false,"sender":{"key":"greg@hurrell.net","avatar":"https://avatars.githubusercontent.com/u/7074?v=4"},"body":"On 08/11/2010, at 16:16, Matthijs Kooijman wrote:\n\n> Hi folks,\n> \n> I recently discovered --word-diff (or rather, --color-words and found\n> --word-diff when I started to hack on the git master version) and I had\n> hoped it would make the unified diffs generated by git-diff more\n> readable.\n\n[snip]\n\n> Inexact output\n> --------------\n> Secondly, the --word-diff output currently never displays any changes to\n> the non-word (whitespace) parts of a file. This makes sense for the\n> LaTeX case, but sometimes you might want to get exact diff output\n> instead. At first glance this seems possible by specifiying a word-regex\n> of \".\" or something similar (i.e., make sure that the word regex matches\n> everything). But this is problematic for newlines. The documentation\n> states that stuff gets silently ignored if a newline ends up inside a\n> word. For the --word-diff=color format, this is probably a fixed\n> limitation of the otput format: you can't give a color to a newline (or\n> a space, for that matter).\n\n[snip]\n\n> Porcelain format\n> ----------------\n> Lastly, the \"porcelain\" word-diff format seems a bit weird to me. Is\n> the format specified somewhere, or are there any programs that use it\n> currently? I couldn't find any users inside the git.git tree itself?\n> \n> Looking at the format itself, it's a bit unclear to me what the ~ lines\n> mean exactly. Commit 882749, which introduced the format says the mean\n> \"newlines in the input\", but I'm not sure if this means the old file,\n> new file or both.\n\nI posted along similar lines back in September:\n\n  http://article.gmane.org/gmane.comp.version-control.git/156375\n\nI found I couldn't really use the porcelain format due to information loss and ended up having to roll my own, parse the normal diff output and using that.\n\nCheers,\nWincent\n"},{"id":"155414","messageId":"201011082022.13678.trast@student.ethz.ch","threadId":"25674","inReplyTo":"20101108151601.GF22067@login.drsnuggles.stderr.nl","subject":"Re: Understanding and improving --word-diff","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-11-08T19:22:13Z","receivedAt":"2010-11-08T19:22:13Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Matthijs Kooijman wrote:\n> This is somewhat expected, of course, since the --word-diff formats are\n> documented to show only changes to words, not to non-words/whitespace.\n> So I guess it is expected that the output is ambigious wrt whitespace,\n> but if so, what is the use of this porcelain format? Wouldn't it be make\n> a lot more sense to make the format unambiguous and make it do\n> word-based diff at the same time? I think this should be possible\n> because of the explicit notation used for the newline.\n\nIt's not just the newlines.  You will note that when diffing e.g.\n\n  a x  c\nvs.\n  a  b c\n\nthen because the word-diff engine doesn't bother with the non-word\nparts, you will end up with (using ordinary --word-diff)\n\n  a  [-x-]{+b+} c\n\nor using the porcelain format\n\n  $ git diff --no-index --word-diff=porcelain a b | cat -E\n  diff --git 1/a 2/b$\n  index 4f47486..b040ae0 100644$\n  --- 1/a$\n  +++ 2/b$\n  @@ -1 +1 @@$\n   a  $\n  -x$\n  +b$\n    c$\n  ~$\n\nNote how the second space after 'x' disappeared.  The case that\nWincent explains in the email he linked (thanks; I didn't even see it\nthe first time around) shows a more drastic example.  So the logic\nthat inserts the space again is too simplistic.\n\nWhat it does not is something like (haven't checked again) always\ninsert the postimage space that went with the preceding word.  What it\n*would* have to do is actually compute differences in the space, while\nat the same time ignoring them for the purposes of the LCS algorithm.\nIf it weren't for the customizable word-regex, it might be enough to\nput the spaces in the words again but enable the internal equivalent\nof -b.\n\n> Looking at the format itself, it's a bit unclear to me what the ~ lines\n> mean exactly. Commit 882749, which introduced the format says the mean\n> \"newlines in the input\", but I'm not sure if this means the old file,\n> new file or both.\n\nAs Matthieu already said, the history of --word-diff=porcelain is\njust: I needed a way to pipe word-diff output to gitk for coloring,\nwithout any messing around with the color strings.  The ~ is the\nminimal required feature to at least let git communicate *some*\nlinebreaks.\n\nYou are probably right in that while the format leaves open the\npossibility of showing addition and removal of space (although the\ncurrent engine does not really attempt to compute it), I did not think\nfar enough to let it show addition and removal of whitespace.\n\n> For example, Specifying the ~ lines to mean a newline in the old, new or\n> both files depending on the previous +, - or space prefixed line is\n> probably enough for this. By generating empty +, - or space prefixed\n> lines when needed, every occurence of ~ could be disambiguated.\n\nAFAICS that breaks down if you couple a newline-change-aware consumer\nwith a non-aware producer (i.e., a new frontend with an old git),\nsince there is currently no guarantee that the line before a ~ is\ncontext.\n\nSo you would have to introduce a new format (porcelain2?) anyway, and\nif you're already going that route, then for simplicity I'd rather\nhave something like ~+ (or so) instead of requiring the parser to\ntrack state.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"155541","messageId":"20101109220136.GA5617@sigill.intra.peff.net","threadId":"25674","inReplyTo":"20101108151601.GF22067@login.drsnuggles.stderr.nl","subject":"Re: Understanding and improving --word-diff","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-11-09T22:01:36Z","receivedAt":"2010-11-09T22:01:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 08, 2010 at 04:16:01PM +0100, Matthijs Kooijman wrote:\n\n> E.g., I would like to see:\n> \n> -a <r>b</r> c\n> +a <g>x</g> c\n> \n> Unfortunately, all --word-diff types currently departs from line-based -\n> and + lines and show the new version of the file with the changed words\n> (both old and new versions) shown inline, marked with coloring or\n> {- ...  -} kind of syntax. E.g., with --word-diff=color, the above would\n> look like:\n> \n> a <r>b</r><g>x</g> c\n> \n> Personally, I think that the first example above is easier to read than\n> the second one (at least for diffs of code).\n\nYeah, as you figured out, word diff is really about formats that aren't\nline oriented. I have seen other systems do what you are asking for, but\nI'm not sure they are nearly as complex as a true word diff. In\nparticular, I have noticed two simplifications:\n\n  1. Only highlight between two lines that are in part of a hunk by\n     themselves. In other words, highlight on this:\n\n        context\n       -foo bar\n       +foo baz\n        context\n\n     but not this:\n\n        context\n       -foo bar\n       -another line\n       +foo baz\n       +another changed line\n        context\n\n     Even though you could perhaps match up pairs of lines (in this\n     case, the first removed line obviously corresponds with the first\n     added line, etc), there are a ton of cases where it would be\n     uselessly distracting, as the lines simply aren't related to each\n     other (of course, there are one-liner changes where that is the\n     case, too, but they are hopefully less common, and since it is only\n     a single pair of lines, there is no guessing about how they were\n     matched up).\n\n  2. Don't do real LCS matching between the pairs. Just find the common\n     prefix and suffix, and highlight everything in the middle. This\n     would yield something like:\n\n       -a <r>b c d</r> e\n       +a <g>x c z</g> e\n\n     when you could in theory produce:\n\n       -a <r>b</r> c <r>d</r> e\n       -a <g>x</g> c <g>z</g> e\n\n     but I suspect in practice it covers many situations.\n\nFor fun, I hacked up the perl script below which will highlight (using\nANSI reverse-codes!) a diff using both of these simplifications. It\nlooks for diff hunks on stdin, so you can do:\n\n  git log | perl /path/to/script.pl\n\nI'm curious to see how the output looks on some real-world diffs. I\nadmit to not having given this problem much thought, but instead just\nsat down and started hacking. :)\n\nThe script tries to handle color in the input sanely, so you can see\nyour highlighting in color. But there are two caveats:\n\n  1. You have to pass --color to git explicitly, since piping stdout\n     turns off git's color autodetection.\n\n  2. There is a buglet in git's color emitting code. For added lines, it\n     produces \"<g>+</g><g>rest of line</g>\" which is annoying to parse.\n     It is fixed by the patch below:\n\ndiff --git a/diff.c b/diff.c\nindex b1b0d3d..0efcdb7 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -363,9 +363,9 @@ static void emit_add_line(const char *reset,\n \t\temit_line_0(ecbdata->opt, ws, reset, '+', line, len);\n \telse {\n \t\t/* Emit just the prefix, then the rest. */\n-\t\temit_line_0(ecbdata->opt, set, reset, '+', \"\", 0);\n+\t\temit_line_0(ecbdata->opt, set, \"\", '+', \"\", 0);\n \t\tws_check_emit(line, len, ecbdata->ws_rule,\n-\t\t\t      ecbdata->opt->file, set, reset, ws);\n+\t\t\t      ecbdata->opt->file, \"\", reset, ws);\n \t}\n }\n \n\nThis patch, btw, is much easier to read when piped through my script. :)\n\nHere's the script:\n\n-- >8 --\n#!/usr/bin/perl\n\nuse strict;\nmy $COLOR = qr/\\x1b\\[[0-9;]*m/;\n\nmy @window;\n\nwhile (<>) {\n  chomp;\n  s/^($COLOR)*//; my $prefix = $&;\n  s/($COLOR)*$//; my $suffix = $&;\n\n  push @window, [$prefix, $_, $suffix];\n\n  if ($window[0] && $window[0]->[1] =~ /^(\\@| )/ &&\n      $window[1] && $window[1]->[1] =~ /^-/ &&\n      $window[2] && $window[2]->[1] =~ /^\\+/ &&\n      $window[3] && $window[3]->[1] !~ /^\\+/) {\n    show_line(shift @window);\n    show_pair(shift @window, shift @window);\n  }\n\n  if (@window >= 4) {\n    show_line(shift @window);\n  }\n}\n\nif (@window == 3 &&\n    $window[0] && $window[0]->[1] =~ /^(\\@| )/ &&\n    $window[1] && $window[1]->[1] =~ /^-/ &&\n    $window[2] && $window[2]->[1] =~ /^\\+/) {\n  show_line(shift @window);\n  show_pair(shift @window, shift @window);\n}\nwhile (@window) {\n  show_line(shift @window);\n}\n\nexit 0;\n\nsub show_line {\n  my $line = shift;\n  print @$line, \"\\n\";\n}\n\nsub show_pair {\n  my ($from, $to) = @_;\n  my @from = split //, $from->[1];\n  my @to = split //, $to->[1];\n\n  my $prefix = 1;\n  while ($from[$prefix] eq $to[$prefix]) {\n    $prefix++;\n  }\n  my $suffix = 0;\n  while ($from[$#from-$suffix] eq $to[$#to-$suffix]) {\n    $suffix++;\n  }\n\n  print $from->[0], highlight($from->[1], $prefix, $suffix), $from->[2], \"\\n\";\n  print $to->[0], highlight($to->[1], $prefix, $suffix), $to->[2], \"\\n\";\n}\n\nsub highlight {\n  my ($line, $prefix, $suffix) = @_;\n  my $end = length($line) - $suffix;\n  return join('',\n    substr($line, 0, $prefix),\n    \"\\x1b[7m\",\n    substr($line, $prefix, $end - $prefix),\n    \"\\x1b[27m\",\n    substr($line, $end)\n  );\n}\n"},{"id":"155550","messageId":"alpine.DEB.1.00.1011100102540.2226@bonsai2","threadId":"25674","inReplyTo":"20101109220136.GA5617@sigill.intra.peff.net","subject":"Re: Understanding and improving --word-diff","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-11-10T00:05:56Z","receivedAt":"2010-11-10T00:05:56Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 9 Nov 2010, Jeff King wrote:\n\n> On Mon, Nov 08, 2010 at 04:16:01PM +0100, Matthijs Kooijman wrote:\n> \n> > E.g., I would like to see:\n> > \n> > -a <r>b</r> c\n> > +a <g>x</g> c\n> > \n> > Unfortunately, all --word-diff types currently departs from line-based \n> > - and + lines and show the new version of the file with the changed \n> > words (both old and new versions) shown inline, marked with coloring \n> > or {- ...  -} kind of syntax. E.g., with --word-diff=color, the above \n> > would look like:\n> > \n> > a <r>b</r><g>x</g> c\n> > \n> > Personally, I think that the first example above is easier to read \n> > than the second one (at least for diffs of code).\n> \n> Yeah, as you figured out, word diff is really about formats that aren't \n> line oriented.\n\nNot really correct. While the _current_ way to show word diffs is \nimitating GNU wdiff, the internal data structure does allow for more \nsophisticated output.\n\nCiao,\nDscho\n\nP.S.: Peff, I hope you're fine with me responding to the first sentence \nonly. After all, you know that my attention span is 7.2 seconds.\n\nSome people taught me to put a smiley after such a statement, so: ;-)\n"},{"id":"155570","messageId":"20101110041705.GB8955@sigill.intra.peff.net","threadId":"25674","inReplyTo":"alpine.DEB.1.00.1011100102540.2226@bonsai2","subject":"Re: Understanding and improving --word-diff","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-11-10T04:17:05Z","receivedAt":"2010-11-10T04:17:05Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 10, 2010 at 01:05:56AM +0100, Johannes Schindelin wrote:\n\n> > Yeah, as you figured out, word diff is really about formats that aren't \n> > line oriented.\n> \n> Not really correct. While the _current_ way to show word diffs is \n> imitating GNU wdiff, the internal data structure does allow for more \n> sophisticated output.\n\nFair enough. I was basing my statement on the output. If somebody would\nlike to demonstrate how the code can be used to get line-oriented diffs\nwith highlighting, I would be very happy to eat my words. :)\n\n> P.S.: Peff, I hope you're fine with me responding to the first sentence \n> only. After all, you know that my attention span is 7.2 seconds.\n\nHeh. I am happy you responded at all. You are very quiet these days. :)\n\n-Peff\n"},{"id":"156081","messageId":"20101118064050.GA12825@sigill.intra.peff.net","threadId":"25674","inReplyTo":"20101109220136.GA5617@sigill.intra.peff.net","subject":"Re: Understanding and improving --word-diff","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-11-18T06:40:50Z","receivedAt":"2010-11-18T06:40:50Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 09, 2010 at 05:01:36PM -0500, Jeff King wrote:\n\n>   2. There is a buglet in git's color emitting code. For added lines, it\n>      produces \"<g>+</g><g>rest of line</g>\" which is annoying to parse.\n>      It is fixed by the patch below:\n> \n> diff --git a/diff.c b/diff.c\n> index b1b0d3d..0efcdb7 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -363,9 +363,9 @@ static void emit_add_line(const char *reset,\n>  \t\temit_line_0(ecbdata->opt, ws, reset, '+', line, len);\n>  \telse {\n>  \t\t/* Emit just the prefix, then the rest. */\n> -\t\temit_line_0(ecbdata->opt, set, reset, '+', \"\", 0);\n> +\t\temit_line_0(ecbdata->opt, set, \"\", '+', \"\", 0);\n>  \t\tws_check_emit(line, len, ecbdata->ws_rule,\n> -\t\t\t      ecbdata->opt->file, set, reset, ws);\n> +\t\t\t      ecbdata->opt->file, \"\", reset, ws);\n>  \t}\n>  }\n\nFYI, I looked further at this patch, and it should not be used. There\nare cases where ws_check_emit will output a reset (e.g., because it has\ncolored some whitespace), and needs to be able to set the proper color\nagain. So while I think the intent of this patch is fine (to avoid\nduplicate colorizing in the output), the actual implementation would\nneed to be much more complex.\n\nInstead, here's an updated version of my color highlighting script that\nhandles this case.\n\nWith the patch I posted earlier today:\n\n  [PATCH 2/2] allow command-specific pagers in pager.<cmd>\n  http://article.gmane.org/gmane.comp.version-control.git/161624\n\nyou can do:\n\n  git config pager.log '/path/to/diff-highlight.pl | less'\n\nand get automagic coloring. Still, this is a bit of a hack, and there\nare some funny corner cases, so something that used the internal diff\nmachinery would be much nicer.\n\n-Peff\n\n-- >8 --\n#!/usr/bin/perl\n\nuse strict;\nmy $COLOR = qr/\\x1b\\[[0-9;]*m/;\nmy $RESET = \"\\x1b[0m\";\nmy $RED = \"\\x1b[31m\";\nmy $GREEN = \"\\x1b[32m\";\nmy $REVERSE = \"\\x1b[7m\";\nmy $UNREVERSE = \"\\x1b[27m\";\n\nmy @window;\n\nwhile (<>) {\n  chomp;\n  my $plain = $_;\n  $plain =~ s/$COLOR//g;\n\n  push @window, [$_, $plain];\n\n  if ($window[0] && $window[0]->[1] =~ /^(\\@| )/ &&\n      $window[1] && $window[1]->[1] =~ /^-/ &&\n      $window[2] && $window[2]->[1] =~ /^\\+/ &&\n      $window[3] && $window[3]->[1] !~ /^\\+/) {\n    show_line(shift @window);\n    show_pair(shift @window, shift @window);\n  }\n\n  if (@window >= 4) {\n    show_line(shift @window);\n  }\n}\n\nif (@window == 3 &&\n    $window[0] && $window[0]->[1] =~ /^(\\@| )/ &&\n    $window[1] && $window[1]->[1] =~ /^-/ &&\n    $window[2] && $window[2]->[1] =~ /^\\+/) {\n  show_line(shift @window);\n  show_pair(shift @window, shift @window);\n}\nwhile (@window) {\n  show_line(shift @window);\n}\n\nexit 0;\n\nsub show_line {\n  my $line = shift;\n  print $line->[0], \"\\n\";\n}\n\nsub show_pair {\n  my ($from, $to) = @_;\n  my @from = split //, $from->[1];\n  my @to = split //, $to->[1];\n\n  my $prefix = 1;\n  while ($from[$prefix] eq $to[$prefix]) {\n    $prefix++;\n  }\n  my $suffix = 0;\n  while ($from[$#from-$suffix] eq $to[$#to-$suffix]) {\n    $suffix++;\n  }\n\n  print $RED, highlight($from->[1], $prefix, $suffix), $RESET, \"\\n\";\n  print $GREEN, highlight($to->[1], $prefix, $suffix), $RESET, \"\\n\";\n}\n\nsub highlight {\n  my ($line, $prefix, $suffix) = @_;\n  my $end = length($line) - $suffix;\n  return $line unless $end > $prefix;\n  return join('',\n    substr($line, 0, $prefix),\n    $REVERSE,\n    substr($line, $prefix, $end - $prefix),\n    $UNREVERSE,\n    substr($line, $end)\n  );\n}\n"}]}