{"thread":{"id":"59588","subject":"similarity index vs. whitespaces","startedAt":"2023-04-13T01:56:56Z","lastAt":"2023-04-13T02:36:16Z","messageCount":2,"participants":["Mike Hommey","Chris Torek"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"475262","messageId":"20230413015640.h6npzp47tnob7bq2@glandium.org","threadId":"59588","inReplyTo":null,"subject":"similarity index vs. whitespaces","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2023-04-13T01:56:40Z","receivedAt":"2023-04-13T01:56:56Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Hi,\n\nI hit an interesting case of copy-detection \"failure\" after renaming a\npython script and changing indentation in it at the same time: git show\n-C wouldn't detect it as a rename unless I lowered the similarity index\nsignificantly. But from a human perspective, the similarity index feels\nwrong.\n\nHere's a small illustrative example:\n$ git init qux\n$ cd qux\n$ cat > qux <<EOF\na\nb\nc\nd\ne\nf\nEOF\n$ git add qux\n$ git commit -a -m qux\n$ git mv qux hoge\n$ sed -i 's/[cde]/  \\0/' hoge\n$ git commit -a -m hoge\n$ git diff -C HEAD^!\ndiff --git a/hoge b/hoge\nnew file mode 100644\nindex 0000000..9ab6fcc\n--- /dev/null\n+++ b/hoge\n@@ -0,0 +1,6 @@\n+a\n+b\n+  c\n+  d\n+  e\n+f\ndiff --git a/qux b/qux\ndeleted file mode 100644\nindex 0fdf397..0000000\n--- a/qux\n+++ /dev/null\n@@ -1,6 +0,0 @@\n-a\n-b\n-c\n-d\n-e\n-f\n\n$ git diff -C10% HEAD^!\ndiff --git a/qux b/hoge\nsimilarity index 33%\nrename from qux\nrename to hoge\nindex 0fdf397..9ab6fcc 100644\n--- a/qux\n+++ b/hoge\n@@ -1,6 +1,6 @@\n a\n b\n-c\n-d\n-e\n+  c\n+  d\n+  e\n f\n\nFrom a human perspective 33% similarity feels way too low here. I know\nit's essentially counting lines in the diff, but that feels limited.\n\nWhat do you think?\n\nMike\n"},{"id":"475263","messageId":"CAPx1Gve5b3yoUFK1VObOjjkFUCG4emaYUnJ-Q7bRyfURK74DCQ@mail.gmail.com","threadId":"59588","inReplyTo":"20230413015640.h6npzp47tnob7bq2@glandium.org","subject":"Re: similarity index vs. whitespaces","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2023-04-13T02:35:58Z","receivedAt":"2023-04-13T02:36:16Z","isPatch":false,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Wed, Apr 12, 2023 at 7:01 PM Mike Hommey <mh@glandium.org> wrote:\n[example of Python script diff snipped]\n> From a human perspective 33% similarity feels way too low here. I know\n> it's essentially counting lines in the diff, but that feels limited.\n\nTechnically, the similarity index isn't based on lines.  Instead\nit's based on byte-by-byte matching, broken into segments (these\ndo use LF or CRLF to break up segments as appropriate but very\n*long* lines get broken up without such line terminators).  This\nshares some code with the delta compression algorithm used to\npack objects.  One of the goals here is to consider a lot of\n*moved* lines to be \"very similar\", despite such moves generally\nproducing large-ish diffs.\n\nCRs are, if I recall correctly, discarded during similarty index\ncomputation.  It would be somewhat easy to discard leading white\nspace -- and even easier to discard *all* white space, but I'd\nsuggest that would be wrong -- due to the special case already\nin place for line terminators here.\n\nIt's not clear to me that discarding leading white space would\nbe correct in *all* cases, but it does seem very appropriate for\nPython code.  Whether the earlier discussion about diff algorithms\nbeing adjusted based on .gitattributes entries might apply here,\nI'll leave for others to argue about. :-)  (That is, I'm just\ntossing it out as an idea for the mix.)\n\nChris\n"}]}