{"thread":{"id":"49958","subject":"A case where diff.colorMoved=plain is more sensible than diff.colorMoved=zebra & others","startedAt":"2018-12-06T13:54:41Z","lastAt":"2018-12-11T00:55:09Z","messageCount":5,"participants":["Ævar Arnfjörð Bjarmason","Phillip Wood","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"364686","messageId":"87zhtiyd45.fsf@evledraar.gmail.com","threadId":"49958","inReplyTo":null,"subject":"A case where diff.colorMoved=plain is more sensible than diff.colorMoved=zebra & others","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-12-06T13:54:34Z","receivedAt":"2018-12-06T13:54:41Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Let's ignore how bad this patch is for git.git, and just focus on how\ndiff.colorMoved treats it:\n\n    diff --git a/builtin/add.c b/builtin/add.c\n    index f65c172299..d1155322ef 100644\n    --- a/builtin/add.c\n    +++ b/builtin/add.c\n    @@ -6,5 +6,3 @@\n     #include \"cache.h\"\n    -#include \"config.h\"\n     #include \"builtin.h\"\n    -#include \"lockfile.h\"\n     #include \"dir.h\"\n    diff --git a/builtin/am.c b/builtin/am.c\n    index 8f27f3375b..eded15aa8a 100644\n    --- a/builtin/am.c\n    +++ b/builtin/am.c\n    @@ -6,3 +6,2 @@\n     #include \"cache.h\"\n    -#include \"config.h\"\n     #include \"builtin.h\"\n    diff --git a/builtin/blame.c b/builtin/blame.c\n    index 06a7163ffe..44a754f190 100644\n    --- a/builtin/blame.c\n    +++ b/builtin/blame.c\n    @@ -8,3 +8,2 @@\n     #include \"cache.h\"\n    -#include \"config.h\"\n     #include \"color.h\"\n    diff --git a/cache.h b/cache.h\n    index ca36b44ee0..ea8d60b94a 100644\n    --- a/cache.h\n    +++ b/cache.h\n    @@ -4,2 +4,4 @@\n     #include \"git-compat-util.h\"\n    +#include \"config.h\"\n    +#include \"new.h\"\n     #include \"strbuf.h\"\n\nThis is a common thing that's useful to have highlighted, e.g. we move\nincludes of config.h to some common file, so I want to se all the\ndeleted config.h lines as moved into the cache.h line, and then the\n\"lockfile.h\" I removed while I was at it plain remove, and the new\n\"new.h\" plain added.\n\nExactly that is what you get with diff.colorMoved=plain, but the default\nof diff.colorMoved=zebra gets confused by this and highlights no moves\nat all, same or \"blocks\" and \"dimmed-zebra\".\n\nSo at first I thought this had something to do with the many->one\ndetection, but it seems to be simpler, we just don't detect a move of\n1-line with anything but plain, e.g. this works as expected in all modes\nand detects the many->one:\n\n    diff --git a/builtin/add.c b/builtin/add.c\n    index f65c172299..f4fda75890 100644\n    --- a/builtin/add.c\n    +++ b/builtin/add.c\n    @@ -5,4 +5,2 @@\n      */\n    -#include \"cache.h\"\n    -#include \"config.h\"\n     #include \"builtin.h\"\n    diff --git a/builtin/branch.c b/builtin/branch.c\n    index 0c55f7f065..52e39924d3 100644\n    --- a/builtin/branch.c\n    +++ b/builtin/branch.c\n    @@ -7,4 +7,2 @@\n\n    -#include \"cache.h\"\n    -#include \"config.h\"\n     #include \"color.h\"\n    diff --git a/cache.h b/cache.h\n    index ca36b44ee0..d4146dbf8a 100644\n    --- a/cache.h\n    +++ b/cache.h\n    @@ -3,2 +3,4 @@\n\n    +#include \"cache.h\"\n    +#include \"config.h\"\n     #include \"git-compat-util.h\"\n\nSo is there some \"must be at least two consecutive lines\" condition for\nnot-plain, or is something else going on here?\n"},{"id":"364689","messageId":"b23422fa-8a1d-e508-a008-a2fe27b7b49f@talktalk.net","threadId":"49958","inReplyTo":"87zhtiyd45.fsf@evledraar.gmail.com","subject":"Re: A case where diff.colorMoved=plain is more sensible than diff.colorMoved=zebra & others","fromName":"Phillip Wood","fromEmail":"phillip.wood@talktalk.net","sentAt":"2018-12-06T14:58:03Z","receivedAt":"2018-12-06T14:58:08Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 06/12/2018 13:54, Ævar Arnfjörð Bjarmason wrote:\n> Let's ignore how bad this patch is for git.git, and just focus on how\n> diff.colorMoved treats it:\n> \n>      diff --git a/builtin/add.c b/builtin/add.c\n>      index f65c172299..d1155322ef 100644\n>      --- a/builtin/add.c\n>      +++ b/builtin/add.c\n>      @@ -6,5 +6,3 @@\n>       #include \"cache.h\"\n>      -#include \"config.h\"\n>       #include \"builtin.h\"\n>      -#include \"lockfile.h\"\n>       #include \"dir.h\"\n>      diff --git a/builtin/am.c b/builtin/am.c\n>      index 8f27f3375b..eded15aa8a 100644\n>      --- a/builtin/am.c\n>      +++ b/builtin/am.c\n>      @@ -6,3 +6,2 @@\n>       #include \"cache.h\"\n>      -#include \"config.h\"\n>       #include \"builtin.h\"\n>      diff --git a/builtin/blame.c b/builtin/blame.c\n>      index 06a7163ffe..44a754f190 100644\n>      --- a/builtin/blame.c\n>      +++ b/builtin/blame.c\n>      @@ -8,3 +8,2 @@\n>       #include \"cache.h\"\n>      -#include \"config.h\"\n>       #include \"color.h\"\n>      diff --git a/cache.h b/cache.h\n>      index ca36b44ee0..ea8d60b94a 100644\n>      --- a/cache.h\n>      +++ b/cache.h\n>      @@ -4,2 +4,4 @@\n>       #include \"git-compat-util.h\"\n>      +#include \"config.h\"\n>      +#include \"new.h\"\n>       #include \"strbuf.h\"\n> \n> This is a common thing that's useful to have highlighted, e.g. we move\n> includes of config.h to some common file, so I want to se all the\n> deleted config.h lines as moved into the cache.h line, and then the\n> \"lockfile.h\" I removed while I was at it plain remove, and the new\n> \"new.h\" plain added.\n> \n> Exactly that is what you get with diff.colorMoved=plain, but the default\n> of diff.colorMoved=zebra gets confused by this and highlights no moves\n> at all, same or \"blocks\" and \"dimmed-zebra\".\n> \n> So at first I thought this had something to do with the many->one\n> detection, but it seems to be simpler, we just don't detect a move of\n> 1-line with anything but plain, e.g. this works as expected in all modes\n> and detects the many->one:\n> \n>      diff --git a/builtin/add.c b/builtin/add.c\n>      index f65c172299..f4fda75890 100644\n>      --- a/builtin/add.c\n>      +++ b/builtin/add.c\n>      @@ -5,4 +5,2 @@\n>        */\n>      -#include \"cache.h\"\n>      -#include \"config.h\"\n>       #include \"builtin.h\"\n>      diff --git a/builtin/branch.c b/builtin/branch.c\n>      index 0c55f7f065..52e39924d3 100644\n>      --- a/builtin/branch.c\n>      +++ b/builtin/branch.c\n>      @@ -7,4 +7,2 @@\n> \n>      -#include \"cache.h\"\n>      -#include \"config.h\"\n>       #include \"color.h\"\n>      diff --git a/cache.h b/cache.h\n>      index ca36b44ee0..d4146dbf8a 100644\n>      --- a/cache.h\n>      +++ b/cache.h\n>      @@ -3,2 +3,4 @@\n> \n>      +#include \"cache.h\"\n>      +#include \"config.h\"\n>       #include \"git-compat-util.h\"\n> \n> So is there some \"must be at least two consecutive lines\" condition for\n> not-plain, or is something else going on here?\n\nTo be considered a block has to have 20 alphanumeric characters - see \ncommit f0b8fb6e59 (\"diff: define block by number of alphanumeric chars\", \n2017-08-15). This stops things like random '}' lines being marked as \nmoved on their own. It might be better to use some kind of frequency \ninformation (a bit like python's difflib junk parameter) instead so that \n(fairly) unique short lines also get marked properly.\n\nBest Wishes\n\nPhillip\n"},{"id":"364695","messageId":"CAGZ79kZVfMNELXuir+t9U8bSg2PVF=oX8aya-OqmRaP0gHRgFw@mail.gmail.com","threadId":"49958","inReplyTo":"b23422fa-8a1d-e508-a008-a2fe27b7b49f@talktalk.net","subject":"Re: A case where diff.colorMoved=plain is more sensible than diff.colorMoved=zebra & others","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-12-06T18:11:13Z","receivedAt":"2018-12-06T18:11:28Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Dec 6, 2018 at 6:58 AM Phillip Wood <phillip.wood@talktalk.net> wrote:\n\n> > So is there some \"must be at least two consecutive lines\" condition for\n> > not-plain, or is something else going on here?\n>\n> To be considered a block has to have 20 alphanumeric characters - see\n> commit f0b8fb6e59 (\"diff: define block by number of alphanumeric chars\",\n> 2017-08-15). This stops things like random '}' lines being marked as\n> moved on their own.\n\nThis is spot on.\n\nAll but the \"plain\" mode use the concept of \"blocks\" of code\n(there is even one mode called \"blocks\", which adds to the confusion).\n\n> It might be better to use some kind of frequency\n> information (a bit like python's difflib junk parameter) instead so that\n> (fairly) unique short lines also get marked properly.\n\nYes that is what I was initially thinking about. However to have good\ninformation, you'd need to index a whole lot (the whole repository,\ni.e. all text blobs in existence?) to get an accurate picture of frequency\ninformation, which I'd prefer to call entropy as I come from a background\nfamiliar with https://en.wikipedia.org/wiki/Information_theory, I am not\nsure where 'frequency information' comes from -- it sounds like the\nsame concept.\n\nOf course it is too expensive to run an operation O(repository size)\njust for this diff, so maybe we could get away with some smaller\ncorpus to build up this information on what is sufficient for coloring.\n\nWhen only looking at the given diff, I would imagine that each line\nwould not carry a whole lot of information as its characters occur\nrather frequently compared to the rest of the diff.\n\nBest,\nStefan\n"},{"id":"364922","messageId":"70f4fab1-9352-49ac-f911-da3e5a0ca172@talktalk.net","threadId":"49958","inReplyTo":"CAGZ79kZVfMNELXuir+t9U8bSg2PVF=oX8aya-OqmRaP0gHRgFw@mail.gmail.com","subject":"Re: A case where diff.colorMoved=plain is more sensible than diff.colorMoved=zebra & others","fromName":"Phillip Wood","fromEmail":"phillip.wood@talktalk.net","sentAt":"2018-12-10T14:43:08Z","receivedAt":"2018-12-10T14:43:13Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 06/12/2018 18:11, Stefan Beller wrote:\n> On Thu, Dec 6, 2018 at 6:58 AM Phillip Wood <phillip.wood@talktalk.net> wrote:\n> \n>>> So is there some \"must be at least two consecutive lines\" condition for\n>>> not-plain, or is something else going on here?\n>>\n>> To be considered a block has to have 20 alphanumeric characters - see\n>> commit f0b8fb6e59 (\"diff: define block by number of alphanumeric chars\",\n>> 2017-08-15). This stops things like random '}' lines being marked as\n>> moved on their own.\n> \n> This is spot on.\n> \n> All but the \"plain\" mode use the concept of \"blocks\" of code\n> (there is even one mode called \"blocks\", which adds to the confusion).\n> \n>> It might be better to use some kind of frequency\n>> information (a bit like python's difflib junk parameter) instead so that\n>> (fairly) unique short lines also get marked properly.\n> \n> Yes that is what I was initially thinking about. However to have good\n> information, you'd need to index a whole lot (the whole repository,\n> i.e. all text blobs in existence?) to get an accurate picture of frequency\n> information, which I'd prefer to call entropy as I come from a background\n> familiar with https://en.wikipedia.org/wiki/Information_theory, I am not\n> sure where 'frequency information' comes from -- it sounds like the\n> same concept.\n> \n> Of course it is too expensive to run an operation O(repository size)\n> just for this diff, so maybe we could get away with some smaller\n> corpus to build up this information on what is sufficient for coloring.\n> \n> When only looking at the given diff, I would imagine that each line\n> would not carry a whole lot of information as its characters occur\n> rather frequently compared to the rest of the diff.\n\nI was thinking of using lines rather than characters as the unit of \ninformation (if that's the right phrase). I was hoping that seeing how \noften a given line occurs within the set of files being diffed would be \ngood enough to tell is if it is an \"interesting\" move or not. In the \nmean time I wonder if decreasing the block limit to 10 alphanumeric \ncharacters would be enough to prevent too much noise in the output \nwithout suppressing matches that it would be useful to highlight.\n\nBest Wishes\n\nPhillip\n\n> \n> Best,\n> Stefan\n> \n\n"},{"id":"364999","messageId":"CAGZ79kap2Mja61tktcR+cKU589vFKU6jp3pBFoUrhAv1Oi3=Qg@mail.gmail.com","threadId":"49958","inReplyTo":"70f4fab1-9352-49ac-f911-da3e5a0ca172@talktalk.net","subject":"Re: A case where diff.colorMoved=plain is more sensible than diff.colorMoved=zebra & others","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-12-11T00:54:54Z","receivedAt":"2018-12-11T00:55:09Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Dec 10, 2018 at 6:43 AM Phillip Wood <phillip.wood@talktalk.net> wrote:\n>\n> On 06/12/2018 18:11, Stefan Beller wrote:\n> > On Thu, Dec 6, 2018 at 6:58 AM Phillip Wood <phillip.wood@talktalk.net> wrote:\n> >\n> >>> So is there some \"must be at least two consecutive lines\" condition for\n> >>> not-plain, or is something else going on here?\n> >>\n> >> To be considered a block has to have 20 alphanumeric characters - see\n> >> commit f0b8fb6e59 (\"diff: define block by number of alphanumeric chars\",\n> >> 2017-08-15). This stops things like random '}' lines being marked as\n> >> moved on their own.\n> >\n> > This is spot on.\n> >\n> > All but the \"plain\" mode use the concept of \"blocks\" of code\n> > (there is even one mode called \"blocks\", which adds to the confusion).\n> >\n> >> It might be better to use some kind of frequency\n> >> information (a bit like python's difflib junk parameter) instead so that\n> >> (fairly) unique short lines also get marked properly.\n> >\n> > Yes that is what I was initially thinking about. However to have good\n> > information, you'd need to index a whole lot (the whole repository,\n> > i.e. all text blobs in existence?) to get an accurate picture of frequency\n> > information, which I'd prefer to call entropy as I come from a background\n> > familiar with https://en.wikipedia.org/wiki/Information_theory, I am not\n> > sure where 'frequency information' comes from -- it sounds like the\n> > same concept.\n> >\n> > Of course it is too expensive to run an operation O(repository size)\n> > just for this diff, so maybe we could get away with some smaller\n> > corpus to build up this information on what is sufficient for coloring.\n> >\n> > When only looking at the given diff, I would imagine that each line\n> > would not carry a whole lot of information as its characters occur\n> > rather frequently compared to the rest of the diff.\n>\n> I was thinking of using lines rather than characters as the unit of\n> information (if that's the right phrase). I was hoping that seeing how\n> often a given line occurs within the set of files being diffed would be\n> good enough to tell is if it is an \"interesting\" move or not.\n\nThat sounds reasonable. We should try that.\n\n> In the\n> mean time I wonder if decreasing the block limit to 10 alphanumeric\n> characters would be enough to prevent too much noise in the output\n> without suppressing matches that it would be useful to highlight.\n\nJonathan elegantly deferred the need to come up with data on why 20\nis a good choice, but rather claimed previous art in git-blame, see\nf0b8fb6e59 (diff: define block by number of alphanumeric chars,\n2017-08-15), which seems to say we'd want to follow the\nmodel of having a blame_entry_score (that counts the number\nof alnum() characters per line) and the\nBLAME_DEFAULT_MOVE_SCORE, which came into existence in\n4a0fc95f18 (git-pickaxe: introduce heuristics to avoid \"trivial\" chunks,\n2006-10-20), simply stating\n\n    The current heuristics are quite simple and may need to be\n    tweaked later, but we need to start somewhere.\n\nso I guess replacing 20 by 10 is totally doable, but the proof on\nwhy 10 is better than 20 is on you. ;-(\nProbably it doesn't need to be as fancy as in 433860f3d0\n(diff: improve positioning of add/delete blocks in diffs, 2016-09-05)\nbut we'd need to gather *some* data to convince\n(s/convince/fool/)  ourselves that it is better.\n\nIt could also be the case that we need to fine tune differently\nfor blame as for move detection, but we could still reuse\nsome code to process it.\n"}]}