{"thread":{"id":"54330","subject":"git blame --ignore-rev does not work","startedAt":"2020-09-30T21:15:50Z","lastAt":"2020-10-03T00:56:32Z","messageCount":5,"participants":["Harrison McCullough","René Scharfe","Barret Rhoden"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"406713","messageId":"CAHLeu+x-z4ntmBezcVUWssZrCm03Md6ZR8-ZQmjkeB5YT89caQ@mail.gmail.com","threadId":"54330","inReplyTo":null,"subject":"git blame --ignore-rev does not work","fromName":"Harrison McCullough","fromEmail":"mccullough.harrison@gmail.com","sentAt":"2020-09-30T21:15:32Z","receivedAt":"2020-09-30T21:15:50Z","isPatch":false,"sender":{"key":"mccullough.harrison@gmail.com","avatar":null},"body":"What did you do before the bug happened?\n\n1. Commit changes to <FILE>\n2. Observe that this commit has a hash of <HASH>, e.g. through git rev-parse\n   HEAD\n3. Run `echo <HASH> > .git-blame-ignore-revs`\n4. Run `git config blame.ignoreRevsFile .git-blame-ignore-revs`\n5. Run `git blame <FILE>`\n6. Run `git blame --ignore-revs-file=.git-blame-ignore-revs <FILE>`\n7. Run `git blame --ignore-rev=<HASH> <FILE>`\n\n\nWhat did you expect to happen? (Expected behavior)\n\nThe three git blame commands should attribute each line of the source file to a\ncommit, but none of those commits should be the one specified by <HASH>.\n\n\nWhat happened instead? (Actual behavior)\n\nAll three git blame commands included lines attributed to <HASH>.\n\n\nWhat's different between what you expected and what actually happened?\n\nThe commit identified by <HASH> was not ignored.\n\n\nAnything else you want to add:\n\nI tried this in a brand new repository and everything worked as expected. I do\nnot know why it is only failing in this repository. It is a large repository I\nuse for work, but I'm using the same version of Git in both places.\n\n\n[System Info]\ngit version:\ngit version 2.28.0\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Mon Apr 27 20:09:39\nPDT 2020; root:xnu-4903.278.35~1/RELEASE_X86_64 x86_64\ncompiler info: clang: 11.0.0 (clang-1100.0.33.17)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /usr/local/bin/bash\n\n\n[Enabled Hooks]\npost-commit\npost-checkout\npost-merge\npre-push\n"},{"id":"406846","messageId":"d805f025-fbfb-0249-a50c-ff857dc2e29d@web.de","threadId":"54330","inReplyTo":"CAHLeu+x-z4ntmBezcVUWssZrCm03Md6ZR8-ZQmjkeB5YT89caQ@mail.gmail.com","subject":"Re: git blame --ignore-rev does not work","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-10-02T21:40:27Z","receivedAt":"2020-10-02T21:40:47Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 30.09.20 um 23:15 schrieb Harrison McCullough:\n> What did you do before the bug happened?\n>\n> 1. Commit changes to <FILE>\n> 2. Observe that this commit has a hash of <HASH>, e.g. through git rev-parse\n>    HEAD\n> 3. Run `echo <HASH> > .git-blame-ignore-revs`\n> 4. Run `git config blame.ignoreRevsFile .git-blame-ignore-revs`\n> 5. Run `git blame <FILE>`\n> 6. Run `git blame --ignore-revs-file=.git-blame-ignore-revs <FILE>`\n> 7. Run `git blame --ignore-rev=<HASH> <FILE>`\n>\n>\n> What did you expect to happen? (Expected behavior)\n>\n> The three git blame commands should attribute each line of the source file to a\n> commit, but none of those commits should be the one specified by <HASH>.\n>\n>\n> What happened instead? (Actual behavior)\n>\n> All three git blame commands included lines attributed to <HASH>.\n>\n>\n> What's different between what you expected and what actually happened?\n>\n> The commit identified by <HASH> was not ignored.\n\nWell, your expectation sounds reasonable, but apparently it's not that\neasy.  Consider this sentence from the description of --ignore-rev on\nthe manpage of git blame:\n\n    If the `blame.markUnblamableLines` config option is set, then those\n    lines touched by an ignored commit that we could not attribute to\n    another revision are marked with a '*'.\n\nSo some commits just cannot be ignored by the current version of git\nblame.  The commit message of ae3f36dea1 (blame: add the ability to\nignore commits and their changes, 2019-05-15), which introduced that\nfeature, mentions an example.  And this silly script finds that 365 of\nthe 765 commits blamed for Git's own Makefile are examples as well:\n\n  file=Makefile\n  rev=v2.28.0\n  hashes=$(git blame \"$rev\" \"$file\" | awk '{print $1}' | sort -u)\n  echo \"$hashes\" | wc -l\n  for hash in $hashes\n  do\n     if git blame --ignore-rev=$hash \"$rev\" \"$file\" | grep -q \"^$hash \"\n     then\n       echo $hash\n     fi\n  done | wc -l\n\nI don't know if these revisions are not ignored due to bugs or because\nthe feature just isn't strong enough, yet, but I would expect your\nparticular case to be represented by at least one of these...\n\n> Anything else you want to add:\n>\n> I tried this in a brand new repository and everything worked as expected. I do\n> not know why it is only failing in this repository. It is a large repository I\n> use for work, but I'm using the same version of Git in both places.\n\n... so this might not be a problem, as finding public examples seems\nto be easy.\n\n>\n>\n> [System Info]\n> git version:\n> git version 2.28.0\n> cpu: x86_64\n> no commit associated with this build\n> sizeof-long: 8\n> sizeof-size_t: 8\n> shell-path: /bin/sh\n> uname: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Mon Apr 27 20:09:39\n> PDT 2020; root:xnu-4903.278.35~1/RELEASE_X86_64 x86_64\n> compiler info: clang: 11.0.0 (clang-1100.0.33.17)\n> libc info: no libc information available\n> $SHELL (typically, interactive shell): /usr/local/bin/bash\n>\n>\n> [Enabled Hooks]\n> post-commit\n> post-checkout\n> post-merge\n> pre-push\n>\n\n"},{"id":"406851","messageId":"3e9b34f9-f61e-f3f1-45a3-6352641e434a@google.com","threadId":"54330","inReplyTo":"d805f025-fbfb-0249-a50c-ff857dc2e29d@web.de","subject":"Re: git blame --ignore-rev does not work","fromName":"Barret Rhoden","fromEmail":"brho@google.com","sentAt":"2020-10-02T22:44:01Z","receivedAt":"2020-10-02T22:44:06Z","isPatch":false,"sender":{"key":"brho@google.com","avatar":null},"body":"Hi -\n\nOn 10/2/20 5:40 PM, René Scharfe wrote:\n[snip]\n> I don't know if these revisions are not ignored due to bugs or because\n> the feature just isn't strong enough, yet, but I would expect your\n> particular case to be represented by at least one of these...\n\nCorrect.\n\nWhen skipping a revision, the algorithm attempts to find another \nrevision that could be responsible for the change.  But it might not be \nable to find anything.  Consider a commit that just adds a few lines to \na file with only 'foo' and 'bar':\n\ncommit: \"Adding Lines\"\n-------------\n  foo\n+No commit\n+ever touched\n+these lines\n  bar\n\nIf we ignored that revision, which commit do we assign those lines to? \nIf they were \"similar\" to the existing lines, then the algorithm might \nmatch.  But in general, we can't find 'correct' (as defined by a user) \nmatches for arbitrary changes.\n\nI usually run git with these settings:\n\n[blame]\n         ignorerevsfile = .git-blame-ignore-revs\n         markIgnoredLines = true\n         markUnblamableLines = true\n\nWhich points out when --ignore-revs is doing something.\n\nThanks,\n\nBarret\n\n\n"},{"id":"406852","messageId":"CAHLeu+zSaTwPEDQ=CuFua0NdEppM+OjFaREU+Yiy9udK1OUK4w@mail.gmail.com","threadId":"54330","inReplyTo":"3e9b34f9-f61e-f3f1-45a3-6352641e434a@google.com","subject":"Re: git blame --ignore-rev does not work","fromName":"Harrison McCullough","fromEmail":"mccullough.harrison@gmail.com","sentAt":"2020-10-02T22:52:45Z","receivedAt":"2020-10-02T22:52:59Z","isPatch":false,"sender":{"key":"mccullough.harrison@gmail.com","avatar":null},"body":"Thank you for your feedback! I do have some more information to\nprovide that is confusing.\n\nI tried running `git blame -w`, and this correctly ignores the\nrevision I tried to ignore with `--ignore-rev`, etc. So it appears\nthat the algorithm to attribute lines to commits is capable of\nignoring the commit in question (in the lines I've inspected) but it's\nnot doing it when I use the \"ignore-rev\" capability—only the \"ignore\nwhitespace changes\" capability.\n\nDoes anyone have any ideas about why that may be the case? Does the\n\"ignore whitespace\" and \"ignore commit\" algorithms use different\nlogic? I would have assumed that they shared most of the logic.\n\nI would love to provide a concrete example, but the only time I've\nbeen able to reproduce this is with proprietary code. I'll try to\ncreate a new repository with a similar commit and see if I can ignore\nit there.\n\nFor the information of those interested, the commit I'm trying to\nignore is a \"reformat the world\" commit. We introduced the tool\n\"astyle\" into our codebase, and as part of that effort I ran astyle\nover our entire codebase.\n\nIs it possible that the commit isn't being ignored because it's too\nbig? It did change over 1300 files....\n\nOn Fri, Oct 2, 2020 at 4:44 PM Barret Rhoden <brho@google.com> wrote:\n>\n> Hi -\n>\n> On 10/2/20 5:40 PM, René Scharfe wrote:\n> [snip]\n> > I don't know if these revisions are not ignored due to bugs or because\n> > the feature just isn't strong enough, yet, but I would expect your\n> > particular case to be represented by at least one of these...\n>\n> Correct.\n>\n> When skipping a revision, the algorithm attempts to find another\n> revision that could be responsible for the change.  But it might not be\n> able to find anything.  Consider a commit that just adds a few lines to\n> a file with only 'foo' and 'bar':\n>\n> commit: \"Adding Lines\"\n> -------------\n>   foo\n> +No commit\n> +ever touched\n> +these lines\n>   bar\n>\n> If we ignored that revision, which commit do we assign those lines to?\n> If they were \"similar\" to the existing lines, then the algorithm might\n> match.  But in general, we can't find 'correct' (as defined by a user)\n> matches for arbitrary changes.\n>\n> I usually run git with these settings:\n>\n> [blame]\n>          ignorerevsfile = .git-blame-ignore-revs\n>          markIgnoredLines = true\n>          markUnblamableLines = true\n>\n> Which points out when --ignore-revs is doing something.\n>\n> Thanks,\n>\n> Barret\n>\n>\n\n\n-- \n-Harrison McCullough\n"},{"id":"406853","messageId":"c77eb54a-ecfd-508b-ed6a-030e66d8e257@google.com","threadId":"54330","inReplyTo":"CAHLeu+zSaTwPEDQ=CuFua0NdEppM+OjFaREU+Yiy9udK1OUK4w@mail.gmail.com","subject":"Re: git blame --ignore-rev does not work","fromName":"Barret Rhoden","fromEmail":"brho@google.com","sentAt":"2020-10-03T00:56:27Z","receivedAt":"2020-10-03T00:56:32Z","isPatch":false,"sender":{"key":"brho@google.com","avatar":null},"body":"Hi -\n\nOn 10/2/20 6:52 PM, Harrison McCullough wrote:\n> Thank you for your feedback! I do have some more information to\n> provide that is confusing.\n> \n> I tried running `git blame -w`, and this correctly ignores the\n> revision I tried to ignore with `--ignore-rev`, etc. So it appears\n> that the algorithm to attribute lines to commits is capable of\n> ignoring the commit in question (in the lines I've inspected) but it's\n> not doing it when I use the \"ignore-rev\" capability—only the \"ignore\n> whitespace changes\" capability.\n> \n> Does anyone have any ideas about why that may be the case? Does the\n> \"ignore whitespace\" and \"ignore commit\" algorithms use different\n> logic? I would have assumed that they shared most of the logic.\n\nYeah, the logic is a little different.  IIRC, the \"whitespace ignore\" \naffects individual diff_hunks that are generated during the blame \nprocess, generated by calling into xdiff.  That's when blame tries and \nfigure out what lines change from target/child to parent.  So the \"blame \nchunk\" that gets analyzed is different, depending on whitespace-ignore \nor not.\n\nThe \"blame ignore\" logic happens after that, and it operates on the \noutput of xdiff.  If the 'target' is one of the ignored commits, it \nlooks at those chunks produced by xdiff (differences from target commit \nto parent) and for each line in the target commit, attempts to figure \nout what line in the parent to match it to.\n\nThe matching process is imperfect.  It uses a fingerprinting algorithm \nto find a likely candidate line in the parent's diff hunk.  The \nfingerprinting is based on matching the two-character pairs in each \nstring.  (I didn't write the fingerprinting, btw).  It does some ranking \nof the lines, picks the best one, and a few other 'search quality' \nthings.  etc.  If it fails to find a matching line in the hunk, it'll do \na simple O(n) scan in the entire file.  But when it looks in the entire \nfile, it has a threshold of similarity, so you don't match arbitrary \nthings.  (Threshold is 10 two-letter pairs, I think).\n\nAnyway, my guess is that when you use the ignore-whitespace option, it \nchanges the diff hunks enough that blame_chunk() gives a different \nresult.  This could be because there are larger diff hunks (more search \nspace for the \"good\" initial scan).  Or maybe because you have a \ndifferent set of two-character pairs or something.\n\n> I would love to provide a concrete example, but the only time I've\n> been able to reproduce this is with proprietary code. I'll try to\n> create a new repository with a similar commit and see if I can ignore\n> it there.\n\nI'd be glad to take a look, though I don't know how fixable it is. \nMy guess is you're hitting right on the edge of the \"find a similar line \ninside a hunk\" and \"couldn't find a good change in the entire file\" \nthreshold, and any change would just be tuning it for your repo.  But \nmaybe not, and I'd like for --ignore-rev to work for you.\n\n> For the information of those interested, the commit I'm trying to\n> ignore is a \"reformat the world\" commit. We introduced the tool\n> \"astyle\" into our codebase, and as part of that effort I ran astyle\n> over our entire codebase.\n\nSame situation for me - reformatted a lot of code and didn't want to \nbreak blame.  =)  That was the original motivation.  I've actually used \nit a lot for other things now, such as finding out which commit changed \na line in a particular way.  git blame, look around.  git show XXX, \nrealize i want and older commit, git blame --ignore-rev XXX, etc.\n\n> Is it possible that the commit isn't being ignored because it's too\n> big? It did change over 1300 files....\n\nThat should be OK.  The algorithm doesn't care about the other files in \nthe commit - only the one that you are blaming.\n\nThanks,\n\nBarret\n\n\n> \n> On Fri, Oct 2, 2020 at 4:44 PM Barret Rhoden <brho@google.com> wrote:\n>>\n>> Hi -\n>>\n>> On 10/2/20 5:40 PM, René Scharfe wrote:\n>> [snip]\n>>> I don't know if these revisions are not ignored due to bugs or because\n>>> the feature just isn't strong enough, yet, but I would expect your\n>>> particular case to be represented by at least one of these...\n>>\n>> Correct.\n>>\n>> When skipping a revision, the algorithm attempts to find another\n>> revision that could be responsible for the change.  But it might not be\n>> able to find anything.  Consider a commit that just adds a few lines to\n>> a file with only 'foo' and 'bar':\n>>\n>> commit: \"Adding Lines\"\n>> -------------\n>>    foo\n>> +No commit\n>> +ever touched\n>> +these lines\n>>    bar\n>>\n>> If we ignored that revision, which commit do we assign those lines to?\n>> If they were \"similar\" to the existing lines, then the algorithm might\n>> match.  But in general, we can't find 'correct' (as defined by a user)\n>> matches for arbitrary changes.\n>>\n>> I usually run git with these settings:\n>>\n>> [blame]\n>>           ignorerevsfile = .git-blame-ignore-revs\n>>           markIgnoredLines = true\n>>           markUnblamableLines = true\n>>\n>> Which points out when --ignore-revs is doing something.\n>>\n>> Thanks,\n>>\n>> Barret\n>>\n>>\n> \n> \n\n"}]}