{"thread":{"id":"46354","subject":"Weirdness with git change detection","startedAt":"2017-07-10T23:45:18Z","lastAt":"2017-08-18T11:57:59Z","messageCount":11,"participants":["Peter Eckersley","Torsten Bögershausen","Jeff King","Junio C Hamano","Michael J Gruber"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"324137","messageId":"CAOYJvnJ1FxevnWf+Aj0-5r6WfmfPkzWhOk_n4y4Yz7HvvfvaBQ@mail.gmail.com","threadId":"46354","inReplyTo":null,"subject":"Weirdness with git change detection","fromName":"Peter Eckersley","fromEmail":"peter.eckersley@gmail.com","sentAt":"2017-07-10T23:45:12Z","receivedAt":"2017-07-10T23:45:18Z","isPatch":false,"sender":{"key":"peter.eckersley@gmail.com","avatar":null},"body":"I have a local git repo that's in some weird state where changes\nappear to be detected by \"git diff\" and prevent operations like \"git\ncheckout\" from switching branches, but those changes are not removed\nby a \"git reset --hard\" or \"git stash\".\n\nHere's an example of the behaviour, with \"git reset --hard\" failing to\nclear a diff in the index:\n\nhttps://paste.debian.net/975811/\n\nHappy to collect additional debugging information if it's useful.\n-- \nPeter\n"},{"id":"324147","messageId":"d047a884-b698-c25b-59e9-3a2704f79fb3@web.de","threadId":"46354","inReplyTo":"CAOYJvnJ1FxevnWf+Aj0-5r6WfmfPkzWhOk_n4y4Yz7HvvfvaBQ@mail.gmail.com","subject":"Re: Weirdness with git change detection","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-07-11T04:15:17Z","receivedAt":"2017-07-11T04:14:40Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n\nOn 11/07/17 01:45, Peter Eckersley wrote:\n> I have a local git repo that's in some weird state where changes\n> appear to be detected by \"git diff\" and prevent operations like \"git\n> checkout\" from switching branches, but those changes are not removed\n> by a \"git reset --hard\" or \"git stash\".\n> \n> Here's an example of the behaviour, with \"git reset --hard\" failing to\n> clear a diff in the index:\n> \n> https://paste.debian.net/975811/\n> \n> Happy to collect additional debugging information if it's useful.\n> \nIf possible, we need to clone the repo and debug ourselfs - in other\nwords, the problem must be reproducible for others.\n\nIt the repo public ?\nWhich OS, Git version are you using ?\n\n"},{"id":"324154","messageId":"20170711070618.dq52o67ixvjudafg@sigill.intra.peff.net","threadId":"46354","inReplyTo":"d047a884-b698-c25b-59e9-3a2704f79fb3@web.de","subject":"Re: Weirdness with git change detection","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-11T07:06:19Z","receivedAt":"2017-07-11T07:06:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 11, 2017 at 06:15:17AM +0200, Torsten Bögershausen wrote:\n\n> On 11/07/17 01:45, Peter Eckersley wrote:\n> > I have a local git repo that's in some weird state where changes\n> > appear to be detected by \"git diff\" and prevent operations like \"git\n> > checkout\" from switching branches, but those changes are not removed\n> > by a \"git reset --hard\" or \"git stash\".\n> > \n> > Here's an example of the behaviour, with \"git reset --hard\" failing to\n> > clear a diff in the index:\n> > \n> > https://paste.debian.net/975811/\n> > \n> > Happy to collect additional debugging information if it's useful.\n> > \n> If possible, we need to clone the repo and debug ourselfs - in other\n> words, the problem must be reproducible for others.\n> \n> It the repo public ?\n\nIt looks like https://github.com/AI-metrics/AI-metrics.\n\nI notice it has a .gitattributes file with:\n\n  *.ipynb filter=clean_ipynb\n\nThere's a config snippet in the repo with:\n\n  [filter \"clean_ipynb\"]\n    clean = ipynb_drop_output\n    smudge = cat\n\nand the drop_output script is included. From the paste we can see that\nPeter was at commit c464aaa. Checking out that commit and running the\nscript shows that it produces the differences that Git is showing.\n\nThe problem is that the currently committed contents do not match the\noutput of the clean filter. So even when \"git reset --hard\" puts the\ncontent from the repository back into the working tree (putting it\nthrough the smudge filter, which is a noop), running the clean filter on\nthe result will always have a difference. Either the filter needs to be\ndisabled, or the cleaned contents committed.\n\n-Peff\n"},{"id":"324158","messageId":"CAOYJvnLCwR7CkG32E_r5BcVm5tFt5zsgYe7TUW3hVGBtQBHzGw@mail.gmail.com","threadId":"46354","inReplyTo":"20170711070618.dq52o67ixvjudafg@sigill.intra.peff.net","subject":"Re: Weirdness with git change detection","fromName":"Peter Eckersley","fromEmail":"peter.eckersley@gmail.com","sentAt":"2017-07-11T07:31:25Z","receivedAt":"2017-07-11T07:31:44Z","isPatch":false,"sender":{"key":"peter.eckersley@gmail.com","avatar":null},"body":"I did try to test that hypothesis by editing the filter to be a no-op,\nbut it's possible I go that wrong. My apologies for bugging the list!\n\nOn 11 July 2017 at 00:06, Jeff King <peff@peff.net> wrote:\n> On Tue, Jul 11, 2017 at 06:15:17AM +0200, Torsten Bögershausen wrote:\n>\n>> On 11/07/17 01:45, Peter Eckersley wrote:\n>> > I have a local git repo that's in some weird state where changes\n>> > appear to be detected by \"git diff\" and prevent operations like \"git\n>> > checkout\" from switching branches, but those changes are not removed\n>> > by a \"git reset --hard\" or \"git stash\".\n>> >\n>> > Here's an example of the behaviour, with \"git reset --hard\" failing to\n>> > clear a diff in the index:\n>> >\n>> > https://paste.debian.net/975811/\n>> >\n>> > Happy to collect additional debugging information if it's useful.\n>> >\n>> If possible, we need to clone the repo and debug ourselfs - in other\n>> words, the problem must be reproducible for others.\n>>\n>> It the repo public ?\n>\n> It looks like https://github.com/AI-metrics/AI-metrics.\n>\n> I notice it has a .gitattributes file with:\n>\n>   *.ipynb filter=clean_ipynb\n>\n> There's a config snippet in the repo with:\n>\n>   [filter \"clean_ipynb\"]\n>     clean = ipynb_drop_output\n>     smudge = cat\n>\n> and the drop_output script is included. From the paste we can see that\n> Peter was at commit c464aaa. Checking out that commit and running the\n> script shows that it produces the differences that Git is showing.\n>\n> The problem is that the currently committed contents do not match the\n> output of the clean filter. So even when \"git reset --hard\" puts the\n> content from the repository back into the working tree (putting it\n> through the smudge filter, which is a noop), running the clean filter on\n> the result will always have a difference. Either the filter needs to be\n> disabled, or the cleaned contents committed.\n>\n> -Peff\n\n\n\n-- \nPeter\n"},{"id":"324159","messageId":"20170711073433.ghcz3lku3ux54fib@sigill.intra.peff.net","threadId":"46354","inReplyTo":"CAOYJvnLCwR7CkG32E_r5BcVm5tFt5zsgYe7TUW3hVGBtQBHzGw@mail.gmail.com","subject":"Re: Weirdness with git change detection","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-11T07:34:33Z","receivedAt":"2017-07-11T07:35:40Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 11, 2017 at 12:31:25AM -0700, Peter Eckersley wrote:\n\n> I did try to test that hypothesis by editing the filter to be a no-op,\n> but it's possible I go that wrong. My apologies for bugging the list!\n\nNo problem. I actually think it would be interesting if Git could\nsomehow detect and warn about this situation. But the obvious way to do\nthat would be to re-run the clean filter directly after checkout. And\ndoing that all the time is expensive.\n\nPerhaps some kind of \"lint\" program would be interesting to warn of\npossible misconfigurations. Of course people would have to run it for it\nto be useful. :)\n\n-Peff\n"},{"id":"324164","messageId":"decbd316-5194-120d-824e-b5cee1f0822c@web.de","threadId":"46354","inReplyTo":"20170711073433.ghcz3lku3ux54fib@sigill.intra.peff.net","subject":"Re: Weirdness with git change detection","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-07-11T08:20:43Z","receivedAt":"2017-07-11T08:20:16Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n\nOn 11/07/17 09:34, Jeff King wrote:\n> On Tue, Jul 11, 2017 at 12:31:25AM -0700, Peter Eckersley wrote:\n> \n>> I did try to test that hypothesis by editing the filter to be a no-op,\n>> but it's possible I go that wrong. My apologies for bugging the list!\n\nActually I like this kind of feedback, to see how people are using Git,\nincluding the problems they have.\n\n> \n> No problem. I actually think it would be interesting if Git could\n> somehow detect and warn about this situation. But the obvious way to do\n> that would be to re-run the clean filter directly after checkout. And\n> doing that all the time is expensive.\n\nWould it be possible to limit the round-trip-check to \"git reset --hard\" ?\nIf yes, possibly many users are willing to pay the price, if Git tells\nthe \"your filters don't round-trip\". (Including CRLF conversions)\n\n> \n> Perhaps some kind of \"lint\" program would be interesting to warn of\n> possible misconfigurations. Of course people would have to run it for it\n> to be useful. :)\n\nWhat do you have in mind here ?\nDon't we need to run some content through the filter(s)?\n"},{"id":"324165","messageId":"20170711082435.a53iuw26oxawcfyq@sigill.intra.peff.net","threadId":"46354","inReplyTo":"decbd316-5194-120d-824e-b5cee1f0822c@web.de","subject":"Re: Weirdness with git change detection","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-11T08:24:35Z","receivedAt":"2017-07-11T08:24:43Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 11, 2017 at 10:20:43AM +0200, Torsten Bögershausen wrote:\n\n> > No problem. I actually think it would be interesting if Git could\n> > somehow detect and warn about this situation. But the obvious way to do\n> > that would be to re-run the clean filter directly after checkout. And\n> > doing that all the time is expensive.\n> \n> Would it be possible to limit the round-trip-check to \"git reset --hard\" ?\n> If yes, possibly many users are willing to pay the price, if Git tells\n> the \"your filters don't round-trip\". (Including CRLF conversions)\n\nAnything's possible, I suppose. But I don't think I'd want that feature\nturned on myself.\n\n> > Perhaps some kind of \"lint\" program would be interesting to warn of\n> > possible misconfigurations. Of course people would have to run it for it\n> > to be useful. :)\n> \n> What do you have in mind here ?\n> Don't we need to run some content through the filter(s)?\n\nI was thinking of a tool that could run a series of checks on the\nrepository and nag about potential problems. One of them could be doing\na round-trip repo->clean->smudge for each file.\n\nAnother one might be warning about files that differ only in case.\n\nThe idea being that users could run \"git lint\" if they suspect something\nfunny is going on. I dunno. It may be a dead-end. Most such\noddities are better detected and handled during actual git operations if\nwe can. So this would really just be for things that are too expensive\nto detect in normal operations.\n\n-Peff\n"},{"id":"324200","messageId":"xmqqy3rv2cfv.fsf@gitster.mtv.corp.google.com","threadId":"46354","inReplyTo":"CAOYJvnJ1FxevnWf+Aj0-5r6WfmfPkzWhOk_n4y4Yz7HvvfvaBQ@mail.gmail.com","subject":"Re: Weirdness with git change detection","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-11T15:57:56Z","receivedAt":"2017-07-11T15:58:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Peter Eckersley <peter.eckersley@gmail.com> writes:\n\n> I have a local git repo that's in some weird state where changes\n> appear to be detected by \"git diff\" and prevent operations like \"git\n> checkout\" from switching branches, but those changes are not removed\n> by a \"git reset --hard\" or \"git stash\".\n>\n> Here's an example of the behaviour, with \"git reset --hard\" failing to\n> clear a diff in the index:\n>\n> https://paste.debian.net/975811/\n>\n> Happy to collect additional debugging information if it's useful.\n\nDo you have any funny clean-smudge pair that do not round-trip?\n"},{"id":"324202","messageId":"xmqqpod72c5p.fsf@gitster.mtv.corp.google.com","threadId":"46354","inReplyTo":"xmqqy3rv2cfv.fsf@gitster.mtv.corp.google.com","subject":"Re: Weirdness with git change detection","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-11T16:04:02Z","receivedAt":"2017-07-11T16:04:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Peter Eckersley <peter.eckersley@gmail.com> writes:\n>\n>> I have a local git repo that's in some weird state where changes\n>> appear to be detected by \"git diff\" and prevent operations like \"git\n>> checkout\" from switching branches, but those changes are not removed\n>> by a \"git reset --hard\" or \"git stash\".\n>>\n>> Here's an example of the behaviour, with \"git reset --hard\" failing to\n>> clear a diff in the index:\n>>\n>> https://paste.debian.net/975811/\n>>\n>> Happy to collect additional debugging information if it's useful.\n>\n> Do you have any funny clean-smudge pair that do not round-trip?\n\nAh, nevermind.  Peff's analysis looks correct.  Thanks for a report\nto provoke a good discussion.\n"},{"id":"326677","messageId":"fa4b367d-a984-6f42-a2bd-6bfcabed7c2a@grubix.eu","threadId":"46354","inReplyTo":"20170711082435.a53iuw26oxawcfyq@sigill.intra.peff.net","subject":"Re: Weirdness with git change detection","fromName":"Michael J Gruber","fromEmail":"git@grubix.eu","sentAt":"2017-08-18T06:56:03Z","receivedAt":"2017-08-18T06:56:13Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Jeff King venit, vidit, dixit 11.07.2017 10:24:\n> On Tue, Jul 11, 2017 at 10:20:43AM +0200, Torsten Bögershausen wrote:\n> \n>>> No problem. I actually think it would be interesting if Git could\n>>> somehow detect and warn about this situation. But the obvious way to do\n>>> that would be to re-run the clean filter directly after checkout. And\n>>> doing that all the time is expensive.\n>>\n>> Would it be possible to limit the round-trip-check to \"git reset --hard\" ?\n>> If yes, possibly many users are willing to pay the price, if Git tells\n>> the \"your filters don't round-trip\". (Including CRLF conversions)\n> \n> Anything's possible, I suppose. But I don't think I'd want that feature\n> turned on myself.\n> \n>>> Perhaps some kind of \"lint\" program would be interesting to warn of\n>>> possible misconfigurations. Of course people would have to run it for it\n>>> to be useful. :)\n>>\n>> What do you have in mind here ?\n>> Don't we need to run some content through the filter(s)?\n> \n> I was thinking of a tool that could run a series of checks on the\n> repository and nag about potential problems. One of them could be doing\n> a round-trip repo->clean->smudge for each file.\n> \n> Another one might be warning about files that differ only in case.\n> \n> The idea being that users could run \"git lint\" if they suspect something\n> funny is going on. I dunno. It may be a dead-end. Most such\n> oddities are better detected and handled during actual git operations if\n> we can. So this would really just be for things that are too expensive\n> to detect in normal operations.\n> \n> -Peff\n> \n\nTypically, that problem arises when you turn a filter on or off at some\npoint in your history. Since \"attributes\" can come from various sources,\nespecially the versioned \".gitattributes\" file, unversioned per-repo\n.git/info/attributes, and global attributes, \"git diff\" may apply\ndifferent attributes depending on what you diff (versioned blob, workdir\nfile, out-of-tree file).\n\nThis is not made easier by the fact that unversioned config (per repo,\nper user, global) defines the filter action, and that even upgrades of\nyour filter tools may change the output. So, \"filter off/on\" is by no\nmeans the only possible source of discrepancies.\n\nI've found that when I decide to use a filter like that, the best\napproach is to either apply it retroactively (filter-branch,\nunversionsed attributes, that is clean all stored blobs) or make a\ncommit where I specifically note the switch (versioned .gitattributes\nplus affected blob changes) and what config should go along with it.\n\nAll of this is difficult to check or correct automatically, since it\ndepends on user decisions.\n\nAbout the only thing we could do is checking that\n\"clean(smudge(foo))=clean(foo)\" at a specific \"point in time\"\n(attributes, config) for specific foo, but that wouldn't catch the case\nabove, even if we iterated over all commits which affect files that the\nfilter (currently) applies to.\n\nKeep in mind that filters are a killer feature, so if you shoot yourself\nin the foot: it could have come worse ;)\n\nMichael\n"},{"id":"326687","messageId":"20170818115749.vgfcaa44az5upm2s@sigill.intra.peff.net","threadId":"46354","inReplyTo":"fa4b367d-a984-6f42-a2bd-6bfcabed7c2a@grubix.eu","subject":"Re: Weirdness with git change detection","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-08-18T11:57:49Z","receivedAt":"2017-08-18T11:57:59Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 18, 2017 at 08:56:03AM +0200, Michael J Gruber wrote:\n\n> > The idea being that users could run \"git lint\" if they suspect something\n> > funny is going on. I dunno. It may be a dead-end. Most such\n> > oddities are better detected and handled during actual git operations if\n> > we can. So this would really just be for things that are too expensive\n> > to detect in normal operations.\n> \n> Typically, that problem arises when you turn a filter on or off at some\n> point in your history. Since \"attributes\" can come from various sources,\n> especially the versioned \".gitattributes\" file, unversioned per-repo\n> .git/info/attributes, and global attributes, \"git diff\" may apply\n> different attributes depending on what you diff (versioned blob, workdir\n> file, out-of-tree file).\n> [...]\n\nYeah, I agree that we cannot catch every problematic case (for all the\nreason you give here). It does seem like a large number of problems\npeople report have to do with checkout of in-tree .gitattributes,\nthough. So my thinking was if we could cover that case, it might help\npeople (even if we leave many problems unnoticed).\n\nBut...\n\n> I've found that when I decide to use a filter like that, the best\n> approach is to either apply it retroactively (filter-branch,\n> unversionsed attributes, that is clean all stored blobs) or make a\n> commit where I specifically note the switch (versioned .gitattributes\n> plus affected blob changes) and what config should go along with it.\n\nOne problem is that people need to know to run the lint command. And if\nthey know enough that this is a problem worthy of checking via a linter,\nthen they could perhaps just as easily do the in-tree blob changes.\n\nI say \"perhaps\" because I don't think it's as easy as running a single\n\"git fix-my-stale-blobs\". I wonder if rather than a linter, we ought to\njust have an option to \"git checkout\" or something to ignore stat data\nand re-checkout any entries for which convert_to_working_tree() isn't a\nnoop.\n\nThat serves both as a repair tool and as a linter (since running \"git\ndiff\" on the result would show you what needs to be fixed). It wouldn't\nsolve the user-education problem, but at least it would give a simple\nsolution that could be passed along to users.\n\nI dunno. I don't do line ending conversion, so I don't really run into\nthis issue myself.\n\n-Peff\n"}]}