{"thread":{"id":"41652","subject":"git smudge filter fails","startedAt":"2016-03-09T18:29:31Z","lastAt":"2016-03-15T16:48:37Z","messageCount":7,"participants":["Stephen Morton","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"280481","messageId":"CAH8BJxHwxp2BtzGBqi6J24Kh0TTGEdCx=-Scu+bx5N-ZVpBZNQ@mail.gmail.com","threadId":"41652","inReplyTo":null,"subject":"git smudge filter fails","fromName":"Stephen Morton","fromEmail":"stephen.c.morton@gmail.com","sentAt":"2016-03-09T18:29:31Z","receivedAt":"2016-03-09T18:29:31Z","isPatch":false,"sender":{"key":"stephen.c.morton@gmail.com","avatar":"https://gravatar.com/avatar/3b826f06cabc070cbd5d53071880caad8badc3b29e0789b510353faca59ffb7b?d=mp&s=160"},"body":"A git smudge filter, at least one that relies on the results from 'git\nlog' does not seem to work\non file A when doing a 'git update' from a revision where file A\ndoesn't exist to a revision where\nit does exist.\n\nBelow is a simple recipe to reproduce.\n\nThis appears to me to be a bug. If not, why is it expected and is\nthere anything I can do to\nwork around this behaviour?\n\nSteve\n\nmkdir git_test\ncd git_test/\ngit init .\ntouch bar.c\ngit add .\ngit commit -am \"Initial commit. foo.c not here yet.\"\ngit tag no_foo\n\ntouch foo.c\ngit add .\ngit commit -am \"Add foo, no content\"\necho 'Date is $Date$' >> foo.c\ngit commit -am \"Add date to foo.c\"\necho 'foo.c filter=dater' > .git/info/attributes\ngit config --local filter.dater.smudge 'myDate=`git log\n--pretty=format:\"%cd\" --date=iso -1 -- %f`; sed -e\n\"s/\\(\\\\$\\)Date[^\\\\$]*\\\\$/\\1Date: $myDate \\\\$/g\"'\ngit config --local filter.dater.clean 'sed -e\n\"s/\\(\\\\$\\)Date[^\\\\$]*\\\\$/\\1Date\\\\$/g\"'\nrm -f foo.c\ngit checkout -- foo.c\ncat foo.c\n# observe keyword expansion\n\ngit checkout no_foo\ngit checkout master\ncat foo.c\n#observe keyword expansion lost\n"},{"id":"280528","messageId":"20160310015939.GA12709@sigill.intra.peff.net","threadId":"41652","inReplyTo":"CAH8BJxHwxp2BtzGBqi6J24Kh0TTGEdCx=-Scu+bx5N-ZVpBZNQ@mail.gmail.com","subject":"Re: git smudge filter fails","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-03-10T01:59:40Z","receivedAt":"2016-03-10T01:59:40Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 09, 2016 at 01:29:31PM -0500, Stephen Morton wrote:\n\n> git config --local filter.dater.smudge 'myDate=`git log\n> --pretty=format:\"%cd\" --date=iso -1 -- %f`; sed -e\n> \"s/\\(\\\\$\\)Date[^\\\\$]*\\\\$/\\1Date: $myDate \\\\$/g\"'\n\nYour filter is running \"git log\" without a revision parameter, which\nmeans it is looking at HEAD.\n\nAnd here....\n\n> git checkout no_foo\n> git checkout master\n> cat foo.c\n> #observe keyword expansion lost\n\nYou are expecting this second one to do:\n\n 1. Switch HEAD to \"master\".\n\n 2. Checkout files which need updating. Looking at HEAD in your filter\n then examines \"master\", and you see the commit timestamp of the\n destination.\n\nBut that isn't how it is implemented. Checkout will handle the file\ncheckout _first_, as that is the part that is likely to run into\nproblems (e.g., rejecting a switch because it would lose changes in the\nworking tree). Only at the very end, after the change to the working\ntree has succeeded, do we update HEAD.\n\nI think the order you are expecting is conceptually cleaner, but I don't\nthink we would want to switch it around (for reasons of efficiency and\nrobustness). And I don't think we would want to make a promise about the\nordering to callers either way, as it binds our implementation.\n\nSo is there a way to reliably know the destination of a checkout? My\nfirst thought was that we could add a placeholder similar to \"%f\" that\nyour filter could use. I'm not sure how we would handle the corner cases\nthere, though (e.g., do we always have a \"destination\" to report? If\nnot, what do we give the script?).\n\nI suspect you could also hack something together with a post-checkout\nscript, though it would probably be a lot less efficient (and might also\nhave some weird corner cases).\n\n-Peff\n"},{"id":"280558","messageId":"CAH8BJxFmAQtoF+1Q7Ub5qWnz5UewrPS4e8JQWms254hO_E05Hw@mail.gmail.com","threadId":"41652","inReplyTo":"20160310015939.GA12709@sigill.intra.peff.net","subject":"Re: git smudge filter fails","fromName":"Stephen Morton","fromEmail":"stephen.c.morton@gmail.com","sentAt":"2016-03-10T14:45:19Z","receivedAt":"2016-03-10T14:45:19Z","isPatch":false,"sender":{"key":"stephen.c.morton@gmail.com","avatar":"https://gravatar.com/avatar/3b826f06cabc070cbd5d53071880caad8badc3b29e0789b510353faca59ffb7b?d=mp&s=160"},"body":"I am a bit confused because this is basically the example used in\nProGit [1] and it is fundamentally broken. In fact, if I understand\ncorrectly, this means that smudge filters cannot be relied upon to\nprovide any 'keyword expansion' type tasks because they will all by\nnature have to query the file with 'git log'.\n\n\n(Note that although in my example I used 'git checkout', with an only\nslightly more complicated example I can make it fail on 'git pull'\nwhich is perhaps a much more realistic use case. That was probably\nimplied in your answer, I just wanted to mention it.)\n\nSteve\n\n[1] https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes\n\n\n\nOn Wed, Mar 9, 2016 at 8:59 PM, Jeff King wrote:\n> On Wed, Mar 09, 2016 at 01:29:31PM -0500, Stephen Morton wrote:\n>\n>> git config --local filter.dater.smudge 'myDate=`git log\n>> --pretty=format:\"%cd\" --date=iso -1 -- %f`; sed -e\n>> \"s/\\(\\\\$\\)Date[^\\\\$]*\\\\$/\\1Date: $myDate \\\\$/g\"'\n>\n> Your filter is running \"git log\" without a revision parameter, which\n> means it is looking at HEAD.\n>\n> And here....\n>\n>> git checkout no_foo\n>> git checkout master\n>> cat foo.c\n>> #observe keyword expansion lost\n>\n> You are expecting this second one to do:\n>\n> 1. Switch HEAD to \"master\".\n>\n> 2. Checkout files which need updating. Looking at HEAD in your filter\n> then examines \"master\", and you see the commit timestamp of the\n> destination.\n>\n> But that isn't how it is implemented. Checkout will handle the file\n> checkout _first_, as that is the part that is likely to run into\n> problems (e.g., rejecting a switch because it would lose changes in the\n> working tree). Only at the very end, after the change to the working\n> tree has succeeded, do we update HEAD.\n>\n> I think the order you are expecting is conceptually cleaner, but I don't\n> think we would want to switch it around (for reasons of efficiency and\n> robustness). And I don't think we would want to make a promise about the\n> ordering to callers either way, as it binds our implementation.\n>\n> So is there a way to reliably know the destination of a checkout? My\n> first thought was that we could add a placeholder similar to \"%f\" that\n> your filter could use. I'm not sure how we would handle the corner cases\n> there, though (e.g., do we always have a \"destination\" to report? If\n> not, what do we give the script?).\n>\n> I suspect you could also hack something together with a post-checkout\n> script, though it would probably be a lot less efficient (and might also\n> have some weird corner cases).\n>\n> -Peff\n"},{"id":"280572","messageId":"20160310210544.GB30595@sigill.intra.peff.net","threadId":"41652","inReplyTo":"CAH8BJxFmAQtoF+1Q7Ub5qWnz5UewrPS4e8JQWms254hO_E05Hw@mail.gmail.com","subject":"Re: git smudge filter fails","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-03-10T21:05:44Z","receivedAt":"2016-03-10T21:05:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 10, 2016 at 09:45:19AM -0500, Stephen Morton wrote:\n\n> I am a bit confused because this is basically the example used in\n> ProGit [1] and it is fundamentally broken. In fact, if I understand\n> correctly, this means that smudge filters cannot be relied upon to\n> provide any 'keyword expansion' type tasks because they will all by\n> nature have to query the file with 'git log'.\n\nInteresting. Perhaps I am missing something (I am far from an expert in\nclean/smudge filters, which I do not generally use myself), but the\nexample in ProGit looks kind of bogus to me. I don't think it ever would\nhave worked reliably, under any version of git.\n\n> (Note that although in my example I used 'git checkout', with an only\n> slightly more complicated example I can make it fail on 'git pull'\n> which is perhaps a much more realistic use case. That was probably\n> implied in your answer, I just wanted to mention it.)\n\nYeah, I think the issue is basically the same for several commands which\nupdate the worktree and the HEAD. Most of them are going to do the\nworktree first.\n\n-Peff\n"},{"id":"280583","messageId":"xmqqtwkec7lk.fsf@gitster.mtv.corp.google.com","threadId":"41652","inReplyTo":"20160310210544.GB30595@sigill.intra.peff.net","subject":"Re: git smudge filter fails","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-10T22:04:55Z","receivedAt":"2016-03-10T22:04:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Thu, Mar 10, 2016 at 09:45:19AM -0500, Stephen Morton wrote:\n>\n>> I am a bit confused because this is basically the example used in\n>> ProGit [1] and it is fundamentally broken. In fact, if I understand\n>> correctly, this means that smudge filters cannot be relied upon to\n>> provide any 'keyword expansion' type tasks because they will all by\n>> nature have to query the file with 'git log'.\n>\n> Interesting. Perhaps I am missing something (I am far from an expert in\n> clean/smudge filters, which I do not generally use myself), but the\n> example in ProGit looks kind of bogus to me. I don't think it ever would\n> have worked reliably, under any version of git.\n>\n>> (Note that although in my example I used 'git checkout', with an only\n>> slightly more complicated example I can make it fail on 'git pull'\n>> which is perhaps a much more realistic use case. That was probably\n>> implied in your answer, I just wanted to mention it.)\n>\n> Yeah, I think the issue is basically the same for several commands which\n> update the worktree and the HEAD. Most of them are going to do the\n> worktree first.\n\nYou can have a pair of branches A and B that have forked long time\nago, and have a path F that has been changed identically since they\nforked, perhaps by cherry-picking the same change. This happens all\nthe time.\n\nIf there were some mechanism that modifies the checked out version\nof F with information that depends on the history that leads to A\n(e.g. \"which commit that is an ancestor of A last modified F?\")\nwhen you check out branch A, it will become invalid when you do \"git\ncheckout B\", because the path F will not change because they are the\nsame between the branches. In short, CVS $Id$-style substitutions\nthat depend on the history fundamentally does not work, unless you\nare willing to always rewrite the whole working tree every time you\nswitch branches.\n\nThe smudge and clean filters are given _only_ the blob contents and\nnothing else, not \"which commit (or tree) the blob is taken from\",\nnot \"which path this blob sits in that tree-ish\", not \"what branch\nam I on\" and this is a very much deliberate design decision made in\norder to avoid leading people to a misguided attempt to mimick CVS\n$Id$-style substitutions.\n"},{"id":"280785","messageId":"CAH8BJxHvg1cnzdsnVsHLYm2BAoqNxp8-mNyVxRy4CHqv0rgN6g@mail.gmail.com","threadId":"41652","inReplyTo":"xmqqtwkec7lk.fsf@gitster.mtv.corp.google.com","subject":"Re: git smudge filter fails","fromName":"Stephen Morton","fromEmail":"stephen.c.morton@gmail.com","sentAt":"2016-03-15T16:17:16Z","receivedAt":"2016-03-15T16:17:16Z","isPatch":false,"sender":{"key":"stephen.c.morton@gmail.com","avatar":"https://gravatar.com/avatar/3b826f06cabc070cbd5d53071880caad8badc3b29e0789b510353faca59ffb7b?d=mp&s=160"},"body":"On Thu, Mar 10, 2016 at 5:04 PM, Junio C Hamano wrote:\n> Jeff King writes:\n>\n>> On Thu, Mar 10, 2016 at 09:45:19AM -0500, Stephen Morton wrote:\n>>\n>>> I am a bit confused because this is basically the example used in\n>>> ProGit [1] and it is fundamentally broken. In fact, if I understand\n>>> correctly, this means that smudge filters cannot be relied upon to\n>>> provide any 'keyword expansion' type tasks because they will all by\n>>> nature have to query the file with 'git log'.\n>>\n>> Interesting. Perhaps I am missing something (I am far from an expert in\n>> clean/smudge filters, which I do not generally use myself), but the\n>> example in ProGit looks kind of bogus to me. I don't think it ever would\n>> have worked reliably, under any version of git.\n>>\n>>> (Note that although in my example I used 'git checkout', with an only\n>>> slightly more complicated example I can make it fail on 'git pull'\n>>> which is perhaps a much more realistic use case. That was probably\n>>> implied in your answer, I just wanted to mention it.)\n>>\n>> Yeah, I think the issue is basically the same for several commands which\n>> update the worktree and the HEAD. Most of them are going to do the\n>> worktree first.\n>\n> You can have a pair of branches A and B that have forked long time\n> ago, and have a path F that has been changed identically since they\n> forked, perhaps by cherry-picking the same change. This happens all\n> the time.\n>\n> If there were some mechanism that modifies the checked out version\n> of F with information that depends on the history that leads to A\n> (e.g. \"which commit that is an ancestor of A last modified F?\")\n> when you check out branch A, it will become invalid when you do \"git\n> checkout B\", because the path F will not change because they are the\n> same between the branches. In short, CVS $Id$-style substitutions\n> that depend on the history fundamentally does not work, unless you\n> are willing to always rewrite the whole working tree every time you\n> switch branches.\n>\n> The smudge and clean filters are given _only_ the blob contents and\n> nothing else, not \"which commit (or tree) the blob is taken from\",\n> not \"which path this blob sits in that tree-ish\", not \"what branch\n> am I on\" and this is a very much deliberate design decision made in\n> order to avoid leading people to a misguided attempt to mimick CVS\n> $Id$-style substitutions.\n>\n\n\nI will raise an Issue with ProGit.\n\nIt's perhaps beyond the scope of my original question, but for\nsituations where I need a \"last change date\" embedded in a file (e.g.\nbecause a protocol standard requires it), is there any recommended way\nto do so? We've the hard way that hardcoding makes\nmerging/cherry-picking a bit of a nightmare and should be avoided. Is\na post-checkout hook the way to go? I've actually found the smudge\nfilter to be very slow for this application as each file is processed\nin series; a post-commit hook that could operate on files in parallel\nwould likely be substantially faster.\n\nStephen\n\n(Sorry about the earlier top-posting. I didn't realize what gmail was\ndoing until after it had happened.)\n"},{"id":"280807","messageId":"xmqqfuvr8z6i.fsf@gitster.mtv.corp.google.com","threadId":"41652","inReplyTo":"CAH8BJxHvg1cnzdsnVsHLYm2BAoqNxp8-mNyVxRy4CHqv0rgN6g@mail.gmail.com","subject":"Re: git smudge filter fails","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-03-15T16:48:37Z","receivedAt":"2016-03-15T16:48:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stephen Morton writes:\n\n> It's perhaps beyond the scope of my original question, but for\n> situations where I need a \"last change date\" embedded in a file (e.g.\n> because a protocol standard requires it), is there any recommended way\n> to do so? We've the hard way that hardcoding makes\n> merging/cherry-picking a bit of a nightmare and should be avoided.\n\nDoes that \"last change date\" have to be embedded in a file with\nother stuff in there, or can it be a standalone file by itself\n(which may be used by other things via linking or inclusion)?\n\nIf it can be a standalone file, a custom ll-merge driver that knows\nhow yoru datestring looks like and takes the later of the versions\nin the two branches being merged would not be too hard to write to\neliminate the \"nightmare\", I would think.\n"}]}