{"thread":{"id":"20508","subject":"Implementing $Date$ substitution - problem with git status","startedAt":"2009-08-09T19:37:48Z","lastAt":"2009-08-13T06:47:25Z","messageCount":8,"participants":["Peter Krefting","Johannes Sixt","Michael J Gruber"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"120086","messageId":"alpine.DEB.2.00.0908092127420.17141@perkele.intern.softwolves.pp.se","threadId":"20508","inReplyTo":null,"subject":"Implementing $Date$ substitution - problem with git status","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-08-09T19:37:48Z","receivedAt":"2009-08-09T19:37:48Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Hi!\n\nI finally got around to moving my web site repository over from CVS to Git, \nand to not have to redo everything about how it is set up, I set up a filter \nthat fakes CVS' handling of \"$Date$\" keywords. Since I cannot access any \ninformation about the file from the \"smudge\" filter, I set up a \"clean\" \nfilter that edits the $Date$ token on \"git add\":\n\n   #!/usr/bin/perl -wTp -i.bak\n   if (/\\$Date:?[^\\$]*\\$/)\n   {\n     my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();\n     my $now = sprintf(\"%04d-%02d-%02d %02d:%02d:%02d\", $year + 1900,\n                       $mon + 1, $mday, $hour, $min, $sec);\n     s/\\$Date:?[^\\$]*\\$/\\$Date: ${now} \\$/;\n   }\n\nIn .gitattributes I have lines like:\n\n   *.html filter=date crlf=true ident\n\nand filter.date.clean is set to point to the script above in .git/config.\n\nThis works fine, except that I have some problems with files that I cannot \nget to be \"clean\" in git status. \"git diff\" shows only a difference in the \n\"$Date$\" token, which is different each time I run \"git diff\" (the \"+\" line \nshows the current time):\n\n   diff --git a/.htaccess b/.htaccess\n   index 2dfbc32..d8ac343 100644\n   --- a/.htaccess\n   +++ b/.htaccess\n   @@ -2,7 +2,7 @@\n    # -----------------------------------------------------\n\n    # $Id$\n   -# $Date: 2009-08-09 19:52:30 $\n   +# $Date: 2009-08-09 20:31:56 $\n   [...]\n\nStrangely, the $Id$ line above (which is handled by me adding the \"ident\" \nattribute to all text files), does not exhibit the same problem.\n\nDoing \"git reset --hard\" or \"git checkout master filename\" does not help, \nthe file is still believed to be modified by git.\n\n\nDoes anyone know if there is a way to get this to work, or am I screwed for \ntrying to do things I'm not supposed to? :-)\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"120088","messageId":"200908092252.58363.j6t@kdbg.org","threadId":"20508","inReplyTo":"alpine.DEB.2.00.0908092127420.17141@perkele.intern.softwolves.pp.se","subject":"Re: Implementing $Date$ substitution - problem with git status","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2009-08-09T20:52:58Z","receivedAt":"2009-08-09T20:52:58Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Sonntag, 9. August 2009, Peter Krefting wrote:\n> I finally got around to moving my web site repository over from CVS to Git,\n> and to not have to redo everything about how it is set up, I set up a\n> filter that fakes CVS' handling of \"$Date$\" keywords. Since I cannot access\n> any information about the file from the \"smudge\" filter, I set up a \"clean\"\n> filter that edits the $Date$ token on \"git add\":\n>\n>    #!/usr/bin/perl -wTp -i.bak\n>    if (/\\$Date:?[^\\$]*\\$/)\n>    {\n>      my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =\n> localtime(); my $now = sprintf(\"%04d-%02d-%02d %02d:%02d:%02d\", $year +\n> 1900, $mon + 1, $mday, $hour, $min, $sec);\n>      s/\\$Date:?[^\\$]*\\$/\\$Date: ${now} \\$/;\n>    }\n>\n> In .gitattributes I have lines like:\n>\n>    *.html filter=date crlf=true ident\n>\n> and filter.date.clean is set to point to the script above in .git/config.\n>\n> This works fine, except that I have some problems with files that I cannot\n> get to be \"clean\" in git status. \"git diff\" shows only a difference in the\n> \"$Date$\" token, which is different each time I run \"git diff\" (the \"+\" line\n> shows the current time):\n>\n>    diff --git a/.htaccess b/.htaccess\n>    index 2dfbc32..d8ac343 100644\n>    --- a/.htaccess\n>    +++ b/.htaccess\n>    @@ -2,7 +2,7 @@\n>     # -----------------------------------------------------\n>\n>     # $Id$\n>    -# $Date: 2009-08-09 19:52:30 $\n>    +# $Date: 2009-08-09 20:31:56 $\n>    [...]\n\nThe clean filter is applied before 'git diff' generates the diff. Since you \nhave unstaged changes (I take [...] to mean this), you see a new date on \nevery invocation.\n\nAfter 'git add .htaccess' subsequent 'git diff' should not show any changes.\n\n> Strangely, the $Id$ line above (which is handled by me adding the \"ident\"\n> attribute to all text files), does not exhibit the same problem.\n>\n> Doing \"git reset --hard\" or \"git checkout master filename\" does not help,\n> the file is still believed to be modified by git.\n\nNow, that's an entirely different problem, and I think that there is a bug. I \nhave observed this as well with my own clean filter sometimes, but not \nalways. I haven't found a recipe that reliably exhibits the problem.\n\n-- Hannes\n"},{"id":"120108","messageId":"alpine.DEB.2.00.0908100535150.17908@ds9.cixit.se","threadId":"20508","inReplyTo":"200908092252.58363.j6t@kdbg.org","subject":"Re: Implementing $Date$ substitution - problem with git status","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-08-10T04:41:01Z","receivedAt":"2009-08-10T04:41:01Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Johannes Sixt:\n\n> The clean filter is applied before 'git diff' generates the diff. Since you\n> have unstaged changes (I take [...] to mean this), you see a new date on\n> every invocation.\n\nSorry for being unclear; no, the $Date$ line is the only change that shows \nup. I removed the surrounding context because I didn't want to publish them.\n\nOn another machine, there is another file that exhibits the same problem.\n\n> After 'git add .htaccess' subsequent 'git diff' should not show any changes.\n\nBut then I need to check it in again, and I'd like to avoid that, if possible.\n\n\nIs it possible to have a filter run on \"commit\" instead of \"add\"? (Or is \n\"commit\" just moving changes from the index to the repository, making that \nimpossible?)\n\n>> Doing \"git reset --hard\" or \"git checkout master filename\" does not \n>> help, the file is still believed to be modified by git.\n>\n> Now, that's an entirely different problem, and I think that there is a \n> bug. I have observed this as well with my own clean filter sometimes, but \n> not always. I haven't found a recipe that reliably exhibits the problem.\n\nIt seems to be triggered by using my date script and doing\n\n   git add file\n   git commit file\n   git reset --hard file\n\nthe last command gives me a local copy with the keyword expanded, but marked \ndirty.\n\n\nIs it possible to have \"git add\" expand the keyword (both the Git-internal \n\"ident\", i.e \"$Id$\", and my \"$Date$\")? Would duplicating the \"clean\" filter \nas a \"smudge\" one accomplish that?\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"120379","messageId":"alpine.DEB.2.00.0908120751500.30907@ds9.cixit.se","threadId":"20508","inReplyTo":"200908092252.58363.j6t@kdbg.org","subject":"Re: Implementing $Date$ substitution - problem with git status","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-08-12T06:54:44Z","receivedAt":"2009-08-12T06:54:44Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Johannes Sixt:\n\n>> Doing \"git reset --hard\" or \"git checkout master filename\" does not \n>> help, the file is still believed to be modified by git.\n\n> Now, that's an entirely different problem, and I think that there is a \n> bug. I have observed this as well with my own clean filter sometimes, but \n> not always. I haven't found a recipe that reliably exhibits the problem.\n\nAfter som examination, it seems to be caused by the way I imported the CVS \nhistory: I kept all the $ keywords expanded in history (so that if I check \nout an old version from Git, it looks like it did in CVS). This means that \nstill in the latest revision of several files, I have \"$Id$\" lines checked \nin in CVS format.\n\nHowever, as I have set the \"ident\" attribute, Git wants to expand it itself \nand check in files with \"$Id$\". When I do a reset, it seems it records the \nentry as clean against a version stored with just \"$Id$\", but the record in \nhistory has an expanded \"$Id$\", and the entry is thus never deemed clean.\n\nI can probably work around this by removing the \"$Id$\" attributes, or by \nremoving the \"ident\" rule.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"120389","messageId":"alpine.DEB.2.00.0908120856110.30907@ds9.cixit.se","threadId":"20508","inReplyTo":"alpine.DEB.2.00.0908120751500.30907@ds9.cixit.se","subject":"Problems with filters and git status - reproduction steps","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-08-12T08:36:22Z","receivedAt":"2009-08-12T08:36:22Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Peter Krefting:\n\n> However, as I have set the \"ident\" attribute, Git wants to expand it itself \n> and check in files with \"$Id$\". When I do a reset, it seems it records the \n> entry as clean against a version stored with just \"$Id$\", but the record in \n> history has an expanded \"$Id$\", and the entry is thus never deemed clean.\n\nActually, that is not the case. It seems to be the filter that causes the \nproblems, all by itself. I cannot seem to reproduce this *reliably*. I tried \nsetting up a minimal repository and a reproduction recipe, but I get \ndifferent behaviour when I perform the steps manually, and when I run it \nfrom a shell script.\n\nHere is the reproduction recipe:\n\n#!/bin/bash\n# Reproduction recipe for $Date$ dirty issue\n\nORIGINREPO=git://git.debian.org/users/peterk/gitfilterproblem.git\nDESTINATIONREPO=gitfilterproblem-testrepo\n\n# Set up repository\necho -- Cloning\ngit clone ${ORIGINREPO} ${DESTINATIONREPO}\ncd ${DESTINATIONREPO}\n\n# Status should be clean\necho -- After cloning, status should be clean\ngit status\n\n# Set up filter\necho -- Set up filter, status should be clean\n./reposetup.sh \ngit status\n\n# Create tracking branch\necho -- Create tracking branch for changed file, status should be clean\ngit branch --track changed-text-files origin/changed-text-files \ngit status\n\n# Merge the branch\necho -- Merge the changes from the branch, now status gets unclean\ngit merge changed-text-files\necho\ngit status\ngit diff\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"120391","messageId":"4A828368.5010206@drmicha.warpmail.net","threadId":"20508","inReplyTo":"alpine.DEB.2.00.0908120856110.30907@ds9.cixit.se","subject":"Re: Problems with filters and git status - reproduction steps","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2009-08-12T08:55:04Z","receivedAt":"2009-08-12T08:55:04Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Peter Krefting venit, vidit, dixit 12.08.2009 10:36:\n> ORIGINREPO=git://git.debian.org/users/peterk/gitfilterproblem.git\n> DESTINATIONREPO=gitfilterproblem-testrepo\n> \n> # Set up repository\n> echo -- Cloning\n> git clone ${ORIGINREPO} ${DESTINATIONREPO}\n> cd ${DESTINATIONREPO}\n> \n> # Status should be clean\n> echo -- After cloning, status should be clean\n> git status\n> \n> # Set up filter\n> echo -- Set up filter, status should be clean\n> ./reposetup.sh \n> git status\n\nI get \"Changed but not updated\" already here!\n\nAlso, what's your git version? There have been some fix-ups recently\nregarding the interaction between filters/textconv and assumptions of\nthe code about dirtiness of the worktree.\n\n> \n> # Create tracking branch\n> echo -- Create tracking branch for changed file, status should be clean\n> git branch --track changed-text-files origin/changed-text-files \n> git status\n> \n> # Merge the branch\n> echo -- Merge the changes from the branch, now status gets unclean\n> git merge changed-text-files\n> echo\n> git status\n> git diff\n\nDo you really want the date in the checked-in version of the file? I\nwould assume otherwise. Then your clean filter should really be the\nsmudge filter, and you would need a clean filter to go with it (remove\nthe date and restore the keyword).\n\nMichael\n"},{"id":"120393","messageId":"alpine.DEB.2.00.0908121020390.30907@ds9.cixit.se","threadId":"20508","inReplyTo":"4A828368.5010206@drmicha.warpmail.net","subject":"Re: Problems with filters and git status - reproduction steps","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-08-12T09:25:37Z","receivedAt":"2009-08-12T09:25:37Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Michael J Gruber:\n\n> I get \"Changed but not updated\" already here!\n\nWith git 1.5.6.5, I do get that. With git 1.6.4, I seem to only get it after \nthe last step.\n\nI have 1.5.6.5 on the server with the master repo, and am running 1.6.4 on \nthe client (although when I ran the recipe through a shell script, I got the \nunclean status earlier, which made me think it ran an earlier version I have \ninstalled in another directory).\n\n> Do you really want the date in the checked-in version of the file?\n\nYes. This way, the checked out copy that makes my web server can do its job \nwithout requiring any of the filters being installed. And the files in \nhistory are marked as necessary.\n\n> I would assume otherwise. Then your clean filter should really be the \n> smudge filter, and you would need a clean filter to go with it (remove the \n> date and restore the keyword).\n\nThe problem with that approach is that the \"smudge\" filter does not have \naccess to the file name, and so can not look up the last change date of the \nfile it is re-writing. And I want the last time the file was *changed*, not \nthe time it was checked out.\n\nAlso, I imported my entire CVS history with keywords expanded to allow for \nthis (and \"checked out\" the Git-generated tree over my CVS check-out to not \nhave Git update all the time-stamps).\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"120494","messageId":"alpine.DEB.2.00.0908130742470.18241@ds9.cixit.se","threadId":"20508","inReplyTo":"alpine.DEB.2.00.0908092127420.17141@perkele.intern.softwolves.pp.se","subject":"Implementing $Date$ substitution - solution using pre-commit","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-08-13T06:47:25Z","receivedAt":"2009-08-13T06:47:25Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Hi!\n\nSo, my filter-based approach at doing RCS-like $Date$ expansion didn't work \nout. Since I wanted to have the Date expand on check-in, not on check-out, \nand be expanded in history, I had made it into a \"clean\" filter. But due to \nhow Git internally checks if files are clean, that did not work properly.\n\nThe solution to this was instead to make the Date expansion into a \npre-commit hook. Using .gitattributes, I set \"datereplace=true\" on all the \nfiles I want to have \"$Date$\" expand in, and then install the following \nfile as .git/hooks/pre-commit:\n\n===8< pre-commit >8===\n#!/bin/bash -e\n\n# Find base commit\nif git-rev-parse --verify HEAD >/dev/null 2>&1\nthen\n \tagainst=HEAD\nelse\n \t# Initial commit: diff against an empty tree object\n \tagainst=4b825dc642cb6eb9a060e54bf8d69288fbee4904\nfi\n\n# Set up Date substitution\nexport NOW=$(date +\"%Y-%m-%d %H:%M:%S\")\nfor file in $(git diff-index --cached --diff-filter=AM --name-only $against); do\n   if git check-attr datereplace -- \"$file\" | grep 'datereplace: true' > /dev/null; then\n     perl -w -i.bak -e 'my $now = $ENV{\"NOW\"};\n$now =~ s/[^-:0-9 ]//g;\nwhile (<>)\n{\n  if (/\\$Date:?[^\\$]*\\$/)\n  {\n    s/\\$Date:?[^\\$]*\\$/\\$Date: ${now} \\$/;\n  }\n  print\n}' \"$file\"\n     git update-index --add \"$file\"\n   fi\ndone\n\nexit 0\n===8< pre-commit >8===\n\nThis has the added bonus over a filter that it all files committed at the \nsame time will have the same date stamp, whereas the filter would expand on \nthe time \"git add\" is executed.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"}]}