Implementing $Date$ substitution - problem with git status

Hi!

I finally got around to moving my web site repository over from CVS to Git, and to not have to redo everything about how it is set up, I set up a filter that fakes CVS' handling of "$Date$" keywords. Since I cannot access any information about the file from the "smudge" filter, I set up a "clean" filter that edits the $Date$ token on "git add":

   #!/usr/bin/perl -wTp -i.bak
   if (/\$Date:?[^\$]*\$/)
   {
     my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
     my $now = sprintf("%04d-%02d-%02d %02d:%02d:%02d", $year + 1900,
                       $mon + 1, $mday, $hour, $min, $sec);
     s/\$Date:?[^\$]*\$/\$Date: ${now} \$/;
   }

In .gitattributes I have lines like:

   *.html filter=date crlf=true ident

and filter.date.clean is set to point to the script above in .git/config.

This works fine, except that I have some problems with files that I cannot get to be "clean" in git status. "git diff" shows only a difference in the "$Date$" token, which is different each time I run "git diff" (the "+" line shows the current time):

   diff --git a/.htaccess b/.htaccess
   index 2dfbc32..d8ac343 100644
   --- a/.htaccess
   +++ b/.htaccess
   @@ -2,7 +2,7 @@
    # -----------------------------------------------------

    # $Id$
   -# $Date: 2009-08-09 19:52:30 $
   +# $Date: 2009-08-09 20:31:56 $
   [...]

Strangely, the $Id$ line above (which is handled by me adding the "ident" attribute to all text files), does not exhibit the same problem.

Doing "git reset --hard" or "git checkout master filename" does not help, the file is still believed to be modified by git.

Does anyone know if there is a way to get this to work, or am I screwed for trying to do things I'm not supposed to? :-)

On Sonntag, 9. August 2009, Peter Krefting wrote:

The clean filter is applied before 'git diff' generates the diff. Since you have unstaged changes (I take [...] to mean this), you see a new date on every invocation.

After 'git add .htaccess' subsequent 'git diff' should not show any changes.

Now, that's an entirely different problem, and I think that there is a bug. I have observed this as well with my own clean filter sometimes, but not always. I haven't found a recipe that reliably exhibits the problem.

-- Hannes

Johannes Sixt:

Sorry for being unclear; no, the $Date$ line is the only change that shows up. I removed the surrounding context because I didn't want to publish them.

On another machine, there is another file that exhibits the same problem.

But then I need to check it in again, and I'd like to avoid that, if possible.

Is it possible to have a filter run on "commit" instead of "add"? (Or is "commit" just moving changes from the index to the repository, making that impossible?)

It seems to be triggered by using my date script and doing

   git add file
   git commit file
   git reset --hard file

the last command gives me a local copy with the keyword expanded, but marked dirty.

Is it possible to have "git add" expand the keyword (both the Git-internal "ident", i.e "$Id$", and my "$Date$")? Would duplicating the "clean" filter as a "smudge" one accomplish that?

Johannes Sixt:

After som examination, it seems to be caused by the way I imported the CVS history: I kept all the $ keywords expanded in history (so that if I check out an old version from Git, it looks like it did in CVS). This means that still in the latest revision of several files, I have "$Id$" lines checked in in CVS format.

However, as I have set the "ident" attribute, Git wants to expand it itself and check in files with "$Id$". When I do a reset, it seems it records the entry as clean against a version stored with just "$Id$", but the record in history has an expanded "$Id$", and the entry is thus never deemed clean.

I can probably work around this by removing the "$Id$" attributes, or by removing the "ident" rule.

Peter Krefting:

Actually, that is not the case. It seems to be the filter that causes the problems, all by itself. I cannot seem to reproduce this *reliably*. I tried setting up a minimal repository and a reproduction recipe, but I get different behaviour when I perform the steps manually, and when I run it from a shell script.

Here is the reproduction recipe:

#!/bin/bash # Reproduction recipe for $Date$ dirty issue

ORIGINREPO=git://git.debian.org/users/peterk/gitfilterproblem.git DESTINATIONREPO=gitfilterproblem-testrepo

# Set up repository echo -- Cloning git clone ${ORIGINREPO} ${DESTINATIONREPO} cd ${DESTINATIONREPO}

# Status should be clean echo -- After cloning, status should be clean git status

# Set up filter echo -- Set up filter, status should be clean ./reposetup.sh git status

# Create tracking branch echo -- Create tracking branch for changed file, status should be clean git branch --track changed-text-files origin/changed-text-files git status

# Merge the branch echo -- Merge the changes from the branch, now status gets unclean git merge changed-text-files echo git status git diff

Peter Krefting venit, vidit, dixit 12.08.2009 10:36:

I get "Changed but not updated" already here!

Also, what's your git version? There have been some fix-ups recently regarding the interaction between filters/textconv and assumptions of the code about dirtiness of the worktree.

Do you really want the date in the checked-in version of the file? I would assume otherwise. Then your clean filter should really be the smudge filter, and you would need a clean filter to go with it (remove the date and restore the keyword).

Michael

Michael J Gruber:

With git 1.5.6.5, I do get that. With git 1.6.4, I seem to only get it after the last step.

I have 1.5.6.5 on the server with the master repo, and am running 1.6.4 on the client (although when I ran the recipe through a shell script, I got the unclean status earlier, which made me think it ran an earlier version I have installed in another directory).

Yes. This way, the checked out copy that makes my web server can do its job without requiring any of the filters being installed. And the files in history are marked as necessary.

The problem with that approach is that the "smudge" filter does not have access to the file name, and so can not look up the last change date of the file it is re-writing. And I want the last time the file was *changed*, not the time it was checked out.

Also, I imported my entire CVS history with keywords expanded to allow for this (and "checked out" the Git-generated tree over my CVS check-out to not have Git update all the time-stamps).

Hi!

So, my filter-based approach at doing RCS-like $Date$ expansion didn't work out. Since I wanted to have the Date expand on check-in, not on check-out, and be expanded in history, I had made it into a "clean" filter. But due to how Git internally checks if files are clean, that did not work properly.

The solution to this was instead to make the Date expansion into a pre-commit hook. Using .gitattributes, I set "datereplace=true" on all the files I want to have "$Date$" expand in, and then install the following file as .git/hooks/pre-commit:

===8< pre-commit >8=== #!/bin/bash -e

# Find base commit
if git-rev-parse --verify HEAD >/dev/null 2>&1
then
 	against=HEAD
else
 	# Initial commit: diff against an empty tree object
 	against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
fi

# Set up Date substitution
export NOW=$(date +"%Y-%m-%d %H:%M:%S")
for file in $(git diff-index --cached --diff-filter=AM --name-only $against); do
   if git check-attr datereplace -- "$file" | grep 'datereplace: true' > /dev/null; then
     perl -w -i.bak -e 'my $now = $ENV{"NOW"};
$now =~ s/[^-:0-9 ]//g;
while (<>)
{
  if (/\$Date:?[^\$]*\$/)
  {
    s/\$Date:?[^\$]*\$/\$Date: ${now} \$/;
  }
  print
}' "$file"
     git update-index --add "$file"
   fi
done

exit 0 ===8< pre-commit >8===

This has the added bonus over a filter that it all files committed at the same time will have the same date stamp, whereas the filter would expand on the time "git add" is executed.