git with custom diff for commits

Hello folks,

I do a nightly mysqldump of a database and check it into a git repository. mysqldump generates a timestamp as part of that output which is causing git to think that the file changes every night when it really doesn't. The timestamp is simply in an SQL comment.

So what I'd like to do is teach git to ignore that particular SQL timestamp comment. I've tried to set up an external diff script that runs diff -I "<<sql timestamp comment>>" that effectively ignores the timestamp. While this works with "git diff", it seems when git commits, it still sees the differences.

How do I properly teach git to ignore these types of differences?

Gerald.

Hi,

On Mon, 17 Dec 2007, Gerald Gutierrez wrote:

You might be interested in reading Documentation/gitattributes.txt, look for "diff driver".

Hth, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

It will show an empty output for "git diff", but I doubt thit will change anything at commit time. Probably the "filter" thing on the same file (also "man gitattributes") can help though.

Hi,

On Tue, 18 Dec 2007, Matthieu Moy wrote:

Ah, right. I completely missed that you were talking about git-commit, not git-log on git commits.

Yes, setting up a "clean" filter that removes the timestamps is probably the reasonable thing to do here.

Sorry for the noise, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

I wouldn't do filters for something like that. Can you guarantee that the output from corresopnding smudge filter will load cleanly back to the mysql database?

Just do not make the commit if you made only meaningless changes and nothing else. pre-commit hook would probably be a good place to do so.

Junio C Hamano <gitster@pobox.com> writes:

The original poster said the date was an SQL comment, so, it should be safe to strip it. That said, precommit hook is probably a good or better solution. At least, if you're not sure you didn't make a mistake writting it, you'll notice either a good commit or no commit at all. With filters, you might well notice you corrupted the commit too late :-(.

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

Yeah. After writing the message you are responding to, I realized that I already do something like that to manage html/man branches that are autogenerated, and it is exactly the same situation as Gerald is in.

The autogeneration of documentation and updating of html/man branches is controlled by dodoc.sh script in 'todo' branch, installed as post-update hook in my public repository. It builds the documentation in a checkout of 'master' branch, installs the result into another checkout of 'html' and 'man' branches, and commits.

The twist is in the "install" step (this is different from my suggestion of preventing the commit in pre-commit hook). When the newly generated documentation files are different from the previously generated ones only by datestamp, they are not installed into 'html'/'man' checkouts, and such a change is not included in the commits.

So my updated suggestion would be:

 - Keep a checkout of that mysql dump;

 - Make the dump procedure to dump to that checkout area;

 - Run "git diff HEAD", perhaps with the external diff that filters out
   the cruft as suggested in the thread, and run "git checkout $path"
   for paths that had only meaningless changes;

 - Run "git commit" if the above step did not nuke all changes.

I ended up doing the following, which sounds similar to the above. Instead of doing mysqldump into data.sql, it goes into data.sql.2 which I compare with the checked in data.sql using "diff -I <timestamp RE>". If there are no differences, I delete data.sql.2. If there are differences, I move data.sql.2 into data.sql and check in. Perhaps not as elegant but certainly works.

Gerald.

"Gerald Gutierrez" <ggmlfs@gmail.com> writes:

Heh, that's essentially how automated html/man branches are managed ;-)

Yes, but wouldn't it be slick to actually teach git's internal diff to do things like GNU diff can, like the ignore option -I, case insensitivity, etc. I thought that's what the external diff capability is for, but it is not so.

Gerald.

Hi,

On Tue, 18 Dec 2007, Gerald Gutierrez wrote:

I disagree. Your statement is misleading when you say you want the internal diff to do things like GNU diff can.

What you want to do _in fact_ is not only modify the diff _output_, but the commit _input_.

And I am not so keen on such filters. If I do not want to commit, I do not change it to begin with. Yes, your mysqldump is a special case here. But it is special anyway, as it is not source code.

Ciao, Dscho

I don't understand how it is a benefit to have "git commit" and "git diff" do different diff functions. It confuses me. A scenario: I keep doing git diff's and it says there is no difference, but then git commit keeps telling me there is.

While I understand it was built for the kernel and everybody likes the plain text format, it is not a stretch of the imagination to have files that are semantically identical but have byte-wise differences. OpenDocument files, image files, XML, etc. Cases can be made either way that "the same" means semantically or byte-wise. It's perfectly fine that the default is byte-wise. But, if the user takes the time to build a custom diff, then wouldn't it be reasonable to assume that user means for git to do semantic comparisons for diff, commit and other git functions, even if there are inconsequential byte-wise differences in the file itself?

Gerald.

Hi,

On Tue, 18 Dec 2007, Gerald Gutierrez wrote:

Matthieu said it better than I ever could.

Ciao, Dscho

"Gerald Gutierrez" <ggmlfs@gmail.com> writes:

It certainly is. Your external diff gets all the information it needs to generate whatever custom diff computation it may want to do.

"Gerald Gutierrez" <ggmlfs@gmail.com> writes:

Don't confuse diff and commit.

Git is really snapshot oriented. When you commit, you don't say "record the changes I made since HEAD", you say "record the current state of the working tree, and also record the fact that this state is based on HEAD".

See what a commit object looks like:

$ git cat-file -p 183f84365d tree 3f9d576b4adc78188a411104bc21159d459fa3f4 parent b9c506f7d9f05a630fa7e31b77a9cf5081d7dbba author Shun Kei Leung <kevinlsk@gmail.com> 1195614079 +0800 committer Junio C Hamano <gitster@pobox.com> 1195630064 -0800

git-p4: Fix typo in --detect-labels

Signed-off-by: Kevin Leung <kevinlsk@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
$ _

The "tree" tells you the state of the project as recorded by commit, and the "parent" tells you where the HEAD was pointing to at commit time. That's really the raw information recorded by git.

Now, "diff" comes into the picture for two things: 1) storage efficiency (you prefer a 1Mb pack file to a 1Gb uncompressed repository), but that doesn't change the concept, and 2) user interface.

When you say "git show HEAD", for example, git will recompute a diff, and show it to you because this is what you expect. But this information is computed at "git show" time, not at "git commit" time.

What you seem to expect is to tell git "commit the changes except this portion", what you have to tell git is actually "commit the new state, but not if this portion is the only change". That sounds identical, but it's actually a bit different.

It's not what I'm saying.

Here's what the git-commit manpage on kernel.org says: "git-commit - Record changes to the repository".

What I am saying is that if the user defines a custom diff, it is reasonable to believe that the user wants to teach git a new definition of "changes". If that new definition says that the files have changed, then go ahead and commit the new state. If that definition says there are no changes, then there is nothing to commit. In other words:

if (user_redefined_changed)
  if (user_defn_of_changed(work, head)) commit(work);
else
  if (builtin_changed(work, head)) commit(work);

As it is, it's not a big deal to me. Instead of teaching git the different concept of "changes", I teach the script that runs git instead. But teaching git directly just seems cleaner and more intuitive.

Gerald.

Hi,

On Tue, 18 Dec 2007, Gerald Gutierrez wrote:

[who said that?  (I know, but would you know in a few weeks?  Right.)]

Yes. Changes, as in "take this _file_ instead". Not "edit this file, remove those lines, add these here, etc.".

So I think that you were saying exactly what Matthieu suggested.

Ciao, Dscho

On Dec 19, 2007 11:48 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Exactly. GIT is a "content tracker" that doesn't care about any semantic meaning in your files. Here's the file, git will store it, as is. No conversions, not fancy interpretations. Strict, safe and fast. (There's a small exception there for DOS-style newlines, which was much resisted.)

You can add external machinery that is aware of your content semantics -- as you've done with calling GNU diff with ignore patterns to decide whether to commit or not. But when you tell GIT to commit something, there's no guessing or transformation involved.

A lot of what GIT achieves is based on that founding principle. Remove it, and we're toast. For starters, the internal machinery is based on SHA1 digests that change if you flip a single byte. That's what makes git fast and realiable. It's not merely an end-user thing.

cheers,

m

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

That said, I think the wording of the man page could be better. Perhaps "Record new snapshot to the repository" or "Record current state of the working tree to the repository".

I can't find something wich is both accurate and newbie friendly ... ("accurate" would probably be "Record index content into the repository")

Hi,

On Wed, 19 Dec 2007, Matthieu Moy wrote:

How about "record a new revision"?

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

Sounds good, yes.

"Gerald Gutierrez" <ggmlfs@gmail.com> writes:

That would lead to even less intuitive semantics.

Suppose your working tree has two changes, like

That didn't come out quite right. It should be:

if (user_redefined_changed) {
  if (user_defn_of_changed(work, head)) { commit(work); }
} else {
  if (builtin_changed(work, head)) { commit(work); }
}

I read about the filter too, but have no idea how to implement it. Any examples that I could follow?

Thanks, Gerald.