[Question] Diff text filters and git add

Hi all,

I am trying to do something a bit strange and wonder about the best way to go. I have a text filter that presents content of very special binary file formats using textconv. What I am wondering is whether using the textconv mechanism is sufficient to have git calculate the file signature or whether I need to use an external diff engine, so that git add behaves in a stable manner (i.e., does git internally use the textconv mechanism for evaluating whether a file changed or whether the external diff engine is required, or whether this is even possible at all).

The basic use case is that there is a timestamp embedded in the binary file that I want to forever ignore when committing. I only need this done on one specific machine, which is under Jenkins control, so it's not something developers would deal with at all (so the filter config is in one place). When the binary generator runs, if the two file images are "similar enough" (as in: the same except for the generated timestamp, and a couple of other annoying bits of metadata), I want git to think that they are the same in an automated way, so that when I am constructing commits, I do not want what would be considered a duplicate of what is essentially the same file.

Sadly, I cannot modify the generator, so I'm stuck with the files being wonky. I also cannot run the generator anywhere downstream, so doing so on the deployment engine is also not an option (don't ask, the generator is limited on where it can be run). Suggestions are welcome, please.

Thanks, Randall

On Tue, Jul 09, 2019 at 05:43:05PM -0400, Randall S. Becker wrote:

No, textconv only applies when generating a diff to output, and will never impact what's stored in Git.

It sounds like you might want a clean filter instead, to sanitize the file contents as they come into Git (and perhaps a matching smudge filter to convert back to the working-tree version if necessary).

You're talking about "the diff engine" here, but note that git-add would never do a diff at all. It cares only about full sha1s (and optimizes out re-computing the sha1 on each invocation by using stat data). So outside of clean/smudge, there's nothing else going on.

-Peff

On July 9, 2019 5:51 PM, Peff wrote:

Thanks. I can script this instead. Will do an external diff then --assume-unchanged when I detect an equivalence.

Appreciate the advice and info, Randall

"Randall S. Becker" <rsbecker@nexbridge.com> writes:

[...]

If you want to ignore changes, --assume-unchanged (i.e. lying to Git) is a wrong solution, as it can lead to data loss. It is meant as performance optimization.

A better solution would be to use --skip-worktree, which though meant for sparse checkout can be used for ignoring changes. The only problem is that it can prevent some safe operations, like git-stash, because git thinks that it could lead to data loss.

Though I am not sure if they are needed with clean/smudge filter.

Best,