threads / discuss / 21695

How to make git diff-* ignore some patterns?

Subject: How to make git diff-* ignore some patterns?

## tl;dr

4 messages between Nov 21, 2009 and Nov 22, 2009.

replies: 3people: 3as markdown or json

Dirk Süsserott· Nov 21, 2009, 16:40 UTC · lore

Hi list,

is there a way to tell "git diff-index" to ignore some special patterns, such that /^-- Dump completed on .*$/ is NOT recognized as a difference and "git diff-index" returns 0 if that's the only difference?

     -- Dirk

<Background> I have a mySQL database which I backup daily using mysqldump (cronjob). The result is a text file (*.sql) with all the "create" and "insert" statements and some metadata. I used to use tar and gzip to backup these files and got a huge collection of backups in the last tree years (500+ MB). Then I switched to Git and recorded only the diffs between day X and day X-1. My repository shrunk to 16 MB for the very same data, which was great!

My database doesn't change every day, but I backup it anway and store the backup files with Git and a cronjob. It does:

---------------
mysqldump ... -r <backupfile> # that's the output file ;-)
git add <backupfile>
if ! git diff-index --quiet HEAD --; then
     git commit -m "Backup of <database> at <timestamp>"
fi
---------------

This way, a new commit is only done when the backupfile has changed. So far, so perfect. A few days ago my web hoster (where the database actually resides) changed the mySQL version. mysqldump now writes "-- Dump completed on <timestamp>" to the file and Git correctly recognizes this as a change and my script creates a new commit. Every day, even if only that line has changed.

I'd like to skip these commits if only the "Dump completed" line has changed. </Background>

Michael J Gruber· Nov 21, 2009, 17:31 UTC · re: Dirk Süsserott · lore

Re: How to make git diff-* ignore some patterns?

Dirk Süsserott venit, vidit, dixit 21.11.2009 17:40:

Show 39 quoted lines

> Hi list,
> 
> is there a way to tell "git diff-index" to ignore some special patterns, 
> such that /^-- Dump completed on .*$/ is NOT recognized as a difference 
> and "git diff-index" returns 0 if that's the only difference?
> 
>      -- Dirk
> 
> <Background>
> I have a mySQL database which I backup daily using mysqldump (cronjob).
> The result is a text file (*.sql) with all the "create" and "insert"
> statements and some metadata.
> I used to use tar and gzip to backup these files and got a huge
> collection of backups in the last tree years (500+ MB).
> Then I switched to Git and recorded only the diffs between day X and day
> X-1. My repository shrunk to 16 MB for the very same data, which was great!
> 
> My database doesn't change every day, but I backup it anway and store 
> the backup files with Git and a cronjob. It does:
> 
> ---------------
> mysqldump ... -r <backupfile> # that's the output file ;-)
> git add <backupfile>
> if ! git diff-index --quiet HEAD --; then
>      git commit -m "Backup of <database> at <timestamp>"
> fi
> ---------------
> 
> This way, a new commit is only done when the backupfile has changed. So 
> far, so perfect.
> A few days ago my web hoster (where the database actually resides) 
> changed the mySQL version.
> mysqldump now writes "-- Dump completed on <timestamp>" to the file and 
> Git correctly recognizes this as a change and my script creates a new 
> commit. Every day, even if only that line has changed.
> 
> I'd like to skip these commits if only the "Dump completed" line has 
> changed.
> </Background>

Is the dump guaranteed to be in a specific order? If yes then this procedure makes sense. (pdfs etc. are problematic because of reordering.)

You can either egrep -v through the output of git diff-index, or define a diff driver: set an attribute, say "dumpdiff", for dump files (see gitattributes) and define diff driver as git config diff.dumpdiff.textconv = dumpdiff.sh where dumpdiff.sh is "egrep -v ...". You may need to call diff-index with --ext-diff. I haven't tried, though ;)

Cheers, Michael

Björn Steinbrink· Nov 21, 2009, 18:07 UTC · re: Dirk Süsserott · lore

Re: How to make git diff-* ignore some patterns?

On 2009.11.21 17:40:14 +0100, Dirk Süsserott wrote:

> is there a way to tell "git diff-index" to ignore some special
> patterns, such that /^-- Dump completed on .*$/ is NOT recognized as
> a difference and "git diff-index" returns 0 if that's the only
> difference?

If you don't mind losing that line, you could use a clean filter via .gitattributes:

echo '*.sql filter=mysql_dump' >> .gitattributes git config filter.mysql_dump.clean "sed -e '/^-- Dump completed on .*$/d'"

That way, git will filter all *.sql paths through that sed command before storing them as blobs, dropping that "Dump completed" line from the data stored in the repo.

Björn

Dirk Süsserott· Nov 22, 2009, 15:51 UTC · re: Björn Steinbrink · lore

Re: How to make git diff-* ignore some patterns?

Am 21.11.2009 19:07 schrieb Björn Steinbrink:

Show 18 quoted lines

> On 2009.11.21 17:40:14 +0100, Dirk Süsserott wrote:
>> is there a way to tell "git diff-index" to ignore some special
>> patterns, such that /^-- Dump completed on .*$/ is NOT recognized as
>> a difference and "git diff-index" returns 0 if that's the only
>> difference?
> 
> If you don't mind losing that line, you could use a clean filter via
> .gitattributes:
> 
> echo '*.sql filter=mysql_dump' >> .gitattributes
> git config filter.mysql_dump.clean "sed -e '/^-- Dump completed on .*$/d'"
> 
> That way, git will filter all *.sql paths through that sed command
> before storing them as blobs, dropping that "Dump completed" line from
> the data stored in the repo.
> 
> Björn
>

Thank you Björn and Michael,

Your suggestions were really helpful. I decided to use Björn's 'clean filter' approach. It works great.

-- Dirk