How to simulate a real checkout to test a new smudge filter?

I am looking for a way to force smudge filter to run by simulating a real life checkout. Let's say I just created a new branch and did not modify any files but want to test my new smudge filter. According to some answers such as https://stackoverflow.com/questions/22909620/git-smudge-clean-filter-between-branches and https://stackoverflow.com/questions/21652242/git-re-checkout-files-after-creating-smudge-filter it should be possible by running:

git checkout HEAD --

but in doesn't work with git 2.9.0. Method suggested in accepted answer here https://stackoverflow.com/questions/17223527/how-do-i-force-git-to-checkout-the-master-branch-and-remove-carriage-returns-aft works but I don't like because it seems fragile. Is there a safe way to do what I want to do in Git still today?

On 06.09.16 19:47, john smith wrote:

It depends what you mean with "safe way".

git checkout, git checkout -f or other combinations will only

overwrite/rewrite the files in the working tree, if, and only if,

git comes to the conclusion that "git add" will do something,

like replace a blob for a file in the index.

(And by running "rm .git/index git will evaluate the "clean" filters,

and the CRLF->LF conversion).

If you want to test a smudge filter, simply remove the file:

mv file /tmp && git checkout file

On 9/6/16, Torsten Bögershausen <tboegi@web.de> wrote:

I want to store all my dotfiles in a single repoitory. The problem is that that some specific pieces of these files are different on different machines. I have a special .conf file that is different on every branch and contains machine-specific definitions of some variables such as EMAIL or SMTP server. In my smudge filter I call a script which parses .conf file and replace all template variable definitions saved in the given file with correct definitions. For example in my ~/.bashrc I have this on all branches:

export EMAIL="@EMAIL@"

and in my .conf file on `home' branch

EMAIL=home@address.com

and on `work' branch:

EMAIL=work@address.com

And in .gitattributes on both branches:

bash/.bashrc     filter=make-usable

I also have single `master' branch that only contains template dotfiles and no .conf. When setting up a new machine I could just create a new branch off master branch and add a new .conf.

In turn, clean filter replace all correct definitions in the given dotfiles back into template definitions.

I'd prefer smudge/clean filters instead of `make' scripts etc. to convert template dotfiles into something usable and back because filters:

1. could be run automatically

2. do not modify files as shown by `git show HEAD:<file>' and
therefore no files are reported as modified by git status and also
there are not conflicts when merging master into work/home branch.

I have problems because with point 1 because apparently smudge filter is not run automatically every time when branch is changed if files listed in .gitattributes do not change. As the last resort I could force smudge/clean filter to run just to keep advantage specified in point 2.

john smith venit, vidit, dixit 06.09.2016 23:01:

I'm afraid smudge/clean are not a good match for your use case: smudge can do anything that clean removes again, i.e.: if you smudge a blob, then clean it, it has to be the same blob.

Your smudge filter seems to give different results for the same blob depending on other variables (your .conf), but git doesn't care as long as clean output is the same. Also, git cannot possibly know when "it is time" to rerun smudge.

Maybe "source"-ing .conf files would be an alternative approach for you, or using hooks?

Cheers Michael

W dniu 06.09.2016 o 23:01, john smith pisze:

Couldn't you use post-checkout hook plus clean filter instead of clean/smudge filter pair, if the smudge part depends on the branch?

Or make post-checkout hook invoke smudge filter... though `git cat-file --filters` is not in any released version, I think...

Best,

On 9/8/16, Jakub Narębski <jnareb@gmail.com> wrote:

I moved smudge-filter functionality to post-checkout hook and only left clean filter. Now post-checkout hooks is called every time when changing branches but it's not working together with clean filter when changing branches. For example, when I have this on master in bash/.bashrc:

export EMAIL="@EMAIL@"

After moving to `work' it becomes this thanks to post-checkout hook which replaces @EMAIL@ with correct definition:

export EMAIL="work@address.com"

But when I move to `home' branch from `work' I still get this:

export EMAIL="work@address.com"

post-checkout didn't work here because it can't see @EMAIL@ part in bash/.bashrc because clean filter has not been run. I need to run this single command to force `clean' filter to run to replace work@address.com with @EMAIL@ and let post-checkout replace it with correct value:

git checkout .

And now I get the correct definition:

export EMAIL="home@address.com"

So it seems that clean filter is only run when checking out paths but not when checking out branches. Is my thinking correct?

john smith <wempwer@gmail.com> writes:

If the new branch you are going to and your current branch record different contents for a path, that path will have to be checked out to the working tree, and smudge will be involved in the process to turn the "clean" contents stored in a Git object needs to be smudged for use in the outside world. If both branches have the same contents, then there is no need to overwrite the path in the working tree with the same thing, so it is not touched.

The clean and smudge operations should look _only_ at the contents they are filtering, and nothing else, and the clean/smudge filtering mechanism is designed to support that use case. It is not designed to do things like embedding the name of the branch that is being checked out into the result.

On 9/10/16, Junio C Hamano <gitster@pobox.com> wrote:

Ok, I think I get it. It was actually my original problem with smudge filters because I wanted them to be run after *every* checkout even if file contents stayed the same (hence the subject).

Now Jakub suggested post-checkout hook in conjunction with only clean filter for my problem of managing dotfiles but I think I don't fully get it. The problem is that in the scenario presented in my last e-mail clean filter is run in the situation which doesn't like a checkin to me. Is `git checkout <PATH>' doing a *checkin*" under the hood so that the clean filter is called? What does actually `checkin' mean in Git? I thought that checkin it's the same as committing a file into the repository.

john smith <wempwer@gmail.com> writes:

I do not know what Jakub meant.

When we need to see if we need to update a working tree file, and when we cannot tell if you modified the working tree file since you checked it out, we may have to run the clean filter (and other conversions collectively called "convert-to-git") on the contents in it and see if the resulting blob matches what is recorded in the index (if they match, there is no need to update the working tree file; if they don't match, we either report that you have a local modification, or we overwrite it with the contents, depending on what operation we are doing for the user).

W dniu 10.09.2016 o 01:07, john smith pisze:

And that's not the case, for performance reasons.

I was wrong, I'm sorry. My mistake.

You would need post-checkout hook together with clean / smudge filters (though you could get by without smudge filter, at least in theory...). The `post-checkout` hook could run e.g. "git checkout -- '*.conf'" to force use of smudge filter, after checking that it was a change of branch ("git checkout <commit-ish>"). See githooks(5) for details (last, third parameter passed to hook script is 1, rather than 0).

Unfortunately I don't see a way to query .gitattributes files to find out all patterns that match specific attribute; you would need to duplicate configuration:

  .gitattributes:
  *.conf filter=transform

  .git/config
  [filter "transform"]
  	clean  = replace-with-placeholder %f
  	smudge = expand-with-branchname %f

  .git/hooks/post-checkout
  #!/bin/sh

  test "$3" -eq "1" && git checkout -- '*.conf'

Or something like that.

On 9/10/16, Jakub Narębski <jnareb@gmail.com> wrote:

Neat idea. I could still have an advantage of filter as git status wouldn't show that files are changed and there would be no merge conflicts and additionally filters would run after every checkout even if files stayed the same. But are sure that `git checkout -- <FILE>' actually forces checkout and runs smudge filter? It doesn't work for me neither with 2.9.0 nor 2.10.0.85.gcda1bbd. For a minimal example, let's say that I have this in .gitattributes:

a     filter=test-filter

And this in .git/config:

smudge = touch /tmp/SMUDGE_RUN && cat

And also have a tracked file named `a' in the repository. Now this should create /tmp/SMUDGE_RUN:

git checkout  -- a

but it doesn't.

This also doesn't work:

git checkout -f -- a

But this does:

git checkout HEAD^ && git co -

It's a secondary issue, but - in my repository I only have one .conf file on every branch with real definitions of machine-specific variables and a number of various config files for various programs such as bash/.bashrc or screen/.screenrc so in my case it would be:

  .gitattributes:
  bash/.basrc filter=transform
  git/.gitconfig filter=transform
  (...)

  .git/config
  [filter "transform"]
        clean  = replace-with-placeholder %f
        smudge = expand-with-branchname %f

  .git/hooks/post-checkout
  #!/bin/sh

  test "$3" -eq "1" && (
       git checkout -- bash/.bashrc &&
       git checkout -- git/.gitconfig
       (...)
       )

Of course a list of files in .git/hooks/post-checkout could be generated at runtime from first column of .gitattributes.