# gitattributes - clean filter invoked on pull? 10 messages from 2011-04-11 to 2011-04-11. Participants: Miklos Vajna, Ramkumar Ramachandra, Michael J Gruber, Johannes Sixt, Dmitry Potapov. Thread: https://gitlist.dev/t/27056 ## Miklos Vajna, 2011-04-11 08:42 Subject: gitattributes - clean filter invoked on pull? Message-ID: <20110411084229.GW5146@genesis.frugalware.org> URL: https://gitlist.dev/e/20110411084229.GW5146%40genesis.frugalware.org ``` Hi, Background: We at LibreOffice are trying to use the 'filter' gitattributes feature to clean up line wrappings in po files. The problem is that it seems the clean filter - which is supposed to be invoked only in case a new blob is created - is invoked even on clone/pull, and other developers are claiming that it slows down their workflow. Is this a bug? I don't exactly understand why this would be necessary. Here is a short script to reproduce the issue: ---- rm -rf client* mkdir client cd client git init git config filter.po.clean 'echo foo >&2 && cat' git config filter.po.smudge cat echo '*.po filter=po' > .gitattributes touch foo.po git add .gitattributes foo.po git commit -m foo cd .. git clone client client2 cd client2 git config filter.po.clean 'echo foo >&2 && cat' git config filter.po.smudge cat cd .. cd client echo aaa > foo.po git commit -am second cd .. cd client2/ git pull ---- Its output here with 1.7.4.4: ---- $ sh test.sh Initialized empty Git repository in /home/vmiklos/git/t/client/.git/ foo foo [master (root-commit) bbf8490] foo foo 1 files changed, 1 insertions(+), 0 deletions(-) create mode 100644 .gitattributes create mode 100644 foo.po Cloning into client2... done. foo [master foo foo e37f5ab] second foo foo 1 files changed, 1 insertions(+), 0 deletions(-) remote: Counting objects: 5, done. remote: Compressing objects: 100% (2/2), done. remote: Total 3 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. From /home/vmiklos/git/t/client bbf8490..e37f5ab master -> origin/master Updating bbf8490..e37f5ab foo Fast-forward foo foo.po | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) ---- Any thoughts why the clean filter is invoked on pull? Thanks. ``` ## Ramkumar Ramachandra, 2011-04-11 09:19 Subject: Re: gitattributes - clean filter invoked on pull? Message-ID: <20110411091919.GE28959@kytes> URL: https://gitlist.dev/e/20110411091919.GE28959%40kytes In-Reply-To: <20110411084229.GW5146@genesis.frugalware.org> ``` Hi Miklos, Miklos Vajna writes: > Background: We at LibreOffice are trying to use the 'filter' > gitattributes feature to clean up line wrappings in po files. > > The problem is that it seems the clean filter - which is supposed to be > invoked only in case a new blob is created - is invoked even on > clone/pull, and other developers are claiming that it slows down their > workflow. > > Is this a bug? I don't exactly understand why this would be necessary. >From config.txt: - 'clean' is "The command which is used to convert the content of a worktree file to a blob upon checkin". - 'smudge' is "The command which is used to convert the content of a blob object to a worktree file upon checkout." According to the documentation, 'smudge' is *supposed* to be invoked on a clone/ pull, since it involves a checkout. I don't see how you can avoid running these filters on every checkin/ checkout unless you cache the result somewhere. -- Ram ``` ## Miklos Vajna, 2011-04-11 09:31 Subject: Re: gitattributes - clean filter invoked on pull? Message-ID: <20110411093114.GY5146@genesis.frugalware.org> URL: https://gitlist.dev/e/20110411093114.GY5146%40genesis.frugalware.org In-Reply-To: <20110411091919.GE28959@kytes> ``` On Mon, Apr 11, 2011 at 02:49:21PM +0530, Ramkumar Ramachandra wrote: > > Is this a bug? I don't exactly understand why this would be necessary. > > From config.txt: > - 'clean' is "The command which is used to convert the content of a > worktree file to a blob upon checkin". > - 'smudge' is "The command which is used to convert the content of a > blob object to a worktree file upon checkout." > > According to the documentation, 'smudge' is *supposed* to be invoked > on a clone/ pull, since it involves a checkout. I don't see how you > can avoid running these filters on every checkin/ checkout unless you > cache the result somewhere. That's not a problem - the issue I pointed out is that the 'clean' one is invoked on pull/clone, and it takes time if it's applied to several files. 'smudge' is just a 'cat', I don't care about it. :) Thanks. ``` ## Ramkumar Ramachandra, 2011-04-11 09:50 Subject: Re: gitattributes - clean filter invoked on pull? Message-ID: <20110411095001.GG28959@kytes> URL: https://gitlist.dev/e/20110411095001.GG28959%40kytes In-Reply-To: <20110411093114.GY5146@genesis.frugalware.org> ``` Hi Miklos, Miklos Vajna writes: > On Mon, Apr 11, 2011 at 02:49:21PM +0530, Ramkumar Ramachandra wrote: > > > Is this a bug? I don't exactly understand why this would be necessary. > > > > From config.txt: > > - 'clean' is "The command which is used to convert the content of a > > worktree file to a blob upon checkin". > > - 'smudge' is "The command which is used to convert the content of a > > blob object to a worktree file upon checkout." > > > > According to the documentation, 'smudge' is *supposed* to be invoked > > on a clone/ pull, since it involves a checkout. I don't see how you > > can avoid running these filters on every checkin/ checkout unless you > > cache the result somewhere. > > That's not a problem - the issue I pointed out is that the 'clean' one > is invoked on pull/clone, and it takes time if it's applied to several > files. > > 'smudge' is just a 'cat', I don't care about it. :) Ah, sorry about that. There actually seems to be a bug :| -- Ram ``` ## Michael J Gruber, 2011-04-11 09:50 Subject: Re: gitattributes - clean filter invoked on pull? Message-ID: <4DA2CED1.6070107@drmicha.warpmail.net> URL: https://gitlist.dev/e/4DA2CED1.6070107%40drmicha.warpmail.net In-Reply-To: <20110411091919.GE28959@kytes> ``` Ramkumar Ramachandra venit, vidit, dixit 11.04.2011 11:19: > Hi Miklos, > > Miklos Vajna writes: >> Background: We at LibreOffice are trying to use the 'filter' >> gitattributes feature to clean up line wrappings in po files. >> >> The problem is that it seems the clean filter - which is supposed to be >> invoked only in case a new blob is created - is invoked even on >> clone/pull, and other developers are claiming that it slows down their >> workflow. >> >> Is this a bug? I don't exactly understand why this would be necessary. > > From config.txt: > - 'clean' is "The command which is used to convert the content of a > worktree file to a blob upon checkin". > - 'smudge' is "The command which is used to convert the content of a > blob object to a worktree file upon checkout." > > According to the documentation, 'smudge' is *supposed* to be invoked > on a clone/ pull, since it involves a checkout. I don't see how you > can avoid running these filters on every checkin/ checkout unless you > cache the result somewhere. Exactly that is why it's surprising that the clean filter is invoked on pull - clean is about checking in, pull only checks out. If you run your script with GIT_TRACE=1 you see that the two last clean invocations come from merge and gc. They go away when you do a fetch only. Note that with clean/smudge, a check "worktree == repo" requires a conversion of "worktree" to what would be checked in, and that uses "clean". That's why it's invoked not only for "commit". But maybe there is a better solution for your actual use case? Do you want to ignore line wrap or normalise it? Michael ``` ## Johannes Sixt, 2011-04-11 10:00 Subject: Re: gitattributes - clean filter invoked on pull? Message-ID: <4DA2D14D.6010707@viscovery.net> URL: https://gitlist.dev/e/4DA2D14D.6010707%40viscovery.net In-Reply-To: <20110411093114.GY5146@genesis.frugalware.org> ``` Am 4/11/2011 11:31, schrieb Miklos Vajna: > On Mon, Apr 11, 2011 at 02:49:21PM +0530, Ramkumar Ramachandra wrote: >>> Is this a bug? I don't exactly understand why this would be necessary. >> >> From config.txt: >> - 'clean' is "The command which is used to convert the content of a >> worktree file to a blob upon checkin". >> - 'smudge' is "The command which is used to convert the content of a >> blob object to a worktree file upon checkout." >> >> According to the documentation, 'smudge' is *supposed* to be invoked >> on a clone/ pull, since it involves a checkout. I don't see how you >> can avoid running these filters on every checkin/ checkout unless you >> cache the result somewhere. > > That's not a problem - the issue I pointed out is that the 'clean' one > is invoked on pull/clone, and it takes time if it's applied to several > files. The invocation is only needed when files are marked as "racily clean", because in this case git has to check whether the worktree contents are what is recorded in the index or not. This can happen a lot when you have a fast machine where many worktree files and the index itself can be written within the same (wall clock) second. You example is so short that it triggers this case almost reliably. When git pull merges the fetched commit, it has to determine whether there are no changes in any of the files that are to be updated by the merge. If one such file is marked as racily clean, the worktree contents must be inspected, which in turn means that the clean filter has to be used. If you insert before the final 'git pull': sleep 1 git reset sleep 1 you will notice that some clean filter calls happen before the 'git pull' because git 'git reset' rectifies the racily-clean entry. This just explains what you observed. I haven't thought about how you should change your workflow to avoid this behavior. My guess is that the extra clean filters called by 'git pull' don't actually happen that frequently during normal interactive work that touches the index. Perhaps you are also bitten by a regression in 'git status', which does not correct the racily-clean entries even though it should (fixed in git 1.7.4.4.), and therefore the clean filter is run more often than necessary. > > 'smudge' is just a 'cat', I don't care about it. :) Then you can just remove it from the config and save a fork(). You don't have to configure both clean and smudge filters. -- Hannes ``` ## Dmitry Potapov, 2011-04-11 10:04 Subject: Re: gitattributes - clean filter invoked on pull? Message-ID: URL: https://gitlist.dev/e/BANLkTi%3DL0rTse2TTNaaO36GUNj5AR1hBXA%40mail.gmail.com In-Reply-To: <20110411084229.GW5146@genesis.frugalware.org> ``` Hi, On Mon, Apr 11, 2011 at 12:42 PM, Miklos Vajna wrote: > > The problem is that it seems the clean filter - which is supposed to be > invoked only in case a new blob is created - is invoked even on > clone/pull, and other developers are claiming that it slows down their > workflow. > > Is this a bug? I don't exactly understand why this would be necessary. No, it is not a bug. Git may invoke the clean filter when a file is not changed to make sure that the file is not changed. It is necessary to prevent a race when a file is changed so quickly that its timestamp does not change. So, what git does is compare timestamp of your file and the index file. Because the index file is written after all files, its timestamp should be later than any file in the repository. However, if the timestamp resolution is not sufficient (i.e. timestamp is the same), git may re-read recently checkout file to make sure that there were no changes to it. During this reading, the clean filter will be invoked. So, clean filter may be invoked extra time, but smudge filter should not. Dmitry ``` ## Miklos Vajna, 2011-04-11 10:16 Subject: Re: gitattributes - clean filter invoked on pull? Message-ID: <20110411101614.GB5146@genesis.frugalware.org> URL: https://gitlist.dev/e/20110411101614.GB5146%40genesis.frugalware.org In-Reply-To: <4DA2CED1.6070107@drmicha.warpmail.net> ``` On Mon, Apr 11, 2011 at 11:50:09AM +0200, Michael J Gruber wrote: > But maybe there is a better solution for your actual use case? Do you > want to ignore line wrap or normalise it? I want to avoid those, so we get readable diffs, whatever line width the different translator tools are using. (If tool1 is using 72 and tool2 is using 80, then it would reformat the whole file.) Do you have a better idea than git config filter.po.clean 'msgcat - --no-wrap' ? Thanks. ``` ## Michael J Gruber, 2011-04-11 10:41 Subject: Re: gitattributes - clean filter invoked on pull? Message-ID: <4DA2DAC6.1010009@drmicha.warpmail.net> URL: https://gitlist.dev/e/4DA2DAC6.1010009%40drmicha.warpmail.net In-Reply-To: <20110411101614.GB5146@genesis.frugalware.org> ``` Miklos Vajna venit, vidit, dixit 11.04.2011 12:16: > On Mon, Apr 11, 2011 at 11:50:09AM +0200, Michael J Gruber wrote: >> But maybe there is a better solution for your actual use case? Do you >> want to ignore line wrap or normalise it? > > I want to avoid those, so we get readable diffs, whatever line width the > different translator tools are using. (If tool1 is using 72 and tool2 is > using 80, then it would reformat the whole file.) > > Do you have a better idea than > > git config filter.po.clean 'msgcat - --no-wrap' > > ? git config diff.po.textconv 'msgcat - --no-wrap' git config diff.po.cachetextconv true If you want to normalise the repo, you may want to look at hooks instead of clean/smudge if they are a performance problem. Michael ``` ## Miklos Vajna, 2011-04-11 11:14 Subject: Re: gitattributes - clean filter invoked on pull? Message-ID: <20110411111428.GD5146@genesis.frugalware.org> URL: https://gitlist.dev/e/20110411111428.GD5146%40genesis.frugalware.org In-Reply-To: <4DA2DAC6.1010009@drmicha.warpmail.net> ``` On Mon, Apr 11, 2011 at 12:41:10PM +0200, Michael J Gruber wrote: > git config diff.po.textconv 'msgcat - --no-wrap' > git config diff.po.cachetextconv true > > If you want to normalise the repo, you may want to look at hooks instead > of clean/smudge if they are a performance problem. Ah - using hooks instead is indeed a better option. Thanks! ```