{"thread":{"id":"27056","subject":"gitattributes - clean filter invoked on pull?","startedAt":"2011-04-11T08:42:29Z","lastAt":"2011-04-11T11:14:28Z","messageCount":10,"participants":["Miklos Vajna","Ramkumar Ramachandra","Michael J Gruber","Johannes Sixt","Dmitry Potapov"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"165606","messageId":"20110411084229.GW5146@genesis.frugalware.org","threadId":"27056","inReplyTo":null,"subject":"gitattributes - clean filter invoked on pull?","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2011-04-11T08:42:29Z","receivedAt":"2011-04-11T08:42:29Z","isPatch":false,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"Hi,\n\nBackground: We at LibreOffice are trying to use the 'filter'\ngitattributes feature to clean up line wrappings in po files.\n\nThe problem is that it seems the clean filter - which is supposed to be\ninvoked only in case a new blob is created - is invoked even on\nclone/pull, and other developers are claiming that it slows down their\nworkflow.\n\nIs this a bug? I don't exactly understand why this would be necessary.\n\nHere is a short script to reproduce the issue:\n\n----\nrm -rf client*\nmkdir client\ncd client\ngit init\ngit config filter.po.clean 'echo foo >&2 && cat'\ngit config filter.po.smudge cat\necho '*.po filter=po' > .gitattributes\ntouch foo.po\ngit add .gitattributes foo.po\ngit commit -m foo\ncd ..\ngit clone client client2\ncd client2\ngit config filter.po.clean 'echo foo >&2 && cat'\ngit config filter.po.smudge cat \ncd ..\ncd client\necho aaa > foo.po \ngit commit -am second\ncd ..\ncd client2/\ngit pull\n----\n\nIts output here with 1.7.4.4:\n\n----\n$ sh test.sh \nInitialized empty Git repository in /home/vmiklos/git/t/client/.git/\nfoo\nfoo\n[master (root-commit) bbf8490] foo\nfoo\n 1 files changed, 1 insertions(+), 0 deletions(-)\n create mode 100644 .gitattributes\n create mode 100644 foo.po\nCloning into client2...\ndone.\nfoo\n[master foo\nfoo\ne37f5ab] second\nfoo\nfoo\n 1 files changed, 1 insertions(+), 0 deletions(-)\nremote: Counting objects: 5, done.\nremote: Compressing objects: 100% (2/2), done.\nremote: Total 3 (delta 0), reused 0 (delta 0)\nUnpacking objects: 100% (3/3), done.\nFrom /home/vmiklos/git/t/client\n bbf8490..e37f5ab master -> origin/master\nUpdating bbf8490..e37f5ab\nfoo\nFast-forward\nfoo\n foo.po | 1 +\n 1 files changed, 1 insertions(+), 0 deletions(-)\n----\n\nAny thoughts why the clean filter is invoked on pull?\n\nThanks.\n"},{"id":"165612","messageId":"20110411091919.GE28959@kytes","threadId":"27056","inReplyTo":"20110411084229.GW5146@genesis.frugalware.org","subject":"Re: gitattributes - clean filter invoked on pull?","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-11T09:19:21Z","receivedAt":"2011-04-11T09:19:21Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Miklos,\n\nMiklos Vajna writes:\n> Background: We at LibreOffice are trying to use the 'filter'\n> gitattributes feature to clean up line wrappings in po files.\n> \n> The problem is that it seems the clean filter - which is supposed to be\n> invoked only in case a new blob is created - is invoked even on\n> clone/pull, and other developers are claiming that it slows down their\n> workflow.\n> \n> Is this a bug? I don't exactly understand why this would be necessary.\n\n>From config.txt:\n- 'clean' is \"The command which is used to convert the content of a\nworktree file to a blob upon checkin\".\n- 'smudge' is \"The command which is used to convert the content of a\nblob object to a worktree file upon checkout.\"\n\nAccording to the documentation, 'smudge' is *supposed* to be invoked\non a clone/ pull, since it involves a checkout. I don't see how you\ncan avoid running these filters on every checkin/ checkout unless you\ncache the result somewhere.\n\n-- Ram\n"},{"id":"165613","messageId":"20110411093114.GY5146@genesis.frugalware.org","threadId":"27056","inReplyTo":"20110411091919.GE28959@kytes","subject":"Re: gitattributes - clean filter invoked on pull?","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2011-04-11T09:31:14Z","receivedAt":"2011-04-11T09:31:14Z","isPatch":false,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"On Mon, Apr 11, 2011 at 02:49:21PM +0530, Ramkumar Ramachandra wrote:\n> > Is this a bug? I don't exactly understand why this would be necessary.\n> \n> From config.txt:\n> - 'clean' is \"The command which is used to convert the content of a\n> worktree file to a blob upon checkin\".\n> - 'smudge' is \"The command which is used to convert the content of a\n> blob object to a worktree file upon checkout.\"\n> \n> According to the documentation, 'smudge' is *supposed* to be invoked\n> on a clone/ pull, since it involves a checkout. I don't see how you\n> can avoid running these filters on every checkin/ checkout unless you\n> cache the result somewhere.\n\nThat's not a problem - the issue I pointed out is that the 'clean' one\nis invoked on pull/clone, and it takes time if it's applied to several\nfiles.\n\n'smudge' is just a 'cat', I don't care about it. :)\n\nThanks.\n"},{"id":"165615","messageId":"20110411095001.GG28959@kytes","threadId":"27056","inReplyTo":"20110411093114.GY5146@genesis.frugalware.org","subject":"Re: gitattributes - clean filter invoked on pull?","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-04-11T09:50:03Z","receivedAt":"2011-04-11T09:50:03Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Miklos,\n\nMiklos Vajna writes:\n> On Mon, Apr 11, 2011 at 02:49:21PM +0530, Ramkumar Ramachandra wrote:\n> > > Is this a bug? I don't exactly understand why this would be necessary.\n> > \n> > From config.txt:\n> > - 'clean' is \"The command which is used to convert the content of a\n> > worktree file to a blob upon checkin\".\n> > - 'smudge' is \"The command which is used to convert the content of a\n> > blob object to a worktree file upon checkout.\"\n> > \n> > According to the documentation, 'smudge' is *supposed* to be invoked\n> > on a clone/ pull, since it involves a checkout. I don't see how you\n> > can avoid running these filters on every checkin/ checkout unless you\n> > cache the result somewhere.\n> \n> That's not a problem - the issue I pointed out is that the 'clean' one\n> is invoked on pull/clone, and it takes time if it's applied to several\n> files.\n> \n> 'smudge' is just a 'cat', I don't care about it. :)\n\nAh, sorry about that. There actually seems to be a bug :|\n\n-- Ram\n"},{"id":"165614","messageId":"4DA2CED1.6070107@drmicha.warpmail.net","threadId":"27056","inReplyTo":"20110411091919.GE28959@kytes","subject":"Re: gitattributes - clean filter invoked on pull?","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2011-04-11T09:50:09Z","receivedAt":"2011-04-11T09:50:09Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Ramkumar Ramachandra venit, vidit, dixit 11.04.2011 11:19:\n> Hi Miklos,\n> \n> Miklos Vajna writes:\n>> Background: We at LibreOffice are trying to use the 'filter'\n>> gitattributes feature to clean up line wrappings in po files.\n>>\n>> The problem is that it seems the clean filter - which is supposed to be\n>> invoked only in case a new blob is created - is invoked even on\n>> clone/pull, and other developers are claiming that it slows down their\n>> workflow.\n>>\n>> Is this a bug? I don't exactly understand why this would be necessary.\n> \n> From config.txt:\n> - 'clean' is \"The command which is used to convert the content of a\n> worktree file to a blob upon checkin\".\n> - 'smudge' is \"The command which is used to convert the content of a\n> blob object to a worktree file upon checkout.\"\n> \n> According to the documentation, 'smudge' is *supposed* to be invoked\n> on a clone/ pull, since it involves a checkout. I don't see how you\n> can avoid running these filters on every checkin/ checkout unless you\n> cache the result somewhere.\n\nExactly that is why it's surprising that the clean filter is invoked on\npull - clean is about checking in, pull only checks out.\n\nIf you run your script with GIT_TRACE=1 you see that the two last clean\ninvocations come from merge and gc. They go away when you do a fetch only.\n\nNote that with clean/smudge, a check \"worktree == repo\" requires a\nconversion of \"worktree\" to what would be checked in, and that uses\n\"clean\". That's why it's invoked not only for \"commit\".\n\nBut maybe there is a better solution for your actual use case? Do you\nwant to ignore line wrap or normalise it?\n\nMichael\n"},{"id":"165616","messageId":"4DA2D14D.6010707@viscovery.net","threadId":"27056","inReplyTo":"20110411093114.GY5146@genesis.frugalware.org","subject":"Re: gitattributes - clean filter invoked on pull?","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2011-04-11T10:00:45Z","receivedAt":"2011-04-11T10:00:45Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 4/11/2011 11:31, schrieb Miklos Vajna:\n> On Mon, Apr 11, 2011 at 02:49:21PM +0530, Ramkumar Ramachandra wrote:\n>>> Is this a bug? I don't exactly understand why this would be necessary.\n>>\n>> From config.txt:\n>> - 'clean' is \"The command which is used to convert the content of a\n>> worktree file to a blob upon checkin\".\n>> - 'smudge' is \"The command which is used to convert the content of a\n>> blob object to a worktree file upon checkout.\"\n>>\n>> According to the documentation, 'smudge' is *supposed* to be invoked\n>> on a clone/ pull, since it involves a checkout. I don't see how you\n>> can avoid running these filters on every checkin/ checkout unless you\n>> cache the result somewhere.\n> \n> That's not a problem - the issue I pointed out is that the 'clean' one\n> is invoked on pull/clone, and it takes time if it's applied to several\n> files.\n\nThe invocation is only needed when files are marked as \"racily clean\",\nbecause in this case git has to check whether the worktree contents are\nwhat is recorded in the index or not. This can happen a lot when you have\na fast machine where many worktree files and the index itself can be\nwritten within the same (wall clock) second. You example is so short that\nit triggers this case almost reliably.\n\nWhen git pull merges the fetched commit, it has to determine whether there\nare no changes in any of the files that are to be updated by the merge. If\none such file is marked as racily clean, the worktree contents must be\ninspected, which in turn means that the clean filter has to be used.\n\nIf you insert before the final 'git pull':\n\nsleep 1\ngit reset\nsleep 1\n\nyou will notice that some clean filter calls happen before the 'git pull'\nbecause git 'git reset' rectifies the racily-clean entry.\n\nThis just explains what you observed. I haven't thought about how you\nshould change your workflow to avoid this behavior. My guess is that the\nextra clean filters called by 'git pull' don't actually happen that\nfrequently during normal interactive work that touches the index.\n\nPerhaps you are also bitten by a regression in 'git status', which does\nnot correct the racily-clean entries even though it should (fixed in git\n1.7.4.4.), and therefore the clean filter is run more often than necessary.\n\n> \n> 'smudge' is just a 'cat', I don't care about it. :)\n\nThen you can just remove it from the config and save a fork(). You don't\nhave to configure both clean and smudge filters.\n\n-- Hannes\n"},{"id":"165617","messageId":"BANLkTi=L0rTse2TTNaaO36GUNj5AR1hBXA@mail.gmail.com","threadId":"27056","inReplyTo":"20110411084229.GW5146@genesis.frugalware.org","subject":"Re: gitattributes - clean filter invoked on pull?","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2011-04-11T10:04:18Z","receivedAt":"2011-04-11T10:04:18Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"Hi,\n\nOn Mon, Apr 11, 2011 at 12:42 PM, Miklos Vajna wrote:\n>\n> The problem is that it seems the clean filter - which is supposed to be\n> invoked only in case a new blob is created - is invoked even on\n> clone/pull, and other developers are claiming that it slows down their\n> workflow.\n>\n> Is this a bug? I don't exactly understand why this would be necessary.\n\nNo, it is not a bug. Git may invoke the clean filter when a file is not\nchanged to make sure that the file is not changed. It is necessary to\nprevent a race when a file is changed so quickly that its timestamp does\nnot change. So, what git does is compare timestamp of your file and the\nindex file. Because the index file is written after all files, its\ntimestamp should be later than any file in the repository. However, if\nthe timestamp resolution is not sufficient (i.e. timestamp is the same),\ngit may re-read recently checkout file to make sure that there were no\nchanges to it. During this reading, the clean filter will be invoked.\n\nSo, clean filter may be invoked extra time, but smudge filter should not.\n\n\nDmitry\n"},{"id":"165618","messageId":"20110411101614.GB5146@genesis.frugalware.org","threadId":"27056","inReplyTo":"4DA2CED1.6070107@drmicha.warpmail.net","subject":"Re: gitattributes - clean filter invoked on pull?","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2011-04-11T10:16:14Z","receivedAt":"2011-04-11T10:16:14Z","isPatch":false,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"On Mon, Apr 11, 2011 at 11:50:09AM +0200, Michael J Gruber wrote:\n> But maybe there is a better solution for your actual use case? Do you\n> want to ignore line wrap or normalise it?\n\nI want to avoid those, so we get readable diffs, whatever line width the\ndifferent translator tools are using. (If tool1 is using 72 and tool2 is\nusing 80, then it would reformat the whole file.)\n\nDo you have a better idea than\n\n git config filter.po.clean 'msgcat - --no-wrap'\n\n?\n\nThanks.\n"},{"id":"165619","messageId":"4DA2DAC6.1010009@drmicha.warpmail.net","threadId":"27056","inReplyTo":"20110411101614.GB5146@genesis.frugalware.org","subject":"Re: gitattributes - clean filter invoked on pull?","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2011-04-11T10:41:10Z","receivedAt":"2011-04-11T10:41:10Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Miklos Vajna venit, vidit, dixit 11.04.2011 12:16:\n> On Mon, Apr 11, 2011 at 11:50:09AM +0200, Michael J Gruber wrote:\n>> But maybe there is a better solution for your actual use case? Do you\n>> want to ignore line wrap or normalise it?\n> \n> I want to avoid those, so we get readable diffs, whatever line width the\n> different translator tools are using. (If tool1 is using 72 and tool2 is\n> using 80, then it would reformat the whole file.)\n> \n> Do you have a better idea than\n> \n> git config filter.po.clean 'msgcat - --no-wrap'\n> \n> ?\n\ngit config diff.po.textconv 'msgcat - --no-wrap'\ngit config diff.po.cachetextconv true\n\nIf you want to normalise the repo, you may want to look at hooks instead\nof clean/smudge if they are a performance problem.\n\nMichael\n"},{"id":"165621","messageId":"20110411111428.GD5146@genesis.frugalware.org","threadId":"27056","inReplyTo":"4DA2DAC6.1010009@drmicha.warpmail.net","subject":"Re: gitattributes - clean filter invoked on pull?","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2011-04-11T11:14:28Z","receivedAt":"2011-04-11T11:14:28Z","isPatch":false,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"On Mon, Apr 11, 2011 at 12:41:10PM +0200, Michael J Gruber wrote:\n> git config diff.po.textconv 'msgcat - --no-wrap'\n> git config diff.po.cachetextconv true\n> \n> If you want to normalise the repo, you may want to look at hooks instead\n> of clean/smudge if they are a performance problem.\n\nAh - using hooks instead is indeed a better option.\n\nThanks!\n"}]}