{"thread":{"id":"13202","subject":"Git on Windows, CRLF issues","startedAt":"2008-04-21T19:48:58Z","lastAt":"2008-04-24T16:56:40Z","messageCount":30,"participants":["Peter Karlsson","Johannes Schindelin","Jakub Narebski","Avery Pennarun","Jeff King","Johannes Sixt","Johan Herland","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"74882","messageId":"alpine.DEB.1.00.0804212145570.21181@perkele.intern.softwolves.pp.se","threadId":"13202","inReplyTo":null,"subject":"Git on Windows, CRLF issues","fromName":"Peter Karlsson","fromEmail":"peter@softwolves.pp.se","sentAt":"2008-04-21T19:48:58Z","receivedAt":"2008-04-21T19:48:58Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Hi!\n\nI have began moving old repositories for Windows-based software to Git. \nSince the tool I am moving from stores everything with CRLF line endings and \nhave RCS-like keyword expansion, I'm treating it all as binary data when \nexporting to Git, i.e I have CRLF in the checked-in data (and I do want \nthat, since this is Windows-only sources).\n\nNow the latests msysgit comes along and (finally!) sets core.autocrlf to \ntrue by default.\n\nHow do I handle this without having everyone breaking check-ins? I can't \nrequire everyone to do unset core.autocrlf globally. Can I do that with \ngitattributes?\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"74884","messageId":"alpine.DEB.1.00.0804212104560.2298@eeepc-johanness","threadId":"13202","inReplyTo":"alpine.DEB.1.00.0804212145570.21181@perkele.intern.softwolves.pp.se","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-04-21T20:07:43Z","receivedAt":"2008-04-21T20:07:43Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 21 Apr 2008, Peter Karlsson wrote:\n\n> Now the latests msysgit comes along and (finally!) sets core.autocrlf to \n> true by default.\n\nIt is actually nice to hear at least _somebody_ not insulting us for this \ndecision. Thank you!\n\n> How do I handle this without having everyone breaking check-ins? I can't \n> require everyone to do unset core.autocrlf globally. Can I do that with \n> gitattributes?\n\nI think that the only solution to this is (sorry!) to have one single big \ncheckin which converts all CR/LF to LF line endings...\n\nDesole,\nDscho\n"},{"id":"74886","messageId":"m3bq42ri56.fsf@localhost.localdomain","threadId":"13202","inReplyTo":"alpine.DEB.1.00.0804212145570.21181@perkele.intern.softwolves.pp.se","subject":"Re: Git on Windows, CRLF issues","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-04-21T21:51:37Z","receivedAt":"2008-04-21T21:51:37Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Peter Karlsson writes:\n\n> I have began moving old repositories for Windows-based software to\n> Git. Since the tool I am moving from stores everything with CRLF line\n> endings and have RCS-like keyword expansion, I'm treating it all as\n> binary data when exporting to Git, i.e I have CRLF in the checked-in\n> data (and I do want that, since this is Windows-only sources).\n> \n> Now the latests msysgit comes along and (finally!) sets core.autocrlf\n> to true by default.\n> \n> How do I handle this without having everyone breaking check-ins? I\n> can't require everyone to do unset core.autocrlf globally. Can I do\n> that with gitattributes?\n\nI think you can, by unsetting `crlf` attribute, i.e. putting the\nfollowing in .gitattributes:\n\n * -crlf\n\nSee gitattributes(5):\n\n `crlf`\n ^^^^^^\n\n This attribute controls the line-ending convention.\n\n [...]\n\n Unset::\n\n Unsetting the `crlf` attribute on a path is meant to\n mark the path as a \"binary\" file. The path never goes\n through line endings conversion upon checkin/checkout.\n\nNot tested!\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"74887","messageId":"32541b130804211453x77f3fd49hef645a417a9919ca@mail.gmail.com","threadId":"13202","inReplyTo":"alpine.DEB.1.00.0804212104560.2298@eeepc-johanness","subject":"Re: Git on Windows, CRLF issues","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-04-21T21:53:34Z","receivedAt":"2008-04-21T21:53:34Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 4/21/08, Johannes Schindelin wrote:\n> I think that the only solution to this is (sorry!) to have one single big\n> checkin which converts all CR/LF to LF line endings...\n\nIf it were me (and I hope it will be, soon, if we can entirely shut\ndown svn internally), I would prefer to use git-filter-branch to go\nthrough *all* my checkins and fix up the CRLFs in all of them. That\nway the history will be clean and diffs/annotates/merges will go more\nsmoothly.\n\nDoes anyone know the most efficient way to do this with\ngit-filter-branch, when there are already thousands of files in the\nrepo with CRLF in them? Running dos2unix on all the files for every\nsingle revision could take a *very* long time.\n\nHave fun,\n\nAvery\n"},{"id":"74899","messageId":"20080422023918.GA5402@sigill.intra.peff.net","threadId":"13202","inReplyTo":"32541b130804211453x77f3fd49hef645a417a9919ca@mail.gmail.com","subject":"Re: Git on Windows, CRLF issues","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-04-22T02:39:18Z","receivedAt":"2008-04-22T02:39:18Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 21, 2008 at 05:53:34PM -0400, Avery Pennarun wrote:\n\n> Does anyone know the most efficient way to do this with\n> git-filter-branch, when there are already thousands of files in the\n> repo with CRLF in them? Running dos2unix on all the files for every\n> single revision could take a *very* long time.\n\nYes, a tree filter would probably be quite slow due to checking out, and\nthen munging all of the files.\n\nYou could maybe do an index filter that gets the blob SHA1 of each file\nthat is new, and just munges those. But I think it is even simpler to\njust keep a cache of original blob hashes mapping to munged blob hashes.\n\nSomething like:\n\n git filter-branch --index-filter '\n git ls-files --stage |\n perl /path/to/caching-munger |\n git update-index --index-info\n '\n\nwhere your caching munger looks something like:\n\n-- >8 --\n#!/usr/bin/perl\n\nuse strict;\nuse DB_File;\nuse Fcntl;\ntie my %cache, 'DB_File', \"$ENV{HOME}/filter-cache\", O_RDWR|O_CREAT, 0666\n or die \"unable to open db: $!\";\n\nwhile(<>) {\n my ($mode, $hash, $path) = /^(\\d+) ([0-9a-f]{40}) \\d\\t(.*)/\n or die \"bad ls-files line: $_\";\n $cache{$hash} = munge($hash)\n unless exists $cache{$hash};\n print \"$mode $cache{$hash}\\t$path\\n\";\n}\n\nsub munge {\n my $h = shift;\n my $r = scalar `git show $h | sed 's/\\$/\\\\r/' | git hash-object -w --stdin`;\n chomp $r;\n return $r;\n}\n-- 8< --\n\nso we keep a dbm of the hash mapping, and do no work if we have already\nseen this blob. If we don't, then we actually do the expensive 'show |\nmunge | hash-object'. And here our munge adds a CR, but you should be\nable to do an arbitrary transformation.\n\n-Peff\n"},{"id":"74913","messageId":"480D863B.1040009@viscovery.net","threadId":"13202","inReplyTo":"alpine.DEB.1.00.0804212145570.21181@perkele.intern.softwolves.pp.se","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-22T06:31:23Z","receivedAt":"2008-04-22T06:31:23Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Peter Karlsson schrieb:\n> I have began moving old repositories for Windows-based software to Git.\n> Since the tool I am moving from stores everything with CRLF line endings\n> and have RCS-like keyword expansion, I'm treating it all as binary data\n> when exporting to Git, i.e I have CRLF in the checked-in data (and I do\n> want that, since this is Windows-only sources).\n> \n> Now the latests msysgit comes along and (finally!) sets core.autocrlf to\n> true by default.\n> \n> How do I handle this without having everyone breaking check-ins? I can't\n> require everyone to do unset core.autocrlf globally. Can I do that with\n> gitattributes?\n\nI see 2 other options:\n\n1. Create a custom setup of msysgit that has core.autocrlf set to false.\n\n2. You are still converting repositories? Convert the files in your\nrepository to LF. I did it like this, but it was a week -or more- worth of\nlabor to get the scripts in a shape that I could reproduce the conversion\n(and it all happened before core.autocrlf even existed).\n\n-- Hannes\n"},{"id":"74914","messageId":"480D88AA.1020004@viscovery.net","threadId":"13202","inReplyTo":"32541b130804211453x77f3fd49hef645a417a9919ca@mail.gmail.com","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-22T06:41:46Z","receivedAt":"2008-04-22T06:41:46Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Avery Pennarun schrieb:\n> Does anyone know the most efficient way to [convert CRLF] with\n> git-filter-branch, when there are already thousands of files in the\n> repo with CRLF in them? Running dos2unix on all the files for every\n> single revision could take a *very* long time.\n\nI chose to write a custom script. Otherwise, a file that stays the same\nthroughout the history would still have been converted on each commit. My\nscript converted each unique file only once, then reconstructed the tree\nobjects and then changed the commits.\n\nIn the end I don't think it payed off. It took me a week or so to convert\nthe repo; I just could have let filter-branch run for a week, too. But I\nalso have to mention that I did the CVS->git conversion a few times to get\na suitable history, and I also repeated the CRLF conversion sometimes, and\nback then git-filter-branch did not exist in its current shape.\n\n-- Hannes\n"},{"id":"74919","messageId":"Pine.LNX.4.64.0804220749080.22731@ds9.cixit.se","threadId":"13202","inReplyTo":"m3bq42ri56.fsf@localhost.localdomain","subject":"Re: Git on Windows, CRLF issues","fromName":"Peter Karlsson","fromEmail":"peter@softwolves.pp.se","sentAt":"2008-04-22T06:52:34Z","receivedAt":"2008-04-22T06:52:34Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Jakub Narebski:\n\n> I think you can, by unsetting `crlf` attribute, i.e. putting the\n> following in .gitattributes:\n> \n> * -crlf\n\nYeah, that does indeed seem to work, no matter how core.autocrlf is\nconfigured globally. I think this is the best way to go for the\nrepositories I am working on (as they are very much DOS/Windows-only).\n\nDoes anyone know how to hack an existing repository so that I can add\nsuch a .gitattributes file to all commits? I've tried reading the\ngit-filter-branch manual page a few times, but I am still confused by\nit.\n\nI guess I need some combination of \"git filter-branch --tree-filter\"\nand \"git update-index --add\".\n\nIt doesn't matter much that the all commits are re-written, as I am\nstill the only one to have cloned them.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"74925","messageId":"Pine.LNX.4.64.0804220940320.22731@ds9.cixit.se","threadId":"13202","inReplyTo":"480D863B.1040009@viscovery.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Peter Karlsson","fromEmail":"peter@softwolves.pp.se","sentAt":"2008-04-22T08:42:19Z","receivedAt":"2008-04-22T08:42:19Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Johannes Sixt:\n\n> 2. You are still converting repositories? Convert the files in your\n> repository to LF.\n\nOr, perhaps, this is the way to go.\n\nGot to figure out how to get CRLF->LF conversion working without having\nRCS keyword expansion going haywire. I'm using RCS format as a\nmiddle-man between the old repositories (PVCS) I'm converting and\nparsecvs which imports them into Git. The old repository has expanded\nkeywords, and I must avoid having RCS/CVS expand them as they would\nexpand in a different manner... :-/\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"74928","messageId":"480DAA1A.10507@viscovery.net","threadId":"13202","inReplyTo":"Pine.LNX.4.64.0804220749080.22731@ds9.cixit.se","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-22T09:04:26Z","receivedAt":"2008-04-22T09:04:26Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Peter Karlsson schrieb:\n> Jakub Narebski:\n> \n>> I think you can, by unsetting `crlf` attribute, i.e. putting the\n>> following in .gitattributes:\n>>\n>> * -crlf\n> \n> Yeah, that does indeed seem to work, no matter how core.autocrlf is\n> configured globally. I think this is the best way to go for the\n> repositories I am working on (as they are very much DOS/Windows-only).\n> \n> Does anyone know how to hack an existing repository so that I can add\n> such a .gitattributes file to all commits? I've tried reading the\n> git-filter-branch manual page a few times, but I am still confused by\n> it.\n\nSomething like (untested, using bash):\n\nX=$(echo \"* -crlf\" | git hash-object -w --stdin)\ngit filter-branch \\\n\t--index-filter $'git-update-index --index-info <<< \\\n\t\t\t\t\"100644 $X\\t.gitattributes\"' \\\n\t-- --all\n\n-- Hannes\n"},{"id":"74979","messageId":"32541b130804220951p224c9be7ya4e8de5056481fd1@mail.gmail.com","threadId":"13202","inReplyTo":"20080422023918.GA5402@sigill.intra.peff.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-04-22T16:51:14Z","receivedAt":"2008-04-22T16:51:14Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 4/21/08, Jeff King wrote:\n> You could maybe do an index filter that gets the blob SHA1 of each file\n> that is new, and just munges those. But I think it is even simpler to\n> just keep a cache of original blob hashes mapping to munged blob hashes.\n> [...]\n\nThanks, this is really cool. I'll try it next time I'm messing with\nour repositories (this week is unfortunately a bit too busy).\n\nDo you think git would benefit from having a generalized version of\nthis script? Basically, the user provides a \"munge\" script on the\ncommand line, and there's a git-filter-branch mode for auto-munging\n(with a cache) every file in every checkin. Even if it's *only* ever\nused for CRLF, I can imagine this being useful to a lot of people.\n\nThanks,\n\nAvery\n"},{"id":"75037","messageId":"Pine.LNX.4.64.0804230806220.18401@ds9.cixit.se","threadId":"13202","inReplyTo":"32541b130804220951p224c9be7ya4e8de5056481fd1@mail.gmail.com","subject":"Re: Git on Windows, CRLF issues","fromName":"Peter Karlsson","fromEmail":"peter@softwolves.pp.se","sentAt":"2008-04-23T07:11:49Z","receivedAt":"2008-04-23T07:11:49Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Avery Pennarun:\n\n> Do you think git would benefit from having a generalized version of\n> this script?\n\nDefinitely. Also, something that would work with a) several branches\n(i.e traverse all the branches; keeping the points at which they\ndiverge), and b) submodules (i.e apply the same changes to the\nsubmodules and updating the submodule index accordingly).\n\nI ended up doing CRLF conversion for most of the repositories I had\nconverted. Fortunately, most of them had a single branch, so after\nhaving created a small script that did CRLF->LF for the text files, I\ncould do a\n\n git filter-branch --tree-filter 'c:/temp/crlf2lf.sh' \\\n --tag-name-filter 'cat' HEAD\n\non each repository and get everything converted during my lunch break.\n\n\nWhat I couldn't figure out is why, after converting everything,\nremoving all references to the repositories I cloned from, and removing\nreferences to the old objects in the reflogs, why\n\n git fsck --unreachable\n\ndid not report any unreachable objects? I would have guessed the entire\nold history and its objects would now be invalidated and could be\nkilled off.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"75041","messageId":"20080423080826.GA11935@sigill.intra.peff.net","threadId":"13202","inReplyTo":"32541b130804220951p224c9be7ya4e8de5056481fd1@mail.gmail.com","subject":"Re: Git on Windows, CRLF issues","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-04-23T08:08:26Z","receivedAt":"2008-04-23T08:08:26Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 22, 2008 at 12:51:14PM -0400, Avery Pennarun wrote:\n\n> Do you think git would benefit from having a generalized version of\n> this script? Basically, the user provides a \"munge\" script on the\n> command line, and there's a git-filter-branch mode for auto-munging\n> (with a cache) every file in every checkin. Even if it's *only* ever\n> used for CRLF, I can imagine this being useful to a lot of people.\n\nIt was easy enough to work up the patch below, which allows\n\n git filter-branch --blob-filter 'tr a-z A-Z'\n\nHowever, it's _still_ horribly slow. Shell script is nice and flexible,\nbut running a tight loop like this is just painful. I suspect\nfilter-branch in something like perl would be a lot faster and just as\nflexible (you could even do it in C, but you'd probably have to invent a\nlittle domain-specific scripting language).\n\nIt is still much better performance than a tree filter, though:\n\n $ cd git && time git filter-branch --tree-filter '\n find . -type f | while read f; do\n tr a-z A-Z <\"$f\" >tmp\n mv tmp \"$f\"\n done\n ' HEAD~10..HEAD\n\n real 4m38.626s\n user 1m32.726s\n sys 2m51.163s\n\n $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD\n real 1m40.809s\n user 0m36.822s\n sys 1m14.273s\n\nLots of system time in both. I'm sure we spend a fair bit of time\nhitting our very large map and blob-cache directories, which would be\nmuch more nicely implemented as associative arrays in memory (if we were\nusing a more featureful language).\n\nAnyway, here is the patch. I don't know if it is even worth applying,\nsince it is still painfully slow.\n\n---\n git-filter-branch.sh | 30 ++++++++++++++++++++++++++++++\n 1 files changed, 30 insertions(+), 0 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 333f6a8..0602b25 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -54,6 +54,23 @@ EOF\n \n eval \"$functions\"\n \n+munge_blobs() {\n+\twhile read mode sha1 stage path\n+\tdo\n+\t\tif ! test -r \"$workdir/../blob-cache/$sha1\"\n+\t\tthen\n+\t\t\tnew=`git cat-file blob $sha1 |\n+\t\t\t eval \"$filter_blob\" |\n+\t\t\t git hash-object -w --stdin`\n+\t\t\tprintf $new >$workdir/../blob-cache/$sha1\n+\t\tfi\n+\t\tprintf \"%s %s\\t%s\\n\" \\\n+\t\t\t\"$mode\" \\\n+\t\t\t$(cat \"$workdir/../blob-cache/$sha1\") \\\n+\t\t\t\"$path\"\n+\tdone\n+}\n+\n # When piped a commit, output a script to set the ident of either\n # \"author\" or \"committer\n \n@@ -105,6 +122,7 @@ tempdir=.git-rewrite\n filter_env=\n filter_tree=\n filter_index=\n+filter_blob=\n filter_parent=\n filter_msg=cat\n filter_commit='git commit-tree \"$@\"'\n@@ -150,6 +168,9 @@ do\n \t--index-filter)\n \t\tfilter_index=\"$OPTARG\"\n \t\t;;\n+\t--blob-filter)\n+\t\tfilter_blob=\"$OPTARG\"\n+\t\t;;\n \t--parent-filter)\n \t\tfilter_parent=\"$OPTARG\"\n \t\t;;\n@@ -227,6 +248,9 @@ ret=0\n # map old->new commit ids for rewriting parents\n mkdir ../map || die \"Could not create map/ directory\"\n \n+# cache rewritten blobs for blob filter\n+mkdir ../blob-cache || die \"Could not create blob-cache/ directory\"\n+\n case \"$filter_subdir\" in\n \"\")\n \tgit rev-list --reverse --topo-order --default HEAD \\\n@@ -295,6 +319,12 @@ while read commit parents; do\n \teval \"$filter_index\" < /dev/null ||\n \t\tdie \"index filter failed: $filter_index\"\n \n+\tif test -n \"$filter_blob\"; then\n+\t\tgit ls-files --stage |\n+\t\tmunge_blobs |\n+\t\tgit update-index --index-info\n+\tfi\n+\n \tparentstr=\n \tfor parent in $parents; do\n \t\tfor reparent in $(map \"$parent\"); do\n-- \n1.5.5.1.144.g4c416.dirty\n"},{"id":"75042","messageId":"20080423081025.GB11935@sigill.intra.peff.net","threadId":"13202","inReplyTo":"Pine.LNX.4.64.0804230806220.18401@ds9.cixit.se","subject":"Re: Git on Windows, CRLF issues","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-04-23T08:10:26Z","receivedAt":"2008-04-23T08:10:26Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 23, 2008 at 08:11:49AM +0100, Peter Karlsson wrote:\n\n> I ended up doing CRLF conversion for most of the repositories I had\n> converted. Fortunately, most of them had a single branch, so after\n> having created a small script that did CRLF->LF for the text files, I\n> could do a\n> \n> git filter-branch --tree-filter 'c:/temp/crlf2lf.sh' \\\n> --tag-name-filter 'cat' HEAD\n> \n> on each repository and get everything converted during my lunch break.\n\nSure, but that is quite slow on a larger tree, since it has to do a\nfull checkout for each commit. The idea of the specialized filter was to\navoid that. But if your project was small enough to do it that way, that\ncertainly works.\n\n> What I couldn't figure out is why, after converting everything,\n> removing all references to the repositories I cloned from, and removing\n> references to the old objects in the reflogs, why\n> \n> git fsck --unreachable\n> \n> did not report any unreachable objects? I would have guessed the entire\n> old history and its objects would now be invalidated and could be\n> killed off.\n\nDid you remove refs/original/ ?\n\n-Peff\n"},{"id":"75052","messageId":"alpine.DEB.1.00.0804231110080.4181@eeepc-johanness","threadId":"13202","inReplyTo":"20080423080826.GA11935@sigill.intra.peff.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-04-23T10:13:27Z","receivedAt":"2008-04-23T10:13:27Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Apr 2008, Jeff King wrote:\n\n> On Tue, Apr 22, 2008 at 12:51:14PM -0400, Avery Pennarun wrote:\n> \n> > Do you think git would benefit from having a generalized version of \n> > this script? Basically, the user provides a \"munge\" script on the \n> > command line, and there's a git-filter-branch mode for auto-munging \n> > (with a cache) every file in every checkin. Even if it's *only* ever \n> > used for CRLF, I can imagine this being useful to a lot of people.\n> \n> It was easy enough to work up the patch below, which allows\n> \n> git filter-branch --blob-filter 'tr a-z A-Z'\n> \n> However, it's _still_ horribly slow.\n\nYou create a quite huge blob-cache, so you are pretty heavy on disk-I/O. \nHave you tried (as suggested in the man page) to run this on a huge RAM \ndisk? That should blow you away.\n\n> Shell script is nice and flexible, but running a tight loop like this is \n> just painful. I suspect filter-branch in something like perl would be a \n> lot faster and just as flexible (you could even do it in C, but you'd \n> probably have to invent a little domain-specific scripting language).\n\nI hoped that the rewrite-commits attempt was more than just that: an \nattempt. So there is a point you could start from, doing things in C.\n\nBut I doubt that you get any joy: either your language is too limited, or \nyou will get the same problems (fork() overhead) again.\n\n> Anyway, here is the patch. I don't know if it is even worth applying, \n> since it is still painfully slow.\n\nI like your patch:\n\nAcked-by: Johannes Schindelin \n\nCiao,\nDscho\n"},{"id":"75053","messageId":"20080423105838.GA15310@sigill.intra.peff.net","threadId":"13202","inReplyTo":"alpine.DEB.1.00.0804231110080.4181@eeepc-johanness","subject":"Re: Git on Windows, CRLF issues","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-04-23T10:58:38Z","receivedAt":"2008-04-23T10:58:38Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 23, 2008 at 11:13:27AM +0100, Johannes Schindelin wrote:\n\n> > It was easy enough to work up the patch below, which allows\n> > \n> > git filter-branch --blob-filter 'tr a-z A-Z'\n> > \n> > However, it's _still_ horribly slow.\n> \n> You create a quite huge blob-cache, so you are pretty heavy on disk-I/O. \n> Have you tried (as suggested in the man page) to run this on a huge RAM \n> disk? That should blow you away.\n\nNo, I didn't. But the disk I/O is pretty minimal. The blob cache is only\na few megabytes, and it stays entirely in Linux's disk cache. My disk\nlight only blinks every 5-10 seconds to flush dirty pages to disk.\n\n> I hoped that the rewrite-commits attempt was more than just that: an \n> attempt. So there is a point you could start from, doing things in C.\n>\n> But I doubt that you get any joy: either your language is too limited, or \n> you will get the same problems (fork() overhead) again.\n\nAh, right. I totally forgot about that effort. I will take a peek next\ntime I need to do some filtering.\n\n> > Anyway, here is the patch. I don't know if it is even worth applying, \n> > since it is still painfully slow.\n> \n> I like your patch:\n> \n> Acked-by: Johannes Schindelin \n\nI think it could use some documentation updates. Avery, do you want to\ntry adding a CRLF example to the manpage?\n\n-Peff\n"},{"id":"75054","messageId":"480F1671.2060602@viscovery.net","threadId":"13202","inReplyTo":"20080423080826.GA11935@sigill.intra.peff.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-23T10:58:57Z","receivedAt":"2008-04-23T10:58:57Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Jeff King schrieb:\n> It was easy enough to work up the patch below, which allows\n> \n> git filter-branch --blob-filter 'tr a-z A-Z'\n...\n> +munge_blobs() {\n> +\twhile read mode sha1 stage path\n> +\tdo\n> +\t\tif ! test -r \"$workdir/../blob-cache/$sha1\"\n> +\t\tthen\n> +\t\t\tnew=`git cat-file blob $sha1 |\n> +\t\t\t eval \"$filter_blob\" |\n> +\t\t\t git hash-object -w --stdin`\n> +\t\t\tprintf $new >$workdir/../blob-cache/$sha1\n> +\t\tfi\n> +\t\tprintf \"%s %s\\t%s\\n\" \\\n> +\t\t\t\"$mode\" \\\n> +\t\t\t$(cat \"$workdir/../blob-cache/$sha1\") \\\n> +\t\t\t\"$path\"\n> +\tdone\n> +}\n\nIn practice, this is not sufficient. The blob filter must have an\nopportunity to decide what it wants to do, not just blindly munge every\nblob. The minimum is a path name, e.g. in $1:\n\n\tnew=$(git cat-file blob $sha1 |\n\t\t$SHELL_PATH -c \"$filter_blob\" ignored \"$path\" |\n\t\tgit hash-object -w --stdin)\n\n-- Hannes\n"},{"id":"75055","messageId":"20080423110402.GA27437@sigill.intra.peff.net","threadId":"13202","inReplyTo":"480F1671.2060602@viscovery.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-04-23T11:04:03Z","receivedAt":"2008-04-23T11:04:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 23, 2008 at 12:58:57PM +0200, Johannes Sixt wrote:\n\n> In practice, this is not sufficient. The blob filter must have an\n> opportunity to decide what it wants to do, not just blindly munge every\n> blob. The minimum is a path name, e.g. in $1:\n> \n> \tnew=$(git cat-file blob $sha1 |\n> \t\t$SHELL_PATH -c \"$filter_blob\" ignored \"$path\" |\n> \t\tgit hash-object -w --stdin)\n\nI intentionally left that out, because:\n\n - I assumed if you were going to do trickery with pathnames, you\n should just be doing an index filter\n\n - it violates the cache assumption, which is that blob $X is always\n transformed the same way\n\nI assume you are wanting to do something like:\n\n git filter-branch --blob-filter '\n case \"$1\" in\n *.jpg) cat ;;\n *) tr a-z A-Z ;;\n esac\n '\n\nObviously it is unlikely to get the same blob sha1 as \"foo.jpg\" and\n\"foo.txt\", but it just feels a little wrong.\n\n-Peff\n"},{"id":"75057","messageId":"480F218C.3060703@viscovery.net","threadId":"13202","inReplyTo":"20080423110402.GA27437@sigill.intra.peff.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-23T11:46:20Z","receivedAt":"2008-04-23T11:46:20Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Jeff King schrieb:\n> On Wed, Apr 23, 2008 at 12:58:57PM +0200, Johannes Sixt wrote:\n> \n>> In practice, this is not sufficient. The blob filter must have an\n>> opportunity to decide what it wants to do, not just blindly munge every\n>> blob. The minimum is a path name, e.g. in $1:\n>>\n>> \tnew=$(git cat-file blob $sha1 |\n>> \t\t$SHELL_PATH -c \"$filter_blob\" ignored \"$path\" |\n>> \t\tgit hash-object -w --stdin)\n> \n> I intentionally left that out, because:\n> \n> - I assumed if you were going to do trickery with pathnames, you\n> should just be doing an index filter\n> \n> - it violates the cache assumption, which is that blob $X is always\n> transformed the same way\n> \n> I assume you are wanting to do something like:\n> \n> git filter-branch --blob-filter '\n> case \"$1\" in\n> *.jpg) cat ;;\n> *) tr a-z A-Z ;;\n> esac\n> '\n> \n> Obviously it is unlikely to get the same blob sha1 as \"foo.jpg\" and\n> \"foo.txt\", but it just feels a little wrong.\n\nYes, that's how I intended it to work. What's wrong here? The fact that a\nuser might name a JPEG foo.txt instead of foo.jpg? Or that the same blob\nmight appear with entirely different names, including different suffixes?\nWell, tough luck. Use an index filter. But without any sort of hint what\nthe blob is about, your original --blob-filter is useless except for the\nmost simplistic repositories.\n\n-- Hannes\n"},{"id":"75063","messageId":"Pine.LNX.4.64.0804231445550.18401@ds9.cixit.se","threadId":"13202","inReplyTo":"20080423081025.GB11935@sigill.intra.peff.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Peter Karlsson","fromEmail":"peter@softwolves.pp.se","sentAt":"2008-04-23T13:47:34Z","receivedAt":"2008-04-23T13:47:34Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Jeff King:\n\n> Sure, but that is quite slow on a larger tree, since it has to do a\n> full checkout for each commit.\n\nIndeed. That's why I would welcome a script such as the one you\nmentioned :-) Fortunately, the repositories I worked on were small\nenough to not suffer too much (even when using Git on Windows, which is\na bit slower than on Linux).\n\n[Not seeing any unreachable objects]\n> Did you remove refs/original/ ?\n\nThat, and cloned the repository to a new location after the conversion,\nand removing the references to \"origin\" there. It does seem that the\nobjects are still there, but I can't see them with \"gitk --all\".\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"75066","messageId":"200804231624.09465.johan@herland.net","threadId":"13202","inReplyTo":"Pine.LNX.4.64.0804231445550.18401@ds9.cixit.se","subject":"Re: Git on Windows, CRLF issues","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-04-23T14:24:09Z","receivedAt":"2008-04-23T14:24:09Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Wednesday 23 April 2008, Peter Karlsson wrote:\n> Jeff King:\n> > Sure, but that is quite slow on a larger tree, since it has to do a\n> > full checkout for each commit.\n>\n> Indeed. That's why I would welcome a script such as the one you\n> mentioned :-) Fortunately, the repositories I worked on were small\n> enough to not suffer too much (even when using Git on Windows, which\n> is a bit slower than on Linux).\n>\n> [Not seeing any unreachable objects]\n>\n> > Did you remove refs/original/ ?\n>\n> That, and cloned the repository to a new location after the\n> conversion, and removing the references to \"origin\" there. It does\n> seem that the objects are still there, but I can't see them with\n> \"gitk --all\".\n\nMaybe they are kept alive by reflogs?\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"75068","messageId":"480F51CF.8080200@viscovery.net","threadId":"13202","inReplyTo":"Pine.LNX.4.64.0804231445550.18401@ds9.cixit.se","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-23T15:12:15Z","receivedAt":"2008-04-23T15:12:15Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Peter Karlsson schrieb:\n> [Not seeing any unreachable objects]\n> Jeff King:\n>> Did you remove refs/original/ ?\n> \n> That, and cloned the repository to a new location after the conversion,\n> and removing the references to \"origin\" there. It does seem that the\n> objects are still there, but I can't see them with \"gitk --all\".\n\nDid you clone locally? Then you must use the file:// protocol, otherwise\neverything is hard-linked from the origin.\n\n-- Hannes\n"},{"id":"75098","messageId":"32541b130804231302i732dbf08v88f8909f241ab99@mail.gmail.com","threadId":"13202","inReplyTo":"480F1671.2060602@viscovery.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-04-23T20:02:50Z","receivedAt":"2008-04-23T20:02:50Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 4/23/08, Johannes Sixt wrote:\n> In practice, this is not sufficient. The blob filter must have an\n> opportunity to decide what it wants to do, not just blindly munge every\n> blob. The minimum is a path name, e.g. in $1:\n\nActually, it may not have been intentional, but because of the way\n'eval' works, the munge script will find that $path already contains\nthe path of the file being munged. Works for me.\n\nHave fun,\n\nAvery\n"},{"id":"75107","messageId":"20080423214745.GA30057@sigill.intra.peff.net","threadId":"13202","inReplyTo":"480F218C.3060703@viscovery.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-04-23T21:47:45Z","receivedAt":"2008-04-23T21:47:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 23, 2008 at 01:46:20PM +0200, Johannes Sixt wrote:\n\n> > I assume you are wanting to do something like:\n> > \n> > git filter-branch --blob-filter '\n> > case \"$1\" in\n> > *.jpg) cat ;;\n> > *) tr a-z A-Z ;;\n> > esac\n> > '\n> > \n> > Obviously it is unlikely to get the same blob sha1 as \"foo.jpg\" and\n> > \"foo.txt\", but it just feels a little wrong.\n> \n> Yes, that's how I intended it to work. What's wrong here? The fact that a\n> user might name a JPEG foo.txt instead of foo.jpg? Or that the same blob\n> might appear with entirely different names, including different suffixes?\n> Well, tough luck. Use an index filter. But without any sort of hint what\n> the blob is about, your original --blob-filter is useless except for the\n> most simplistic repositories.\n\nYes, the script produces incorrect results if you have the same blob\nwith different names. IOW, if I accidentally add a JPEG as 'foo', and\nthen later rename it to 'foo.jpg', it will munge the blob the first time\nit sees it, and then use the munged value for 'foo.jpg', since we never\neven run the case statement. Yes, this is not terribly likely, but it\ndoes seem like an awful (and hard to diagnose!) bug to have hiding in\nthe script.\n\nThe correct fix is either:\n\n - the blob cache needs to take into account sha1 _and_ path\n\n - the cache lookup needs to be _inside_ the path filter. In that case\n you would either have to support it in the script (e.g.,\n --blob-ignore jpg), or you could make the caching an optional part\n of the blob filter (the way you can call 'map' explicitly from your\n filters).\n\n-Peff\n"},{"id":"75115","messageId":"7vprsgqiq1.fsf@gitster.siamese.dyndns.org","threadId":"13202","inReplyTo":"20080423214745.GA30057@sigill.intra.peff.net","subject":"Re: Git on Windows, CRLF issues","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-04-23T23:01:10Z","receivedAt":"2008-04-23T23:01:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> The correct fix is either:\n>\n> - the blob cache needs to take into account sha1 _and_ path\n>\n> - the cache lookup needs to be _inside_ the path filter. In that case\n> you would either have to support it in the script (e.g.,\n> --blob-ignore jpg), or you could make the caching an optional part\n> of the blob filter (the way you can call 'map' explicitly from your\n> filters).\n\nBut once you start saying \"even originally the same blob (i.e. identified\nby one object name) can be rewritten into different result, depending on\nwhere in the tree it appears\", would it make sense to have blob filters to\nbegin with?\n\nShouldn't that kind of of context sensitive (in the space dimension -- you\ncan introduce the context sensitivity in the time dimension by saying\nthere may even be cases where you would want to filter differently\ndepending on the path and which commit the blob appears, which is even\nworse) filtering be best left to the tree or index filter?\n"},{"id":"75116","messageId":"32541b130804231604r4c439fa5va5c01bfd53a22cc7@mail.gmail.com","threadId":"13202","inReplyTo":"7vprsgqiq1.fsf@gitster.siamese.dyndns.org","subject":"Re: Git on Windows, CRLF issues","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-04-23T23:04:48Z","receivedAt":"2008-04-23T23:04:48Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 4/23/08, Junio C Hamano wrote:\n> But once you start saying \"even originally the same blob (i.e. identified\n> by one object name) can be rewritten into different result, depending on\n> where in the tree it appears\", would it make sense to have blob filters to\n> begin with?\n>\n> Shouldn't that kind of of context sensitive (in the space dimension -- you\n> can introduce the context sensitivity in the time dimension by saying\n> there may even be cases where you would want to filter differently\n> depending on the path and which commit the blob appears, which is even\n> worse) filtering be best left to the tree or index filter?\n\nWhat I really want is the equivalent of \"dos2unix --recursive *.c\n*.txt etc\" for all commits.\n\nTheoretically, a .txt file might be renamed to a .jpg file, in which\ncase funny things would happen with such a filter, depending which\ncommit was seen first.\n\nI'm pretty confident that this will never happen to me, but it's a\nvalid concern.\n\nHave fun,\n\nAvery\n"},{"id":"75128","messageId":"20080424013722.GB31146@sigill.intra.peff.net","threadId":"13202","inReplyTo":"7vprsgqiq1.fsf@gitster.siamese.dyndns.org","subject":"Re: Git on Windows, CRLF issues","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-04-24T01:37:22Z","receivedAt":"2008-04-24T01:37:22Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 23, 2008 at 04:01:10PM -0700, Junio C Hamano wrote:\n\n> But once you start saying \"even originally the same blob (i.e. identified\n> by one object name) can be rewritten into different result, depending on\n> where in the tree it appears\", would it make sense to have blob filters to\n> begin with?\n> \n> Shouldn't that kind of of context sensitive (in the space dimension -- you\n> can introduce the context sensitivity in the time dimension by saying\n> there may even be cases where you would want to filter differently\n> depending on the path and which commit the blob appears, which is even\n> worse) filtering be best left to the tree or index filter?\n\nYes, that was my original reasoning. But I think the problem then is\nthat the blob filter isn't terribly useful. IOW, it is not really a\nseparate filter, but rather an optimizing pattern for an index filter,\nso maybe calling it a blob filter is the wrong approach, and it would be\nbetter as a short perl script in contrib/filter-branch. Then you could\ncall:\n\n git filter-branch --index-filter '\n /path/to/git/contrib/filter-branch/dos2unix \\\n \"*.txt\" \"*.c\"\n '\n\n-Peff\n"},{"id":"75140","messageId":"481027F4.5060905@viscovery.net","threadId":"13202","inReplyTo":"32541b130804231302i732dbf08v88f8909f241ab99@mail.gmail.com","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-04-24T06:25:56Z","receivedAt":"2008-04-24T06:25:56Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Avery Pennarun schrieb:\n> On 4/23/08, Johannes Sixt wrote:\n>> In practice, this is not sufficient. The blob filter must have an\n>> opportunity to decide what it wants to do, not just blindly munge every\n>> blob. The minimum is a path name, e.g. in $1:\n> \n> Actually, it may not have been intentional, but because of the way\n> 'eval' works, the munge script will find that $path already contains\n> the path of the file being munged. Works for me.\n\nYes, of course! So I stand corrected, and Jeff's patch makes sense.\n\nFor consistency's sake, the path should be made available in, say,\nGIT_BLOB_PATH just like the commit is available in GIT_COMMIT.\n\n-- Hannes\n"},{"id":"75143","messageId":"alpine.DEB.1.00.0804240909220.26553@eeepc-johanness","threadId":"13202","inReplyTo":"32541b130804231604r4c439fa5va5c01bfd53a22cc7@mail.gmail.com","subject":"Re: Git on Windows, CRLF issues","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-04-24T08:11:07Z","receivedAt":"2008-04-24T08:11:07Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 23 Apr 2008, Avery Pennarun wrote:\n\n> What I really want is the equivalent of \"dos2unix --recursive *.c *.txt \n> etc\" for all commits.\n\nI start to wonder if \"git fast-export --all | my-intelligent-perl-script | \ngit fast-import\" would not be a better solution here.\n\nAll you would have to do is to detect when a blob begins, and how long it \nis, and work with that. If your trees do not contain any binary files, it \nshould be trivial.\n\nCiao,\nDscho\n"},{"id":"75154","messageId":"32541b130804240956u2ec5817ey7b9652447489bc86@mail.gmail.com","threadId":"13202","inReplyTo":"alpine.DEB.1.00.0804240909220.26553@eeepc-johanness","subject":"Re: Git on Windows, CRLF issues","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-04-24T16:56:40Z","receivedAt":"2008-04-24T16:56:40Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 4/24/08, Johannes Schindelin wrote:\n> On Wed, 23 Apr 2008, Avery Pennarun wrote:\n> > What I really want is the equivalent of \"dos2unix --recursive *.c *.txt\n> > etc\" for all commits.\n>\n> I start to wonder if \"git fast-export --all | my-intelligent-perl-script |\n> git fast-import\" would not be a better solution here.\n>\n> All you would have to do is to detect when a blob begins, and how long it\n> is, and work with that. If your trees do not contain any binary files, it\n> should be trivial.\n\nErr, yes... as long as there are no binary files. I'm not so lucky,\nand life is a little more complex in that case. It also gives no easy\nway of selectively applying the blob filter based on filename, which\nis pretty important when you do have some binary files and you're\ntrying to decide whether to run dos2unix.\n\n(In contrast, the *other* objection, which is that the same blob might\nhave multiple filenames, doesn't bother me at all, since I'm sure I\ndon't have any .txt files that were accidentally named .jpg at some\npoint.)\n\nI agree that a working solution based on\ngit-fast-export/git-fast-import should run faster than any of the\nother proposed solutions, but my version of Jeff's patch is quite fast\nand it's easy to compose simple command lines that \"make simple things\nsimple and hard things possible\":\n\n git-filter-branch --blob-filter dos2unix HEAD\n git-filter-branch --blob-filter 'case \"$path\" in *.c) expand -8;; *)\ncat;; esac' HEAD\n\nIt sure beats writing a perl script every time you want to do something.\n\nJeff wrote:\n> But I think the problem then is\n> that the blob filter isn't terribly useful. IOW, it is not really a\n> separate filter, but rather an optimizing pattern for an index filter,\n> so maybe calling it a blob filter is the wrong approach\n\nThe problem is that doing an optimization on an index filter is kind\nof hard for a user to express, and each user will have to implement\nthe caching logic by hand every time. Using --index-filter at all\nrequires extremely high levels of shell and git knowledge.\n\nThe fact that the blob transformation might \"slightly depend on\" the\npath is not actually very important; fundamentally we're still\ntransforming blobs, not paths. We're just using the filename as a\n*hint* about what kind of transformation we need to do on that\nparticular blob.\n\nI think the measure of a good idea here is how straightforward it is\nto express what you want on the command line, and --blob-filter makes\nit easy to express a certain class of filters.\n\nHave fun,\n\nAvery\n"}]}