Avery Pennarun[PATCH v2 1/3] filter-branch: add new --blob-filter option.
From: Jeff King <peff@peff.net>
On Tue, Apr 22, 2008 at 12:51:14PM -0400, Avery Pennarun wrote:
Show 5 quoted lines
> Do you think git would benefit from having a generalized version of > this script? Basically, the user provides a "munge" script on the > command line, and there's a git-filter-branch mode for auto-munging > (with a cache) every file in every checkin. Even if it's *only* ever > used for CRLF, I can imagine this being useful to a lot of people.
It was easy enough to work up the patch below, which allows
git filter-branch --blob-filter 'tr a-z A-Z'
However, it's _still_ horribly slow. Shell script is nice and flexible, but running a tight loop like this is just painful. I suspect filter-branch in something like perl would be a lot faster and just as flexible (you could even do it in C, but you'd probably have to invent a little domain-specific scripting language).
It is still much better performance than a tree filter, though:
$ cd git && time git filter-branch --tree-filter '
find . -type f | while read f; do
tr a-z A-Z <"$f" >tmp
mv tmp "$f"
done
' HEAD~10..HEADreal 4m38.626s user 1m32.726s sys 2m51.163s
$ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD real 1m40.809s user 0m36.822s sys 1m14.273s
Lots of system time in both. I'm sure we spend a fair bit of time hitting our very large map and blob-cache directories, which would be much more nicely implemented as associative arrays in memory (if we were using a more featureful language).
Anyway, here is the patch. I don't know if it is even worth applying, since it is still painfully slow.
Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de> --- git-filter-branch.sh | 30 ++++++++++++++++++++++++++++++ 1 files changed, 30 insertions(+), 0 deletions(-)
Show changes to git-filter-branch.sh +30 −0
diff --git a/git-filter-branch.sh b/git-filter-branch.sh index d04c346..a0d9a79 100755 --- a/git-filter-branch.sh +++ b/git-filter-branch.sh @@ -54,6 +54,23 @@ EOF eval "$functions" +munge_blobs() { + while read mode sha1 stage path + do + if ! test -r "$workdir/../blob-cache/$sha1" + then + new=`git cat-file blob $sha1 | + eval "$filter_blob" | + git hash-object -w --stdin` + printf $new >$workdir/../blob-cache/$sha1 + fi + printf "%s %s\t%s\n" \ + "$mode" \ + $(cat "$workdir/../blob-cache/$sha1") \ + "$path" + done +} + # When piped a commit, output a script to set the ident of either # "author" or "committer @@ -105,6 +122,7 @@ tempdir=.git-rewrite filter_env= filter_tree= filter_index= +filter_blob= filter_parent= filter_msg=cat filter_commit='git commit-tree "$@"' @@ -150,6 +168,9 @@ do --index-filter) filter_index="$OPTARG" ;; + --blob-filter) + filter_blob="$OPTARG" + ;; --parent-filter) filter_parent="$OPTARG" ;; @@ -227,6 +248,9 @@ ret=0 # map old->new commit ids for rewriting parents mkdir ../map || die "Could not create map/ directory" +# cache rewritten blobs for blob filter +mkdir ../blob-cache || die "Could not create blob-cache/ directory" + case "$filter_subdir" in "") git rev-list --reverse --topo-order --default HEAD \ @@ -295,6 +319,12 @@ while read commit parents; do eval "$filter_index" < /dev/null || die "index filter failed: $filter_index" + if test -n "$filter_blob"; then + git ls-files --stage | + munge_blobs | + git update-index --index-info + fi + parentstr= for parent in $parents; do for reparent in $(map "$parent"); do
-- 1.5.6.rc2.29.g4717e
