Re: Performance issue exposed by git-filter-branch

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Dec 17, 2010, 03:37 UTC
Message-ID: <20101217033715.GA7302@burratino>
In-Reply-To: <20101217032232.GC7003@burratino>

Jonathan Nieder wrote:

> Ken Brownfield wrote:

>> The thread titled "git and larger trees, not so fast?".
>
> Here it is[1].  Sorry to say, the improvements discussed there
> were made right away and indeed had a dramatic effect.

Of course I missed your point. :)

filter-branch --index-filter works a little like this: for each commit:

. find the underlying tree
. read-tree: unpack that tree and all of its subtrees into
the index file.  That is, convert from a recursive structure
   /:
	COPYING
	Documentation/
	INSTALL
	Makefile
	...

   Documentation/:
	CodingGuidelines
	Makefile
	...

into a flat structure

	COPYING
	Documentation/CodingGuideLines
	Documentation/Makefile
	Documentation/RelNotes/1.5.0.txt
	...
. rm: find entries matching certain patterns and remove them
from the index file.  This takes two passes through the index:
first to find matching entries, second to write the result to
disk.
. write-tree: write new trees for the object store.  That is,
convert from a flat structure back to a recursive structure.

This is convenient, but it does not sound to me like the most efficient way to eliminate a few subtrees from each commit. That is why I was suggesting a method that avoids unpacking some trees altogether.

That said, speedups for read-tree, rm, and write-tree would certainly be nice to have. One project of interest to some people is to give the index file a recursive structure, so finding the entries to remove in the "git rm" example could be faster.