Re: Performance issue exposed by git-filter-branch

From: Elijah Newren <newren@gmail.com>
Date: Dec 17, 2010, 05:39 UTC
Message-ID: <AANLkTi=-0Sj9c5b778jchn+pgw26xCbioQ2K4tNgtm_G@mail.gmail.com>
In-Reply-To: <20101217030855.GB7003@burratino>

On Thu, Dec 16, 2010 at 8:08 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

Show 12 quoted lines

> Ken Brownfield wrote:
>
>> I had considered this approach (and the one mentioned by Jonathan)
>> but there are no git tools to actually perform the filter I wanted
>> on the export in this form.
>
> Keep in mind that the two suggestions were subtly different from one
> another.
>
> For the "filter fast-import stream" technique, apparently there is a
> tool called reposurgeon[1] to do that.  git_fast_filter[2] has the
> same purpose, too, if I remember correctly.

Yes, git_fast_filter was written precisely because git-filter-branch took waaaaaay too long. IIRC, git-filter-branch would have taken about 2-3 months for our use case (there's no way we could have shut down the repositories for that long), whereas git_fast_filter (serving along with fast-export and fast-import) allowed us to drop that to about an hour (we couldn't use --index-filter with filter-branch as we needed to do a number of operations on the actual file contents as well).

All git_fast_filter really does is parse the fast-export output into some basic python data structures, making it easy for you to modify those structures as necessary (assuming basic python skills, though if you only need to do what one of the examples shows then you could even get away without that), and then pipes the results back out in the format fast-import expects. It has a few examples with it; removing existing files is one of the simple examples.

I haven't really bothered keeping the public repository up-to-date since there hasn't been any prior external interest in it, but we haven't modified it much internally either, and most of those modifications are likely for niche stuff that you wouldn't need.

Elijah