Re: git-svn with big subversion repository

From: Jason Miller <jason@milr.com>
Date: Mar 9, 2011, 05:53 UTC
Message-ID: <20110308215342.35b8aac1@naru.jasonmmiller.org>
In-Reply-To: <C992EE5B.CBFB%jkristian@linkedin.com>

On Wed, 2 Mar 2011 02:43:23 +0000 John Kristian <jkristian@linkedin.com> wrote:

> How do you recommend using git to work with branches of a large, busy
> subversion repository? In general, how can small teams use git for
> their tasks, and use subversion to coordinate with a larger
> organization?

I don't know if this is the same problem that you are having, but...

I had to clone a repository with 200k revisions and 12000 branches+tags. It was going to take weeks and weeks with a local svnsync mirror on a high-end workstation with fast disks.

I've never touched perl-code previous to this, but a friend pointed me at a good perl profiler, and I found pretty quickly the offending line of code in git-svn:

3515         return unless ::verify_ref($self->refname.'^0');

This was basically doing a
  system("git rev-parse --verify some-reference^0")

several times per revision fetched per branch. When you have 12000 branches, that really, really adds up. I made a change that seems to speed it up by a factor of about 10-20x on my repository, but I'm still digging around in git to see if I'm doing it correctly.

My basic logic is that if the above one-liner returns true then either
one of the following files will exist:
    $ENV{GIT_DIR}/refs/remotes/$refname
    $ENV{GIT_DIR}/refs/$refname
    $ENV{GIT_DIR}/refs/heads/$refname
or there will be an entry for the reference in
    $ENV{GIT_DIR}/packed-refs

Furthermore since packed-refs changes infrequently, you can cache its contents.

I'm still digging around in the plumbing to see if this is assumption is true or not. If I find it is true, I'll likely submit a patch. Now that it "works on my machine" I've backburnered it a bit since git is more a tool I use than a project I hack on.

-Jason