git-svn with big subversion repository

How do you recommend using git to work with branches of a large, busy subversion repository? In general, how can small teams use git for their tasks, and use subversion to coordinate with a larger organization?

git-svn has some trouble, I find. For example, this tries to copy the entire repo starting with revision 1:

git svn clone --stdlayout svn+ssh://server/repo/project

This would take weeks, I estimate for my subversion repository.

Choosing a subset of the repository enables git svn clone to cope, but then git svn fetch will stall after processing a few revisions. For example:

git svn clone --no-follow-parent --no-minimize-url \
 --branches=branches \
 --ignore-paths="^(?!branches/(TEAM_|RELEASE_))" \
 -r $BASE svn+ssh://server/repo/project
git svn fetch --no-follow-parent # stalls

I don't why it stalls. I guess it's doing something that requires processing the entire subversion repository.

The best I can do is clone each subversion branch into a separate svn-remote section of the .git/config file, for example:

git svn clone --no-follow-parent --no-minimize-url \
 --svn-remote=TEAM_FOO --id=TEAM_FOO \
 -r $BASE svn+ssh://server/repo/project/branches/TEAM_FOO
git svn fetch --no-follow-parent

The clone runs about as long as svn checkout, and the fetch replays the later revisions briskly. Sadly, the relationship between branches isn't fetched: git log won't tell me how a given subversion branch was copied from another. I use svn for that.

I'm using git version 1.7.4, git-svn version 1.7.4 (svn 1.6.5), svn version 1.6.0 (r36650) and Mac OS X version 10.6.5. I got git from MacPorts.

- John Kristian

Hi John,

I've successfully run git svn clone on a repository with about 100k revisions. The clone was not of the whole repository, but rather a subdirectory for a project using the trunk/tags/branches structure. The project is about 200k files and about 4GB.

The initial clone took hours and hours (on my macbook). I basically had to leave it on over night (the svn server is here on the LAN, running over https).

The only problem I had was that the clone would occasionally exit (not stall, as you say). This is a know problem described here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=526989

The solution is to just run git svn fetch, as the cloning will pick up where it stopped. To keep from having to do this yourself, loop the fetch in a shell script. I blogged about it here: <http://blog.tfnico.com/2010/07/living-with-subversion-and-git-in.html>

And there are also some more tricks and tips for living with git+svn here: <http://www.tfnico.com/presentations/git-and-subversion>

You could also investigate how the Apache folks have made their git mirrors here: http://git.apache.org/ - at least they have an SVN repo with over a million revisions. I think they did something like svn-dump + git fast-import, but I couldn't find any details on the fly.

On Wed, Mar 2, 2011 at 3:43 AM, John Kristian <jkristian@linkedin.com> wrote:

On 03/01/2011 09:43 PM, John Kristian wrote:

My initial git-svn clone took several days and many restarts. It was much faster on my laptop. I found out later I had a flaky router and it was dropping about 20% of my packets. Replaced the router and the clone dropped to a reasonable couple-of-hours. Is it just me?

You can optimize by cloning specific paths inside the svn repo and then merging in git later.

Phil

* John Kristian:

I've used svnsync to a local repository and git-svn against that. This meant that my experiments do not cause excessive load on the server.

You should definitely coordinate this because this could be considered leaching IP if the repository is not public.

On Wed, 2 Mar 2011 02:43:23 +0000 John Kristian <jkristian@linkedin.com> wrote:

I don't know if this is the same problem that you are having, but...

I had to clone a repository with 200k revisions and 12000 branches+tags. It was going to take weeks and weeks with a local svnsync mirror on a high-end workstation with fast disks.

I've never touched perl-code previous to this, but a friend pointed me at a good perl profiler, and I found pretty quickly the offending line of code in git-svn:

3515         return unless ::verify_ref($self->refname.'^0');

This was basically doing a
  system("git rev-parse --verify some-reference^0")

several times per revision fetched per branch. When you have 12000 branches, that really, really adds up. I made a change that seems to speed it up by a factor of about 10-20x on my repository, but I'm still digging around in git to see if I'm doing it correctly.

My basic logic is that if the above one-liner returns true then either
one of the following files will exist:
    $ENV{GIT_DIR}/refs/remotes/$refname
    $ENV{GIT_DIR}/refs/$refname
    $ENV{GIT_DIR}/refs/heads/$refname
or there will be an entry for the reference in
    $ENV{GIT_DIR}/packed-refs

Furthermore since packed-refs changes infrequently, you can cache its contents.

I'm still digging around in the plumbing to see if this is assumption is true or not. If I find it is true, I'll likely submit a patch. Now that it "works on my machine" I've backburnered it a bit since git is more a tool I use than a project I hack on.

-Jason

On 18:14 Wed 09 Mar , John Kristian wrote: Mr Kristian:

Indeed I was, it took about 48 hours to to the initial import. However, I forgot to mention one other important thing that was a problem.

There is a pattern in svn of doing the following:

/trunk/module1 /trunk/module2 /trunk/module3

Then some branches will be like this:
	svn cp /trunk/ /branches/mybranch1
and others might be:
	svn cp /trunk/module2 /brancyes/mybranchofmodule2

If this hasn't ever been done on your repository, you can stop reading now.

There is no way to represent this in Git directly, so the correct thing to do here is to create a git repository for each module. Now the hard thing is telling git-svn how to handle this. I ended up writing a python script that reads in the SVN changelog and finds all of the children of e.g. /trunk/module1. Any that were copied from /trunk, it appends "/module1" to the path, and any that were copied from /trunk/module1, it leaves alone. This then goes in the git configuration file as the list of branches to fetch.

-Jason