Re: Importing Mozilla CVS into git

From: Jon Smirl <jonsmirl@gmail.com>
Date: Jun 6, 2006, 15:13 UTC
Message-ID: <9e4733910606060813r41037467u74235f7a9386c1e0@mail.gmail.com>
In-Reply-To: <46a038f90606052255s62cda81bt62d7442beb26658a@mail.gmail.com>

On 6/6/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:

Show 13 quoted lines

> On 6/3/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > On 6/1/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > > With the attached patch you can parse the entire Mozilla tree. The
> > > tree has over 100,000 files in it and about 300 branches.
> >
> > I was a little low with these counts, more like 110,000 files and some
> > parts of the tree have 1,000 branches. Total tree size is 3GB.
>
> I don't think it really has that many branches. If I am to believe
> cvsps (which took 3GB to walk the history), it has some branches with
> recursive loops in their ancestry (MANG_MATH_BRANCH and
> SpiderMonkey140_BRANCH have eachother as ancestors!?), 197969 commits
> and 796 branches.

It probably is 796 and not a 1,000. The branch names were scrolling across my screen and I just estimated.

> This repository has been mangled quite badly. Don't know what you guys
> did with it, but it sure isn't pretty. I'm working on getting
> git-cvsimport to get through a complete import.

The repository is close to 10 years old and it has gone through a number of corporate reorgs. Who knows what has happened to it over that length of time.

Have you looked at the SVN CVS import tool? It imported Mozilla on the first try. If you download the source they have built about 40 test repositories with various errors. Those would make a good test suite for cvsps. http://cvs2svn.tigris.org

I have been working on converting the svn tool to do git commands but my git knowledge is limited so it has been slow going. The last stage, pass 8, is very similar to what the git tools do. The svn commands just need to be swapped for git ones.

If you get git-cvsimport working I'll use it instead. Will the cvsps process stay small enough to run on a 32b machine? The svn tools are very RAM efficient since they use an external db. Can cvsps read from a local copy of the repository without using a CVS server?

We are going to have to develop some kind of incremental mechanism for updating the new git tree. It can take up to two days to convert the repository, Mozilla development can't be shut down that long for a transition. Git will also need to mirror the CVS repository (check-in still going to CVS) for a long time while we convince everyone on the merits of switching.

My imported svn version of Mozilla has a lot of performance problems. One of the directories has over 200,000 files in it slowing downing the filesystem. The repository went from 3GB CVS to 8GB svn, probably due to svn using 1000s of tiny files. I'll look around and see if svn has a pack feature like git.

-- 
Jon Smirl
jonsmirl@gmail.com