Re: cvs2svn conversion directly to git ready for experimentation

From: Jon Smirl <jonsmirl@gmail.com>
Date: Aug 3, 2007, 20:27 UTC
Message-ID: <9e4733910708031327u7df2205ap56a7ad5430380fb@mail.gmail.com>
In-Reply-To: <9e4733910708031316x1b7d2a40n5d0298cedd6cf97c@mail.gmail.com>

On 8/3/07, Jon Smirl <jonsmirl@gmail.com> wrote:

> Make a bulk importer for SVN like git-fastimport. I measured some SVN
> imports and the bulk of the time was spent forking off SVN. Before
> git-fast import it would have taken git two weeks to import Mozilla
> CVS.

And add a CVS parser to cvs2svn. Use the one I posted or write it again. Fork is not a very fast operation, millions of forks take a week to run.

In the cvs2git code I did there was one process running cvs2svn and it parsed the CVS files internally. A second process ran git-fastimport. Nothing else was forked.

When I first started we were forking both git and cvs. When I ran oprofile on it 95% of the CPU time was being spent in the kernel. Linus helped me figure out what was going on. It was the overhead of page table copies associated with millions of forks that was taking so long. The solution is to eliminate the forks.

My first try with forks for both cvs and git took about a week to import Mozilla CVS. After all the forks were eliminated I could import Mozilla CVS in four hours.

Show 11 quoted lines

>
> >
> > Michael
> >
> >
>
>
> --
> Jon Smirl
> jonsmirl@gmail.com
>

-- 
Jon Smirl
jonsmirl@gmail.com