RE: Best way to merge two repos with same content, differenthistory

From: Kelly F. Hickel <kfh@mqsoftware.com>
Date: Jun 5, 2009, 20:08 UTC
Message-ID: <63BEA5E623E09F4D92233FB12A9F794303117DD2@emailmn.mqsoftware.com>
In-Reply-To: <robbat2-20090605T194802-473902673Z@orbis-terrarum.net>

Show 18 quoted lines

> -----Original Message-----
> From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On
> Behalf Of Robin H. Johnson
> Sent: Friday, June 05, 2009 3:02 PM
> To: Git Mailing List
> Subject: Re: Best way to merge two repos with same content,
> differenthistory
> 
> On Fri, Jun 05, 2009 at 02:06:25PM -0500, Kelly F. Hickel wrote:
> > Robin,
> > 	That's all good news, I have an 8 way box with 32gb of ram
> running a
> > 64 bit Linux, a box with 4 gb of ram panics during the conversion.
> Thanks for your data.
> 
> For comparison, our conversion box is also 8-way, but only 16GiB RAM.
> 
> I'm surprised at how long pass1 is for you, especially since you've

got

Show 11 quoted lines

> a lot less CVS Files and CVS Revisions than the Gentoo repo (I do
> deduce
> that your individual revisions are larger, averaging at 15KiB vs. our
> 711 bytes).
> 
> I think there's something odd in the total CVS branches/tags count
> however, as the counts there imply an average of 67 branches and 173
> tags per CVS revision. You might want to dig into that part manually
> and
> see about it (not sure of your Python skills). That would probably cut
> down both your pass1 and pass4 times significantly.

Robin, I'm not much with python, so haven't dug into the code much at all. The numbers are high, although we do create a lot of branches (had to contribute a fix a year or two to CVS to get the branching time down from the 2.5 hours it was taking). At one point I carefully examined the symbol file that cvs2git was outputting and convinced myself that it was doing the right thing, but that was awhile ago.

Show 14 quoted lines

> 
> Hopefully mhagger will get the external blob stuff committed soon, I
> was
> working on validating it's results.
> 
> In doing so discovered a testcase where RCSRevisionReader and
> CVSRevisionReader gave different output themselves, the latter (which
> is
> documented as more accurate otherwise) missing the contents of an
> entire
> file. It's on the cvs2svn-dev mailing list now. Tracing that first,
> thereafter comparing it to the new Git side.
> 
> > git repack -a -d -f --depth=4000 --window=4000 && git pack-refs

--all

> Did those extreme depth/window values actually help size much? The
> Gentoo ones actually didn't improve significantly over

depth=window=50.

I know that they were still (apparently) improving after the 200 mark, it took long enough at 200 that I just decided to crank the numbers way up and let it run over the weekend.

Show 6 quoted lines

> 
> --
> Robin Hugh Johnson
> Gentoo Linux Developer & Infra Guy
> E-Mail     : robbat2@gentoo.org
> GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

I'll be looking forward to a newer faster cvs2git, although I did just get the graft idea working, so not sure if we'll wait that long or not (would be nice not to have to muck around with it though).

Thanks, Kelly