RE: Best way to merge two repos with same content, differenthistory

From: Kelly F. Hickel <kfh@mqsoftware.com>
Date: Jun 5, 2009, 19:06 UTC
Message-ID: <63BEA5E623E09F4D92233FB12A9F794303117DCB@emailmn.mqsoftware.com>
In-Reply-To: <robbat2-20090605T183716-227340397Z@orbis-terrarum.net>

Show 17 quoted lines

> -----Original Message-----
> From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On
> Behalf Of Robin H. Johnson
> Sent: Friday, June 05, 2009 1:46 PM
> To: Git Mailing List
> Subject: Re: Best way to merge two repos with same content,
> differenthistory
> 
> On Fri, Jun 05, 2009 at 12:10:30PM -0500, Kelly F. Hickel wrote:
> > Bost,
> > 	Thanks, but I'm already working with a local copy of the CVS
> repo.
> > 	I've corresponded with Michael Haggerty about the time this
> takes,
> > 	and there just doesn't seem to be any way to improve the speed,
> > 	without making some fairly drastic changes to cvs2git.
> I've been working with mhagger lately as it also pertains to the

Gentoo

> conversion. We've made some very good progress.
> 
> A couple of comments in that regard:
> - Make really sure your box is not short of RAM. Throw some

measurement

Show 6 quoted lines

>   tools onto there to see it. A couple of GiB is worthwhile. After we
>   found this early on, and switched boxes, we dropped from our initial
>   multiple days to 20 hours.
> - His latest ExternalBlobGenerator code (_NOT_ available in SVN yet)
>   reduced our pass1 time from 36204 seconds to 1598 seconds, with
>   a potential to be much faster now, as parallelization of part of

that

>   is now trivial.
> - pass9 is still the remaining large time-eater for us. I've started

to

Show 11 quoted lines

>   look at it, but I haven't made any actual developments yet.
> 
> Would you mind posting your cvs2svn stats like these?
> http://archives.gentoo.org/gentoo-
> scm/msg_b69b2f6ecee0ec7bb402d31b372b945b.xml
> 
> --
> Robin Hugh Johnson
> Gentoo Linux Developer & Infra Guy
> E-Mail     : robbat2@gentoo.org
> GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Robin, 
	That's all good news, I have an 8 way box with 32gb of ram
running a 64 bit Linux, a box with 4 gb of ram panics during the
conversion.

My conversion data is below...

Thanks, Kelly

cvs2svn Statistics:
------------------
Total CVS Files:             18488
Total CVS Revisions:        225208
Total CVS Branches:       15203751
Total CVS Tags:           39079236
Total Unique Tags:           11453
Total Unique Branches:        4364
CVS Repos Size in KB:      3355895
Total SVN Commits:           49967
First Revision Date:    Mon Nov  8 02:26:51 1999
Last Revision Date:     Wed Apr 22 17:59:44 2009
------------------
Timings (seconds):
------------------
251546   pass1    CollectRevsPass
     4   pass2    CleanMetadataPass
   142   pass3    CollateSymbolsPass
 53491   pass4    FilterSymbolsPass
     4   pass5    SortRevisionSummaryPass
   163   pass6    SortSymbolSummaryPass
  8825   pass7    InitializeChangesetsPass
   418   pass8    BreakRevisionChangesetCyclesPass
   418   pass9    RevisionTopologicalSortPass
  4256   pass10   BreakSymbolChangesetCyclesPass
  4914   pass11   BreakAllChangesetCyclesPass
  4575   pass12   TopologicalSortPass
  3111   pass13   CreateRevsPass
   270   pass14   SortSymbolsPass
   154   pass15   IndexSymbolsPass
  5517   pass16   OutputPass
337808   total
251783.89user 80800.42system 93:50:11elapsed 98%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (3major+3132264023minor)pagefaults 0swaps

git-fast-import statistics: 
---------------------------------------------------------------------
Alloc'd objects:     415000
Total objects:       410079 (   2628078 duplicates                  )
      blobs  :       152002 (     68715 duplicates     135125 deltas)
      trees  :       213636 (   2559363 duplicates     164052 deltas)
      commits:        44441 (         0 duplicates          0 deltas)
      tags   :            0 (         0 duplicates          0 deltas)
Total branches:       15822 (      6184 loads     )
      marks:     1073741824 (    265158 unique    )
      atoms:          14807
Memory total:         32402 KiB
       pools:         16192 KiB
     objects:         16210 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 8589934592
pack_report: pack_used_ctr            =     311525
pack_report: pack_mmap_calls          =      13303
pack_report: pack_open_windows        =          1 /          1
pack_report: pack_mapped              =  403041230 /  403041230
---------------------------------------------------------------------

git repack -a -d -f --depth=4000 --window=4000 && git pack-refs --all Counting objects: 409458, done.5/119582) Compressing objects: 100% (119582/119582), done. Writing objects: 100% (128713/128713), done. Total 128713 (delta 82330), reused 0 (delta 0) Compressing objects: 100% (384171/384171), done. Writing objects: 100% (409458/409458), done. Total 409458 (delta 309214), reused 0 (delta 0)