Best way to merge two repos with same content, different history

Hi all,
	We're converting out of CVS after 10 years... The cvs2git
conversion takes around 4-5 days, and there doesn't seem to be any way
to speed that up.  So, our current plan is to take the tip of the
branches that we need for the next week, and import each of those
branches into its own clean git repo (top-skim).  Then we'll work out of
those repos while the conversion is in process (presumably creating
branches in those repos as needed).  Once the conversion is finished,
we'll need to get all the work done in the top-skimmed repos merged into
the converted repo so that we end up with little to no developer down
time, and all of our history, pre and post conversion in one repo.
	I'm testing all of this in advance, of course, and the tricky
part at the moment is how to "stitch" the commits from the top-skim
repos back onto the converted repo when the conversion is done.  The
file content of the initial commit for the skimmed repo is identical to
the last commit for the respective branch in the converted repo, but the
SHA1s are different, presumably because the history of the content is
different.

	Stated another way, I have two repositories, "new" and "old",
where the files in the initial commit on branch "B1" in "new" have
exactly the same content as the last commit on branch "B1" in "old".
There also exist various branches in "new" based on "B1".  I'd like to
merge all the commits from "new" into "old", but the SHA1s are
different, presumably because the history leading up to those points are
different.

	Other than using manually format-patch on every branch in new,
then applying the patches (presumably with regular old patch, since the
ancestor commit IDs won't match), is there any "good" way to merge "new"
into "old"?

Thanks,	

--

Kelly F. Hickel Senior Product Architect MQSoftware, Inc. 952-345-8677 Office 952-345-8721 Fax kfh@mqsoftware.com www.mqsoftware.com Certified IBM SOA Specialty Your Full Service Provider for IBM WebSphere Learn more at www.mqsoftware.com

On Fri, Jun 5, 2009 at 18:30, Kelly F. Hickel<kfh@mqsoftware.com> wrote:

try google: "cvs2git migration - cloning CVS repository"

Bost

Bost, 
	Thanks, but I'm already working with a local copy of the CVS repo.  I've corresponded with Michael Haggerty about the time this takes, and there just doesn't seem to be any way to improve the speed, without making some fairly drastic changes to cvs2git.

Thanks, Kelly

On Fri, Jun 5, 2009 at 19:10, Kelly F. Hickel<kfh@mqsoftware.com> wrote:

uhm, see below

Bost

On Tue, Feb 3, 2009 at 14:55, Johannes Schindelin<Johannes.Schindelin@gmx.de> wrote:

On Fri, Jun 05, 2009 at 12:10:30PM -0500, Kelly F. Hickel wrote:

I've been working with mhagger lately as it also pertains to the Gentoo conversion. We've made some very good progress.

A couple of comments in that regard:
- Make really sure your box is not short of RAM. Throw some measurement
  tools onto there to see it. A couple of GiB is worthwhile. After we
  found this early on, and switched boxes, we dropped from our initial
  multiple days to 20 hours.
- His latest ExternalBlobGenerator code (_NOT_ available in SVN yet)
  reduced our pass1 time from 36204 seconds to 1598 seconds, with
  a potential to be much faster now, as parallelization of part of that
  is now trivial.
- pass9 is still the remaining large time-eater for us. I've started to
  look at it, but I haven't made any actual developments yet.

Would you mind posting your cvs2svn stats like these? http://archives.gentoo.org/gentoo-scm/msg_b69b2f6ecee0ec7bb402d31b372b945b.xml

Gentoo

measurement

that

to

Robin, 
	That's all good news, I have an 8 way box with 32gb of ram
running a 64 bit Linux, a box with 4 gb of ram panics during the
conversion.

My conversion data is below...

Thanks, Kelly

cvs2svn Statistics:
------------------
Total CVS Files:             18488
Total CVS Revisions:        225208
Total CVS Branches:       15203751
Total CVS Tags:           39079236
Total Unique Tags:           11453
Total Unique Branches:        4364
CVS Repos Size in KB:      3355895
Total SVN Commits:           49967
First Revision Date:    Mon Nov  8 02:26:51 1999
Last Revision Date:     Wed Apr 22 17:59:44 2009
------------------
Timings (seconds):
------------------
251546   pass1    CollectRevsPass
     4   pass2    CleanMetadataPass
   142   pass3    CollateSymbolsPass
 53491   pass4    FilterSymbolsPass
     4   pass5    SortRevisionSummaryPass
   163   pass6    SortSymbolSummaryPass
  8825   pass7    InitializeChangesetsPass
   418   pass8    BreakRevisionChangesetCyclesPass
   418   pass9    RevisionTopologicalSortPass
  4256   pass10   BreakSymbolChangesetCyclesPass
  4914   pass11   BreakAllChangesetCyclesPass
  4575   pass12   TopologicalSortPass
  3111   pass13   CreateRevsPass
   270   pass14   SortSymbolsPass
   154   pass15   IndexSymbolsPass
  5517   pass16   OutputPass
337808   total
251783.89user 80800.42system 93:50:11elapsed 98%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (3major+3132264023minor)pagefaults 0swaps

git-fast-import statistics: 
---------------------------------------------------------------------
Alloc'd objects:     415000
Total objects:       410079 (   2628078 duplicates                  )
      blobs  :       152002 (     68715 duplicates     135125 deltas)
      trees  :       213636 (   2559363 duplicates     164052 deltas)
      commits:        44441 (         0 duplicates          0 deltas)
      tags   :            0 (         0 duplicates          0 deltas)
Total branches:       15822 (      6184 loads     )
      marks:     1073741824 (    265158 unique    )
      atoms:          14807
Memory total:         32402 KiB
       pools:         16192 KiB
     objects:         16210 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 8589934592
pack_report: pack_used_ctr            =     311525
pack_report: pack_mmap_calls          =      13303
pack_report: pack_open_windows        =          1 /          1
pack_report: pack_mapped              =  403041230 /  403041230
---------------------------------------------------------------------

git repack -a -d -f --depth=4000 --window=4000 && git pack-refs --all Counting objects: 409458, done.5/119582) Compressing objects: 100% (119582/119582), done. Writing objects: 100% (128713/128713), done. Total 128713 (delta 82330), reused 0 (delta 0) Compressing objects: 100% (384171/384171), done. Writing objects: 100% (409458/409458), done. Total 409458 (delta 309214), reused 0 (delta 0)

On Fri, Jun 05, 2009 at 02:06:25PM -0500, Kelly F. Hickel wrote:

Thanks for your data.

For comparison, our conversion box is also 8-way, but only 16GiB RAM.

I'm surprised at how long pass1 is for you, especially since you've got a lot less CVS Files and CVS Revisions than the Gentoo repo (I do deduce that your individual revisions are larger, averaging at 15KiB vs. our 711 bytes).

I think there's something odd in the total CVS branches/tags count however, as the counts there imply an average of 67 branches and 173 tags per CVS revision. You might want to dig into that part manually and see about it (not sure of your Python skills). That would probably cut down both your pass1 and pass4 times significantly.

Hopefully mhagger will get the external blob stuff committed soon, I was working on validating it's results.

In doing so discovered a testcase where RCSRevisionReader and CVSRevisionReader gave different output themselves, the latter (which is documented as more accurate otherwise) missing the contents of an entire file. It's on the cvs2svn-dev mailing list now. Tracing that first, thereafter comparing it to the new Git side.

Did those extreme depth/window values actually help size much? The Gentoo ones actually didn't improve significantly over depth=window=50.

got

Robin, I'm not much with python, so haven't dug into the code much at all. The numbers are high, although we do create a lot of branches (had to contribute a fix a year or two to CVS to get the branching time down from the 2.5 hours it was taking). At one point I carefully examined the symbol file that cvs2git was outputting and convinced myself that it was doing the right thing, but that was awhile ago.

--all

depth=window=50.

I know that they were still (apparently) improving after the 200 mark, it took long enough at 200 that I just decided to crank the numbers way up and let it run over the weekend.

I'll be looking forward to a newer faster cvs2git, although I did just get the graft idea working, so not sure if we'll wait that long or not (would be nice not to have to muck around with it though).

Thanks, Kelly

Robin H. Johnson wrote:

In case anybody is following this, the issue that Robin found was in release 2.2.0 of cvs2svn/cvs2git, but had already been fixed in the current trunk version...

Conclusion 1: Please use the trunk version of cvs2git, not release 2.2.0. Trunk is usually the most stable version available, especially for 2git conversions.

Conclusion 2: I've really got to get around to making a new release :-/

Michael

On Fri, Jun 5, 2009 at 12:30 PM, Kelly F. Hickel <kfh@mqsoftware.com> wrote:

The usual replacement for "manually using format-patch" is to use "git rebase." It does pretty much exactly what you're describing, assuming you don't do too many complicated merges in the meantime.

Another option is to use the .git/info/grafts file. Here's a brief intro: http://git.or.cz/gitwiki/GraftPoint

You'd use that to pretend the parent of your top-skimmed branch is actually the equivalent commit in your new branch. Then could run "git filter-branch" to make the graft permanent, and get all your users to switch to the new repository.

Or you could skip the filter-branch stuff and keep the really hold history somewhere else, available for use if someone installs the graft in their local repo. This would lead to a smaller repository in the general case. (I gather that's what the Linux kernel does for per-2.6.11 versions.)

Have fun,

Avery

Thanks Avery,
	This appears to be just what I was looking for!  I'll fiddle with it a bit to see if I can convince it to work for me.

Thanks,
	Kelly

Kelly F. Hickel, 05.06.2009:

If rebasing 'new' on top of 'old' isn't an option, then you could try:

    $ git checkout new
    $ git merge -s ours old

It's the other way round (not merging 'new' into 'old', but vice versa), but there is now merge strategy "theirs".

Markus