Re: [PATCH] cvsimport: introduce -L<imit> option to workaround memory leaks

From: Linus Torvalds <torvalds@osdl.org>
Date: May 26, 2006, 05:20 UTC
Message-ID: <Pine.LNX.4.64.0605252204590.5623@g5.osdl.org>
In-Reply-To: <46a038f90605251742p2435ae23k8bfbb98409a30c1c@mail.gmail.com>

On Fri, 26 May 2006, Martin Langhoff wrote:

> 
> Call me slow, but I am still running and rerunning that gentoo import. ;-)

I'm doing it too, just for fun.

Of course, since I'm doing this on a machine that basically has a laptop disk, the "just for fun" part is a bit sad. It's waiting for disk about 25% of the time ;/

And it's slow as hell. I really wish we could do better on the CVS import front.

> The current import has reached ~200K commits, and .git is 450MB, while
> the checked out tree is 230MB (680MB with .git). At this stage, git
> repack -a -d is too memory hungry.

I've got 2GB in that puppy, and "repack -a -d" is fine for me. I'm not quite up to 200k commits yet (I'm at 160k), but the repacking is certainly faster than the rest of the import.. Gaah.

It's "git-rev-list --objects" that is the memory sucker for me, the packing itself doesn't seem to be too bad.

The biggest cost seems to be git-write-tree, which is about 0.225 seconds for me on that tree on that machine. Which _should_ mean that we could do 4 commits a second, but that sure as hell ain't how it works out. It seems to do about 1.71 commits a second for me on that tree, which is pretty damn pitiful. Some cvs overhead, and probably some other git overhead too.

(That's a 2GHz Merom, so the fact that you get ~6k commits per hour on your 2GHz Opteron is about the same speed - I suspect you're also at least partly limited by disk, our numbers seem to match pretty well).

200k commits at 6k commits per hour is about a day and a half (plus the occasional packing load). Taking that long to import a CVS archive is horrible. But I guess it _is_ several years of work, and I guess you really have to do it only once, but still.

			Linus