Re: git-svnimport failed and now git-repack hates me

From: Eric Wong <normalperson@yhbt.net>
Date: Jan 4, 2007, 02:33 UTC
Message-ID: <20070104023350.GA1194@localdomain>
In-Reply-To: <Pine.LNX.4.64.0701031737300.4989@woody.osdl.org>

Linus Torvalds <torvalds@osdl.org> wrote:

> On Wed, 3 Jan 2007, Chris Lee wrote:
> > First issue I ran into: On a machine with 4GB of RAM, when I tried to
> > do a full import, git-svnimport died after 309906 revisions, saying
> > that it couldn't fork.

Managing memory with the Perl SVN libraries has been very painful in my experience.

Part of it is Perl, which (as far as I know) never frees allocated memory back to the OS (although Perl can reuse the allocated memory for other things). I'm CC-ing the resident Perl guru on this...

I'm also fairly certain that most higher-level languages have this problem.

Show 9 quoted lines

> I suspect you'd have been better off just re-starting, and using something 
> like
> 
> 	while :
> 	do
> 		git svnimport -l 1000 <...>
> 		.. figure out some way to decide if it's all done ..
> 		git repack -d
> 	done

> However, I don't know what the proper magic is for svnimport to do that 
> sane "do it in chunks and tell when you're all done". Or even better - to 
> just make it repack properly and not keep everything in memory.

<shameless self-promotion>
	git-svn already does this chunking internally

	Just set the repack interval to something smaller than 1000;
	(--repack=100) if you experience timeouts.
</shameless self-promotion>

-- 
Eric Wong