Re: Errors GITtifying GCC and Binutils

From: Eric Wong <normalperson@yhbt.net>
Date: Mar 25, 2006, 08:25 UTC
Message-ID: <20060325082521.GA17473@hand.yhbt.net>
In-Reply-To: <20060324182504.GI31387@lug-owl.de>

Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:

> On Wed, 2006-03-22 14:33:37 +0100, Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:
> 
> Since it seems nobody looked at the GCC import run (which means to use
> the svnimport), I ran it again, under strace control:

If you don't care for automated branch handling, how about trying git-svn? under the contrib/ directory in git.git

	git-svn init svn://gcc.gnu.org/svn/gcc
	git-svn fetch

Show 16 quoted lines

> > GCC
> > ~~~
> > $ /home/jbglaw/bin/git svnimport -C gcc -v svn://gcc.gnu.org/svn/gcc
> 
> > Committed change 3936:/ 1993-03-31 05:44:03)
> > Commit ID ceff85145f8671fb2a9d826a761cedc2a507bd1e
> > Writing to refs/heads/origin
> > DONE: 3936 origin ceff85145f8671fb2a9d826a761cedc2a507bd1e
> > ... 3937 trunk/gcc/final.c ...
> > Can't fork at /home/jbglaw/bin/git-svnimport line 379.
> 
> ... 4279 trunk/gcc/config/i386/xm-sco.h ...
> 
> This time it broke at a different revision, so I guess it's not a SVN
> or git / git-svnimport problem, but rather a problem of my Perl
> installation or the kernel itself?

I've known of SVN library bindings leaking memory in the past, but I thought it's been solved. Afaik, any memory allocated by the Perl interpreter is never released back to the kernel, either. (At least that seems to be the case with my setup (Debian unstable, Perl 5.8.8, 2.6 kernel, x86 machine).

Show 10 quoted lines

> What are possible reasons for clone() to fail with -ENOMEN? I have to
> admit that the box _is_ loaded a bit all the time:
> 
> jbglaw@bixie:~/vax/git-conversion$ uptime
>  19:23:58 up 136 days,  7:46, 20 users,  load average: 4.45, 4.25, 3.05
> jbglaw@bixie:~/vax/git-conversion$ free
>              total       used       free     shared    buffers     cached
> Mem:        507308     501760       5548          0       2184      16900
> -/+ buffers/cache:     482676      24632
> Swap:      2441872    1295512    1146360

Some importers (my own git-svn included) aren't amazingly efficient when handling lots of history which gcc has. It looks like (from what I understand of the SVN api used in git-svnimport) is that the entire log for the 100k+ revisions in the tree is slurped down into memory before any processing is done.

git-svn does this too, but by parsing the output of the svn binary instead of using the library, so at least it won't have issues with the svn bindings and libraries to worry about.

My git-svn process running on the SVN tree just finished parsing the svn log output, and it's maxed out at 74M RSS (on a 32-bit x86). It'll probably take a while to import it all (which I won't do), but I could have just as easily done the following to reduce memory usage by ~half:

	git-svn fetch -r0:50000		# import the first 50000k
	git-svn fetch			# now import the remaining

Afaik, there's no way to do something like the above with git-svnimport for memory-starved setups.

-- 
Eric Wong