Re: Why so much time in the kernel?

From: Linus Torvalds <torvalds@osdl.org>
Date: Jun 16, 2006, 15:06 UTC
Message-ID: <Pine.LNX.4.64.0606160755170.5498@g5.osdl.org>
In-Reply-To: <9e4733910606160749t4d7a541ev72a67383e96d86da@mail.gmail.com>

On Fri, 16 Jun 2006, Jon Smirl wrote:

> 
> I am spending over 40% of the time in the kernel. This looks to be
> caused from forks and starting small tasks, is that the correct
> interpretation?

Yes. Your kernel profile is all for stuff related to setting up and tearing down process space (well, __mutex_lock_slowpath at 1.88% and __d_lookup at 1.3% is not, but every single one before that does seem to be about fork/exec/exit).

I think it's both the CVS server that continually forks/exits (it doesn't actually do a exec at all - it seem sto be using fork/exit as a way to control its memory usage - knowing that the OS will free all the temporary memory on exit - I think the newer CVS development trees don't do this, but that also seems to be why they leak memory like mad and eventually run out ;).

AND it's git-cvsimport forking and exec'ing git helper processes.

So that process overhead is expected.

What I would _not_ have expected is:

>   933646  2.0983 /usr/local/bin/git-read-tree

I don't see why git-read-tree is so hot for you. We should never need to read a tree when we're importing something, unless there are tons of branches and we switch back and forth between them.

I guess mozilla really does use a fair number of branches?

Martin sent out a patch (that I don't think has been merged yet) to avoid the git-read-tree overhead when switching branches. Look for an email with a subject like "cvsimport: keep one index per branch during import", I suspect that would speed up the git part a lot.

(It will also avoid a few fork/exec's, but you'll still have most of them, so I don't think you'll see any really _fundamental_ changes to this, but the git-read-tree overhead should be basically gone, and some of the libz.so pressure would also be gone with it. It should also avoid rewriting the index file, so you'd get lower disk pressure, but it looks like none of your problems are really due to IO, so again, that probably won't make much of a difference for you).

			Linus