Re: [PATCH] cvsimport: introduce -L<imit> option to workaround memory leaks

From: Linus Torvalds <torvalds@osdl.org>
Date: May 23, 2006, 16:05 UTC
Message-ID: <Pine.LNX.4.64.0605230848060.5623@g5.osdl.org>
In-Reply-To: <20060523153636.GA21506@thunk.org>

On Tue, 23 May 2006, Theodore Tso wrote:

Show 7 quoted lines

> On Mon, May 22, 2006 at 07:28:37PM -0700, Linus Torvalds wrote:
> > 
> > I actually think that I found a real ext3 performance bug from trying to 
> > determine why git sometimes slows down ridiculously when the tree has been 
> > allowed to go too long without a repack.
> 
> Do you have dir_index (the hashed btree) feature enabled by any chance?

No, and I know I probably should, since it would hopefully help git usage.

But my problem actually happens even with moderately sized directories: they were just 40kB or so in size, and the problem isn't high system CPU usage, but tons of extra IO. I ran things on a machine with 2GB of RAM, and as far as I could tell, the working set _should_ have fit into memory, but CPU utilization was consistently in the 1% range.

Now, it's possible that I'm just wrong, and it really didn't fit in memory, but I I _suspect_ that the issue is that ext3 directory handling still uses the "buffer_head" thing rather than the page cache, and that we simply don't LRU the memory appropriately so we don't let the memory pressure expand the buffer cache.

Now, using buffer cache in this day and age is insane and horrible (there's a reason I suspect the LRU doesn't work that well: the buffer heads aren't supposed to be used as a cache, and people are supposed to use the page cache for it these days), but Andrew tells me that the whole JBD thing basically requires it. Whatever.

Now, repacking obviously hides it entirely (because then the load becomes entirely a page-cache load, and the kernel does _that_ beautifully), but I'm a bit bummed that I think I hit an ext3 braindamage.

So an unpacked git archive on ext3 (but not ext2, I believe: ext2 should use the page cache for directories) ends up being very buffer-cache intensive. And the buffer cache is basically deprecated..

		Linus

PS. I'll see if I can figure out the problem, and maybe the good news is that I'll be able to just fix a real kernel performance issue. Still, there's a _reason_ we tried to get away from the buffer heads as a caching entity..