Re: Excessive memory consumption.

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mar 17, 2008, 17:04 UTC
Message-ID: <alpine.LFD.1.00.0803170956430.3020@woody.linux-foundation.org>
In-Reply-To: <20080317154353.GA9062@old.davidb.org>

On Mon, 17 Mar 2008, David Brown wrote:

> 
> phase, git-pack-objects quickly grows to 3 or more GB in size.  I've tried
> setting  pack.windowMemory = 512m, and pack.deltaCacheLimit = 512m.  Any
> other suggested things to try?

This *might* be the malloc() fragmentation issue. If you have lots of large files with deltas in between them, you might well end up with a lot of the allocations having been 'free()'d, but even if you have "only" one gig in active use, there might be lots of holes in the allocation patterns with "free" memory.

Does lowering the limits more aggressively make any difference (ie try with those limits set to just a few tens of megs?) It will cause much worse deltas (since now the delta machinery won't work well on those big tar-files), so you probably don't want to do this for real, but as a test of whether it's those allocations or soemthing else it would still be interesting.

But using valgrind (--tool=massif) to show the real allocations might also give an even better view. Maybe valgrind tells you only 1 gig is used, and then yes, it would be about some kind of fragmentation issue. Or maybe valgrind points out some other memory user entirely..

		Linus