Re: RAM consumption when working with the gcc repo

From: Martin Koegler <mkoegler@auto.tuwien.ac.at>
Date: Dec 8, 2007, 17:24 UTC
Message-ID: <20071208172437.GA24442@auto.tuwien.ac.at>
In-Reply-To: <alpine.LFD.0.99999.0712071529580.555@xanadu.home>

On Fri, Dec 07, 2007 at 03:46:30PM -0500, Nicolas Pitre wrote:

Show 22 quoted lines

> On Fri, 7 Dec 2007, david@lang.hm wrote:
> > On Fri, 7 Dec 2007, Jon Smirl wrote:
> > 
> > > I noticed two things when doing a repack of the gcc repo. First is
> > > that the git process is getting to be way too big. Turning off the
> > > delta caches had minimal impact. Why does the process still grow to
> > > 4.8GB?
> > > 
> > > Putting this in perspective, this is a 4.8GB process constructing a
> > > 330MB file. Something isn't right. Memory leak or inefficient data
> > > structure?
> > 
> > keep in mind that that 330MB file is _very_ heavily compressed. the simple
> > zlib compression is probably getting you 10:1 or 20:1 compression and the
> > delta compression is a significant multiplier on top of that.
> 
> Doesn't matter.  Something is indeed fishy.
> 
> The bulk of pack-objects memory consumption can be estimated as follows:
> 
> 1M objects * sizeof(struct object_entry) ~= 100MB
> 256 window entries with data (assuming a big 1MB per entry) = 256MB

For each (uncompressed) object in the delta window, a delta index is created. It can have the same size as the uncompressed object.

Each thread has its own window, so using 4 threads means having 1024 objects in memory => 1 GB

Show 12 quoted lines

> Delta result caching was disabled therefore 0MB
> read-side delta cache limited to 16MB
> 
> So the purely ram allocation might get to roughly 400MB.
> 
> Then add the pack and index map, which, depending on the original pack 
> size,
> might be 2GB.
> 
> So we're pessimistically talking of about 2.5GB of virtual space.
> 
> The other 2.3GB is hard to explain.

mfg Martin Kögler