Re: [PATCH] pack-objects: only throw away data during memory pressure

From: Nicolas Pitre <nico@cam.org>
Date: Feb 11, 2008, 16:00 UTC
Message-ID: <alpine.LFD.1.00.0802111054080.2732@xanadu.home>
In-Reply-To: <alpine.LSU.1.00.0802111519300.3870@racer.site>

On Mon, 11 Feb 2008, Johannes Schindelin wrote:

Show 21 quoted lines

> Hi,
> 
> On Mon, 11 Feb 2008, Martin Koegler wrote:
> 
> > What about this not really tested patch for dealing with memory pressure 
> > in git-pack-objects?
> > 
> > It will slow down the repack in the case of memory pressure, but missing 
> > memory will not affect the results.
> 
> It almost helped:
> 
> $ /usr/bin/time git repack -a -d -f --window=250 --depth=250
> Counting objects: 2477715, done.
> fatal: Out of memory, malloc failed411764)
> Command exited with non-zero status 1
> 10050.12user 240.63system 2:53:37elapsed 98%CPU (0avgtext+0avgdata 
> 0maxresident)k
> 0inputs+0outputs (29555major+94032945minor)pagefaults 0swaps
> 
> So, it ran longer until it ran out of memory.

What it can do for you is to limit the window memory usage much more without affecting the end result, say to 128MB. Of course the repack is then going to progress much slower if active purging of the window memory is involved.

If you still run out of memory at that point then there is not much more to do besides using a new memory allocator that doesn't suffer as much from memory fragmentation, or find the possible memory leak that no one else found so far.

Nicolas