Re: [PATCH] Correct pack memory leak causing git gc to try to exceed ulimit

From: Nicolas Pitre <nico@cam.org>
Date: Jul 10, 2008, 05:37 UTC
Message-ID: <alpine.LFD.1.10.0807100119360.12484@xanadu.home>
In-Reply-To: <20080709071007.GA31974@spearce.org>

On Wed, 9 Jul 2008, Shawn O. Pearce wrote:

Show 18 quoted lines

> When recursing to unpack a delta base we must unuse_pack() so that
> the pack window for the current object does not remain pinned in
> memory while the delta base is itself being unpacked and materialized
> for our use.
> 
> On a long delta chain of 50 objects we may need to access 6 different
> windows from a very large (>3G) pack file in order to obtain all
> of the delta base content.  If the process ulimit permits us to
> map/allocate only 1.5G we must release windows during this recursion
> to ensure we stay within the ulimit and transition memory from pack
> cache to standard malloc, or other mmap needs.
> 
> Inserting an unuse_pack() call prior to the recursion allows us to
> avoid pinning the current window, making it available for garbage
> collection if memory runs low.
> 
> This has been broken since at least before 1.5.1-rc1, and very
> likely earlier than that.  Its fixed now.  :)

Well well... I suspect this might have been the cause of our strange out-of-memory issue when trying to repack the gcc repository a while ago.

I updated my copy of git://git.infradead.org/gcc.git and re-attempted a full repack with window=500 and depth=500, and this time it worked for me on a 32-bit machine! Took about 6h30 single-threaded, and produced a mere 331MB pack file containing 1254664 objects across 142328 commits.

Nicolas