Re: [PATCH] RFC: git lazy clone proof-of-concept

From: Nicolas Pitre <nico@cam.org>
Date: Feb 12, 2008, 21:05 UTC
Message-ID: <alpine.LFD.1.00.0802121553120.2732@xanadu.home>
In-Reply-To: <alpine.LSU.1.00.0802122036150.3870@racer.site>

On Tue, 12 Feb 2008, Johannes Schindelin wrote:

Show 22 quoted lines

> Hi,
> 
> On Sun, 10 Feb 2008, Johannes Schindelin wrote:
> 
> > $ /usr/bin/time git repack -a -d -f --window=150 --depth=150
> > Counting objects: 2477715, done.
> > Compressing objects:  19% (481551/2411764)
> > Compressing objects:  19% (482333/2411764)
> > fatal: Out of memory, malloc failed411764)
> > Command exited with non-zero status 1
> > 7118.37user 54.15system 2:01:44elapsed 98%CPU (0avgtext+0avgdata 
> > 0maxresident)k
> > 0inputs+0outputs (29834major+17122977minor)pagefaults 0swaps
> 
> I made the window much smaller (512 megabyte), and it still runs, after 27 
> hours:
> 
> Compressing objects:  20% (484132/2411764)
> 
> However, it seems that it only worked on about 4000 objects in the last 
> 20(!) hours.  So, the first 19% were relatively quick.  The next percent 
> not at all.

Yeah... this repo is really a pain to repack. I have access to a 8-processor machine with 8GB of ram and all my repack attempts so far were killed after using too much memory, despite the window memory limit. Those were threaded repack attempts, so the first 98% was really quick, like less than 15 minutes, but then all threads converged on this small fraction of the object space which appears to cause problems. And then I'm presuming I ran into the same threaded memory fragmentation issue. Might be worth attaching gdb to it and extract a sample of the object SHA1's populating the delta window when the slowdown occurs to see what they actually are...

I'm attempting a single-threaded repack now.

Nicolas