Re: 'git gc --aggressive' effectively unusable

From: Jeff King <peff@peff.net>
Date: Apr 4, 2010, 20:38 UTC
Message-ID: <20100404203850.GA8798@coredump.intra.peff.net>
In-Reply-To: <87zl1js248.fsf@troilus.org>

On Sun, Apr 04, 2010 at 10:50:47AM -0400, Michael Poole wrote:

Show 18 quoted lines

> > Are you sure it doesn't subsequently speed up again?
> 
> I have seen asymptotic slowdown as "git gc --aggressive" progresses on
> certain repositories.  It is particularly bad with
> git://git.infradead.org/gcc.git (on an x86-64 system with 4 GB RAM).
> git seemed to be thrashing swap badly as time went on.  I don't know
> that git gc --aggressive would *never* finish on my gcc-git repository.
> I just know that it got to about 80% done in less than an hour, to 90%
> after twelve hours, and about 94% after another twelve hours.  (The same
> operation on linux-2.6.git takes about 40 minutes with all the default
> settings.)
> 
> I may have been dreaming, but I thought with some 1.6.x version of git,
> reducing core.packedGitLimit and pack.windowLimit (now windowMemory?)
> mostly made the thrashing go away.  When I try again with v1.7.0.2,
> though, it doesn't seem to help very much -- there is still a lot of
> swapping, and the git process got to about 7 GB virtual size before I
> killed it after about 10 hours of operation.

I packed Frans' sample kernel repo with "git gc --aggressive" last night. It did finish after about 9 hours. I didn't take memory usage measurements, but here's what time said:

  real    535m38.898s
  user    216m46.437s
  sys     0m24.186s

That's 3.6 hours of CPU time over almost 9 hours (on a dual-core machine). The non-agressive pack was about 680M, and the result was 480M. The machine has 2G of RAM, and not much else running. So I would really not expect there to be much disk I/O required, but clearly we were waiting quite a bit.

I'll try tweaking a few of the pack memory limits and try again.

-Peff