Re: 'git gc --aggressive' effectively unusable

From: Michael Poole <mdpoole@troilus.org>
Date: Apr 4, 2010, 14:50 UTC
Message-ID: <87zl1js248.fsf@troilus.org>
In-Reply-To: <87oci0m5v4.fsf@catnip.gol.com>

Miles Bader writes:

Show 15 quoted lines

> Frans Pop <elendil@planet.nl> writes:
>>> > I haven't had the patience to let it finish
>>>
>>> There's your problem.
>>
>> Yes, I had seen that. But there's a difference between taking much more 
>> time and slowing down to such an extend that it never finishes.
>>
>> I've tried it today on my linux-2.6 repo as well and the same thing 
>> happened. At first the progress is not fast but reasonable. When it gets 
>> to about 45% percent it starts slowing down a lot: from ~1500 objects per 
>> update of the counters to ~300 objects per update. And who knows what the 
>> progress is going to be when it reaches 70% or 90%: 10 per update?
>
> Are you sure it doesn't subsequently speed up again?

I have seen asymptotic slowdown as "git gc --aggressive" progresses on certain repositories. It is particularly bad with git://git.infradead.org/gcc.git (on an x86-64 system with 4 GB RAM). git seemed to be thrashing swap badly as time went on. I don't know that git gc --aggressive would *never* finish on my gcc-git repository. I just know that it got to about 80% done in less than an hour, to 90% after twelve hours, and about 94% after another twelve hours. (The same operation on linux-2.6.git takes about 40 minutes with all the default settings.)

I may have been dreaming, but I thought with some 1.6.x version of git, reducing core.packedGitLimit and pack.windowLimit (now windowMemory?) mostly made the thrashing go away. When I try again with v1.7.0.2, though, it doesn't seem to help very much -- there is still a lot of swapping, and the git process got to about 7 GB virtual size before I killed it after about 10 hours of operation.

Michael Poole