Re: git gc expanding packed data?

From: Nicolas Pitre <nico@cam.org>
Date: Aug 12, 2009, 15:35 UTC
Message-ID: <alpine.LFD.2.00.0908121118420.10633@xanadu.home>
In-Reply-To: <3ace41890908120745y305b2a59hd453b7e146e4a774@mail.gmail.com>

On Wed, 12 Aug 2009, Hin-Tak Leung wrote:

Show 19 quoted lines

> On Tue, Aug 11, 2009 at 10:33 PM, Nicolas Pitre<nico@cam.org> wrote:
> <snipped>
> 
> > From git v1.6.3 the --aggressive switch makes for 'git repack' to be
> > called with --window=250 --depth=250, meaning the equivalent of:
> >
> >        git repack -a -d -f --window=250 --depth=250
> >
> > Do you still get a huge pack with the above?
> >
> >> I guess --aggressive doesn't always save space...
> >
> > If so that is (and was) a bug.
> 
> I tried 'git repack -a -d -f --window=250 --depth=250' with 1.6.2.5
> (fc11.x86_64) and it took half a day, swallowed up all the memory -
> 3GB virtual & 1.3GB resident - and finally the kernel oom killer
> killed it at a last message of (601460/957910). Left no temp files.
> Would git 1.6.3 use less memory? :-(

Probably not.  However you should try:

	git config pack.deltaCacheSize 1

That limits the delta cache size to one byte (effectively disabling it) instead of the default of 0 which means unlimited. With that I'm able to repack that repository using the above git repack command on an x86-64 system with 4GB of RAM and using 4 threads (this is a quad core). Resident memory usage grows to nearly 3.3GB though.

If your machine is SMP and you don't have sufficient RAM then you can reduce the number of threads to only one:

	git config pack.threads 1

Additionally, you can further limit memory usage with the --window-memory argument to 'git repack'. For example, using --window-memory=128M should keep a reasonable upper bound on the delta search memory usage although this can result in less optimal delta match if the repo contains lots of large files (and I think this is the case for the gcc repo).

Nicolas