Re: pack operation is thrashing my server

From: Jon Smirl <jonsmirl@gmail.com>
Date: Sep 7, 2008, 20:33 UTC
Message-ID: <9e4733910809071333t57d03257m34fd6a752e40177e@mail.gmail.com>
In-Reply-To: <alpine.LFD.1.10.0809071304130.23787@xanadu.home>

On 9/7/08, Nicolas Pitre <nico@cam.org> wrote:

Show 26 quoted lines

> On Sun, 7 Sep 2008, Jon Smirl wrote:
>
>  > On 9/7/08, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>  > >
>  > >
>  > >  On Sat, 6 Sep 2008, Jon Smirl wrote:
>  > >  >
>  > >
>  > > > When I was playing with those giant Mozilla packs speed of zlib wasn't
>  > >  > a big problem. Number one problem was the repack process exceeding 3GB
>  > >  > which forced me to get 64b hardware and 8GB of memory. If you start
>  > >  > swapping in a repack, kill it, it will probably take a month to
>  > >  > finish.
>  > >
>  > >
>  > > .. and you'd make things much much WORSE?
>  >
>  > My observations on the Mozilla packs indicated that the problems were
>  > elsewhere in git, not in the decompression algorithms. Why does a
>  > single 2000 delta chain take 15% of the entire pack time? Something
>  > isn't right when long chains are processed which triggers far more
>  > decompressions than needed.
>
>
> Please have a look at commit eac12e2d4d7f.  This fix improved things for
>  my gcc repack tests.

Do you have any test numbers for something like a 2000 delta chain before and after?

You can get to Mozilla CVS with rsync. https://wiki.mozilla.org/How_to_Create_a_CVS_Mirror I think it was the master Mozilla makefile with the 2000 deltas. The whole repo is 15GB so you probably just want the Makefile,v

There's no point in working with Mozilla except for testing purposes since they went with Mercurial and abandoned their history.

-- 
Jon Smirl
jonsmirl@gmail.com