Re: pack operation is thrashing my server

From: Jon Smirl <jonsmirl@gmail.com>
Date: Sep 7, 2008, 02:50 UTC
Message-ID: <9e4733910809061950g6d9d2cf1g708f8faf0c06108@mail.gmail.com>
In-Reply-To: <alpine.LFD.1.10.0809061812090.3117@nehalem.linux-foundation.org>

On 9/6/08, Linus Torvalds <torvalds@linux-foundation.org> wrote:

Show 10 quoted lines

>  I'm starting to hate zlib. I actually spent almost a week trying to clean
>  up the zlib source code and make it something that gcc can compile into
>  clean code, but the fact is, zlib isn't amenable to that. The whole "shift
>  <n> bits in from the buffer" approach means that there is no way to make
>  zlib generate good code unless you are an insanely competent assembly
>  hacker or have tons of registers to keep all the temporaries live in.
>
>  Now, I still do think that all my reasons for choosing zlib were pretty
>  solid (it's a well-tested piece of code and it is _everywhere_ and easy to
>  use), but boy do I wish there had been alternatives.

Some alternative algorithms are here... http://cs.fit.edu/~mmahoney/compression It is possible to beat zlib by 2x at the cost of CPU time and memory.

Of course switching to these algorithms would involve a lot of testing and benchmarking. I'm also not sure how PAQ would fare on lots of small git objects instead of large files.

Turning a 500MB packfile into a 250MB has lots of advantages in IO reduction so it is worth some CPU/memory to create it.

You can even win 50'000€ for a better algorithm. http://prize.hutter1.net/

-- 
Jon Smirl
jonsmirl@gmail.com