Re: pack operation is thrashing my server

From: Junio C Hamano <gitster@pobox.com>
Date: Sep 7, 2008, 01:03 UTC
Message-ID: <7vk5dorclv.fsf@gitster.siamese.dyndns.org>
In-Reply-To: <alpine.LFD.1.10.0808151729070.3324@nehalem.linux-foundation.org>

Linus Torvalds <torvalds@linux-foundation.org> writes:

Show 13 quoted lines

> Interestingly, it turns out that git also hits a sad performance downside 
> of using zlib.
>
> We always tend to set "stream.avail_out" to the exact size of the expected 
> output. And it turns out that that means that the fast-path case of 
> inffast.c doesn't trigger as often as it could. This (idiotic) patch 
> actually seems to help performance on git rev-list by about 5%.
>
> But maybe it's just me seeing things. But I did this because of the entry 
> assumptions in inflate_fast(), that code only triggers for the case of 
> strm->avail_out >= 258.
>
> Sad, if true.

This is reproducible "rev-list --objects --all" in my copy of the kernel repo takes around 47-48 seconds user time, and with the (idiotic) patch it is cut down to 41-42 seconds.

(with patch) 41.41user 0.51system 0:41.93elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+134411minor)pagefaults 0swaps

(without patch) 47.21user 0.64system 0:47.85elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+134935minor)pagefaults 0swaps

One funny thing about your patch is that it also reduces the number of minor faults; I would have expected that the additional memory wastage (even though most of the allocated object buffer memory would be freed immediately as soon as the caller is done with it) would result in larger number of faults, not smaller, which is puzzling.