Re: obnoxious CLI complaints

From: René Scharfe <rene.scharfe@lsrfire.ath.cx>
Date: Sep 11, 2009, 22:01 UTC
Message-ID: <4AAAC8CE.8020302@lsrfire.ath.cx>
In-Reply-To: <alpine.LFD.2.01.0909110744030.3654@localhost.localdomain>

Am 11.09.2009 16:47, schrieb Linus Torvalds:

Show 18 quoted lines

>
>
> On Fri, 11 Sep 2009, René Scharfe wrote:
>>
>> Using zlib directly avoids the overhead of a pipe and of buffering the
>> output for blocked writes; surprisingly (to me), it isn't any faster.
>
> In fact, it should be slower.
>
> On SMP, you're quite likely better off using the pipe, and compressing on
> another CPU. Of course, it's usually the case that the compression is _so_
> much slower than generating the tar-file (especially for the hot-cache
> case) that it doesn't matter or the pipe overhead is even a slowdown.
>
> But especially if generating the tar-file has some delays in it
> (cold-cache object lookup, whatever), the "compress in separate process"
> is likely simply better, because you can compress while the other process
> is looking up data for the tar.

Yes, that makes sense and can be seen here (quad core, Fedora 11, best of five consecutive runs, Linux kernel repo):

	# git v1.6.5-rc0
	$ time git archive --format=tar v2.6.31 | gzip -6 >/dev/null

	real	0m16.591s
	user	0m19.769s
	sys	0m0.474s

	# git v1.6.5-rc0 + patch
	$ time ../git/git archive --format=tar.gz -6 v2.6.31 >/dev/null

	real	0m20.390s
	user	0m20.299s
	sys	0m0.088s

User time is quite similar, real time is lower when using a pipe.

But what has bugged me since I added zip support is this result:

	# git v1.6.5-rc0
	$ time git archive --format=zip -6 v2.6.31 >/dev/null

	real	0m16.471s
	user	0m16.340s
	sys	0m0.128s

I'd have expected this to be the slowest case, because it's compressing all files separately, i.e. it needs to create and flush the compression context lots of times instead of only once as in the two cases above. And it's sequential and uses zlib, just like the tar.gz format. I suspect the convenience function gzwrite() adds this overhead.

Oh, I just discovered pigz (http://zlib.net/pigz/), a parallel gzip:

	# git v1.6.5-rc0, pigz 2.1.5
	$ time git archive --format=tar v2.6.31 | pigz -6 >/dev/null

	real	0m6.251s
	user	0m21.383s
	sys	0m0.547s

So pipes win. :) Still need to investigate why zip is as (relatively) fast as it is, though.

René