Re: [PATCHv4] Add Gitweb support for XZ compressed snapshots

From: Jakub Narebski <jnareb@gmail.com>
Date: Jul 31, 2009, 18:48 UTC
Message-ID: <m34osssn7o.fsf@localhost.localdomain>
In-Reply-To: <828BD9FC-1238-4B2E-858D-248977F04D31@uwaterloo.ca>

Mark A Rada <marada@uwaterloo.ca> writes:

Show 8 quoted lines

> Ok, so I got a good nights sleep now, and reviewed the results of my
> benchmarks to make sure they were consistent (turns out I had the
> archive sizes in the wrong order for the XZ repository tests).
> 
> I also reworded a number of things and added a conclusion to the
> benchmarks.
> 
> Let me know what you think.

Well separated change.  Very detailed commit message; that's good!.

[...]

Show 23 quoted lines

> Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259) 	349M
> gzip 	23.70s user    	0.47s system  	99% cpu  24.227 total 	76M
> gunzip 	3.74s user     	0.74s system  	94% cpu  4.741 total
> bzip2 	130.96s user   	0.53s system  	99% cpu  2:11.97 total 	59M
> bunzip2 31.05s user    	1.02s system  	99% cpu  32.355 total
> xz 	448.78s user 	0.91s system  	99% cpu  7:31.28 total 	51M
> unxz 	7.67s user     	0.80s system  	98% cpu  8.607 total
> 
> Git (0a53e9ddeaddad63ad106860237bbf53411d11a7) 			11M
> gzip 	0.77s user 	0.03s system 	99% cpu  0.792 total 	2.5M
> gunzip 	0.12s user 	0.02s system 	98% cpu  0.142 total
> bzip2 	3.42s user 	0.02s system 	99% cpu  3.454 total 	2.1M
> bunzip2 0.95s user 	0.03s system 	99% cpu  0.984 total
> xz 	12.88s user 	0.14s system 	98% cpu  13.239 total 	1.9M
> unxz 	0.27s user 	0.03s system 	99% cpu  0.298 total
> 
> XZ (669413bb2db954bbfde3c4542fddbbab53891eb4) 			1.8M
> gzip  	0.12s user 	0.00s system 	95% cpu  0.132 total 	442K
> gunzip 	0.02s user 	0.00s system 	97% cpu  0.027 total
> bzip2  	1.28s user 	0.01s system 	99% cpu  1.298 total 	363K
> bunzip2 0.15s user 	0.01s system 	100% cpu 0.157 total
> xz 	1.62s user 	0.03s system 	99% cpu  1.652 total 	347K
> unxz   	0.05s user 	0.00s system 	99% cpu  0.058 total

Note that for me the above results are not aligned in table. This is a cosmetic issue.

Show 21 quoted lines

> Purely from a time and memory perspective, nothing compares to GZip in
> each of the three tests. Though, if you have an average upload speed of
> 20KB/s, it would take ~400 seconds longer to transfer the kernel
> snapshot
> that was BZip2 compressed than it would the XZ compressed snapshot, the
> transfer time difference is even greater when compared to the GZip
> compressed snapshot. The wall clock time savings are relatively the same
> for all test cases, but less dramatic for the smaller repositories.
> 
> The obvious downside for XZ compressed snapshots is the large CPU and
> memory load put on the server to actualy generate the snapshot. Though
> XZ
> will eventually have good threading support, and I suspect then that the
> wall clock time for making an XZ compressed snapshot would go down
> considerably if the server had a beefy multi-core CPU.
> 
> I have not enabled XZ compression by default because the current default
> is GZip, and XZ is only really competitive with BZip2. Also, the XZ
> format
> is still fairly new (the format was declared stable about 6 months ago),
> and there have been no "stable" releases of the utils yet.

Those above three paragraphs are strangely wrapped, with single word for a whole line ('snapshot', 'XZ', 'format'). This is a cosmetic issue.

> 
> Signed-off-by: Mark Rada <marada@uwaterloo.ca>

-- 
Jakub Narebski
Git User's Survey 2009: http://tinyurl.com/GitSurvey2009