{"thread":{"id":"20340","subject":"[PATCHv2 1/2] gitweb: add support for XZ compressed snapshots","startedAt":"2009-08-01T20:06:13Z","lastAt":"2009-08-01T20:06:13Z","messageCount":1,"participants":["Mark A Rada"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"119354","messageId":"F58C96F8-01A3-48F9-B2A1-E4FB3908B399@uwaterloo.ca","threadId":"20340","inReplyTo":null,"subject":"[PATCHv2 1/2] gitweb: add support for XZ compressed snapshots","fromName":"Mark A Rada","fromEmail":"marada@uwaterloo.ca","sentAt":"2009-08-01T20:06:13Z","receivedAt":"2009-08-01T20:06:13Z","isPatch":false,"sender":{"key":"marada@uwaterloo.ca","avatar":"https://avatars.githubusercontent.com/u/38430?v=4"},"body":"Another go at it....no changes for this commit.\n\n--\nMark A Rada (ferrous26)\nmarada@uwaterloo.ca\n\n---------->8-------------\nFrom: Mark Rada <marada@uwaterloo.ca>\nDate: Thu, 30 Jul 2009 08:56:42 -0400\nSubject: [PATCH 1/2] gitweb: add support for XZ compressed snapshots\n\nThe XZ compression format uses the LZMA2 compression algorithm, which\noften yields higher compression ratios than both GZip and BZip2 at the\ncost of using more CPU time and RAM. Though, while XZ is the slowest\nfor compression, it is much faster than BZip2 for decompression, almost\ncomparable to GZip (see benchmarks below).\n\nYou can enable XZ compressed snapshots by adding 'txz' to the list of\ndefault options for snapshots in your $GITWEB_CONFIG.\n\nI did some simple benchmarks, starting with an already tarballed\narchive of the repos listed below. Memory usage seemed to be consistent\nfor any given algorithm at their default compression level. Timings were\ngathered using the `time' command.\n\nCPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)\nVirtual Memory Usage\n         GZip: 4152K        BZip2: 13352K        XZ: 102M\n\nLinux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)   349M\ngzip    23.70s user   0.47s system  99% cpu   24.227 total    76M\ngunzip  3.74s user    0.74s system  94% cpu   4.741 total\nbzip2   130.96s user  0.53s system  99% cpu   2:11.97 total   59M\nbunzip2 31.05s user   1.02s system  99% cpu   32.355 total\nxz      448.78s user  0.91s system  99% cpu   7:31.28 total   51M\nunxz    7.67s user    0.80s system  98% cpu   8.607 total\n\nGit (0a53e9ddeaddad63ad106860237bbf53411d11a7)                11M\ngzip    0.77s user    0.03s system  99% cpu   0.792 total     2.5M\ngunzip  0.12s user    0.02s system  98% cpu   0.142 total\nbzip2   3.42s user    0.02s system  99% cpu   3.454 total     2.1M\nbunzip2 0.95s user    0.03s system  99% cpu   0.984 total\nxz      12.88s user   0.14s system  98% cpu   13.239 total    1.9M\nunxz    0.27s user    0.03s system  99% cpu   0.298 total\n\nXZ (669413bb2db954bbfde3c4542fddbbab53891eb4)                 1.8M\ngzip    0.12s user    0.00s system  95% cpu   0.132 total     442K\ngunzip  0.02s user    0.00s system  97% cpu   0.027 tota\nbzip2   1.28s user    0.01s system  99% cpu   1.298 total     363K\nbunzip2 0.15s user    0.01s system  100% cpu  0.157 total\nxz      1.62s user    0.03s system  99% cpu   1.652 total     347K\nunxz    0.05s user    0.00s system  99% cpu   0.058 total\n\nPurely from a time and memory perspective, nothing compares to GZip in\neach of the three tests. Though, if you have an average upload speed of\n20KB/s, it would take ~400 seconds longer to transfer the BZip2\ncompressed kernel snapshot than it would the XZ compressed snapshot,\nthe transfer time difference between GZip and XZ is even greater. The\nwall clock time savings are relatively the same for all test cases, but\nless dramatic for the smaller repositories.\n\nThe obvious downside for XZ compressed snapshots is the large CPU and\nmemory load put on the server to actualy generate a compressed\nsnapshot. Though XZ will eventually have good threading support, and I\nsuspect then that the wall clock time for making an XZ compressed\nsnapshot would go down considerably if the server had a beefy\nmulti-core CPU.\n\nI have not enabled XZ compression by default because the current default\nis GZip, and XZ is only really competitive with BZip2. Also, the XZ\nformat is still fairly new (the format was declared stable about 6\nmonths ago), and there have been no \"stable\" releases of the utils yet.\n\nSigned-off-by: Mark Rada <marada@uwaterloo.ca>\n---\n  gitweb/gitweb.perl |    8 ++++++++\n  1 files changed, 8 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 7fbd5ff..3398163 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -176,6 +176,13 @@ our %known_snapshot_formats = (\n  \t\t'format' => 'tar',\n  \t\t'compressor' => ['bzip2']},\n\n+\t'txz' => {\n+\t\t'display' => 'tar.xz',\n+\t\t'type' => 'application/x-xz',\n+\t\t'suffix' => '.tar.xz',\n+\t\t'format' => 'tar',\n+\t\t'compressor' => ['xz']},\n+\n  \t'zip' => {\n  \t\t'display' => 'zip',\n  \t\t'type' => 'application/x-zip',\n@@ -188,6 +195,7 @@ our %known_snapshot_formats = (\n  our %known_snapshot_format_aliases = (\n  \t'gzip'  => 'tgz',\n  \t'bzip2' => 'tbz2',\n+\t'xz'    => 'txz',\n\n  \t# backward compatibility: legacy gitweb config support\n  \t'x-gzip' => undef, 'gz' => undef,\n-- \n1.6.4\n"}]}