{"thread":{"id":"20438","subject":"[PATCH 3/3] gitweb: add support for XZ compressed snapshots","startedAt":"2009-08-06T14:28:25Z","lastAt":"2009-08-06T14:28:25Z","messageCount":1,"participants":["Mark A Rada"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"119780","messageId":"070B9CE6-BD6F-49B2-BE20-7AF231537C13@uwaterloo.ca","threadId":"20438","inReplyTo":null,"subject":"[PATCH 3/3] gitweb: add support for XZ compressed snapshots","fromName":"Mark A Rada","fromEmail":"marada@uwaterloo.ca","sentAt":"2009-08-06T14:28:25Z","receivedAt":"2009-08-06T14:28:25Z","isPatch":true,"sender":{"key":"marada@uwaterloo.ca","avatar":"https://avatars.githubusercontent.com/u/38430?v=4"},"body":"The XZ compression format uses the LZMA2 compression algorithm, which\noften yields higher compression ratios than both GZip and BZip2 at the\ncost of using more CPU time and RAM. XZ is the slowest for compression,\nbut still much faster than BZip2 for decompression, almost comparable\nto GZip (see benchmarks below).\n\nSome simple benchmarks show the pros and cons of using XZ compression;\nstarting with an already tarball'd archive of the repos listed below.\nMemory usage seemed to be consistent for any given algorithm at their\nrespective default compression levels.\n\nCPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)\nVirtual Memory Usage\n        GZip: 4152K        BZip2: 13352K        XZ: 102M\n\nLinux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)  349M\ngzip    23.70s user   0.47s system  99% cpu   24.227 total    76M\ngunzip  3.74s user    0.74s system  94% cpu   4.741 total\nbzip2   130.96s user  0.53s system  99% cpu   2:11.97 total   59M\nbunzip2 31.05s user   1.02s system  99% cpu   32.355 total\nxz      448.78s user  0.91s system  99% cpu   7:31.28 total   51M\nunxz    7.67s user    0.80s system  98% cpu   8.607 total\n\nGit (0a53e9ddeaddad63ad106860237bbf53411d11a7)                11M\ngzip    0.77s user    0.03s system  99% cpu   0.792 total    2.5M\ngunzip  0.12s user    0.02s system  98% cpu   0.142 total\nbzip2   3.42s user    0.02s system  99% cpu   3.454 total    2.1M\nbunzip2 0.95s user    0.03s system  99% cpu   0.984 total\nxz      12.88s user   0.14s system  98% cpu   13.239 total   1.9M\nunxz    0.27s user    0.03s system  99% cpu   0.298 total\n\nXZ (669413bb2db954bbfde3c4542fddbbab53891eb4)                1.8M\ngzip    0.12s user    0.00s system  95% cpu   0.132 total    442K\ngunzip  0.02s user    0.00s system  97% cpu   0.027 total\nbzip2   1.28s user    0.01s system  99% cpu   1.298 total    363K\nbunzip2 0.15s user    0.01s system  100% cpu  0.157 total\nxz      1.62s user    0.03s system  99% cpu   1.652 total    347K\nunxz    0.05s user    0.00s system  99% cpu   0.058 total\n\n From a time and memory perspective, nothing compares to GZip, but if\ngiven an average upload speed of 20KB/s, it would take ~400 seconds\nlonger to transfer the BZip2'd kernel snapshot than the XZ snapshot;\nthe transfer time difference is even greater between GZip and XZ. The\nreal time savings are relatively the same for all test cases, but less\ndramatic for smaller repositories.\n\nXZ decompresses ~1.8-2 times slower than GZip, and ~2.7-3.75 times\nfaster than BZip2; XZ gets relatively faster as snapshots get larger.\nHowever, XZ takes relatively longer to compress as snapshots get larger.\n\nThe downside for XZ'd snapshots is the large CPU and memory load put on\nthe server to generate the compressed snapshot, though XZ will  \neventually\nhave threading support, and the real clock time for making XZ'd  \nsnapshots\nwould decrease if the server had a beefy multi-core CPU.\n\nXZ compression is disabled by default to allow upgrades to take place\nwithout any surprises, as the CPU and memory requirements will be an\nissue for high load or lightweight servers. Also, the XZ format is still\nnew (format declared stable ~6 months ago), and there have been no\n\"stable\" releases of the utils yet.\n\nSigned-off-by: Mark Rada <marada@uwaterloo.ca>\n---\n  gitweb/gitweb.perl |    9 +++++++++\n  1 files changed, 9 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex a0cdf31..84659f5 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -177,6 +177,14 @@ our %known_snapshot_formats = (\n  \t\t'format' => 'tar',\n  \t\t'compressor' => ['bzip2']},\n\n+\t'txz' => {\n+\t\t'display' => 'tar.xz',\n+\t\t'type' => 'application/x-xz',\n+\t\t'suffix' => '.tar.xz',\n+\t\t'format' => 'tar',\n+\t\t'compressor' => ['xz'],\n+\t\t'disabled' => 1},\n+\n  \t'zip' => {\n  \t\t'display' => 'zip',\n  \t\t'type' => 'application/x-zip',\n@@ -189,6 +197,7 @@ our %known_snapshot_formats = (\n  our %known_snapshot_format_aliases = (\n  \t'gzip'  => 'tgz',\n  \t'bzip2' => 'tbz2',\n+\t'xz'    => 'txz',\n\n  \t# backward compatibility: legacy gitweb config support\n  \t'x-gzip' => undef, 'gz' => undef,\n-- \n1.6.4\n"}]}