{"thread":{"id":"20319","subject":"[PATCHv4] Add Gitweb support for XZ compressed snapshots","startedAt":"2009-07-31T13:32:38Z","lastAt":"2009-07-31T18:48:29Z","messageCount":4,"participants":["Mark A Rada","J.H.","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"119270","messageId":"828BD9FC-1238-4B2E-858D-248977F04D31@uwaterloo.ca","threadId":"20319","inReplyTo":null,"subject":"[PATCHv4] Add Gitweb support for XZ compressed snapshots","fromName":"Mark A Rada","fromEmail":"marada@uwaterloo.ca","sentAt":"2009-07-31T13:32:38Z","receivedAt":"2009-07-31T13:32:38Z","isPatch":false,"sender":{"key":"marada@uwaterloo.ca","avatar":"https://avatars.githubusercontent.com/u/38430?v=4"},"body":"Ok, so I got a good nights sleep now, and reviewed the results of my\nbenchmarks to make sure they were consistent (turns out I had the\narchive sizes in the wrong order for the XZ repository tests).\n\nI also reworded a number of things and added a conclusion to the\nbenchmarks.\n\nLet me know what you think.\n\n\n--\nMark A Rada (ferrous26)\nmarada@uwaterloo.ca\n\n\n------->8--------------\nFrom: Mark Rada <marada@uwaterloo.ca>\nDate: Thu, 30 Jul 2009 08:56:42 -0400\nSubject: [PATCH] Add Gitweb support for XZ compressed snapshots\n\nThe XZ compression format uses the LZMA2 compression algorithm, which\noften yields higher compression ratios than both GZip and BZip2 at the\ncost of using more CPU time and RAM. Though, while XZ is the slowest\nfor compression, it is much faster than BZip2 for decompression, almost\ncomparable to GZip (see benchmarks below).\n\nYou can enable XZ compressed snapshots by adding 'txz' to the list of\ndefault options for snapshots in your $GITWEB_CONFIG.\n\nI did some simple benchmarks, starting with an already tarballed\narchive of the repos listed below. Memory usage seemed to be consistent\nfor any given algorithm at their default compression level. Timings were\ngathered using the `time' command.\n\nCPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)\nVirtual Memory Usage\n\tGZip: 4152K\tBZip2: 13352K\tXZ: 102M\n\nLinux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259) \t349M\ngzip \t23.70s user    \t0.47s system  \t99% cpu  24.227 total \t76M\ngunzip \t3.74s user     \t0.74s system  \t94% cpu  4.741 total\nbzip2 \t130.96s user   \t0.53s system  \t99% cpu  2:11.97 total \t59M\nbunzip2 31.05s user    \t1.02s system  \t99% cpu  32.355 total\nxz \t448.78s user \t0.91s system  \t99% cpu  7:31.28 total \t51M\nunxz \t7.67s user     \t0.80s system  \t98% cpu  8.607 total\n\nGit (0a53e9ddeaddad63ad106860237bbf53411d11a7) \t\t\t11M\ngzip \t0.77s user \t0.03s system \t99% cpu  0.792 total \t2.5M\ngunzip \t0.12s user \t0.02s system \t98% cpu  0.142 total\nbzip2 \t3.42s user \t0.02s system \t99% cpu  3.454 total \t2.1M\nbunzip2 0.95s user \t0.03s system \t99% cpu  0.984 total\nxz \t12.88s user \t0.14s system \t98% cpu  13.239 total \t1.9M\nunxz \t0.27s user \t0.03s system \t99% cpu  0.298 total\n\nXZ (669413bb2db954bbfde3c4542fddbbab53891eb4) \t\t\t1.8M\ngzip  \t0.12s user \t0.00s system \t95% cpu  0.132 total \t442K\ngunzip \t0.02s user \t0.00s system \t97% cpu  0.027 total\nbzip2  \t1.28s user \t0.01s system \t99% cpu  1.298 total \t363K\nbunzip2 0.15s user \t0.01s system \t100% cpu 0.157 total\nxz \t1.62s user \t0.03s system \t99% cpu  1.652 total \t347K\nunxz   \t0.05s user \t0.00s system \t99% cpu  0.058 total\n\nPurely from a time and memory perspective, nothing compares to GZip in\neach of the three tests. Though, if you have an average upload speed of\n20KB/s, it would take ~400 seconds longer to transfer the kernel  \nsnapshot\nthat was BZip2 compressed than it would the XZ compressed snapshot, the\ntransfer time difference is even greater when compared to the GZip\ncompressed snapshot. The wall clock time savings are relatively the same\nfor all test cases, but less dramatic for the smaller repositories.\n\nThe obvious downside for XZ compressed snapshots is the large CPU and\nmemory load put on the server to actualy generate the snapshot. Though  \nXZ\nwill eventually have good threading support, and I suspect then that the\nwall clock time for making an XZ compressed snapshot would go down\nconsiderably if the server had a beefy multi-core CPU.\n\nI have not enabled XZ compression by default because the current default\nis GZip, and XZ is only really competitive with BZip2. Also, the XZ  \nformat\nis still fairly new (the format was declared stable about 6 months ago),\nand there have been no \"stable\" releases of the utils yet.\n\nSigned-off-by: Mark Rada <marada@uwaterloo.ca>\n---\n  gitweb/gitweb.perl |    8 ++++++++\n  1 files changed, 8 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 7fbd5ff..3398163 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -176,6 +176,13 @@ our %known_snapshot_formats = (\n  \t\t'format' => 'tar',\n  \t\t'compressor' => ['bzip2']},\n\n+\t'txz' => {\n+\t\t'display' => 'tar.xz',\n+\t\t'type' => 'application/x-xz',\n+\t\t'suffix' => '.tar.xz',\n+\t\t'format' => 'tar',\n+\t\t'compressor' => ['xz']},\n+\n  \t'zip' => {\n  \t\t'display' => 'zip',\n  \t\t'type' => 'application/x-zip',\n@@ -188,6 +195,7 @@ our %known_snapshot_formats = (\n  our %known_snapshot_format_aliases = (\n  \t'gzip'  => 'tgz',\n  \t'bzip2' => 'tbz2',\n+\t'xz'    => 'txz',\n\n  \t# backward compatibility: legacy gitweb config support\n  \t'x-gzip' => undef, 'gz' => undef,\n-- \n1.6.4\n"},{"id":"119296","messageId":"4A7332F3.50908@eaglescrag.net","threadId":"20319","inReplyTo":"828BD9FC-1238-4B2E-858D-248977F04D31@uwaterloo.ca","subject":"Re: [PATCHv4] Add Gitweb support for XZ compressed snapshots","fromName":"J.H.","fromEmail":"warthog19@eaglescrag.net","sentAt":"2009-07-31T18:07:47Z","receivedAt":"2009-07-31T18:07:47Z","isPatch":false,"sender":{"key":"warthog19@eaglescrag.net","avatar":null},"body":"Mark A Rada wrote:\n> Ok, so I got a good nights sleep now, and reviewed the results of my\n> benchmarks to make sure they were consistent (turns out I had the\n> archive sizes in the wrong order for the XZ repository tests).\n> \n> I also reworded a number of things and added a conclusion to the\n> benchmarks.\n> \n> Let me know what you think.\n> \n> \n> -- \n> Mark A Rada (ferrous26)\n> marada@uwaterloo.ca\n> \n> \n> ------->8--------------\n> From: Mark Rada <marada@uwaterloo.ca>\n> Date: Thu, 30 Jul 2009 08:56:42 -0400\n> Subject: [PATCH] Add Gitweb support for XZ compressed snapshots\n> \n> The XZ compression format uses the LZMA2 compression algorithm, which\n> often yields higher compression ratios than both GZip and BZip2 at the\n> cost of using more CPU time and RAM. Though, while XZ is the slowest\n> for compression, it is much faster than BZip2 for decompression, almost\n> comparable to GZip (see benchmarks below).\n> \n> You can enable XZ compressed snapshots by adding 'txz' to the list of\n> default options for snapshots in your $GITWEB_CONFIG.\n> \n> I did some simple benchmarks, starting with an already tarballed\n> archive of the repos listed below. Memory usage seemed to be consistent\n> for any given algorithm at their default compression level. Timings were\n> gathered using the `time' command.\n> \n> CPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)\n> Virtual Memory Usage\n>     GZip: 4152K    BZip2: 13352K    XZ: 102M\n> \n> Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)     349M\n> gzip     23.70s user        0.47s system      99% cpu  24.227 total     76M\n> gunzip     3.74s user         0.74s system      94% cpu  4.741 total\n> bzip2     130.96s user       0.53s system      99% cpu  2:11.97 \n> total     59M\n> bunzip2 31.05s user        1.02s system      99% cpu  32.355 total\n> xz     448.78s user     0.91s system      99% cpu  7:31.28 total     51M\n> unxz     7.67s user         0.80s system      98% cpu  8.607 total\n> \n> Git (0a53e9ddeaddad63ad106860237bbf53411d11a7)             11M\n> gzip     0.77s user     0.03s system     99% cpu  0.792 total     2.5M\n> gunzip     0.12s user     0.02s system     98% cpu  0.142 total\n> bzip2     3.42s user     0.02s system     99% cpu  3.454 total     2.1M\n> bunzip2 0.95s user     0.03s system     99% cpu  0.984 total\n> xz     12.88s user     0.14s system     98% cpu  13.239 total     1.9M\n> unxz     0.27s user     0.03s system     99% cpu  0.298 total\n> \n> XZ (669413bb2db954bbfde3c4542fddbbab53891eb4)             1.8M\n> gzip      0.12s user     0.00s system     95% cpu  0.132 total     442K\n> gunzip     0.02s user     0.00s system     97% cpu  0.027 total\n> bzip2      1.28s user     0.01s system     99% cpu  1.298 total     363K\n> bunzip2 0.15s user     0.01s system     100% cpu 0.157 total\n> xz     1.62s user     0.03s system     99% cpu  1.652 total     347K\n> unxz       0.05s user     0.00s system     99% cpu  0.058 total\n> \n> Purely from a time and memory perspective, nothing compares to GZip in\n> each of the three tests. Though, if you have an average upload speed of\n> 20KB/s, it would take ~400 seconds longer to transfer the kernel snapshot\n> that was BZip2 compressed than it would the XZ compressed snapshot, the\n> transfer time difference is even greater when compared to the GZip\n> compressed snapshot. The wall clock time savings are relatively the same\n> for all test cases, but less dramatic for the smaller repositories.\n> \n> The obvious downside for XZ compressed snapshots is the large CPU and\n> memory load put on the server to actualy generate the snapshot. Though XZ\n> will eventually have good threading support, and I suspect then that the\n> wall clock time for making an XZ compressed snapshot would go down\n> considerably if the server had a beefy multi-core CPU.\n> \n> I have not enabled XZ compression by default because the current default\n> is GZip, and XZ is only really competitive with BZip2. Also, the XZ format\n> is still fairly new (the format was declared stable about 6 months ago),\n> and there have been no \"stable\" releases of the utils yet.\n\nOne thing that would concern me greatly, is not so much the CPU time \n(though that's a *huge* change in comparison to gz) but the memory \nusage.  Where gzip and bzip2 are chewing 4M and 13M respectively, xz \nchews 102M.  From a 'beefy' server perspective chewing up that much \nmemory per snapshot for that long could be bad.  This is likely \nsomething that needs to have some sort of enable/disable switch if it's \ngoing to be included.\n\nGood analysis btw, though I'll admit it makes me leary of something \ndynamically generating xz compressed files.\n\n- John 'Warthog9' Hawley\n"},{"id":"119299","messageId":"m38wi4so66.fsf@localhost.localdomain","threadId":"20319","inReplyTo":"4A7332F3.50908@eaglescrag.net","subject":"Re: [PATCHv4] Add Gitweb support for XZ compressed snapshots","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-07-31T18:27:50Z","receivedAt":"2009-07-31T18:27:50Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"J.H.\" <warthog19@eaglescrag.net> writes:\n> Mark A Rada wrote:\n\n> > I have not enabled XZ compression by default because the current\n> > default is GZip, and XZ is only really competitive with\n> > BZip2. Also, the XZ format is still fairly new (the format was\n> > declared stable about 6 months ago), and there have been no\n> > \"stable\" releases of the utils yet.\n> \n> One thing that would concern me greatly, is not so much the CPU time\n> (though that's a *huge* change in comparison to gz) but the memory\n> usage.  Where gzip and bzip2 are chewing 4M and 13M respectively, xz\n> chews 102M.  From a 'beefy' server perspective chewing up that much\n> memory per snapshot for that long could be bad.  This is likely\n> something that needs to have some sort of enable/disable switch if\n> it's going to be included.\n\nFirst, by default project specific override for 'snapshot' feature is\ndisabled, and only 'tgz' ('.tar.gz') snapshot links are generated.\n\nSecond, if you want to enable project specific override for 'snapshot'\nfeature, you can still control what snapshot formats (among 'tgz',\n'tbz2', 'zip' and now 'txz') from $GITWEB_CONFIG by simply deleting\nformats you don't want to be used from %known_snapshot_formats.\n\nFor example\n\n  delete $known_snapshot_formats{'txz'};\n\n> \n> Good analysis btw, though I'll admit it makes me leary of something\n> dynamically generating xz compressed files.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"119301","messageId":"m34osssn7o.fsf@localhost.localdomain","threadId":"20319","inReplyTo":"828BD9FC-1238-4B2E-858D-248977F04D31@uwaterloo.ca","subject":"Re: [PATCHv4] Add Gitweb support for XZ compressed snapshots","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-07-31T18:48:29Z","receivedAt":"2009-07-31T18:48:29Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Mark A Rada <marada@uwaterloo.ca> writes:\n\n> Ok, so I got a good nights sleep now, and reviewed the results of my\n> benchmarks to make sure they were consistent (turns out I had the\n> archive sizes in the wrong order for the XZ repository tests).\n> \n> I also reworded a number of things and added a conclusion to the\n> benchmarks.\n> \n> Let me know what you think.\n\nWell separated change.  Very detailed commit message; that's good!.\n\n[...]\n> Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259) \t349M\n> gzip \t23.70s user    \t0.47s system  \t99% cpu  24.227 total \t76M\n> gunzip \t3.74s user     \t0.74s system  \t94% cpu  4.741 total\n> bzip2 \t130.96s user   \t0.53s system  \t99% cpu  2:11.97 total \t59M\n> bunzip2 31.05s user    \t1.02s system  \t99% cpu  32.355 total\n> xz \t448.78s user \t0.91s system  \t99% cpu  7:31.28 total \t51M\n> unxz \t7.67s user     \t0.80s system  \t98% cpu  8.607 total\n> \n> Git (0a53e9ddeaddad63ad106860237bbf53411d11a7) \t\t\t11M\n> gzip \t0.77s user \t0.03s system \t99% cpu  0.792 total \t2.5M\n> gunzip \t0.12s user \t0.02s system \t98% cpu  0.142 total\n> bzip2 \t3.42s user \t0.02s system \t99% cpu  3.454 total \t2.1M\n> bunzip2 0.95s user \t0.03s system \t99% cpu  0.984 total\n> xz \t12.88s user \t0.14s system \t98% cpu  13.239 total \t1.9M\n> unxz \t0.27s user \t0.03s system \t99% cpu  0.298 total\n> \n> XZ (669413bb2db954bbfde3c4542fddbbab53891eb4) \t\t\t1.8M\n> gzip  \t0.12s user \t0.00s system \t95% cpu  0.132 total \t442K\n> gunzip \t0.02s user \t0.00s system \t97% cpu  0.027 total\n> bzip2  \t1.28s user \t0.01s system \t99% cpu  1.298 total \t363K\n> bunzip2 0.15s user \t0.01s system \t100% cpu 0.157 total\n> xz \t1.62s user \t0.03s system \t99% cpu  1.652 total \t347K\n> unxz   \t0.05s user \t0.00s system \t99% cpu  0.058 total\n\nNote that for me the above results are not aligned in table.\nThis is a cosmetic issue.\n\n> Purely from a time and memory perspective, nothing compares to GZip in\n> each of the three tests. Though, if you have an average upload speed of\n> 20KB/s, it would take ~400 seconds longer to transfer the kernel\n> snapshot\n> that was BZip2 compressed than it would the XZ compressed snapshot, the\n> transfer time difference is even greater when compared to the GZip\n> compressed snapshot. The wall clock time savings are relatively the same\n> for all test cases, but less dramatic for the smaller repositories.\n> \n> The obvious downside for XZ compressed snapshots is the large CPU and\n> memory load put on the server to actualy generate the snapshot. Though\n> XZ\n> will eventually have good threading support, and I suspect then that the\n> wall clock time for making an XZ compressed snapshot would go down\n> considerably if the server had a beefy multi-core CPU.\n> \n> I have not enabled XZ compression by default because the current default\n> is GZip, and XZ is only really competitive with BZip2. Also, the XZ\n> format\n> is still fairly new (the format was declared stable about 6 months ago),\n> and there have been no \"stable\" releases of the utils yet.\n\nThose above three paragraphs are strangely wrapped, with single word\nfor a whole line ('snapshot', 'XZ', 'format').  This is a cosmetic issue.\n\n> \n> Signed-off-by: Mark Rada <marada@uwaterloo.ca>\n\n-- \nJakub Narebski\nGit User's Survey 2009: http://tinyurl.com/GitSurvey2009\n"}]}