{"thread":{"id":"20324","subject":"[PATCHv5] Add Gitweb support for XZ compressed snapshots","startedAt":"2009-08-01T00:40:33Z","lastAt":"2009-08-03T17:26:11Z","messageCount":10,"participants":["Mark A Rada","J.H.","demerphq","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"119311","messageId":"B05AF655-7430-420A-A22E-389601558B0D@uwaterloo.ca","threadId":"20324","inReplyTo":null,"subject":"[PATCHv5] Add Gitweb support for XZ compressed snapshots","fromName":"Mark A Rada","fromEmail":"marada@uwaterloo.ca","sentAt":"2009-08-01T00:40:33Z","receivedAt":"2009-08-01T00:40:33Z","isPatch":false,"sender":{"key":"marada@uwaterloo.ca","avatar":"https://avatars.githubusercontent.com/u/38430?v=4"},"body":"\n >Note that for me the above results are not aligned in table.\n >This is a cosmetic issue.\n\nThe table formatting issue was due to a bad habit of mixing tabs and  \nspaces,\nI decided to go with spaces this time :)\n\n\n >One thing that would concern me greatly, is not so much the CPU time  \n(though that's a *huge* change in comparison to gz) but the memory  \nusage.  Where gzip and bzip2 are chewing 4M and 13M respectively, xz  \nchews 102M.  >From a 'beefy' server perspective chewing up that much  \nmemory per snapshot for that long could be bad.  This is likely  \nsomething that needs to have some sort of enable/disable switch if  \nit's going to be included.\n\nTrue, and there are two solutions I can think of for this \"problem\".\n\n\t1. My tests were at the default compression level, the XZ documentation\n\tsays that at lower levels you will get resource usage and compression\n\tratios that are comparable to BZip2. However, I'm not sure where you\n\twould change the compression level variable for this (globally for the\n\tsystem, somewhere in $GITWEB_CONFIG, a git config variable). Does\n\tsomeone know the correct answer here?\n\n\t2. Implement snapshot caching for Gitweb.\n\n\n--\nMark A Rada (ferrous26)\nmarada@uwaterloo.ca\n\n\nFrom: Mark Rada <marada@uwaterloo.ca>\nDate: Thu, 30 Jul 2009 08:56:42 -0400\nSubject: [PATCH] Add Gitweb support for XZ compressed snapshots\n\nThe XZ compression format uses the LZMA2 compression algorithm, which\noften yields higher compression ratios than both GZip and BZip2 at the\ncost of using more CPU time and RAM. Though, while XZ is the slowest\nfor compression, it is much faster than BZip2 for decompression, almost\ncomparable to GZip (see benchmarks below).\n\nYou can enable XZ compressed snapshots by adding 'txz' to the list of\ndefault options for snapshots in your $GITWEB_CONFIG.\n\nI did some simple benchmarks, starting with an already tarballed\narchive of the repos listed below. Memory usage seemed to be consistent\nfor any given algorithm at their default compression level. Timings were\ngathered using the `time' command.\n\nCPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)\nVirtual Memory Usage\n\tGZip: 4152K\tBZip2: 13352K\tXZ: 102M\n\nLinux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)   349M\ngzip    23.70s user   0.47s system  99% cpu   24.227 total    76M\ngunzip  3.74s user    0.74s system  94% cpu   4.741 total\nbzip2   130.96s user  0.53s system  99% cpu   2:11.97 total   59M\nbunzip2 31.05s user   1.02s system  99% cpu   32.355 total\nxz      448.78s user  0.91s system  99% cpu   7:31.28 total   51M\nunxz    7.67s user    0.80s system  98% cpu   8.607 total\n\nGit (0a53e9ddeaddad63ad106860237bbf53411d11a7)                11M\ngzip    0.77s user    0.03s system  99% cpu   0.792 total     2.5M\ngunzip  0.12s user    0.02s system  98% cpu   0.142 total\nbzip2   3.42s user    0.02s system  99% cpu   3.454 total     2.1M\nbunzip2 0.95s user    0.03s system  99% cpu   0.984 total\nxz      12.88s user   0.14s system  98% cpu   13.239 total    1.9M\nunxz    0.27s user    0.03s system  99% cpu   0.298 total\n\nXZ (669413bb2db954bbfde3c4542fddbbab53891eb4)                 1.8M\ngzip    0.12s user    0.00s system  95% cpu   0.132 total     442K\ngunzip  0.02s user    0.00s system  97% cpu   0.027 tota\nbzip2   1.28s user    0.01s system  99% cpu   1.298 total     363K\nbunzip2 0.15s user    0.01s system  100% cpu  0.157 total\nxz      1.62s user    0.03s system  99% cpu   1.652 total     347K\nunxz    0.05s user    0.00s system  99% cpu   0.058 total\n\nPurely from a time and memory perspective, nothing compares to GZip in\neach of the three tests. Though, if you have an average upload speed of\n20KB/s, it would take ~400 seconds longer to transfer the kernel\nsnapshot that was BZip2 compressed than it would the XZ compressed\nsnapshot, the transfer time difference is even greater when compared to\nthe GZip compressed snapshot. The wall clock time savings are\nrelatively the same for all test cases, but less dramatic for the\nsmaller repositories.\n\nThe obvious downside for XZ compressed snapshots is the large CPU and\nmemory load put on the server to actualy generate the snapshot. Though\nXZ will eventually have good threading support, and I suspect then that\nthe wall clock time for making an XZ compressed snapshot would go down\nconsiderably if the server had a beefy multi-core CPU.\n\nI have not enabled XZ compression by default because the current default\nis GZip, and XZ is only really competitive with BZip2. Also, the XZ\nformat is still fairly new (the format was declared stable about 6\nmonths ago), and there have been no \"stable\" releases of the utils yet.\n\nSigned-off-by: Mark Rada <marada@uwaterloo.ca>\n---\n  gitweb/gitweb.perl |    8 ++++++++\n  1 files changed, 8 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 7fbd5ff..3398163 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -176,6 +176,13 @@ our %known_snapshot_formats = (\n  \t\t'format' => 'tar',\n  \t\t'compressor' => ['bzip2']},\n\n+\t'txz' => {\n+\t\t'display' => 'tar.xz',\n+\t\t'type' => 'application/x-xz',\n+\t\t'suffix' => '.tar.xz',\n+\t\t'format' => 'tar',\n+\t\t'compressor' => ['xz']},\n+\n  \t'zip' => {\n  \t\t'display' => 'zip',\n  \t\t'type' => 'application/x-zip',\n@@ -188,6 +195,7 @@ our %known_snapshot_formats = (\n  our %known_snapshot_format_aliases = (\n  \t'gzip'  => 'tgz',\n  \t'bzip2' => 'tbz2',\n+\t'xz'    => 'txz',\n\n  \t# backward compatibility: legacy gitweb config support\n  \t'x-gzip' => undef, 'gz' => undef,\n-- \n1.6.4\n"},{"id":"119312","messageId":"4A739087.1090301@eaglescrag.net","threadId":"20324","inReplyTo":"B05AF655-7430-420A-A22E-389601558B0D@uwaterloo.ca","subject":"Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots","fromName":"J.H.","fromEmail":"warthog19@eaglescrag.net","sentAt":"2009-08-01T00:47:03Z","receivedAt":"2009-08-01T00:47:03Z","isPatch":false,"sender":{"key":"warthog19@eaglescrag.net","avatar":null},"body":"Mark A Rada wrote:\n> \n>  >Note that for me the above results are not aligned in table.\n>  >This is a cosmetic issue.\n> \n> The table formatting issue was due to a bad habit of mixing tabs and \n> spaces,\n> I decided to go with spaces this time :)\n> \n> \n>  >One thing that would concern me greatly, is not so much the CPU time \n> (though that's a *huge* change in comparison to gz) but the memory \n> usage.  Where gzip and bzip2 are chewing 4M and 13M respectively, xz \n> chews 102M.  >From a 'beefy' server perspective chewing up that much \n> memory per snapshot for that long could be bad.  This is likely \n> something that needs to have some sort of enable/disable switch if it's \n> going to be included.\n> \n> True, and there are two solutions I can think of for this \"problem\".\n> \n>     1. My tests were at the default compression level, the XZ documentation\n>     says that at lower levels you will get resource usage and compression\n>     ratios that are comparable to BZip2. However, I'm not sure where you\n>     would change the compression level variable for this (globally for the\n>     system, somewhere in $GITWEB_CONFIG, a git config variable). Does\n>     someone know the correct answer here?\n\nWell you can always call xz with -[1-9] to change the compression level \n(same as gzip and bzip2) though I think a full disabling would be 'more' \npreferable, though I'm not sure I like Jakub's suggestion of just \ndeleting it after the fact, it would work.\n\n>     2. Implement snapshot caching for Gitweb.\n\nI think it's slightly broken in my version (binary files don't work \nright apparently, it's on my todo list to fix in my upcoming update) but \nboth Lea and I have done this long ago (caching layers in Gitweb), which \nwould be an acceptable workaround for this, create once and serve many - \nthough this has the downside of trading cpu for diskspace.  At least \nwith xz there's less diskspace used ;-)\n\nI think more my concern is more what's enabled by default, and since xz \nis still new (as was pointed out) it's probably worth only enabling if \nthe admin selects it to be enabled.\n\n- John 'Warthog9' Hawley\n\n\n> -- \n> Mark A Rada (ferrous26)\n> marada@uwaterloo.ca\n> \n> \n> From: Mark Rada <marada@uwaterloo.ca>\n> Date: Thu, 30 Jul 2009 08:56:42 -0400\n> Subject: [PATCH] Add Gitweb support for XZ compressed snapshots\n> \n> The XZ compression format uses the LZMA2 compression algorithm, which\n> often yields higher compression ratios than both GZip and BZip2 at the\n> cost of using more CPU time and RAM. Though, while XZ is the slowest\n> for compression, it is much faster than BZip2 for decompression, almost\n> comparable to GZip (see benchmarks below).\n> \n> You can enable XZ compressed snapshots by adding 'txz' to the list of\n> default options for snapshots in your $GITWEB_CONFIG.\n> \n> I did some simple benchmarks, starting with an already tarballed\n> archive of the repos listed below. Memory usage seemed to be consistent\n> for any given algorithm at their default compression level. Timings were\n> gathered using the `time' command.\n> \n> CPU: AMD Sempron 3400+ (1 core @ 1.8GHz with 256K L2 cache)\n> Virtual Memory Usage\n>     GZip: 4152K    BZip2: 13352K    XZ: 102M\n> \n> Linux 2.6 series (f5886c7f96f2542382d3a983c5f13e03d7fc5259)   349M\n> gzip    23.70s user   0.47s system  99% cpu   24.227 total    76M\n> gunzip  3.74s user    0.74s system  94% cpu   4.741 total\n> bzip2   130.96s user  0.53s system  99% cpu   2:11.97 total   59M\n> bunzip2 31.05s user   1.02s system  99% cpu   32.355 total\n> xz      448.78s user  0.91s system  99% cpu   7:31.28 total   51M\n> unxz    7.67s user    0.80s system  98% cpu   8.607 total\n> \n> Git (0a53e9ddeaddad63ad106860237bbf53411d11a7)                11M\n> gzip    0.77s user    0.03s system  99% cpu   0.792 total     2.5M\n> gunzip  0.12s user    0.02s system  98% cpu   0.142 total\n> bzip2   3.42s user    0.02s system  99% cpu   3.454 total     2.1M\n> bunzip2 0.95s user    0.03s system  99% cpu   0.984 total\n> xz      12.88s user   0.14s system  98% cpu   13.239 total    1.9M\n> unxz    0.27s user    0.03s system  99% cpu   0.298 total\n> \n> XZ (669413bb2db954bbfde3c4542fddbbab53891eb4)                 1.8M\n> gzip    0.12s user    0.00s system  95% cpu   0.132 total     442K\n> gunzip  0.02s user    0.00s system  97% cpu   0.027 tota\n> bzip2   1.28s user    0.01s system  99% cpu   1.298 total     363K\n> bunzip2 0.15s user    0.01s system  100% cpu  0.157 total\n> xz      1.62s user    0.03s system  99% cpu   1.652 total     347K\n> unxz    0.05s user    0.00s system  99% cpu   0.058 total\n> \n> Purely from a time and memory perspective, nothing compares to GZip in\n> each of the three tests. Though, if you have an average upload speed of\n> 20KB/s, it would take ~400 seconds longer to transfer the kernel\n> snapshot that was BZip2 compressed than it would the XZ compressed\n> snapshot, the transfer time difference is even greater when compared to\n> the GZip compressed snapshot. The wall clock time savings are\n> relatively the same for all test cases, but less dramatic for the\n> smaller repositories.\n> \n> The obvious downside for XZ compressed snapshots is the large CPU and\n> memory load put on the server to actualy generate the snapshot. Though\n> XZ will eventually have good threading support, and I suspect then that\n> the wall clock time for making an XZ compressed snapshot would go down\n> considerably if the server had a beefy multi-core CPU.\n> \n> I have not enabled XZ compression by default because the current default\n> is GZip, and XZ is only really competitive with BZip2. Also, the XZ\n> format is still fairly new (the format was declared stable about 6\n> months ago), and there have been no \"stable\" releases of the utils yet.\n> \n> Signed-off-by: Mark Rada <marada@uwaterloo.ca>\n> ---\n>  gitweb/gitweb.perl |    8 ++++++++\n>  1 files changed, 8 insertions(+), 0 deletions(-)\n> \n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 7fbd5ff..3398163 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -176,6 +176,13 @@ our %known_snapshot_formats = (\n>          'format' => 'tar',\n>          'compressor' => ['bzip2']},\n> \n> +    'txz' => {\n> +        'display' => 'tar.xz',\n> +        'type' => 'application/x-xz',\n> +        'suffix' => '.tar.xz',\n> +        'format' => 'tar',\n> +        'compressor' => ['xz']},\n> +\n>      'zip' => {\n>          'display' => 'zip',\n>          'type' => 'application/x-zip',\n> @@ -188,6 +195,7 @@ our %known_snapshot_formats = (\n>  our %known_snapshot_format_aliases = (\n>      'gzip'  => 'tgz',\n>      'bzip2' => 'tbz2',\n> +    'xz'    => 'txz',\n> \n>      # backward compatibility: legacy gitweb config support\n>      'x-gzip' => undef, 'gz' => undef,\n"},{"id":"119319","messageId":"9b18b3110908010112va6e2ceap727c4129a054ebda@mail.gmail.com","threadId":"20324","inReplyTo":"4A739087.1090301@eaglescrag.net","subject":"Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2009-08-01T08:12:12Z","receivedAt":"2009-08-01T08:12:12Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"2009/8/1 J.H. <warthog19@eaglescrag.net>:\n> Mark A Rada wrote:\n>>\n>>  >Note that for me the above results are not aligned in table.\n>>  >This is a cosmetic issue.\n>>\n>> The table formatting issue was due to a bad habit of mixing tabs and\n>> spaces,\n>> I decided to go with spaces this time :)\n>>\n>>\n>>  >One thing that would concern me greatly, is not so much the CPU time\n>> (though that's a *huge* change in comparison to gz) but the memory usage.\n>>  Where gzip and bzip2 are chewing 4M and 13M respectively, xz chews 102M.\n>>  >From a 'beefy' server perspective chewing up that much memory per snapshot\n>> for that long could be bad.  This is likely something that needs to have\n>> some sort of enable/disable switch if it's going to be included.\n>>\n>> True, and there are two solutions I can think of for this \"problem\".\n>>\n>>    1. My tests were at the default compression level, the XZ documentation\n>>    says that at lower levels you will get resource usage and compression\n>>    ratios that are comparable to BZip2. However, I'm not sure where you\n>>    would change the compression level variable for this (globally for the\n>>    system, somewhere in $GITWEB_CONFIG, a git config variable). Does\n>>    someone know the correct answer here?\n>\n> Well you can always call xz with -[1-9] to change the compression level\n> (same as gzip and bzip2) though I think a full disabling would be 'more'\n> preferable, though I'm not sure I like Jakub's suggestion of just deleting\n> it after the fact, it would work.\n>\n>>    2. Implement snapshot caching for Gitweb.\n>\n> I think it's slightly broken in my version (binary files don't work right\n> apparently, it's on my todo list to fix in my upcoming update) but both Lea\n> and I have done this long ago (caching layers in Gitweb), which would be an\n> acceptable workaround for this, create once and serve many - though this has\n> the downside of trading cpu for diskspace.  At least with xz there's less\n> diskspace used ;-)\n>\n> I think more my concern is more what's enabled by default, and since xz is\n> still new (as was pointed out) it's probably worth only enabling if the\n> admin selects it to be enabled.\n\nFWIW the perl project ripped out all the snapshot generation logic\nfrom gitweb, and replaced it with a tool that generates snapshots\ncorrectly for our requirements (if the build process needs additional\nfiles /currently/ git-archive does not support adding them), this\nincludes a disk level cache for the snapshots since creating the tar,\nadding the additional files, then gziping is quite slow.\n\nIf its interesting to people I can post it and the other changes here,\nalthough its not a \"nice\" change, as I literally ripped out the\nexisting code.\n\ncheers,\nYves\n\n\n\n\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"119320","messageId":"m3zlakq7bt.fsf@localhost.localdomain","threadId":"20324","inReplyTo":"4A739087.1090301@eaglescrag.net","subject":"Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-01T08:14:35Z","receivedAt":"2009-08-01T08:14:35Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"J.H.\" <warthog19@eaglescrag.net> writes:\n\n> Well you can always call xz with -[1-9] to change the compression\n> level (same as gzip and bzip2) though I think a full disabling would\n> be 'more' preferable, though I'm not sure I like Jakub's suggestion of\n> just deleting it after the fact, it would work.\n[...]\n\nThe problem is that 'keys %known_snapshot_formats' serves also as list\nof allowed snapshot formats, if project specific override is enabled.\nWe can add another optional flag ('disabled' => 1) if you don't want\nto delete from %known_snapshot_formats in $GITWEB_CONFIG, though I\ndon't know if it is worth it.  Anyway such mechanism can be added, and\nIMHO should be added, in a separate commit.\n\n> I think more my concern is more what's enabled by default, and since\n> xz is still new (as was pointed out) it's probably worth only enabling\n> if the admin selects it to be enabled.\n\nBy default (i.e. as in gitweb in git.git) 'snapshot' feature has\ndisabled projects specific override.  Which means only 'tgz' snapshot\nis enabled / used.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"119322","messageId":"m3vdl7rjev.fsf@localhost.localdomain","threadId":"20324","inReplyTo":"9b18b3110908010112va6e2ceap727c4129a054ebda@mail.gmail.com","subject":"Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-01T09:08:48Z","receivedAt":"2009-08-01T09:08:48Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"demerphq <demerphq@gmail.com> writes:\n\n> FWIW the perl project ripped out all the snapshot generation logic\n> from gitweb, and replaced it with a tool that generates snapshots\n> correctly for our requirements (if the build process needs additional\n> files /currently/ git-archive does not support adding them), this\n> includes a disk level cache for the snapshots since creating the tar,\n> adding the additional files, then gziping is quite slow.\n> \n> If its interesting to people I can post it and the other changes here,\n> although its not a \"nice\" change, as I literally ripped out the\n> existing code.\n\nDo you mean gitweb serving http://perl5.git.perl.org/ ?\n\nWell, at least publish this version of gitweb somewhere (snapshot or\nbetter git repository), and add it to the list of gitweb forks at\nhttp://git.or.cz/gitwiki/Gitweb wiki page.\n\nSidenote: snapshot support appeared, as far as I know, first at fork of\ngitweb that had been used by XMMS2, but the snapshot was generated by\na separate snapshot.cgi script... written in Python.  (Now they use\nstock (if old) gitweb, I think.)\n\nP.S. Perhaps you or other Perl Mongers would be interested in creating\nyet another git web interface, in Perl, but contrary to (current)\ngitweb modular and using more of CPAN modules?  For example convert\nSVN::Web or Insurrection (or other Perl module) from Subversion (or\nother SCM) to Git (and rename it to Git::Web, or ???).\n\nP.P.S. Could you per chance post announcement of Git User's Survey 2009\non http://perl5.git.perl.org, just like it is done on http://repo.or.cz\nand http://git.kernel.org?  TIA.\n\n-- \nJakub Narebski\nShadeHawk on #git\n\nGit User's Survey 2009\nhttp://tinyurl.com/GitSurvey2009\n"},{"id":"119324","messageId":"9b18b3110908010313x38991dffx3e4a2f2a5c5fff9f@mail.gmail.com","threadId":"20324","inReplyTo":"m3vdl7rjev.fsf@localhost.localdomain","subject":"Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2009-08-01T10:13:52Z","receivedAt":"2009-08-01T10:13:52Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"2009/8/1 Jakub Narebski <jnareb@gmail.com>:\n> demerphq <demerphq@gmail.com> writes:\n>\n>> FWIW the perl project ripped out all the snapshot generation logic\n>> from gitweb, and replaced it with a tool that generates snapshots\n>> correctly for our requirements (if the build process needs additional\n>> files /currently/ git-archive does not support adding them), this\n>> includes a disk level cache for the snapshots since creating the tar,\n>> adding the additional files, then gziping is quite slow.\n>>\n>> If its interesting to people I can post it and the other changes here,\n>> although its not a \"nice\" change, as I literally ripped out the\n>> existing code.\n>\n> Do you mean gitweb serving http://perl5.git.perl.org/ ?\n\nYes. I updated it to a relatively recent version just the other week.\n\n> Well, at least publish this version of gitweb somewhere (snapshot or\n> better git repository), and add it to the list of gitweb forks at\n> http://git.or.cz/gitwiki/Gitweb wiki page.\n\nSure, we have been discussing doing that recently.\n\nI need to cleanup how I set up the repository for it tho.\n\nAlso probably there are one or two patches that should be pushed back to you.\n\n> Sidenote: snapshot support appeared, as far as I know, first at fork of\n> gitweb that had been used by XMMS2, but the snapshot was generated by\n> a separate snapshot.cgi script... written in Python.  (Now they use\n> stock (if old) gitweb, I think.)\n\nAh, I'm not generating it with a separate cgi script, I just replaced\nthe call to git-archive to be a call to my own tool.\n\n> P.S. Perhaps you or other Perl Mongers would be interested in creating\n> yet another git web interface, in Perl, but contrary to (current)\n> gitweb modular and using more of CPAN modules?  For example convert\n> SVN::Web or Insurrection (or other Perl module) from Subversion (or\n> other SCM) to Git (and rename it to Git::Web, or ???).\n\nI think that there is some interest in doing that and likely over time\nthere will be more. It has come up at least a few times in our\ndiscussion forums. I think you could easily make a public call for\nsupport on places like Perlmonks to get more action tho.\n\nA better git api toolset for perl would make things a lot easier.\nEspecially an XS one.\n\n>\n> P.P.S. Could you per chance post announcement of Git User's Survey 2009\n> on http://perl5.git.perl.org, just like it is done on http://repo.or.cz\n> and http://git.kernel.org?  TIA.\n\nSure.\n\ncheers,\nYves\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"119386","messageId":"200908030125.53534.jnareb@gmail.com","threadId":"20324","inReplyTo":"9b18b3110908010313x38991dffx3e4a2f2a5c5fff9f@mail.gmail.com","subject":"Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-02T23:25:51Z","receivedAt":"2009-08-02T23:25:51Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sat, 1 Aug 2009, demerphq wrote:\n> 2009/8/1 Jakub Narebski <jnareb@gmail.com>:\n>> demerphq <demerphq@gmail.com> writes:\n>>\n>>> FWIW the perl project ripped out all the snapshot generation logic\n>>> from gitweb, and replaced it with a tool that generates snapshots\n>>> correctly for our requirements (if the build process needs additional\n>>> files /currently/ git-archive does not support adding them), this\n>>> includes a disk level cache for the snapshots since creating the tar,\n>>> adding the additional files, then gziping is quite slow.\n>>>\n>>> If its interesting to people I can post it and the other changes here,\n>>> although its not a \"nice\" change, as I literally ripped out the\n>>> existing code.\n>>\n>> Do you mean gitweb serving http://perl5.git.perl.org/ ?\n> \n> Yes. I updated it to a relatively recent version just the other week.\n\nBy the way, if you don't mind, how do you keep your changes on top\nof changing target, i.e. on top of updating baseline to recent \nversions?\n\n> \n>> Well, at least publish this version of gitweb somewhere (snapshot or\n>> better git repository), and add it to the list of gitweb forks at\n>> http://git.or.cz/gitwiki/Gitweb wiki page.\n> \n> Sure, we have been discussing doing that recently.\n> \n> I need to cleanup how I set up the repository for it tho.\n\nO.K.\n\n> \n> Also probably there are one or two patches that should be pushed back\n> to you. \n\nWhat do those patches consist of?\n \n>> Sidenote: snapshot support appeared, as far as I know, first at fork of\n>> gitweb that had been used by XMMS2, but the snapshot was generated by\n>> a separate snapshot.cgi script... written in Python.  (Now they use\n>> stock (if old) gitweb, I think.)\n> \n> Ah, I'm not generating it with a separate cgi script, I just replaced\n> the call to git-archive to be a call to my own tool.\n\nO.K.  \n\nBy the way, this is one of the only two places where we have to\nuse quote_command and 3-argument form of open, instead of list for of\nmagic open.  It is because of pipeline, piping git-archive output\ninto compressor.  The other such place is git_object, to redirect error\nstream to /dev/null (to discard stderr).\n\nI thought about replacing it by list form of open somewhat (you can\nfind it in git mailing list archive), but it is not easy.  And IPC::Run\nis IMVHO a bit of overkill, especially for \"minimal dependencies\"\ngitweb (perhaps for Git::Web?).\n\n> \n>> P.S. Perhaps you or other Perl Mongers would be interested in creating\n>> yet another git web interface, in Perl, but contrary to (current)\n>> gitweb modular and using more of CPAN modules?  For example convert\n>> SVN::Web or Insurrection (or other Perl module) from Subversion (or\n>> other SCM) to Git (and rename it to Git::Web, or ???).\n> \n> I think that there is some interest in doing that and likely over time\n> there will be more. It has come up at least a few times in our\n> discussion forums. I think you could easily make a public call for\n> support on places like Perlmonks to get more action tho.\n\nThanks in advance.\n\n> \n> A better git api toolset for perl would make things a lot easier.\n> Especially an XS one.\n\nWell, in the beginnings of Git.pm there was XS interface (to \ngit-hash-object or git-cat-file), but it was dropped because it\nused -fPIC, which is not portable enough for Git (this is not present\nin perl/Git.pm history in git.git repository).  Git.pm was created\nby Petr 'Pasky' Baudis, author of Cogito porcelain, and creator and\nadmin of repo.or.cz (and also creator of first Git homepage).  \nBut he is not a Perl hacker (correct me if I am wrong, Pasky); both\nCogito and repo.or.cz duct tape (under name of Girocco) are written\nin bash, not in Perl.  Which can be seen for example by using \nError::Simple in Git.pm...\n\nAlso I don't think that Pasky has time for maintaining Git.pm, \nnevermind modernizing it / adding new features.\n\nThere was another attempt to make Perl interface for Git, more \nobject-oriented, by Lea Wiemann during GSoC 2008 project 'gitweb caching'.\nYou can see results in Lea repository at repo.or.cz[1], and also in \ngit.kernel.org gitweb code[2].\n\n[1]: http://repo.or.cz/w/git/gitweb-caching.git\n[2]: http://git.kernel.org/?p=git/warthog9/gitweb.git\n \n>>\n>> P.P.S. Could you per chance post announcement of Git User's Survey 2009\n>> on http://perl5.git.perl.org, just like it is done on http://repo.or.cz\n>> and http://git.kernel.org?  TIA.\n> \n> Sure.\n\nHmmm... I don't see \"Git User's Survey 2009\" announcement at \nhttp://perl5.git.perl.org/.  Neither at Planet Perl Iron Man, nor at\nPerlsphere\n\n-- \nJakub Narebski\n\nGit User's Survey 2009:\nhttp://tinyurl.com/GitSurvey2009\n"},{"id":"119387","messageId":"9b18b3110908021655r18203816u2864ffb95aabb354@mail.gmail.com","threadId":"20324","inReplyTo":"200908030125.53534.jnareb@gmail.com","subject":"Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2009-08-02T23:55:37Z","receivedAt":"2009-08-02T23:55:37Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"2009/8/3 Jakub Narebski <jnareb@gmail.com>:\n> On Sat, 1 Aug 2009, demerphq wrote:\n>> 2009/8/1 Jakub Narebski <jnareb@gmail.com>:\n>>> demerphq <demerphq@gmail.com> writes:\n>>>\n>>>> FWIW the perl project ripped out all the snapshot generation logic\n>>>> from gitweb, and replaced it with a tool that generates snapshots\n>>>> correctly for our requirements (if the build process needs additional\n>>>> files /currently/ git-archive does not support adding them), this\n>>>> includes a disk level cache for the snapshots since creating the tar,\n>>>> adding the additional files, then gziping is quite slow.\n>>>>\n>>>> If its interesting to people I can post it and the other changes here,\n>>>> although its not a \"nice\" change, as I literally ripped out the\n>>>> existing code.\n>>>\n>>> Do you mean gitweb serving http://perl5.git.perl.org/ ?\n>>\n>> Yes. I updated it to a relatively recent version just the other week.\n>\n> By the way, if you don't mind, how do you keep your changes on top\n> of changing target, i.e. on top of updating baseline to recent\n> versions?\n\nThings broke. I fixed them. :-)\n\n\n>> Also probably there are one or two patches that should be pushed back\n>> to you.\n>\n> What do those patches consist of?\n\nOne has to do with layout problems in the blame, and I think one deals\nwith broken line number anchor links.\n\nIll look closer when i have more time.\n\n> By the way, this is one of the only two places where we have to\n> use quote_command and 3-argument form of open, instead of list for of\n> magic open.  It is because of pipeline, piping git-archive output\n> into compressor.  The other such place is git_object, to redirect error\n> stream to /dev/null (to discard stderr).\n>\n> I thought about replacing it by list form of open somewhat (you can\n> find it in git mailing list archive), but it is not easy.  And IPC::Run\n> is IMVHO a bit of overkill, especially for \"minimal dependencies\"\n> gitweb (perhaps for Git::Web?).\n\nLately I have been playing with Capture::Tiny, which seems to fill\nsome nice niches in this respect.\n\n>> A better git api toolset for perl would make things a lot easier.\n>> Especially an XS one.\n>\n> Well, in the beginnings of Git.pm there was XS interface (to\n> git-hash-object or git-cat-file), but it was dropped because it\n> used -fPIC, which is not portable enough for Git (this is not present\n> in perl/Git.pm history in git.git repository).  Git.pm was created\n> by Petr 'Pasky' Baudis, author of Cogito porcelain, and creator and\n> admin of repo.or.cz (and also creator of first Git homepage).\n> But he is not a Perl hacker (correct me if I am wrong, Pasky); both\n> Cogito and repo.or.cz duct tape (under name of Girocco) are written\n> in bash, not in Perl.  Which can be seen for example by using\n> Error::Simple in Git.pm...\n>\n> Also I don't think that Pasky has time for maintaining Git.pm,\n> nevermind modernizing it / adding new features.\n>\n> There was another attempt to make Perl interface for Git, more\n> object-oriented, by Lea Wiemann during GSoC 2008 project 'gitweb caching'.\n> You can see results in Lea repository at repo.or.cz[1], and also in\n> git.kernel.org gitweb code[2].\n>\n> [1]: http://repo.or.cz/w/git/gitweb-caching.git\n> [2]: http://git.kernel.org/?p=git/warthog9/gitweb.git\n\nYeah I recall that being discussed. I see that Git.pm has recevied\nsome attention since the last time I looked. Which is good to see.\n\nHaving lurked on the list I understand that there is some kind of\neffort for a git library.  Perhaps at some point that will end up with\nan XS interface.\n\n>>>\n>>> P.P.S. Could you per chance post announcement of Git User's Survey 2009\n>>> on http://perl5.git.perl.org, just like it is done on http://repo.or.cz\n>>> and http://git.kernel.org?  TIA.\n>>\n>> Sure.\n>\n> Hmmm... I don't see \"Git User's Survey 2009\" announcement at\n> http://perl5.git.perl.org/.  Neither at Planet Perl Iron Man, nor at\n> Perlsphere\n\nAh, sorry, I posted it to the perl5-porters mailing list and asked the\nrelevant people to forward the request on or post it on their site. I\nknow at least one perl committer already did it. However tomorrow the\nmain european perl conference starts (YAPC::EU in Lisbon) and I think\na big chunk of the \"active\" Perl community will be there, so probably\nthere wont be much action from them until they get back.\n\nTo be honest I'm not familiar with Planet Perl Iron Man nor\nPerlsphere. Ill have to check them out i guess. :-)\n\nI will try to remember to put up a note on the website as soon as possible.\n\ncheers,\nYves\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"119389","messageId":"200908030227.46550.jnareb@gmail.com","threadId":"20324","inReplyTo":"9b18b3110908021655r18203816u2864ffb95aabb354@mail.gmail.com","subject":"Re: [PATCHv5] Add Gitweb support for XZ compressed snapshots","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-03T00:27:45Z","receivedAt":"2009-08-03T00:27:45Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, 3 August 2009, demerphq wrote:\n> 2009/8/3 Jakub Narebski <jnareb@gmail.com>:\n>> On Sat, 1 Aug 2009, demerphq wrote:\n>>> 2009/8/1 Jakub Narebski <jnareb@gmail.com>:\n\n>>>> Do you mean gitweb serving http://perl5.git.perl.org/ ?\n>>>\n>>> Yes. I updated it to a relatively recent version just the other week.\n>>\n>> By the way, if you don't mind, how do you keep your changes on top\n>> of changing target, i.e. on top of updating baseline to recent\n>> versions?\n> \n> Things broke. I fixed them. :-)\n\nErr... I meant here to ask if you use merging into your branch, rebasing\nyour branch on top of upstream, using some patch management interface like\nStGIT, Guilt ot Quilt, or topic branch management tool namely TopGit?\n \n>>> Also probably there are one or two patches that should be pushed back\n>>> to you.\n>>\n>> What do those patches consist of?\n> \n> One has to do with layout problems in the blame, and I think one deals\n> with broken line number anchor links.\n> \n> Ill look closer when i have more time.\n\nAbout the one with broken 'linenr' links: you should look at patch which\nmakes gitweb use information in \"previous\" header from git-blame -p, \nwhich was send to git mailing list and is currently in 'pu':\n  [PATCHv2 03/10] gitweb: Use \"previous\" header of git-blame -p in 'blame' view\n  Message-Id: <1248475450-5668-4-git-send-email-jnareb@gmail.com>\n  http://article.gmane.org/gmane.comp.version-control.git/123961\n\n\nWhat layout problems in 'blame' view did you notice? \n\n>> By the way, this is one of the only two places where we have to\n>> use quote_command and 3-argument form of open, instead of list for of\n>> magic open.  It is because of pipeline, piping git-archive output\n>> into compressor.  The other such place is git_object, to redirect error\n>> stream to /dev/null (to discard stderr).\n>>\n>> I thought about replacing it by list form of open somewhat (you can\n>> find it in git mailing list archive), but it is not easy.  And IPC::Run\n>> is IMVHO a bit of overkill, especially for \"minimal dependencies\"\n>> gitweb (perhaps for Git::Web?).\n> \n> Lately I have been playing with Capture::Tiny, which seems to fill\n> some nice niches in this respect.\n\nI don't quite see how you plan to use Capture::Tiny here.  The output\nof git-archive has to be passed to compressor, and then dumped to stdout,\nnot captured.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"119415","messageId":"200908031926.15255.jnareb@gmail.com","threadId":"20324","inReplyTo":"9b18b3110908010413w51e901dfk5a6f1666e5c3197f@mail.gmail.com","subject":"Re: Working on gitweb (was: [PATCHv5] Add Gitweb support for XZ compressed snapshots)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-08-03T17:26:11Z","receivedAt":"2009-08-03T17:26:11Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dnia sobota 1. sierpnia 2009 13:13, demerphq napisał:\n\n> One thing i notice is that most/all of the existing repos are forks of git.\n> \n> And the changes are to gitweb.perl, persumably on the assumption that\n> people upgrade using the install process.\n> \n> That doesn't make it easy to track changes that are made to the\n> production version (\"installed\" version), unless im missing something.\n> \n> How does one set up gitweb so that one can hack its sources and track\n> them in a sane way when you are not using make install-gitweb\n> (whatever) all the time?\n\nThere are at least two possible options. First (that is the one I use)\nis to create script to update gitweb, which would run \"make gitweb/gitweb.cgi\",\nwith appropriate option, then copy files, like gitweb-update.sh script below.\n\nSecond option is to do like t/t9500* gitweb test, which means providing\nconfig file by using GITWEB_CONFIG environment variable, and set all \nrequired options/variables like $GIT via config file instead of via \nbuild time configuration.  See also gitweb-run.sh script below.\n\n-- [gitweb-update.sh] --\n#!/bin/bash\n\n#BINDIR=\"/usr/bin\"\nBINDIR=\"/home/local/git\"\n\nfunction make_gitweb()\n{\n\tpushd \"/home/jnareb/git/\"\n\n\tmake GITWEB_PROJECTROOT=\"/home/local/scm\" \\\n\t     GITWEB_CSS=\"/gitweb/gitweb.css\" \\\n\t     GITWEB_LOGO=\"/gitweb/git-logo.png\" \\\n\t     GITWEB_FAVICON=\"/gitweb/git-favicon.png\" \\\n\t     GITWEB_BLAMEJS=\"/gitweb/blame.js\" \\\n\t     GITWEB_HOMETEXT=\"/home/local/scm/indextext.html\" \\\n\t     bindir=$BINDIR \\\n\t     gitweb/gitweb.cgi\n\n\tpopd\n}\n\nfunction copy_gitweb()\n{\n\tcp -fv ~/git/gitweb/gitweb.{cgi,css} /home/local/gitweb/\n}\n\nmake_gitweb\ncopy_gitweb\n\n# end of gitweb-update.sh\n\n-- [gitweb-run.sh] --\n#!/bin/bash\n\nexport GATEWAY_INTERFACE=\"CGI/1.1\"\nexport HTTP_ACCEPT=\"*/*\"\nexport REQUEST_METHOD=\"GET\"\nexport QUERY_STRING=\"\"$1\"\"\nexport PATH_INFO=\"\"$2\"\"\n\nexport GITWEB_CONFIG=\"~/git/gitweb/gitweb_config.perl\"\n\nperl -- ~/git/gitweb/gitweb.perl\n\n-- [gitweb_config.perl] --\n#!/usr/bin/perl\n\n# gitweb configuration\n\nour $version = \"current\";\n#our $GIT = \"/usr/bin/git\";\nour $GIT = \"/home/local/git/git\";\nour $projectroot = \"/home/local/scm\";\nour $home_link_str = \"projects\";\nour $site_name = \"[localhost]\";\nour $site_header = \"\";\nour $site_footer = \"\";\nour $home_text = \"indextext.html\";\nour @stylesheets = (\"file:///home/user/git/gitweb/gitweb.css\");\nour $logo = \"file:///home/user/git/gitweb/git-logo.png\";\nour $favicon = \"file:///home/user/git/gitweb/git-favicon.png\";\nour $blamejs = \"file:///home/user/git/gitweb/blame.js\";\nour $projects_list = \"\";\nour $export_ok = \"\";\nour $strict_export = \"\";\nour $project_maxdepth = 2007;\nour @git_base_url_list = (\"/home/user/git\");\n\n$feature{'blame'}{'default'} = [1];\n$feature{'grep'}{'default'} = [1];\n$feature{'pickaxe'}{'default'} = [1];\n\n# end of gitweb_config.perl\n-- \nJakub Narebski\nPoland\n"}]}