{"thread":{"id":"22216","subject":"[PATCH 1/9] gitweb: Load checking","startedAt":"2010-01-14T01:22:56Z","lastAt":"2010-02-07T21:32:06Z","messageCount":60,"participants":["John 'Warthog9' Hawley","Jakub Narebski","J.H.","Junio C Hamano","Petr Baudis"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"131564","messageId":"1263432185-21334-1-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":null,"subject":"[PATCH 0/9] Gitweb caching v5","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:22:56Z","receivedAt":"2010-01-14T01:22:56Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"Afternoon everyone,\n \nThis is the latest incarnation of gitweb w/ caching.  This is\nfinally at the point where it should probably start either being\nconsidered for inclusion or mainline, or I need to accept that this\nwill never get in and more perminantely fork (as is the case with\nFedora where this is going in as gitweb-caching as a parrallel rpm\npackage).\n\nThat said this brings the base up to mainline (again), it updates a\nnumber of elements in the caching engine, and this is a much cleaner\nbreak-out of the tree vs. what I am currently developing against.\n\nv5:\n\t- Missed a couple of things that were in my local tree, and\n\t  added them back in.\n\t- Split up the die_error and the version matching patch\n\t- Set version matching to be on by default - otherwise this\n\t  really is code that will never get checked, or at best\n\t  enabled by default by distributions\n\t- Added a minor code cleanup with respect to $site_header\n\t  that was already in my tree\n\t- Applied against a more recent git tree vs. 1.6.6-rc2\n\t- Removed breakout patch for now (did that in v4 actually)\n\t  and will deal with that separately \n\n\thttp://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v5\n\nv4:\n\t- major re-working of the caching layer to use file handle\n\t  redirection instead of buffering output\n\t- other minor improvements\n\n\thttp://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v4\nv3:\n\t- various minor re-works based on mailing list feedback,\n\t  this series was not sent to the mailing list.\nv2:\n\t- Better breakout\n\t- You can actually disable the cache now\n\n- John 'Warthog9' Hawley \n\n\n\nJohn 'Warthog9' Hawley (9):\n  gitweb: Load checking\n  gitweb: change die_error to take \"extra\" argument for extended die\n    information\n  gitweb: Add option to force version match\n  gitweb: Makefile improvements\n  gitweb: add a get function to compliment print_local_time\n  gitweb: add a get function to compliment print_sort_th\n  gitweb: cleanup error message produced by undefined $site_header\n  gitweb: Convert output to using indirect file handle\n  gitweb: File based caching layer (from git.kernel.org)\n\n Makefile           |   65 +---\n gitweb/Makefile    |  129 +++++++\n gitweb/README      |   10 +-\n gitweb/cache.pm    |  283 ++++++++++++++\n gitweb/gitweb.css  |    6 +\n gitweb/gitweb.perl | 1034 ++++++++++++++++++++++++++++++----------------------\n 6 files changed, 1030 insertions(+), 497 deletions(-)\n create mode 100644 gitweb/Makefile\n create mode 100644 gitweb/cache.pm\n"},{"id":"131559","messageId":"1263432185-21334-2-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":"1263432185-21334-1-git-send-email-warthog9@eaglescrag.net","subject":"[PATCH 1/9] gitweb: Load checking","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:22:57Z","receivedAt":"2010-01-14T01:22:57Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"From: John 'Warthog9' Hawley <warthog9@kernel.org>\n\nThis changes slightly the behavior of gitweb, so that it verifies\nthat the box isn't inundated with before attempting to serve gitweb.\nIf the box is overloaded, it basically returns a 503 Server Unavailable\nuntil the load falls below the defined threshold.  This helps dramatically\nif you have a box that's I/O bound, reaches a certain load and you\ndon't want gitweb, the I/O hog that it is, increasing the pain the\nserver is already undergoing.\n\nThis behavior is controlled by $maxload configuration variable.\nDefault is a load of 300, which for most cases should never be hit.\nUnset it (set it to undefined value, i.e. undef) to turn off checking.\n\nCurrently it requires that '/proc/loadavg' file exists, otherwise the\nload check is bypassed (load is taken to be 0).  So platforms that do\nnot implement '/proc/loadavg' currently cannot use this feature.\n(provisions are included for additional checks to be added by others)\n\nSigned-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\n gitweb/README      |    7 ++++++-\n gitweb/gitweb.perl |   45 +++++++++++++++++++++++++++++++++++++++++----\n 2 files changed, 47 insertions(+), 5 deletions(-)\n\ndiff --git a/gitweb/README b/gitweb/README\nindex e34ee79..6c2c8e1 100644\n--- a/gitweb/README\n+++ b/gitweb/README\n@@ -174,7 +174,7 @@ not include variables usually directly set during build):\n    Base URL for relative URLs in pages generated by gitweb,\n    (e.g. $logo, $favicon, @stylesheets if they are relative URLs),\n    needed and used only for URLs with nonempty PATH_INFO via\n-   <base href=\"$base_url>.  Usually gitweb sets its value correctly,\n+   <base href=\"$base_url\">.  Usually gitweb sets its value correctly,\n    and there is no need to set this variable, e.g. to $my_uri or \"/\".\n  * $home_link\n    Target of the home link on top of all pages (the first part of view\n@@ -228,6 +228,11 @@ not include variables usually directly set during build):\n    repositories from launching cross-site scripting (XSS) attacks.  Set this\n    to true if you don't trust the content of your repositories. The default\n    is false.\n+ * $maxload\n+   Used to set the maximum load that we will still respond to gitweb queries.\n+   If server load exceed this value then return \"503 Service Unavaliable\" error.\n+   Server load is taken to be 0 if gitweb cannot determine its value.  Set it to\n+   undefined value to turn it off.  The default is 300.\n \n \n Projects list file format\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 7e477af..0a07d3a 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -221,6 +221,12 @@ our %avatar_size = (\n \t'double'  => 32\n );\n \n+# Used to set the maximum load that we will still respond to gitweb queries.\n+# If server load exceed this value then return \"503 server busy\" error.\n+# If gitweb cannot determined server load, it is taken to be 0.\n+# Leave it undefined (or set to 'undef') to turn off load checking.\n+our $maxload = 300;\n+\n # You define site-wide feature defaults here; override them with\n # $GITWEB_CONFIG as necessary.\n our %feature = (\n@@ -551,6 +557,32 @@ if (-e $GITWEB_CONFIG) {\n \tdo $GITWEB_CONFIG_SYSTEM if -e $GITWEB_CONFIG_SYSTEM;\n }\n \n+# Get loadavg of system, to compare against $maxload.\n+# Currently it requires '/proc/loadavg' present to get loadavg;\n+# if it is not present it returns 0, which means no load checking.\n+sub get_loadavg {\n+\tif( -e '/proc/loadavg' ){\n+\t\topen my $fd, '<', '/proc/loadavg'\n+\t\t\tor return 0;\n+\t\tmy @load = split(/\\s+/, scalar <$fd>);\n+\t\tclose $fd;\n+\n+\t\t# The first three columns measure CPU and IO utilization of the last one,\n+\t\t# five, and 10 minute periods.  The fourth column shows the number of\n+\t\t# currently running processes and the total number of processes in the m/n\n+\t\t# format.  The last column displays the last process ID used.\n+\t\treturn $load[0] || 0;\n+\t}\n+\t# additional checks for load average should go here for things that don't export\n+\t# /proc/loadavg\n+\n+\treturn 0;\n+}\n+\n+if (defined $maxload && get_loadavg() > $maxload) {\n+\tdie_error(503, \"The load average on the server is too high\");\n+}\n+\n # version of the core git binary\n our $git_version = qx(\"$GIT\" --version) =~ m/git version (.*)$/ ? $1 : \"unknown\";\n $number_of_git_cmds++;\n@@ -3354,14 +3386,19 @@ sub git_footer_html {\n # 500: The server isn't configured properly, or\n #      an internal error occurred (e.g. failed assertions caused by bugs), or\n #      an unknown error occurred (e.g. the git binary died unexpectedly).\n+# 503: The server is currently unavailable (because it is overloaded,\n+#      or down for maintenance).  Generally, this is a temporary state.\n sub die_error {\n \tmy $status = shift || 500;\n \tmy $error = shift || \"Internal server error\";\n \n-\tmy %http_responses = (400 => '400 Bad Request',\n-\t\t\t      403 => '403 Forbidden',\n-\t\t\t      404 => '404 Not Found',\n-\t\t\t      500 => '500 Internal Server Error');\n+\tmy %http_responses = (\n+\t\t400 => '400 Bad Request',\n+\t\t403 => '403 Forbidden',\n+\t\t404 => '404 Not Found',\n+\t\t500 => '500 Internal Server Error',\n+\t\t503 => '503 Service Unavailable',\n+\t);\n \tgit_header_html($http_responses{$status});\n \tprint <<EOF;\n <div class=\"page_body\">\n-- \n1.6.5.2\n"},{"id":"131562","messageId":"1263432185-21334-3-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":"1263432185-21334-2-git-send-email-warthog9@eaglescrag.net","subject":"[PATCH 2/9] gitweb: change die_error to take \"extra\" argument for extended die information","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:22:58Z","receivedAt":"2010-01-14T01:22:58Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"This is a small change that just adds a 3rd, optional, parameter to die_error\nthat allows for extended error information to be output along with what the\nerror was.\n---\n gitweb/gitweb.perl |    8 +++++++-\n 1 files changed, 7 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 0a07d3a..8298de5 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -3391,6 +3391,7 @@ sub git_footer_html {\n sub die_error {\n \tmy $status = shift || 500;\n \tmy $error = shift || \"Internal server error\";\n+\tmy $extra = shift;\n \n \tmy %http_responses = (\n \t\t400 => '400 Bad Request',\n@@ -3405,8 +3406,13 @@ sub die_error {\n <br /><br />\n $status - $error\n <br />\n-</div>\n EOF\n+\tif (defined $extra) {\n+\t\tprint \"<hr />\\n\" .\n+\t\t\t\"$extra\\n\";\n+\t}\n+\tprint \"</div>\\n\";\n+\n \tgit_footer_html();\n \texit;\n }\n-- \n1.6.5.2\n"},{"id":"131561","messageId":"1263432185-21334-4-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":"1263432185-21334-3-git-send-email-warthog9@eaglescrag.net","subject":"[PATCH 3/9] gitweb: Add option to force version match","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:22:59Z","receivedAt":"2010-01-14T01:22:59Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"This adds $git_versions_must_match variable, which is set to true,\nchecks that we are running on the same version of git that we\nshipped with, and if not throw '500 Internal Server Error' error.\nWhat is checked is the version of gitweb (embedded in building\ngitweb.cgi), against version of runtime git binary used.\n\nGitweb can usually run with a mismatched git install.  This is more\nhere to give an obvious warning as to whats going on vs. silently\nfailing.\n\nBy default this feature is turned on.\n\nSigned-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\n gitweb/README      |    3 +++\n gitweb/gitweb.perl |   23 +++++++++++++++++++++++\n 2 files changed, 26 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/README b/gitweb/README\nindex 6c2c8e1..03151d2 100644\n--- a/gitweb/README\n+++ b/gitweb/README\n@@ -233,6 +233,9 @@ not include variables usually directly set during build):\n    If server load exceed this value then return \"503 Service Unavaliable\" error.\n    Server load is taken to be 0 if gitweb cannot determine its value.  Set it to\n    undefined value to turn it off.  The default is 300.\n+ * $git_versions_must_match\n+   If set, gitweb fails with 500 Internal Server Error if the version of gitweb\n+   doesn't match version of git binary.  The default is true.\n \n \n Projects list file format\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 8298de5..b41bc33 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -221,6 +221,9 @@ our %avatar_size = (\n \t'double'  => 32\n );\n \n+# If it is true, exit if gitweb version and git binary version don't match\n+our $git_versions_must_match = 1;\n+\n # Used to set the maximum load that we will still respond to gitweb queries.\n # If server load exceed this value then return \"503 server busy\" error.\n # If gitweb cannot determined server load, it is taken to be 0.\n@@ -587,6 +590,26 @@ if (defined $maxload && get_loadavg() > $maxload) {\n our $git_version = qx(\"$GIT\" --version) =~ m/git version (.*)$/ ? $1 : \"unknown\";\n $number_of_git_cmds++;\n \n+# Throw an error if git versions does not match, if $git_versions_must_match is true.\n+if ($git_versions_must_match &&\n+    $git_version ne $version) {\n+\tmy $admin_contact =\n+\t\tdefined $ENV{'SERVER_ADMIN'} ? \", $ENV{'SERVER_ADMIN'},\" : '';\n+\tmy $err_msg = <<EOT;\n+<h1 align=\"center\">*** Warning ***</h1>\n+<p>\n+This version of gitweb was compiled for <b>@{[esc_html($version)]}</b>,\n+however git version <b>@{[esc_html($git_version)]}</b> was found on server,\n+and administrator requested strict version checking.\n+</p>\n+<p>\n+Please contact the server administrator${admin_contact} to either configure\n+gitweb to allow mismatched versions, or update git or gitweb installation.\n+</p>\n+EOT\n+\tdie_error(500, 'Internal server error', $err_msg);\n+}\n+\n $projects_list ||= $projectroot;\n \n # ======================================================================\n-- \n1.6.5.2\n"},{"id":"131565","messageId":"1263432185-21334-5-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":"1263432185-21334-4-git-send-email-warthog9@eaglescrag.net","subject":"[PATCH 4/9] gitweb: Makefile improvements","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:23:00Z","receivedAt":"2010-01-14T01:23:00Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"From: John 'Warthog9' Hawley <warthog9@kernel.org>\n\nThis commit adjust the main Makefile so you can simply run\n\n     make gitweb\n\nwhich in turn calls gitweb/Makefile.  This means that in order to\ngenerate gitweb, you can simply run 'make' from gitweb subdirectory:\n\n     cd gitweb\n     make\n\nTargets gitweb/gitweb.cgi and (dependent on JSMIN being defined)\ngitweb/gitweb.min.js in main Makefile are preserved for backward\ncompatibility.\n\nSigned-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\n Makefile        |   65 +++++----------------------\n gitweb/Makefile |  129 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 141 insertions(+), 53 deletions(-)\n create mode 100644 gitweb/Makefile\n\ndiff --git a/Makefile b/Makefile\nindex be18389..9f069a8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -282,29 +282,6 @@ pathsep = :\n # JavaScript minifier invocation that can function as filter\n JSMIN =\n \n-# default configuration for gitweb\n-GITWEB_CONFIG = gitweb_config.perl\n-GITWEB_CONFIG_SYSTEM = /etc/gitweb.conf\n-GITWEB_HOME_LINK_STR = projects\n-GITWEB_SITENAME =\n-GITWEB_PROJECTROOT = /pub/git\n-GITWEB_PROJECT_MAXDEPTH = 2007\n-GITWEB_EXPORT_OK =\n-GITWEB_STRICT_EXPORT =\n-GITWEB_BASE_URL =\n-GITWEB_LIST =\n-GITWEB_HOMETEXT = indextext.html\n-GITWEB_CSS = gitweb.css\n-GITWEB_LOGO = git-logo.png\n-GITWEB_FAVICON = git-favicon.png\n-ifdef JSMIN\n-GITWEB_JS = gitweb.min.js\n-else\n-GITWEB_JS = gitweb.js\n-endif\n-GITWEB_SITE_HEADER =\n-GITWEB_SITE_FOOTER =\n-\n export prefix bindir sharedir sysconfdir\n \n CC = gcc\n@@ -1535,6 +1512,11 @@ $(patsubst %.perl,%,$(SCRIPT_PERL)): % : %.perl\n \tchmod +x $@+ && \\\n \tmv $@+ $@\n \n+\n+.PHONY: gitweb\n+gitweb:\n+\t$(QUIET_SUBDIR0)gitweb $(QUIET_SUBDIR1) all\n+\n ifdef JSMIN\n OTHER_PROGRAMS += gitweb/gitweb.cgi   gitweb/gitweb.min.js\n gitweb/gitweb.cgi: gitweb/gitweb.perl gitweb/gitweb.min.js\n@@ -1542,30 +1524,13 @@ else\n OTHER_PROGRAMS += gitweb/gitweb.cgi\n gitweb/gitweb.cgi: gitweb/gitweb.perl\n endif\n-\t$(QUIET_GEN)$(RM) $@ $@+ && \\\n-\tsed -e '1s|#!.*perl|#!$(PERL_PATH_SQ)|' \\\n-\t    -e 's|++GIT_VERSION++|$(GIT_VERSION)|g' \\\n-\t    -e 's|++GIT_BINDIR++|$(bindir)|g' \\\n-\t    -e 's|++GITWEB_CONFIG++|$(GITWEB_CONFIG)|g' \\\n-\t    -e 's|++GITWEB_CONFIG_SYSTEM++|$(GITWEB_CONFIG_SYSTEM)|g' \\\n-\t    -e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \\\n-\t    -e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \\\n-\t    -e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \\\n-\t    -e 's|\"++GITWEB_PROJECT_MAXDEPTH++\"|$(GITWEB_PROJECT_MAXDEPTH)|g' \\\n-\t    -e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \\\n-\t    -e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \\\n-\t    -e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \\\n-\t    -e 's|++GITWEB_LIST++|$(GITWEB_LIST)|g' \\\n-\t    -e 's|++GITWEB_HOMETEXT++|$(GITWEB_HOMETEXT)|g' \\\n-\t    -e 's|++GITWEB_CSS++|$(GITWEB_CSS)|g' \\\n-\t    -e 's|++GITWEB_LOGO++|$(GITWEB_LOGO)|g' \\\n-\t    -e 's|++GITWEB_FAVICON++|$(GITWEB_FAVICON)|g' \\\n-\t    -e 's|++GITWEB_JS++|$(GITWEB_JS)|g' \\\n-\t    -e 's|++GITWEB_SITE_HEADER++|$(GITWEB_SITE_HEADER)|g' \\\n-\t    -e 's|++GITWEB_SITE_FOOTER++|$(GITWEB_SITE_FOOTER)|g' \\\n-\t    $< >$@+ && \\\n-\tchmod +x $@+ && \\\n-\tmv $@+ $@\n+\t$(QUIET_SUBDIR0)gitweb $(QUIET_SUBDIR1) $(patsubst gitweb/%,%,$@)\n+\n+ifdef JSMIN\n+gitweb/gitweb.min.js: gitweb/gitweb.js\n+\t$(QUIET_SUBDIR0)gitweb $(QUIET_SUBDIR1) $(patsubst gitweb/%,%,$@)\n+endif # JSMIN\n+\n \n git-instaweb: git-instaweb.sh gitweb/gitweb.cgi gitweb/gitweb.css gitweb/gitweb.js\n \t$(QUIET_GEN)$(RM) $@ $@+ && \\\n@@ -1592,12 +1557,6 @@ $(patsubst %.perl,%,$(SCRIPT_PERL)) git-instaweb: % : unimplemented.sh\n \tmv $@+ $@\n endif # NO_PERL\n \n-\n-ifdef JSMIN\n-gitweb/gitweb.min.js: gitweb/gitweb.js\n-\t$(QUIET_GEN)$(JSMIN) <$< >$@\n-endif # JSMIN\n-\n ifndef NO_PYTHON\n $(patsubst %.py,%,$(SCRIPT_PYTHON)): GIT-CFLAGS\n $(patsubst %.py,%,$(SCRIPT_PYTHON)): % : %.py\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nnew file mode 100644\nindex 0000000..c9eb1ee\n--- /dev/null\n+++ b/gitweb/Makefile\n@@ -0,0 +1,129 @@\n+# The default target of this Makefile is...\n+all::\n+\n+# Define V=1 to have a more verbose compile.\n+#\n+# Define JSMIN to point to JavaScript minifier that functions as\n+# a filter to have gitweb.js minified.\n+#\n+\n+prefix ?= $(HOME)\n+bindir ?= $(prefix)/bin\n+RM ?= rm -f\n+\n+# JavaScript minifier invocation that can function as filter\n+JSMIN ?=\n+\n+# default configuration for gitweb\n+GITWEB_CONFIG = gitweb_config.perl\n+GITWEB_CONFIG_SYSTEM = /etc/gitweb.conf\n+GITWEB_HOME_LINK_STR = projects\n+GITWEB_SITENAME =\n+GITWEB_PROJECTROOT = /pub/git\n+GITWEB_PROJECT_MAXDEPTH = 2007\n+GITWEB_EXPORT_OK =\n+GITWEB_STRICT_EXPORT =\n+GITWEB_BASE_URL =\n+GITWEB_LIST =\n+GITWEB_HOMETEXT = indextext.html\n+GITWEB_CSS = gitweb.css\n+GITWEB_LOGO = git-logo.png\n+GITWEB_FAVICON = git-favicon.png\n+ifdef JSMIN\n+GITWEB_JS = gitweb.min.js\n+else\n+GITWEB_JS = gitweb.js\n+endif\n+GITWEB_SITE_HEADER =\n+GITWEB_SITE_FOOTER =\n+\n+# include user config\n+-include ../config.mak.autogen\n+-include ../config.mak\n+\n+# determine version\n+../GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE\n+\t$(QUIET_SUBDIR0)../ $(QUIET_SUBDIR1) GIT-VERSION-FILE\n+\n+-include ../GIT-VERSION-FILE\n+\n+### Build rules\n+\n+SHELL_PATH ?= $(SHELL)\n+PERL_PATH  ?= /usr/bin/perl\n+\n+# Shell quote;\n+bindir_SQ = $(subst ','\\'',$(bindir))         #'\n+SHELL_PATH_SQ = $(subst ','\\'',$(SHELL_PATH)) #'\n+PERL_PATH_SQ  = $(subst ','\\'',$(PERL_PATH))  #'\n+\n+# Quiet generation (unless V=1)\n+QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir\n+QUIET_SUBDIR1  =\n+\n+ifneq ($(findstring $(MAKEFLAGS),w),w)\n+PRINT_DIR = --no-print-directory\n+else # \"make -w\"\n+NO_SUBDIR = :\n+endif\n+\n+ifneq ($(findstring $(MAKEFLAGS),s),s)\n+ifndef V\n+\tQUIET          = @\n+\tQUIET_GEN      = $(QUIET)echo '   ' GEN $@;\n+\tQUIET_SUBDIR0  = +@subdir=\n+\tQUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \\\n+\t                 $(MAKE) $(PRINT_DIR) -C $$subdir\n+\texport V\n+\texport QUIET\n+\texport QUIET_GEN\n+\texport QUIET_SUBDIR0\n+\texport QUIET_SUBDIR1\n+endif\n+endif\n+\n+all:: gitweb.cgi\n+\n+ifdef JSMIN\n+FILES=gitweb.cgi gitweb.min.js\n+gitweb.cgi: gitweb.perl gitweb.min.js\n+else # !JSMIN\n+FILES=gitweb.cgi\n+gitweb.cgi: gitweb.perl\n+endif # JSMIN\n+\n+gitweb.cgi:\n+\t$(QUIET_GEN)$(RM) $@ $@+ && \\\n+\tsed -e '1s|#!.*perl|#!$(PERL_PATH_SQ)|' \\\n+\t    -e 's|++GIT_VERSION++|$(GIT_VERSION)|g' \\\n+\t    -e 's|++GIT_BINDIR++|$(bindir)|g' \\\n+\t    -e 's|++GITWEB_CONFIG++|$(GITWEB_CONFIG)|g' \\\n+\t    -e 's|++GITWEB_CONFIG_SYSTEM++|$(GITWEB_CONFIG_SYSTEM)|g' \\\n+\t    -e 's|++GITWEB_HOME_LINK_STR++|$(GITWEB_HOME_LINK_STR)|g' \\\n+\t    -e 's|++GITWEB_SITENAME++|$(GITWEB_SITENAME)|g' \\\n+\t    -e 's|++GITWEB_PROJECTROOT++|$(GITWEB_PROJECTROOT)|g' \\\n+\t    -e 's|\"++GITWEB_PROJECT_MAXDEPTH++\"|$(GITWEB_PROJECT_MAXDEPTH)|g' \\\n+\t    -e 's|++GITWEB_EXPORT_OK++|$(GITWEB_EXPORT_OK)|g' \\\n+\t    -e 's|++GITWEB_STRICT_EXPORT++|$(GITWEB_STRICT_EXPORT)|g' \\\n+\t    -e 's|++GITWEB_BASE_URL++|$(GITWEB_BASE_URL)|g' \\\n+\t    -e 's|++GITWEB_LIST++|$(GITWEB_LIST)|g' \\\n+\t    -e 's|++GITWEB_HOMETEXT++|$(GITWEB_HOMETEXT)|g' \\\n+\t    -e 's|++GITWEB_CSS++|$(GITWEB_CSS)|g' \\\n+\t    -e 's|++GITWEB_LOGO++|$(GITWEB_LOGO)|g' \\\n+\t    -e 's|++GITWEB_FAVICON++|$(GITWEB_FAVICON)|g' \\\n+\t    -e 's|++GITWEB_JS++|$(GITWEB_JS)|g' \\\n+\t    -e 's|++GITWEB_SITE_HEADER++|$(GITWEB_SITE_HEADER)|g' \\\n+\t    -e 's|++GITWEB_SITE_FOOTER++|$(GITWEB_SITE_FOOTER)|g' \\\n+\t    $< >$@+ && \\\n+\tchmod +x $@+ && \\\n+\tmv $@+ $@\n+\n+ifdef JSMIN\n+gitweb.min.js: gitweb.js\n+\t$(QUIET_GEN)$(JSMIN) <$< >$@\n+endif # JSMIN\n+\n+clean:\n+\t$(RM) $(FILES)\n+\n+.PHONY: all clean .FORCE-GIT-VERSION-FILE\n-- \n1.6.5.2\n"},{"id":"131567","messageId":"1263432185-21334-6-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":"1263432185-21334-5-git-send-email-warthog9@eaglescrag.net","subject":"[PATCH 5/9] gitweb: add a get function to compliment print_local_time","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:23:01Z","receivedAt":"2010-01-14T01:23:01Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"This adds a get function for print_local_time so that the basic\nfunction can be used outside of their straight printing operation.\n---\n gitweb/gitweb.perl |   11 +++++++++--\n 1 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex b41bc33..d38aad6 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -3537,14 +3537,21 @@ sub git_print_header_div {\n }\n \n sub print_local_time {\n+\tprint get_local_time(@_);\n+}\n+\n+sub get_local_time {\n+\tmy $localtime = \"\";\n \tmy %date = @_;\n \tif ($date{'hour_local'} < 6) {\n-\t\tprintf(\" (<span class=\\\"atnight\\\">%02d:%02d</span> %s)\",\n+\t\t$localtime .= sprintf(\" (<span class=\\\"atnight\\\">%02d:%02d</span> %s)\",\n \t\t\t$date{'hour_local'}, $date{'minute_local'}, $date{'tz_local'});\n \t} else {\n-\t\tprintf(\" (%02d:%02d %s)\",\n+\t\t$localtime .= sprintf(\" (%02d:%02d %s)\",\n \t\t\t$date{'hour_local'}, $date{'minute_local'}, $date{'tz_local'});\n \t}\n+\n+\treturn $localtime\n }\n \n # Outputs the author name and date in long form\n-- \n1.6.5.2\n"},{"id":"131560","messageId":"1263432185-21334-7-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":"1263432185-21334-6-git-send-email-warthog9@eaglescrag.net","subject":"[PATCH 6/9] gitweb: add a get function to compliment print_sort_th","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:23:02Z","receivedAt":"2010-01-14T01:23:02Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"This adds a get function for print_sort_th so that the basic\nfunction can be used outside of their straight printing operation.\n---\n gitweb/gitweb.perl |   11 +++++++++--\n 1 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex d38aad6..07fdeb5 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -4375,17 +4375,24 @@ sub fill_project_list_info {\n # print 'sort by' <th> element, generating 'sort by $name' replay link\n # if that order is not selected\n sub print_sort_th {\n+\tprint get_sort_th(@_);\n+}\n+\n+sub get_sort_th {\n \tmy ($name, $order, $header) = @_;\n+\tmy $sortth = \"\";\n \t$header ||= ucfirst($name);\n \n \tif ($order eq $name) {\n-\t\tprint \"<th>$header</th>\\n\";\n+\t\t$sortth .= \"<th>$header</th>\\n\";\n \t} else {\n-\t\tprint \"<th>\" .\n+\t\t$sortth .= \"<th>\" .\n \t\t      $cgi->a({-href => href(-replay=>1, order=>$name),\n \t\t               -class => \"header\"}, $header) .\n \t\t      \"</th>\\n\";\n \t}\n+\n+\treturn $sortth;\n }\n \n sub git_project_list_body {\n-- \n1.6.5.2\n"},{"id":"131563","messageId":"1263432185-21334-8-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":"1263432185-21334-7-git-send-email-warthog9@eaglescrag.net","subject":"[PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:23:03Z","receivedAt":"2010-01-14T01:23:03Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"If $site_header is not defined you get extraneous errors in the web\nlogs:\n\n[Wed Jan 13 16:55:42 2010] [error] [client ::1] [Wed Jan 13 16:55:42 2010] gitweb.cgi: Use of uninitialized value $site_header in -f at /var/www/gitweb/gitweb.cgi line 3287., referer: http://git/gitweb.cgi\n\nfor example.  This ensures that the variable is defined before trying to use it.\n---\n gitweb/gitweb.perl |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 07fdeb5..c4a177d 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -3277,7 +3277,7 @@ EOF\n \tprint \"</head>\\n\" .\n \t      \"<body>\\n\";\n \n-\tif (-f $site_header) {\n+\tif ($site_header && -f $site_header) {\n \t\tinsert_file($site_header);\n \t}\n \n-- \n1.6.5.2\n"},{"id":"131566","messageId":"1263432185-21334-9-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":"1263432185-21334-8-git-send-email-warthog9@eaglescrag.net","subject":"[PATCH 8/9] gitweb: Convert output to using indirect file handle","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:23:04Z","receivedAt":"2010-01-14T01:23:04Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"This converts the output handling of gitweb to using an indirect\nfile handle.  This is in preparation to add the caching layer.  This\nis a slight modification to the way I was originally doing it by\npassing the output around.  This should be a nop and this shouldn't\nchange the behavior of gitweb.  This does leave error reporting\nfunctions (die_error specifically) continuing to output directly\nas I want to garauntee those will report their errors regardless of\nwhat may be going on with respect to the rest of the output.\n---\n gitweb/gitweb.perl |  880 ++++++++++++++++++++++++++--------------------------\n 1 files changed, 448 insertions(+), 432 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex c4a177d..8bb323c 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -450,6 +450,13 @@ our %feature = (\n \t\t'default' => [0]},\n );\n \n+# Basic file handler for all of gitweb, there are two of them.  The first\n+# is the basic text/html file handler which is used for everything other\n+# then the binary files, that uses a separate file handler though\n+# these are both set to STDOUT for the time being.\n+our $output_handler = *STDOUT;\n+our $output_handler_bin = *STDOUT;\n+\n sub gitweb_get_feature {\n \tmy ($name) = @_;\n \treturn unless exists $feature{$name};\n@@ -3081,7 +3088,7 @@ sub insert_file {\n \tmy $filename = shift;\n \n \topen my $fd, '<', $filename;\n-\tprint map { to_utf8($_) } <$fd>;\n+\tprint {$output_handler} map { to_utf8($_) } <$fd>;\n \tclose $fd;\n }\n \n@@ -3198,10 +3205,10 @@ sub git_header_html {\n \t} else {\n \t\t$content_type = 'text/html';\n \t}\n-\tprint $cgi->header(-type=>$content_type, -charset => 'utf-8',\n+\tprint {$output_handler} $cgi->header(-type=>$content_type, -charset => 'utf-8',\n \t                   -status=> $status, -expires => $expires);\n \tmy $mod_perl_version = $ENV{'MOD_PERL'} ? \" $ENV{'MOD_PERL'}\" : '';\n-\tprint <<EOF;\n+\tprint {$output_handler} <<EOF;\n <?xml version=\"1.0\" encoding=\"utf-8\"?>\n <!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\n <html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en-US\" lang=\"en-US\">\n@@ -3216,16 +3223,16 @@ EOF\n \t# the stylesheet, favicon etc urls won't work correctly with path_info\n \t# unless we set the appropriate base URL\n \tif ($ENV{'PATH_INFO'}) {\n-\t\tprint \"<base href=\\\"\".esc_url($base_url).\"\\\" />\\n\";\n+\t\tprint {$output_handler} \"<base href=\\\"\".esc_url($base_url).\"\\\" />\\n\";\n \t}\n \t# print out each stylesheet that exist, providing backwards capability\n \t# for those people who defined $stylesheet in a config file\n \tif (defined $stylesheet) {\n-\t\tprint '<link rel=\"stylesheet\" type=\"text/css\" href=\"'.$stylesheet.'\"/>'.\"\\n\";\n+\t\tprint {$output_handler} '<link rel=\"stylesheet\" type=\"text/css\" href=\"'.$stylesheet.'\"/>'.\"\\n\";\n \t} else {\n \t\tforeach my $stylesheet (@stylesheets) {\n \t\t\tnext unless $stylesheet;\n-\t\t\tprint '<link rel=\"stylesheet\" type=\"text/css\" href=\"'.$stylesheet.'\"/>'.\"\\n\";\n+\t\t\tprint {$output_handler} '<link rel=\"stylesheet\" type=\"text/css\" href=\"'.$stylesheet.'\"/>'.\"\\n\";\n \t\t}\n \t}\n \tif (defined $project) {\n@@ -3244,7 +3251,7 @@ EOF\n \n \t\t\t$href_params{'action'} = $type;\n \t\t\t$link_attr{'-href'} = href(%href_params);\n-\t\t\tprint \"<link \".\n+\t\t\tprint {$output_handler} \"<link \".\n \t\t\t      \"rel=\\\"$link_attr{'-rel'}\\\" \".\n \t\t\t      \"title=\\\"$link_attr{'-title'}\\\" \".\n \t\t\t      \"href=\\\"$link_attr{'-href'}\\\" \".\n@@ -3254,7 +3261,7 @@ EOF\n \t\t\t$href_params{'extra_options'} = '--no-merges';\n \t\t\t$link_attr{'-href'} = href(%href_params);\n \t\t\t$link_attr{'-title'} .= ' (no merges)';\n-\t\t\tprint \"<link \".\n+\t\t\tprint {$output_handler} \"<link \".\n \t\t\t      \"rel=\\\"$link_attr{'-rel'}\\\" \".\n \t\t\t      \"title=\\\"$link_attr{'-title'}\\\" \".\n \t\t\t      \"href=\\\"$link_attr{'-href'}\\\" \".\n@@ -3263,37 +3270,37 @@ EOF\n \t\t}\n \n \t} else {\n-\t\tprintf('<link rel=\"alternate\" title=\"%s projects list\" '.\n+\t\tprintf( {$output_handler} '<link rel=\"alternate\" title=\"%s projects list\" '.\n \t\t       'href=\"%s\" type=\"text/plain; charset=utf-8\" />'.\"\\n\",\n \t\t       $site_name, href(project=>undef, action=>\"project_index\"));\n-\t\tprintf('<link rel=\"alternate\" title=\"%s projects feeds\" '.\n+\t\tprintf( {$output_handler} '<link rel=\"alternate\" title=\"%s projects feeds\" '.\n \t\t       'href=\"%s\" type=\"text/x-opml\" />'.\"\\n\",\n \t\t       $site_name, href(project=>undef, action=>\"opml\"));\n \t}\n \tif (defined $favicon) {\n-\t\tprint qq(<link rel=\"shortcut icon\" href=\"$favicon\" type=\"image/png\" />\\n);\n+\t\tprint {$output_handler} qq(<link rel=\"shortcut icon\" href=\"$favicon\" type=\"image/png\" />\\n);\n \t}\n \n-\tprint \"</head>\\n\" .\n+\tprint {$output_handler} \"</head>\\n\" .\n \t      \"<body>\\n\";\n \n \tif ($site_header && -f $site_header) {\n \t\tinsert_file($site_header);\n \t}\n \n-\tprint \"<div class=\\\"page_header\\\">\\n\" .\n+\tprint {$output_handler} \"<div class=\\\"page_header\\\">\\n\" .\n \t      $cgi->a({-href => esc_url($logo_url),\n \t               -title => $logo_label},\n \t              qq(<img src=\"$logo\" width=\"72\" height=\"27\" alt=\"git\" class=\"logo\"/>));\n-\tprint $cgi->a({-href => esc_url($home_link)}, $home_link_str) . \" / \";\n+\tprint {$output_handler} $cgi->a({-href => esc_url($home_link)}, $home_link_str) . \" / \";\n \tif (defined $project) {\n-\t\tprint $cgi->a({-href => href(action=>\"summary\")}, esc_html($project));\n+\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"summary\")}, esc_html($project));\n \t\tif (defined $action) {\n-\t\t\tprint \" / $action\";\n+\t\t\tprint {$output_handler} \" / $action\";\n \t\t}\n-\t\tprint \"\\n\";\n+\t\tprint {$output_handler} \"\\n\";\n \t}\n-\tprint \"</div>\\n\";\n+\tprint {$output_handler} \"</div>\\n\";\n \n \tmy $have_search = gitweb_check_feature('search');\n \tif (defined $project && $have_search) {\n@@ -3313,7 +3320,7 @@ EOF\n \t\tif ($use_pathinfo) {\n \t\t\t$action .= \"/\".esc_url($project);\n \t\t}\n-\t\tprint $cgi->startform(-method => \"get\", -action => $action) .\n+\t\tprint {$output_handler} $cgi->startform(-method => \"get\", -action => $action) .\n \t\t      \"<div class=\\\"search\\\">\\n\" .\n \t\t      (!$use_pathinfo &&\n \t\t      $cgi->input({-name=>\"p\", -value=>$project, -type=>\"hidden\"}) . \"\\n\") .\n@@ -3336,11 +3343,11 @@ EOF\n sub git_footer_html {\n \tmy $feed_class = 'rss_logo';\n \n-\tprint \"<div class=\\\"page_footer\\\">\\n\";\n+\tprint {$output_handler} \"<div class=\\\"page_footer\\\">\\n\";\n \tif (defined $project) {\n \t\tmy $descr = git_get_project_description($project);\n \t\tif (defined $descr) {\n-\t\t\tprint \"<div class=\\\"page_footer_text\\\">\" . esc_html($descr) . \"</div>\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"page_footer_text\\\">\" . esc_html($descr) . \"</div>\\n\";\n \t\t}\n \n \t\tmy %href_params = get_feed_info();\n@@ -3351,22 +3358,22 @@ sub git_footer_html {\n \n \t\tforeach my $format qw(RSS Atom) {\n \t\t\t$href_params{'action'} = lc($format);\n-\t\t\tprint $cgi->a({-href => href(%href_params),\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(%href_params),\n \t\t\t              -title => \"$href_params{'-title'} $format feed\",\n \t\t\t              -class => $feed_class}, $format).\"\\n\";\n \t\t}\n \n \t} else {\n-\t\tprint $cgi->a({-href => href(project=>undef, action=>\"opml\"),\n+\t\tprint {$output_handler} $cgi->a({-href => href(project=>undef, action=>\"opml\"),\n \t\t              -class => $feed_class}, \"OPML\") . \" \";\n-\t\tprint $cgi->a({-href => href(project=>undef, action=>\"project_index\"),\n+\t\tprint {$output_handler} $cgi->a({-href => href(project=>undef, action=>\"project_index\"),\n \t\t              -class => $feed_class}, \"TXT\") . \"\\n\";\n \t}\n-\tprint \"</div>\\n\"; # class=\"page_footer\"\n+\tprint {$output_handler} \"</div>\\n\"; # class=\"page_footer\"\n \n \tif (defined $t0 && gitweb_check_feature('timed')) {\n-\t\tprint \"<div id=\\\"generating_info\\\">\\n\";\n-\t\tprint 'This page took '.\n+\t\tprint {$output_handler} \"<div id=\\\"generating_info\\\">\\n\";\n+\t\tprint {$output_handler} 'This page took '.\n \t\t      '<span id=\"generating_time\" class=\"time_span\">'.\n \t\t      Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).\n \t\t      ' seconds </span>'.\n@@ -3375,26 +3382,26 @@ sub git_footer_html {\n \t\t      $number_of_git_cmds.\n \t\t      '</span> git commands '.\n \t\t      \" to generate.\\n\";\n-\t\tprint \"</div>\\n\"; # class=\"page_footer\"\n+\t\tprint {$output_handler} \"</div>\\n\"; # class=\"page_footer\"\n \t}\n \n \tif (-f $site_footer) {\n \t\tinsert_file($site_footer);\n \t}\n \n-\tprint qq!<script type=\"text/javascript\" src=\"$javascript\"></script>\\n!;\n+\tprint {$output_handler} qq!<script type=\"text/javascript\" src=\"$javascript\"></script>\\n!;\n \tif ($action eq 'blame_incremental') {\n-\t\tprint qq!<script type=\"text/javascript\">\\n!.\n+\t\tprint {$output_handler} qq!<script type=\"text/javascript\">\\n!.\n \t\t      qq!startBlame(\"!. href(action=>\"blame_data\", -replay=>1) .qq!\",\\n!.\n \t\t      qq!           \"!. href() .qq!\");\\n!.\n \t\t      qq!</script>\\n!;\n \t} elsif (gitweb_check_feature('javascript-actions')) {\n-\t\tprint qq!<script type=\"text/javascript\">\\n!.\n+\t\tprint {$output_handler} qq!<script type=\"text/javascript\">\\n!.\n \t\t      qq!window.onload = fixLinks;\\n!.\n \t\t      qq!</script>\\n!;\n \t}\n \n-\tprint \"</body>\\n\" .\n+\tprint {$output_handler} \"</body>\\n\" .\n \t      \"</html>\";\n }\n \n@@ -3416,6 +3423,14 @@ sub die_error {\n \tmy $error = shift || \"Internal server error\";\n \tmy $extra = shift;\n \n+\t# The output handlers for die_error need to be reset to STDOUT\n+\t# so that half the message isn't being output to random and \n+\t# half to STDOUT as expected.  This is mainly for the benefit\n+\t# of using git_header_html() and git_footer_html() since those\n+\t# internaly use the indirect print handler.\n+\t$output_handler = *STDOUT;\n+\t$output_handler_bin = *STDOUT;\n+\n \tmy %http_responses = (\n \t\t400 => '400 Bad Request',\n \t\t403 => '403 Forbidden',\n@@ -3484,12 +3499,12 @@ sub git_print_page_nav {\n \t\t$arg{$label}{'_href'} = $link;\n \t}\n \n-\tprint \"<div class=\\\"page_nav\\\">\\n\" .\n+\tprint {$output_handler} \"<div class=\\\"page_nav\\\">\\n\" .\n \t\t(join \" | \",\n \t\t map { $_ eq $current ?\n \t\t       $_ : $cgi->a({-href => ($arg{$_}{_href} ? $arg{$_}{_href} : href(%{$arg{$_}}))}, \"$_\")\n \t\t } @navs);\n-\tprint \"<br/>\\n$extra<br/>\\n\" .\n+\tprint {$output_handler} \"<br/>\\n$extra<br/>\\n\" .\n \t      \"</div>\\n\";\n }\n \n@@ -3530,14 +3545,14 @@ sub git_print_header_div {\n \t$args{'hash'} = $hash if $hash;\n \t$args{'hash_base'} = $hash_base if $hash_base;\n \n-\tprint \"<div class=\\\"header\\\">\\n\" .\n+\tprint {$output_handler} \"<div class=\\\"header\\\">\\n\" .\n \t      $cgi->a({-href => href(%args), -class => \"title\"},\n \t      $title ? $title : $action) .\n \t      \"\\n</div>\\n\";\n }\n \n sub print_local_time {\n-\tprint get_local_time(@_);\n+\tprint {$output_handler} get_local_time(@_);\n }\n \n sub get_local_time {\n@@ -3562,11 +3577,11 @@ sub git_print_authorship {\n \tmy $author = $co->{'author_name'};\n \n \tmy %ad = parse_date($co->{'author_epoch'}, $co->{'author_tz'});\n-\tprint \"<$tag class=\\\"author_date\\\">\" .\n+\tprint {$output_handler} \"<$tag class=\\\"author_date\\\">\" .\n \t      format_search_author($author, \"author\", esc_html($author)) .\n \t      \" [$ad{'rfc2822'}\";\n-\tprint_local_time(%ad) if ($opts{-localtime});\n-\tprint \"]\" . git_get_avatar($co->{'author_email'}, -pad_before => 1)\n+\tprint {$output_handler} get_local_time(%ad) if ($opts{-localtime});\n+\tprint {$output_handler} \"]\" . git_get_avatar($co->{'author_email'}, -pad_before => 1)\n \t\t  . \"</$tag>\\n\";\n }\n \n@@ -3582,7 +3597,7 @@ sub git_print_authorship_rows {\n \t@people = ('author', 'committer') unless @people;\n \tforeach my $who (@people) {\n \t\tmy %wd = parse_date($co->{\"${who}_epoch\"}, $co->{\"${who}_tz\"});\n-\t\tprint \"<tr><td>$who</td><td>\" .\n+\t\tprint {$output_handler} \"<tr><td>$who</td><td>\" .\n \t\t      format_search_author($co->{\"${who}_name\"}, $who,\n \t\t\t       esc_html($co->{\"${who}_name\"})) . \" \" .\n \t\t      format_search_author($co->{\"${who}_email\"}, $who,\n@@ -3592,8 +3607,8 @@ sub git_print_authorship_rows {\n \t\t      \"</td></tr>\\n\" .\n \t\t      \"<tr>\" .\n \t\t      \"<td></td><td> $wd{'rfc2822'}\";\n-\t\tprint_local_time(%wd);\n-\t\tprint \"</td>\" .\n+\t\tprint {$output_handler} get_local_time(%wd);\n+\t\tprint {$output_handler} \"</td>\" .\n \t\t      \"</tr>\\n\";\n \t}\n }\n@@ -3603,11 +3618,10 @@ sub git_print_page_path {\n \tmy $type = shift;\n \tmy $hb = shift;\n \n-\n-\tprint \"<div class=\\\"page_path\\\">\";\n-\tprint $cgi->a({-href => href(action=>\"tree\", hash_base=>$hb),\n+\tprint {$output_handler} \"<div class=\\\"page_path\\\">\";\n+\tprint {$output_handler} $cgi->a({-href => href(action=>\"tree\", hash_base=>$hb),\n \t              -title => 'tree root'}, to_utf8(\"[$project]\"));\n-\tprint \" / \";\n+\tprint {$output_handler} \" / \";\n \tif (defined $name) {\n \t\tmy @dirname = split '/', $name;\n \t\tmy $basename = pop @dirname;\n@@ -3615,25 +3629,25 @@ sub git_print_page_path {\n \n \t\tforeach my $dir (@dirname) {\n \t\t\t$fullname .= ($fullname ? '/' : '') . $dir;\n-\t\t\tprint $cgi->a({-href => href(action=>\"tree\", file_name=>$fullname,\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"tree\", file_name=>$fullname,\n \t\t\t                             hash_base=>$hb),\n \t\t\t              -title => $fullname}, esc_path($dir));\n-\t\t\tprint \" / \";\n+\t\t\tprint {$output_handler} \" / \";\n \t\t}\n \t\tif (defined $type && $type eq 'blob') {\n-\t\t\tprint $cgi->a({-href => href(action=>\"blob_plain\", file_name=>$file_name,\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob_plain\", file_name=>$file_name,\n \t\t\t                             hash_base=>$hb),\n \t\t\t              -title => $name}, esc_path($basename));\n \t\t} elsif (defined $type && $type eq 'tree') {\n-\t\t\tprint $cgi->a({-href => href(action=>\"tree\", file_name=>$file_name,\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"tree\", file_name=>$file_name,\n \t\t\t                             hash_base=>$hb),\n \t\t\t              -title => $name}, esc_path($basename));\n-\t\t\tprint \" / \";\n+\t\t\tprint {$output_handler} \" / \";\n \t\t} else {\n-\t\t\tprint esc_path($basename);\n+\t\t\tprint {$output_handler} esc_path($basename);\n \t\t}\n \t}\n-\tprint \"<br/></div>\\n\";\n+\tprint {$output_handler} \"<br/></div>\\n\";\n }\n \n sub git_print_log {\n@@ -3657,7 +3671,7 @@ sub git_print_log {\n \t\t\t$signoff = 1;\n \t\t\t$empty = 0;\n \t\t\tif (! $opts{'-remove_signoff'}) {\n-\t\t\t\tprint \"<span class=\\\"signoff\\\">\" . esc_html($line) . \"</span><br/>\\n\";\n+\t\t\t\tprint {$output_handler} \"<span class=\\\"signoff\\\">\" . esc_html($line) . \"</span><br/>\\n\";\n \t\t\t\tnext;\n \t\t\t} else {\n \t\t\t\t# remove signoff lines\n@@ -3676,12 +3690,12 @@ sub git_print_log {\n \t\t\t$empty = 0;\n \t\t}\n \n-\t\tprint format_log_line_html($line) . \"<br/>\\n\";\n+\t\tprint {$output_handler} format_log_line_html($line) . \"<br/>\\n\";\n \t}\n \n \tif ($opts{'-final_empty_line'}) {\n \t\t# end with single empty line\n-\t\tprint \"<br/>\\n\" unless $empty;\n+\t\tprint {$output_handler} \"<br/>\\n\" unless $empty;\n \t}\n }\n \n@@ -3754,12 +3768,12 @@ sub git_print_tree_entry {\n \t# the mode of the entry, list is the name of the entry, an href,\n \t# and link is the action links of the entry.\n \n-\tprint \"<td class=\\\"mode\\\">\" . mode_str($t->{'mode'}) . \"</td>\\n\";\n+\tprint {$output_handler} \"<td class=\\\"mode\\\">\" . mode_str($t->{'mode'}) . \"</td>\\n\";\n \tif (exists $t->{'size'}) {\n-\t\tprint \"<td class=\\\"size\\\">$t->{'size'}</td>\\n\";\n+\t\tprint {$output_handler} \"<td class=\\\"size\\\">$t->{'size'}</td>\\n\";\n \t}\n \tif ($t->{'type'} eq \"blob\") {\n-\t\tprint \"<td class=\\\"list\\\">\" .\n+\t\tprint {$output_handler} \"<td class=\\\"list\\\">\" .\n \t\t\t$cgi->a({-href => href(action=>\"blob\", hash=>$t->{'hash'},\n \t\t\t                       file_name=>\"$basedir$t->{'name'}\", %base_key),\n \t\t\t        -class => \"list\"}, esc_path($t->{'name'}));\n@@ -3768,71 +3782,71 @@ sub git_print_tree_entry {\n \t\t\tif ($link_target) {\n \t\t\t\tmy $norm_target = normalize_link_target($link_target, $basedir);\n \t\t\t\tif (defined $norm_target) {\n-\t\t\t\t\tprint \" -> \" .\n+\t\t\t\t\tprint {$output_handler} \" -> \" .\n \t\t\t\t\t      $cgi->a({-href => href(action=>\"object\", hash_base=>$hash_base,\n \t\t\t\t\t                             file_name=>$norm_target),\n \t\t\t\t\t               -title => $norm_target}, esc_path($link_target));\n \t\t\t\t} else {\n-\t\t\t\t\tprint \" -> \" . esc_path($link_target);\n+\t\t\t\t\tprint {$output_handler} \" -> \" . esc_path($link_target);\n \t\t\t\t}\n \t\t\t}\n \t\t}\n-\t\tprint \"</td>\\n\";\n-\t\tprint \"<td class=\\\"link\\\">\";\n-\t\tprint $cgi->a({-href => href(action=>\"blob\", hash=>$t->{'hash'},\n+\t\tprint {$output_handler} \"</td>\\n\";\n+\t\tprint {$output_handler} \"<td class=\\\"link\\\">\";\n+\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\", hash=>$t->{'hash'},\n \t\t                             file_name=>\"$basedir$t->{'name'}\", %base_key)},\n \t\t              \"blob\");\n \t\tif ($have_blame) {\n-\t\t\tprint \" | \" .\n+\t\t\tprint {$output_handler} \" | \" .\n \t\t\t      $cgi->a({-href => href(action=>\"blame\", hash=>$t->{'hash'},\n \t\t\t                             file_name=>\"$basedir$t->{'name'}\", %base_key)},\n \t\t\t              \"blame\");\n \t\t}\n \t\tif (defined $hash_base) {\n-\t\t\tprint \" | \" .\n+\t\t\tprint {$output_handler} \" | \" .\n \t\t\t      $cgi->a({-href => href(action=>\"history\", hash_base=>$hash_base,\n \t\t\t                             hash=>$t->{'hash'}, file_name=>\"$basedir$t->{'name'}\")},\n \t\t\t              \"history\");\n \t\t}\n-\t\tprint \" | \" .\n+\t\tprint {$output_handler} \" | \" .\n \t\t\t$cgi->a({-href => href(action=>\"blob_plain\", hash_base=>$hash_base,\n \t\t\t                       file_name=>\"$basedir$t->{'name'}\")},\n \t\t\t        \"raw\");\n-\t\tprint \"</td>\\n\";\n+\t\tprint {$output_handler} \"</td>\\n\";\n \n \t} elsif ($t->{'type'} eq \"tree\") {\n-\t\tprint \"<td class=\\\"list\\\">\";\n-\t\tprint $cgi->a({-href => href(action=>\"tree\", hash=>$t->{'hash'},\n+\t\tprint {$output_handler} \"<td class=\\\"list\\\">\";\n+\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"tree\", hash=>$t->{'hash'},\n \t\t                             file_name=>\"$basedir$t->{'name'}\",\n \t\t                             %base_key)},\n \t\t              esc_path($t->{'name'}));\n-\t\tprint \"</td>\\n\";\n-\t\tprint \"<td class=\\\"link\\\">\";\n-\t\tprint $cgi->a({-href => href(action=>\"tree\", hash=>$t->{'hash'},\n+\t\tprint {$output_handler} \"</td>\\n\";\n+\t\tprint {$output_handler} \"<td class=\\\"link\\\">\";\n+\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"tree\", hash=>$t->{'hash'},\n \t\t                             file_name=>\"$basedir$t->{'name'}\",\n \t\t                             %base_key)},\n \t\t              \"tree\");\n \t\tif (defined $hash_base) {\n-\t\t\tprint \" | \" .\n+\t\t\tprint {$output_handler} \" | \" .\n \t\t\t      $cgi->a({-href => href(action=>\"history\", hash_base=>$hash_base,\n \t\t\t                             file_name=>\"$basedir$t->{'name'}\")},\n \t\t\t              \"history\");\n \t\t}\n-\t\tprint \"</td>\\n\";\n+\t\tprint {$output_handler} \"</td>\\n\";\n \t} else {\n \t\t# unknown object: we can only present history for it\n \t\t# (this includes 'commit' object, i.e. submodule support)\n-\t\tprint \"<td class=\\\"list\\\">\" .\n+\t\tprint {$output_handler} \"<td class=\\\"list\\\">\" .\n \t\t      esc_path($t->{'name'}) .\n \t\t      \"</td>\\n\";\n-\t\tprint \"<td class=\\\"link\\\">\";\n+\t\tprint {$output_handler} \"<td class=\\\"link\\\">\";\n \t\tif (defined $hash_base) {\n-\t\t\tprint $cgi->a({-href => href(action=>\"history\",\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"history\",\n \t\t\t                             hash_base=>$hash_base,\n \t\t\t                             file_name=>\"$basedir$t->{'name'}\")},\n \t\t\t              \"history\");\n \t\t}\n-\t\tprint \"</td>\\n\";\n+\t\tprint {$output_handler} \"</td>\\n\";\n \t}\n }\n \n@@ -3879,13 +3893,13 @@ sub git_difftree_body {\n \tmy ($difftree, $hash, @parents) = @_;\n \tmy ($parent) = $parents[0];\n \tmy $have_blame = gitweb_check_feature('blame');\n-\tprint \"<div class=\\\"list_head\\\">\\n\";\n+\tprint {$output_handler} \"<div class=\\\"list_head\\\">\\n\";\n \tif ($#{$difftree} > 10) {\n-\t\tprint(($#{$difftree} + 1) . \" files changed:\\n\");\n+\t\tprint {$output_handler} (($#{$difftree} + 1) . \" files changed:\\n\");\n \t}\n-\tprint \"</div>\\n\";\n+\tprint {$output_handler} \"</div>\\n\";\n \n-\tprint \"<table class=\\\"\" .\n+\tprint {$output_handler} \"<table class=\\\"\" .\n \t      (@parents > 1 ? \"combined \" : \"\") .\n \t      \"diff_tree\\\">\\n\";\n \n@@ -3893,11 +3907,11 @@ sub git_difftree_body {\n \tmy $has_header = @$difftree && @parents > 1 && $action eq 'commitdiff';\n \tif ($has_header) {\n \t\t# table header\n-\t\tprint \"<thead><tr>\\n\" .\n+\t\tprint {$output_handler} \"<thead><tr>\\n\" .\n \t\t       \"<th></th><th></th>\\n\"; # filename, patchN link\n \t\tfor (my $i = 0; $i < @parents; $i++) {\n \t\t\tmy $par = $parents[$i];\n-\t\t\tprint \"<th>\" .\n+\t\t\tprint {$output_handler} \"<th>\" .\n \t\t\t      $cgi->a({-href => href(action=>\"commitdiff\",\n \t\t\t                             hash=>$hash, hash_parent=>$par),\n \t\t\t               -title => 'commitdiff to parent number ' .\n@@ -3905,7 +3919,7 @@ sub git_difftree_body {\n \t\t\t              $i+1) .\n \t\t\t      \"&nbsp;</th>\\n\";\n \t\t}\n-\t\tprint \"</tr></thead>\\n<tbody>\\n\";\n+\t\tprint {$output_handler} \"</tr></thead>\\n<tbody>\\n\";\n \t}\n \n \tmy $alternate = 1;\n@@ -3914,9 +3928,9 @@ sub git_difftree_body {\n \t\tmy $diff = parsed_difftree_line($line);\n \n \t\tif ($alternate) {\n-\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n-\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t}\n \t\t$alternate ^= 1;\n \n@@ -3927,14 +3941,14 @@ sub git_difftree_body {\n \n \t\t\tif (!is_deleted($diff)) {\n \t\t\t\t# file exists in the result (child) commit\n-\t\t\t\tprint \"<td>\" .\n+\t\t\t\tprint {$output_handler} \"<td>\" .\n \t\t\t\t      $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n \t\t\t\t                             file_name=>$diff->{'to_file'},\n \t\t\t\t                             hash_base=>$hash),\n \t\t\t\t              -class => \"list\"}, esc_path($diff->{'to_file'})) .\n \t\t\t\t      \"</td>\\n\";\n \t\t\t} else {\n-\t\t\t\tprint \"<td>\" .\n+\t\t\t\tprint {$output_handler} \"<td>\" .\n \t\t\t\t      esc_path($diff->{'to_file'}) .\n \t\t\t\t      \"</td>\\n\";\n \t\t\t}\n@@ -3942,7 +3956,7 @@ sub git_difftree_body {\n \t\t\tif ($action eq 'commitdiff') {\n \t\t\t\t# link to patch\n \t\t\t\t$patchno++;\n-\t\t\t\tprint \"<td class=\\\"link\\\">\" .\n+\t\t\t\tprint {$output_handler} \"<td class=\\\"link\\\">\" .\n \t\t\t\t      $cgi->a({-href => \"#patch$patchno\"}, \"patch\") .\n \t\t\t\t      \" | \" .\n \t\t\t\t      \"</td>\\n\";\n@@ -3960,9 +3974,9 @@ sub git_difftree_body {\n \t\t\t\t$not_deleted ||= ($status ne 'D');\n \n \t\t\t\tif ($status eq 'A') {\n-\t\t\t\t\tprint \"<td  class=\\\"link\\\" align=\\\"right\\\"> | </td>\\n\";\n+\t\t\t\t\tprint {$output_handler} \"<td  class=\\\"link\\\" align=\\\"right\\\"> | </td>\\n\";\n \t\t\t\t} elsif ($status eq 'D') {\n-\t\t\t\t\tprint \"<td class=\\\"link\\\">\" .\n+\t\t\t\t\tprint {$output_handler} \"<td class=\\\"link\\\">\" .\n \t\t\t\t\t      $cgi->a({-href => href(action=>\"blob\",\n \t\t\t\t\t                             hash_base=>$hash,\n \t\t\t\t\t                             hash=>$from_hash,\n@@ -3971,11 +3985,11 @@ sub git_difftree_body {\n \t\t\t\t\t      \" | </td>\\n\";\n \t\t\t\t} else {\n \t\t\t\t\tif ($diff->{'to_id'} eq $from_hash) {\n-\t\t\t\t\t\tprint \"<td class=\\\"link nochange\\\">\";\n+\t\t\t\t\t\tprint {$output_handler} \"<td class=\\\"link nochange\\\">\";\n \t\t\t\t\t} else {\n-\t\t\t\t\t\tprint \"<td class=\\\"link\\\">\";\n+\t\t\t\t\t\tprint {$output_handler} \"<td class=\\\"link\\\">\";\n \t\t\t\t\t}\n-\t\t\t\t\tprint $cgi->a({-href => href(action=>\"blobdiff\",\n+\t\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blobdiff\",\n \t\t\t\t\t                             hash=>$diff->{'to_id'},\n \t\t\t\t\t                             hash_parent=>$from_hash,\n \t\t\t\t\t                             hash_base=>$hash,\n@@ -3987,24 +4001,24 @@ sub git_difftree_body {\n \t\t\t\t}\n \t\t\t}\n \n-\t\t\tprint \"<td class=\\\"link\\\">\";\n+\t\t\tprint {$output_handler} \"<td class=\\\"link\\\">\";\n \t\t\tif ($not_deleted) {\n-\t\t\t\tprint $cgi->a({-href => href(action=>\"blob\",\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\",\n \t\t\t\t                             hash=>$diff->{'to_id'},\n \t\t\t\t                             file_name=>$diff->{'to_file'},\n \t\t\t\t                             hash_base=>$hash)},\n \t\t\t\t              \"blob\");\n-\t\t\t\tprint \" | \" if ($has_history);\n+\t\t\t\tprint {$output_handler} \" | \" if ($has_history);\n \t\t\t}\n \t\t\tif ($has_history) {\n-\t\t\t\tprint $cgi->a({-href => href(action=>\"history\",\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"history\",\n \t\t\t\t                             file_name=>$diff->{'to_file'},\n \t\t\t\t                             hash_base=>$hash)},\n \t\t\t\t              \"history\");\n \t\t\t}\n-\t\t\tprint \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"</td>\\n\";\n \n-\t\t\tprint \"</tr>\\n\";\n+\t\t\tprint {$output_handler} \"</tr>\\n\";\n \t\t\tnext; # instead of 'else' clause, to avoid extra indent\n \t\t}\n \t\t# else ordinary diff\n@@ -4030,51 +4044,51 @@ sub git_difftree_body {\n \t\t\tmy $mode_chng = \"<span class=\\\"file_status new\\\">[new $to_file_type\";\n \t\t\t$mode_chng   .= \" with mode: $to_mode_str\" if $to_mode_str;\n \t\t\t$mode_chng   .= \"]</span>\";\n-\t\t\tprint \"<td>\";\n-\t\t\tprint $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n+\t\t\tprint {$output_handler} \"<td>\";\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n \t\t\t                             hash_base=>$hash, file_name=>$diff->{'file'}),\n \t\t\t              -class => \"list\"}, esc_path($diff->{'file'}));\n-\t\t\tprint \"</td>\\n\";\n-\t\t\tprint \"<td>$mode_chng</td>\\n\";\n-\t\t\tprint \"<td class=\\\"link\\\">\";\n+\t\t\tprint {$output_handler} \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"<td>$mode_chng</td>\\n\";\n+\t\t\tprint {$output_handler} \"<td class=\\\"link\\\">\";\n \t\t\tif ($action eq 'commitdiff') {\n \t\t\t\t# link to patch\n \t\t\t\t$patchno++;\n-\t\t\t\tprint $cgi->a({-href => \"#patch$patchno\"}, \"patch\");\n-\t\t\t\tprint \" | \";\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => \"#patch$patchno\"}, \"patch\");\n+\t\t\t\tprint {$output_handler} \" | \";\n \t\t\t}\n-\t\t\tprint $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n \t\t\t                             hash_base=>$hash, file_name=>$diff->{'file'})},\n \t\t\t              \"blob\");\n-\t\t\tprint \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"</td>\\n\";\n \n \t\t} elsif ($diff->{'status'} eq \"D\") { # deleted\n \t\t\tmy $mode_chng = \"<span class=\\\"file_status deleted\\\">[deleted $from_file_type]</span>\";\n-\t\t\tprint \"<td>\";\n-\t\t\tprint $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'from_id'},\n+\t\t\tprint {$output_handler} \"<td>\";\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'from_id'},\n \t\t\t                             hash_base=>$parent, file_name=>$diff->{'file'}),\n \t\t\t               -class => \"list\"}, esc_path($diff->{'file'}));\n-\t\t\tprint \"</td>\\n\";\n-\t\t\tprint \"<td>$mode_chng</td>\\n\";\n-\t\t\tprint \"<td class=\\\"link\\\">\";\n+\t\t\tprint {$output_handler} \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"<td>$mode_chng</td>\\n\";\n+\t\t\tprint {$output_handler} \"<td class=\\\"link\\\">\";\n \t\t\tif ($action eq 'commitdiff') {\n \t\t\t\t# link to patch\n \t\t\t\t$patchno++;\n-\t\t\t\tprint $cgi->a({-href => \"#patch$patchno\"}, \"patch\");\n-\t\t\t\tprint \" | \";\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => \"#patch$patchno\"}, \"patch\");\n+\t\t\t\tprint {$output_handler} \" | \";\n \t\t\t}\n-\t\t\tprint $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'from_id'},\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'from_id'},\n \t\t\t                             hash_base=>$parent, file_name=>$diff->{'file'})},\n \t\t\t              \"blob\") . \" | \";\n \t\t\tif ($have_blame) {\n-\t\t\t\tprint $cgi->a({-href => href(action=>\"blame\", hash_base=>$parent,\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blame\", hash_base=>$parent,\n \t\t\t\t                             file_name=>$diff->{'file'})},\n \t\t\t\t              \"blame\") . \" | \";\n \t\t\t}\n-\t\t\tprint $cgi->a({-href => href(action=>\"history\", hash_base=>$parent,\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"history\", hash_base=>$parent,\n \t\t\t                             file_name=>$diff->{'file'})},\n \t\t\t              \"history\");\n-\t\t\tprint \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"</td>\\n\";\n \n \t\t} elsif ($diff->{'status'} eq \"M\" || $diff->{'status'} eq \"T\") { # modified, or type changed\n \t\t\tmy $mode_chnge = \"\";\n@@ -4092,39 +4106,39 @@ sub git_difftree_body {\n \t\t\t\t}\n \t\t\t\t$mode_chnge .= \"]</span>\\n\";\n \t\t\t}\n-\t\t\tprint \"<td>\";\n-\t\t\tprint $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n+\t\t\tprint {$output_handler} \"<td>\";\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n \t\t\t                             hash_base=>$hash, file_name=>$diff->{'file'}),\n \t\t\t              -class => \"list\"}, esc_path($diff->{'file'}));\n-\t\t\tprint \"</td>\\n\";\n-\t\t\tprint \"<td>$mode_chnge</td>\\n\";\n-\t\t\tprint \"<td class=\\\"link\\\">\";\n+\t\t\tprint {$output_handler} \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"<td>$mode_chnge</td>\\n\";\n+\t\t\tprint {$output_handler} \"<td class=\\\"link\\\">\";\n \t\t\tif ($action eq 'commitdiff') {\n \t\t\t\t# link to patch\n \t\t\t\t$patchno++;\n-\t\t\t\tprint $cgi->a({-href => \"#patch$patchno\"}, \"patch\") .\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => \"#patch$patchno\"}, \"patch\") .\n \t\t\t\t      \" | \";\n \t\t\t} elsif ($diff->{'to_id'} ne $diff->{'from_id'}) {\n \t\t\t\t# \"commit\" view and modified file (not onlu mode changed)\n-\t\t\t\tprint $cgi->a({-href => href(action=>\"blobdiff\",\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blobdiff\",\n \t\t\t\t                             hash=>$diff->{'to_id'}, hash_parent=>$diff->{'from_id'},\n \t\t\t\t                             hash_base=>$hash, hash_parent_base=>$parent,\n \t\t\t\t                             file_name=>$diff->{'file'})},\n \t\t\t\t              \"diff\") .\n \t\t\t\t      \" | \";\n \t\t\t}\n-\t\t\tprint $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n \t\t\t                             hash_base=>$hash, file_name=>$diff->{'file'})},\n \t\t\t               \"blob\") . \" | \";\n \t\t\tif ($have_blame) {\n-\t\t\t\tprint $cgi->a({-href => href(action=>\"blame\", hash_base=>$hash,\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blame\", hash_base=>$hash,\n \t\t\t\t                             file_name=>$diff->{'file'})},\n \t\t\t\t              \"blame\") . \" | \";\n \t\t\t}\n-\t\t\tprint $cgi->a({-href => href(action=>\"history\", hash_base=>$hash,\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"history\", hash_base=>$hash,\n \t\t\t                             file_name=>$diff->{'file'})},\n \t\t\t              \"history\");\n-\t\t\tprint \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"</td>\\n\";\n \n \t\t} elsif ($diff->{'status'} eq \"R\" || $diff->{'status'} eq \"C\") { # renamed or copied\n \t\t\tmy %status_name = ('R' => 'moved', 'C' => 'copied');\n@@ -4134,7 +4148,7 @@ sub git_difftree_body {\n \t\t\t\t# mode also for directories, so we cannot use $to_mode_str\n \t\t\t\t$mode_chng = sprintf(\", mode: %04o\", $to_mode_oct & 0777);\n \t\t\t}\n-\t\t\tprint \"<td>\" .\n+\t\t\tprint {$output_handler} \"<td>\" .\n \t\t\t      $cgi->a({-href => href(action=>\"blob\", hash_base=>$hash,\n \t\t\t                             hash=>$diff->{'to_id'}, file_name=>$diff->{'to_file'}),\n \t\t\t              -class => \"list\"}, esc_path($diff->{'to_file'})) . \"</td>\\n\" .\n@@ -4147,35 +4161,35 @@ sub git_difftree_body {\n \t\t\tif ($action eq 'commitdiff') {\n \t\t\t\t# link to patch\n \t\t\t\t$patchno++;\n-\t\t\t\tprint $cgi->a({-href => \"#patch$patchno\"}, \"patch\") .\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => \"#patch$patchno\"}, \"patch\") .\n \t\t\t\t      \" | \";\n \t\t\t} elsif ($diff->{'to_id'} ne $diff->{'from_id'}) {\n \t\t\t\t# \"commit\" view and modified file (not only pure rename or copy)\n-\t\t\t\tprint $cgi->a({-href => href(action=>\"blobdiff\",\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blobdiff\",\n \t\t\t\t                             hash=>$diff->{'to_id'}, hash_parent=>$diff->{'from_id'},\n \t\t\t\t                             hash_base=>$hash, hash_parent_base=>$parent,\n \t\t\t\t                             file_name=>$diff->{'to_file'}, file_parent=>$diff->{'from_file'})},\n \t\t\t\t              \"diff\") .\n \t\t\t\t      \" | \";\n \t\t\t}\n-\t\t\tprint $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\", hash=>$diff->{'to_id'},\n \t\t\t                             hash_base=>$parent, file_name=>$diff->{'to_file'})},\n \t\t\t              \"blob\") . \" | \";\n \t\t\tif ($have_blame) {\n-\t\t\t\tprint $cgi->a({-href => href(action=>\"blame\", hash_base=>$hash,\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blame\", hash_base=>$hash,\n \t\t\t\t                             file_name=>$diff->{'to_file'})},\n \t\t\t\t              \"blame\") . \" | \";\n \t\t\t}\n-\t\t\tprint $cgi->a({-href => href(action=>\"history\", hash_base=>$hash,\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"history\", hash_base=>$hash,\n \t\t\t                            file_name=>$diff->{'to_file'})},\n \t\t\t              \"history\");\n-\t\t\tprint \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"</td>\\n\";\n \n \t\t} # we should not encounter Unmerged (U) or Unknown (X) status\n-\t\tprint \"</tr>\\n\";\n+\t\tprint {$output_handler} \"</tr>\\n\";\n \t}\n-\tprint \"</tbody>\" if $has_header;\n-\tprint \"</table>\\n\";\n+\tprint {$output_handler} \"</tbody>\" if $has_header;\n+\tprint {$output_handler} \"</table>\\n\";\n }\n \n sub git_patchset_body {\n@@ -4190,7 +4204,7 @@ sub git_patchset_body {\n \tmy $to_name;\n \tmy (%from, %to);\n \n-\tprint \"<div class=\\\"patchset\\\">\\n\";\n+\tprint {$output_handler} \"<div class=\\\"patchset\\\">\\n\";\n \n \t# skip to first patch\n \twhile ($patch_line = <$fd>) {\n@@ -4218,7 +4232,7 @@ sub git_patchset_body {\n \t\t# and parse raw git-diff line if needed\n \t\tif (is_patch_split($diffinfo, { 'to_file' => $to_name })) {\n \t\t\t# this is continuation of a split patch\n-\t\t\tprint \"<div class=\\\"patch cont\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"patch cont\\\">\\n\";\n \t\t} else {\n \t\t\t# advance raw git-diff output if needed\n \t\t\t$patch_idx++ if defined $diffinfo;\n@@ -4230,7 +4244,7 @@ sub git_patchset_body {\n \t\t\t# find which patch (using pathname of result) we are at now;\n \t\t\tif ($is_combined) {\n \t\t\t\twhile ($to_name ne $diffinfo->{'to_file'}) {\n-\t\t\t\t\tprint \"<div class=\\\"patch\\\" id=\\\"patch\". ($patch_idx+1) .\"\\\">\\n\" .\n+\t\t\t\t\tprint {$output_handler} \"<div class=\\\"patch\\\" id=\\\"patch\". ($patch_idx+1) .\"\\\">\\n\" .\n \t\t\t\t\t      format_diff_cc_simplified($diffinfo, @hash_parents) .\n \t\t\t\t\t      \"</div>\\n\";  # class=\"patch\"\n \n@@ -4247,7 +4261,7 @@ sub git_patchset_body {\n \n \t\t\t# this is first patch for raw difftree line with $patch_idx index\n \t\t\t# we index @$difftree array from 0, but number patches from 1\n-\t\t\tprint \"<div class=\\\"patch\\\" id=\\\"patch\". ($patch_idx+1) .\"\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"patch\\\" id=\\\"patch\". ($patch_idx+1) .\"\\\">\\n\";\n \t\t}\n \n \t\t# git diff header\n@@ -4255,25 +4269,25 @@ sub git_patchset_body {\n \t\t#assert($patch_line !~ m!$/$!) if DEBUG; # is chomp-ed\n \t\t$patch_number++;\n \t\t# print \"git diff\" header\n-\t\tprint format_git_diff_header_line($patch_line, $diffinfo,\n+\t\tprint {$output_handler} format_git_diff_header_line($patch_line, $diffinfo,\n \t\t                                  \\%from, \\%to);\n \n \t\t# print extended diff header\n-\t\tprint \"<div class=\\\"diff extended_header\\\">\\n\";\n+\t\tprint {$output_handler} \"<div class=\\\"diff extended_header\\\">\\n\";\n \tEXTENDED_HEADER:\n \t\twhile ($patch_line = <$fd>) {\n \t\t\tchomp $patch_line;\n \n \t\t\tlast EXTENDED_HEADER if ($patch_line =~ m/^--- |^diff /);\n \n-\t\t\tprint format_extended_diff_header_line($patch_line, $diffinfo,\n+\t\t\tprint {$output_handler} format_extended_diff_header_line($patch_line, $diffinfo,\n \t\t\t                                       \\%from, \\%to);\n \t\t}\n-\t\tprint \"</div>\\n\"; # class=\"diff extended_header\"\n+\t\tprint {$output_handler} \"</div>\\n\"; # class=\"diff extended_header\"\n \n \t\t# from-file/to-file diff header\n \t\tif (! $patch_line) {\n-\t\t\tprint \"</div>\\n\"; # class=\"patch\"\n+\t\t\tprint {$output_handler} \"</div>\\n\"; # class=\"patch\"\n \t\t\tlast PATCH;\n \t\t}\n \t\tnext PATCH if ($patch_line =~ m/^diff /);\n@@ -4284,7 +4298,7 @@ sub git_patchset_body {\n \t\tchomp $patch_line;\n \t\t#assert($patch_line =~ m/^\\+\\+\\+/) if DEBUG;\n \n-\t\tprint format_diff_from_to_header($last_patch_line, $patch_line,\n+\t\tprint {$output_handler} format_diff_from_to_header($last_patch_line, $patch_line,\n \t\t                                 $diffinfo, \\%from, \\%to,\n \t\t                                 @hash_parents);\n \n@@ -4295,11 +4309,11 @@ sub git_patchset_body {\n \n \t\t\tnext PATCH if ($patch_line =~ m/^diff /);\n \n-\t\t\tprint format_diff_line($patch_line, \\%from, \\%to);\n+\t\t\tprint {$output_handler} format_diff_line($patch_line, \\%from, \\%to);\n \t\t}\n \n \t} continue {\n-\t\tprint \"</div>\\n\"; # class=\"patch\"\n+\t\tprint {$output_handler} \"</div>\\n\"; # class=\"patch\"\n \t}\n \n \t# for compact combined (--cc) format, with chunk and patch simpliciaction\n@@ -4311,7 +4325,7 @@ sub git_patchset_body {\n \t\t$diffinfo = parsed_difftree_line($difftree->[$patch_idx]);\n \n \t\t# generate anchor for \"patch\" links in difftree / whatchanged part\n-\t\tprint \"<div class=\\\"patch\\\" id=\\\"patch\". ($patch_idx+1) .\"\\\">\\n\" .\n+\t\tprint {$output_handler} \"<div class=\\\"patch\\\" id=\\\"patch\". ($patch_idx+1) .\"\\\">\\n\" .\n \t\t      format_diff_cc_simplified($diffinfo, @hash_parents) .\n \t\t      \"</div>\\n\";  # class=\"patch\"\n \n@@ -4320,13 +4334,13 @@ sub git_patchset_body {\n \n \tif ($patch_number == 0) {\n \t\tif (@hash_parents > 1) {\n-\t\t\tprint \"<div class=\\\"diff nodifferences\\\">Trivial merge</div>\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"diff nodifferences\\\">Trivial merge</div>\\n\";\n \t\t} else {\n-\t\t\tprint \"<div class=\\\"diff nodifferences\\\">No differences found</div>\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"diff nodifferences\\\">No differences found</div>\\n\";\n \t\t}\n \t}\n \n-\tprint \"</div>\\n\"; # class=\"patchset\"\n+\tprint {$output_handler} \"</div>\\n\"; # class=\"patchset\"\n }\n \n # . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .\n@@ -4375,7 +4389,7 @@ sub fill_project_list_info {\n # print 'sort by' <th> element, generating 'sort by $name' replay link\n # if that order is not selected\n sub print_sort_th {\n-\tprint get_sort_th(@_);\n+\tprint {$output_handler} get_sort_th(@_);\n }\n \n sub get_sort_th {\n@@ -4428,20 +4442,20 @@ sub git_project_list_body {\n \t\t\t}\n \t\t}\n \t\tmy $cloud = git_populate_project_tagcloud(\\%ctags);\n-\t\tprint git_show_project_tagcloud($cloud, 64);\n+\t\tprint {$output_handler} git_show_project_tagcloud($cloud, 64);\n \t}\n \n-\tprint \"<table class=\\\"project_list\\\">\\n\";\n+\tprint {$output_handler} \"<table class=\\\"project_list\\\">\\n\";\n \tunless ($no_header) {\n-\t\tprint \"<tr>\\n\";\n+\t\tprint {$output_handler} \"<tr>\\n\";\n \t\tif ($check_forks) {\n-\t\t\tprint \"<th></th>\\n\";\n+\t\t\tprint {$output_handler} \"<th></th>\\n\";\n \t\t}\n-\t\tprint_sort_th('project', $order, 'Project');\n-\t\tprint_sort_th('descr', $order, 'Description');\n-\t\tprint_sort_th('owner', $order, 'Owner');\n-\t\tprint_sort_th('age', $order, 'Last Change');\n-\t\tprint \"<th></th>\\n\" . # for links\n+\t\tprint {$output_handler} get_sort_th('project', $order, 'Project');\n+\t\tprint {$output_handler} get_sort_th('descr', $order, 'Description');\n+\t\tprint {$output_handler} get_sort_th('owner', $order, 'Owner');\n+\t\tprint {$output_handler} get_sort_th('age', $order, 'Last Change');\n+\t\tprint {$output_handler} \"<th></th>\\n\" . # for links\n \t\t      \"</tr>\\n\";\n \t}\n \tmy $alternate = 1;\n@@ -4461,26 +4475,26 @@ sub git_project_list_body {\n \t\t}\n \n \t\tif ($alternate) {\n-\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n-\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t}\n \t\t$alternate ^= 1;\n \t\tif ($check_forks) {\n-\t\t\tprint \"<td>\";\n+\t\t\tprint {$output_handler} \"<td>\";\n \t\t\tif ($pr->{'forks'}) {\n-\t\t\t\tprint \"<!-- $pr->{'forks'} -->\\n\";\n-\t\t\t\tprint $cgi->a({-href => href(project=>$pr->{'path'}, action=>\"forks\")}, \"+\");\n+\t\t\t\tprint {$output_handler} \"<!-- $pr->{'forks'} -->\\n\";\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(project=>$pr->{'path'}, action=>\"forks\")}, \"+\");\n \t\t\t}\n-\t\t\tprint \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"</td>\\n\";\n \t\t}\n-\t\tprint \"<td>\" . $cgi->a({-href => href(project=>$pr->{'path'}, action=>\"summary\"),\n-\t\t                        -class => \"list\"}, esc_html($pr->{'path'})) . \"</td>\\n\" .\n+\t\tprint {$output_handler} \"<td>\" . $cgi->a({-href => href(project=>$pr->{'path'}, action=>\"summary\"),\n+\t\t                        -class => \"list\"}, esc_html($pr->{'path'})) .\"</td>\\n\".\n \t\t      \"<td>\" . $cgi->a({-href => href(project=>$pr->{'path'}, action=>\"summary\"),\n \t\t                        -class => \"list\", -title => $pr->{'descr_long'}},\n \t\t                        esc_html($pr->{'descr'})) . \"</td>\\n\" .\n \t\t      \"<td><i>\" . chop_and_escape_str($pr->{'owner'}, 15) . \"</i></td>\\n\";\n-\t\tprint \"<td class=\\\"\". age_class($pr->{'age'}) . \"\\\">\" .\n+\t\tprint {$output_handler} \"<td class=\\\"\". age_class($pr->{'age'}) . \"\\\">\" .\n \t\t      (defined $pr->{'age_string'} ? $pr->{'age_string'} : \"No commits\") . \"</td>\\n\" .\n \t\t      \"<td class=\\\"link\\\">\" .\n \t\t      $cgi->a({-href => href(project=>$pr->{'path'}, action=>\"summary\")}, \"summary\")   . \" | \" .\n@@ -4492,14 +4506,14 @@ sub git_project_list_body {\n \t\t      \"</tr>\\n\";\n \t}\n \tif (defined $extra) {\n-\t\tprint \"<tr>\\n\";\n+\t\tprint {$output_handler} \"<tr>\\n\";\n \t\tif ($check_forks) {\n-\t\t\tprint \"<td></td>\\n\";\n+\t\t\tprint {$output_handler} \"<td></td>\\n\";\n \t\t}\n-\t\tprint \"<td colspan=\\\"5\\\">$extra</td>\\n\" .\n+\t\tprint {$output_handler} \"<td colspan=\\\"5\\\">$extra</td>\\n\" .\n \t\t      \"</tr>\\n\";\n \t}\n-\tprint \"</table>\\n\";\n+\tprint {$output_handler} \"</table>\\n\";\n }\n \n sub git_log_body {\n@@ -4519,7 +4533,7 @@ sub git_log_body {\n \t\t               \"<span class=\\\"age\\\">$co{'age_string'}</span>\" .\n \t\t               esc_html($co{'title'}) . $ref,\n \t\t               $commit);\n-\t\tprint \"<div class=\\\"title_text\\\">\\n\" .\n+\t\tprint {$output_handler} \"<div class=\\\"title_text\\\">\\n\" .\n \t\t      \"<div class=\\\"log_link\\\">\\n\" .\n \t\t      $cgi->a({-href => href(action=>\"commit\", hash=>$commit)}, \"commit\") .\n \t\t      \" | \" .\n@@ -4529,16 +4543,16 @@ sub git_log_body {\n \t\t      \"<br/>\\n\" .\n \t\t      \"</div>\\n\";\n \t\t      git_print_authorship(\\%co, -tag => 'span');\n-\t\t      print \"<br/>\\n</div>\\n\";\n+\t\t      print {$output_handler} \"<br/>\\n</div>\\n\";\n \n-\t\tprint \"<div class=\\\"log_body\\\">\\n\";\n+\t\tprint {$output_handler} \"<div class=\\\"log_body\\\">\\n\";\n \t\tgit_print_log($co{'comment'}, -final_empty_line=> 1);\n-\t\tprint \"</div>\\n\";\n+\t\tprint {$output_handler} \"</div>\\n\";\n \t}\n \tif ($extra) {\n-\t\tprint \"<div class=\\\"page_nav\\\">\\n\";\n-\t\tprint \"$extra\\n\";\n-\t\tprint \"</div>\\n\";\n+\t\tprint {$output_handler} \"<div class=\\\"page_nav\\\">\\n\";\n+\t\tprint {$output_handler} \"$extra\\n\";\n+\t\tprint {$output_handler} \"</div>\\n\";\n \t}\n }\n \n@@ -4549,41 +4563,41 @@ sub git_shortlog_body {\n \t$from = 0 unless defined $from;\n \t$to = $#{$commitlist} if (!defined $to || $#{$commitlist} < $to);\n \n-\tprint \"<table class=\\\"shortlog\\\">\\n\";\n+\tprint {$output_handler} \"<table class=\\\"shortlog\\\">\\n\";\n \tmy $alternate = 1;\n \tfor (my $i = $from; $i <= $to; $i++) {\n \t\tmy %co = %{$commitlist->[$i]};\n \t\tmy $commit = $co{'id'};\n \t\tmy $ref = format_ref_marker($refs, $commit);\n \t\tif ($alternate) {\n-\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n-\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t}\n \t\t$alternate ^= 1;\n \t\t# git_summary() used print \"<td><i>$co{'age_string'}</i></td>\\n\" .\n-\t\tprint \"<td title=\\\"$co{'age_string_age'}\\\"><i>$co{'age_string_date'}</i></td>\\n\" .\n+\t\tprint {$output_handler} \"<td title=\\\"$co{'age_string_age'}\\\"><i>$co{'age_string_date'}</i></td>\\n\" .\n \t\t      format_author_html('td', \\%co, 10) . \"<td>\";\n-\t\tprint format_subject_html($co{'title'}, $co{'title_short'},\n+\t\tprint {$output_handler} format_subject_html($co{'title'}, $co{'title_short'},\n \t\t                          href(action=>\"commit\", hash=>$commit), $ref);\n-\t\tprint \"</td>\\n\" .\n+\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t      \"<td class=\\\"link\\\">\" .\n \t\t      $cgi->a({-href => href(action=>\"commit\", hash=>$commit)}, \"commit\") . \" | \" .\n \t\t      $cgi->a({-href => href(action=>\"commitdiff\", hash=>$commit)}, \"commitdiff\") . \" | \" .\n \t\t      $cgi->a({-href => href(action=>\"tree\", hash=>$commit, hash_base=>$commit)}, \"tree\");\n \t\tmy $snapshot_links = format_snapshot_links($commit);\n \t\tif (defined $snapshot_links) {\n-\t\t\tprint \" | \" . $snapshot_links;\n+\t\t\tprint {$output_handler} \" | \" . $snapshot_links;\n \t\t}\n-\t\tprint \"</td>\\n\" .\n+\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t      \"</tr>\\n\";\n \t}\n \tif (defined $extra) {\n-\t\tprint \"<tr>\\n\" .\n+\t\tprint {$output_handler} \"<tr>\\n\" .\n \t\t      \"<td colspan=\\\"4\\\">$extra</td>\\n\" .\n \t\t      \"</tr>\\n\";\n \t}\n-\tprint \"</table>\\n\";\n+\tprint {$output_handler} \"</table>\\n\";\n }\n \n sub git_history_body {\n@@ -4594,7 +4608,7 @@ sub git_history_body {\n \t$from = 0 unless defined $from;\n \t$to = $#{$commitlist} unless (defined $to && $to <= $#{$commitlist});\n \n-\tprint \"<table class=\\\"history\\\">\\n\";\n+\tprint {$output_handler} \"<table class=\\\"history\\\">\\n\";\n \tmy $alternate = 1;\n \tfor (my $i = $from; $i <= $to; $i++) {\n \t\tmy %co = %{$commitlist->[$i]};\n@@ -4606,18 +4620,18 @@ sub git_history_body {\n \t\tmy $ref = format_ref_marker($refs, $commit);\n \n \t\tif ($alternate) {\n-\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n-\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t}\n \t\t$alternate ^= 1;\n-\t\tprint \"<td title=\\\"$co{'age_string_age'}\\\"><i>$co{'age_string_date'}</i></td>\\n\" .\n+\t\tprint {$output_handler} \"<td title=\\\"$co{'age_string_age'}\\\"><i>$co{'age_string_date'}</i></td>\\n\" .\n \t# shortlog:   format_author_html('td', \\%co, 10)\n \t\t      format_author_html('td', \\%co, 15, 3) . \"<td>\";\n \t\t# originally git_history used chop_str($co{'title'}, 50)\n-\t\tprint format_subject_html($co{'title'}, $co{'title_short'},\n+\t\tprint {$output_handler} format_subject_html($co{'title'}, $co{'title_short'},\n \t\t                          href(action=>\"commit\", hash=>$commit), $ref);\n-\t\tprint \"</td>\\n\" .\n+\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t      \"<td class=\\\"link\\\">\" .\n \t\t      $cgi->a({-href => href(action=>$ftype, hash_base=>$commit, file_name=>$file_name)}, $ftype) . \" | \" .\n \t\t      $cgi->a({-href => href(action=>\"commitdiff\", hash=>$commit)}, \"commitdiff\");\n@@ -4627,7 +4641,7 @@ sub git_history_body {\n \t\t\tmy $blob_parent  = git_get_hash_by_path($commit, $file_name);\n \t\t\tif (defined $blob_current && defined $blob_parent &&\n \t\t\t\t\t$blob_current ne $blob_parent) {\n-\t\t\t\tprint \" | \" .\n+\t\t\t\tprint {$output_handler} \" | \" .\n \t\t\t\t\t$cgi->a({-href => href(action=>\"blobdiff\",\n \t\t\t\t\t                       hash=>$blob_current, hash_parent=>$blob_parent,\n \t\t\t\t\t                       hash_base=>$hash_base, hash_parent_base=>$commit,\n@@ -4635,15 +4649,15 @@ sub git_history_body {\n \t\t\t\t\t        \"diff to current\");\n \t\t\t}\n \t\t}\n-\t\tprint \"</td>\\n\" .\n+\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t      \"</tr>\\n\";\n \t}\n \tif (defined $extra) {\n-\t\tprint \"<tr>\\n\" .\n+\t\tprint {$output_handler} \"<tr>\\n\" .\n \t\t      \"<td colspan=\\\"4\\\">$extra</td>\\n\" .\n \t\t      \"</tr>\\n\";\n \t}\n-\tprint \"</table>\\n\";\n+\tprint {$output_handler} \"</table>\\n\";\n }\n \n sub git_tags_body {\n@@ -4652,7 +4666,7 @@ sub git_tags_body {\n \t$from = 0 unless defined $from;\n \t$to = $#{$taglist} if (!defined $to || $#{$taglist} < $to);\n \n-\tprint \"<table class=\\\"tags\\\">\\n\";\n+\tprint {$output_handler} \"<table class=\\\"tags\\\">\\n\";\n \tmy $alternate = 1;\n \tfor (my $i = $from; $i <= $to; $i++) {\n \t\tmy $entry = $taglist->[$i];\n@@ -4663,50 +4677,50 @@ sub git_tags_body {\n \t\t\t$comment_short = chop_str($comment, 30, 5);\n \t\t}\n \t\tif ($alternate) {\n-\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n-\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t}\n \t\t$alternate ^= 1;\n \t\tif (defined $tag{'age'}) {\n-\t\t\tprint \"<td><i>$tag{'age'}</i></td>\\n\";\n+\t\t\tprint {$output_handler} \"<td><i>$tag{'age'}</i></td>\\n\";\n \t\t} else {\n-\t\t\tprint \"<td></td>\\n\";\n+\t\t\tprint {$output_handler} \"<td></td>\\n\";\n \t\t}\n-\t\tprint \"<td>\" .\n+\t\tprint {$output_handler} \"<td>\" .\n \t\t      $cgi->a({-href => href(action=>$tag{'reftype'}, hash=>$tag{'refid'}),\n \t\t               -class => \"list name\"}, esc_html($tag{'name'})) .\n \t\t      \"</td>\\n\" .\n \t\t      \"<td>\";\n \t\tif (defined $comment) {\n-\t\t\tprint format_subject_html($comment, $comment_short,\n+\t\t\tprint {$output_handler} format_subject_html($comment, $comment_short,\n \t\t\t                          href(action=>\"tag\", hash=>$tag{'id'}));\n \t\t}\n-\t\tprint \"</td>\\n\" .\n+\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t      \"<td class=\\\"selflink\\\">\";\n \t\tif ($tag{'type'} eq \"tag\") {\n-\t\t\tprint $cgi->a({-href => href(action=>\"tag\", hash=>$tag{'id'})}, \"tag\");\n+\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"tag\", hash=>$tag{'id'})}, \"tag\");\n \t\t} else {\n-\t\t\tprint \"&nbsp;\";\n+\t\t\tprint {$output_handler} \"&nbsp;\";\n \t\t}\n-\t\tprint \"</td>\\n\" .\n+\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t      \"<td class=\\\"link\\\">\" . \" | \" .\n \t\t      $cgi->a({-href => href(action=>$tag{'reftype'}, hash=>$tag{'refid'})}, $tag{'reftype'});\n \t\tif ($tag{'reftype'} eq \"commit\") {\n-\t\t\tprint \" | \" . $cgi->a({-href => href(action=>\"shortlog\", hash=>$tag{'fullname'})}, \"shortlog\") .\n+\t\t\tprint {$output_handler} \" | \" . $cgi->a({-href => href(action=>\"shortlog\", hash=>$tag{'fullname'})}, \"shortlog\") .\n \t\t\t      \" | \" . $cgi->a({-href => href(action=>\"log\", hash=>$tag{'fullname'})}, \"log\");\n \t\t} elsif ($tag{'reftype'} eq \"blob\") {\n-\t\t\tprint \" | \" . $cgi->a({-href => href(action=>\"blob_plain\", hash=>$tag{'refid'})}, \"raw\");\n+\t\t\tprint {$output_handler} \" | \" . $cgi->a({-href => href(action=>\"blob_plain\", hash=>$tag{'refid'})}, \"raw\");\n \t\t}\n-\t\tprint \"</td>\\n\" .\n+\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t      \"</tr>\";\n \t}\n \tif (defined $extra) {\n-\t\tprint \"<tr>\\n\" .\n+\t\tprint {$output_handler} \"<tr>\\n\" .\n \t\t      \"<td colspan=\\\"5\\\">$extra</td>\\n\" .\n \t\t      \"</tr>\\n\";\n \t}\n-\tprint \"</table>\\n\";\n+\tprint {$output_handler} \"</table>\\n\";\n }\n \n sub git_heads_body {\n@@ -4715,19 +4729,19 @@ sub git_heads_body {\n \t$from = 0 unless defined $from;\n \t$to = $#{$headlist} if (!defined $to || $#{$headlist} < $to);\n \n-\tprint \"<table class=\\\"heads\\\">\\n\";\n+\tprint {$output_handler} \"<table class=\\\"heads\\\">\\n\";\n \tmy $alternate = 1;\n \tfor (my $i = $from; $i <= $to; $i++) {\n \t\tmy $entry = $headlist->[$i];\n \t\tmy %ref = %$entry;\n \t\tmy $curr = $ref{'id'} eq $head;\n \t\tif ($alternate) {\n-\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n-\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t}\n \t\t$alternate ^= 1;\n-\t\tprint \"<td><i>$ref{'age'}</i></td>\\n\" .\n+\t\tprint {$output_handler} \"<td><i>$ref{'age'}</i></td>\\n\" .\n \t\t      ($curr ? \"<td class=\\\"current_head\\\">\" : \"<td>\") .\n \t\t      $cgi->a({-href => href(action=>\"shortlog\", hash=>$ref{'fullname'}),\n \t\t               -class => \"list name\"},esc_html($ref{'name'})) .\n@@ -4740,11 +4754,11 @@ sub git_heads_body {\n \t\t      \"</tr>\";\n \t}\n \tif (defined $extra) {\n-\t\tprint \"<tr>\\n\" .\n+\t\tprint {$output_handler} \"<tr>\\n\" .\n \t\t      \"<td colspan=\\\"3\\\">$extra</td>\\n\" .\n \t\t      \"</tr>\\n\";\n \t}\n-\tprint \"</table>\\n\";\n+\tprint {$output_handler} \"</table>\\n\";\n }\n \n sub git_search_grep_body {\n@@ -4752,7 +4766,7 @@ sub git_search_grep_body {\n \t$from = 0 unless defined $from;\n \t$to = $#{$commitlist} if (!defined $to || $#{$commitlist} < $to);\n \n-\tprint \"<table class=\\\"commit_search\\\">\\n\";\n+\tprint {$output_handler} \"<table class=\\\"commit_search\\\">\\n\";\n \tmy $alternate = 1;\n \tfor (my $i = $from; $i <= $to; $i++) {\n \t\tmy %co = %{$commitlist->[$i]};\n@@ -4761,12 +4775,12 @@ sub git_search_grep_body {\n \t\t}\n \t\tmy $commit = $co{'id'};\n \t\tif ($alternate) {\n-\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n-\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t}\n \t\t$alternate ^= 1;\n-\t\tprint \"<td title=\\\"$co{'age_string_age'}\\\"><i>$co{'age_string_date'}</i></td>\\n\" .\n+\t\tprint {$output_handler} \"<td title=\\\"$co{'age_string_age'}\\\"><i>$co{'age_string_date'}</i></td>\\n\" .\n \t\t      format_author_html('td', \\%co, 15, 5) .\n \t\t      \"<td>\" .\n \t\t      $cgi->a({-href => href(action=>\"commit\", hash=>$co{'id'}),\n@@ -4786,25 +4800,25 @@ sub git_search_grep_body {\n \t\t\t\t$match = esc_html($match);\n \t\t\t\t$trail = esc_html($trail);\n \n-\t\t\t\tprint \"$lead<span class=\\\"match\\\">$match</span>$trail<br />\";\n+\t\t\t\tprint {$output_handler} \"$lead<span class=\\\"match\\\">$match</span>$trail<br />\";\n \t\t\t}\n \t\t}\n-\t\tprint \"</td>\\n\" .\n+\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t      \"<td class=\\\"link\\\">\" .\n \t\t      $cgi->a({-href => href(action=>\"commit\", hash=>$co{'id'})}, \"commit\") .\n \t\t      \" | \" .\n \t\t      $cgi->a({-href => href(action=>\"commitdiff\", hash=>$co{'id'})}, \"commitdiff\") .\n \t\t      \" | \" .\n \t\t      $cgi->a({-href => href(action=>\"tree\", hash=>$co{'tree'}, hash_base=>$co{'id'})}, \"tree\");\n-\t\tprint \"</td>\\n\" .\n+\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t      \"</tr>\\n\";\n \t}\n \tif (defined $extra) {\n-\t\tprint \"<tr>\\n\" .\n+\t\tprint {$output_handler} \"<tr>\\n\" .\n \t\t      \"<td colspan=\\\"3\\\">$extra</td>\\n\" .\n \t\t      \"</tr>\\n\";\n \t}\n-\tprint \"</table>\\n\";\n+\tprint {$output_handler} \"</table>\\n\";\n }\n \n ## ======================================================================\n@@ -4824,11 +4838,11 @@ sub git_project_list {\n \n \tgit_header_html();\n \tif (-f $home_text) {\n-\t\tprint \"<div class=\\\"index_include\\\">\\n\";\n+\t\tprint {$output_handler} \"<div class=\\\"index_include\\\">\\n\";\n \t\tinsert_file($home_text);\n-\t\tprint \"</div>\\n\";\n+\t\tprint {$output_handler} \"</div>\\n\";\n \t}\n-\tprint $cgi->startform(-method => \"get\") .\n+\tprint {$output_handler} $cgi->startform(-method => \"get\") .\n \t      \"<p class=\\\"projsearch\\\">Search:\\n\" .\n \t      $cgi->textfield(-name => \"s\", -value => $searchtext) . \"\\n\" .\n \t      \"</p>\" .\n@@ -4858,7 +4872,7 @@ sub git_forks {\n sub git_project_index {\n \tmy @projects = git_get_projects_list($project);\n \n-\tprint $cgi->header(\n+\tprint {$output_handler} $cgi->header(\n \t\t-type => 'text/plain',\n \t\t-charset => 'utf-8',\n \t\t-content_disposition => 'inline; filename=\"index.aux\"');\n@@ -4875,7 +4889,7 @@ sub git_project_index {\n \t\t$path  =~ s/ /\\+/g;\n \t\t$owner =~ s/ /\\+/g;\n \n-\t\tprint \"$path $owner\\n\";\n+\t\tprint {$output_handler} \"$path $owner\\n\";\n \t}\n }\n \n@@ -4902,12 +4916,12 @@ sub git_summary {\n \tgit_header_html();\n \tgit_print_page_nav('summary','', $head);\n \n-\tprint \"<div class=\\\"title\\\">&nbsp;</div>\\n\";\n-\tprint \"<table class=\\\"projects_list\\\">\\n\" .\n+\tprint {$output_handler} \"<div class=\\\"title\\\">&nbsp;</div>\\n\";\n+\tprint {$output_handler} \"<table class=\\\"projects_list\\\">\\n\" .\n \t      \"<tr id=\\\"metadata_desc\\\"><td>description</td><td>\" . esc_html($descr) . \"</td></tr>\\n\" .\n \t      \"<tr id=\\\"metadata_owner\\\"><td>owner</td><td>\" . esc_html($owner) . \"</td></tr>\\n\";\n \tif (defined $cd{'rfc2822'}) {\n-\t\tprint \"<tr id=\\\"metadata_lchange\\\"><td>last change</td><td>$cd{'rfc2822'}</td></tr>\\n\";\n+\t\tprint {$output_handler} \"<tr id=\\\"metadata_lchange\\\"><td>last change</td><td>$cd{'rfc2822'}</td></tr>\\n\";\n \t}\n \n \t# use per project git URL list in $projectroot/$project/cloneurl\n@@ -4917,7 +4931,7 @@ sub git_summary {\n \t@url_list = map { \"$_/$project\" } @git_base_url_list unless @url_list;\n \tforeach my $git_url (@url_list) {\n \t\tnext unless $git_url;\n-\t\tprint \"<tr class=\\\"metadata_url\\\"><td>$url_tag</td><td>$git_url</td></tr>\\n\";\n+\t\tprint {$output_handler} \"<tr class=\\\"metadata_url\\\"><td>$url_tag</td><td>$git_url</td></tr>\\n\";\n \t\t$url_tag = \"\";\n \t}\n \n@@ -4926,23 +4940,23 @@ sub git_summary {\n \tif ($show_ctags) {\n \t\tmy $ctags = git_get_project_ctags($project);\n \t\tmy $cloud = git_populate_project_tagcloud($ctags);\n-\t\tprint \"<tr id=\\\"metadata_ctags\\\"><td>Content tags:<br />\";\n-\t\tprint \"</td>\\n<td>\" unless %$ctags;\n-\t\tprint \"<form action=\\\"$show_ctags\\\" method=\\\"post\\\"><input type=\\\"hidden\\\" name=\\\"p\\\" value=\\\"$project\\\" />Add: <input type=\\\"text\\\" name=\\\"t\\\" size=\\\"8\\\" /></form>\";\n-\t\tprint \"</td>\\n<td>\" if %$ctags;\n-\t\tprint git_show_project_tagcloud($cloud, 48);\n-\t\tprint \"</td></tr>\";\n+\t\tprint {$output_handler} \"<tr id=\\\"metadata_ctags\\\"><td>Content tags:<br />\";\n+\t\tprint {$output_handler} \"</td>\\n<td>\" unless %$ctags;\n+\t\tprint {$output_handler} \"<form action=\\\"$show_ctags\\\" method=\\\"post\\\"><input type=\\\"hidden\\\" name=\\\"p\\\" value=\\\"$project\\\" />Add: <input type=\\\"text\\\" name=\\\"t\\\" size=\\\"8\\\" /></form>\";\n+\t\tprint {$output_handler} \"</td>\\n<td>\" if %$ctags;\n+\t\tprint {$output_handler} git_show_project_tagcloud($cloud, 48);\n+\t\tprint {$output_handler} \"</td></tr>\";\n \t}\n \n-\tprint \"</table>\\n\";\n+\tprint {$output_handler} \"</table>\\n\";\n \n \t# If XSS prevention is on, we don't include README.html.\n \t# TODO: Allow a readme in some safe format.\n \tif (!$prevent_xss && -s \"$projectroot/$project/README.html\") {\n-\t\tprint \"<div class=\\\"title\\\">readme</div>\\n\" .\n+\t\tprint {$output_handler} \"<div class=\\\"title\\\">readme</div>\\n\" .\n \t\t      \"<div class=\\\"readme\\\">\\n\";\n \t\tinsert_file(\"$projectroot/$project/README.html\");\n-\t\tprint \"\\n</div>\\n\"; # class=\"readme\"\n+\t\tprint {$output_handler} \"\\n</div>\\n\"; # class=\"readme\"\n \t}\n \n \t# we need to request one more than 16 (0..15) to check if\n@@ -4991,7 +5005,7 @@ sub git_tag {\n \t}\n \n \tgit_print_header_div('commit', esc_html($tag{'name'}), $hash);\n-\tprint \"<div class=\\\"title_text\\\">\\n\" .\n+\tprint {$output_handler} \"<div class=\\\"title_text\\\">\\n\" .\n \t      \"<table class=\\\"object_header\\\">\\n\" .\n \t      \"<tr>\\n\" .\n \t      \"<td>object</td>\\n\" .\n@@ -5003,15 +5017,15 @@ sub git_tag {\n \tif (defined($tag{'author'})) {\n \t\tgit_print_authorship_rows(\\%tag, 'author');\n \t}\n-\tprint \"</table>\\n\\n\" .\n+\tprint {$output_handler} \"</table>\\n\\n\" .\n \t      \"</div>\\n\";\n-\tprint \"<div class=\\\"page_body\\\">\";\n+\tprint {$output_handler} \"<div class=\\\"page_body\\\">\";\n \tmy $comment = $tag{'comment'};\n \tforeach my $line (@$comment) {\n \t\tchomp $line;\n-\t\tprint esc_html($line, -nbsp=>1) . \"<br/>\\n\";\n+\t\tprint {$output_handler} esc_html($line, -nbsp=>1) . \"<br/>\\n\";\n \t}\n-\tprint \"</div>\\n\";\n+\tprint {$output_handler} \"</div>\\n\";\n \tgit_footer_html();\n }\n \n@@ -5062,21 +5076,23 @@ sub git_blame_common {\n \n \t# incremental blame data returns early\n \tif ($format eq 'data') {\n-\t\tprint $cgi->header(\n+\t\tprint {$output_handler} $cgi->header(\n \t\t\t-type=>\"text/plain\", -charset => \"utf-8\",\n \t\t\t-status=> \"200 OK\");\n \t\tlocal $| = 1; # output autoflush\n-\t\tprint while <$fd>;\n+\t\twhile (<$fd>) {\n+\t\t\t print {$output_handler} $_;\n+\t\t}\n \t\tclose $fd\n-\t\t\tor print \"ERROR $!\\n\";\n+\t\t\tor die_error(500, \"ERROR $!\\n\");\n \n-\t\tprint 'END';\n+\t\tprint {$output_handler} 'END';\n \t\tif (defined $t0 && gitweb_check_feature('timed')) {\n-\t\t\tprint ' '.\n+\t\t\tprint {$output_handler} ' '.\n \t\t\t      Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).\n \t\t\t      ' '.$number_of_git_cmds;\n \t\t}\n-\t\tprint \"\\n\";\n+\t\tprint {$output_handler} \"\\n\";\n \n \t\treturn;\n \t}\n@@ -5109,20 +5125,20 @@ sub git_blame_common {\n \n \t# page body\n \tif ($format eq 'incremental') {\n-\t\tprint \"<noscript>\\n<div class=\\\"error\\\"><center><b>\\n\".\n+\t\tprint {$output_handler} \"<noscript>\\n<div class=\\\"error\\\"><center><b>\\n\".\n \t\t      \"This page requires JavaScript to run.\\n Use \".\n \t\t      $cgi->a({-href => href(action=>'blame',javascript=>0,-replay=>1)},\n \t\t              'this page').\n \t\t      \" instead.\\n\".\n \t\t      \"</b></center></div>\\n</noscript>\\n\";\n \n-\t\tprint qq!<div id=\"progress_bar\" style=\"width: 100%; background-color: yellow\"></div>\\n!;\n+\t\tprint {$output_handler} qq!<div id=\"progress_bar\" style=\"width: 100%; background-color: yellow\"></div>\\n!;\n \t}\n \n-\tprint qq!<div class=\"page_body\">\\n!;\n-\tprint qq!<div id=\"progress_info\">... / ...</div>\\n!\n+\tprint {$output_handler} qq!<div class=\"page_body\">\\n!;\n+\tprint {$output_handler} qq!<div id=\"progress_info\">... / ...</div>\\n!\n \t\tif ($format eq 'incremental');\n-\tprint qq!<table id=\"blame_table\" class=\"blame\" width=\"100%\">\\n!.\n+\tprint {$output_handler} qq!<table id=\"blame_table\" class=\"blame\" width=\"100%\">\\n!.\n \t      #qq!<col width=\"5.5em\" /><col width=\"2.5em\" /><col width=\"*\" />\\n!.\n \t      qq!<thead>\\n!.\n \t      qq!<tr><th>Commit</th><th>Line</th><th>Data</th></tr>\\n!.\n@@ -5143,12 +5159,12 @@ sub git_blame_common {\n \t\t\tchomp $line;\n \t\t\t$linenr++;\n \n-\t\t\tprint qq!<tr id=\"l$linenr\" class=\"$color_class\">!.\n+\t\t\tprint {$output_handler} qq!<tr id=\"l$linenr\" class=\"$color_class\">!.\n \t\t\t      qq!<td class=\"sha1\"><a href=\"\"> </a></td>!.\n \t\t\t      qq!<td class=\"linenr\">!.\n \t\t\t      qq!<a class=\"linenr\" href=\"\">$linenr</a></td>!;\n-\t\t\tprint qq!<td class=\"pre\">! . esc_html($line) . \"</td>\\n\";\n-\t\t\tprint qq!</tr>\\n!;\n+\t\t\tprint {$output_handler} qq!<td class=\"pre\">! . esc_html($line) . \"</td>\\n\";\n+\t\t\tprint {$output_handler} qq!</tr>\\n!;\n \t\t}\n \n \t} else { # porcelain, i.e. ordinary blame\n@@ -5189,25 +5205,25 @@ sub git_blame_common {\n \t\t\t$tr_class .= ' boundary' if (exists $meta->{'boundary'});\n \t\t\t$tr_class .= ' no-previous' if ($meta->{'nprevious'} == 0);\n \t\t\t$tr_class .= ' multiple-previous' if ($meta->{'nprevious'} > 1);\n-\t\t\tprint \"<tr id=\\\"l$lineno\\\" class=\\\"$tr_class\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr id=\\\"l$lineno\\\" class=\\\"$tr_class\\\">\\n\";\n \t\t\tif ($group_size) {\n-\t\t\t\tprint \"<td class=\\\"sha1\\\"\";\n-\t\t\t\tprint \" title=\\\"\". esc_html($author) . \", $date\\\"\";\n-\t\t\t\tprint \" rowspan=\\\"$group_size\\\"\" if ($group_size > 1);\n-\t\t\t\tprint \">\";\n-\t\t\t\tprint $cgi->a({-href => href(action=>\"commit\",\n+\t\t\t\tprint {$output_handler} \"<td class=\\\"sha1\\\"\";\n+\t\t\t\tprint {$output_handler} \" title=\\\"\". esc_html($author) . \", $date\\\"\";\n+\t\t\t\tprint {$output_handler} \" rowspan=\\\"$group_size\\\"\" if ($group_size > 1);\n+\t\t\t\tprint {$output_handler} \">\";\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"commit\",\n \t\t\t\t                             hash=>$full_rev,\n \t\t\t\t                             file_name=>$file_name)},\n \t\t\t\t              esc_html($short_rev));\n \t\t\t\tif ($group_size >= 2) {\n \t\t\t\t\tmy @author_initials = ($author =~ /\\b([[:upper:]])\\B/g);\n \t\t\t\t\tif (@author_initials) {\n-\t\t\t\t\t\tprint \"<br />\" .\n+\t\t\t\t\t\tprint {$output_handler} \"<br />\" .\n \t\t\t\t\t\t      esc_html(join('', @author_initials));\n \t\t\t\t\t\t#           or join('.', ...)\n \t\t\t\t\t}\n \t\t\t\t}\n-\t\t\t\tprint \"</td>\\n\";\n+\t\t\t\tprint {$output_handler} \"</td>\\n\";\n \t\t\t}\n \t\t\t# 'previous' <sha1 of parent commit> <filename at commit>\n \t\t\tif (exists $meta->{'previous'} &&\n@@ -5224,23 +5240,23 @@ sub git_blame_common {\n \t\t\tmy $blamed = href(action => 'blame',\n \t\t\t                  file_name => $linenr_filename,\n \t\t\t                  hash_base => $linenr_commit);\n-\t\t\tprint \"<td class=\\\"linenr\\\">\";\n-\t\t\tprint $cgi->a({ -href => \"$blamed#l$orig_lineno\",\n+\t\t\tprint {$output_handler} \"<td class=\\\"linenr\\\">\";\n+\t\t\tprint {$output_handler} $cgi->a({ -href => \"$blamed#l$orig_lineno\",\n \t\t\t                -class => \"linenr\" },\n \t\t\t              esc_html($lineno));\n-\t\t\tprint \"</td>\";\n-\t\t\tprint \"<td class=\\\"pre\\\">\" . esc_html($data) . \"</td>\\n\";\n-\t\t\tprint \"</tr>\\n\";\n+\t\t\tprint {$output_handler} \"</td>\";\n+\t\t\tprint {$output_handler} \"<td class=\\\"pre\\\">\" . esc_html($data) . \"</td>\\n\";\n+\t\t\tprint {$output_handler} \"</tr>\\n\";\n \t\t} # end while\n \n \t}\n \n \t# footer\n-\tprint \"</tbody>\\n\".\n+\tprint {$output_handler} \"</tbody>\\n\".\n \t      \"</table>\\n\"; # class=\"blame\"\n-\tprint \"</div>\\n\";   # class=\"blame_body\"\n+\tprint {$output_handler} \"</div>\\n\";   # class=\"blame_body\"\n \tclose $fd\n-\t\tor print \"Reading blob failed\\n\";\n+\t\tor print {$output_handler} \"Reading blob failed\\n\";\n \n \tgit_footer_html();\n }\n@@ -5323,16 +5339,17 @@ sub git_blob_plain {\n \tmy $sandbox = $prevent_xss &&\n \t\t$type !~ m!^(?:text/plain|image/(?:gif|png|jpeg))$!;\n \n-\tprint $cgi->header(\n+\tprint {$output_handler} $cgi->header(\n \t\t-type => $type,\n \t\t-expires => $expires,\n \t\t-content_disposition =>\n \t\t\t($sandbox ? 'attachment' : 'inline')\n \t\t\t. '; filename=\"' . $save_as . '\"');\n \tlocal $/ = undef;\n-\tbinmode STDOUT, ':raw';\n-\tprint <$fd>;\n-\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n+\n+\tbinmode $output_handler_bin, ':raw';\n+\tprint {$output_handler_bin} <$fd>;\n+\tbinmode $output_handler_bin, ':utf8'; # as set at the beginning of gitweb.cgi\n \tclose $fd;\n }\n \n@@ -5391,18 +5408,18 @@ sub git_blob {\n \t\tgit_print_page_nav('','', $hash_base,$co{'tree'},$hash_base, $formats_nav);\n \t\tgit_print_header_div('commit', esc_html($co{'title'}), $hash_base);\n \t} else {\n-\t\tprint \"<div class=\\\"page_nav\\\">\\n\" .\n+\t\tprint {$output_handler} \"<div class=\\\"page_nav\\\">\\n\" .\n \t\t      \"<br/><br/></div>\\n\" .\n \t\t      \"<div class=\\\"title\\\">$hash</div>\\n\";\n \t}\n \tgit_print_page_path($file_name, \"blob\", $hash_base);\n-\tprint \"<div class=\\\"page_body\\\">\\n\";\n+\tprint {$output_handler} \"<div class=\\\"page_body\\\">\\n\";\n \tif ($mimetype =~ m!^image/!) {\n-\t\tprint qq!<img type=\"$mimetype\"!;\n+\t\tprint {$output_handler} qq!<img type=\"$mimetype\"!;\n \t\tif ($file_name) {\n-\t\t\tprint qq! alt=\"$file_name\" title=\"$file_name\"!;\n+\t\t\tprint {$output_handler} qq! alt=\"$file_name\" title=\"$file_name\"!;\n \t\t}\n-\t\tprint qq! src=\"! .\n+\t\tprint {$output_handler} qq! src=\"! .\n \t\t      href(action=>\"blob_plain\", hash=>$hash,\n \t\t           hash_base=>$hash_base, file_name=>$file_name) .\n \t\t      qq!\" />\\n!;\n@@ -5412,14 +5429,14 @@ sub git_blob {\n \t\t\tchomp $line;\n \t\t\t$nr++;\n \t\t\t$line = untabify($line);\n-\t\t\tprintf \"<div class=\\\"pre\\\"><a id=\\\"l%i\\\" href=\\\"\" . href(-replay => 1)\n+\t\t\tprintf {$output_handler} \"<div class=\\\"pre\\\"><a id=\\\"l%i\\\" href=\\\"\" . href(-replay => 1)\n \t\t\t\t. \"#l%i\\\" class=\\\"linenr\\\">%4i</a> %s</div>\\n\",\n \t\t\t       $nr, $nr, $nr, esc_html($line, -nbsp=>1);\n \t\t}\n \t}\n \tclose $fd\n-\t\tor print \"Reading blob failed.\\n\";\n-\tprint \"</div>\";\n+\t\tor print {$output_handler} \"Reading blob failed.\\n\";\n+\tprint {$output_handler} \"</div>\";\n \tgit_footer_html();\n }\n \n@@ -5474,9 +5491,9 @@ sub git_tree {\n \t\tgit_print_header_div('commit', esc_html($co{'title'}) . $ref, $hash_base);\n \t} else {\n \t\tundef $hash_base;\n-\t\tprint \"<div class=\\\"page_nav\\\">\\n\";\n-\t\tprint \"<br/><br/></div>\\n\";\n-\t\tprint \"<div class=\\\"title\\\">$hash</div>\\n\";\n+\t\tprint {$output_handler} \"<div class=\\\"page_nav\\\">\\n\";\n+\t\tprint {$output_handler} \"<br/><br/></div>\\n\";\n+\t\tprint {$output_handler} \"<div class=\\\"title\\\">$hash</div>\\n\";\n \t}\n \tif (defined $file_name) {\n \t\t$basedir = $file_name;\n@@ -5485,16 +5502,16 @@ sub git_tree {\n \t\t}\n \t\tgit_print_page_path($file_name, 'tree', $hash_base);\n \t}\n-\tprint \"<div class=\\\"page_body\\\">\\n\";\n-\tprint \"<table class=\\\"tree\\\">\\n\";\n+\tprint {$output_handler} \"<div class=\\\"page_body\\\">\\n\";\n+\tprint {$output_handler} \"<table class=\\\"tree\\\">\\n\";\n \tmy $alternate = 1;\n \t# '..' (top directory) link if possible\n \tif (defined $hash_base &&\n \t    defined $file_name && $file_name =~ m![^/]+$!) {\n \t\tif ($alternate) {\n-\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n-\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t}\n \t\t$alternate ^= 1;\n \n@@ -5502,33 +5519,33 @@ sub git_tree {\n \t\t$up =~ s!/?[^/]+$!!;\n \t\tundef $up unless $up;\n \t\t# based on git_print_tree_entry\n-\t\tprint '<td class=\"mode\">' . mode_str('040000') . \"</td>\\n\";\n-\t\tprint '<td class=\"size\">&nbsp;</td>'.\"\\n\" if $show_sizes;\n-\t\tprint '<td class=\"list\">';\n-\t\tprint $cgi->a({-href => href(action=>\"tree\",\n+\t\tprint {$output_handler} '<td class=\"mode\">' . mode_str('040000') . \"</td>\\n\";\n+\t\tprint {$output_handler} '<td class=\"size\">&nbsp;</td>'.\"\\n\" if $show_sizes;\n+\t\tprint {$output_handler} '<td class=\"list\">';\n+\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"tree\",\n \t\t                             hash_base=>$hash_base,\n \t\t                             file_name=>$up)},\n \t\t              \"..\");\n-\t\tprint \"</td>\\n\";\n-\t\tprint \"<td class=\\\"link\\\"></td>\\n\";\n+\t\tprint {$output_handler} \"</td>\\n\";\n+\t\tprint {$output_handler} \"<td class=\\\"link\\\"></td>\\n\";\n \n-\t\tprint \"</tr>\\n\";\n+\t\tprint {$output_handler} \"</tr>\\n\";\n \t}\n \tforeach my $line (@entries) {\n \t\tmy %t = parse_ls_tree_line($line, -z => 1, -l => $show_sizes);\n \n \t\tif ($alternate) {\n-\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t} else {\n-\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t}\n \t\t$alternate ^= 1;\n \n \t\tgit_print_tree_entry(\\%t, $basedir, $hash_base, $have_blame);\n \n-\t\tprint \"</tr>\\n\";\n+\t\tprint {$output_handler} \"</tr>\\n\";\n \t}\n-\tprint \"</table>\\n\" .\n+\tprint {$output_handler} \"</table>\\n\" .\n \t      \"</div>\";\n \tgit_footer_html();\n }\n@@ -5605,16 +5622,16 @@ sub git_snapshot {\n \t}\n \n \t$filename =~ s/([\"\\\\])/\\\\$1/g;\n-\tprint $cgi->header(\n+\tprint {$output_handler} $cgi->header(\n \t\t-type => $known_snapshot_formats{$format}{'type'},\n \t\t-content_disposition => 'inline; filename=\"' . $filename . '\"',\n \t\t-status => '200 OK');\n \n \topen my $fd, \"-|\", $cmd\n \t\tor die_error(500, \"Execute git-archive failed\");\n-\tbinmode STDOUT, ':raw';\n-\tprint <$fd>;\n-\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n+\tbinmode $output_handler_bin, ':raw';\n+\tprint {$output_handler_bin} <$fd>;\n+\tbinmode $output_handler_bin, ':utf8'; # as set at the beginning of gitweb.cgi\n \tclose $fd;\n }\n \n@@ -5765,11 +5782,11 @@ sub git_commit {\n \t} else {\n \t\tgit_print_header_div('tree', esc_html($co{'title'}) . $ref, $co{'tree'}, $hash);\n \t}\n-\tprint \"<div class=\\\"title_text\\\">\\n\" .\n+\tprint {$output_handler} \"<div class=\\\"title_text\\\">\\n\" .\n \t      \"<table class=\\\"object_header\\\">\\n\";\n \tgit_print_authorship_rows(\\%co);\n-\tprint \"<tr><td>commit</td><td class=\\\"sha1\\\">$co{'id'}</td></tr>\\n\";\n-\tprint \"<tr>\" .\n+\tprint {$output_handler} \"<tr><td>commit</td><td class=\\\"sha1\\\">$co{'id'}</td></tr>\\n\";\n+\tprint {$output_handler} \"<tr>\" .\n \t      \"<td>tree</td>\" .\n \t      \"<td class=\\\"sha1\\\">\" .\n \t      $cgi->a({-href => href(action=>\"tree\", hash=>$co{'tree'}, hash_base=>$hash),\n@@ -5780,13 +5797,13 @@ sub git_commit {\n \t              \"tree\");\n \tmy $snapshot_links = format_snapshot_links($hash);\n \tif (defined $snapshot_links) {\n-\t\tprint \" | \" . $snapshot_links;\n+\t\tprint {$output_handler} \" | \" . $snapshot_links;\n \t}\n-\tprint \"</td>\" .\n+\tprint {$output_handler} \"</td>\" .\n \t      \"</tr>\\n\";\n \n \tforeach my $par (@$parents) {\n-\t\tprint \"<tr>\" .\n+\t\tprint {$output_handler} \"<tr>\" .\n \t\t      \"<td>parent</td>\" .\n \t\t      \"<td class=\\\"sha1\\\">\" .\n \t\t      $cgi->a({-href => href(action=>\"commit\", hash=>$par),\n@@ -5799,12 +5816,12 @@ sub git_commit {\n \t\t      \"</td>\" .\n \t\t      \"</tr>\\n\";\n \t}\n-\tprint \"</table>\".\n+\tprint {$output_handler} \"</table>\".\n \t      \"</div>\\n\";\n \n-\tprint \"<div class=\\\"page_body\\\">\\n\";\n+\tprint {$output_handler} \"<div class=\\\"page_body\\\">\\n\";\n \tgit_print_log($co{'comment'});\n-\tprint \"</div>\\n\";\n+\tprint {$output_handler} \"</div>\\n\";\n \n \tgit_difftree_body(\\@difftree, $hash, @$parents);\n \n@@ -5852,7 +5869,7 @@ sub git_object {\n \t\tdie_error(400, \"Not enough information to find object\");\n \t}\n \n-\tprint $cgi->redirect(-uri => href(action=>$type, -full=>1,\n+\tprint {$output_handler} $cgi->redirect(-uri => href(action=>$type, -full=>1,\n \t                                  hash=>$hash, hash_base=>$hash_base,\n \t                                  file_name=>$file_name),\n \t                     -status => '302 Found');\n@@ -5943,23 +5960,23 @@ sub git_blobdiff {\n \t\t\tgit_print_page_nav('','', $hash_base,$co{'tree'},$hash_base, $formats_nav);\n \t\t\tgit_print_header_div('commit', esc_html($co{'title'}), $hash_base);\n \t\t} else {\n-\t\t\tprint \"<div class=\\\"page_nav\\\"><br/>$formats_nav<br/></div>\\n\";\n-\t\t\tprint \"<div class=\\\"title\\\">$hash vs $hash_parent</div>\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"page_nav\\\"><br/>$formats_nav<br/></div>\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"title\\\">$hash vs $hash_parent</div>\\n\";\n \t\t}\n \t\tif (defined $file_name) {\n \t\t\tgit_print_page_path($file_name, \"blob\", $hash_base);\n \t\t} else {\n-\t\t\tprint \"<div class=\\\"page_path\\\"></div>\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"page_path\\\"></div>\\n\";\n \t\t}\n \n \t} elsif ($format eq 'plain') {\n-\t\tprint $cgi->header(\n+\t\tprint {$output_handler} $cgi->header(\n \t\t\t-type => 'text/plain',\n \t\t\t-charset => 'utf-8',\n \t\t\t-expires => $expires,\n \t\t\t-content_disposition => 'inline; filename=\"' . \"$file_name\" . '.patch\"');\n \n-\t\tprint \"X-Git-Url: \" . $cgi->self_url() . \"\\n\\n\";\n+\t\tprint {$output_handler} \"X-Git-Url: \" . $cgi->self_url() . \"\\n\\n\";\n \n \t} else {\n \t\tdie_error(400, \"Unknown blobdiff format\");\n@@ -5967,12 +5984,12 @@ sub git_blobdiff {\n \n \t# patch\n \tif ($format eq 'html') {\n-\t\tprint \"<div class=\\\"page_body\\\">\\n\";\n+\t\tprint {$output_handler} \"<div class=\\\"page_body\\\">\\n\";\n \n \t\tgit_patchset_body($fd, [ \\%diffinfo ], $hash_base, $hash_parent_base);\n \t\tclose $fd;\n \n-\t\tprint \"</div>\\n\"; # class=\"page_body\"\n+\t\tprint {$output_handler} \"</div>\\n\"; # class=\"page_body\"\n \t\tgit_footer_html();\n \n \t} else {\n@@ -5980,12 +5997,12 @@ sub git_blobdiff {\n \t\t\t$line =~ s!a/($hash|$hash_parent)!'a/'.esc_path($diffinfo{'from_file'})!eg;\n \t\t\t$line =~ s!b/($hash|$hash_parent)!'b/'.esc_path($diffinfo{'to_file'})!eg;\n \n-\t\t\tprint $line;\n+\t\t\tprint {$output_handler} $line;\n \n \t\t\tlast if $line =~ m!^\\+\\+\\+!;\n \t\t}\n \t\tlocal $/ = undef;\n-\t\tprint <$fd>;\n+\t\tprint {$output_handler} <$fd>;\n \t\tclose $fd;\n \t}\n }\n@@ -6148,16 +6165,16 @@ sub git_commitdiff {\n \t\tgit_header_html(undef, $expires);\n \t\tgit_print_page_nav('commitdiff','', $hash,$co{'tree'},$hash, $formats_nav);\n \t\tgit_print_header_div('commit', esc_html($co{'title'}) . $ref, $hash);\n-\t\tprint \"<div class=\\\"title_text\\\">\\n\" .\n+\t\tprint {$output_handler} \"<div class=\\\"title_text\\\">\\n\" .\n \t\t      \"<table class=\\\"object_header\\\">\\n\";\n \t\tgit_print_authorship_rows(\\%co);\n-\t\tprint \"</table>\".\n+\t\tprint {$output_handler} \"</table>\".\n \t\t      \"</div>\\n\";\n-\t\tprint \"<div class=\\\"page_body\\\">\\n\";\n+\t\tprint {$output_handler} \"<div class=\\\"page_body\\\">\\n\";\n \t\tif (@{$co{'comment'}} > 1) {\n-\t\t\tprint \"<div class=\\\"log\\\">\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"log\\\">\\n\";\n \t\t\tgit_print_log($co{'comment'}, -final_empty_line=> 1, -remove_title => 1);\n-\t\t\tprint \"</div>\\n\"; # class=\"log\"\n+\t\t\tprint {$output_handler} \"</div>\\n\"; # class=\"log\"\n \t\t}\n \n \t} elsif ($format eq 'plain') {\n@@ -6165,27 +6182,27 @@ sub git_commitdiff {\n \t\tmy $tagname = git_get_rev_name_tags($hash);\n \t\tmy $filename = basename($project) . \"-$hash.patch\";\n \n-\t\tprint $cgi->header(\n+\t\tprint {$output_handler} $cgi->header(\n \t\t\t-type => 'text/plain',\n \t\t\t-charset => 'utf-8',\n \t\t\t-expires => $expires,\n \t\t\t-content_disposition => 'inline; filename=\"' . \"$filename\" . '\"');\n \t\tmy %ad = parse_date($co{'author_epoch'}, $co{'author_tz'});\n-\t\tprint \"From: \" . to_utf8($co{'author'}) . \"\\n\";\n-\t\tprint \"Date: $ad{'rfc2822'} ($ad{'tz_local'})\\n\";\n-\t\tprint \"Subject: \" . to_utf8($co{'title'}) . \"\\n\";\n+\t\tprint {$output_handler} \"From: \" . to_utf8($co{'author'}) . \"\\n\";\n+\t\tprint {$output_handler} \"Date: $ad{'rfc2822'} ($ad{'tz_local'})\\n\";\n+\t\tprint {$output_handler} \"Subject: \" . to_utf8($co{'title'}) . \"\\n\";\n \n-\t\tprint \"X-Git-Tag: $tagname\\n\" if $tagname;\n-\t\tprint \"X-Git-Url: \" . $cgi->self_url() . \"\\n\\n\";\n+\t\tprint {$output_handler} \"X-Git-Tag: $tagname\\n\" if $tagname;\n+\t\tprint {$output_handler} \"X-Git-Url: \" . $cgi->self_url() . \"\\n\\n\";\n \n \t\tforeach my $line (@{$co{'comment'}}) {\n-\t\t\tprint to_utf8($line) . \"\\n\";\n+\t\t\tprint {$output_handler} to_utf8($line) . \"\\n\";\n \t\t}\n-\t\tprint \"---\\n\\n\";\n+\t\tprint {$output_handler} \"---\\n\\n\";\n \t} elsif ($format eq 'patch') {\n \t\tmy $filename = basename($project) . \"-$hash.patch\";\n \n-\t\tprint $cgi->header(\n+\t\tprint {$output_handler} $cgi->header(\n \t\t\t-type => 'text/plain',\n \t\t\t-charset => 'utf-8',\n \t\t\t-expires => $expires,\n@@ -6198,24 +6215,24 @@ sub git_commitdiff {\n \t\t\t$hash_parent eq '-c' || $hash_parent eq '--cc';\n \t\tgit_difftree_body(\\@difftree, $hash,\n \t\t                  $use_parents ? @{$co{'parents'}} : $hash_parent);\n-\t\tprint \"<br/>\\n\";\n+\t\tprint {$output_handler} \"<br/>\\n\";\n \n \t\tgit_patchset_body($fd, \\@difftree, $hash,\n \t\t                  $use_parents ? @{$co{'parents'}} : $hash_parent);\n \t\tclose $fd;\n-\t\tprint \"</div>\\n\"; # class=\"page_body\"\n+\t\tprint {$output_handler} \"</div>\\n\"; # class=\"page_body\"\n \t\tgit_footer_html();\n \n \t} elsif ($format eq 'plain') {\n \t\tlocal $/ = undef;\n-\t\tprint <$fd>;\n+\t\tprint {$output_handler} <$fd>;\n \t\tclose $fd\n-\t\t\tor print \"Reading git-diff-tree failed\\n\";\n+\t\t\tor print {$output_handler} \"Reading git-diff-tree failed\\n\";\n \t} elsif ($format eq 'patch') {\n \t\tlocal $/ = undef;\n-\t\tprint <$fd>;\n+\t\tprint {$output_handler} <$fd>;\n \t\tclose $fd\n-\t\t\tor print \"Reading git-format-patch failed\\n\";\n+\t\t\tor print {$output_handler} \"Reading git-format-patch failed\\n\";\n \t}\n }\n \n@@ -6318,7 +6335,7 @@ sub git_search {\n \t\tgit_print_page_nav('','', $hash,$co{'tree'},$hash);\n \t\tgit_print_header_div('commit', esc_html($co{'title'}), $hash);\n \n-\t\tprint \"<table class=\\\"pickaxe search\\\">\\n\";\n+\t\tprint {$output_handler} \"<table class=\\\"pickaxe search\\\">\\n\";\n \t\tmy $alternate = 1;\n \t\tlocal $/ = \"\\n\";\n \t\topen my $fd, '-|', git_cmd(), '--no-pager', 'log', @diff_opts,\n@@ -6334,24 +6351,24 @@ sub git_search {\n \t\t\tif (defined $set{'commit'}) {\n \t\t\t\t# finish previous commit\n \t\t\t\tif (%co) {\n-\t\t\t\t\tprint \"</td>\\n\" .\n+\t\t\t\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t\t\t\t      \"<td class=\\\"link\\\">\" .\n \t\t\t\t\t      $cgi->a({-href => href(action=>\"commit\", hash=>$co{'id'})}, \"commit\") .\n \t\t\t\t\t      \" | \" .\n \t\t\t\t\t      $cgi->a({-href => href(action=>\"tree\", hash=>$co{'tree'}, hash_base=>$co{'id'})}, \"tree\");\n-\t\t\t\t\tprint \"</td>\\n\" .\n+\t\t\t\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t\t\t\t      \"</tr>\\n\";\n \t\t\t\t}\n \n \t\t\t\tif ($alternate) {\n-\t\t\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t\t\t} else {\n-\t\t\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t\t\t}\n \t\t\t\t$alternate ^= 1;\n \t\t\t\t%co = parse_commit($set{'commit'});\n \t\t\t\tmy $author = chop_and_escape_str($co{'author_name'}, 15, 5);\n-\t\t\t\tprint \"<td title=\\\"$co{'age_string_age'}\\\"><i>$co{'age_string_date'}</i></td>\\n\" .\n+\t\t\t\tprint {$output_handler} \"<td title=\\\"$co{'age_string_age'}\\\"><i>$co{'age_string_date'}</i></td>\\n\" .\n \t\t\t\t      \"<td><i>$author</i></td>\\n\" .\n \t\t\t\t      \"<td>\" .\n \t\t\t\t      $cgi->a({-href => href(action=>\"commit\", hash=>$co{'id'}),\n@@ -6360,7 +6377,7 @@ sub git_search {\n \t\t\t} elsif (defined $set{'to_id'}) {\n \t\t\t\tnext if ($set{'to_id'} =~ m/^0{40}$/);\n \n-\t\t\t\tprint $cgi->a({-href => href(action=>\"blob\", hash_base=>$co{'id'},\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(action=>\"blob\", hash_base=>$co{'id'},\n \t\t\t\t                             hash=>$set{'to_id'}, file_name=>$set{'to_file'}),\n \t\t\t\t              -class => \"list\"},\n \t\t\t\t              \"<span class=\\\"match\\\">\" . esc_path($set{'file'}) . \"</span>\") .\n@@ -6371,23 +6388,23 @@ sub git_search {\n \n \t\t# finish last commit (warning: repetition!)\n \t\tif (%co) {\n-\t\t\tprint \"</td>\\n\" .\n+\t\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t\t      \"<td class=\\\"link\\\">\" .\n \t\t\t      $cgi->a({-href => href(action=>\"commit\", hash=>$co{'id'})}, \"commit\") .\n \t\t\t      \" | \" .\n \t\t\t      $cgi->a({-href => href(action=>\"tree\", hash=>$co{'tree'}, hash_base=>$co{'id'})}, \"tree\");\n-\t\t\tprint \"</td>\\n\" .\n+\t\t\tprint {$output_handler} \"</td>\\n\" .\n \t\t\t      \"</tr>\\n\";\n \t\t}\n \n-\t\tprint \"</table>\\n\";\n+\t\tprint {$output_handler} \"</table>\\n\";\n \t}\n \n \tif ($searchtype eq 'grep') {\n \t\tgit_print_page_nav('','', $hash,$co{'tree'},$hash);\n \t\tgit_print_header_div('commit', esc_html($co{'title'}), $hash);\n \n-\t\tprint \"<table class=\\\"grep_search\\\">\\n\";\n+\t\tprint {$output_handler} \"<table class=\\\"grep_search\\\">\\n\";\n \t\tmy $alternate = 1;\n \t\tmy $matches = 0;\n \t\tlocal $/ = \"\\n\";\n@@ -6406,21 +6423,21 @@ sub git_search {\n \t\t\t\t(undef, $file, $lno, $ltext) = split(/:/, $line, 4);\n \t\t\t}\n \t\t\tif ($file ne $lastfile) {\n-\t\t\t\t$lastfile and print \"</td></tr>\\n\";\n+\t\t\t\t$lastfile and print {$output_handler} \"</td></tr>\\n\";\n \t\t\t\tif ($alternate++) {\n-\t\t\t\t\tprint \"<tr class=\\\"dark\\\">\\n\";\n+\t\t\t\t\tprint {$output_handler} \"<tr class=\\\"dark\\\">\\n\";\n \t\t\t\t} else {\n-\t\t\t\t\tprint \"<tr class=\\\"light\\\">\\n\";\n+\t\t\t\t\tprint {$output_handler} \"<tr class=\\\"light\\\">\\n\";\n \t\t\t\t}\n-\t\t\t\tprint \"<td class=\\\"list\\\">\".\n+\t\t\t\tprint {$output_handler} \"<td class=\\\"list\\\">\".\n \t\t\t\t\t$cgi->a({-href => href(action=>\"blob\", hash=>$co{'hash'},\n \t\t\t\t\t\t\t       file_name=>\"$file\"),\n \t\t\t\t\t\t-class => \"list\"}, esc_path($file));\n-\t\t\t\tprint \"</td><td>\\n\";\n+\t\t\t\tprint {$output_handler} \"</td><td>\\n\";\n \t\t\t\t$lastfile = $file;\n \t\t\t}\n \t\t\tif ($binary) {\n-\t\t\t\tprint \"<div class=\\\"binary\\\">Binary file</div>\\n\";\n+\t\t\t\tprint {$output_handler} \"<div class=\\\"binary\\\">Binary file</div>\\n\";\n \t\t\t} else {\n \t\t\t\t$ltext = untabify($ltext);\n \t\t\t\tif ($ltext =~ m/^(.*)($search_regexp)(.*)$/i) {\n@@ -6432,7 +6449,7 @@ sub git_search {\n \t\t\t\t} else {\n \t\t\t\t\t$ltext = esc_html($ltext, -nbsp=>1);\n \t\t\t\t}\n-\t\t\t\tprint \"<div class=\\\"pre\\\">\" .\n+\t\t\t\tprint {$output_handler} \"<div class=\\\"pre\\\">\" .\n \t\t\t\t\t$cgi->a({-href => href(action=>\"blob\", hash=>$co{'hash'},\n \t\t\t\t\t\t\t       file_name=>\"$file\").'#l'.$lno,\n \t\t\t\t\t\t-class => \"linenr\"}, sprintf('%4i', $lno))\n@@ -6440,16 +6457,16 @@ sub git_search {\n \t\t\t}\n \t\t}\n \t\tif ($lastfile) {\n-\t\t\tprint \"</td></tr>\\n\";\n+\t\t\tprint {$output_handler} \"</td></tr>\\n\";\n \t\t\tif ($matches > 1000) {\n-\t\t\t\tprint \"<div class=\\\"diff nodifferences\\\">Too many matches, listing trimmed</div>\\n\";\n+\t\t\t\tprint {$output_handler} \"<div class=\\\"diff nodifferences\\\">Too many matches, listing trimmed</div>\\n\";\n \t\t\t}\n \t\t} else {\n-\t\t\tprint \"<div class=\\\"diff nodifferences\\\">No matches found</div>\\n\";\n+\t\t\tprint {$output_handler} \"<div class=\\\"diff nodifferences\\\">No matches found</div>\\n\";\n \t\t}\n \t\tclose $fd;\n \n-\t\tprint \"</table>\\n\";\n+\t\tprint {$output_handler} \"</table>\\n\";\n \t}\n \tgit_footer_html();\n }\n@@ -6457,7 +6474,7 @@ sub git_search {\n sub git_search_help {\n \tgit_header_html();\n \tgit_print_page_nav('','', $hash,$hash,$hash);\n-\tprint <<EOT;\n+\tprint {$output_handler} <<EOT;\n <p><strong>Pattern</strong> is by default a normal string that is matched precisely (but without\n regard to case, except in the case of pickaxe). However, when you check the <em>re</em> checkbox,\n the pattern entered is recognized as the POSIX extended\n@@ -6469,7 +6486,7 @@ insensitive).</p>\n EOT\n \tmy $have_grep = gitweb_check_feature('grep');\n \tif ($have_grep) {\n-\t\tprint <<EOT;\n+\t\tprint {$output_handler} <<EOT;\n <dt><b>grep</b></dt>\n <dd>All files in the currently selected tree (HEAD unless you are explicitly browsing\n     a different one) are searched for the given pattern. On large trees, this search can take\n@@ -6478,7 +6495,7 @@ due to git-grep peculiarity, currently if regexp mode is turned off, the matches\n case-sensitive.</dd>\n EOT\n \t}\n-\tprint <<EOT;\n+\tprint {$output_handler} <<EOT;\n <dt><b>author</b></dt>\n <dd>Name and e-mail of the change author and date of birth of the patch will be scanned for the given pattern.</dd>\n <dt><b>committer</b></dt>\n@@ -6486,7 +6503,7 @@ EOT\n EOT\n \tmy $have_pickaxe = gitweb_check_feature('pickaxe');\n \tif ($have_pickaxe) {\n-\t\tprint <<EOT;\n+\t\tprint {$output_handler} <<EOT;\n <dt><b>pickaxe</b></dt>\n <dd>All commits that caused the string to appear or disappear from any file (changes that\n added, removed or \"modified\" the string) will be listed. This search can take a while and\n@@ -6494,7 +6511,7 @@ takes a lot of strain on the server, so please use it wisely. Note that since yo\n interested even in changes just changing the case as well, this search is case sensitive.</dd>\n EOT\n \t}\n-\tprint \"</dl>\\n\";\n+\tprint {$output_handler} \"</dl>\\n\";\n \tgit_footer_html();\n }\n \n@@ -6541,7 +6558,7 @@ sub git_feed {\n \t\t\t\t$since = Time::ParseDate::parsedate($if_modified, GMT => 1);\n \t\t\t}\n \t\t\tif (defined $since && $latest_epoch <= $since) {\n-\t\t\t\tprint $cgi->header(\n+\t\t\t\tprint {$output_handler} $cgi->header(\n \t\t\t\t\t-type => $content_type,\n \t\t\t\t\t-charset => 'utf-8',\n \t\t\t\t\t-last_modified => $latest_date{'rfc2822'},\n@@ -6549,12 +6566,12 @@ sub git_feed {\n \t\t\t\treturn;\n \t\t\t}\n \t\t}\n-\t\tprint $cgi->header(\n+\t\tprint {$output_handler} $cgi->header(\n \t\t\t-type => $content_type,\n \t\t\t-charset => 'utf-8',\n \t\t\t-last_modified => $latest_date{'rfc2822'});\n \t} else {\n-\t\tprint $cgi->header(\n+\t\tprint {$output_handler} $cgi->header(\n \t\t\t-type => $content_type,\n \t\t\t-charset => 'utf-8');\n \t}\n@@ -6598,13 +6615,13 @@ sub git_feed {\n \t} else {\n \t\t$alt_url = href(-full=>1, action=>\"summary\");\n \t}\n-\tprint qq!<?xml version=\"1.0\" encoding=\"utf-8\"?>\\n!;\n+\tprint {$output_handler} qq!<?xml version=\"1.0\" encoding=\"utf-8\"?>\\n!;\n \tif ($format eq 'rss') {\n-\t\tprint <<XML;\n+\t\tprint {$output_handler} <<XML;\n <rss version=\"2.0\" xmlns:content=\"http://purl.org/rss/1.0/modules/content/\">\n <channel>\n XML\n-\t\tprint \"<title>$title</title>\\n\" .\n+\t\tprint {$output_handler} \"<title>$title</title>\\n\" .\n \t\t      \"<link>$alt_url</link>\\n\" .\n \t\t      \"<description>$descr</description>\\n\" .\n \t\t      \"<language>en</language>\\n\" .\n@@ -6614,22 +6631,22 @@ XML\n \t\t\t# prefer the logo to the favicon, since RSS\n \t\t\t# doesn't allow both\n \t\t\tmy $img = esc_url($logo || $favicon);\n-\t\t\tprint \"<image>\\n\" .\n+\t\t\tprint {$output_handler} \"<image>\\n\" .\n \t\t\t      \"<url>$img</url>\\n\" .\n \t\t\t      \"<title>$title</title>\\n\" .\n \t\t\t      \"<link>$alt_url</link>\\n\" .\n \t\t\t      \"</image>\\n\";\n \t\t}\n \t\tif (%latest_date) {\n-\t\t\tprint \"<pubDate>$latest_date{'rfc2822'}</pubDate>\\n\";\n-\t\t\tprint \"<lastBuildDate>$latest_date{'rfc2822'}</lastBuildDate>\\n\";\n+\t\t\tprint {$output_handler} \"<pubDate>$latest_date{'rfc2822'}</pubDate>\\n\";\n+\t\t\tprint {$output_handler} \"<lastBuildDate>$latest_date{'rfc2822'}</lastBuildDate>\\n\";\n \t\t}\n-\t\tprint \"<generator>gitweb v.$version/$git_version</generator>\\n\";\n+\t\tprint {$output_handler} \"<generator>gitweb v.$version/$git_version</generator>\\n\";\n \t} elsif ($format eq 'atom') {\n-\t\tprint <<XML;\n+\t\tprint {$output_handler} <<XML;\n <feed xmlns=\"http://www.w3.org/2005/Atom\">\n XML\n-\t\tprint \"<title>$title</title>\\n\" .\n+\t\tprint {$output_handler} \"<title>$title</title>\\n\" .\n \t\t      \"<subtitle>$descr</subtitle>\\n\" .\n \t\t      '<link rel=\"alternate\" type=\"text/html\" href=\"' .\n \t\t      $alt_url . '\" />' . \"\\n\" .\n@@ -6639,19 +6656,19 @@ XML\n \t\t      # use project owner for feed author\n \t\t      \"<author><name>$owner</name></author>\\n\";\n \t\tif (defined $favicon) {\n-\t\t\tprint \"<icon>\" . esc_url($favicon) . \"</icon>\\n\";\n+\t\t\tprint {$output_handler} \"<icon>\" . esc_url($favicon) . \"</icon>\\n\";\n \t\t}\n \t\tif (defined $logo_url) {\n \t\t\t# not twice as wide as tall: 72 x 27 pixels\n-\t\t\tprint \"<logo>\" . esc_url($logo) . \"</logo>\\n\";\n+\t\t\tprint {$output_handler} \"<logo>\" . esc_url($logo) . \"</logo>\\n\";\n \t\t}\n \t\tif (! %latest_date) {\n \t\t\t# dummy date to keep the feed valid until commits trickle in:\n-\t\t\tprint \"<updated>1970-01-01T00:00:00Z</updated>\\n\";\n+\t\t\tprint {$output_handler} \"<updated>1970-01-01T00:00:00Z</updated>\\n\";\n \t\t} else {\n-\t\t\tprint \"<updated>$latest_date{'iso-8601'}</updated>\\n\";\n+\t\t\tprint {$output_handler} \"<updated>$latest_date{'iso-8601'}</updated>\\n\";\n \t\t}\n-\t\tprint \"<generator version='$version/$git_version'>gitweb</generator>\\n\";\n+\t\tprint {$output_handler} \"<generator version='$version/$git_version'>gitweb</generator>\\n\";\n \t}\n \n \t# contents\n@@ -6676,7 +6693,7 @@ XML\n \t\t# print element (entry, item)\n \t\tmy $co_url = href(-full=>1, action=>\"commitdiff\", hash=>$commit);\n \t\tif ($format eq 'rss') {\n-\t\t\tprint \"<item>\\n\" .\n+\t\t\tprint {$output_handler} \"<item>\\n\" .\n \t\t\t      \"<title>\" . esc_html($co{'title'}) . \"</title>\\n\" .\n \t\t\t      \"<author>\" . esc_html($co{'author'}) . \"</author>\\n\" .\n \t\t\t      \"<pubDate>$cd{'rfc2822'}</pubDate>\\n\" .\n@@ -6686,22 +6703,22 @@ XML\n \t\t\t      \"<content:encoded>\" .\n \t\t\t      \"<![CDATA[\\n\";\n \t\t} elsif ($format eq 'atom') {\n-\t\t\tprint \"<entry>\\n\" .\n+\t\t\tprint {$output_handler} \"<entry>\\n\" .\n \t\t\t      \"<title type=\\\"html\\\">\" . esc_html($co{'title'}) . \"</title>\\n\" .\n \t\t\t      \"<updated>$cd{'iso-8601'}</updated>\\n\" .\n \t\t\t      \"<author>\\n\" .\n \t\t\t      \"  <name>\" . esc_html($co{'author_name'}) . \"</name>\\n\";\n \t\t\tif ($co{'author_email'}) {\n-\t\t\t\tprint \"  <email>\" . esc_html($co{'author_email'}) . \"</email>\\n\";\n+\t\t\t\tprint {$output_handler} \"  <email>\" . esc_html($co{'author_email'}) . \"</email>\\n\";\n \t\t\t}\n-\t\t\tprint \"</author>\\n\" .\n+\t\t\tprint {$output_handler} \"</author>\\n\" .\n \t\t\t      # use committer for contributor\n \t\t\t      \"<contributor>\\n\" .\n \t\t\t      \"  <name>\" . esc_html($co{'committer_name'}) . \"</name>\\n\";\n \t\t\tif ($co{'committer_email'}) {\n-\t\t\t\tprint \"  <email>\" . esc_html($co{'committer_email'}) . \"</email>\\n\";\n+\t\t\t\tprint {$output_handler} \"  <email>\" . esc_html($co{'committer_email'}) . \"</email>\\n\";\n \t\t\t}\n-\t\t\tprint \"</contributor>\\n\" .\n+\t\t\tprint {$output_handler} \"</contributor>\\n\" .\n \t\t\t      \"<published>$cd{'iso-8601'}</published>\\n\" .\n \t\t\t      \"<link rel=\\\"alternate\\\" type=\\\"text/html\\\" href=\\\"$co_url\\\" />\\n\" .\n \t\t\t      \"<id>$co_url</id>\\n\" .\n@@ -6709,19 +6726,19 @@ XML\n \t\t\t      \"<div xmlns=\\\"http://www.w3.org/1999/xhtml\\\">\\n\";\n \t\t}\n \t\tmy $comment = $co{'comment'};\n-\t\tprint \"<pre>\\n\";\n+\t\tprint {$output_handler} \"<pre>\\n\";\n \t\tforeach my $line (@$comment) {\n \t\t\t$line = esc_html($line);\n-\t\t\tprint \"$line\\n\";\n+\t\t\tprint {$output_handler} \"$line\\n\";\n \t\t}\n-\t\tprint \"</pre><ul>\\n\";\n+\t\tprint {$output_handler} \"</pre><ul>\\n\";\n \t\tforeach my $difftree_line (@difftree) {\n \t\t\tmy %difftree = parse_difftree_raw_line($difftree_line);\n \t\t\tnext if !$difftree{'from_id'};\n \n \t\t\tmy $file = $difftree{'file'} || $difftree{'to_file'};\n \n-\t\t\tprint \"<li>\" .\n+\t\t\tprint {$output_handler} \"<li>\" .\n \t\t\t      \"[\" .\n \t\t\t      $cgi->a({-href => href(-full=>1, action=>\"blobdiff\",\n \t\t\t                             hash=>$difftree{'to_id'}, hash_parent=>$difftree{'from_id'},\n@@ -6729,26 +6746,26 @@ XML\n \t\t\t                             file_name=>$file, file_parent=>$difftree{'from_file'}),\n \t\t\t              -title => \"diff\"}, 'D');\n \t\t\tif ($have_blame) {\n-\t\t\t\tprint $cgi->a({-href => href(-full=>1, action=>\"blame\",\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(-full=>1, action=>\"blame\",\n \t\t\t\t                             file_name=>$file, hash_base=>$commit),\n \t\t\t\t              -title => \"blame\"}, 'B');\n \t\t\t}\n \t\t\t# if this is not a feed of a file history\n \t\t\tif (!defined $file_name || $file_name ne $file) {\n-\t\t\t\tprint $cgi->a({-href => href(-full=>1, action=>\"history\",\n+\t\t\t\tprint {$output_handler} $cgi->a({-href => href(-full=>1, action=>\"history\",\n \t\t\t\t                             file_name=>$file, hash=>$commit),\n \t\t\t\t              -title => \"history\"}, 'H');\n \t\t\t}\n \t\t\t$file = esc_path($file);\n-\t\t\tprint \"] \".\n+\t\t\tprint {$output_handler} \"] \".\n \t\t\t      \"$file</li>\\n\";\n \t\t}\n \t\tif ($format eq 'rss') {\n-\t\t\tprint \"</ul>]]>\\n\" .\n+\t\t\tprint {$output_handler} \"</ul>]]>\\n\" .\n \t\t\t      \"</content:encoded>\\n\" .\n \t\t\t      \"</item>\\n\";\n \t\t} elsif ($format eq 'atom') {\n-\t\t\tprint \"</ul>\\n</div>\\n\" .\n+\t\t\tprint {$output_handler} \"</ul>\\n</div>\\n\" .\n \t\t\t      \"</content>\\n\" .\n \t\t\t      \"</entry>\\n\";\n \t\t}\n@@ -6756,9 +6773,9 @@ XML\n \n \t# end of feed\n \tif ($format eq 'rss') {\n-\t\tprint \"</channel>\\n</rss>\\n\";\n+\t\tprint {$output_handler} \"</channel>\\n</rss>\\n\";\n \t} elsif ($format eq 'atom') {\n-\t\tprint \"</feed>\\n\";\n+\t\tprint {$output_handler} \"</feed>\\n\";\n \t}\n }\n \n@@ -6772,13 +6789,12 @@ sub git_atom {\n \n sub git_opml {\n \tmy @list = git_get_projects_list();\n-\n-\tprint $cgi->header(\n+\tprint {$output_handler} $cgi->header(\n \t\t-type => 'text/xml',\n \t\t-charset => 'utf-8',\n \t\t-content_disposition => 'inline; filename=\"opml.xml\"');\n \n-\tprint <<XML;\n+\tprint {$output_handler} <<XML;\n <?xml version=\"1.0\" encoding=\"utf-8\"?>\n <opml version=\"1.0\">\n <head>\n@@ -6803,9 +6819,9 @@ XML\n \t\tmy $path = esc_html(chop_str($proj{'path'}, 25, 5));\n \t\tmy $rss  = href('project' => $proj{'path'}, 'action' => 'rss', -full => 1);\n \t\tmy $html = href('project' => $proj{'path'}, 'action' => 'summary', -full => 1);\n-\t\tprint \"<outline type=\\\"rss\\\" text=\\\"$path\\\" title=\\\"$path\\\" xmlUrl=\\\"$rss\\\" htmlUrl=\\\"$html\\\"/>\\n\";\n+\t\tprint {$output_handler} \"<outline type=\\\"rss\\\" text=\\\"$path\\\" title=\\\"$path\\\" xmlUrl=\\\"$rss\\\" htmlUrl=\\\"$html\\\"/>\\n\";\n \t}\n-\tprint <<XML;\n+\tprint {$output_handler} <<XML;\n </outline>\n </body>\n </opml>\n-- \n1.6.5.2\n"},{"id":"131568","messageId":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","threadId":"22216","inReplyTo":"1263432185-21334-9-git-send-email-warthog9@eaglescrag.net","subject":"[PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)","fromName":"John 'Warthog9' Hawley","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-14T01:23:05Z","receivedAt":"2010-01-14T01:23:05Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"This is a very large patch that implements the file based\ncaching layer that is used on such large sites as kernel.org and\nsoon git.fedoraproject.org.  This provides a simple, and straight\nforward caching mechanism that scales dramatically better than\nGitweb by itself.\n\nThe caching layer basically buffers the output that Gitweb would\nnormally return, and saves that output to a cache file on the local\ndisk.  When the file is requested it attempts to gain a shared lock\non the cache file and cat it out to the client.  Should an exclusive\nlock be on a file (it's being updated) the code has a choice to either\nupdate in the background and go ahead and show the stale page while\nupdate is being performed, or stall the client(s) until the page\nis generated.\n\nThere are two forms of stalling involved here, background building\nand non-background building, both of which are discussed in the\nconfiguration page.\n\nThere are still a few known \"issues\" with respect to this:\n- Code needs to be added to be \"browser\" aware so\n  that clients like wget that are trying to get a\n  binary blob don't obtain a \"Generating...\" page\n- There is an intermittent flushing issue that has yet\n  to be tracked down\n\nCaching is disabled by default with the $cache_enable variable,\nsetting this to 1 will enable file based caching.  It is expected\nthat this will be extended to include additional types of caching\n(like memcached) in the future and should not be exclusively\nconsidered a binary value.\n---\n gitweb/cache.pm    |  283 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n gitweb/gitweb.css  |    6 +\n gitweb/gitweb.perl |   58 ++++++++++-\n 3 files changed, 344 insertions(+), 3 deletions(-)\n create mode 100644 gitweb/cache.pm\n\ndiff --git a/gitweb/cache.pm b/gitweb/cache.pm\nnew file mode 100644\nindex 0000000..d08bcec\n--- /dev/null\n+++ b/gitweb/cache.pm\n@@ -0,0 +1,283 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2006, John 'Warthog9' Hawley <warthog19@eaglescrag.net>\n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Gitweb caching engine\n+#\n+\n+use File::Path qw(make_path remove_tree);\n+use Digest::MD5 qw(md5 md5_hex md5_base64);\n+use Fcntl ':flock';\n+\n+sub cache_fetch {\n+\tmy ($action) = @_;\n+\tmy $cacheTime = 0;\n+\n+\t# Deal with cache being disabled\n+\tif( $cache_enable == 0 ){\n+\t\t$output_handler = *STDOUT;\n+\t\t$output_handler_bin = *STDOUT;\n+\t\t$actions{$action}->();\n+\t\treturn;\n+\t}elsif( $cache_enable == 1 ){\n+\t\t#obviously we are using file based caching\n+\n+\t\tif(! -d $cachedir){\n+\t\t\tprint \"*** Warning ***: Caching enabled but cache directory does not exsist.  ($cachedir)\\n\";\n+\t\t\tmkdir (\"cache\", 0665) || die \"Cannot create cache dir - you will need to manually create\";\n+\t\t\tprint \"Cache directory created successfully\\n\";\n+\t\t}\n+\n+\t\tour $full_url = \"$my_url?\". $ENV{'QUERY_STRING'};\n+\t\tour $urlhash = md5_hex($full_url);\n+\t\tour $fullhashdir = \"$cachedir/\". substr( $urlhash, 0, 2) .\"/\";\n+\n+\t\tmy $numdirs = make_path( $fullhashdir, { mode => 0777, error => \\my $mkdirerr, } );\n+\t\tif( @$mkdirerr ){\n+\t\t\tmy $mkdirerrmsg = \"\";\n+\t\t\tfor my $diag (@$mkdirerr) {\n+\t\t\t\tmy ($file, $message) = %$diag;\n+\t\t\t\tif($file eq '' ){\n+\t\t\t\t\t$mkdirerrmsg .= \"general error: $message\\n\";\n+\t\t\t\t}else{\n+\t\t\t\t\t$mkdirerrmsg .= \"problem unlinking $file: $message\\n\";\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tdie_error(500, \"Could not create cache directory | $mkdirerrmsg\");\n+\t\t}\n+\t\t$fullhashpath = \"$fullhashdir/\". substr( $urlhash, 2 );\n+\t\t$fullhashbinpath = \"$fullhashpath.bin\";\n+\t} # done dealing with cache enabled / disabled\n+\n+\tif(! -e \"$fullhashpath\" ){\n+\t\tif(! defined(my $childPid = fork()) ){\n+\t\t\tcacheUpdate($action,0);\n+\t\t\tcacheDisplay($action);\n+\t\t} elsif ( $childPid == 0 ){\n+\t\t\t#run the updater\n+\t\t\tcacheUpdate($action,1);\n+\t\t}else{\n+\t\t\tcacheWaitForUpdate($action);\n+\t\t}\n+\t}else{\n+\t\t#if cache is out dated, update\n+\t\t#else displayCache();\n+\t\topen(cacheFile, '<', \"$fullhashpath\");\n+\t\tstat(cacheFile);\n+\t\tclose(cacheFile);\n+\t\t$cacheTime = get_loadavg() * 60;\n+\t\tif( $cacheTime > $maxCacheTime ){\n+\t\t\t$cacheTime = $maxCacheTime;\n+\t\t}\n+\t\tif( $cacheTime < $minCacheTime ){\n+\t\t\t$cacheTime = $minCacheTime;\n+\t\t}\n+\t\tif( (stat(_))[9] < (time - $cacheTime) ){\n+\t\t\tif( ! defined(my $childPid = fork()) ){\n+\t\t\t\tcacheUpdate($action,0);\n+\t\t\t\tcacheDisplay($action);\n+\t\t\t} elsif ( $childPid == 0 ){\n+\t\t\t\t#run the updater\n+\t\t\t\t#print \"Running updater\\n\";\n+\t\t\t\tcacheUpdate($action,1);\n+\t\t\t}else{\n+\t\t\t\t#print \"Waiting for update\\n\";\n+\t\t\t\tcacheWaitForUpdate($action);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tcacheDisplay($action);\n+\t\t}\n+\n+\n+\t}\n+\n+\t#\n+\t# If all of the caching failes - lets go ahead and press on without it and fall back to 'default'\n+\t# non-caching behavior.  This is the softest of the failure conditions.\n+\t#\n+\t#$actions{$action}->();\n+}\n+\n+sub cacheUpdate {\n+\tmy ($action,$areForked) = @_;\n+\tmy $lockingStatus;\n+\tmy $fileData = \"\";\n+\n+\tif($backgroundCache){\n+\t\topen(cacheFileBG, '>:utf8', \"$fullhashpath.bg\");\n+\t\tmy $lockStatBG = flock(cacheFileBG,LOCK_EX|LOCK_NB);\n+\n+\t\t$lockStatus = $lockStatBG;\n+\t}else{\n+\t\topen(cacheFile, '>:utf8', \"$fullhashpath\");\n+\t\tmy $lockStat = flock(cacheFile,LOCK_EX|LOCK_NB);\n+\n+\t\t$lockStatus = $lockStat;\n+\t}\n+\t#print \"lock status: $lockStat\\n\";\n+\n+\n+\tif (! $lockStatus ){\n+\t\tif ( $areForked ){\n+\t\t\texit(0);\n+\t\t}else{\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tif(\n+\t\t$action eq \"snapshot\"\n+\t\t||\n+\t\t$action eq \"blob_plain\"\n+\t){\n+\t\topen cacheFileBin, '>', $fullhashbinpath or die_error(500, \"Could not open bin dump file\");\n+\t\t$output_handler_bin = *cacheFileBin;\n+\t}\n+\n+\t$output_handler = *cacheFile;\n+\n+\tif($backgroundCache){\n+\t\topen(cacheFile, '>:utf8', \"$fullhashpath\");\n+\t\t$lockStat = flock(cacheFile,LOCK_EX);\n+\n+\t\tif (! $lockStat ){\n+\t\t\tif ( $areForked ){\n+\t\t\t\texit(0);\n+\t\t\t}else{\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\t$actions{$action}->();\n+\n+\tif(\n+\t\t$action eq \"snapshot\"\n+\t\t||\n+\t\t$action eq \"blob_plain\"\n+\t){\n+\t\tclose(cacheFileBin);\n+\t}\n+\n+\tflock(cacheFile,LOCK_UN);\n+\tclose(cacheFile);\n+\n+\tif($backgroundCache){\n+\t\tflock(cacheFileBG,LOCK_UN);\n+\t\tclose(cacheFileBG);\n+\t}\n+\n+\tif ( $areForked ){\n+\t\texit(0);\n+\t} else {\n+\t\treturn;\n+\t}\n+}\n+\n+\n+sub cacheWaitForUpdate {\n+\tmy ($action) = @_;\n+\tmy $x = 0;\n+\tmy $max = 10;\n+\tmy $lockStat = 0;\n+\n+\tif( $backgroundCache ){\n+\t\tif( -e \"$fullhashpath\" ){\n+\t\t\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n+\t\t\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n+\t\t\tstat(cacheFile);\n+\t\t\tclose(cacheFile);\n+\n+\t\t\tif( $lockStat && ( (stat(_))[9] > (time - $maxCacheLife) ) ){\n+\t\t\t\tcacheDisplay($action);\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif(\n+\t\t$action eq \"atom\"\n+\t\t||\n+\t\t$action eq \"rss\"\n+\t\t||\n+\t\t$action eq \"opml\"\n+\t){\n+\t\tdo {\n+\t\t\tsleep 2 if $x > 0;\n+\t\t\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n+\t\t\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n+\t\t\tclose(cacheFile);\n+\t\t\t$x++;\n+\t\t\t$combinedLockStat = $lockStat;\n+\t\t} while ((! $combinedLockStat) && ($x < $max));\n+\n+\t\tif( $x != $max ){\n+\t\t\tcacheDisplay($action);\n+\t\t}\n+\t\treturn;\n+\t}\n+\n+\t$| = 1;\n+\n+\tprint $::cgi->header(-type=>'text/html', -charset => 'utf-8',\n+\t                   -status=> 200, -expires => 'never');\n+\n+\tprint <<EOF;\n+<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www/w3.porg/TR/html4/strict.dtd\">\n+<!-- git web w/caching interface version $version, (C) 2006-2010, John 'Warthog9' Hawley <warthog9\\@kernel.org> -->\n+<!-- git core binaries version $git_version -->\n+<head>\n+<meta http-equiv=\"content-type\" content=\"$content_type; charset=utf-8\"/>\n+<meta name=\"generator\" content=\"gitweb/$version git/$git_version\"/>\n+<meta name=\"robots\" content=\"index, nofollow\"/>\n+<meta http-equiv=\"refresh\" content=\"0\"/>\n+<title>$title</title>\n+</head>\n+<body>\n+EOF\n+\n+\tprint \"Generating..\";\n+\tdo {\n+\t\tprint \".\";\n+\t\tsleep 2 if $x > 0;\n+\t\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n+\t\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n+\t\tclose(cacheFile);\n+\t\t$x++;\n+\t\t$combinedLockStat = $lockStat;\n+\t} while ((! $combinedLockStat) && ($x < $max));\n+\tprint <<EOF;\n+</body>\n+</html>\n+EOF\n+\treturn;\n+}\n+\n+sub cacheDisplay {\n+\tmy ($action) = @_;\n+\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n+\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n+\tif (! $lockStat ){\n+\t\tclose(cacheFile);\n+\t\tcacheWaitForUpdate($action);\n+\t}\n+\n+\twhile( <cacheFile> ){\n+\t\tprint $_;\n+\t}\n+\tif(\n+\t\t$action eq \"snapshot\"\n+\t\t||\n+\t\t$action eq \"blob_plain\"\n+\t){\n+\t\topen(cacheFileBin, '<', \"$fullhashbinpath\");\n+\t\tbinmode STDOUT, ':raw';\n+\t\tprint <cacheFileBin>;\n+\t\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n+\t\tclose(cacheFileBin);\n+\t}\n+\tclose(cacheFile);\n+}\ndiff --git a/gitweb/gitweb.css b/gitweb/gitweb.css\nindex 50067f2..f809f16 100644\n--- a/gitweb/gitweb.css\n+++ b/gitweb/gitweb.css\n@@ -67,6 +67,12 @@ div.page_path {\n \tborder-width: 0px 0px 1px;\n }\n \n+div.cachetime {\n+\tfloat: left;\n+\tmargin-right: 10px;\n+\tcolor: #555555;\n+}\n+\n div.page_footer {\n \theight: 17px;\n \tpadding: 4px 8px;\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 8bb323c..ec95bb9 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -230,6 +230,50 @@ our $git_versions_must_match = 1;\n # Leave it undefined (or set to 'undef') to turn off load checking.\n our $maxload = 300;\n \n+# This enables/disables the caching layer in gitweb.  This currently only supports the\n+# 'dumb' file based caching layer, primarily used on git.kernel.org.  this is reasonably\n+# effective but it has the downside of requiring a huge amount of disk space if there\n+# are a number of repositories involved.  It is not uncommon for git.kernel.org to have\n+# on the order of 80G - 120G accumulate over the course of a few months.  It is recommended\n+# that the cache directory be periodically completely deleted, and this is safe to perform.\n+# Suggested mechanism\n+# mv $cacheidr $cachedir.flush;mkdir $cachedir;rm -rf $cachedir.flush\n+# Value is binary. 0 = disabled (default), 1 = enabled.\n+#\n+# Values of caching:\n+# \t1 = 'dumb' file based caching used on git.kernel.org\n+our $cache_enable = 0;\n+\n+# Used to set the minimum cache timeout for the dynamic caching algorithm.  Basically\n+# if we calculate the cache to be under this number of seconds we set the cache timeout\n+# to this minimum.\n+# Value is in seconds.  1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour\n+our $minCacheTime = 20;\n+\n+# Used to set the maximum cache timeout for the dynamic caching algorithm.  Basically\n+# if we calculate the cache to exceed this number of seconds we set the cache timeout\n+# to this maximum.\n+# Value is in seconds.  1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour\n+our $maxCacheTime = 1200;\n+\n+# If you need to change the location of the caching directory, override this\n+# otherwise this will probably do fine for you\n+our $cachedir = 'cache';\n+\n+# If this is set (to 1) cache will do it's best to always display something instead\n+# of making someone wait for the cache to update.  This will launch the cacheUpdate\n+# into the background and it will lock a <file>.bg file and will only lock the\n+# actual cache file when it needs to write into it.  In theory this will make\n+# gitweb seem more responsive at the price of possibly stale data.\n+our $backgroundCache = 1;\n+\n+# Used to set the maximum cache file life.  If a cache files last modify time exceeds\n+# this value, it will assume that the data is just too old, and HAS to be regenerated\n+# instead of trying to display the existing cache data.\n+# Value is in seconds.  1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour\n+# 18000 = 5 hours\n+our $maxCacheLife = 18000;\n+\n # You define site-wide feature defaults here; override them with\n # $GITWEB_CONFIG as necessary.\n our %feature = (\n@@ -593,6 +637,11 @@ if (defined $maxload && get_loadavg() > $maxload) {\n \tdie_error(503, \"The load average on the server is too high\");\n }\n \n+#\n+# Includes\n+#\n+do 'cache.pm';\n+\n # version of the core git binary\n our $git_version = qx(\"$GIT\" --version) =~ m/git version (.*)$/ ? $1 : \"unknown\";\n $number_of_git_cmds++;\n@@ -994,7 +1043,7 @@ if ($action !~ m/^(?:opml|project_list|project_index)$/ &&\n     !$project) {\n \tdie_error(400, \"Project needed\");\n }\n-$actions{$action}->();\n+cache_fetch($action);\n exit;\n \n ## ======================================================================\n@@ -3200,7 +3249,9 @@ sub git_header_html {\n \t# support xhtml+xml but choking when it gets what it asked for.\n \tif (defined $cgi->http('HTTP_ACCEPT') &&\n \t    $cgi->http('HTTP_ACCEPT') =~ m/(,|;|\\s|^)application\\/xhtml\\+xml(,|;|\\s|$)/ &&\n-\t    $cgi->Accept('application/xhtml+xml') != 0) {\n+\t    $cgi->Accept('application/xhtml+xml') != 0\n+\t    &&\n+\t    $cache_enable == 0) {\n \t\t$content_type = 'application/xhtml+xml';\n \t} else {\n \t\t$content_type = 'text/html';\n@@ -3344,6 +3395,7 @@ sub git_footer_html {\n \tmy $feed_class = 'rss_logo';\n \n \tprint {$output_handler} \"<div class=\\\"page_footer\\\">\\n\";\n+\tprint {$output_handler} \"<div class=\\\"cachetime\\\">Cache Last Updated: \". gmtime( time ) .\" GMT</div>\\n\";\n \tif (defined $project) {\n \t\tmy $descr = git_get_project_description($project);\n \t\tif (defined $descr) {\n@@ -3424,7 +3476,7 @@ sub die_error {\n \tmy $extra = shift;\n \n \t# The output handlers for die_error need to be reset to STDOUT\n-\t# so that half the message isn't being output to random and \n+\t# so that half the message isn't being output to random and\n \t# half to STDOUT as expected.  This is mainly for the benefit\n \t# of using git_header_html() and git_footer_html() since those\n \t# internaly use the indirect print handler.\n-- \n1.6.5.2\n"},{"id":"131698","messageId":"m37hrkdu4k.fsf@localhost.localdomain","threadId":"22216","inReplyTo":"1263432185-21334-1-git-send-email-warthog9@eaglescrag.net","subject":"Re: [PATCH 0/9] Gitweb caching v5","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-15T01:40:08Z","receivedAt":"2010-01-15T01:40:08Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n\n> Afternoon everyone,\n>  \n> This is the latest incarnation of gitweb w/ caching.  This is\n> finally at the point where it should probably start either being\n> considered for inclusion or mainline, or I need to accept that this\n> will never get in and more perminantely fork (as is the case with\n> Fedora where this is going in as gitweb-caching as a parrallel rpm\n> package).\n> \n> That said this brings the base up to mainline (again),\n\nCould you tell us which commit is the base of this series (like in\ngit-request-pull output), i.e. which commit this series is rebased\nagainst?\n\n> it updates a\n> number of elements in the caching engine, and this is a much cleaner\n> break-out of the tree vs. what I am currently developing against.\n\nIs caching engine part changed since v2?\n\n> v5:\n> \t- Missed a couple of things that were in my local tree, and\n> \t  added them back in.\n\nThat doesn't tell us much.\n\n> \t- Split up the die_error and the version matching patch\n> \t- Set version matching to be on by default - otherwise this\n> \t  really is code that will never get checked, or at best\n> \t  enabled by default by distributions\n> \t- Added a minor code cleanup with respect to $site_header\n> \t  that was already in my tree\n> \t- Applied against a more recent git tree vs. 1.6.6-rc2\n> \t- Removed breakout patch for now (did that in v4 actually)\n> \t  and will deal with that separately \n> \n> \thttp://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v5\n \n----\nShort comments about patches in this series; I will be sending\ndetailed comments for each patch individually.\n\n> John 'Warthog9' Hawley (9):\n>   gitweb: Load checking\nLooks good.\n\n>   gitweb: change die_error to take \"extra\" argument for extended die\n>     information\nCommit message could be better (summary should really be shorter), and\nI think there is some indent typo, but otherwise looks good.\n\n>   gitweb: Add option to force version match\nFor me it needs to be disabled in gitweb test suite (t/gitweb-lib.sh),\nif it is enabled by default.  I really like that I can test current\ngitweb without need to recompile git.\n\nAlso it should have tests that it works as intended (both for matching\nand non-matching versions) in t/t9501-gitweb-standalone-http-status.sh\n\n>   gitweb: Makefile improvements\nDoes it differ from my proposal (i.e. gitweb/Makefile doing the work),\nbased on your idea (\"make gitweb\" for Makefile and gitweb/Makefile)?\n\n>   gitweb: add a get function to compliment print_local_time\n>   gitweb: add a get function to compliment print_sort_th\nThose two looks O.K. from what I seen.\n\n>   gitweb: cleanup error message produced by undefined $site_header\nShouldn't there be such protection for other such variables, like\n$site_footer and $home_text (and a bit diferent protection against\nundefined $projects_list)?  By the way, how did you arrived at\nundefined $site_header: deafult build configuration leaves it empty,\nbut defined. \n\n>   gitweb: Convert output to using indirect file handle\nI have alternate solution, using shorter filehandle name (just $out)\nin\n\n  git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel\n  http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel\n\nI would have to think a bit about separate handle for binary files;\nI am not sure if it is really required.\n\n>   gitweb: File based caching layer (from git.kernel.org)\nI am working (time permitting) in spliting this large code drop into\nsmaller commits, namely:\n\n - href(..., -path_info => 0)          (for cache key)\n - simple file based caching + tests\n - global expire time + tests\n - output caching in gitweb            (WIP)\n - adaptive expiration time            (planned)\n - tee output / cache write            (planned)\n - expire time variation from CHI      (planned)\n - locking for single writer           (planned)\n - server-side generating info         (planned)\n - AJAX-y generating info              (wishlist)\n\nwhile ensuring that it pass all existing gitweb tests, and adding new\ntests for new features.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"131700","messageId":"4B4FEF33.2040605@eaglescrag.net","threadId":"22216","inReplyTo":"m37hrkdu4k.fsf@localhost.localdomain","subject":"Re: [PATCH 0/9] Gitweb caching v5","fromName":"J.H.","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-15T04:29:39Z","receivedAt":"2010-01-15T04:29:39Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"On 01/14/2010 05:40 PM, Jakub Narebski wrote:\n> \"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n> \n>> Afternoon everyone,\n>>  \n>> This is the latest incarnation of gitweb w/ caching.  This is\n>> finally at the point where it should probably start either being\n>> considered for inclusion or mainline, or I need to accept that this\n>> will never get in and more perminantely fork (as is the case with\n>> Fedora where this is going in as gitweb-caching as a parrallel rpm\n>> package).\n>>\n>> That said this brings the base up to mainline (again),\n> \n> Could you tell us which commit is the base of this series (like in\n> git-request-pull output), i.e. which commit this series is rebased\n> against?\n\nThis series was based on\ngit://git.kernel.org/pub/scm/git/git.git\n054d2fa05cf0bc55fe1556c9e87d58d67a144f44\n\nhttp://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v5\n\n> \n>> it updates a\n>> number of elements in the caching engine, and this is a much cleaner\n>> break-out of the tree vs. what I am currently developing against.\n> \n> Is caching engine part changed since v2?\n\nSlightly, not dramatically.  Only changes were to modify the caching\nengine to deal with the change in output mechanism (I.E. my $output ->\nprint {$fh})\n\n<snip>\n>> John 'Warthog9' Hawley (9):\n>>   gitweb: Load checking\n> Looks good.\n> \n>>   gitweb: change die_error to take \"extra\" argument for extended die\n>>     information\n> Commit message could be better (summary should really be shorter), and\n> I think there is some indent typo, but otherwise looks good.\n> \n>>   gitweb: Add option to force version match\n> For me it needs to be disabled in gitweb test suite (t/gitweb-lib.sh),\n> if it is enabled by default.  I really like that I can test current\n> gitweb without need to recompile git.\n> \n> Also it should have tests that it works as intended (both for matching\n> and non-matching versions) in t/t9501-gitweb-standalone-http-status.sh\n\nI'll get t9501 cleaned up and make sure that the tests default to\nturning it off, and I'll add a test to confirm that this works.\n\n>>   gitweb: Makefile improvements\n> Does it differ from my proposal (i.e. gitweb/Makefile doing the work),\n> based on your idea (\"make gitweb\" for Makefile and gitweb/Makefile)?\n\nI think this is taken straight from the version you had, I don't think\nI've modified it.\n\n> \n>>   gitweb: add a get function to compliment print_local_time\n>>   gitweb: add a get function to compliment print_sort_th\n> Those two looks O.K. from what I seen.\n> \n>>   gitweb: cleanup error message produced by undefined $site_header\n> Shouldn't there be such protection for other such variables, like\n> $site_footer and $home_text (and a bit diferent protection against\n> undefined $projects_list)?  By the way, how did you arrived at\n> undefined $site_header: deafult build configuration leaves it empty,\n> but defined.\n\nI would have to go back and figure it out, but it's something I hit\nyears ago and added that check to keep it from spewing all over my logs.\n Could easily add it to the others mentioned.\n\n>>   gitweb: Convert output to using indirect file handle\n> I have alternate solution, using shorter filehandle name (just $out)\n> in\n> \n>   git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel\n>   http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel\n> \n> I would have to think a bit about separate handle for binary files;\n> I am not sure if it is really required.\n\nFor caching you have to have it.  When your outputing the data back from\nthe cache you need to switch the output mode for the browser to receive\nthe data properly.  Otherwise the resulting output from the caching\nengine is going to be garbage.  The caching engine explicitly stores the\nbinary files separate from the rest of the response.\n\n>>   gitweb: File based caching layer (from git.kernel.org)\n> I am working (time permitting) in spliting this large code drop into\n> smaller commits, namely:\n\nfirst up, is there a reason not to take the caching layer as it stands\nwhile you work on these?  I'm fine with adding test cases for what's\nthere now if you want, but I guess I'm confused about explicitly wanting\nto break these into smaller commits.\n\n>  - href(..., -path_info => 0)          (for cache key)\n\nYou actually *really* want to have the full url vs. just the path_info.\n While I accept that this means that you will end up with multiple\ncopies of data being stored it helps dramatically if you have multiple\nsites pointing into the same caching space.  If you happen to have two\ndistinct trees\n\nhttp://git.public.com/?p=test.git;a=summary\nhttp://git.private.com/?p=test.git;a=summary\n\nThat respectively point to:\n\n/group/public/git/test.git\n/group/private/git/test.git\n\nyou'll end up squashing the cache files needlessly and erroneously as\nwhat's in the cache file will depend on what last site was hit that\ngenerated the file.\n\n>  - simple file based caching + tests\n>  - global expire time + tests\n>  - output caching in gitweb            (WIP)\n>  - adaptive expiration time            (planned)\n>  - tee output / cache write            (planned)\n\nYou sadly can't 'tee' the output as this would re-introduce the\nstampeding heard problem which is one of the reasons the caching layer\ncame about in the first place.  Suppose you could give one person the\noutput but make everyone else wait for the cache to finish writing out,\nor have the waiting client processes tail the file while it's generated\nbut those both seem a little excessive vs. just waiting.\n\n>  - expire time variation from CHI      (planned)\n>  - locking for single writer           (planned)\n>  - server-side generating info         (planned)\n>  - AJAX-y generating info              (wishlist)\n\nIf it's helpful I can genuinely devote several more days to this to get\nthese cleaned up.  Which would save you a fair amount of time in\nbreaking this up.\n\n- John 'Warthog9' Hawley\n"},{"id":"131712","messageId":"201001151128.11211.jnareb@gmail.com","threadId":"22216","inReplyTo":"4B4FEF33.2040605@eaglescrag.net","subject":"Re: [PATCH 0/9] Gitweb caching v5","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-15T10:28:10Z","receivedAt":"2010-01-15T10:28:10Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Fri, 15 Jan 2010, J.H. wrote:\n> On 01/14/2010 05:40 PM, Jakub Narebski wrote:\n>> \"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n\n>>> it updates a\n>>> number of elements in the caching engine, and this is a much cleaner\n>>> break-out of the tree vs. what I am currently developing against.\n>> \n>> Is caching engine part changed since v2?\n> \n> Slightly, not dramatically.  Only changes were to modify the caching\n> engine to deal with the change in output mechanism (I.E. my $output ->\n> print {$fh})\n\nO.K.\n\nIt is still quite large code dump...\n\n>>> John 'Warthog9' Hawley (9):\n[...]\n>>>   gitweb: Add option to force version match\n>> For me it needs to be disabled in gitweb test suite (t/gitweb-lib.sh),\n>> if it is enabled by default.  I really like that I can test current\n>> gitweb without need to recompile git.\n>> \n>> Also it should have tests that it works as intended (both for matching\n>> and non-matching versions) in t/t9501-gitweb-standalone-http-status.sh\n> \n> I'll get t9501 cleaned up and make sure that the tests default to\n> turning it off, and I'll add a test to confirm that this works.\n\nI'll send you proposed fixup to squash, so if you want you can simply\nwait for a bit.\n\n[...] \n>>>   gitweb: cleanup error message produced by undefined $site_header\n>> Shouldn't there be such protection for other such variables, like\n>> $site_footer and $home_text (and a bit diferent protection against\n>> undefined $projects_list)?  By the way, how did you arrived at\n>> undefined $site_header: deafult build configuration leaves it empty,\n>> but defined.\n> \n> I would have to go back and figure it out, but it's something I hit\n> years ago and added that check to keep it from spewing all over my logs.\n> Could easily add it to the others mentioned.\n\nPlease remember that while for $home_text, $site_header and $site_footer\nit would be enough to replace\n\n  if (-f $file_variable) {\n  \t...\n  }\n\nwith\n\n  if (defined $file_variable && -f $file_variable) {\n  \t...\n  }\n\nfor undefined $projects_list you have to fail (use die_error).\n\n\nDefensive programming, here protecting against errors in gitweb config,\nis a good idea IMHO.\n\n>>>   gitweb: Convert output to using indirect file handle\n>> I have alternate solution, using shorter filehandle name (just $out)\n>> in\n>> \n>>   git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel\n>>   http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel\n>> \n>> I would have to think a bit about separate handle for binary files;\n>> I am not sure if it is really required.\n> \n> For caching you have to have it.  When your outputing the data back from\n> the cache you need to switch the output mode for the browser to receive\n> the data properly.  Otherwise the resulting output from the caching\n> engine is going to be garbage.  The caching engine explicitly stores the\n> binary files separate from the rest of the response.\n\nI think you don't.  Just read from cache file and print to browser in\nbinary (:raw) mode; IIUC the ':utf8' is needed only when you print from\nPerl.\n\n>>>   gitweb: File based caching layer (from git.kernel.org)\n>> I am working (time permitting) in spliting this large code drop into\n>> smaller commits, namely:\n> \n> first up, is there a reason not to take the caching layer as it stands\n> while you work on these?  I'm fine with adding test cases for what's\n> there now if you want, but I guess I'm confused about explicitly wanting\n> to break these into smaller commits.\n\nIt's there a reason to take this big code dump while we (me and you) are\nworking on it?  As I see there is no other work on gitweb now, and for the\ntime being you can run your fork for a bit.  Or is it about planned\ndeployment on git.fedora.org or something?  Anyway when there is official\nrelease of git including gitweb output caching, it should be easy to switch\noff from caching fork.\n \nThe caching code has also a few issues: coding style, Perl style (e.g.\nnot using $cache_enabled / $caching_enabled as boolean), duplicated code,\nunnecessary open/close dance around stat (you can stat file which is not\nclosed), not using atomic write (I'm not sure how important this issue\nis, especially with locking), not being able to replace cache by some\nother caching module, etc.\n\n\n>>  - href(..., -path_info => 0)          (for cache key)\n> \n> You actually *really* want to have the full url vs. just the path_info.\n\nErrr... it is not about full url (although I errorneously used relative\nURL for human-readable cache key) vs only path_info, but about path_info\n(for example http://git.public.com/test.git) and query (for example\nhttp://git.public.com/?p=test.git) URLs which leads to exactly the same\nview.\n\n> While I accept that this means that you will end up with multiple\n> copies of data being stored it helps dramatically if you have multiple\n> sites pointing into the same caching space.  If you happen to have two\n> distinct trees\n> \n> http://git.public.com/?p=test.git;a=summary\n> http://git.private.com/?p=test.git;a=summary\n> \n> That respectively point to:\n> \n> /group/public/git/test.git\n> /group/private/git/test.git\n> \n> you'll end up squashing the cache files needlessly and erroneously as\n> what's in the cache file will depend on what last site was hit that\n> generated the file.\n\nSo the key (the human-readable key, not it's MD5 digest) would be\n\n  href(-replay=>1, -full_url=>1, -path_info=>0);\n\n>>  - simple file based caching + tests\n>>  - global expire time + tests\n>>  - output caching in gitweb            (WIP)\n>>  - adaptive expiration time            (planned)\n>>  - tee output / cache write            (planned)\n> \n> You sadly can't 'tee' the output as this would re-introduce the\n> stampeding heard problem which is one of the reasons the caching layer\n> came about in the first place.  Suppose you could give one person the\n> output but make everyone else wait for the cache to finish writing out,\n> or have the waiting client processes tail the file while it's generated\n> but those both seem a little excessive vs. just waiting.\n\nThe \"tee\" is about serving [one person] the output, while cache is being\nwritten.  It is a bit orthogonal issue to the 'stampeding herd' protection.\n \n>>  - expire time variation from CHI      (planned)\n\nThis feature is meant to prevent cache miss stampedes[1]\n\n[1] http://search.cpan.org/~jswartz/CHI-0.33/lib/CHI.pm#expires_variance\n\n>>  - locking for single writer           (planned)\n\n...or rather locking to avoid 'stampeding herd' problem.\n\nBTW I don't recall if it isn't the solution you used, but if there exists\nstale cache entry, the other clients can use stale version instead of\nforcing to wait till the writer finishes updating cache.\n\n>>  - server-side generating info         (planned)\n>>  - AJAX-y generating info              (wishlist)\n> \n> If it's helpful I can genuinely devote several more days to this to get\n> these cleaned up.  Which would save you a fair amount of time in\n> breaking this up.\n\nI'm in the middle of working on gitweb caching, with the intent of arriving\nat (super)set of features that are in your patch series.\n\nSee http://repo.or.cz/w/?p=git/jnareb-git.git;a=shortlog;h=refs/heads/gitweb/cache-kernel\n(this branch is rebased / amended / rewritten).\n\n-- \nJakub Narebski\nPoland\n"},{"id":"131794","messageId":"m33a27dmsk.fsf@localhost.localdomain","threadId":"22216","inReplyTo":"1263432185-21334-2-git-send-email-warthog9@eaglescrag.net","subject":"Re: [PATCH 1/9] gitweb: Load checking","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-15T22:30:25Z","receivedAt":"2010-01-15T22:30:25Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This one looks good, but while examining other patch in this series\nI have noticed rare situation where we would get Perl error with\nthis patch.\n\nI have added fix-up for this issue, although I guess that better\nsolution might be not to add any <script> element for git_footer_html\ncalled from die_error.\n\nI'm sorry I haven't noticed this earlier.\n\n\n\"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n\n> From: John 'Warthog9' Hawley <warthog9@kernel.org>\n> \n> This changes slightly the behavior of gitweb, so that it verifies\n> that the box isn't inundated with before attempting to serve gitweb.\n> If the box is overloaded, it basically returns a 503 Server Unavailable\n> until the load falls below the defined threshold.  This helps dramatically\n> if you have a box that's I/O bound, reaches a certain load and you\n> don't want gitweb, the I/O hog that it is, increasing the pain the\n> server is already undergoing.\n> \n> This behavior is controlled by $maxload configuration variable.\n> Default is a load of 300, which for most cases should never be hit.\n> Unset it (set it to undefined value, i.e. undef) to turn off checking.\n> \n> Currently it requires that '/proc/loadavg' file exists, otherwise the\n> load check is bypassed (load is taken to be 0).  So platforms that do\n> not implement '/proc/loadavg' currently cannot use this feature.\n> (provisions are included for additional checks to be added by others)\n>\nWhile at it check that $action is defined before comparing it in\ngit_footer_html() subroutine.  Until this patch there were no direct\nor indirect (via die_error) invocation of git_footer_html() with\n$action undefined; each call was after dispatch, which sets $action to\ndefault value if it is undefined.\n\nThis would cause Perl error (\"Use of uninitialized value in string eq\")\nif load is too high _and_ gitweb was invoked without action parameter\nexplicitly set (e.g. for projects list).\n \n> Signed-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n> Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n> ---\n>  gitweb/README      |    7 ++++++-\n>  gitweb/gitweb.perl |   45 +++++++++++++++++++++++++++++++++++++++++----\n>  2 files changed, 47 insertions(+), 5 deletions(-)\n> \n> diff --git a/gitweb/README b/gitweb/README\n> index e34ee79..6c2c8e1 100644\n> --- a/gitweb/README\n> +++ b/gitweb/README\n> @@ -174,7 +174,7 @@ not include variables usually directly set during build):\n>     Base URL for relative URLs in pages generated by gitweb,\n>     (e.g. $logo, $favicon, @stylesheets if they are relative URLs),\n>     needed and used only for URLs with nonempty PATH_INFO via\n> -   <base href=\"$base_url>.  Usually gitweb sets its value correctly,\n> +   <base href=\"$base_url\">.  Usually gitweb sets its value correctly,\n>     and there is no need to set this variable, e.g. to $my_uri or \"/\".\n>   * $home_link\n>     Target of the home link on top of all pages (the first part of view\n> @@ -228,6 +228,11 @@ not include variables usually directly set during build):\n>     repositories from launching cross-site scripting (XSS) attacks.  Set this\n>     to true if you don't trust the content of your repositories. The default\n>     is false.\n> + * $maxload\n> +   Used to set the maximum load that we will still respond to gitweb queries.\n> +   If server load exceed this value then return \"503 Service Unavaliable\" error.\n> +   Server load is taken to be 0 if gitweb cannot determine its value.  Set it to\n> +   undefined value to turn it off.  The default is 300.\n>  \n>  \n>  Projects list file format\n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 7e477af..0a07d3a 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -221,6 +221,12 @@ our %avatar_size = (\n>  \t'double'  => 32\n>  );\n>  \n> +# Used to set the maximum load that we will still respond to gitweb queries.\n> +# If server load exceed this value then return \"503 server busy\" error.\n> +# If gitweb cannot determined server load, it is taken to be 0.\n> +# Leave it undefined (or set to 'undef') to turn off load checking.\n> +our $maxload = 300;\n> +\n>  # You define site-wide feature defaults here; override them with\n>  # $GITWEB_CONFIG as necessary.\n>  our %feature = (\n> @@ -551,6 +557,32 @@ if (-e $GITWEB_CONFIG) {\n>  \tdo $GITWEB_CONFIG_SYSTEM if -e $GITWEB_CONFIG_SYSTEM;\n>  }\n>  \n> +# Get loadavg of system, to compare against $maxload.\n> +# Currently it requires '/proc/loadavg' present to get loadavg;\n> +# if it is not present it returns 0, which means no load checking.\n> +sub get_loadavg {\n> +\tif( -e '/proc/loadavg' ){\n> +\t\topen my $fd, '<', '/proc/loadavg'\n> +\t\t\tor return 0;\n> +\t\tmy @load = split(/\\s+/, scalar <$fd>);\n> +\t\tclose $fd;\n> +\n> +\t\t# The first three columns measure CPU and IO utilization of the last one,\n> +\t\t# five, and 10 minute periods.  The fourth column shows the number of\n> +\t\t# currently running processes and the total number of processes in the m/n\n> +\t\t# format.  The last column displays the last process ID used.\n> +\t\treturn $load[0] || 0;\n> +\t}\n> +\t# additional checks for load average should go here for things that don't export\n> +\t# /proc/loadavg\n> +\n> +\treturn 0;\n> +}\n> +\n> +if (defined $maxload && get_loadavg() > $maxload) {\n> +\tdie_error(503, \"The load average on the server is too high\");\n> +}\n> +\n>  # version of the core git binary\n>  our $git_version = qx(\"$GIT\" --version) =~ m/git version (.*)$/ ? $1 : \"unknown\";\n>  $number_of_git_cmds++;\n> @@ -3354,14 +3386,19 @@ sub git_footer_html {\n>  # 500: The server isn't configured properly, or\n>  #      an internal error occurred (e.g. failed assertions caused by bugs), or\n>  #      an unknown error occurred (e.g. the git binary died unexpectedly).\n> +# 503: The server is currently unavailable (because it is overloaded,\n> +#      or down for maintenance).  Generally, this is a temporary state.\n>  sub die_error {\n>  \tmy $status = shift || 500;\n>  \tmy $error = shift || \"Internal server error\";\n>  \n> -\tmy %http_responses = (400 => '400 Bad Request',\n> -\t\t\t      403 => '403 Forbidden',\n> -\t\t\t      404 => '404 Not Found',\n> -\t\t\t      500 => '500 Internal Server Error');\n> +\tmy %http_responses = (\n> +\t\t400 => '400 Bad Request',\n> +\t\t403 => '403 Forbidden',\n> +\t\t404 => '404 Not Found',\n> +\t\t500 => '500 Internal Server Error',\n> +\t\t503 => '503 Service Unavailable',\n> +\t);\n>  \tgit_header_html($http_responses{$status});\n>  \tprint <<EOF;\n>  <div class=\"page_body\">\n@@ -3354,7 +3354,8 @@ sub git_footer_html {\n \t}\n \n \tprint qq!<script type=\"text/javascript\" src=\"$javascript\"></script>\\n!;\n-\tif ($action eq 'blame_incremental') {\n+\tif (defined $action &&\n+\t    $action eq 'blame_incremental') {\n \t\tprint qq!<script type=\"text/javascript\">\\n!.\n \t\t      qq!startBlame(\"!. href(action=>\"blame_data\", -replay=>1) .qq!\",\\n!.\n \t\t      qq!           \"!. href() .qq!\");\\n!.\n\n> -- \n> 1.6.5.2\n> \n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"131798","messageId":"m3y6jzc7re.fsf@localhost.localdomain","threadId":"22216","inReplyTo":"1263432185-21334-3-git-send-email-warthog9@eaglescrag.net","subject":"Re: [PATCH 2/9] gitweb: change die_error to take \"extra\" argument for extended die information","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-15T22:40:32Z","receivedAt":"2010-01-15T22:40:32Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n\nDon't you want kernel.org address also for authorship?  Also commit\nsummary can be written in shorter way, see proposal below.\n\nFrom: \"John 'Warthog9' Hawley\" <warthog9@kernel.org>\nSubject: gitweb: Allow for longer error explanation in die_error()\n\n> This is a small change that just adds a 3rd, optional, parameter to die_error\n> that allows for extended error information to be output along with what the\n> error was.\n\nSinged-off-by: \"John 'Warthog9' Hawley\" <warthog9@kernel.org>\n> ---\n>  gitweb/gitweb.perl |    8 +++++++-\n>  1 files changed, 7 insertions(+), 1 deletions(-)\n> \n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 0a07d3a..8298de5 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -3391,6 +3391,7 @@ sub git_footer_html {\n>  sub die_error {\n>  \tmy $status = shift || 500;\n>  \tmy $error = shift || \"Internal server error\";\n> +\tmy $extra = shift;\n>  \n>  \tmy %http_responses = (\n>  \t\t400 => '400 Bad Request',\n> @@ -3405,8 +3406,13 @@ sub die_error {\n>  <br /><br />\n>  $status - $error\n>  <br />\n> -</div>\n>  EOF\n> +\tif (defined $extra) {\n> +\t\tprint \"<hr />\\n\" .\n> +\t\t\t\"$extra\\n\";\n\nFollowing gitweb whitespace convention (tab for indent, space for\nalign), it should be:\n\n +\t\tprint \"<hr />\\n\" .\n +\t\t      \"$extra\\n\";\n\n(6 x space in place of last tab).\n\n> +\t}\n> +\tprint \"</div>\\n\";\n> +\n>  \tgit_footer_html();\n>  \texit;\n>  }\n> -- \n> 1.6.5.2\n> \n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"131812","messageId":"m3tyunc55t.fsf@localhost.localdomain","threadId":"22216","inReplyTo":"1263432185-21334-4-git-send-email-warthog9@eaglescrag.net","subject":"Re: [PATCH 3/9] gitweb: Add option to force version match","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-15T23:36:45Z","receivedAt":"2010-01-15T23:36:45Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n\n> This adds $git_versions_must_match variable, which is set to true,\n> checks that we are running on the same version of git that we\n> shipped with, and if not throw '500 Internal Server Error' error.\n> What is checked is the version of gitweb (embedded in building\n> gitweb.cgi), against version of runtime git binary used.\n> \n> Gitweb can usually run with a mismatched git install.  This is more\n> here to give an obvious warning as to whats going on vs. silently\n> failing.\n> \n> By default this feature is turned on.\n\nIf this feature is turned on, then I would prefer for gitweb tests to\nhave it explicitly turned off, so I don't need to rebuild git to test\ngitweb.  \n\nAlso it would be nice to have some tests for this new feature.\n\nSee patch below (proposed to be squashed with this one).\n\n\nP.S. t is there where I have noticed the issue with undefined $action\nin git_footer_html(), mentioned in reply to patch 1/9.\n \n> Signed-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n> Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n> ---\n\nHere could be information about differences from previous versions of\nthis patch.\n\n>  gitweb/README      |    3 +++\n>  gitweb/gitweb.perl |   23 +++++++++++++++++++++++\n>  2 files changed, 26 insertions(+), 0 deletions(-)\n> \n> diff --git a/gitweb/README b/gitweb/README\n> index 6c2c8e1..03151d2 100644\n> --- a/gitweb/README\n> +++ b/gitweb/README\n> @@ -233,6 +233,9 @@ not include variables usually directly set during build):\n>     If server load exceed this value then return \"503 Service Unavaliable\" error.\n>     Server load is taken to be 0 if gitweb cannot determine its value.  Set it to\n>     undefined value to turn it off.  The default is 300.\n> + * $git_versions_must_match\n> +   If set, gitweb fails with 500 Internal Server Error if the version of gitweb\n> +   doesn't match version of git binary.  The default is true.\n>  \n>  \n>  Projects list file format\n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 8298de5..b41bc33 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -221,6 +221,9 @@ our %avatar_size = (\n>  \t'double'  => 32\n>  );\n>  \n> +# If it is true, exit if gitweb version and git binary version don't match\n> +our $git_versions_must_match = 1;\n> +\n>  # Used to set the maximum load that we will still respond to gitweb queries.\n>  # If server load exceed this value then return \"503 server busy\" error.\n>  # If gitweb cannot determined server load, it is taken to be 0.\n> @@ -587,6 +590,26 @@ if (defined $maxload && get_loadavg() > $maxload) {\n>  our $git_version = qx(\"$GIT\" --version) =~ m/git version (.*)$/ ? $1 : \"unknown\";\n>  $number_of_git_cmds++;\n>  \n> +# Throw an error if git versions does not match, if $git_versions_must_match is true.\n> +if ($git_versions_must_match &&\n> +    $git_version ne $version) {\n> +\tmy $admin_contact =\n> +\t\tdefined $ENV{'SERVER_ADMIN'} ? \", $ENV{'SERVER_ADMIN'},\" : '';\n> +\tmy $err_msg = <<EOT;\n> +<h1 align=\"center\">*** Warning ***</h1>\n> +<p>\n> +This version of gitweb was compiled for <b>@{[esc_html($version)]}</b>,\n> +however git version <b>@{[esc_html($git_version)]}</b> was found on server,\n> +and administrator requested strict version checking.\n> +</p>\n> +<p>\n> +Please contact the server administrator${admin_contact} to either configure\n> +gitweb to allow mismatched versions, or update git or gitweb installation.\n> +</p>\n> +EOT\n> +\tdie_error(500, 'Internal server error', $err_msg);\n> +}\n> +\n>  $projects_list ||= $projectroot;\n>  \n>  # ======================================================================\ndiff --git a/t/gitweb-lib.sh b/t/gitweb-lib.sh\nindex 76d8b7b..d9ffc90\n--- a/t/gitweb-lib.sh\n+++ b/t/gitweb-lib.sh\n@@ -25,6 +25,7 @@ our \\$favicon = 'file:///$TEST_DIRECTORY/../gitweb/git-favicon.png';\n our \\$projects_list = '';\n our \\$export_ok = '';\n our \\$strict_export = '';\n+our \\$git_versions_must_match = 0;\n \n EOF\n \ndiff --git a/t/t9501-gitweb-standalone-http-status.sh b/t/t9501-gitweb-standalone-http-status.sh\nindex 0688a57..721900e 100755\n--- a/t/t9501-gitweb-standalone-http-status.sh\n+++ b/t/t9501-gitweb-standalone-http-status.sh\n@@ -113,5 +113,31 @@ test_expect_success 'snapshots: bad object id' '\n '\n test_debug 'cat gitweb.output'\n \n+# ======================================================================\n+# check $git_versions_must_match feature\n+# should be last section, just in case\n+cp -f gitweb_config.perl gitweb_config.perl.bak\n+echo 'our $git_versions_must_match = 1;' >>gitweb_config.perl\n+\n+cat <<\\EOF >>gitweb_config.perl\n+our $version = \"current\";\n+EOF\n+test_expect_success 'force version match: no match' '\n+\tgitweb_run \"p=.git\" &&\n+\tgrep \"500 - Internal Server Error\" gitweb.output\n+'\n+test_debug 'cat gitweb.headers'\n+\n+cat <<\\EOF >>gitweb_config.perl\n+# must be kept in sync with code in gitweb/gitweb.perl\n+our $version = qx(\"$GIT\" --version) =~ m/git version (.*)$/ ? $1 : \"unknown\";\n+EOF\n+test_expect_success 'force version match: match' '\n+\tgitweb_run \"p=.git\" &&\n+\tgrep \"Status: 200 OK\" gitweb.output\n+'\n+test_debug 'cat gitweb.headers'\n+\n+mv -f gitweb_config.perl.bak gitweb_config.perl\n \n test_done\n\n> -- \n> 1.6.5.2\n> \n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"131814","messageId":"m3pr5bc4jv.fsf@localhost.localdomain","threadId":"22216","inReplyTo":"1263432185-21334-8-git-send-email-warthog9@eaglescrag.net","subject":"Re: [PATCH 7/9] gitweb: cleanup error message produced by undefined $site_header","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-15T23:49:53Z","receivedAt":"2010-01-15T23:49:53Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n\nI have modified commit message a bit\n\n> If $site_header is not defined you get extraneous errors in the web\n> logs:\n> \n> [Wed Jan 13 16:55:42 2010] [error] [client ::1] [Wed Jan 13 16:55:42 2010]\n>  gitweb.cgi: Use of uninitialized value $site_header in -f at .../gitweb.cgi line 3287.,\n>  referer: http://git/gitweb.cgi\n> \n> for example (line wrapped for better readibility).  This commit\n> ensures that the variable is defined before trying to use it.\n\n  Ordinarily build procedure ensures that $site_header is defined (but\n  empty, therefore false-ish), so this issue might happen only because\n  of errors in gitweb config file.  Nevetheless it is better to code\n  defensively.\n\n  Signed-off-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n> ---\n>  gitweb/gitweb.perl |    2 +-\n>  1 files changed, 1 insertions(+), 1 deletions(-)\n> \n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 07fdeb5..c4a177d 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -3277,7 +3277,7 @@ EOF\n>  \tprint \"</head>\\n\" .\n>  \t      \"<body>\\n\";\n>  \n> -\tif (-f $site_header) {\n> +\tif ($site_header && -f $site_header) {\n\nIt should be, I think\n\n+\tif (defined $site_header && -f $site_header) {\n\nalthough I guess that nobody would name $site_header file \"0\".\n\n>  \t\tinsert_file($site_header);\n>  \t}\n>  \n> -- \n\nMore important is the fact that it is not the only variable holding\nfile name, that is not checked that it is defined before use.  You\nshould do the same change also for $site_footer and $home_text, and\ndie_error(500, \"Gitweb misconfigured\") or just die_error(500) if\n$projects_list is not defined.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"131825","messageId":"m3ljfydgmt.fsf@localhost.localdomain","threadId":"22216","inReplyTo":"1263432185-21334-9-git-send-email-warthog9@eaglescrag.net","subject":"Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-16T00:43:32Z","receivedAt":"2010-01-16T00:43:32Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n\n> This converts the output handling of gitweb to using an indirect\n> file handle.  This is in preparation to add the caching layer.  This\n> is a slight modification to the way I was originally doing it by\n> passing the output around.  This should be a nop and this shouldn't\n> change the behavior of gitweb.  This does leave error reporting\n> functions (die_error specifically) continuing to output directly\n> as I want to garauntee those will report their errors regardless of\n> what may be going on with respect to the rest of the output.\n\nSignoff?\n\nCompare with my version of this patch:\n  http://repo.or.cz/w/git/jnareb-git.git/commitdiff/0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557\nin the gitweb/cache-kernel branch:\n  http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel\n\n\nMy commit message looks like the following:\n\n....\ngitweb: Print to explicit filehandle (preparing for caching)\n\nThis means replacing\n\n  print <something>;\nby\n  print {$out} <something>;\n\nand\n\n  binmode STDOUT, <layer>;\nby\n  binmode $out, <layer>;\n\nwhere $out is global variable set to \\*STDOUT at the beginning of\ngitweb, but after reading gitweb config.  This way it would be simple\nto e.g. tie output filehandle or use PerlIO layers to simultaneously\nwrite to standard output and to some specified file (like \"tee\"\nutility does), or redirect output to a scalar, or a file.\n\ndie_error (re)sets $out to \\*STDOUT; we would (probably) want to treat\nerrors in a special way, and do not cache them.\n\nThe only other differences are reindent of continued lines (if needed),\nand sometimes word-wrapping lines which this change made too long.\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n....\n\n> ---\n>  gitweb/gitweb.perl |  880 ++++++++++++++++++++++++++--------------------------\n>  1 files changed, 448 insertions(+), 432 deletions(-)\n> \n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index c4a177d..8bb323c 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -450,6 +450,13 @@ our %feature = (\n>  \t\t'default' => [0]},\n>  );\n>  \n> +# Basic file handler for all of gitweb, there are two of them.  The first\n> +# is the basic text/html file handler which is used for everything other\n> +# then the binary files, that uses a separate file handler though\n> +# these are both set to STDOUT for the time being.\n> +our $output_handler = *STDOUT;\n> +our $output_handler_bin = *STDOUT;\n> +\n\nFirst it is not file handleR, but filehandle.\n\nSecond, there is no need for separate filehandle for binary files, if\nyou do it correctly (i.e. call binmode on filehandle, and not on\nSTDOUT).  When caching is enabled, and 'print {$output_handle} <sth>'\nprints to in-memory file (or even directly to cache file) it would do\nconversion, so when reading from cache file we can dump it raw, in\nbinary mode.\n\nThird, wouldn't it be better to use shorter variable name, e.g. $out\nor $oh, instead of $output_handle?  We would be able to align print(f)\nstatements without making lines much longer.\n\nFourth, there is slight difference between\n  our $out = *STDOUT;\nand\n  out $out = \\*STDOUT;\nIn the former we have global variable, in latter we have indirect\nfilehandle.  CGI::Cache uses the latter form, IIRC.\n\n> @@ -3313,7 +3320,7 @@ EOF\n>  \t\tif ($use_pathinfo) {\n>  \t\t\t$action .= \"/\".esc_url($project);\n>  \t\t}\n> -\t\tprint $cgi->startform(-method => \"get\", -action => $action) .\n> +\t\tprint {$output_handler} $cgi->startform(-method => \"get\", -action => $action) .\n>  \t\t      \"<div class=\\\"search\\\">\\n\" .\n>  \t\t      (!$use_pathinfo &&\n>  \t\t      $cgi->input({-name=>\"p\", -value=>$project, -type=>\"hidden\"}) . \"\\n\") .\n\nHere for example after change gitweb source stops being nicely aligned.\nOTOH it makes for bigger patch.  In my version I did realign.\n\nYou can always check for true differences with \"diff -w\".\n\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"131830","messageId":"7vmy0eoogx.fsf@alter.siamese.dyndns.org","threadId":"22216","inReplyTo":"m3ljfydgmt.fsf@localhost.localdomain","subject":"Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-16T00:58:06Z","receivedAt":"2010-01-16T00:58:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> This means replacing\n>\n>   print <something>;\n> by\n>   print {$out} <something>;\n\nJust out of curiosity, how is this different from\n\n    print $out <something>;\n"},{"id":"131838","messageId":"201001160214.58167.jnareb@gmail.com","threadId":"22216","inReplyTo":"7vmy0eoogx.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-16T01:14:55Z","receivedAt":"2010-01-16T01:14:55Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sat, 16 Jan 2010, Junio C Hamano wrote:\n> Jakub Narebski <jnareb@gmail.com> writes:\n> \n> > This means replacing\n> >\n> >   print <something>;\n> > by\n> >   print {$out} <something>;\n> \n> Just out of curiosity, how is this different from\n> \n>     print $out <something>;\n\nActually there is no difference.  It doesn't matter one way or other in\nsituations in gitweb.\n\nI have thought however (but I might be mistaken) that \"print {$fh} <sth>\"\nis idiomatic Perl.\n\n'perldoc -f print' says:\n    Note that if you're storing FILEHANDLES in an array or other expression,\n    you will have to use a block returning its value instead:\n\n           print { $files[$i] } \"stuff\\n\";\n           print { $OK ? STDOUT : STDERR } \"stuff\\n\";\n\nAlso, there is no \",\" between FILEHANDLE and LIST in \"print FILEHANDLE LIST\"\n-- \nJakub Narebski\nPoland\n"},{"id":"131841","messageId":"7v8wbyn7vy.fsf@alter.siamese.dyndns.org","threadId":"22216","inReplyTo":"201001160214.58167.jnareb@gmail.com","subject":"Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-16T01:41:37Z","receivedAt":"2010-01-16T01:41:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> I have thought however (but I might be mistaken) that \"print {$fh} <sth>\"\n> is idiomatic Perl.\n>\n> 'perldoc -f print' says:\n>     Note that if you're storing FILEHANDLES in an array or other expression,\n>     you will have to use a block returning its value instead:\n\nNote that \"in an array or other expression\".  I've always thought the\nintention of this phrase was \"you _could_ help the parser by doing this,\nif you have expression more complex than a simple scalar variable\nreference\".\n\nIOW, I know that {} _can_ be used there, but I haven't seen people write\n{$a_single_variable}, especially without a space around the \"expression\"\n(technically, a single variable is an expression), when\n\n\tprint $fh <stuff>\n\nsuffices, and I was curious why you chose to use the syntax when it wasn't\nnecessary.  Besides, {$fh} looks so eh... (hesitates to mention a dirty\nword ^W^W^Wthe name of a different language, but bleeps it out)...\n"},{"id":"131843","messageId":"m3hbqmdau2.fsf@localhost.localdomain","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"Re: [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-16T02:48:59Z","receivedAt":"2010-01-16T02:48:59Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n\n> This is a very large patch\n\nThis is true, and that is why I am woeking on splitting this patch\ninto series of smaller patches, each adding single feature present in\nthis megapatch (this code drop)... and cleaning up (and improving) it\nwhile at it.  This hopefully would make it easier to review.\n\n>                             that implements the file based\n> caching layer that is used on such large sites as kernel.org and\n> soon git.fedoraproject.org.  This provides a simple, and straight\n> forward caching mechanism that scales dramatically better than\n> Gitweb by itself.\n\nDo you have any benchmarks comparing gitweb performace with and\nwithout caching enabled?\n\n> \n> The caching layer basically buffers the output that Gitweb would\n> normally return, and saves that output to a cache file on the local\n> disk.  When the file is requested it attempts to gain a shared lock\n> on the cache file and cat it out to the client.  Should an exclusive\n> lock be on a file (it's being updated) the code has a choice to either\n> update in the background and go ahead and show the stale page while\n> update is being performed, or stall the client(s) until the page\n> is generated.\n\nThe above paragraph is not very clear to me.\n\n\nCorrect me if I am wrong, but as I understand it the cache\narchitecture is as following:\n\n* This patch implements output caching, which means that the whole\n  gitweb response, including HTTP headers, is stored in cache.  (This\n  means that in absence of extra mechanism content-type negotiation\n  should be disabled when caching is turned on).\n\n* Caching engine used implements simple file based caching layer,\n  where cached data is stored verbatim in cache file (no serialization\n  / hibernating / marshalling of data - better performance, and\n  possibility of X-Sendfile support).  Cache expiration is global\n  value, i.e. is not stored along cache entry in file.  Cache entries\n  expire based on mtime of file.\n\n* When there exist cache entry for given request, and it is not\n  expired, gitweb output is served directtly from cache file.\n\n* When there exist cache entry for given request, but it is expired,\n  one process acquires exclusive (writer) lock on file; the rest of\n  clients get served stale data.\n\n* When there does not exist cache entry for given request, one process\n  acquires exclusive (writer) lock on cache file; the rest of clients\n  wait for cache to be filled.\n\n> \n> There are two forms of stalling involved here, background building\n> and non-background building, both of which are discussed in the\n> configuration page.\n\nI'd like to have at least design decisions put into commit message,\nand perhaps also have caching mechanism described in separate section\nin gitweb/README.\n\n> \n> There are still a few known \"issues\" with respect to this:\n> - Code needs to be added to be \"browser\" aware so\n>   that clients like wget that are trying to get a\n>   binary blob don't obtain a \"Generating...\" page\n\nThis issue should be clearly addressed: when do we serve\n\"Generating...\" page, and when we do not.  The issue is not only wget\ntrying to download binary blob or patchset, or snapshot, but also\nbinary blob which is image referenced from a blob which is HTML, and\nthere is issue of web feeds (accessed by feed readers).\n\n> - There is an intermittent flushing issue that has yet\n>   to be tracked down\n\nCould you tell us more where does this shows (what are the\nsymptompts)?\n\nBTW if it was split into small separate commits, you could be able to\nfind bug by bisecting history.  Also troubles with finding this bug\nmight mean that code is not very clean.\n\n> \n> Caching is disabled by default with the $cache_enable variable,\n> setting this to 1 will enable file based caching.  It is expected\n> that this will be extended to include additional types of caching\n> (like memcached) in the future and should not be exclusively\n> considered a binary value.\n\nNot a good idea, IMHO.  In my rewrite of this patch there is _boolean_\n$caching_enabled variable which controls if (output) caching is\nenabled or not, and $cache variable holding instance of cache engine,\nwhich might be used to select different caching that simple file-based\ncaching.\n\nSignoff?\n\n> ---\n>  gitweb/cache.pm    |  283 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n>  gitweb/gitweb.css  |    6 +\n>  gitweb/gitweb.perl |   58 ++++++++++-\n>  3 files changed, 344 insertions(+), 3 deletions(-)\n>  create mode 100644 gitweb/cache.pm\n\nVery large patch... but no updates to gitweb/README, no updates to\nt/gitweb-lib.sh (I guess that gitweb tests are no longer working).\n \n> diff --git a/gitweb/cache.pm b/gitweb/cache.pm\n> new file mode 100644\n> index 0000000..d08bcec\n> --- /dev/null\n> +++ b/gitweb/cache.pm\n> @@ -0,0 +1,283 @@\n> +# gitweb - simple web interface to track changes in git repositories\n> +#\n> +# (C) 2006, John 'Warthog9' Hawley <warthog19@eaglescrag.net>\n> +#\n> +# This program is licensed under the GPLv2\n> +\n> +#\n> +# Gitweb caching engine\n> +#\n> +\n> +use File::Path qw(make_path remove_tree);\n\nUsing make_path (you do not use remove_tree, so there is no need for\nimporting it) instead of older mkdir interface requires File::Path\nversion 2.0 (which meant that I had to upgrade File::Path).  This at\nleast should be mentioned in the comment, perhaps also in\ngitweb/INSTALL.\n\n> +use Digest::MD5 qw(md5 md5_hex md5_base64);\n\nYou use only md5_hex; no need to import other functions.\n\n> +use Fcntl ':flock';\n> +\n> +sub cache_fetch {\n> +\tmy ($action) = @_;\n> +\tmy $cacheTime = 0;\n> +\n> +\t# Deal with cache being disabled\n> +\tif( $cache_enable == 0 ){\n\nStyle:\n\n  +\tif ($cache_enable == 0) {\n\nor better\n\n  +\tif ($cache_enabled) {\n\n> +\t\t$output_handler = *STDOUT;\n> +\t\t$output_handler_bin = *STDOUT;\n\nThere should be no need for that, as $output_handle is set to *STDOUT\n(or \\*STDOUT) anyway.\n\n> +\t\t$actions{$action}->();\n> +\t\treturn;\n\nAnyway I think that the whole block should be _outside_ cache_fetch,\nwhich should be invoked only if caching is enabled.  For example in\ngitweb.perl:\n\n  if ($caching_enabled) {\n  \tdo $cache_pm;\n  \tdie $@ if $@;\n  \n  \t# ...\n  \n\tcache_fetch($cache, $action);\n  } else {\n  \t$actions{$action}->();\n  }\n\n> +\t}elsif( $cache_enable == 1 ){\n\nStyle.\n\n> +\t\t#obviously we are using file based caching\n\nSee my comment about using $cache_enable as enum selecting cache type\n(blergh).  BTW what's with 'obviously'?\n\n> +\n> +\t\tif(! -d $cachedir){\n\nStyle.\n\n> +\t\t\tprint \"*** Warning ***: Caching enabled but cache directory does not exsist.  ($cachedir)\\n\";\n\nWhy this warning?  Is it really necessary?\n\n> +\t\t\tmkdir (\"cache\", 0665) || die \"Cannot create cache dir - you will need to manually create\";\n> +\t\t\tprint \"Cache directory created successfully\\n\";\n> +\t\t}\n> +\n> +\t\tour $full_url = \"$my_url?\". $ENV{'QUERY_STRING'};\n\nWouldn't work if you client uses path_info URL, e.g.\n\n  http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel\n\nThat's why I use href(-replay=>1, -full_url=>1, -path_info=>0) for\ncache key for request (you could use freeze(\\%input_params) instead,\nwhere freeze is from Storable module).\n\n> +\t\tour $urlhash = md5_hex($full_url);\n> +\t\tour $fullhashdir = \"$cachedir/\". substr( $urlhash, 0, 2) .\"/\";\n\nIs depth 2 enough for cache?\n\n> +\n> +\t\tmy $numdirs = make_path( $fullhashdir, { mode => 0777, error => \\my $mkdirerr, } );\n> +\t\tif( @$mkdirerr ){\n> +\t\t\tmy $mkdirerrmsg = \"\";\n> +\t\t\tfor my $diag (@$mkdirerr) {\n> +\t\t\t\tmy ($file, $message) = %$diag;\n> +\t\t\t\tif($file eq '' ){\n> +\t\t\t\t\t$mkdirerrmsg .= \"general error: $message\\n\";\n> +\t\t\t\t}else{\n> +\t\t\t\t\t$mkdirerrmsg .= \"problem unlinking $file: $message\\n\";\n> +\t\t\t\t}\n> +\t\t\t}\n> +\t\t\tdie_error(500, \"Could not create cache directory | $mkdirerrmsg\");\n> +\t\t}\n> +\t\t$fullhashpath = \"$fullhashdir/\". substr( $urlhash, 2 );\n> +\t\t$fullhashbinpath = \"$fullhashpath.bin\";\n> +\t} # done dealing with cache enabled / disabled\n\nNote also if dealing with caching enabled / disabled was outside\ncache_fetch you would have less nested code.\n\n> +\n> +\tif(! -e \"$fullhashpath\" ){\n> +\t\tif(! defined(my $childPid = fork()) ){\n\nStyle.\n\n> +\t\t\tcacheUpdate($action,0);\n> +\t\t\tcacheDisplay($action);\n\nWhy camelCase Java/JavaScript-like convention, quite different from\nthe C-like naming convention used elsewhere in gitweb?\n\n> +\t\t} elsif ( $childPid == 0 ){\n> +\t\t\t#run the updater\n> +\t\t\tcacheUpdate($action,1);\n\ncacheUpdate($action,0) vs cacheUpdate($action,1) is very cryptic\ndistinctions.  It would be better to use \"named parameter\" and/or\nseparate, differently named, [wrapper] functions.\n\n> +\t\t}else{\n> +\t\t\tcacheWaitForUpdate($action);\n> +\t\t}\n\nThis whole block should probably be in a separate function.\n\n> +\t}else{\n> +\t\t#if cache is out dated, update\n> +\t\t#else displayCache();\n> +\t\topen(cacheFile, '<', \"$fullhashpath\");\n> +\t\tstat(cacheFile);\n> +\t\tclose(cacheFile);\n\nYou don't need to open file to stat it.\n\n> +\t\t$cacheTime = get_loadavg() * 60;\n> +\t\tif( $cacheTime > $maxCacheTime ){\n> +\t\t\t$cacheTime = $maxCacheTime;\n> +\t\t}\n> +\t\tif( $cacheTime < $minCacheTime ){\n> +\t\t\t$cacheTime = $minCacheTime;\n> +\t\t}\n\nThis should probably be a separate function (effective cache expiraton\ntime).  Also adaptiveness of caching is not described in commit\nmessage.\n\n> +\t\tif( (stat(_))[9] < (time - $cacheTime) ){\n> +\t\t\tif( ! defined(my $childPid = fork()) ){\n> +\t\t\t\tcacheUpdate($action,0);\n> +\t\t\t\tcacheDisplay($action);\n> +\t\t\t} elsif ( $childPid == 0 ){\n> +\t\t\t\t#run the updater\n> +\t\t\t\t#print \"Running updater\\n\";\n\nRemains of debugging.\n\n> +\t\t\t\tcacheUpdate($action,1);\n> +\t\t\t}else{\n> +\t\t\t\t#print \"Waiting for update\\n\";\n> +\t\t\t\tcacheWaitForUpdate($action);\n> +\t\t\t}\n\nRepeated code (I think).\n\n> +\t\t} else {\n> +\t\t\tcacheDisplay($action);\n> +\t\t}\n> +\n> +\n> +\t}\n> +\n> +\t#\n> +\t# If all of the caching failes - lets go ahead and press on without it and fall back to 'default'\n> +\t# non-caching behavior.  This is the softest of the failure conditions.\n> +\t#\n> +\t#$actions{$action}->();\n\nWhy is this commented out?\n\n> +}\n> +\n> +sub cacheUpdate {\n> +\tmy ($action,$areForked) = @_;\n> +\tmy $lockingStatus;\n> +\tmy $fileData = \"\";\n> +\n> +\tif($backgroundCache){\n> +\t\topen(cacheFileBG, '>:utf8', \"$fullhashpath.bg\");\n> +\t\tmy $lockStatBG = flock(cacheFileBG,LOCK_EX|LOCK_NB);\n> +\n> +\t\t$lockStatus = $lockStatBG;\n> +\t}else{\n> +\t\topen(cacheFile, '>:utf8', \"$fullhashpath\");\n> +\t\tmy $lockStat = flock(cacheFile,LOCK_EX|LOCK_NB);\n> +\n> +\t\t$lockStatus = $lockStat;\n> +\t}\n\nAlmost identical code.  Use of global handles instead of indirect\nfilehandles.\n\n> +\t#print \"lock status: $lockStat\\n\";\n> +\n> +\n> +\tif (! $lockStatus ){\n> +\t\tif ( $areForked ){\n> +\t\t\texit(0);\n> +\t\t}else{\n> +\t\t\treturn;\n> +\t\t}\n> +\t}\n\nThis conditional needs explanation (comment), I think.\n\n> +\n> +\tif(\n> +\t\t$action eq \"snapshot\"\n> +\t\t||\n> +\t\t$action eq \"blob_plain\"\n\nThis condition should be put in a separate function/\n\n> +\t){\n> +\t\topen cacheFileBin, '>', $fullhashbinpath or die_error(500, \"Could not open bin dump file\");\n> +\t\t$output_handler_bin = *cacheFileBin;\n> +\t}\n> +\n> +\t$output_handler = *cacheFile;\n> +\n> +\tif($backgroundCache){\n> +\t\topen(cacheFile, '>:utf8', \"$fullhashpath\");\n\nWhy \"$fullhashpath\" and not simply $fullhashpath?\n\n> +\t\t$lockStat = flock(cacheFile,LOCK_EX);\n> +\n> +\t\tif (! $lockStat ){\n> +\t\t\tif ( $areForked ){\n> +\t\t\t\texit(0);\n> +\t\t\t}else{\n> +\t\t\t\treturn;\n> +\t\t\t}\n> +\t\t}\n\nRepeated code.\n\n> +\t}\n> +\n> +\t$actions{$action}->();\n> +\n> +\tif(\n> +\t\t$action eq \"snapshot\"\n> +\t\t||\n> +\t\t$action eq \"blob_plain\"\n> +\t){\n> +\t\tclose(cacheFileBin);\n> +\t}\n> +\n> +\tflock(cacheFile,LOCK_UN);\n> +\tclose(cacheFile);\n> +\n> +\tif($backgroundCache){\n> +\t\tflock(cacheFileBG,LOCK_UN);\n> +\t\tclose(cacheFileBG);\n> +\t}\n> +\n> +\tif ( $areForked ){\n> +\t\texit(0);\n> +\t} else {\n> +\t\treturn;\n> +\t}\n> +}\n> +\n> +\n> +sub cacheWaitForUpdate {\n> +\tmy ($action) = @_;\n> +\tmy $x = 0;\n> +\tmy $max = 10;\n\nWhat is $x, what is $max?\n\n> +\tmy $lockStat = 0;\n> +\n> +\tif( $backgroundCache ){\n> +\t\tif( -e \"$fullhashpath\" ){\n> +\t\t\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n\nWhy opening with :uft8, and not with :raw?  I don't think we need to\ndo the eventual conversion once again...\n\n> +\t\t\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n> +\t\t\tstat(cacheFile);\n> +\t\t\tclose(cacheFile);\n> +\n> +\t\t\tif( $lockStat && ( (stat(_))[9] > (time - $maxCacheLife) ) ){\n> +\t\t\t\tcacheDisplay($action);\n> +\t\t\t\treturn;\n> +\t\t\t}\n\nWhy do we deal with cache expiration in two places?  If it is not\na bug, it should be explained in a comment.\n\n> +\t\t}\n> +\t}\n> +\n> +\tif(\n> +\t\t$action eq \"atom\"\n> +\t\t||\n> +\t\t$action eq \"rss\"\n> +\t\t||\n> +\t\t$action eq \"opml\"\n> +\t){\n> +\t\tdo {\n> +\t\t\tsleep 2 if $x > 0;\n> +\t\t\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n> +\t\t\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n> +\t\t\tclose(cacheFile);\n> +\t\t\t$x++;\n> +\t\t\t$combinedLockStat = $lockStat;\n> +\t\t} while ((! $combinedLockStat) && ($x < $max));\n\nWhy busy wait instead of _blocking_ lock, i.e. waiting on lock for it\nto be free?  It doesn't look like we _do_ anything in the loop.\n\nAh, I see that we wait at most 2*$max seconds (where interval of 2\nseconds is hardcoded).  Is it really necessary?\n\n> +\n> +\t\tif( $x != $max ){\n> +\t\t\tcacheDisplay($action);\n> +\t\t}\n> +\t\treturn;\n> +\t}\n> +\n> +\t$| = 1;\n> +\n> +\tprint $::cgi->header(-type=>'text/html', -charset => 'utf-8',\n> +\t                   -status=> 200, -expires => 'never');\n> +\n> +\tprint <<EOF;\n> +<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www/w3.porg/TR/html4/strict.dtd\">\n> +<!-- git web w/caching interface version $version, (C) 2006-2010, John 'Warthog9' Hawley <warthog9\\@kernel.org> -->\n> +<!-- git core binaries version $git_version -->\n> +<head>\n> +<meta http-equiv=\"content-type\" content=\"$content_type; charset=utf-8\"/>\n> +<meta name=\"generator\" content=\"gitweb/$version git/$git_version\"/>\n> +<meta name=\"robots\" content=\"index, nofollow\"/>\n> +<meta http-equiv=\"refresh\" content=\"0\"/>\n> +<title>$title</title>\n> +</head>\n> +<body>\n> +EOF\n> +\n> +\tprint \"Generating..\";\n> +\tdo {\n> +\t\tprint \".\";\n> +\t\tsleep 2 if $x > 0;\n> +\t\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n> +\t\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n> +\t\tclose(cacheFile);\n> +\t\t$x++;\n> +\t\t$combinedLockStat = $lockStat;\n> +\t} while ((! $combinedLockStat) && ($x < $max));\n\nThis trick of having http-equiv 'refresh' meta with the delay of 0\nseconds, but not closing the output and therefore not triggering\nredirect should be described in comments, and perhaps also in the\ncommit message.\n\n> +\tprint <<EOF;\n> +</body>\n> +</html>\n> +EOF\n> +\treturn;\n> +}\n> +\n> +sub cacheDisplay {\n> +\tmy ($action) = @_;\n> +\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n> +\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n> +\tif (! $lockStat ){\n> +\t\tclose(cacheFile);\n> +\t\tcacheWaitForUpdate($action);\n> +\t}\n> +\n> +\twhile( <cacheFile> ){\n> +\t\tprint $_;\n> +\t}\n\nWhy not slurp it (local $/ = undef), but write line after line?\n\n> +\tif(\n> +\t\t$action eq \"snapshot\"\n> +\t\t||\n> +\t\t$action eq \"blob_plain\"\n> +\t){\n> +\t\topen(cacheFileBin, '<', \"$fullhashbinpath\");\n> +\t\tbinmode STDOUT, ':raw';\n> +\t\tprint <cacheFileBin>;\n\nWhy not slurp it (local $/ = undef), but write line after line,\nimplicitly?\n\n> +\t\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n> +\t\tclose(cacheFileBin);\n> +\t}\n> +\tclose(cacheFile);\n> +}\n\n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 8bb323c..ec95bb9 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -230,6 +230,50 @@ our $git_versions_must_match = 1;\n>  # Leave it undefined (or set to 'undef') to turn off load checking.\n>  our $maxload = 300;\n>  \n> +# This enables/disables the caching layer in gitweb.  This currently only supports the\n> +# 'dumb' file based caching layer, primarily used on git.kernel.org.  this is reasonably\n> +# effective but it has the downside of requiring a huge amount of disk space if there\n> +# are a number of repositories involved.  It is not uncommon for git.kernel.org to have\n> +# on the order of 80G - 120G accumulate over the course of a few months.  It is recommended\n> +# that the cache directory be periodically completely deleted, and this is safe to perform.\n> +# Suggested mechanism\n> +# mv $cacheidr $cachedir.flush;mkdir $cachedir;rm -rf $cachedir.flush\n> +# Value is binary. 0 = disabled (default), 1 = enabled.\n> +#\n> +# Values of caching:\n> +# \t1 = 'dumb' file based caching used on git.kernel.org\n> +our $cache_enable = 0;\n> +\n> +# Used to set the minimum cache timeout for the dynamic caching algorithm.  Basically\n> +# if we calculate the cache to be under this number of seconds we set the cache timeout\n> +# to this minimum.\n> +# Value is in seconds.  1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour\n> +our $minCacheTime = 20;\n> +\n> +# Used to set the maximum cache timeout for the dynamic caching algorithm.  Basically\n> +# if we calculate the cache to exceed this number of seconds we set the cache timeout\n> +# to this maximum.\n> +# Value is in seconds.  1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour\n> +our $maxCacheTime = 1200;\n> +\n> +# If you need to change the location of the caching directory, override this\n> +# otherwise this will probably do fine for you\n> +our $cachedir = 'cache';\n\nWhy not '/tmp/gitweb-cache', or '/var/cache/gitweb'?  Perhaps use\nTMPDIR / File::Spec->tmpdir() if it is undefined?\n\nNote that this path is relative to the place where we run gitweb from,\nwhich is important for gitweb tests.\n\n> +\n> +# If this is set (to 1) cache will do it's best to always display something instead\n> +# of making someone wait for the cache to update.  This will launch the cacheUpdate\n> +# into the background and it will lock a <file>.bg file and will only lock the\n> +# actual cache file when it needs to write into it.  In theory this will make\n> +# gitweb seem more responsive at the price of possibly stale data.\n> +our $backgroundCache = 1;\n\nDoes it mean that if there exist cache entry for given request, but it\nis expired, also the client that created write lock gets stale data\ninstead of 'Generating...' info, and updates/regenerates cache using\nbackground process?\n\nThis comment is not entirely clear for me.\n\n> +\n> +# Used to set the maximum cache file life.  If a cache files last modify time exceeds\n> +# this value, it will assume that the data is just too old, and HAS to be regenerated\n> +# instead of trying to display the existing cache data.\n> +# Value is in seconds.  1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour\n> +# 18000 = 5 hours\n> +our $maxCacheLife = 18000;\n\nThis should also be mentioned in commit message (modifying what I\nwrote).\n\n> +\n>  # You define site-wide feature defaults here; override them with\n>  # $GITWEB_CONFIG as necessary.\n>  our %feature = (\n> @@ -593,6 +637,11 @@ if (defined $maxload && get_loadavg() > $maxload) {\n>  \tdie_error(503, \"The load average on the server is too high\");\n>  }\n>  \n> +#\n> +# Includes\n> +#\n> +do 'cache.pm';\n\nShould be\n\n  +do \"$cache_pm\";\n\nif you don't use require, where $cache_pm can be overriden in gitweb\nconfig, otherwise gitweb caching tests wouldn't work: they invoke\ngitweb from test directory.\n\n> +\n>  # version of the core git binary\n>  our $git_version = qx(\"$GIT\" --version) =~ m/git version (.*)$/ ? $1 : \"unknown\";\n>  $number_of_git_cmds++;\n> @@ -994,7 +1043,7 @@ if ($action !~ m/^(?:opml|project_list|project_index)$/ &&\n>      !$project) {\n>  \tdie_error(400, \"Project needed\");\n>  }\n> -$actions{$action}->();\n> +cache_fetch($action);\n>  exit;\n>  \n\nAs I wrote, I think cache_fetch should be invoked only when caching is\nenabled.\n\n>  ## ======================================================================\n> @@ -3200,7 +3249,9 @@ sub git_header_html {\n>  \t# support xhtml+xml but choking when it gets what it asked for.\n>  \tif (defined $cgi->http('HTTP_ACCEPT') &&\n>  \t    $cgi->http('HTTP_ACCEPT') =~ m/(,|;|\\s|^)application\\/xhtml\\+xml(,|;|\\s|$)/ &&\n> -\t    $cgi->Accept('application/xhtml+xml') != 0) {\n> +\t    $cgi->Accept('application/xhtml+xml') != 0\n> +\t    &&\n> +\t    $cache_enable == 0) {\n>  \t\t$content_type = 'application/xhtml+xml';\n>  \t} else {\n>  \t\t$content_type = 'text/html';\n\nO.K.\n\n> @@ -3344,6 +3395,7 @@ sub git_footer_html {\n>  \tmy $feed_class = 'rss_logo';\n>  \n>  \tprint {$output_handler} \"<div class=\\\"page_footer\\\">\\n\";\n> +\tprint {$output_handler} \"<div class=\\\"cachetime\\\">Cache Last Updated: \". gmtime( time ) .\" GMT</div>\\n\";\n\nShouldn't this be conditional on $cache_enabled?\n\n>  \tif (defined $project) {\n>  \t\tmy $descr = git_get_project_description($project);\n>  \t\tif (defined $descr) {\n\nBTW. you need, I think, protect timing info and do not show it if\ncaching is enabled.  It doesn't make much sense to show how much time\nit took to generate page... when said page could have been retrieved\nfrom cache.\n\nBut it might make sense; I am not sure.\n\n> @@ -3424,7 +3476,7 @@ sub die_error {\n>  \tmy $extra = shift;\n>  \n>  \t# The output handlers for die_error need to be reset to STDOUT\n> -\t# so that half the message isn't being output to random and \n> +\t# so that half the message isn't being output to random and\n>  \t# half to STDOUT as expected.  This is mainly for the benefit\n>  \t# of using git_header_html() and git_footer_html() since those\n>  \t# internaly use the indirect print handler.\n\nIt looks like spurious change.\n\n> -- \n> 1.6.5.2\n> \n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"132446","messageId":"cover.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 00/10] gitweb: Simple file based output caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:23Z","receivedAt":"2010-01-23T00:27:23Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This 10 patches long patch series is intended as proof of concept\nfor splitting large 'gitweb: File based caching layer (from git.kernel.org)'\nmega-patch by John 'Warthog9' Hawley aka J.H., by starting small and\nadding features piece by piece.\n\nThis patch is meant as replacement for last two patches:\n* [PATCH 8/9] gitweb: Convert output to using indirect file handle\n  Message-ID: <1263432185-21334-9-git-send-email-warthog9@eaglescrag.net>\n* [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)\n  Message-ID: <1263432185-21334-10-git-send-email-warthog9@eaglescrag.net>\n\nin the long patch series by J.H.\n* [PATCH 0/9] Gitweb caching v5\n  http://thread.gmane.org/gmane.comp.version-control.git/136913\n\nNote that this patch series is part of 'gitweb/cache-kernel' branch of\nhttp://repo.or.cz/w/git/jnareb-git.git repository (gitweb link), built\non top of modified patches from 'Gitweb caching v2' series (from\n'gitweb-ml-v2' branch of http://git.kernel.org/?p=git/warthog9/gitweb.git\nrepository).  Therefore they might not apply as straight replacements\non top of early parts of 'gitweb-ml-v5' branch.\n\nThis is work in progress (showing how I see introducing output caching\nto gitweb), it lacks proper documentation (POD for gitweb/cache.pm,\nnew configuration variables in gitweb/README, perhaps \"Gitweb caching\"\nsection in gitweb/README and gitweb/cache.pm mentioned in gitweb/INSTALL),\nand commits/patches marked '(WIP)' lacks proper commit message.\n\nJust food for thought...\n\nTable of contents:\n~~~~~~~~~~~~~~~~~~\n [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing\n                   for caching)\n [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)\n [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching\n [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration\n [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set)\n                   output caching (WIP)\n [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)\n [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)\n [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd'\n                   problem (WIP)\n [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for\n                   filling cache (WIP)\n [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n                   regenerating cache (WIP)\n\n\nDiffstat:\n~~~~~~~~~\n\n gitweb/cache.pm                        |  566 ++++++++++\n gitweb/gitweb.perl                     | 1923 +++++++++++++++++---------------\n t/gitweb-lib.sh                        |    2 +\n t/t9500-gitweb-standalone-no-errors.sh |   13 +\n t/t9503-gitweb-caching.sh              |   32 +\n t/t9503/test_cache_interface.pl        |  195 ++++\n t/test-lib.sh                          |    3 +\n 7 files changed, 1836 insertions(+), 898 deletions(-)\n create mode 100644 gitweb/cache.pm\n create mode 100755 t/t9503-gitweb-caching.sh\n create mode 100755 t/t9503/test_cache_interface.pl\n"},{"id":"132444","messageId":"f71ad6f178dc424571fc3c6c77d18557752bfeeb.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:25Z","receivedAt":"2010-01-23T00:27:25Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"If named boolean option -path_info is passed to href() subroutine, use\nits value, instead of querying 'pathinfo' feature to check whether to\ngenerate path_info URL (if generating path_info link is possible at all).\n\nhref(-replay=>1, -path_info=>0) is meant to be used to generate a key\nfor caching gitweb output; alternate solution would be to use freeze()\nfrom Storable (core module) on %input_params hash (or its reference),\ne.g.:\n  $key = freeze \\%input_params;\nor other serialization technique.\n\nWhile at it document extra options/flags to href().\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nActually after discussion with J.H.\n  http://thread.gmane.org/gmane.comp.version-control.git/136913/focus=137061\nI have changed my mind and now I think that good human-readable key\nfor output caching is href(-replay=>1, -full=>1, -path_info=>0);\n\nThe reason behind -path_info=>0 is that we want to have the same cache\nentry (the same cache file) regardless of whether we use path_info\nURL, or non-path_info URL.\n\nThe reason behind -full_path=>1 (from what I understand from\nJ.H. argument) is that you can have two gitweb deployments sharing the\nsame cache root, but in which the same project name can point to\ndifferent projects (different $projectsroot).\n\n gitweb/gitweb.perl |    7 ++++++-\n 1 files changed, 6 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 2f6a8e1..cd5073c 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1004,6 +1004,10 @@ exit;\n ## ======================================================================\n ## action links\n \n+# possible values of extra options\n+# -full => 0|1      - use absolute/full URL ($my_uri/$my_url as base)\n+# -replay => 1      - start from a current view (replay with modifications)\n+# -path_info => 0|1 - don't use/use path_info URL (if possible)\n sub href {\n \tmy %params = @_;\n \t# default is to use -absolute url() i.e. $my_uri\n@@ -1020,7 +1024,8 @@ sub href {\n \t}\n \n \tmy $use_pathinfo = gitweb_check_feature('pathinfo');\n-\tif ($use_pathinfo and defined $params{'project'}) {\n+\tif (defined $params{'project'} &&\n+\t    (exists $params{-path_info} ? $params{-path_info} : $use_pathinfo)) {\n \t\t# try to put as many parameters as possible in PATH_INFO:\n \t\t#   - project name\n \t\t#   - action\n-- \n1.6.6.1\n"},{"id":"132445","messageId":"b2f4605fcab798843fc6096783873f6808a68af1.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:26Z","receivedAt":"2010-01-23T00:27:26Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This is first step towards implementing file based output (response)\ncaching layer that is used on such large sites as kernel.org.\n\nThis patch introduces GitwebCaching::SimpleFileCache package, which\nfollows Cache::Cache / CHI interface, although do not implement it\nfully.  The intent of following established convention is to be able\nin the future to replace our simple file based cache e.g. by one using\nmemcached.\n\nLike in original patch by John 'Warthog9' Hawley (J.H.) (the one this\ncommit intends to be incremental step to), the data is stored in the\ncase as-is, without adding metadata (like expiration date), and\nwithout serialization (which means only scalar data).\n\nTo be implemented (from original patch by J.H.):\n* cache expiration (based on file stats, current time and global\n  expiration time); currently elements in cache do not expire\n* actually using this cache in gitweb, except error pages\n* adaptive cache expiration, based on average system load\n* optional locking interface, where only one process can update cache\n  (using flock)\n* server-side progress indicator when waiting for filling cache,\n  which in turn requires separating situations (like snapshots and\n  other non-HTML responses) where we should not show 'please wait'\n  message\n\nPossible extensions (beyond what was in original patch):\n* (optionally) show information about cache utilization\n* AJAX (JavaScript-based) progress indicator\n* JavaScript code to update relative dates in cached output\n* make cache size-aware (try to not exceed specified maximum size)\n* utilize X-Sendfile header (or equivalent) to show cached data\n  (optional, as it makes sense only if web server supports sendfile\n  feature and have it enabled)\n* variable expiration feature from CHI, allowing items to expire a bit\n  earlier than the stated expiration time to prevent cache miss\n  stampedes (although locking, if available, should take care of\n  this).\n\nThe code of GitwebCaching::SimpleFileCache package in gitweb/cache.pm\nwas heavily based on file-based cache in Cache::Cache package, i.e.\non Cache::FileCache, Cache::FileBackend and Cache::BaseCache\n(including implementing atomic write, something that original patch\nlacks).\n\nThis patch does not yet enable output caching in gitweb (it doesn't\nhave all required features yet); on the other hand it includes tests,\ncurrently testing only cache Perl API.\n\nInspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nLarge parts of this code are based _heavily_ on Cache::FileCache\nimplementation (including Cache::FileBackend and Cache::BaseCache)\nfrom Cache::Cache distribution (which is dual licensed using\n(Perl) Artistic License and GNU General Public License, like Perl\nitself).  In the final version of code it should probably be\ncleaned up.\n\nAlso although it implements 'compute' interface from CHI (Unified\ncache interface), actual CHI code was not used even as reference.\n(Nevertheless gitweb/cache.pm is meant to use minimal dependencies,\nlike gitweb itself, so it won't use Moose / Mouse for OO).\n\n gitweb/cache.pm                 |  317 +++++++++++++++++++++++++++++++++++++++\n t/t9503-gitweb-caching.sh       |   32 ++++\n t/t9503/test_cache_interface.pl |   77 ++++++++++\n t/test-lib.sh                   |    3 +\n 4 files changed, 429 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/cache.pm\n create mode 100755 t/t9503-gitweb-caching.sh\n create mode 100755 t/t9503/test_cache_interface.pl\n\ndiff --git a/gitweb/cache.pm b/gitweb/cache.pm\nnew file mode 100644\nindex 0000000..ea544b0\n--- /dev/null\n+++ b/gitweb/cache.pm\n@@ -0,0 +1,317 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2006, John 'Warthog9' Hawley <warthog19@eaglescrag.net>\n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Gitweb caching engine\n+#\n+\n+{\n+# Minimalistic cache that stores data in the filesystem, without serialization\n+# and currently without any kind of cache expiration (all keys last forever till\n+# they got explicitely removed)\n+#\n+# It follows Cache::Cache and CHI interface (but does not implement it fully)\n+\n+package GitwebCache::SimpleFileCache;\n+\n+use strict;\n+use warnings;\n+\n+use File::Path qw(make_path);  # requires version >= 2.0\n+use File::Spec;\n+use File::Temp;\n+use Digest::MD5 qw(md5_hex);\n+\n+# by default, the cache nests all entries on the filesystem two\n+# directories deep\n+\n+our $DEFAULT_CACHE_DEPTH = 2;\n+\n+# by default, the root of the cache is located in 'cache'.\n+\n+our $DEFAULT_CACHE_ROOT = \"cache\";\n+\n+# ......................................................................\n+# constructor\n+\n+# The options are set by passing in a reference to a hash containing\n+# any of the following keys:\n+#  * 'namespace'\n+#    The namespace associated with this cache.  This allows easy separation of\n+#    multiple, distinct caches without worrying about key collision.  Defaults\n+#    to '' (which does not allow for simple implementation of clear() method).\n+#  * 'cache_root'\n+#    The location in the filesystem that will hold the root of the cache.\n+#    Defaults to 'cache', relative to gitweb.cgi directory.\n+#  * 'cache_depth'\n+#    The number of subdirectories deep to cache object item.  This should be\n+#    large enough that no cache directory has more than a few hundred objects.\n+#    Defaults to 2 unless explicitly set.\n+sub new {\n+\tmy ($proto, $p_options_hash_ref) = @_;\n+\n+\tmy $class = ref($proto) || $proto;\n+\tmy $self  = {};\n+\t$self = bless($self, $class);\n+\n+\tmy ($root, $depth, $ns);\n+\tif (defined $p_options_hash_ref) {\n+\t\t$root  = $p_options_hash_ref->{'cache_root'};\n+\t\t$depth = $p_options_hash_ref->{'cache_depth'};\n+\t\t$ns    = $p_options_hash_ref->{'namespace'};\n+\t}\n+\t$root  = $DEFAULT_CACHE_ROOT  unless defined($root);\n+\t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n+\t$ns    = '' unless defined($ns);\n+\n+\t$self->set_root($root);\n+\t$self->set_depth($depth);\n+\t$self->set_namespace($ns);\n+\n+\treturn $self;\n+}\n+\n+# ......................................................................\n+# accessors\n+\n+sub get_depth {\n+\tmy ($self) = @_;\n+\n+\treturn $self->{'_Depth'};\n+}\n+\n+sub set_depth {\n+\tmy ($self, $depth) = @_;\n+\n+\t$self->{'_Depth'} = $depth;\n+}\n+\n+sub get_root {\n+\tmy ($self) = @_;\n+\n+\treturn $self->{'_Root'};\n+}\n+\n+\n+sub set_root {\n+\tmy ($self, $root) = @_;\n+\n+\t$self->{'_Root'} = $root;\n+}\n+\n+sub get_namespace {\n+\tmy ($self) = @_;\n+\n+\treturn $self->{'_Namespace'};\n+}\n+\n+\n+sub set_namespace {\n+\tmy ($self, $namespace) = @_;\n+\n+\t$self->{'_Namespace'} = $namespace;\n+}\n+\n+# ----------------------------------------------------------------------\n+# (private) utility functions and methods\n+\n+# Take an human readable key, and create a unique (hashed) key from it\n+sub _Build_Hashed_Key {\n+\tmy ($p_key) = @_;\n+\n+\treturn md5_hex($p_key);\n+}\n+\n+# Take an human readable key, and return file path\n+sub _path_to_key {\n+\tmy ($self, $p_namespace, $p_key) = @_;\n+\n+\treturn $self->_path_to_hashed_key($p_namespace,\n+\t                                    _Build_Hashed_Key($p_key));\n+}\n+\n+# Take hashed key, and return file path\n+sub _path_to_hashed_key {\n+\tmy ($self, $p_namespace, $p_hashed_key) = @_;\n+\n+\treturn File::Spec->catfile($self->get_root(), $p_namespace,\n+\t                           _Split_Word($p_hashed_key, $self->get_depth()));\n+}\n+\n+# Split word into N components, where each component but last is two-letter word\n+# e.g. _Split_Word(\"06b90e786e304a18fdfbd7c7bcc41a6b\", 2) == qw(06 b90e786e304a18fdfbd7c7bcc41a6b);\n+#      _Split_Word(\"06b90e786e304a18fdfbd7c7bcc41a6b\", 3) == qw(06 b9 0e786e304a18fdfbd7c7bcc41a6b);\n+sub _Split_Word {\n+\tmy ($p_word, $p_depth) = @_;\n+\n+\t$p_depth--; # now it is number of leading 2-letter components\n+\treturn unpack(\"(a2)$p_depth a*\", $p_word);\n+}\n+\n+sub _Read_File {\n+\tmy ($p_path) = @_;\n+\n+\t-e $p_path\n+\t\tor return undef;\n+\n+\topen(my $fh, '<', $p_path)\n+\t\tor return undef;\n+\n+\tlocal $/ = undef;\n+\tmy $data = <$fh>;\n+\n+\tclose($fh);\n+\n+\treturn $data;\n+}\n+\n+# write a file atomically, assuming that path leading to file exists\n+sub _Write_File {\n+\tmy ($p_path, $p_data) = @_;\n+\n+\tmy ($volume, $directory, $filename) = File::Spec->splitpath($p_path);\n+\tif (defined $directory and defined $volume) {\n+\t\t$directory = File::Spec->catpath($volume, $directory, '');\n+\t}\n+\n+\tmy $temp = File::Temp->new(DIR => $directory,\n+\t                          TEMPLATE => \"${filename}_XXXXX\",\n+\t                          SUFFIX => '.tmp');\n+\tbinmode($temp);\n+\tprint {$temp} $p_data;\n+\tclose($temp);\n+\n+\trename($temp, $p_path);\n+}\n+\n+# ensures that directory leading to path exists, or dies\n+sub _Make_Path {\n+\tmy ($p_path, $p_dir) = @_;\n+\n+\tmy ($volume, $directory, $filename) = File::Spec->splitpath($p_path);\n+\tif (defined $directory and defined $volume) {\n+\t\t$directory = File::Spec->catpath($volume, $directory, \"\");\n+\t}\n+\n+\treturn\n+\t\tunless (defined $directory and not -d $directory);\n+\n+\tmy $numdirs = make_path($directory,\n+\t                        { mode => 0777, error => \\my $mkdirerr });\n+\tif (@$mkdirerr) {\n+\t\tmy $mkdirerrmsg = \"\";\n+\t\tfor my $diag (@$mkdirerr) {\n+\t\t\tmy ($file, $message) = %$diag;\n+\t\t\tif ($file eq '' ){\n+\t\t\t\t$mkdirerrmsg .= \"general error: $message\\n\";\n+\t\t\t} else {\n+\t\t\t\t$mkdirerrmsg .= \"problem unlinking $file: $message\\n\";\n+\t\t\t}\n+\t\t}\n+\t\t#die_error(500, \"Could not create cache directory | $mkdirerrmsg\");\n+\t}\n+}\n+\n+sub _Remove_File {\n+\tmy ($p_path) = @_;\n+\n+\tif (-f $p_path) {\n+\t\tunlink($p_path);\n+\t}\n+}\n+\n+# _read_data and _write_data methods do deserialization/serialization\n+# in original implementation in Cache::Cache distribution\n+\n+sub _read_data {\n+\tmy ($self, $p_path) = @_;\n+\n+\treturn _Read_File($p_path);\n+}\n+\n+sub _write_data {\n+\tmy ($self, $p_path, $p_data) = @_;\n+\n+\t_Make_Path($p_path);\n+\t_Write_File($p_path, $p_data);\n+}\n+\n+# ----------------------------------------------------------------------\n+# worker methods (explicit namespace)\n+\n+sub restore {\n+\tmy ($self, $p_namespace, $p_key) = @_;\n+\n+\treturn $self->_read_data($self->_path_to_key($p_namespace, $p_key));\n+}\n+\n+sub store {\n+\tmy ($self, $p_namespace, $p_key, $p_data) = @_;\n+\n+\t$self->_write_data($self->_path_to_key($p_namespace, $p_key),\n+\t                   $p_data);\n+}\n+\n+sub delete_key {\n+\tmy ($self, $p_namespace, $p_key) = @_;\n+\n+\t_Remove_File($self->_path_to_key($p_namespace, $p_key));\n+}\n+\n+sub get_size {\n+\tmy ($self, $p_namespace, $p_key) = @_;\n+\n+\tmy $path = $self->_path_to_key($p_namespace, $p_key);\n+\tif (-e $path) {\n+\t\treturn -s $path;\n+\t}\n+\treturn 0;\n+}\n+\n+# ......................................................................\n+# interface methods\n+\n+# Removing and expiring\n+\n+sub remove {\n+\tmy ($self, $p_key) = @_;\n+\n+\t$self->delete_key($self->get_namespace(), $p_key);\n+}\n+\n+# Getting and setting\n+\n+sub set {\n+\tmy ($self, $p_key, $p_data) = @_;\n+\n+\t$self->store($self->get_namespace(), $p_key, $p_data);\n+}\n+\n+sub get {\n+\tmy ($self, $p_key) = @_;\n+\n+\tmy $data = $self->restore($self->get_namespace(), $p_key)\n+\t\tor return undef;\n+\n+\treturn $data;\n+}\n+\n+sub compute {\n+\tmy ($self, $p_key, $p_coderef) = @_;\n+\n+\tmy $data = $self->get($p_key);\n+\tif (!defined $data) {\n+\t\t$data = $p_coderef->($self, $p_key);\n+\t\t$self->set($p_key, $data);\n+\t}\n+\n+\treturn $data;\n+}\n+\n+1;\n+} # end of package GitwebCache::SimpleFileCache;\n+\n+1;\ndiff --git a/t/t9503-gitweb-caching.sh b/t/t9503-gitweb-caching.sh\nnew file mode 100755\nindex 0000000..768080c\n--- /dev/null\n+++ b/t/t9503-gitweb-caching.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Jakub Narebski\n+#\n+\n+test_description='caching interface to be used in gitweb'\n+#test_description='caching interface used in gitweb, gitweb caching\n+#\n+#This test checks cache (interface) used in gitweb caching, caching\n+#infrastructure and gitweb response (output) caching (the last by\n+#running gitweb as CGI script from commandline).'\n+\n+# for now we are running only cache interface tests\n+. ./test-lib.sh\n+\n+# this test is present in gitweb-lib.sh\n+if ! test_have_prereq PERL; then\n+\tsay 'perl not available, skipping test'\n+\ttest_done\n+fi\n+\n+\"$PERL_PATH\" -MTest::More -e 0 >/dev/null 2>&1 || {\n+\tsay 'perl module Test::More unavailable, skipping test'\n+\ttest_done\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+test_external 'GitwebCache::* Perl API (in gitweb/cache.pm)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_cache_interface.pl\n+\n+test_done\ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nnew file mode 100755\nindex 0000000..0b6628b\n--- /dev/null\n+++ b/t/t9503/test_cache_interface.pl\n@@ -0,0 +1,77 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use Test::More;\n+\n+# test source version; there is no installation target for gitweb\n+my $cache_pm = \"$ENV{TEST_DIRECTORY}/../gitweb/cache.pm\";\n+\n+unless (-f \"$cache_pm\") {\n+\tplan skip_all => \"gitweb/cache.pm not found\";\n+}\n+\n+# it is currently not a proper Perl module, so we use 'do FILE'\n+#ok(eval { do \"$cache_pm\"; 1 or die $!; }, \"loading gitweb/cache.pm\");\n+my $return = do \"$cache_pm\";\n+ok(!$@,              \"parse gitweb/cache.pm\");\n+ok(defined $return,  \"do    gitweb/cache.pm\");\n+ok($return,          \"run   gitweb/cache.pm\");\n+# instead of: BEGIN { use_ok('GitwebCache::SimpleFileCache') }\n+\n+# Test creating a cache\n+#\n+my $cache = new_ok('GitwebCache::SimpleFileCache',\n+\t[ { 'cache_root' => 'cache', 'cache_depth' => 2 } ]);\n+\n+# Test that default values are defined\n+#\n+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,\n+\t'$DEFAULT_CACHE_ROOT defined');\n+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,\n+\t'$DEFAULT_CACHE_DEPTH defined');\n+\n+# Test accessors and default values for cache\n+#\n+SKIP: {\n+\tskip 'default values not defined', 3\n+\t\tunless ($GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT &&\n+\t\t        $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH);\n+\n+\tis($cache->get_namespace(), '', \"default namespace is ''\");\n+\tis($cache->get_root(), $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,\n+\t\t\"default cache root is '$GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT'\");\n+\tcmp_ok($cache->get_depth(), '==', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,\n+\t\t\"default cache depth is $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH\");\n+}\n+\n+# Test the getting, setting, and removal of a cached value\n+# (Cache::Cache interface)\n+#\n+my $key = 'Test Key';\n+my $value = 'Test Value';\n+can_ok($cache, qw(get set remove));\n+#ok(!defined($cache->get($key)),        'get before set')\n+#\tor diag(\"get returned '\", $cache->get($key), \"' for $key\");\n+$cache->set($key, $value);\n+is($cache->get($key), $value,          'get after set, returns cached value');\n+$cache->remove($key);\n+ok(!defined($cache->get($key)),        'get after remove, is undefined');\n+\n+# Test the getting and setting of a cached value\n+# (CHI interface)\n+#\n+my $call_count = 0;\n+sub get_value {\n+\t$call_count++;\n+\treturn $value;\n+}\n+can_ok($cache, qw(compute));\n+is($cache->compute($key, \\&get_value), $value, 'compute 1st time (set)');\n+is($cache->compute($key, \\&get_value), $value, 'compute 2nd time (get)');\n+is($cache->compute($key, \\&get_value), $value, 'compute 3rd time (get)');\n+cmp_ok($call_count, '==', 1, 'get_value() is called once');\n+\n+done_testing();\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 142f36f..9282d9e 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -429,6 +429,9 @@ test_external () {\n \t\t# Announce the script to reduce confusion about the\n \t\t# test output that follows.\n \t\tsay_color \"\" \" run $test_count: $descr ($*)\"\n+\t\t# Export TEST_DIRECTORY and TRASH_DIRECTORY\n+\t\t# to be able to use them in script\n+\t\texport TEST_DIRECTORY TRASH_DIRECTORY\n \t\t# Run command; redirect its stderr to &4 as in\n \t\t# test_run_, but keep its stdout on our stdout even in\n \t\t# non-verbose mode.\n-- \n1.6.6.1\n"},{"id":"132438","messageId":"c59ba5d8c0a907340c7f2cff5b3a0c006530293d.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:27Z","receivedAt":"2010-01-23T00:27:27Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add stat-based cache expiration to file-based GitwebCache::SimpleFileCache.\nContrary to the way other caching interfaces such as Cache::Cache and CHI\ndo it, the time cache element expires in is _global_ value associated with\ncache instance, and is not local property of cache entry.  (Currently cache\nentry does not store any metadata associated with entry... which means that\nthere is no need for serialization / marshalling / freezing and thawing.)\nDefault expire time is -1, which means never expire.\n\nTo check if cache entry is expired, GitwebCache::SimpleFileCache compares\ndifference between mtime (last modify time) of a cache file and current time\nwith (global) time to expire.  It is done using CHI-compatibile is_valid()\nmethod.\n\nAdd test checking that expire time of 0 (expire now) works correctly.\n\n\nWhile at it show diagnostic if there were parse errors in gitweb/cache.pm\n(TO BE MOVED TO PREVIOUS COMMIT).\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThe main difference from the way J.H. did it in his patch adding output\ncaching to gitweb is that here I uses imply stat, and not open/stat/close.\n\nNow that cahcing engine supports cache expiration, we can add caching\nsupport to gitweb.\n\n gitweb/cache.pm                 |   47 ++++++++++++++++++++++++++++++++++++++-\n t/t9503/test_cache_interface.pl |   11 ++++++++-\n 2 files changed, 56 insertions(+), 2 deletions(-)\n\ndiff --git a/gitweb/cache.pm b/gitweb/cache.pm\nindex ea544b0..12a7a78 100644\n--- a/gitweb/cache.pm\n+++ b/gitweb/cache.pm\n@@ -50,6 +50,10 @@ our $DEFAULT_CACHE_ROOT = \"cache\";\n #    The number of subdirectories deep to cache object item.  This should be\n #    large enough that no cache directory has more than a few hundred objects.\n #    Defaults to 2 unless explicitly set.\n+#  * 'default_expires_in' (Cache::Cache compatibile),\n+#    'expires_in' (CHI compatibile) [seconds]\n+#    The expiration time for objects place in the cache.\n+#    Defaults to $EXPIRES_NEVER if not explicitly set.\n sub new {\n \tmy ($proto, $p_options_hash_ref) = @_;\n \n@@ -57,19 +61,24 @@ sub new {\n \tmy $self  = {};\n \t$self = bless($self, $class);\n \n-\tmy ($root, $depth, $ns);\n+\tmy ($root, $depth, $ns, $expires_in);\n \tif (defined $p_options_hash_ref) {\n \t\t$root  = $p_options_hash_ref->{'cache_root'};\n \t\t$depth = $p_options_hash_ref->{'cache_depth'};\n \t\t$ns    = $p_options_hash_ref->{'namespace'};\n+\t\t$expires_in =\n+\t\t\t$p_options_hash_ref->{'default_expires_in'} ||\n+\t\t\t$p_options_hash_ref->{'expires_in'};\n \t}\n \t$root  = $DEFAULT_CACHE_ROOT  unless defined($root);\n \t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n \t$ns    = '' unless defined($ns);\n+\t$expires_in = -1 unless defined($expires_in); # <0 means never\n \n \t$self->set_root($root);\n \t$self->set_depth($depth);\n \t$self->set_namespace($ns);\n+\t$self->set_expires_in($expires_in);\n \n \treturn $self;\n }\n@@ -115,6 +124,20 @@ sub set_namespace {\n \t$self->{'_Namespace'} = $namespace;\n }\n \n+sub get_expires_in {\n+\tmy ($self) = @_;\n+\n+\treturn $self->{'_Expires_In'};\n+}\n+\n+\n+sub set_expires_in {\n+\tmy ($self, $expires_in) = @_;\n+\n+\t$self->{'_Expires_In'} = $expires_in;\n+}\n+\n+\n # ----------------------------------------------------------------------\n # (private) utility functions and methods\n \n@@ -282,6 +305,27 @@ sub remove {\n \t$self->delete_key($self->get_namespace(), $p_key);\n }\n \n+# exists in cache and is not expired\n+sub is_valid {\n+\tmy ($self, $p_key) = @_;\n+\n+\t# should there be namespace variant of this function?\n+\tmy $path = $self->_path_to_key($self->get_namespace(), $p_key);\n+\n+\t# does file exists in cache?\n+\treturn 0 unless -f $path;\n+\n+\t# expire time can be set to never\n+\tmy $expires_in = $self->get_expires_in();\n+\treturn 1 unless (defined $expires_in && $expires_in >= 0);\n+\n+\t# is file expired?\n+\tmy $mtime = (stat(_))[9];\n+\tmy $now = time();\n+\n+\treturn (($now - $mtime) < $expires_in);\n+}\n+\n # Getting and setting\n \n sub set {\n@@ -293,6 +337,7 @@ sub set {\n sub get {\n \tmy ($self, $p_key) = @_;\n \n+\treturn undef unless $self->is_valid($p_key);\n \tmy $data = $self->restore($self->get_namespace(), $p_key)\n \t\tor return undef;\n \ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nindex 0b6628b..3644ca8 100755\n--- a/t/t9503/test_cache_interface.pl\n+++ b/t/t9503/test_cache_interface.pl\n@@ -16,7 +16,8 @@ unless (-f \"$cache_pm\") {\n # it is currently not a proper Perl module, so we use 'do FILE'\n #ok(eval { do \"$cache_pm\"; 1 or die $!; }, \"loading gitweb/cache.pm\");\n my $return = do \"$cache_pm\";\n-ok(!$@,              \"parse gitweb/cache.pm\");\n+ok(!$@,              \"parse gitweb/cache.pm\")\n+\tor diag(\"parse error:\\n\", $@);\n ok(defined $return,  \"do    gitweb/cache.pm\");\n ok($return,          \"run   gitweb/cache.pm\");\n # instead of: BEGIN { use_ok('GitwebCache::SimpleFileCache') }\n@@ -74,4 +75,12 @@ is($cache->compute($key, \\&get_value), $value, 'compute 2nd time (get)');\n is($cache->compute($key, \\&get_value), $value, 'compute 3rd time (get)');\n cmp_ok($call_count, '==', 1, 'get_value() is called once');\n \n+# Test cache expiration for 'expire now'\n+#\n+$cache->set_expires_in(0);\n+is($cache->get_expires_in(), 0,        '\"expires in\" is set to now (0)');\n+$cache->set($key, $value);\n+ok(!defined($cache->get($key)),        'cache is expired');\n+\n+\n done_testing();\n-- \n1.6.6.1\n"},{"id":"132447","messageId":"553a364cdfdbb1d5c6dfc75f0545e46143a1be8a.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set) output caching (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:28Z","receivedAt":"2010-01-23T00:27:28Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis patch, as you can see, lack proper commit message: it is work in\nprogress.\n\nAs you can see cache_fetch() subroutine is much, much simpler that the\none in original patch by J.H.:\n  [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)\n  Message-ID: <1263432185-21334-10-git-send-email-warthog9@eaglescrag.net>\n  http://permalink.gmane.org/gmane.comp.version-control.git/136917\n\nAs you can see I have introduced $cache_pm variable, to be able to\ntest caching in t/t9500-gitweb-standalone-no-errors.sh, but also to be\nable to install cache.pm in some other place than along gitweb.cgi.\nThere would be no such problems if we used 'require GitwebCache' or\nsomesuch, in place of 'do \"cache.pm\"' like in original patch by J.H.\nBut at leats for now I have decided to follow J.H. in this issue.\n\nInstead of using binary (sic!) valued $cache_enable as in J.H. patch,\nI use set of two variables: $cache (to be able to select what caching\nengine to use, and what features should be enabled), and\n$caching_enabled to actually enable/disable cache.\n\nWhen caching is enabled gitweb do not output timing info (time to\ngenerate page), as it would contain incorrect information if the page\nwas retrieved from cache; in this place we could put cachetime info\nfrom the original patch by J.H., which means writing\n\n  Cache Last Updated: \". gmtime( time )\n\nI have forgot about adding this feature...\n\n\nATTENTION !!!: I have run both tests (t9500 to check for errors in\ngitweb.perl, and t9503 to test the API), but I haven't actually tested\nthat *gitweb itself* behaves correctly.\n\n gitweb/cache.pm                        |   36 ++++++++++++++++++++++++++++\n gitweb/gitweb.perl                     |   40 +++++++++++++++++++++++++++++--\n t/gitweb-lib.sh                        |    2 +\n t/t9500-gitweb-standalone-no-errors.sh |   13 ++++++++++\n 4 files changed, 88 insertions(+), 3 deletions(-)\n\ndiff --git a/gitweb/cache.pm b/gitweb/cache.pm\nindex 12a7a78..3a33158 100644\n--- a/gitweb/cache.pm\n+++ b/gitweb/cache.pm\n@@ -359,4 +359,41 @@ sub compute {\n 1;\n } # end of package GitwebCache::SimpleFileCache;\n \n+# human readable key identifying gitweb output\n+sub gitweb_output_key {\n+\treturn href(-replay => 1, -full => 1, -path_info => 0);\n+}\n+\n+sub cache_fetch {\n+\tmy ($cache, $action) = @_;\n+\n+\tmy $key = gitweb_output_key();\n+\tmy $data = $cache->get($key);\n+\n+\tif (defined $data) {\n+\t\t# print cached data\n+\t\tbinmode STDOUT, ':raw';\n+\t\tprint STDOUT $data;\n+\n+\t} else {\n+\t\t# calculate data and regenerate data\n+\t\topen my $data_fh, '>', \\$data\n+\t\t\tor die \"Can't open memory file: $!\";\n+\t\t# matches \"binmode STDOUT, ':uft8'\" at beginning\n+\t\tbinmode $data_fh, ':utf8';\n+\n+\t\t$out = $data_fh || \\*STDOUT;\n+\t\t$actions{$action}->();\n+\n+\t\tif (defined $data) {\n+\t\t\t$cache->set($key, $data);\n+\t\t\tbinmode STDOUT, ':raw';\n+\t\t\tlocal $/ = undef;\n+\t\t\tprint STDOUT $data;\n+\t\t}\n+\n+\t\tclose $data_fh;\n+\t}\n+}\n+\n 1;\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex cd5073c..0394dc8 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -234,6 +234,22 @@ our $gitlinkurl_base = (\"++GITWEB_BASE_URL++\" =~ m!^(git://.*)$!) ? $1 : '';\n # Leave it undefined (or set to 'undef') to turn off load checking.\n our $maxload = 300;\n \n+# This enables/disables the caching layer in gitweb.  Currently supported\n+# is only output (response) caching, similar to the one used on git.kernel.org.\n+our $caching_enabled = 0;\n+# Set to _initialized_ instance of cache interface implementing (at least)\n+# get($key) and set($key, $data) methods (Cache::Cache and CHI interfaces).\n+# If unset, GitwebCache::SimpleFileCache would be used, which is 'dumb'\n+# (but fast) file based caching layer, currently without any support for\n+# cache size limiting.  It is therefore recommended that the cache directory\n+# be periodically completely deleted; this operation is safe to perform.\n+# Suggested mechanism:\n+# mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush\n+our $cache;\n+# Locations of 'cache.pm' file; if it is relative path, it is relative to\n+# the directory gitweb is run from\n+our $cache_pm = 'cache.pm';\n+\n # You define site-wide feature defaults here; override them with\n # $GITWEB_CONFIG as necessary.\n our %feature = (\n@@ -998,7 +1014,21 @@ if ($action !~ m/^(?:opml|project_list|project_index)$/ &&\n     !$project) {\n \tdie_error(400, \"Project needed\");\n }\n-$actions{$action}->();\n+\n+if ($caching_enabled) {\n+\tdo $cache_pm;\n+\tdie $@ if $@;\n+\n+\t$cache ||= GitwebCache::SimpleFileCache->new({\n+\t\t'cache_root'  => '/tmp/cache',\n+\t\t'cache_depth' => 2,\n+\t\t'expires_in'  => 20, # in seconds\n+\t});\n+\tcache_fetch($cache, $action);\n+} else {\n+\t$actions{$action}->();\n+}\n+\n exit;\n \n ## ======================================================================\n@@ -3207,7 +3237,9 @@ sub git_header_html {\n \t# 'application/xhtml+xml', otherwise send it as plain old 'text/html'.\n \t# we have to do this because MSIE sometimes globs '*/*', pretending to\n \t# support xhtml+xml but choking when it gets what it asked for.\n-\tif (defined $cgi->http('HTTP_ACCEPT') &&\n+\t# Disable content-type negotiation when caching (use mimetype good for all).\n+\tif (!$caching_enabled &&\n+\t    defined $cgi->http('HTTP_ACCEPT') &&\n \t    $cgi->http('HTTP_ACCEPT') =~ m/(,|;|\\s|^)application\\/xhtml\\+xml(,|;|\\s|$)/ &&\n \t    $cgi->Accept('application/xhtml+xml') != 0) {\n \t\t$content_type = 'application/xhtml+xml';\n@@ -3380,7 +3412,9 @@ sub git_footer_html {\n \t}\n \tprint {$out} \"</div>\\n\"; # class=\"page_footer\"\n \n-\tif (defined $t0 && gitweb_check_feature('timed')) {\n+\t# timing info doesn't make much sense with output (response) caching\n+\tif (!$caching_enabled &&\n+\t    defined $t0 && gitweb_check_feature('timed')) {\n \t\tprint {$out} \"<div id=\\\"generating_info\\\">\\n\";\n \t\tprint {$out} 'This page took '.\n \t\t             '<span id=\"generating_time\" class=\"time_span\">'.\ndiff --git a/t/gitweb-lib.sh b/t/gitweb-lib.sh\nindex d9ffc90..d041083 100755\n--- a/t/gitweb-lib.sh\n+++ b/t/gitweb-lib.sh\n@@ -27,6 +27,8 @@ our \\$export_ok = '';\n our \\$strict_export = '';\n our \\$git_versions_must_match = 0;\n \n+our \\$cache_pm = '$TEST_DIRECTORY/../gitweb/cache.pm';\n+\n EOF\n \n \tcat >.git/description <<EOF\ndiff --git a/t/t9500-gitweb-standalone-no-errors.sh b/t/t9500-gitweb-standalone-no-errors.sh\nindex 2fc7fdb..0f93962 100755\n--- a/t/t9500-gitweb-standalone-no-errors.sh\n+++ b/t/t9500-gitweb-standalone-no-errors.sh\n@@ -639,4 +639,17 @@ test_expect_success \\\n \t gitweb_run \"p=.git;a=summary\"'\n test_debug 'cat gitweb.log'\n \n+# ----------------------------------------------------------------------\n+# caching\n+\n+cat >>gitweb_config.perl <<\\EOF\n+\n+$caching_enabled = 1;\n+EOF\n+test_expect_success \\\n+\t'caching enabled' \\\n+\t'gitweb_run \"p=.git;a=summary\"'\n+test_debug 'cat gitweb.log'\n+\n+\n test_done\n-- \n1.6.6.1\n"},{"id":"132442","messageId":"1021d8b597baf908f9770e36dddde87e9fbf1a0c.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:29Z","receivedAt":"2010-01-23T00:27:29Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis patch, as you can see, lack proper commit message: it is work in\nprogress.\n\nAs it is quick'n'dirty proof of concept patch, it lacks variables in\ngitweb.perl that can be used to control cache behaviour.  In original\npatch in J.H. those were the following variables: $minCacheTime,\n$maxCacheTime.\n\nIn the final version we should probably leave $check_load undefined,\nor set it to 'sub { return 0; }' instead of requiring that there is\nget_loadavg() subroutine in main namespace (see t9503 test change).\nIt would be alos good idea to leave ->set_expires_in() subroutine\nwhich would set min and max range so that adaptive cache lifetime is\neffectively disabled.\n\n gitweb/cache.pm                 |   82 ++++++++++++++++++++++++++++++++++----\n t/t9503/test_cache_interface.pl |    8 +++-\n 2 files changed, 80 insertions(+), 10 deletions(-)\n\ndiff --git a/gitweb/cache.pm b/gitweb/cache.pm\nindex 3a33158..9692e8d 100644\n--- a/gitweb/cache.pm\n+++ b/gitweb/cache.pm\n@@ -61,24 +61,36 @@ sub new {\n \tmy $self  = {};\n \t$self = bless($self, $class);\n \n-\tmy ($root, $depth, $ns, $expires_in);\n+\tmy ($root, $depth, $ns);\n+\tmy ($expires_min, $expires_max, $increase_factor, $check_load);\n \tif (defined $p_options_hash_ref) {\n \t\t$root  = $p_options_hash_ref->{'cache_root'};\n \t\t$depth = $p_options_hash_ref->{'cache_depth'};\n \t\t$ns    = $p_options_hash_ref->{'namespace'};\n-\t\t$expires_in =\n+\t\t$expires_min =\n+\t\t\t$p_options_hash_ref->{'expires_min'} ||\n \t\t\t$p_options_hash_ref->{'default_expires_in'} ||\n \t\t\t$p_options_hash_ref->{'expires_in'};\n+\t\t$expires_max =\n+\t\t\t$p_options_hash_ref->{'expires_max'};\n+\t\t$increase_factor = $p_options_hash_ref->{'expires_factor'};\n+\t\t$check_load      = $p_options_hash_ref->{'check_load'};\n \t}\n \t$root  = $DEFAULT_CACHE_ROOT  unless defined($root);\n \t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n \t$ns    = '' unless defined($ns);\n-\t$expires_in = -1 unless defined($expires_in); # <0 means never\n+\t$expires_min =   20 unless defined($expires_min);\n+\t$expires_max = 1200 unless defined($expires_max);\n+\t$increase_factor = 60 unless defined($increase_factor);\n+\t$check_load = \\&main::get_loadavg unless defined($check_load);\n \n \t$self->set_root($root);\n \t$self->set_depth($depth);\n \t$self->set_namespace($ns);\n-\t$self->set_expires_in($expires_in);\n+\t$self->set_expires_min($expires_min);\n+\t$self->set_expires_max($expires_max);\n+\t$self->set_increase_factor($increase_factor);\n+\t$self->set_check_load($check_load);\n \n \treturn $self;\n }\n@@ -124,19 +136,71 @@ sub set_namespace {\n \t$self->{'_Namespace'} = $namespace;\n }\n \n-sub get_expires_in {\n+\n+sub get_expires_min {\n \tmy ($self) = @_;\n \n-\treturn $self->{'_Expires_In'};\n+\treturn $self->{'_Expires_Min'};\n+}\n+\n+sub set_expires_min {\n+\tmy ($self, $expires_min) = @_;\n+\n+\t$self->{'_Expires_Min'} = $expires_min;\n }\n \n+sub get_expires_max {\n+\tmy ($self) = @_;\n+\n+\treturn $self->{'_Expires_Max'};\n+}\n+\n+sub set_expires_max {\n+\tmy ($self, $expires_max) = @_;\n+\n+\t$self->{'_Expires_Max'} = $expires_max;\n+}\n+\n+sub get_increase_factor {\n+\tmy ($self) = @_;\n+\n+\treturn $self->{'_Increase_Factor'};\n+}\n \n-sub set_expires_in {\n-\tmy ($self, $expires_in) = @_;\n+sub set_increase_factor {\n+\tmy ($self, $increase_factor) = @_;\n \n-\t$self->{'_Expires_In'} = $expires_in;\n+\t$self->{'_Increase_Factor'} = $increase_factor;\n }\n \n+sub get_check_load {\n+\tmy ($self) = @_;\n+\n+\treturn $self->{'_Check_Load'};\n+}\n+\n+sub set_check_load {\n+\tmy ($self, $sub) = @_;\n+\n+\t$self->{'_Check_Load'} = $sub;\n+}\n+\n+# ......................................................................\n+\n+sub get_expires_in {\n+\tmy ($self) = @_;\n+\tmy $expires_in =\n+\t\t#$self->get_expires_min() +\n+\t\t$self->get_increase_factor() * $self->get_check_load()->();\n+\n+\tif ($expires_in < $self->get_expires_min()) {\n+\t\treturn $self->get_expires_min();\n+\t} elsif ($expires_in > $self->get_expires_max()) {\n+\t\treturn $self->get_expires_max();\n+\t}\n+\n+\treturn $expires_in;\n+}\n \n # ----------------------------------------------------------------------\n # (private) utility functions and methods\ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nindex 3644ca8..0870b87 100755\n--- a/t/t9503/test_cache_interface.pl\n+++ b/t/t9503/test_cache_interface.pl\n@@ -6,6 +6,11 @@ use strict;\n \n use Test::More;\n \n+# mockup\n+sub get_loadavg {\n+\treturn 0.0;\n+}\n+\n # test source version; there is no installation target for gitweb\n my $cache_pm = \"$ENV{TEST_DIRECTORY}/../gitweb/cache.pm\";\n \n@@ -77,7 +82,8 @@ cmp_ok($call_count, '==', 1, 'get_value() is called once');\n \n # Test cache expiration for 'expire now'\n #\n-$cache->set_expires_in(0);\n+$cache->set_expires_min(0);\n+$cache->set_expires_max(0);\n is($cache->get_expires_in(), 0,        '\"expires in\" is set to now (0)');\n $cache->set($key, $value);\n ok(!defined($cache->get($key)),        'cache is expired');\n-- \n1.6.6.1\n"},{"id":"132439","messageId":"d878f121da06da6dc9f9b7174b34c62ed754f482.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:30Z","receivedAt":"2010-01-23T00:27:30Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis patch, as you can see, lack proper commit message: it is work in\nprogress.\n\nThe reason behind supporting compute() as interface to cache is that\nwith this interface it is possible (as it can be seen in the following\npatch) to use locking to avoid cache miss stampede (only one process\nregenerates cache).\n\nThe support for $cache which do not provide '->compute($key, $code)'\nmethod is left just in case we would want to use such (external)\ncaching engine.  We should probably add generic_compute() subroutine\nwhich would use get/set, and provide compute-like interface.\n\n gitweb/cache.pm |   53 +++++++++++++++++++++++++++++++++++++++++++----------\n 1 files changed, 43 insertions(+), 10 deletions(-)\n\ndiff --git a/gitweb/cache.pm b/gitweb/cache.pm\nindex 9692e8d..8dd4f39 100644\n--- a/gitweb/cache.pm\n+++ b/gitweb/cache.pm\n@@ -432,6 +432,48 @@ sub cache_fetch {\n \tmy ($cache, $action) = @_;\n \n \tmy $key = gitweb_output_key();\n+\tif ($cache->can('compute')) {\n+\t\tcache_fetch_compute($cache, $action, $key);\n+\t} else {\n+\t\tcache_fetch_get_set($cache, $action, $key);\n+\t}\n+}\n+\n+# calculate data to regenerate cache\n+sub cache_calculate {\n+\tmy ($action) = @_;\n+\n+\tmy $data;\n+\topen my $data_fh, '>', \\$data\n+\t\tor die \"Can't open memory file: $!\";\n+\t# matches \"binmode STDOUT, ':uft8'\" at beginning\n+\tbinmode $data_fh, ':utf8';\n+\n+\t$out = $data_fh || \\*STDOUT;\n+\t$actions{$action}->();\n+\n+\tclose $data_fh;\n+\n+\treturn $data;\n+}\n+\n+# for $cache which can ->compute($key, $code)\n+sub cache_fetch_compute {\n+\tmy ($cache, $action, $key) = @_;\n+\n+\tmy $data = $cache->compute($key, sub { cache_calculate($action) });\n+\n+\tif (defined $data) {\n+\t\t# print cached data\n+\t\tbinmode STDOUT, ':raw';\n+\t\tlocal $/ = undef;\n+\t\tprint STDOUT $data;\n+\t}\n+}\n+\n+# for $cache which can ->get($key) and ->set($key, $data)\n+sub cache_fetch_get_set {\n+\tmy ($cache, $action, $key) = @_;\n+\n \tmy $data = $cache->get($key);\n \n \tif (defined $data) {\n@@ -440,22 +482,13 @@ sub cache_fetch {\n \t\tprint STDOUT $data;\n \n \t} else {\n-\t\t# calculate data and regenerate data\n-\t\topen my $data_fh, '>', \\$data\n-\t\t\tor die \"Can't open memory file: $!\";\n-\t\t# matches \"binmode STDOUT, ':uft8'\" at beginning\n-\t\tbinmode $data_fh, ':utf8';\n-\n-\t\t$out = $data_fh || \\*STDOUT;\n-\t\t$actions{$action}->();\n+\t\t$data = cache_calculate($action);\n \n \t\tif (defined $data) {\n \t\t\t$cache->set($key, $data);\n \t\t\tbinmode STDOUT, ':raw';\n \t\t\tlocal $/ = undef;\n \t\t\tprint STDOUT $data;\n \t\t}\n-\n-\t\tclose $data_fh;\n \t}\n }\n \n-- \n1.6.6.1\n"},{"id":"132440","messageId":"2ae094e73aacb9d855a71fa60d8982645293872c.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd' problem (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:31Z","receivedAt":"2010-01-23T00:27:31Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis patch, as you can see, lack proper commit message: it is work in\nprogress.\n\nAs you can see we can almost do the same for the cache which supports\nonly get/set interface... but for replacing ->retrieve() with ->get()\nin ->compute().  This is argument for generic_compute subroutine,\nmentioned in previous patch.\n\nNote that we actually can and do test that provided mechanism avoid\ncache miss stampede (aka 'stampeding herd') problem... although the\ntest should probably be run only with --long (this would need update\nto t/test-lib.sh to pass GIT_TEST_LONG to external tests in\ntest_external and the like), as we need to sleep at least one second\nto ensure that we would have 'stampeding herd' problem.\n\n gitweb/cache.pm                 |   27 ++++++++++++++--\n t/t9503/test_cache_interface.pl |   66 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 90 insertions(+), 3 deletions(-)\n\ndiff --git a/gitweb/cache.pm b/gitweb/cache.pm\nindex 8dd4f39..f514ee9 100644\n--- a/gitweb/cache.pm\n+++ b/gitweb/cache.pm\n@@ -24,6 +24,7 @@ use File::Path qw(make_path);  # requires version >= 2.0\n use File::Spec;\n use File::Temp;\n use Digest::MD5 qw(md5_hex);\n+use Fcntl qw(:flock);\n \n # by default, the cache nests all entries on the filesystem two\n # directories deep\n@@ -217,7 +218,7 @@ sub _path_to_key {\n \tmy ($self, $p_namespace, $p_key) = @_;\n \n \treturn $self->_path_to_hashed_key($p_namespace,\n-\t                                    _Build_Hashed_Key($p_key));\n+\t                                  _Build_Hashed_Key($p_key));\n }\n \n # Take hashed key, and return file path\n@@ -228,6 +229,13 @@ sub _path_to_hashed_key {\n \t                           _Split_Word($p_hashed_key, $self->get_depth()));\n }\n \n+sub _lockfile_to_key {\n+\tmy ($self, $p_namespace, $p_key) = @_;\n+\n+\treturn $self->_path_to_hashed_key($p_namespace,\n+\t                                  _Build_Hashed_Key($p_key)) . '.lock';\n+}\n+\n # Split word into N components, where each component but last is two-letter word\n # e.g. _Split_Word(\"06b90e786e304a18fdfbd7c7bcc41a6b\", 2) == qw(06 b90e786e304a18fdfbd7c7bcc41a6b);\n #      _Split_Word(\"06b90e786e304a18fdfbd7c7bcc41a6b\", 3) == qw(06 b9 0e786e304a18fdfbd7c7bcc41a6b);\n@@ -412,17 +420,30 @@ sub compute {\n \tmy ($self, $p_key, $p_coderef) = @_;\n \n \tmy $data = $self->get($p_key);\n-\tif (!defined $data) {\n+\treturn $data if defined $data;\n+\n+\tmy $lockfile = $self->_lockfile_to_key($self->get_namespace(), $p_key);\n+\t_Make_Path($lockfile);\n+\topen my $lock_fh, '+>', $lockfile;\n+\t#\tor die \"Can't open lockfile '$lockfile': $!\";\n+\tif (my $lock_state = flock($lock_fh, LOCK_EX | LOCK_NB)) {\n+\t\t# acquired writers lock\n \t\t$data = $p_coderef->($self, $p_key);\n \t\t$self->set($p_key, $data);\n+\t} else {\n+\t\t# get readers lock\n+\t\tflock($lock_fh, LOCK_SH);\n+\t\t$data = $self->restore($self->get_namespace(), $p_key);\n \t}\n-\n+\tclose $lock_fh;\n \treturn $data;\n }\n \n 1;\n } # end of package GitwebCache::SimpleFileCache;\n \n+# ======================================================================\n+\n # human readable key identifying gitweb output\n sub gitweb_output_key {\n \treturn href(-replay => 1, -full => 1, -path_info => 0);\ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nindex 0870b87..43b806d 100755\n--- a/t/t9503/test_cache_interface.pl\n+++ b/t/t9503/test_cache_interface.pl\n@@ -65,6 +65,8 @@ $cache->set($key, $value);\n is($cache->get($key), $value,          'get after set, returns cached value');\n $cache->remove($key);\n ok(!defined($cache->get($key)),        'get after remove, is undefined');\n+eval { $cache->remove('Not-Existent Key'); };\n+ok(!$@,                                'remove on non-existent key doesn\\'t die');\n \n # Test the getting and setting of a cached value\n # (CHI interface)\n@@ -80,6 +82,70 @@ is($cache->compute($key, \\&get_value), $value, 'compute 2nd time (get)');\n is($cache->compute($key, \\&get_value), $value, 'compute 3rd time (get)');\n cmp_ok($call_count, '==', 1, 'get_value() is called once');\n \n+# Test 'stampeding herd' / cache miss stampede problem\n+# (probably should be run only if GIT_TEST_LONG)\n+sub get_value_slow {\n+\t$call_count++;\n+\tsleep 1;\n+\treturn $value;\n+}\n+my ($pid, $kid_fh);\n+\n+$call_count = 0;\n+$cache->remove($key);\n+$pid = open $kid_fh, '-|';\n+SKIP: {\n+\tskip \"cannot fork: $!\", 1\n+\t\tunless defined $pid;\n+\n+\tmy $data = $cache->get($key);\n+\tif (!defined $data) {\n+\t\t$data = get_value_slow();\n+\t\t$cache->set($key, $data);\n+\t}\n+\n+\tif ($pid) {\n+\t\tmy $child_count = <$kid_fh>;\n+\t\tchomp $child_count;\n+\n+\t\twaitpid $pid, 0;\n+\t\tclose $kid_fh;\n+\n+\t\t$call_count += $child_count;\n+\t} else {\n+\t\tprint \"$call_count\\n\";\n+\t\texit 0;\n+\t}\n+\n+\tcmp_ok($call_count, '==', 2, 'parallel get/set: get_value_slow() called twice');\n+}\n+\n+$call_count = 0;\n+$cache->remove($key);\n+$pid = open $kid_fh, '-|';\n+SKIP: {\n+\tskip \"cannot fork: $!\", 1\n+\t\tunless defined $pid;\n+\n+\tmy $data = $cache->compute($key, \\&get_value_slow);\n+\n+\tif ($pid) {\n+\t\tmy $child_count = <$kid_fh>;\n+\t\tchomp $child_count;\n+\n+\t\twaitpid $pid, 0;\n+\t\tclose $kid_fh;\n+\n+\t\t$call_count += $child_count;\n+\t} else {\n+\t\tprint \"$call_count\\n\";\n+\t\texit 0;\n+\t}\n+\n+\tcmp_ok($call_count, '==', 1, 'parallel compute: get_value_slow() called once');\n+}\n+\n+\n # Test cache expiration for 'expire now'\n #\n $cache->set_expires_min(0);\n-- \n1.6.6.1\n"},{"id":"132441","messageId":"dba7265713bde1e403c3cbab6de81345b20f8ab8.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for filling cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:32Z","receivedAt":"2010-01-23T00:27:32Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis patch, as you can see, lack proper commit message: it is work in\nprogress.\n\nNote that contrary to original patch by J.H. we fork unconditionally;\notherwise the process which do recalculate cache entry would be at\ndisadvantage compared to processes which failed race to acquire\nwriters lock.  Perhaps this should be made configurable, though.\n\nAlso note that currently there is no limit on how stale the data which\nis being served while regenerating data can be, i.e. there is no\nequivalent for $maxCacheLife.  Simplest solution (I guess) would be to\njust delete cache entry if it is too stale upfront.\n\n gitweb/cache.pm                 |   32 +++++++++++++++++++++++++++-----\n t/t9503/test_cache_interface.pl |   37 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 64 insertions(+), 5 deletions(-)\n\ndiff --git a/gitweb/cache.pm b/gitweb/cache.pm\nindex f514ee9..1adf467 100644\n--- a/gitweb/cache.pm\n+++ b/gitweb/cache.pm\n@@ -426,14 +426,36 @@ sub compute {\n \t_Make_Path($lockfile);\n \topen my $lock_fh, '+>', $lockfile;\n \t#\tor die \"Can't open lockfile '$lockfile': $!\";\n+\n+\t# try to retrieve stale data\n+\t$data = $self->restore($self->get_namespace(), $p_key);\n+\n \tif (my $lock_state = flock($lock_fh, LOCK_EX | LOCK_NB)) {\n \t\t# acquired writers lock\n-\t\t$data = $p_coderef->($self, $p_key);\n-\t\t$self->set($p_key, $data);\n+\t\tmy $pid = fork() if $data;\n+\t\tif (!defined $pid || $pid) {\n+\t\t\t# parent, or didn't fork\n+\t\t\t$data = $p_coderef->($self, $p_key);\n+\t\t\t$self->set($p_key, $data);\n+\n+\t\t\tif ($pid) {\n+\t\t\t\t# wait for child (which would print) and exit\n+\t\t\t\twaitpid $pid, 0;\n+\t\t\t\texit 0;\n+\t\t\t} else {\n+\t\t\t\t# there is no child, or was no $data to serve in background\n+\t\t\t\t;\n+\t\t\t}\n+\t\t} else {\n+\t\t\t# child to serve $data\n+\t\t\t;\n+\t\t}\n \t} else {\n-\t\t# get readers lock\n-\t\tflock($lock_fh, LOCK_SH);\n-\t\t$data = $self->restore($self->get_namespace(), $p_key);\n+\t\tif (!defined $data) {\n+\t\t\t# get readers lock if there is no stale data to serve\n+\t\t\tflock($lock_fh, LOCK_SH);\n+\t\t\t$data = $self->restore($self->get_namespace(), $p_key);\n+\t\t}\n \t}\n \tclose $lock_fh;\n \treturn $data;\ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nindex 43b806d..788e8f2 100755\n--- a/t/t9503/test_cache_interface.pl\n+++ b/t/t9503/test_cache_interface.pl\n@@ -154,5 +154,42 @@ is($cache->get_expires_in(), 0,        '\"expires in\" is set to now (0)');\n $cache->set($key, $value);\n ok(!defined($cache->get($key)),        'cache is expired');\n \n+# Test that cache returns stale data in existing but expired cache situation\n+# (probably should be run only if GIT_TEST_LONG)\n+$cache->set_expires_min(0);\n+$cache->set_expires_max(0);\n+my $stale_value = 'Stale Value';\n+my $child_data = '';\n+$cache->set($key, $stale_value);\n+$call_count = 0;\n+$pid = open $kid_fh, '-|';\n+SKIP: {\n+\tskip \"cannot fork: $!\", 4\n+\t\tunless defined $pid;\n+\n+\tmy $data = $cache->compute($key, \\&get_value_slow);\n+\n+\tif ($pid) {\n+\t\t$child_data = <$kid_fh>;\n+\t\tchomp $child_data;\n+\n+\t\twaitpid $pid, 0;\n+\t\tclose $kid_fh;\n+\t} else {\n+\t\tprint \"$data\\n\";\n+\t\texit 0;\n+\t}\n+\n+\tis($data,       $stale_value, 'stale data in parent when expired');\n+\tis($child_data, $stale_value, 'stale data in child  when expired');\n+\n+\t# never expire\n+\t$cache->set_expires_min(-1);\n+\t$cache->set_expires_max(-1);\n+\tis($cache->get($key), $value, 'value got set correctly');\n+}\n+$cache->set_expires_min(0);\n+$cache->set_expires_max(0);\n+\n \n done_testing();\n-- \n1.6.6.1\n"},{"id":"132443","messageId":"f4660e9c7ffdb4a62da0c56703de002c9df3b598.1264198194.git.jnareb@gmail.com","threadId":"22216","inReplyTo":"1263432185-21334-10-git-send-email-warthog9@eaglescrag.net","subject":"[RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:27:33Z","receivedAt":"2010-01-23T00:27:33Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis patch, as you can see, lack proper commit message: it is work in\nprogress.\n\nContrary to the patch by J.H. the subroutine that is responsible for\ndoing \"Generating...\" progress info / activity indicator is in\ngitweb.perl, and not in cache.pm.  Also it does not use not declared\nnor defined $title variable ;-)\n\nCompared with original patch by J.H. gitweb prints generating info in\nmore restricted set of situations; the set of actions where gitweb\ndoes not generate activity indicator is larger.  We could probably\nprovide activity indicator also for (possibly) non-HTML output, like\n'blob_plain' or 'patches', provided that 'User-Agent' denotes that we\nare using web browser.\n\nIn the future we would probably want Ajax-y JavaScript-based activity\nindicator, without need for busy-wait.\n\nNOTE that I have not tested that it works, only that it should not\ncontain syntax errors...\n\n gitweb/cache.pm    |   32 ++++++++++++++++++++-\n gitweb/gitweb.perl |   75 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 104 insertions(+), 3 deletions(-)\n\ndiff --git a/gitweb/cache.pm b/gitweb/cache.pm\nindex 1adf467..2c7be3a 100644\n--- a/gitweb/cache.pm\n+++ b/gitweb/cache.pm\n@@ -63,7 +63,8 @@ sub new {\n \t$self = bless($self, $class);\n \n \tmy ($root, $depth, $ns);\n-\tmy ($expires_min, $expires_max, $increase_factor, $check_load);\n+\tmy ($expires_min, $expires_max, $increase_factor,\n+\t    $check_load, $generating_info);\n \tif (defined $p_options_hash_ref) {\n \t\t$root  = $p_options_hash_ref->{'cache_root'};\n \t\t$depth = $p_options_hash_ref->{'cache_depth'};\n@@ -76,6 +77,7 @@ sub new {\n \t\t\t$p_options_hash_ref->{'expires_max'};\n \t\t$increase_factor = $p_options_hash_ref->{'expires_factor'};\n \t\t$check_load      = $p_options_hash_ref->{'check_load'};\n+\t\t$generating_info = $p_options_hash_ref->{'generating_info'};\n \t}\n \t$root  = $DEFAULT_CACHE_ROOT  unless defined($root);\n \t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n@@ -92,6 +94,7 @@ sub new {\n \t$self->set_expires_max($expires_max);\n \t$self->set_increase_factor($increase_factor);\n \t$self->set_check_load($check_load);\n+\t$self->set_generating_info($generating_info);\n \n \treturn $self;\n }\n@@ -186,6 +189,18 @@ sub set_check_load {\n \t$self->{'_Check_Load'} = $sub;\n }\n \n+sub get_generating_info {\n+\tmy ($self) = @_;\n+\n+\treturn $self->{'_Generating_Info'};\n+}\n+\n+sub set_generating_info {\n+\tmy ($self, $sub) = @_;\n+\n+\t$self->{'_Generating_Info'} = $sub;\n+}\n+\n # ......................................................................\n \n sub get_expires_in {\n@@ -203,6 +218,12 @@ sub get_expires_in {\n \treturn $expires_in;\n }\n \n+sub generating_info {\n+\tif (defined $self->get_generating_info()) {\n+\t\t$self->get_generating_info()->($self, @_);\n+\t}\n+}\n+\n # ----------------------------------------------------------------------\n # (private) utility functions and methods\n \n@@ -434,6 +455,9 @@ sub compute {\n \t\t# acquired writers lock\n \t\tmy $pid = fork() if $data;\n \t\tif (!defined $pid || $pid) {\n+\t\t\t# provide \"generating page...\" info if there is no stale data to serve\n+\t\t\t$self->generating_info($p_key, $lock_fh)\n+\t\t\t\tunless ($data);\n \t\t\t# parent, or didn't fork\n \t\t\t$data = $p_coderef->($self, $p_key);\n \t\t\t$self->set($p_key, $data);\n@@ -451,8 +475,12 @@ sub compute {\n \t\t\t;\n \t\t}\n \t} else {\n+\t\t# some else process is (re)generating cache\n \t\tif (!defined $data) {\n-\t\t\t# get readers lock if there is no stale data to serve\n+\t\t\t# there is no stale data to serve\n+\t\t\t# provide \"generating page...\" info\n+\t\t\t$self->generating_info($p_key, $lock_fh);\n+\t\t\t# get readers lock\n \t\t\tflock($lock_fh, LOCK_SH);\n \t\t\t$data = $self->restore($self->get_namespace(), $p_key);\n \t\t}\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 0394dc8..cb46b8c 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -13,7 +13,7 @@ use CGI qw(:standard :escapeHTML -nosticky);\n use CGI::Util qw(unescape);\n use CGI::Carp qw(fatalsToBrowser);\n use Encode;\n-use Fcntl ':mode';\n+use Fcntl qw(:mode :flock);\n use File::Find qw();\n use File::Basename qw(basename);\n binmode STDOUT, ':utf8';\n@@ -1023,6 +1023,7 @@ if ($caching_enabled) {\n \t\t'cache_root'  => '/tmp/cache',\n \t\t'cache_depth' => 2,\n \t\t'expires_in'  => 20, # in seconds\n+\t\t'generating_info' => \\&git_generating_data_html,\n \t});\n \tcache_fetch($cache, $action);\n } else {\n@@ -3215,6 +3216,78 @@ sub blob_contenttype {\n ## ======================================================================\n ## functions printing HTML: header, footer, error page\n \n+sub git_generating_data_html {\n+\tmy ($cache, $key, $lock_fh) = @_;\n+\n+\tif ($action eq 'atom' || $action eq 'rss' || $action eq 'opml' || # feeds\n+\t    $action eq 'blob_plain' || # unknown mimetype\n+\t    $action eq 'commitdiff_plain' || # text/plain\n+\t    $action eq 'patch' || $action eq 'patches' || # text/plain\n+\t    $action eq 'snapshot') { # binary\n+\t\treturn;\n+\t}\n+\n+\tmy $title = \"[Generating...] $site_name\";\n+\tif (defined $project) {\n+\t\t$title .= \" - \" . to_utf8($project);\n+\t\tif (defined $action) {\n+\t\t\t$title .= \"/$action\";\n+\t\t\tif (defined $file_name) {\n+\t\t\t\t$title .= \" - \" . esc_path($file_name);\n+\t\t\t\tif ($action eq \"tree\" && $file_name !~ m|/$|) {\n+\t\t\t\t\t$title .= \"/\";\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tmy $mod_perl_version = $ENV{'MOD_PERL'} ? \" $ENV{'MOD_PERL'}\" : '';\n+\tprint STDOUT $cgi->header(-type => 'text/html', -charset => 'utf-8',\n+\t                          -status=> '200 OK', -expires => 'now');\n+\tprint STDOUT <<\"EOF\";\n+<?xml version=\"1.0\" encoding=\"utf-8\"?>\n+<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"\n+                      \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\n+<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en-US\" lang=\"en-US\">\n+<!-- git web interface version $version -->\n+<!-- git core binaries version $git_version -->\n+<head>\n+<meta http-equiv=\"content-type\" content=\"text/html; charset=utf-8\" />\n+<meta http-equiv=\"refresh\" content=\"0\" />\n+<meta name=\"generator\" content=\"gitweb/$version git/$git_version$mod_perl_version\" />\n+<meta name=\"robots\" content=\"noindex, nofollow\" />\n+<title>$title</title>\n+</head>\n+<body>\n+EOF\n+\tprint STDOUT 'Generating..';\n+\n+\tmy $ready;\n+\tmy ($wait, $wait_step, $wait_max); # in seconds\n+\t$wait = 0;\n+\t$wait_step = 2;\n+\t$wait_max = 20;\n+\t$| = 1; # autoflush\n+\tdo {\n+\t\tprint STDOUT '.';\n+\n+\t\tsleep $wait_step if ($wait > 0);\n+\t\t$wait += $wait_step;\n+\n+\t\t$ready = flock($lock_fh, LOCK_SH|LOCK_NB)\n+\t\t\tif $lock_fh;\n+\n+\t} while (!$ready && ($wait < $wait_max));\n+\n+\tprint STDOUT <<\"EOF\";\n+\n+</body>\n+</html>\n+EOF\n+\t#exit 0;\n+\treturn;\n+}\n+\n sub git_header_html {\n \tmy $status = shift || \"200 OK\";\n \tmy $expires = shift;\n-- \n1.6.6.1\n"},{"id":"132448","messageId":"201001230148.19770.jnareb@gmail.com","threadId":"22216","inReplyTo":"0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557.1264198194.git.jnareb@gmail.com","subject":"Re: [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing for caching)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T00:48:18Z","receivedAt":"2010-01-23T00:48:18Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"As you can (or rather can't ;-)) see this patch didn't made it into list,\nbecause with 119,993 characters in format-patch patch it probably exceeds\na little bit exceeding 100,000 characters message size limit on VGER.\n\nThe problem is that it doesn't make sense to send partial patch... well,\nperhaps reindent and breaking of exceedingly long lines should be split\ninto separate patch...\n\nFor now you can view the patch via gitweb\n  http://repo.or.cz/w/git/jnareb-git.git/commitdiff/0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557\nand apply it from\n  http://repo.or.cz/w/git/jnareb-git.git/patch/0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557\n\nThe comment for this message (for this patch) can be seen below.\n\n-- >8 --\nOn Sat, 23 Jan 2010, Jakub Narebski wrote:\n\n> This means replacing\n> \n>   print <something>;\n> by\n>   print {$out} <something>;\n> \n> and\n> \n>   binmode STDOUT, <layer>;\n> by\n>   binmode $out, <layer>;\n> \n> where $out is global variable set to \\*STDOUT at the beginning of\n> gitweb, but after reading gitweb config.  This way it would be simple\n> to e.g. tie output filehandle or use PerlIO layers to simultaneously\n> write to standard output and to some specified file (like \"tee\"\n> utility does), or redirect output to a scalar, or a file.\n> \n> die_error (re)sets $out to \\*STDOUT; we would (probably) want to treat\n> errors in a special way, and do not cache them.\n> \n> \n> The only other differences are reindent of continued lines (if\n> needed), and sometimes word-wrapping lines which this change made too\n> long.\n> \n> Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n> ---\n> This patch is meant as (straight) replacement for the following patch\n> by J.H. (John 'Warthog9' Hawley):\n> * [PATCH 8/9] gitweb: Convert output to using indirect file handle\n>   Message-ID: <1263432185-21334-9-git-send-email-warthog9@eaglescrag.net>\n>   http://permalink.gmane.org/gmane.comp.version-control.git/136915\n> \n> Actually this patch precedes (was written before) the patch by J.H.\n> \n> This patch was written _before_ comment from Junio that it would be\n> better to simply use\n>   print $out <something>;\n> and do not try to be too clever.\n> \n> \n> Differences from patch by J.H.:\n> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> * Only one output handle, instead of having one output handle for text,\n>   (':utf8'), and another output handler for binary files (:raw).  \n> \n>   I do assume that I can write to handler with appropriate layer:\n>   :utf8/:raw, and then I can simply read from cache file in :raw\n>   binmode, as the data is already converted correctly.\n> \n> * Shorter name for output handle: $out instead of $output_handler\n> \n> * Set output handler to \\*STDOUT (default value in declaration of this\n>   variable, and also in die_error() subroutine), instead of *STDOUT.\n>   This way $out is indirect filehandle, instead of using direct\n>   filehandle which is _global_ to current package (see perlopentut(1)).\n> \n> * Reindent continuation lines, i.e.\n> \n>     print <line1> .\n>           <line2>;\n> \n>   got replaced (reindented) with\n> \n>     print {$out} <line1> .\n>                  <line2>;\n> \n>   In some places lines were broken into two, when after indent the\n>   line got too long.\n> \n> * Slightly different replacement for printf\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132475","messageId":"m3r5phaxd1.fsf@localhost.localdomain","threadId":"22216","inReplyTo":"1263432185-21334-6-git-send-email-warthog9@eaglescrag.net","subject":"Re: [PATCH 5/9] gitweb: add a get function to compliment print_local_time","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-23T11:13:10Z","receivedAt":"2010-01-23T11:13:10Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"John 'Warthog9' Hawley\" <warthog9@eaglescrag.net> writes:\n\nShould authorship be associated with eaglescrag.net or kernel.org?\n\n> This adds a get function for print_local_time so that the basic\n> function can be used outside of their straight printing operation.\n\nSignoff?\n> ---\n>  gitweb/gitweb.perl |   11 +++++++++--\n>  1 files changed, 9 insertions(+), 2 deletions(-)\n> \n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index b41bc33..d38aad6 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -3537,14 +3537,21 @@ sub git_print_header_div {\n>  }\n>  \n>  sub print_local_time {\n> +\tprint get_local_time(@_);\n> +}\n> +\n> +sub get_local_time {\n\nThis new function should be rather named 'format_local_time', to\nfollow gitweb convention of naming subroutines.  (And of course commit\nmessage needs to be changed accordingly.)\n\nThis comment applies also to the following patch.\n\n> +\tmy $localtime = \"\";\n>  \tmy %date = @_;\n>  \tif ($date{'hour_local'} < 6) {\n> -\t\tprintf(\" (<span class=\\\"atnight\\\">%02d:%02d</span> %s)\",\n> +\t\t$localtime .= sprintf(\" (<span class=\\\"atnight\\\">%02d:%02d</span> %s)\",\n>  \t\t\t$date{'hour_local'}, $date{'minute_local'}, $date{'tz_local'});\n>  \t} else {\n> -\t\tprintf(\" (%02d:%02d %s)\",\n> +\t\t$localtime .= sprintf(\" (%02d:%02d %s)\",\n>  \t\t\t$date{'hour_local'}, $date{'minute_local'}, $date{'tz_local'});\n>  \t}\n> +\n> +\treturn $localtime\n\nStyle (terminate statement with ';'): \n\n  +\treturn $localtime;\n\n>  }\n>  \n>  # Outputs the author name and date in long form\n> -- \n> 1.6.5.2\n> \n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"132496","messageId":"4B5B5431.6050808@eaglescrag.net","threadId":"22216","inReplyTo":"cover.1264198194.git.jnareb@gmail.com","subject":"Re: [RFC PATCH 00/10] gitweb: Simple file based output caching","fromName":"J.H.","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-23T19:55:29Z","receivedAt":"2010-01-23T19:55:29Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"Just a heads up I wasn't able to get to these yesterday (Friday), but\nI'll try and work in looking these over come Monday.\n\n- John 'Warthog9' Hawley\n\nOn 01/22/2010 04:27 PM, Jakub Narebski wrote:\n> This 10 patches long patch series is intended as proof of concept\n> for splitting large 'gitweb: File based caching layer (from git.kernel.org)'\n> mega-patch by John 'Warthog9' Hawley aka J.H., by starting small and\n> adding features piece by piece.\n> \n> This patch is meant as replacement for last two patches:\n> * [PATCH 8/9] gitweb: Convert output to using indirect file handle\n>   Message-ID: <1263432185-21334-9-git-send-email-warthog9@eaglescrag.net>\n> * [PATCH 9/9] gitweb: File based caching layer (from git.kernel.org)\n>   Message-ID: <1263432185-21334-10-git-send-email-warthog9@eaglescrag.net>\n> \n> in the long patch series by J.H.\n> * [PATCH 0/9] Gitweb caching v5\n>   http://thread.gmane.org/gmane.comp.version-control.git/136913\n> \n> Note that this patch series is part of 'gitweb/cache-kernel' branch of\n> http://repo.or.cz/w/git/jnareb-git.git repository (gitweb link), built\n> on top of modified patches from 'Gitweb caching v2' series (from\n> 'gitweb-ml-v2' branch of http://git.kernel.org/?p=git/warthog9/gitweb.git\n> repository).  Therefore they might not apply as straight replacements\n> on top of early parts of 'gitweb-ml-v5' branch.\n> \n> This is work in progress (showing how I see introducing output caching\n> to gitweb), it lacks proper documentation (POD for gitweb/cache.pm,\n> new configuration variables in gitweb/README, perhaps \"Gitweb caching\"\n> section in gitweb/README and gitweb/cache.pm mentioned in gitweb/INSTALL),\n> and commits/patches marked '(WIP)' lacks proper commit message.\n> \n> Just food for thought...\n> \n> Table of contents:\n> ~~~~~~~~~~~~~~~~~~\n>  [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing\n>                    for caching)\n>  [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)\n>  [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching\n>  [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration\n>  [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set)\n>                    output caching (WIP)\n>  [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)\n>  [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)\n>  [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd'\n>                    problem (WIP)\n>  [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for\n>                    filling cache (WIP)\n>  [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n>                    regenerating cache (WIP)\n> \n> \n> Diffstat:\n> ~~~~~~~~~\n> \n>  gitweb/cache.pm                        |  566 ++++++++++\n>  gitweb/gitweb.perl                     | 1923 +++++++++++++++++---------------\n>  t/gitweb-lib.sh                        |    2 +\n>  t/t9500-gitweb-standalone-no-errors.sh |   13 +\n>  t/t9503-gitweb-caching.sh              |   32 +\n>  t/t9503/test_cache_interface.pl        |  195 ++++\n>  t/test-lib.sh                          |    3 +\n>  7 files changed, 1836 insertions(+), 898 deletions(-)\n>  create mode 100644 gitweb/cache.pm\n>  create mode 100755 t/t9503-gitweb-caching.sh\n>  create mode 100755 t/t9503/test_cache_interface.pl\n"},{"id":"132537","messageId":"1264341290-11356-1-git-send-email-jnareb@gmail.com","threadId":"22216","inReplyTo":"cover.1264198194.git.jnareb@gmail.com","subject":"[RFC PATCH 11/10] gitweb: Ajax-y \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-24T13:54:50Z","receivedAt":"2010-01-24T13:54:50Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nThis patch, as you can see, lack proper commit message: it is work in\nprogress (and very much proof of concept patch).\n\nThis is Ajax-y JavaScript-based activity indicator.  It is very\nbare-bones, without any error checking or fallback in the case we\nmisdetected and JavaScript is turned off.  Also currently it would\nwork only for subsequent links from gitweb, with 'js=1' added.\n\nWe need to mark Ajax request to _not_ create \"Generating...\" page.\nI have used there custon 'X-Ajax' HTTP header; we could use \n'Pragma: wait', or ejen additional query parameter for that.\n\n gitweb/gitweb.perl |   69 +++++++++++++++++++++++++++++++++++++--------------\n 1 files changed, 50 insertions(+), 19 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex cb46b8c..d422a4b 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -3219,6 +3219,9 @@ sub blob_contenttype {\n sub git_generating_data_html {\n \tmy ($cache, $key, $lock_fh) = @_;\n \n+\tif ($cgi->http('X-Ajax')) {\n+\t\treturn;\n+\t}\n \tif ($action eq 'atom' || $action eq 'rss' || $action eq 'opml' || # feeds\n \t    $action eq 'blob_plain' || # unknown mimetype\n \t    $action eq 'commitdiff_plain' || # text/plain\n@@ -3241,6 +3244,7 @@ sub git_generating_data_html {\n \t\t}\n \t}\n \n+\tmy $use_javascript = $cgi->param('js');\n \tmy $mod_perl_version = $ENV{'MOD_PERL'} ? \" $ENV{'MOD_PERL'}\" : '';\n \tprint STDOUT $cgi->header(-type => 'text/html', -charset => 'utf-8',\n \t                          -status=> '200 OK', -expires => 'now');\n@@ -3253,31 +3257,58 @@ sub git_generating_data_html {\n <!-- git core binaries version $git_version -->\n <head>\n <meta http-equiv=\"content-type\" content=\"text/html; charset=utf-8\" />\n-<meta http-equiv=\"refresh\" content=\"0\" />\n+EOF\n+\tif (!$use_javascript) {\n+\t\tprint STDOUT qq!<meta http-equiv=\"refresh\" content=\"0\" />\\n!;\n+\t}\n+\tprint STDOUT <<\"EOF\";\n <meta name=\"generator\" content=\"gitweb/$version git/$git_version$mod_perl_version\" />\n <meta name=\"robots\" content=\"noindex, nofollow\" />\n <title>$title</title>\n </head>\n <body>\n EOF\n-\tprint STDOUT 'Generating..';\n-\n-\tmy $ready;\n-\tmy ($wait, $wait_step, $wait_max); # in seconds\n-\t$wait = 0;\n-\t$wait_step = 2;\n-\t$wait_max = 20;\n-\t$| = 1; # autoflush\n-\tdo {\n-\t\tprint STDOUT '.';\n-\n-\t\tsleep $wait_step if ($wait > 0);\n-\t\t$wait += $wait_step;\n-\n-\t\t$ready = flock($lock_fh, LOCK_SH|LOCK_NB)\n-\t\t\tif $lock_fh;\n-\n-\t} while (!$ready && ($wait < $wait_max));\n+\tif ($use_javascript) {\n+\t\tmy $url_self = href(-replay=>1);\n+\t\tprint STDOUT <<\"EOF\";\n+Generating...<br />\n+<script type=\"text/javascript\" src=\"$javascript\"></script>\n+<script type=\"text/javascript\">\n+window.onload = function () {\n+\txhr = createRequestObject();\n+\txhr.onreadystatechange = function () {\n+\t\tif (xhr.readyState === 4 && xhr.status === 200) {\n+\t\t\t// intentionally overwrite document\n+\t\t\tdocument.write(xhr.responseText);\n+\t\t}\n+\t}\n+\txhr.open('GET', $url_self);\n+\t// mark request to not show \"Generating...\" page\n+\txhr.setRequestHeader('X-Ajax', 'Yes');\n+\txhr.send(null);\n+}\n+</script>\n+EOF\n+\t} else {\n+\t\tprint STDOUT 'Generating..';\n+\n+\t\tmy $ready;\n+\t\tmy ($wait, $wait_step, $wait_max); # in seconds\n+\t\t$wait = 0;\n+\t\t$wait_step = 2;\n+\t\t$wait_max = 20;\n+\t\t$| = 1; # autoflush\n+\t\tdo {\n+\t\t\tprint STDOUT '.';\n+\t\t\t\n+\t\t\tsleep $wait_step if ($wait > 0);\n+\t\t\t$wait += $wait_step;\n+\t\t\t\n+\t\t\t$ready = flock($lock_fh, LOCK_SH|LOCK_NB)\n+\t\t\t\tif $lock_fh;\n+\t\t\t\n+\t\t} while (!$ready && ($wait < $wait_max));\n+\t}\n \n \tprint STDOUT <<\"EOF\";\n \n-- \n1.6.6.1\n"},{"id":"132562","messageId":"20100124215924.GA9553@machine.or.cz","threadId":"22216","inReplyTo":"1263432185-21334-4-git-send-email-warthog9@eaglescrag.net","subject":"Re: [PATCH 3/9] gitweb: Add option to force version match","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2010-01-24T21:59:24Z","receivedAt":"2010-01-24T21:59:24Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"  Hi!\n\nOn Wed, Jan 13, 2010 at 05:22:59PM -0800, John 'Warthog9' Hawley wrote:\n> +# Throw an error if git versions does not match, if $git_versions_must_match is true.\n> +if ($git_versions_must_match &&\n> +    $git_version ne $version) {\n> +\tmy $admin_contact =\n> +\t\tdefined $ENV{'SERVER_ADMIN'} ? \", $ENV{'SERVER_ADMIN'},\" : '';\n> +\tmy $err_msg = <<EOT;\n> +<h1 align=\"center\">*** Warning ***</h1>\n> +<p>\n> +This version of gitweb was compiled for <b>@{[esc_html($version)]}</b>,\n> +however git version <b>@{[esc_html($git_version)]}</b> was found on server,\n> +and administrator requested strict version checking.\n> +</p>\n> +<p>\n> +Please contact the server administrator${admin_contact} to either configure\n> +gitweb to allow mismatched versions, or update git or gitweb installation.\n> +</p>\n> +EOT\n> +\tdie_error(500, 'Internal server error', $err_msg);\n> +}\n> +\n\n  I get very unhappy when *I* am the server administrator and read\nmessages like this. ;-) Could you please mention the\n\\$git_versions_must_match in the text?\n\n\t\t\t\tPetr \"Pasky\" Baudis\n"},{"id":"132563","messageId":"20100124221444.GB9553@machine.or.cz","threadId":"22216","inReplyTo":"m3ljfydgmt.fsf@localhost.localdomain","subject":"Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2010-01-24T22:14:44Z","receivedAt":"2010-01-24T22:14:44Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Fri, Jan 15, 2010 at 04:43:32PM -0800, Jakub Narebski wrote:\n> Third, wouldn't it be better to use shorter variable name, e.g. $out\n> or $oh, instead of $output_handle?  We would be able to align print(f)\n> statements without making lines much longer.\n\nI totally agree, I was going to comment on this too. Please use\nsomething very short, I think even the three characters might be pushing\nit a bit. Having long idiom will make it real pain to both read and\nwrite gitweb code.\n\nIt would still be nice if we could have some way to avoid this hack\nentirely. Couldn't we layer PerlIO::via over STDOUT and continue to use\nargument-less print as before?\n\nI think we (well, in practice, \"you\" nowadays :) should be less\nperfectionist when accepting gitweb contributions, so if the answer is\n\"it's too hard for me to do right now\", that's fine I guess; I know I\ncan't contribute the code currently. :(\n\n\t\t\t\tPetr \"Pasky\" Baudis\n"},{"id":"132564","messageId":"20100124222417.GC9553@machine.or.cz","threadId":"22216","inReplyTo":"f4660e9c7ffdb4a62da0c56703de002c9df3b598.1264198194.git.jnareb@gmail.com","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2010-01-24T22:24:17Z","receivedAt":"2010-01-24T22:24:17Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"  I have stupid question, common to both the original patch and this\nRFC.\n\n> [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n> regenerating cache (WIP)\n\n  Just why is a \"Generating...\" page appropriate?\n\n  I have to admit I hate it; can you please at least make it\nconfigurable? Why is it needed at all? It generates unnecessary\nredirects, clears my previous page in the browser by a useless\nplaceholder, confuses non-interactive HTTP clients and I just can't\nsee the real purpose.\n\n  What's the gain?\n\n  Thanks,\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nA lot of people have my books on their bookshelves.\nThat's the problem, they need to read them. -- Don Knuth\n"},{"id":"132565","messageId":"m3iqarayar.fsf@localhost.localdomain","threadId":"22216","inReplyTo":"20100124215924.GA9553@machine.or.cz","subject":"Re: [PATCH 3/9] gitweb: Add option to force version match","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-24T23:17:56Z","receivedAt":"2010-01-24T23:17:56Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Petr Baudis <pasky@suse.cz> writes:\n\n> On Wed, Jan 13, 2010 at 05:22:59PM -0800, John 'Warthog9' Hawley wrote:\n\n> > +# Throw an error if git versions does not match, if $git_versions_must_match is true.\n> > +if ($git_versions_must_match &&\n> > +    $git_version ne $version) {\n> > +\tmy $admin_contact =\n> > +\t\tdefined $ENV{'SERVER_ADMIN'} ? \", $ENV{'SERVER_ADMIN'},\" : '';\n> > +\tmy $err_msg = <<EOT;\n> > +<h1 align=\"center\">*** Warning ***</h1>\n> > +<p>\n> > +This version of gitweb was compiled for <b>@{[esc_html($version)]}</b>,\n> > +however git version <b>@{[esc_html($git_version)]}</b> was found on server,\n> > +and administrator requested strict version checking.\n\n+and administrator requested strict version checking by setting\n+\\$git_versions_must_match to false value (e.g. to 0) in gitweb configuration file,\n+'@{[esc_path(-e $GITWEB_CONFIG ? $GITWEB_CONFIG : $GITWEB_CONFIG_SYSTEM)]}'.\n\n> > +</p>\n> > +<p>\n> > +Please contact the server administrator${admin_contact} to either configure\n> > +gitweb to allow mismatched versions, or update git or gitweb installation.\n> > +</p>\n> > +EOT\n> > +\tdie_error(500, 'Internal server error', $err_msg);\n> > +}\n> > +\n> \n>   I get very unhappy when *I* am the server administrator and read\n> messages like this. ;-) Could you please mention the\n> \\$git_versions_must_match in the text?\n\nSomething like the text (extension) proposed above?\n\n\nActually, J.H., if you are changing the default, you should also change the\nmessage to mention that checking versions is the default, and administrator\ncan turn it off[1].  It is incorrect if $git_versions_must_match defaults\nto true.\n\n[1] And you have to take into account that gitweb config file can not\nexist, and neither $GITWEB_CONFIG nor $GITWEB_CONFIG_SYSTEM needs to be\nnon-empty.  J.H., are you really, really sure that it is a good idea to\nhave version check turned on by default, in light of this issue?\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"132569","messageId":"201001250103.38703.jnareb@gmail.com","threadId":"22216","inReplyTo":"20100124222417.GC9553@machine.or.cz","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-25T00:03:37Z","receivedAt":"2010-01-25T00:03:37Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sun, 24 Jan 2010, Petr Baudis wrote:\n\n>   I have stupid question, common to both the original patch and this\n> RFC.\n> \n> > [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n> > regenerating cache (WIP)\n> \n>   Just why is a \"Generating...\" page appropriate?\n> \n>   I have to admit I hate it; can you please at least make it\n> configurable? Why is it needed at all? It generates unnecessary\n> redirects, clears my previous page in the browser by a useless\n> placeholder, confuses non-interactive HTTP clients and I just can't\n> see the real purpose.\n> \n>   What's the gain?\n\nDon't ask me, I am just following original patch by John 'Warthog9' \nHawley (J.H.) here, splitting it into smaller and hopefully easier to \nreview parts, with each separate feature in a separate patch (so that \nfeatures like this one are more visible).  I _think_ that in original \npatch by J.H. this feature was configurable (at least to some extent).  \nAs this series is more proof of concept than ready to apply patches, \nsome of configurability is missing currently.\n\nI think that adding the \"Generating...\" page follows behavior of other \nwell known web applications, such as for example GMail.  Currently\nif caching is enabled, and there is no appropriate entry in cache (not\neven [slightly] stale entry) the page is shown only when it is generated\nin full: there is no streaming (although I planned to add it).  So for\nexample for 'blame' view or for pickaxe search you could wait quite\na bit for a page to be shown.  The \"Generating...\" page is a kind of \nactivity indicator here.\n\nNote that the JavaScript-using Ajax-y version from the following patch \n'[RFC PATCH 11/10] gitweb: Ajax-y \"Generating...\" page when \nregenerating cache (WIP)'\n  http://permalink.gmane.org/gmane.comp.version-control.git/137896\nis not visible in browser history, as it simply replaces \"Generating...\"\npage with the result.  So there is no 'unnecessary redirect', and there\nis no 'useless placeholder' in browser history.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132573","messageId":"201001250217.17743.jnareb@gmail.com","threadId":"22216","inReplyTo":"201001250103.38703.jnareb@gmail.com","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-25T01:17:17Z","receivedAt":"2010-01-25T01:17:17Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jakub Narebski wrote:\n> On Sun, 24 Jan 2010, Petr Baudis wrote:\n> \n> >   I have stupid question, common to both the original patch and this\n> > RFC.\n> > \n> > > [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n> > > regenerating cache (WIP)\n> > \n> >   Just why is a \"Generating...\" page appropriate?\n> > \n> >   I have to admit I hate it; can you please at least make it\n> > configurable? Why is it needed at all? It generates unnecessary\n> > redirects, clears my previous page in the browser by a useless\n> > placeholder, confuses non-interactive HTTP clients and I just can't\n> > see the real purpose.\n\n> Note that the JavaScript-using Ajax-y version from the following patch \n> '[RFC PATCH 11/10] gitweb: Ajax-y \"Generating...\" page when \n> regenerating cache (WIP)'\n>   http://permalink.gmane.org/gmane.comp.version-control.git/137896\n> is not visible in browser history, as it simply replaces \"Generating...\"\n> page with the result.  So there is no 'unnecessary redirect', and there\n> is no 'useless placeholder' in browser history.\n\nActually I have checked and the version used on git.kernel.org does show\n\"Generating...\" page, but it does not appear in the history.  I guess\nthat refresh with no URL does not count as redirect.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132574","messageId":"201001250247.13101.jnareb@gmail.com","threadId":"22216","inReplyTo":"20100124221444.GB9553@machine.or.cz","subject":"Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-25T01:47:12Z","receivedAt":"2010-01-25T01:47:12Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sun, Jan 24, 2010 at 23:14 +0100, Petr Baudis wrote:\n> On Fri, Jan 15, 2010 at 04:43:32PM -0800, Jakub Narebski wrote:\n\n> > Third, wouldn't it be better to use shorter variable name, e.g. $out\n> > or $oh, instead of $output_handle?  We would be able to align print(f)\n> > statements without making lines much longer.\n> \n> I totally agree, I was going to comment on this too. Please use\n> something very short, I think even the three characters might be pushing\n> it a bit. Having long idiom will make it real pain to both read and\n> write gitweb code.\n> \n> It would still be nice if we could have some way to avoid this hack\n> entirely. Couldn't we layer PerlIO::via over STDOUT and continue to use\n> argument-less print as before?\n\nFirst, the original version in v2 series, the one that can be found in\n42641b1 (GITWEB - File based caching layer, 2009-12-10) commit in\ngit://git.kernel.org/pub/scm/git/warthog9/gitweb.git gitweb-ml-v2\nhttp://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v2\nthe change was even larger. It was\n  print <something>;\nreplaced by\n  $output .= <something>;\nand of course make functions that printed sth return formatted output\ninstead.\n\nAlso with original version you paid the price even if caching was turned\noff (not used): the whole output had to fit in memory, and there was no\nstreaming, which meant that the whole output had to be generated before\nit could be displayed (decreasing latency).\n\n\nSecond, PerlIO::via is not a good solution.  One solution would be to\nrequire Capture::Tiny if caching is enabled, and just do\n\n  $out = capture {\n  \t$actions{$action}->();\n  }\n\nor even simply (if it is possible with the way 'capture' is prototyped)\n\n  $out = capture $actions{$action};\n\nAnother solution would be to use PerlIO::Util to redirect output to\nscalar:\n\n  *STDOUT->push_layer(scalar => \\$buffer);\n  # ...\n  *STDOUT->pop_layer();\n  \nYet another solution would be to manipulate *STDOUT directly\n\n  $stdout = *STDOUT;\n  *STDOUT = $buffer_fh;\n  ...\n  *STDOUT = $stdout;\n\nfor which, I think, to be done correctly we would have to duplicate\n(large?) parts of Capture::Tiny.\n\n\nSo there is a bit of rule of preservation of difficulty at work.  Either\nwe have large patch adding explicit filehandle to all print statements\n'print <sth> -> print $out <sth>' but simple code, or have smaller patch\nbut complicated *STDOUT manipulation, or have small patch but rely on \nnon-core CPAN modules present.\n\n> I think we (well, in practice, \"you\" nowadays :) should be less\n> perfectionist when accepting gitweb contributions, so if the answer is\n> \"it's too hard for me to do right now\", that's fine I guess; I know I\n> can't contribute the code currently. :(\n\nTrue.  At least two series ('categories' which I have copy locally\nin http://repo.or.cz/w/git/jnareb-git.git as gitweb/category, and\n'committags' by Marcel M. Cary[1]) that are not yet in gitweb because\nof that...\n\n[1]: http://thread.gmane.org/gmane.comp.version-control.git/100418/focus=133141\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132595","messageId":"201001251246.40237.jnareb@gmail.com","threadId":"22216","inReplyTo":"20100124222417.GC9553@machine.or.cz","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-25T11:46:39Z","receivedAt":"2010-01-25T11:46:39Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sun, 24 Jan 2010, Petr Baudis wrote:\n>   I have stupid question, common to both the original patch and this\n> RFC.\n> \n> > [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n> > regenerating cache (WIP)\n> \n>   Just why is a \"Generating...\" page appropriate?\n> \n>   I have to admit I hate it; can you please at least make it\n> configurable? Why is it needed at all? It [...] confuses\n> non-interactive HTTP clients [...]\n\nFirst, if I understand the code correctly HTTP clients which do not \nhonor metaredirect (http-equiv refresh) would get page which looks\nthe following\n\n  <html>\n  Generating...\n  </html>\n  <html>\n  Gitweb page\n  </html>\n\n\nSecond, gitweb can always check User-Agent header, and serve \n\"Generating...\" page only to web browsers:\n\n  unless (defined $cgi->user_agent() &&\n          $cgi->user_agent() =~ /\\b(Mozilla|Opera)\\b/i) {\n  \treturn;\n  }\n\nor something like that.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132599","messageId":"20100125130230.GE9553@machine.or.cz","threadId":"22216","inReplyTo":"201001251246.40237.jnareb@gmail.com","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2010-01-25T13:02:30Z","receivedAt":"2010-01-25T13:02:30Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Mon, Jan 25, 2010 at 12:46:39PM +0100, Jakub Narebski wrote:\n> On Sun, 24 Jan 2010, Petr Baudis wrote:\n> >   I have stupid question, common to both the original patch and this\n> > RFC.\n> > \n> > > [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n> > > regenerating cache (WIP)\n> > \n> >   Just why is a \"Generating...\" page appropriate?\n> > \n> >   I have to admit I hate it; can you please at least make it\n> > configurable? Why is it needed at all? It [...] confuses\n> > non-interactive HTTP clients [...]\n> \n> First, if I understand the code correctly HTTP clients which do not \n> honor metaredirect (http-equiv refresh) would get page which looks\n> the following\n> \n>   <html>\n>   Generating...\n>   </html>\n>   <html>\n>   Gitweb page\n>   </html>\n\nTo be clear, I conjectured it confuses non-interactive HTTP clients from\none of the TODOs in your patch series - I'm not sure about it myself. :)\n\n> Second, gitweb can always check User-Agent header, and serve \n> \"Generating...\" page only to web browsers:\n> \n>   unless (defined $cgi->user_agent() &&\n>           $cgi->user_agent() =~ /\\b(Mozilla|Opera)\\b/i) {\n>   \treturn;\n>   }\n> \n> or something like that.\n\nI'm not too happy with this. What about Safari? Opera? ELinks? There's a\nlot of web browsers.\n\nMost of the issues can be worked around, but I'm not sure why to go\nthrough all the trouble. I just personally don't see the value in having\nthe placeholder in there at all, to me it is distracting UI even if all\nthe technicalities are put aside.\n\nBut if it will be possible to turn this off eventually, it's all your\ncall whether to bother implementing it. :-)\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nA lot of people have my books on their bookshelves.\nThat's the problem, they need to read them. -- Don Knuth\n"},{"id":"132603","messageId":"201001251448.27513.jnareb@gmail.com","threadId":"22216","inReplyTo":"20100125130230.GE9553@machine.or.cz","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-25T13:48:26Z","receivedAt":"2010-01-25T13:48:26Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, Jan 25, 2010, Petr Baudis wrote:\n> On Mon, Jan 25, 2010 at 12:46:39PM +0100, Jakub Narebski wrote:\n>> On Sun, 24 Jan 2010, Petr Baudis wrote:\n\n>>>   I have stupid question, common to both the original patch and this\n>>> RFC.\n>>> \n>>>> [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n>>>> regenerating cache (WIP)\n>>> \n>>>   Just why is a \"Generating...\" page appropriate?\n>>> \n>>>   I have to admit I hate it; can you please at least make it\n>>> configurable? Why is it needed at all? It [...] confuses\n>>> non-interactive HTTP clients [...]\n\n>> Second, gitweb can always check User-Agent header, and serve \n>> \"Generating...\" page only to web browsers:\n>> \n>>   unless (defined $cgi->user_agent() &&\n>>           $cgi->user_agent() =~ /\\b(Mozilla|Opera)\\b/i) {\n>>   \treturn;\n>>   }\n>> \n>> or something like that.\n> \n> I'm not too happy with this. What about Safari? Opera? ELinks? There's a\n> lot of web browsers.\n\nThe \"Mozilla\" part would catch all \"Mozilla compatibile\" web browsers,\nincluding Firefox (and other Gecko-based web browsers), Internet Explorer,\nWebKit based browsers including Safari and Chrome and Konqueror.\nThe \"Opera\" part would catch Opera.\nhttp://www.nczonline.net/blog/2010/01/12/history-of-the-user-agent-string/\n\nAs to other web browsers like Elinks, Lynx, w3m, Dillo, etc.: the issue\nis whether they honor '<meta http-equiv=\"refresh\" content=\"0\" />'.  \nI think it is better to stay on the safe side; it is not disaster if web\nbrowser is not shown \"Generating...\" page where it could (but see \nexplanation below).\n \n> Most of the issues can be worked around, but I'm not sure why to go\n> through all the trouble. I just personally don't see the value in having\n> the placeholder in there at all, to me it is distracting UI even if all\n> the technicalities are put aside.\n\nThe issue that \"Generating...\" page tries to solve is, I think, the \nfollowing.  \n\nSome actions, like 'blame' view or pickaxe search, or grep search,\ncan take quite a long time to generate, with times counted in\ntens of seconds.  It is not that visible for non-caching, because\ngitweb streams output so we have at least _some_ output upfront quite\nfast.\n\nNow those patches (mine and J.H. both) make gitweb use locking\n(it is IIRC configurable in J.H. patch) to make only one process\ngenerate the page if it is missing from cache, or is stale.  Now\nif it is missing, we have to wait until it is generated in full\nbefore being able to show it to client.  While it is possible to\n\"tee\" output (using PerlIO::tee, or Capture::Tiny, or tie like\nCGI::Cache) writing it simultaneously to browser and to cache for \nthe process that is generating data, it is as far as I understand\nit impossible for processes which are waiting for data.  Therefore\nthe need for \"Generating...\" page, so the user does not think that\nweb server hung or something, and is not generating output.\n \nWe can try to reduce occurrences of cache miss stampedes by using\n'expires_variance' feature[1] from CHI - Unified caching interface.\nWe can also turn off locking and tee output to have some output upfront\nas an activity indicator instead of this \"Generating...\" page.\n\n[1]: http://search.cpan.org/~jswartz/CHI-0.33/lib/CHI.pm#set\n\n> But if it will be possible to turn this off eventually, it's all your\n> call whether to bother implementing it. :-)\n\nIn my implementation it is (or rather would be) as simple as just\nnot passing 'generating_info' => \\&git_generating_data_html in the\nGitwebCache::SimpleFileCache constructor.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132604","messageId":"20100125135653.GN4159@machine.or.cz","threadId":"22216","inReplyTo":"201001251448.27513.jnareb@gmail.com","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2010-01-25T13:56:53Z","receivedAt":"2010-01-25T13:56:53Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Mon, Jan 25, 2010 at 02:48:26PM +0100, Jakub Narebski wrote:\n> Now those patches (mine and J.H. both) make gitweb use locking\n> (it is IIRC configurable in J.H. patch) to make only one process\n> generate the page if it is missing from cache, or is stale.  Now\n> if it is missing, we have to wait until it is generated in full\n> before being able to show it to client.  While it is possible to\n> \"tee\" output (using PerlIO::tee, or Capture::Tiny, or tie like\n> CGI::Cache) writing it simultaneously to browser and to cache for \n> the process that is generating data, it is as far as I understand\n> it impossible for processes which are waiting for data.  Therefore\n> the need for \"Generating...\" page, so the user does not think that\n> web server hung or something, and is not generating output.\n\nAh, ok, so the message is there to cover up for a technical problem. ;-)\nI didn't quite realize. Then, it would be great to tweak the mechanisms\nso that the user does not really have to wait.\n\nSo, I wonder about two things:\n\n(i) How often does it happen that two requests for the same page are\nreceived? Has anyone measured it? Or is at least able to make\na minimally educated guess? IOW, isn't this premature optimization?\n\n(ii) Can't the locked gitwebs do the equivalent of tail -f?\n\n\nP.S.: Again the disclaimer - if this is \"too hard\", it's better to\naccept patches like they are, then improve this later. But perhaps\na better solution would be not to clutter the code by optimizing this\ncase at all if it's not clear it really matters in the real world.\n\n\t\t\t\tPetr \"Pasky\" Baudis\n"},{"id":"132630","messageId":"4B5DFFE5.6060908@eaglescrag.net","threadId":"22216","inReplyTo":"20100125135653.GN4159@machine.or.cz","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"J.H.","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-25T20:32:37Z","receivedAt":"2010-01-25T20:32:37Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"On 01/25/2010 05:56 AM, Petr Baudis wrote:\n> On Mon, Jan 25, 2010 at 02:48:26PM +0100, Jakub Narebski wrote:\n>> Now those patches (mine and J.H. both) make gitweb use locking\n>> (it is IIRC configurable in J.H. patch) to make only one process\n>> generate the page if it is missing from cache, or is stale.  Now\n>> if it is missing, we have to wait until it is generated in full\n>> before being able to show it to client.  While it is possible to\n>> \"tee\" output (using PerlIO::tee, or Capture::Tiny, or tie like\n>> CGI::Cache) writing it simultaneously to browser and to cache for \n>> the process that is generating data, it is as far as I understand\n>> it impossible for processes which are waiting for data.  Therefore\n>> the need for \"Generating...\" page, so the user does not think that\n>> web server hung or something, and is not generating output.\n> \n> Ah, ok, so the message is there to cover up for a technical problem. ;-)\n> I didn't quite realize. Then, it would be great to tweak the mechanisms\n> so that the user does not really have to wait.\n\nNo, that is an incorrect assumption on how the 'Generating...' page\nworks, and your missing a bit of the point.\n\n(1) The message itself 'Generating...' is a que to the user that\nsomething is happening and that the browser is not actually hanging.\nWeb users are at the point where if things are not instantaneous and\nshow immediately they will either browse away completely or hit the\nrefresh button incessantly until content does appear.  While the page is\nusually only seen for about a second, and I'll admit it can be annoying,\nit's nothing more than a 'sit tight a second'.  For things like the\nfront page it can take upwards of 7 seconds to generate for a single\nuser, a lot to ask for a no response scenario.\n\n(2) It prevents the stampeding herd problem, which was very vehemently\ndiscussed 4 years ago by HPA and myself and roughly boils down to this:\n\nWhen a single user comes into the site, in particular the front page, it\nkicks off a process that will start to generate at it, causing a huge\namount of git requests into individual repositories and a lot of disk\ni/o.  A second user will then come in and the same requests will start\nto be done from the beginning again, and so on until you basically kill\nthe machine because the disk i/o goes up enough that it can't ever\nservice the requests fast enough.\n\nThis does 2 things in the end:\n\n1) means there's only 1 copy of the page ever being generated, thus\nmeaning there isn't extraneous and dangerous disk i/o going on on the system\n\n2) prevents a user from reporting to the website that it's broken by\ngiving them a visual que that things aren't broken.\n\n\n> So, I wonder about two things:\n> \n> (i) How often does it happen that two requests for the same page are\n> received? Has anyone measured it? Or is at least able to make\n> a minimally educated guess? IOW, isn't this premature optimization?\n\nFor most pages, not many but it happens more often than you think.  The\ndata I have is much too old to be useful now but the front page could,\nat times, have up to 30 people waiting for it without caching.  This is\na very important patch believe it or not.  For a site the size of\nkernel.org it cannot exist without this.\n\nBut here's a quick stat, in 36 hours git.kernel.org has had\n156099 accesses world wide or about 1.2 accesses a second.\n\nandroid.git.kernel.org, in the same time period has had 115818 accesses.\n\nIf the first request takes 7 seconds to generate, by the time it's done\nthere are now 3 additional requests running.  If it again takes 7\nseconds to generate there are now another 3 requests running, etc.  Very\nquickly you've got so much i/o running the box more or less is useless.\n\n> (ii) Can't the locked gitwebs do the equivalent of tail -f?\n\nNot really going to help much, most of the gitweb operations won't\noutput much of anything beyond the header until it's collected all of\nthe data it needs anyway and then there will be a flurry of output.  It\nalso means that this 'Generating...' page will only work for caching\nschemes that tail can read out of, which I'm not sure it would work all\nthat well with things like memcached or a non-custom caching layer where\nwe don't necessarily have direct access to the file being written to.\n\nAt least the way I had it (and I'll admit I haven't read through Jakub's\nre-working of my patches so I don't know if it's still there) is that\nwith background caching you only get the 'Generating...' page if it's\nnew or the content is grossly out of data.  If it's a popular page and\nit's not grossly out of date it shows you the 'stale' data while it\ngenerates the new content in the background anyway, only locking you out\nwhen the new file is being written.  Or at least that's how I had it.\n\n- John 'Warthog9' Hawley\n"},{"id":"132631","messageId":"4B5E01F5.20402@eaglescrag.net","threadId":"22216","inReplyTo":"201001251448.27513.jnareb@gmail.com","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"J.H.","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-25T20:41:25Z","receivedAt":"2010-01-25T20:41:25Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"On 01/25/2010 05:48 AM, Jakub Narebski wrote:\n> On Mon, Jan 25, 2010, Petr Baudis wrote:\n>> On Mon, Jan 25, 2010 at 12:46:39PM +0100, Jakub Narebski wrote:\n>>> On Sun, 24 Jan 2010, Petr Baudis wrote:\n> \n>>>>   I have stupid question, common to both the original patch and this\n>>>> RFC.\n>>>>\n>>>>> [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n>>>>> regenerating cache (WIP)\n>>>>\n>>>>   Just why is a \"Generating...\" page appropriate?\n>>>>\n>>>>   I have to admit I hate it; can you please at least make it\n>>>> configurable? Why is it needed at all? It [...] confuses\n>>>> non-interactive HTTP clients [...]\n> \n>>> Second, gitweb can always check User-Agent header, and serve \n>>> \"Generating...\" page only to web browsers:\n>>>\n>>>   unless (defined $cgi->user_agent() &&\n>>>           $cgi->user_agent() =~ /\\b(Mozilla|Opera)\\b/i) {\n>>>   \treturn;\n>>>   }\n>>>\n>>> or something like that.\n>>\n>> I'm not too happy with this. What about Safari? Opera? ELinks? There's a\n>> lot of web browsers.\n> \n> The \"Mozilla\" part would catch all \"Mozilla compatibile\" web browsers,\n> including Firefox (and other Gecko-based web browsers), Internet Explorer,\n> WebKit based browsers including Safari and Chrome and Konqueror.\n> The \"Opera\" part would catch Opera.\n> http://www.nczonline.net/blog/2010/01/12/history-of-the-user-agent-string/\n> \n> As to other web browsers like Elinks, Lynx, w3m, Dillo, etc.: the issue\n> is whether they honor '<meta http-equiv=\"refresh\" content=\"0\" />'.  \n> I think it is better to stay on the safe side; it is not disaster if web\n> browser is not shown \"Generating...\" page where it could (but see \n> explanation below).\n\nMost of them do, that particular tag has been around for a long time and\nsince it doesn't require Javascript to do the page refresh it's pretty\nmuch universal.\n\nThe problem is going to be with things like wget when someone wants to\nsnag a binary file.  This works fine if the file is already cached, but\nthe user doesn't get what they are expecting if they get a blob that\nisn't the final file, but the html contents of the page.  I don't know\nof any hint that things like wget would send to the server that you\ncould switch based on, but it would be more or less the non-background\ncaching state.\n\n>> Most of the issues can be worked around, but I'm not sure why to go\n>> through all the trouble. I just personally don't see the value in having\n>> the placeholder in there at all, to me it is distracting UI even if all\n>> the technicalities are put aside.\n> \n> The issue that \"Generating...\" page tries to solve is, I think, the \n> following.  \n> \n> Some actions, like 'blame' view or pickaxe search, or grep search,\n> can take quite a long time to generate, with times counted in\n> tens of seconds.  It is not that visible for non-caching, because\n> gitweb streams output so we have at least _some_ output upfront quite\n> fast.\n> \n> Now those patches (mine and J.H. both) make gitweb use locking\n> (it is IIRC configurable in J.H. patch) to make only one process\n> generate the page if it is missing from cache, or is stale.  Now\n> if it is missing, we have to wait until it is generated in full\n> before being able to show it to client.  While it is possible to\n> \"tee\" output (using PerlIO::tee, or Capture::Tiny, or tie like\n> CGI::Cache) writing it simultaneously to browser and to cache for \n> the process that is generating data, it is as far as I understand\n> it impossible for processes which are waiting for data.  Therefore\n> the need for \"Generating...\" page, so the user does not think that\n> web server hung or something, and is not generating output.\n>  \n> We can try to reduce occurrences of cache miss stampedes by using\n> 'expires_variance' feature[1] from CHI - Unified caching interface.\n> We can also turn off locking and tee output to have some output upfront\n> as an activity indicator instead of this \"Generating...\" page.\n> \n> [1]: http://search.cpan.org/~jswartz/CHI-0.33/lib/CHI.pm#set\n> \n>> But if it will be possible to turn this off eventually, it's all your\n>> call whether to bother implementing it. :-)\n> \n> In my implementation it is (or rather would be) as simple as just\n> not passing 'generating_info' => \\&git_generating_data_html in the\n> GitwebCache::SimpleFileCache constructor.\n> \n\nAt least in mine it was don't allow background caching.  It would force\neveryone to wait on the one process that was actually generating\ncontent.  But it means a few blank pages with a spinning working icon\nuntil the cache releases it's exclusive lock.\n\n- John 'Warthog9' Hawley\n"},{"id":"132632","messageId":"4B5E03AD.1020404@eaglescrag.net","threadId":"22216","inReplyTo":"201001250247.13101.jnareb@gmail.com","subject":"Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle","fromName":"J.H.","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-01-25T20:48:45Z","receivedAt":"2010-01-25T20:48:45Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"> So there is a bit of rule of preservation of difficulty at work.  Either\n> we have large patch adding explicit filehandle to all print statements\n> 'print <sth> -> print $out <sth>' but simple code, or have smaller patch\n> but complicated *STDOUT manipulation, or have small patch but rely on \n> non-core CPAN modules present.\n\nI think depending in non-core CPAN modules is a really bad idea, and\nwill cause some concern and consternation with mainline distributions,\nbesides making it more complicated for users to get this up and running\nquickly and easily.  While I agree there are other ways of handling this\nI think just adding the filehandle to the print statements provides the\neasiest cross section of usability and functionality for everything\ninvolved.\n\n- John 'Warthog9' Hawley\n"},{"id":"132633","messageId":"201001252158.29516.jnareb@gmail.com","threadId":"22216","inReplyTo":"20100125135653.GN4159@machine.or.cz","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-25T20:58:28Z","receivedAt":"2010-01-25T20:58:28Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, Jan 25, 2010 at 14:56 +0100, Petr Baudis wrote:\n> On Mon, Jan 25, 2010 at 02:48:26PM +0100, Jakub Narebski wrote:\n\n> > Now those patches (mine and J.H. both) make gitweb use locking\n> > (it is IIRC configurable in J.H. patch) to make only one process\n> > generate the page if it is missing from cache, or is stale.  Now\n> > if it is missing, we have to wait until it is generated in full\n> > before being able to show it to client.  While it is possible to\n> > \"tee\" output (using PerlIO::tee, or Capture::Tiny, or tie like\n> > CGI::Cache) writing it simultaneously to browser and to cache for \n> > the process that is generating data, it is as far as I understand\n> > it impossible for processes which are waiting for data.  Therefore\n> > the need for \"Generating...\" page, so the user does not think that\n> > web server hung or something, and is not generating output.\n> \n> Ah, ok, so the message is there to cover up for a technical problem. ;-)\n> I didn't quite realize. Then, it would be great to tweak the mechanisms\n> so that the user does not really have to wait.\n\nWell, the mechanism would certainly be configurable in final version\n(current split version is more of proof of concept of splitting).\n\n> \n> So, I wonder about two things:\n> \n> (i) How often does it happen that two requests for the same page are\n> received? Has anyone measured it? Or is at least able to make\n> a minimally educated guess? IOW, isn't this premature optimization?\n\nTo be more exact the question is how often second request for the same\npage appears when earlier request didn't finished processing.  It is\nthe matter of both frequency of given requests, and time it takes to\ngenerate request (which grows with growing load on server).\n\nAs to measurements: Pasky, do you have access logs, or their analysis\na la AWStats, Webalizer and the like, for repo.or.cz?  Warthog9, do\nyou have access logs or analysis for git.kernel.org?  Can you get similar\nfrom fedorahosted?\n \n> (ii) Can't the locked gitwebs do the equivalent of tail -f?\n\nWell, it could, in principle, but it would need some changes.  \n\nFirst, instead of using temporary file to create cache entry atomically\n(write to temporary file, then rename) the process generating data would\nhave to write to file other processes can read from.  It could be e.g.\nlockfile.\n\nSecond, there would be needed extended cache API so that generated data\nis streamed to cache file, ->set($key, $data) ==> ->set($key, $fh) or\n->set_io($key, $fh).  This would mean some complications, but what might\nbe more important is that this trick would not work as far as I can see\nwith other caching backends / caching engines that the one from \ngitweb/cache.pm (like memcached or mmap based ones).\n\nThen the code could look like the following (in pseudocode):\n\n  try to acquire writers lock\n  if (acquired writers lock) {\n  \tgenerate and \"tee\" response\n  \tcreate cache entry\n  } else {\n  \t# <<<<<<\n  \twhile (not acquired writers lock &&\n  \t       sysread something) {\n  \t\tprint <data>;\n  \t}\n  \t# >>>>>>\n  \tretrieve and print (rest) of data\n  }\n\nwhere parts between <<<<<< and >>>>>> are new.\n\nBut there is another complication: gitweb needs to be able to deal with\nthe situation where process generating data got interrupted before \ncreating full output, or process generating data ran die_error which\ndoes not generate any cache entry (e.g. if the URL we are trying to\naccess returns 404 not found - the check for existence of object can\ntake a while if the system is busy, I think).  \n\nNow in current implementation either cache entry is written in full, or it\nis not written at all.  It would be, I think, fairly easy to check with the\ncurrent code whether cache entry got generated when we acquired readers\nlock (when the process get terminated, the lock gets released, which\nis advantage over using atomic creating file with O_EXCL for locking),\nand if we didn't repeat the whole process.  With the \"tee\"/\"tail\" \nsolution if the process generating data got interrupted before end,\nwe can detect such situation, but currently I have no idea what should\nbe done in such situation.  We can as easily as for the current solution\n(which needs \"Generating...\" page for activity indicator) to detect\ndie_error situation, and with some care i.e. with not writing to cache\nfile directly we can ensure that cache entries contain full, correctly\ngenerated data.\n\n> \n> P.S.: Again the disclaimer - if this is \"too hard\", it's better to\n> accept patches like they are, then improve this later. But perhaps\n> a better solution would be not to clutter the code by optimizing this\n> case at all if it's not clear it really matters in the real world.\n\nSee above.\n\n\nP.S. I have noticed that with current implementation (well, I am not sure\nif it is true also for J.H. implementation) there is problem if there is\nmore than one process trying to request URL which result in die_error being\ncalled.  The design decision, present in original patch, was to not cache\n\"die_error\" / non-\"200 OK\" pages; it seems sane, but I don't know if it\nwas a correct decision.  The solution for interrupted generating process,\ndescribed above, works also for die_error pages, although it makes \ndie_error pages slower for such (hopefully rare) situation of simultaneous\nerrorneous request.\n\nP.P.S. Both Pasky's approach to caching projects_list page, and Lea Wiemann\nwork on \"gitweb caching\" project for Google Summer of Code 2008 approached\ncaching in different way: by caching (parsed) data, not by caching output.\nNote however that for some actions like 'snapshot' we would probably want\nto have response/output caching anyway.  Also for output caching we can\nuse X-Sendfile (or like) extension.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132638","messageId":"201001252248.14874.jnareb@gmail.com","threadId":"22216","inReplyTo":"4B5E03AD.1020404@eaglescrag.net","subject":"Re: [PATCH 8/9] gitweb: Convert output to using indirect file handle","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-25T21:48:14Z","receivedAt":"2010-01-25T21:48:14Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, 25 Jan 2010, J.H. wrote:\n\n> > So there is a bit of rule of preservation of difficulty at work.  Either\n> > we have large patch adding explicit filehandle to all print statements\n> > 'print <sth> -> print $out <sth>' but simple code, or have smaller patch\n> > but complicated *STDOUT manipulation, or have small patch but rely on \n> > non-core CPAN modules present.\n> \n> I think depending in non-core CPAN modules is a really bad idea, and\n> will cause some concern and consternation with mainline distributions,\n> besides making it more complicated for users to get this up and running\n> quickly and easily.  \n\nNote: the modules in question are Capture::Tiny or PerlIO::Util.\nCapture::Tiny is available as perl-Capture-Tiny RPM module for Fedora Core\nup from FC11 and in DAG repository, at least. \n\n\nFirst, installing non-core CPAN modules locally for user is very simple\nthanks to local::lib.  After setup it is simply 'cpan -i PerlIO::Util',\nand PerlIO::Util gets installed locally, by default inside ~/perl5.\n\nSecond, we can go the route that Git.pm did, namely include sources for\nextra modules (private-Error.pm in case of Git.pm) and install it if \nneeded.  With proper build system (Module::Install, Module::Build) it is\neven simpler with inc or inc::latest.  And there is always PAR[1] if you\nwant to have everything in one file.\n\n[1]: http://par.perl.org\n\n>                     While I agree there are other ways of handling this \n> I think just adding the filehandle to the print statements provides the\n> easiest cross section of usability and functionality for everything\n> involved.\n\nWell, manipulating *STDOUT shouldn't be, I think, much harder, although\nI guess it would need more care (so that e.g. \"Generating...\" page gets\nwritten to real STDOUT).\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132661","messageId":"201001260249.36678.jnareb@gmail.com","threadId":"22216","inReplyTo":"4B5DFFE5.6060908@eaglescrag.net","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-26T01:49:35Z","receivedAt":"2010-01-26T01:49:35Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, 25 Jan 2010, J.H. wrote:\n> On 01/25/2010 05:56 AM, Petr Baudis wrote:\n>> On Mon, Jan 25, 2010 at 02:48:26PM +0100, Jakub Narebski wrote:\n\n>>> Now those patches (mine and J.H. both) make gitweb use locking\n>>> (it is IIRC configurable in J.H. patch) to make only one process\n>>> generate the page if it is missing from cache, or is stale.  Now\n>>> if it is missing, we have to wait until it is generated in full\n>>> before being able to show it to client.  While it is possible to\n>>> \"tee\" output (using PerlIO::tee, or Capture::Tiny, or tie like\n>>> CGI::Cache) writing it simultaneously to browser and to cache for \n>>> the process that is generating data, it is as far as I understand\n>>> it impossible for processes which are waiting for data.  Therefore\n>>> the need for \"Generating...\" page, so the user does not think that\n>>> web server hung or something, and is not generating output.\n>> \n>> Ah, ok, so the message is there to cover up for a technical problem. ;-)\n>> I didn't quite realize. Then, it would be great to tweak the mechanisms\n>> so that the user does not really have to wait.\n> \n> No, that is an incorrect assumption on how the 'Generating...' page\n> works, and your missing a bit of the point.\n> \n> (1) The message itself 'Generating...' is a cue to the user that\n> something is happening and that the browser is not actually hanging.\n> Web users are at the point where if things are not instantaneous and\n> show immediately they will either browse away completely or hit the\n> refresh button incessantly until content does appear.  While the page is\n> usually only seen for about a second, and I'll admit it can be annoying,\n> it's nothing more than a 'sit tight a second'.  For things like the\n> front page it can take upwards of 7 seconds to generate for a single\n> user, a lot to ask for a no response scenario.\n\nPasky is asking later whether instead of separate \"Generating...\" page\n(be it server side using trick with http-equiv refresh taking place when\npage/connection is closed, or JavaScript Ajax-y one) couldn't gitweb\nuse streaming and have early parts of generated page (in particular the\npage header and navigation bar) serve as activity indicator.  I think\nit is possible, if a bit more complicated, and such feature can be \nintroduced in a separate commit.\n\nAs to 'sit tight for a second': the code can be modified to print/show \n\"Generating...\" page only if we are waiting more than second (or more\nthan half of second with utime from Time::HiRes) for the data.  Pasky,\nwould such change address some of your concerns about \"Generating...\"\npage as activity indicator?\n\n> \n> (2) It prevents the stampeding herd problem, which was very vehemently\n> discussed 4 years ago by HPA and myself and roughly boils down to this:\n> \n> When a single user comes into the site, in particular the front page, it\n> kicks off a process that will start to generate at it, causing a huge\n> amount of git requests into individual repositories and a lot of disk\n> i/o.  A second user will then come in and the same requests will start\n> to be done from the beginning again, and so on until you basically kill\n> the machine because the disk i/o goes up enough that it can't ever\n> service the requests fast enough.\n\nAs you can see from the way my proof of concept (of splitting) caching\nseries is split, I think that those two features are separate.  Using\nlocking to have one writer, and protect against cache miss stampede \nproblem is one issue, and having \"Generating...\" page is another issue.\nThey are not orthogonal as they are now, as \"Generating...\" page depends\non locking to check if page got generated, but it could simply check\nwhether the cache file is present already.\n\n> \n> This does 2 things in the end:\n> \n> 1) means there's only 1 copy of the page ever being generated, thus\n> meaning there isn't extraneous and dangerous disk i/o going on on the system\n> \n> 2) prevents a user from reporting to the website that it's broken by\n> giving them a visual cue that things aren't broken.\n\n1) is about locking ('expires_variance' can help there, too), 2) is about\n\"Generating...\" but could be done with streaming: \"tee\"/\"tail\" solution.\n\n>> So, I wonder about two things:\n>> \n>> (i) How often does it happen that two requests for the same page are\n>> received? Has anyone measured it? Or is at least able to make\n>> a minimally educated guess? IOW, isn't this premature optimization?\n> \n> For most pages, not many but it happens more often than you think.  The\n> data I have is much too old to be useful now but the front page could,\n> at times, have up to 30 people waiting for it without caching.  This is\n> a very important patch believe it or not.  For a site the size of\n> kernel.org it cannot exist without this.\n> \n> But here's a quick stat, in 36 hours git.kernel.org has had\n> 156099 accesses world wide or about 1.2 accesses a second.\n> \n> android.git.kernel.org, in the same time period has had 115818 accesses.\n> \n> If the first request takes 7 seconds to generate, by the time it's done\n> there are now 3 additional requests running.  If it again takes 7\n> seconds to generate there are now another 3 requests running, etc.  Very\n> quickly you've got so much i/o running the box more or less is useless.\n\nOn the other hand do cache for front page (projects list) ever expire\nso that there are no stale data to serve?  So the issue with growing I/O\nis I guess only with priming the cache.  \n\nBTW. I wonder how well the 'expires_variance' solution used in CHI would\nprotect against cache miss stampedes; that probably depends on the ratio\nof variance to time it takes gitweb to generate the page.\n\n>> (ii) Can't the locked gitwebs do the equivalent of tail -f?\n> \n> Not really going to help much, most of the gitweb operations won't\n> output much of anything beyond the header until it's collected all of\n> the data it needs anyway and then there will be a flurry of output.  \n\nWell, it might be enough to have headers as activity indicator... but\non the other hand users can think that server is broken if they see no\nfurther data, as opposed to the \"Generating...\" page.\n\n> It also means that this 'Generating...' page will only work for caching\n> schemes that tail can read out of, which I'm not sure it would work all\n> that well with things like memcached or a non-custom caching layer where\n> we don't necessarily have direct access to the file being written to.\n\nI don't think that caching interfaces such as CHI or Cache::Cache, or\nindividual caching engines such as Cache::Memcached or Cache::FastMmap\noffer streaming access.  OTOH we can read from the side as the data\nis generated (\"tee\" to a temporary file, or something), but it is even\nmore complication.\n\n> \n> At least the way I had it (and I'll admit I haven't read through Jakub's\n> re-working of my patches so I don't know if it's still there) is that\n> with background caching you only get the 'Generating...' page if it's\n> new or the content is grossly out of data.  If it's a popular page and\n> it's not grossly out of date it shows you the 'stale' data while it\n> generates the new content in the background anyway, only locking you out\n> when the new file is being written.  Or at least that's how I had it.\n\nIt is there, i.e. gitweb would serve stale data if it can be found,\nbut it lacks 'too stale to serve as stale data' timeout/expire time.\nI skipped this feature to get split series out of door faster.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132663","messageId":"201001260330.26365.jnareb@gmail.com","threadId":"22216","inReplyTo":"4B5E01F5.20402@eaglescrag.net","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-26T02:30:25Z","receivedAt":"2010-01-26T02:30:25Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, 25 Jan 2010, J.H. wrote:\n> On 01/25/2010 05:48 AM, Jakub Narebski wrote:\n>> On Mon, Jan 25, 2010, Petr Baudis wrote:\n>>> On Mon, Jan 25, 2010 at 12:46:39PM +0100, Jakub Narebski wrote:\n>>>> On Sun, 24 Jan 2010, Petr Baudis wrote:\n>> \n>>>>>   I have stupid question, common to both the original patch and this\n>>>>> RFC.\n>>>>>\n>>>>>> [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n>>>>>> regenerating cache (WIP)\n>>>>>\n>>>>>   Just why is a \"Generating...\" page appropriate?\n>>>>>\n>>>>>   I have to admit I hate it; can you please at least make it\n>>>>> configurable? Why is it needed at all? It [...] confuses\n>>>>> non-interactive HTTP clients [...]\n>> \n>>>> Second, gitweb can always check User-Agent header, and serve \n>>>> \"Generating...\" page only to web browsers:\n>>>>\n>>>>   unless (defined $cgi->user_agent() &&\n>>>>           $cgi->user_agent() =~ /\\b(Mozilla|Opera)\\b/i) {\n>>>>   \treturn;\n>>>>   }\n>>>>\n>>>> or something like that.\n>>>\n>>> I'm not too happy with this. What about Safari? Opera? ELinks? There's a\n>>> lot of web browsers.\n>> \n>> The \"Mozilla\" part would catch all \"Mozilla compatibile\" web browsers,\n>> including Firefox (and other Gecko-based web browsers), Internet Explorer,\n>> WebKit based browsers including Safari and Chrome and Konqueror.\n>> The \"Opera\" part would catch Opera.\n>> http://www.nczonline.net/blog/2010/01/12/history-of-the-user-agent-string/\n>> \n>> As to other web browsers like Elinks, Lynx, w3m, Dillo, etc.: the issue\n>> is whether they honor '<meta http-equiv=\"refresh\" content=\"0\" />'.  \n>> I think it is better to stay on the safe side; it is not disaster if web\n>> browser is not shown \"Generating...\" page where it could (but see \n>> explanation below).\n> \n> Most of them do, that particular tag has been around for a long time and\n> since it doesn't require Javascript to do the page refresh it's pretty\n> much universal.\n\nFor this method to work there is additional assumption about web browser,\nnamely that it would follow refresh only after it gets page in full (only\nwhen connection closes).  But perhaps that is true for all web browsers\nfollowing http-equiv refresh.\n\n> \n> The problem is going to be with things like wget when someone wants to\n> snag a binary file.  This works fine if the file is already cached, but\n> the user doesn't get what they are expecting if they get a blob that\n> isn't the final file, but the html contents of the page.  I don't know\n> of any hint that things like wget would send to the server that you\n> could switch based on, but it would be more or less the non-background\n> caching state.\n\nWell, the above solution would work as long as you don't run wget with\n--user-agent.\n\n[...]\n>>> But if it will be possible to turn this off eventually, it's all your\n>>> call whether to bother implementing it. :-)\n>> \n>> In my implementation it is (or rather would be) as simple as just\n>> not passing 'generating_info' => \\&git_generating_data_html in the\n>> GitwebCache::SimpleFileCache constructor.\n> \n> At least in mine it was don't allow background caching.  It would force\n> everyone to wait on the one process that was actually generating\n> content.  But it means a few blank pages with a spinning working icon\n> until the cache releases it's exclusive lock.\n\nTheoretically at least having \"Generating...\" page is orthogonal to using\nlocking, to background serving and to serving stale data.  In current \nimplementation those features are dependent, serving stale data on locking,\nand \"Generating...\" also on locking.\n\nI think that serving stale data should depend on background caching, as\notherwise we introduce inequality in treating process which regenerates\ncache (and must wait without background caching) and rest of processes\n(which are served stale data).\n\nCertainly in final version of patch all this should be made configurable.\n\n \nP.S. With background caching there is a question whether parent should\ngenerate and child display stale or \"Generating...\", or whether parent\nshould display stale/activity and child (re)generate cache entry.  Also\nthere is an issue whether parent should waitpid for child, would fiddling\nwith SIGCHLD handler be enough, or is it not needed.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"132899","messageId":"20100128173950.GH9553@machine.or.cz","threadId":"22216","inReplyTo":"4B5DFFE5.6060908@eaglescrag.net","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2010-01-28T17:39:50Z","receivedAt":"2010-01-28T17:39:50Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Mon, Jan 25, 2010 at 12:32:37PM -0800, J.H. wrote:\n> This does 2 things in the end:\n> \n> 1) means there's only 1 copy of the page ever being generated, thus\n> meaning there isn't extraneous and dangerous disk i/o going on on the system\n\nBut this has nothing to do with what you _do_ when there are multiple\nrequests, whether you do the same as if caching was disabled (hang until\ncontent is generated) or doing something novel (creating redirects\nthrough \"Generating...\" page).\n\n> 2) prevents a user from reporting to the website that it's broken by\n> giving them a visual que that things aren't broken.\n\nBut this has nothing to do with caching per se, right? I think it\nactually makes _no difference_ if caching is enabled or not to this\nproblem, or am I missing something?\n\n\nMy point is, I guess, that showing the Generating page doesn't seem to\nhave actually anything to do with the caching itself?\n\n> > So, I wonder about two things:\n> > \n> > (i) How often does it happen that two requests for the same page are\n> > received? Has anyone measured it? Or is at least able to make\n> > a minimally educated guess? IOW, isn't this premature optimization?\n> \n> For most pages, not many but it happens more often than you think.  The\n> data I have is much too old to be useful now but the front page could,\n> at times, have up to 30 people waiting for it without caching.  This is\n> a very important patch believe it or not.  For a site the size of\n> kernel.org it cannot exist without this.\n..snip numbers..\n\nOk, I think you convinced me here (even if I think the front page is\nkind of special case).\n\n> > (ii) Can't the locked gitwebs do the equivalent of tail -f?\n> \n> Not really going to help much, most of the gitweb operations won't\n> output much of anything beyond the header until it's collected all of\n> the data it needs anyway and then there will be a flurry of output.  It\n> also means that this 'Generating...' page will only work for caching\n> schemes that tail can read out of, which I'm not sure it would work all\n> that well with things like memcached or a non-custom caching layer where\n> we don't necessarily have direct access to the file being written to.\n> \n> At least the way I had it (and I'll admit I haven't read through Jakub's\n> re-working of my patches so I don't know if it's still there) is that\n> with background caching you only get the 'Generating...' page if it's\n> new or the content is grossly out of data.  If it's a popular page and\n> it's not grossly out of date it shows you the 'stale' data while it\n> generates the new content in the background anyway, only locking you out\n> when the new file is being written.  Or at least that's how I had it.\n\nWell, my user experience with gitweb on kernel.org is that I get\nGenerating... page all the time when I dive deep enough to the object\ntree. I just find it really distracting and sometimes troublesome when\nI want to wget some final object.\n\nI think it's fine to take in the caching support with the Generating...\npage in the bundle, but I do want to declare my intention to get rid of\nit later, at least for caching backends that could do without it - for\npages where content appears incrementally, tail -f, for pages where\ncontent appears all at once, show at least the header and some \"I'm\nbusy\" notification without redirects.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nIf you can't see the value in jet powered ants you should turn in\nyour nerd card. -- Dunbal (464142)\n"},{"id":"133178","messageId":"201001311258.43202.jnareb@gmail.com","threadId":"22216","inReplyTo":"20100128173950.GH9553@machine.or.cz","subject":"Re: [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when regenerating cache (WIP)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-01-31T11:58:41Z","receivedAt":"2010-01-31T11:58:41Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, Jan 28, 2010, Petr Baudis wrote:\n> On Mon, Jan 25, 2010 at 12:32:37PM -0800, J.H. wrote:\n> > This does 2 things in the end:\n> > \n> > 1) means there's only 1 copy of the page ever being generated, thus\n> > meaning there isn't extraneous and dangerous disk i/o going on on the system\n> \n> But this has nothing to do with what you _do_ when there are multiple\n> requests, whether you do the same as if caching was disabled (hang until\n> content is generated) or doing something novel (creating redirects\n> through \"Generating...\" page).\n> \n> > 2) prevents a user from reporting to the website that it's broken by\n> > giving them a visual que that things aren't broken.\n> \n> But this has nothing to do with caching per se, right? I think it\n> actually makes _no difference_ if caching is enabled or not to this\n> problem, or am I missing something?\n> \n> \n> My point is, I guess, that showing the Generating page doesn't seem to\n> have actually anything to do with the caching itself?\n\nThe point is that without caching it is easy to streaming response, and\nto consider early parts of page (like page header, generated before any\nheavy work) to serve as activity indicator.\n\nWith caching it is difficult to have streaming response, both from\ntechnical point of view (writer must generate to client and to cache\nsimultaneously, readers must know when writer finished work to close\nconnection), and from robustness point of view (what happens if writer\nis interrupted / killed before finishing generating output).  With\n\"generate then display\" (which is not exclusive to caching, and is\nanother possible way of generating content even without caching) we\nrather need some kind of activity indicator like \"Generating...\" page.\n\nI think that \"Generating...\" page can be improved in two ways:\n* Show \"Generating...\" page only if we are waiting for response for\n  more than one second.  This might need mucking with alarms, as I think\n  that sleep 1 before $self->generating_info(...) would be not a good\n  solution.\n* Stream response (using PerlIO::tee layer from PerlIO::Util, or \n  Capture::Tiny module, or tied filehandle like in CGI::Cache) for\n  writer (i.e. process generating data), and wait for it to be finished\n  (perhaps with \"Generating...\") in readers.  This way you wouldn't get\n  \"Generating...\" page for rare views/URLs, and for common views/URLs\n  there is high probability that you would not need \"Generating...\"\n  page as there would be slightly stale response to serve.\nOf course one can implement _both_ of those solutions, i.e. wait one\nseconds in readers, and stream in writer.\n\nI am not sure, but there might be another issue why activity indicator\nis more important for the case with caching enabled.  If you interrupt\nwriter, one of readers waiting for finished data would have to take\nrole of writer, which besides need for technical solution to this problem\nwould mean longer wait.\n\n[..] \n> > > (ii) Can't the locked gitwebs do the equivalent of tail -f?\n> > \n> > Not really going to help much, most of the gitweb operations won't\n> > output much of anything beyond the header until it's collected all of\n> > the data it needs anyway and then there will be a flurry of output.  It\n> > also means that this 'Generating...' page will only work for caching\n> > schemes that tail can read out of, which I'm not sure it would work all\n> > that well with things like memcached or a non-custom caching layer where\n> > we don't necessarily have direct access to the file being written to.\n> > \n> > At least the way I had it (and I'll admit I haven't read through Jakub's\n> > re-working of my patches so I don't know if it's still there) is that\n> > with background caching you only get the 'Generating...' page if it's\n> > new or the content is grossly out of data.  If it's a popular page and\n> > it's not grossly out of date it shows you the 'stale' data while it\n> > generates the new content in the background anyway, only locking you out\n> > when the new file is being written.  Or at least that's how I had it.\n> \n> Well, my user experience with gitweb on kernel.org is that I get\n> \"Generating...\" page all the time when I dive deep enough to the object\n> tree. I just find it really distracting and sometimes troublesome when\n> I want to wget some final object.\n\nFirst, the user_agent checking would help there (it's a pity that all\nweb spiders (bots) and all non-interactive downloaders do not say what\nthey are explicitly in User-Agent string).  \n\nSecond, I guess that waiting 1 second (or more) before showing \n\"Generating...\" page would help in most cases.\n\n> \n> I think it's fine to take in the caching support with the Generating...\n> page in the bundle, but I do want to declare my intention to get rid of\n> it later, at least for caching backends that could do without it - for\n> pages where content appears incrementally, tail -f, for pages where\n> content appears all at once, show at least the header and some \"I'm\n> busy\" notification without redirects.\n\nIn the final version this should be fully configurable.  Note that\nthe series of patches I have send were just proof of concept for \nsplitting caching patch into smaller parts / individual features.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"133766","messageId":"4B6CBD05.6040604@eaglescrag.net","threadId":"22216","inReplyTo":"cover.1264198194.git.jnareb@gmail.com","subject":"Re: [RFC PATCH 00/10] gitweb: Simple file based output caching","fromName":"J.H.","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-02-06T00:51:17Z","receivedAt":"2010-02-06T00:51:17Z","isPatch":true,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"> Table of contents:\n> ~~~~~~~~~~~~~~~~~~\n>  [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing\n>                    for caching)\n\nThis looks fine, I did some quick testing to verify that this would work\n- and it does.\n\nThe only caveat that needs to be aware is that if the layer is going to\noutput binary data it needs to flip the whole stream to :raw before\noutputting (this is going to be more specific to the caching layer).\n\nOne advantage to having the file handles separate is that it's easier to\ndistinguish if the data is going to need to be binary data that will\nneed to be flipped properly.\n\nAlso means you could cache the binary data differently than textual data.\n\nI.E. binary data gets saved to disk, but page data gets saved to memcached.\n\nJust food for thought, I'm not sure which way makes more sense\npersonally, though I would have a tendency to err on the side of\nflexibility and have both.\n\n>  [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)\n\nnote: delaying additional comment till I've finished reading through the\nbasics of the following patches.\n\n>  [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching\n\nOk this is quite the departure from what I had, I'm unsure that it's the\nright way to go, but it obviously has merits (I.E. much simpler addition\nof any Cache::Cache or CHI compatible caching layer)\n\nThis patch itself looks fine, and as it states it borrows heavily from\nmy basic implementation - just wraps it differently.  I might have some\nthoughts on extending this a bit to be a bit more flushed out from a\nbasic standpoint.\n\nNeed to dig through it some more, but I'm generally ok with it.\n\n>  [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration\n\nLooks fine to me, though the note about getting the errors should get\nmoved to previous patch, as it says.\n\nNote: I'm going to stop here as the following are WIP and I want to play\naround with this particular direction on my own a little more before\nfurther comment.  There's some ideas running around I want to try and\nget down in code first.  Me moving on and trying these other ideas is\nnot a reflection on the following patches, just some alternative\nthinking before I discuss some other ideas on the following patches.\n\nAlso I've been sitting on this e-mail in this state for almost a week\nwhile I've been playing with this and having to fight other fires and I\nknow that Jakub has been looking for commentary on this.\n\n>  [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set)\n>                    output caching (WIP)\n>  [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)\n>  [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)\n>  [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd'\n>                    problem (WIP)\n>  [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for\n>                    filling cache (WIP)\n>  [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n>                    regenerating cache (WIP)\n"},{"id":"133834","messageId":"201002070056.31665.jnareb@gmail.com","threadId":"22216","inReplyTo":"4B6CBD05.6040604@eaglescrag.net","subject":"Re: [RFC PATCH 00/10] gitweb: Simple file based output caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-02-06T23:56:30Z","receivedAt":"2010-02-06T23:56:30Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sat, 6 Feb 2010, J.H. wrote:\n> > Table of contents:\n> > ~~~~~~~~~~~~~~~~~~\n> >  [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing\n> >                    for caching)\n\nI am working on v2 of this series, where this patch is not necessary.\n\nIt uses *STDOUT->push_layer(scalar => \\$data) and *STDOUT->pop_layer()\nfrom PerlIO::Util if it is available, and manipulation of *STDOUT (which\nmeans *STDOUT = $data_fh and not $out = $data_fh).  But I must say that\ndoing capture of STDOUT (only; STDERR is not captured) without requiring\nextra Perl modules (like recommended Capture::Tiny or e.g. IO::Capture),\nand especially testing that it works correctly with capturing output\nof cache_fetch is serious PITA.\n\nThis patch has the advantage that all operations are simpler.  In\nparticular it is easy to have section which should be not captured,\nor where capture should be turned off (slightly different).\n\nIt has the disadvantage that all future contributions must use\n\"print $out <something>\" / \"print {$out} <something>\", and that\ncontributions from before this change would have to be carefully\nupdated.  (Well, we could probably add the test that would check\nthat everything that needs to go to $out does, and everything that\nshouldn't got to $out but to STDOUT doesn't.)\n\n\nIf I were to have such patch in new version of \"gitweb output\ncaching\" series, I would make the following changes:\n* (optionally) use simpler 'print $out <sth>' instead of visually \n  distinct 'print {$out} <sth>', where from first glance one can\n  see that $out is filehandle and not something to be printed\n* use short filehandle name: $out, or $oh, or $o/$O.\n* split above patch in 2 to 4 patches:\n  - pure mechanical (scripted) change:\n     + print <sth>     ->  print $out <sth>\n     + printf(<sth>)   ->  printf($out <sth>)\n     + binmode STDOUT  ->  binmode $out\n    The last with possible exception of very first binmode call.\n  - realign (purely whitespace change)\n  - wrap too long lines (newlines and whitespace), optional\n  - change $out to $bout/$bin ($binary_output_fh) where needed;\n    but see comment below (optional)\n\n> \n> This looks fine, I did some quick testing to verify that this would work\n> - and it does.\n\nI have only ran test, and didn't actually check that it works correctly.\nThis commit shouldn't change gitweb behaviour at all.\n\n> \n> The only caveat that needs to be aware is that if the layer is going to\n> output binary data it needs to flip the whole stream to :raw before\n> outputting (this is going to be more specific to the caching layer).\n> \n> One advantage to having the file handles separate is that it's easier to\n> distinguish if the data is going to need to be binary data that will\n> need to be flipped properly.\n\nI don't think that it would be needed.  \n\nFirst, all mode changing operations, i.e. calls to binmode are changed\nto act on $out rather than on STDOUT it means.  It means that if we are\nusing 'in memory file' to capture output to scalar variable, then captured\ndata would be properly converted in variable.  So it would be enough to\nsave this variable in :raw mode to file.  If we are saving directly to\ncache file, then of course saved data would go through layer and would\nbe converted properly.  In any case in cache file we would have _already_\n_converted_ data.\n\nThis means that regardless whether $out used ':utf8' (pseudo)layer,\nor ':raw' (pseudo)layer, if we read from cache file in ':raw' (binary mode)\nand print data from cache to original (true) STDOUT also in ':raw' mode,\nwe would print correctly formatted data.\n\n> \n> Also means you could cache the binary data differently than textual data.\n> \n> I.E. binary data gets saved to disk, but page data gets saved to memcached.\n\nThat's true, but on the other hand it would be easy to add some extra\ncommand marking data as binary below binmode.  Or we can examine IO\nlayers (using PerlIO::get_layers($out); the PerlIO module is in Perl\ncore) if there is 'utf8' layer or 'raw' (pseudo)layer.\n\n> \n> Just food for thought, I'm not sure which way makes more sense\n> personally, though I would have a tendency to err on the side of\n> flexibility and have both.\n\nIt might be good idea... but nevertheless I'd like to have short name\nfor binary filehandle, if we decode to keep it.  What should it be?\n$bout, $bin, $B, $bin_out, $out_bin, $bin_fh?\n\n> \n> >  [RFC PATCH 02/10] gitweb: href(..., -path_info => 0|1)\n> \n> note: delaying additional comment till I've finished reading through the\n> basics of the following patches.\n\nThis is to use later _full_ _normalized_ URI as cache key for given page.\nIIRC in your original patch you ignored path_info; but on the other hand\ngit.kernel.org has path_info feature turned off...\n\n> \n> >  [RFC PATCH 03/10] gitweb/cache.pm - Very simple file based caching\n> \n> Ok this is quite the departure from what I had, I'm unsure that it's the\n> right way to go, but it obviously has merits (I.E. much simpler addition\n> of any Cache::Cache or CHI compatible caching layer)\n> \n> This patch itself looks fine, and as it states it borrows heavily from\n> my basic implementation - just wraps it differently.  I might have some\n> thoughts on extending this a bit to be a bit more flushed out from a\n> basic standpoint.\n> \n> Need to dig through it some more, but I'm generally ok with it.\n\nNote that the new implementation in (not send yet) new version of \n\"gitweb output caching\" series is based more on newer and more modern\nCHI unified interface rather than older Cache::Cache interface.  It\nis I think much cleaner and easier to read.\n\nThe major difference from your implementation is that in my version\nthe gitweb caching engine uses \"save to temporary file + rename file\nto final name\" method to have atomic write to cache (atomic cache\nfilling).  It should be more robust, but OTOH it introduces a bit of\nperformance penalty.  With locking and single writer we could use\npredictable temporary file name rather than using tempfile/mkstemp\nor equivalent from File::Temp, or UUID based filename like CHI does\nit.\n\nAlso, tests.\n\n\nCurrent code (even the v2 version) lacks proper error detection, error\nsignalling and logging.\n\n> \n> >  [RFC PATCH 04/10] gitweb/cache.pm - Stat-based cache expiration\n> \n> Looks fine to me, though the note about getting the errors should get\n> moved to previous patch, as it says.\n\nI wanted to get this series out faster, that is why it is not polished.\n\n> \n> Note: I'm going to stop here as the following are WIP and I want to play\n> around with this particular direction on my own a little more before\n> further comment.  There's some ideas running around I want to try and\n> get down in code first.  Me moving on and trying these other ideas is\n> not a reflection on the following patches, just some alternative\n> thinking before I discuss some other ideas on the following patches.\n\nTake a look at gitweb/cache-kernel-v2 branch (the new caching series).\nNote however that it would be subject to rebasing / changes.\n\n> \n> Also I've been sitting on this e-mail in this state for almost a week\n> while I've been playing with this and having to fight other fires and I\n> know that Jakub has been looking for commentary on this.\n\nThank you very much for your commentary, in spite of your heavy load.\n\n> \n> >  [RFC PATCH 05/10] gitweb: Use Cache::Cache compatibile (get, set)\n> >                    output caching (WIP)\n> >  [RFC PATCH 06/10] gitweb/cache.pm - Adaptive cache expiration time (WIP)\n> >  [RFC PATCH 07/10] gitweb: Use CHI compatibile (compute method) caching (WIP)\n> >  [RFC PATCH 08/10] gitweb/cache.pm - Use locking to avoid 'stampeding herd'\n> >                    problem (WIP)\n> >  [RFC PATCH 09/10] gitweb/cache.pm - Serve stale data when waiting for\n> >                    filling cache (WIP)\n> >  [RFC PATCH 10/10] gitweb: Show appropriate \"Generating...\" page when\n> >                    regenerating cache (WIP)\n\nThere is new version of this series in gitweb/cache-kernel-v2 in my\ngit/jnareb-git.git fork (clone) of git.git repository at repo.or.cz.\nNow all commits have proper description (for first series one had to\nread comment section in emails for commit description), and all features\nare tested (at least on API level, and to some extent) -- full tests\ndo require having PerlIO::Util installed (I have done it following\nlocal::lib and installing it from 'cpan' client), though.\n\nAlso all features are fully configurable, to even greater extent than\nin original series by J.H. (this what what v1 was lacking).  And there\nis (see diffstat) section about caching in gitweb/README.\n\nThe following changes since commit d5f8a3d6f4d946c33459e00edf02819f89711777:\n  Junio C Hamano (1):\n        Merge branch 'master' into next\n\nare available in the git repository at:\n\n  git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v2\n\nYou can view it via gitweb at:\n\n  http://repo.or.cz/w/git/jnareb-git.git/shortlog/refs/heads/gitweb/cache-kernel-v2\n  http://repo.or.cz/w/git/jnareb-git.git/log/refs/heads/gitweb/cache-kernel-v2\n\nSHORTLOG (10):\n      gitweb: href(..., -path_info => 0|1)\n      gitweb/cache.pm - Very simple file based caching\n      gitweb/cache.pm - Stat-based cache expiration\n      gitweb: Use Cache::Cache compatibile (get, set) output caching\n      gitweb/cache.pm - Adaptive cache expiration time\n      gitweb: Use CHI compatibile (compute method) caching\n      gitweb/cache.pm - Use locking to avoid 'cache miss stampede' problem\n      gitweb/cache.pm - Serve stale data when waiting for filling cache\n      gitweb/cache.pm - Regenerate (refresh) cache in background\n      gitweb: Show appropriate \"Generating...\" page when regenerating cache\n\n gitweb/README                          |   70 +++++\n gitweb/cache.pm                        |  527 ++++++++++++++++++++++++++++++++\n gitweb/gitweb.perl                     |  301 +++++++++++++++++-\n t/gitweb-lib.sh                        |    2 +\n t/t9500-gitweb-standalone-no-errors.sh |   19 ++\n t/t9503-gitweb-caching.sh              |   32 ++\n t/t9503/test_cache_interface.pl        |  380 +++++++++++++++++++++++\n t/test-lib.sh                          |    3 +\n 8 files changed, 1319 insertions(+), 15 deletions(-)\n create mode 100644 gitweb/cache.pm\n create mode 100755 t/t9503-gitweb-caching.sh\n create mode 100755 t/t9503/test_cache_interface.pl\n\n-- \nJakub Narebski\nPoland\n"},{"id":"133870","messageId":"201002071335.53801.jnareb@gmail.com","threadId":"22216","inReplyTo":"201002070056.31665.jnareb@gmail.com","subject":"Re: [RFC PATCH 00/10] gitweb: Simple file based output caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-02-07T12:35:52Z","receivedAt":"2010-02-07T12:35:52Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sun, 7 Feb 2010, Jakub Narebski wrote:\n\n> There is new version of this series in gitweb/cache-kernel-v2 in my\n> git/jnareb-git.git fork (clone) of git.git repository at repo.or.cz.\n> Now all commits have proper description (for first series one had to\n> read comment section in emails for commit description), [...]\n\nBelow there are commit messages for gitweb/cache-kernel-v2 branch after\nrebase and fixups:\n\ncommit 560e2ab10d0f8457fbeca7a26814ff3e32396f7b\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 11:27:22 2010 +0100\n\n    gitweb: href(..., -path_info => 0|1)\n    \n    If named boolean option -path_info is passed to href() subroutine, use\n    its value to decide whether to generate path_info URL form.  If this\n    option is not passed, href() queries 'pathinfo' feature to check\n    whether to generate path_info URL (if generating path_info link is\n    possible at all).\n    \n    href(-replay=>1, -path_info=>0) is meant to be used to generate a key\n    for caching gitweb output; alternate solution would be to use freeze()\n    from Storable (core module) on %input_params hash (or its reference),\n    e.g.:\n      $key = freeze \\%input_params;\n    or other serialization technique.\n    \n    While at it document extra options/flags to href().\n    \n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/gitweb.perl |    7 ++++++-\n 1 files changed, 6 insertions(+), 1 deletions(-)\n\ncommit dd6e8dc27d5b799bd2a1aed03738195dfe3bc5e7\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 13:13:06 2010 +0100\n\n    gitweb/cache.pm - Very simple file based caching\n    \n    This is first step towards implementing file based output (response)\n    caching layer that is used on such large sites as kernel.org.\n    \n    This patch introduces GitwebCaching::SimpleFileCache package, which\n    follows Cache::Cache / CHI interface, although do not implement it\n    fully.  The intent of following established convention is to be able\n    in the future to replace our simple file based cache e.g. by one using\n    memcached.\n    \n    Like in original patch by John 'Warthog9' Hawley (J.H.) (the one this\n    commit intends to be incremental step to), the data is stored in the\n    case as-is, without adding metadata (like expiration date), and\n    without serialization (which means only scalar data).\n    \n    To be implemented (from original patch by J.H.):\n    * cache expiration (based on file stats, current time and global\n      expiration time); currently elements in cache do not expire\n    * actually using this cache in gitweb, except error pages\n    * adaptive cache expiration, based on average system load\n    * optional locking interface, where only one process can update cache\n      (using flock)\n    * server-side progress indicator when waiting for filling cache,\n      which in turn requires separating situations (like snapshots and\n      other non-HTML responses) where we should not show 'please wait'\n      message\n    \n    Possible extensions (beyond what was in original patch):\n    * (optionally) show information about cache utilization\n    * AJAX (JavaScript-based) progress indicator\n    * JavaScript code to update relative dates in cached output\n    * make cache size-aware (try to not exceed specified maximum size)\n    * utilize X-Sendfile header (or equivalent) to show cached data\n      (optional, as it makes sense only if web server supports sendfile\n      feature and have it enabled)\n    * variable expiration feature from CHI, allowing items to expire a bit\n      earlier than the stated expiration time to prevent cache miss\n      stampedes (although locking, if available, should take care of\n      this).\n    \n    The code of GitwebCaching::SimpleFileCache package in gitweb/cache.pm\n    was heavily based on file-based cache in Cache::Cache package, i.e.\n    on Cache::FileCache, Cache::FileBackend and Cache::BaseCache, and on\n    file-based cache in CHI, i.e. on CHI::Driver::File and CHI::Driver\n    (including implementing atomic write, something that original patch\n    lacks).\n    \n    This patch does not yet enable output caching in gitweb (it doesn't\n    have all required features yet); on the other hand it includes tests,\n    currently testing only cache Perl API.\n    \n    Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/cache.pm                 |  269 +++++++++++++++++++++++++++++++++++++++\n t/t9503-gitweb-caching.sh       |   32 +++++\n t/t9503/test_cache_interface.pl |   84 ++++++++++++\n t/test-lib.sh                   |    3 +\n 4 files changed, 388 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/cache.pm\n create mode 100755 t/t9503-gitweb-caching.sh\n create mode 100755 t/t9503/test_cache_interface.pl\n\ncommit 3914e7da792fec50fcc64c0e644d54cf4451703a\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 13:13:17 2010 +0100\n\n    gitweb/cache.pm - Stat-based cache expiration\n    \n    Add stat-based cache expiration to file-based GitwebCache::SimpleFileCache.\n    Contrary to the way other caching interfaces such as Cache::Cache and CHI\n    do it, the time cache element expires in is _global_ value associated with\n    cache instance, and is not local property of cache entry.  (Currently cache\n    entry does not store any metadata associated with entry... which means that\n    there is no need for serialization / marshalling / freezing and thawing.)\n    Default expire time is -1, which means never expire.\n    \n    To check if cache entry is expired, GitwebCache::SimpleFileCache compares\n    difference between mtime (last modify time) of a cache file and current time\n    with (global) time to expire.  It is done using CHI-compatible is_valid()\n    method.\n    \n    Add some tests checking that expiring works correctly (on the level of API).\n    \n    To be implemented (from original patch by J.H.):\n    * actually using this cache in gitweb, except error pages\n    * adaptive cache expiration, based on average system load\n    * optional locking interface, where only one process can update cache\n      (using flock)\n    * server-side progress indicator when waiting for filling cache,\n      which in turn requires separating situations (like snapshots and\n      other non-HTML responses) where we should not show 'please wait'\n      message\n    \n    Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/cache.pm                 |   34 ++++++++++++++++++++++++++++++++--\n t/t9503/test_cache_interface.pl |   10 ++++++++++\n 2 files changed, 42 insertions(+), 2 deletions(-)\n\ncommit a55625cb0f2d6c08a28e774fd2ddb4e5347a24b3\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 13:13:27 2010 +0100\n\n    gitweb: Use Cache::Cache compatible (get, set) output caching\n    \n    This commit actually adds output caching to gitweb, as we have now\n    minimal features required for it in GitwebCache::SimpleFileCache\n    (a 'dumb' but fast file-based cache engine).  To enable cache you need\n    at least set $caching_enabled to true in gitweb config, and copy cache.pm\n    from gitweb/ alongside gitweb.cgi - this is described in more detail\n    in the new \"Gitweb caching\" section in gitweb/README\n    \n    Currently cache support related subroutines in cache.pm (which are\n    outside GitwebCache::SimpleFileCache package) are not well separated\n    from gitweb script itself; cache.pm lacks encapsulation.  cache.pm\n    assumes that there are href() subroutine and %actions variable, and\n    that there exist $actions{$action} (where $action is parameter passed\n    to cache_fetch), and it is a code reference (see also comments in\n    t/t9503/test_cache_interface.pl).  This is remaining artifact from the\n    original patch by J.H. (which also had cache_fetch() subroutine).\n    \n    Gitweb itself uses directly only cache_fetch, to get page from cache\n    or to generate page and save it to cache, and cache_stop, to be used\n    in die_error subroutine, as currently error pages are not cached.\n    \n    The cache_fetch subroutine captures output (from STDOUT only, as\n    STDERR is usually logged) using either ->push_layer()/->pop_layer()\n    from PerlIO::Util submodule (if it is available), or by setting and\n    restoring *STDOUT.  Note that only the former could be tested reliably\n    to be reliable in t9503 test!\n    \n    Enabling caching causes the following additional changes to gitweb\n    output:\n    * Disables content-type negotiation (choosing between 'text/html'\n      mimetype and 'application/xhtml+xml') when caching, as there is no\n      content-type negotiation done when retrieving page from cache.\n      Use 'text/html' mimetype that can be used by all browsers.\n    * Disable timing info (how much time it took to generate original\n      page, and how many git commands it took), and in its place show when\n      page was originally generated (in GMT / UTC timezone).\n    \n    Add basic tests of caching support to t9500-gitweb-standalone-no-errors\n    test: set $caching_enabled to true and check for errors for first time\n    run (generating cache) and second time run (retrieving from cache) for a\n    single view - summary view for a project.\n    \n    If PerlIO::Util is available (see comments), test that cache_fetch\n    behaves correctly, namely that it saves and restores action output in\n    cache, and that it prints generated output or cached output.\n    \n    \n    To be implemented (from original patch by J.H.):\n    * adaptive cache expiration, based on average system load\n    * optional locking interface, where only one process can update cache\n      (using flock)\n    * server-side progress indicator when waiting for filling cache,\n      which in turn requires separating situations (like snapshots and\n      other non-HTML responses) where we should not show 'please wait'\n      message\n    \n    Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/README                          |   70 ++++++++++++++++++++++\n gitweb/cache.pm                        |   78 ++++++++++++++++++++++++\n gitweb/gitweb.perl                     |  102 ++++++++++++++++++++++++++++----\n t/gitweb-lib.sh                        |    2 +\n t/t9500-gitweb-standalone-no-errors.sh |   19 ++++++\n t/t9503/test_cache_interface.pl        |   93 +++++++++++++++++++++++++++++\n 6 files changed, 352 insertions(+), 12 deletions(-)\n\ncommit 3e471ebd31e881ce1439f23075378c2ec6b95e4d\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 13:13:31 2010 +0100\n\n    gitweb/cache.pm - Adaptive cache expiration time\n    \n    Add to GitwebCache::SimpleFileCache support for adaptive lifetime\n    (cache expiration) control.  Cache lifetime can be increased or\n    decreased by any factor, e.g. load average, through the definition\n    of the 'check_load' callback.\n    \n    Note that using ->set_expires_in, or unsetting 'check_load' via\n    ->set_check_load(undef) turns off adaptive caching.\n    \n    Make gitweb automatically adjust cache lifetime by load, using\n    get_loadavg() function.  Define and describe default parameters for\n    dynamic (adaptive) cache expiration time control.\n    \n    There are some very basic tests of dynamic expiration time in t9503,\n    namely checking if dynamic expire time is within given upper and lower\n    bounds.\n    \n    To be implemented (from original patch by J.H.):\n    * optional locking interface, where only one process can update cache\n      (using flock)\n    * server-side progress indicator when waiting for filling cache,\n      which in turn requires separating situations (like snapshots and\n      other non-HTML responses) where we should not show 'please wait'\n      message\n    \n    Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/cache.pm                 |   55 +++++++++++++++++++++++++++++++++++---\n gitweb/gitweb.perl              |   27 +++++++++++++++++-\n t/t9503/test_cache_interface.pl |   22 +++++++++++++++\n 3 files changed, 97 insertions(+), 7 deletions(-)\n\ncommit 984390f99c33d82cd4ddbfa6e00c721d9e74cddb\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 13:13:52 2010 +0100\n\n    gitweb: Use CHI compatible (compute method) caching\n    \n    If $cache provides CHI compatible ->compute($key, $code) method, use it\n    instead of Cache::Cache compatible ->get($key) and ->set($key, $data).\n    While at it, refactor regenerating cache into cache_calculate subroutine.\n    \n    GitwebCache::SimpleFileCache provides 'compute' method, which currently\n    simply use 'get' and 'set' methods in proscribed manner.  Nevertheless\n    'compute' method can be more flexible in choosing when to refresh cache,\n    and which process is to refresh/(re)generate cache entry.  This method\n    would use (advisory) locking to prevent 'cache miss stampede' (aka\n    'stampeding herd') problem in the next commit.\n    \n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/cache.pm |   39 ++++++++++++++++++++++++++++++++++++---\n 1 files changed, 36 insertions(+), 3 deletions(-)\n\ncommit 7d0109e4379f5187364edf7c25cdbc5247609f64\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 13:18:14 2010 +0100\n\n    gitweb/cache.pm - Use locking to avoid 'cache miss stampede' problem\n    \n    In the ->compute($key, $code) method from GitwebCache::SimpleFileCache,\n    use locking (via flock) to ensure that only one process would generate\n    data to update/fill-in cache; the rest would wait for the cache to\n    be (re)generated and would read data from cache.\n    \n    Currently this feature can not be disabled (via %cache_options).\n    \n    \n    A test in t9503 shows that in the case where there are two clients\n    trying to simultaneously access non-existent or stale cache entry,\n    (and generating data takes (artifically) a bit of time), if they are\n    using ->compute method the data is (re)generated once, as opposed to\n    if those clients are just using ->get/->set methods.\n    \n    To be implemented (from original patch by J.H.):\n    * background building, and showing stale cache\n    * server-side progress indicator when waiting for filling cache,\n      which in turn requires separating situations (like snapshots and\n      other non-HTML responses) where we should not show 'please wait'\n      message\n    \n    Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/cache.pm                 |   29 ++++++++++++++++-\n t/t9503/test_cache_interface.pl |   65 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 92 insertions(+), 2 deletions(-)\n\ncommit e7985f69eb9000860b155939d5fd7040e30f682f\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 13:19:21 2010 +0100\n\n    gitweb/cache.pm - Serve stale data when waiting for filling cache\n    \n    When process fails to acquire exclusive (writers) lock, then instead\n    of waiting for the other process to (re)generate and fill cache, serve\n    stale (expired) data from cache.  This is of course possible only if\n    there is some stale data in cache for given key.\n    \n    This feature of GitwebCache::SimpleFileCache is used only for an\n    ->update($key, $code) method.  It is controlled by 'max_lifetime'\n    cache parameter; you can set it to -1 to always serve stale data\n    if it exists, and you can set it to 0 (or any value smaller than\n    'expires_min') to turn this feature off.\n    \n    This feature, as it is implemented currently, makes ->update() method a\n    bit assymetric with respect to process that acquired writers lock and\n    those processes that didn't, which can be seen in the new test in t9503.\n    The process that is to regenerate (refresh) data in cache must wait for\n    the data to be generated in full before showing anything to client, while\n    the other processes show stale (expired) data immediately.  In order to\n    remove or reduce this assymetry gitweb would need to employ one of the two\n    alternate solutions.  Either data should be (re)generated in background,\n    so that process that acquired writers lock would generate data in\n    background while serving stale data, or alternatively the process that\n    generates data should pass output to original STDOUT while capturing it\n    (\"tee\" otput).\n    \n    When developing this feature, ->is_valid() method acquired additional\n    extra optional parameter, where one cap pass expire time instead of using\n    cache-wode global expire time.\n    \n    To be implemented (from original patch by J.H.):\n    * background building,\n    * server-side progress indicator when waiting for filling cache,\n      which in turn requires separating situations (like snapshots and\n      other non-HTML responses) where we should not show 'please wait'\n      message\n    \n    Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/cache.pm                 |   23 ++++++++++----\n gitweb/gitweb.perl              |    8 +++++\n t/t9503/test_cache_interface.pl |   63 +++++++++++++++++++++++++++++++++++++-\n 3 files changed, 86 insertions(+), 8 deletions(-)\n\ncommit 19911970b8a811a6382e39a10b071bff1dd4bd70\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 13:20:46 2010 +0100\n\n    gitweb/cache.pm - Regenerate (refresh) cache in background\n    \n    This commit removes assymetry in serving stale data (if it exists)\n    when regenerating cache in GitwebCache::SimpleFileCache.  The process\n    that acquired exclusive (writers) lock, and is therefore selected to\n    be the one that (re)generates data to fill the cache, can now generate\n    data in background, while serving stale data.\n    \n    This feature can be enabled or disabled on demand via 'background_cache'\n    cache parameter.  It is turned on by default.\n    \n    To be implemented (from original patch by J.H.):\n    * server-side progress indicator when waiting for filling cache,\n      which in turn requires separating situations (like snapshots and\n      other non-HTML responses) where we should not show 'please wait'\n      message\n    \n    Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/cache.pm                 |   36 +++++++++++++++++++++++++++++-------\n gitweb/gitweb.perl              |    9 +++++++++\n t/t9503/test_cache_interface.pl |   14 ++++++++------\n 3 files changed, 46 insertions(+), 13 deletions(-)\n\ncommit ce97bb5bc1660f6d5c9b9be68c556ac94097978c\nAuthor: Jakub Narebski <jnareb@gmail.com>\nDate:   Sun Feb 7 13:21:10 2010 +0100\n\n    gitweb: Show appropriate \"Generating...\" page when regenerating cache\n    \n    When there exist stale/expired (but not too stale) version of\n    (re)generated page in cache, gitweb returns stale version (and updates\n    cache in background, assuming 'background_cache' is set to true value).\n    When there is no stale version suitable to serve the client, currently\n    we have to wait for the data to be generated in full before showing it.\n    Add to GitwebCache::SimpleFileCache, via 'generating_info' callback,\n    the ability to show user some activity indicator / progress bar, to\n    show that we are working on generating data.\n    \n    Gitweb itself uses \"Generating...\" page as activity indicator, which\n    redirects (via <meta http-equiv=\"Refresh\" ...>) to refreshed version\n    of the page after the cache is filled (via trick of not closing page\n    and therefore not closing connection till data is available in cache,\n    checked by getting shared/readers lock on lockfile for cache entry).\n    The git_generating_data_html() subroutine, which is used by gitweb\n    to implement this feature, is highly configurable: you can choose\n    initial delay, frequency of writing some data so that connection\n    won't get closed, and maximum time to wait for data in \"Generating...\"\n    page (see %generating_options hash).\n    \n    Currently git_generating_data_html() contains hardcoded \"whitelist\" of\n    actions for which such HTML \"Generating...\" page makes sense.\n    \n    \n    This implements final feature from the original gitweb output caching\n    patch by J.H.\n    \n    Inspired-by-code-by: John 'Warthog9' Hawley <warthog9@kernel.org>\n    Signed-off-by: Jakub Narebski <jnareb@gmail.com>\n\n gitweb/cache.pm                 |   23 +++++-\n gitweb/gitweb.perl              |  154 ++++++++++++++++++++++++++++++++++++++-\n t/t9503/test_cache_interface.pl |   45 +++++++++++\n 3 files changed, 216 insertions(+), 6 deletions(-)\n\n-- \nJakub Narebski\nPoland\n"},{"id":"133891","messageId":"201002072232.07378.jnareb@gmail.com","threadId":"22216","inReplyTo":"0dd15cb3f18e2a26fc834fd3b071e6d3ecc00557.1264198194.git.jnareb@gmail.com","subject":"Re: [RFC PATCH 01/10] gitweb: Print to explicit filehandle (preparing for caching)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-02-07T21:32:06Z","receivedAt":"2010-02-07T21:32:06Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sat, 23 Feb 2010, Jakub Narebski wrote:\n\n> This means replacing\n> \n>   print <something>;\n> by\n>   print {$out} <something>;\n> \n> and\n> \n>   binmode STDOUT, <layer>;\n> by\n>   binmode $out, <layer>;\n\n[...]\n\n> This patch was written _before_ comment from Junio that it would be\n> better to simply use\n>   print $out <something>;\n> and do not try to be too clever.\n\nActually 'print FILEHANDLE LIST' form is an example of indirect object\nsyntax in Perl (which for filehandles was present before there were\nobjects in Perl).  In indirect object notation the \"first argument\"\nmust be a name, a scalar variable, or a block (like {$out}).  If instead\nof $out there would be something more complicated, we would have to\nuse block form ('print {$out[$i]} <sth>', for example).\n\n-- \nJakub Narebski\nPoland\n"}]}