{"thread":{"id":"26127","subject":"[RFC PATCH v7 0/9] gitweb: Output caching, with eval/die based error handling","startedAt":"2010-12-22T23:54:32Z","lastAt":"2011-01-05T02:26:59Z","messageCount":34,"participants":["Jakub Narebski","Jonathan Nieder","Junio C Hamano","Jeff King","J.H."],"isPatch":true,"patchVersion":7,"patchTotal":9},"messages":[{"id":"158502","messageId":"20101222234843.7998.87068.stgit@localhost.localdomain","threadId":"26127","inReplyTo":null,"subject":"[RFC PATCH v7 0/9] gitweb: Output caching, with eval/die based error handling","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:54:32Z","receivedAt":"2010-12-22T23:54:32Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This is preliminary (proof of concept) version of shortened series\nintended as replacement (rewrite) of \"Gitweb caching v8\" series from\nJohn 'Warthog9' Hawley (J.H.).\n\nThis series shows how one can manage exception handling using\ndie_error like die, even in the presence of output caching. The\noutput caching engine has an option that allows to turn off (default)\nor on caching of error pages.\n\nThis series is unfinished; it does not include adaptive cache\nlifetime, nor support for other caching engines than the one provided\n(like Cache::Cache or CHI), nor does it support background cache\ngeneration or progress info indicator.\n\nThis is just intended as proof of concept.\n\n---\nJakub Narebski (9):\n gitweb: Add optional output caching\n gitweb/lib - Cache captured output (using compute_fh)\n gitweb/lib - Very simple file based cache\n gitweb/lib - Simple output capture by redirecting STDOUT to file\n t/test-lib.sh: Export also GIT_BUILD_DIR in test_external\n gitweb: Prepare for splitting gitweb\n gitweb: Introduce %actions_info, gathering information about actions\n gitweb: use eval + die for error (exception) handling\n gitweb: Go to DONE_REQUEST rather than DONE_GITWEB in die_error\n\n gitweb/Makefile | 22 +\n gitweb/README | 46 ++\n gitweb/gitweb.perl | 280 +++++++++++++--\n gitweb/lib/GitwebCache/CacheOutput.pm | 84 ++++\n gitweb/lib/GitwebCache/Capture/ToFile.pm | 109 ++++++\n gitweb/lib/GitwebCache/FileCacheWithLocking.pm | 452 ++++++++++++++++++++++++\n t/gitweb-lib.sh | 11 +\n t/t9500-gitweb-standalone-no-errors.sh | 20 +\n t/t9501-gitweb-standalone-http-status.sh | 13 +\n t/t9502-gitweb-standalone-parse-output.sh | 33 ++\n t/t9510-gitweb-capture-interface.sh | 34 ++\n t/t9510/test_capture_interface.pl | 132 +++++++\n t/t9511-gitweb-caching-interface.sh | 34 ++\n t/t9511/test_cache_interface.pl | 381 ++++++++++++++++++++\n t/t9512-gitweb-cache-output-interface.sh | 34 ++\n t/t9512/test_cache_output.pl | 162 +++++++++\n t/test-lib.sh | 4 \n 17 files changed, 1806 insertions(+), 45 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/ToFile.pm\n create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm\n create mode 100755 t/t9510-gitweb-capture-interface.sh\n create mode 100755 t/t9510/test_capture_interface.pl\n create mode 100755 t/t9511-gitweb-caching-interface.sh\n create mode 100755 t/t9511/test_cache_interface.pl\n create mode 100755 t/t9512-gitweb-cache-output-interface.sh\n create mode 100755 t/t9512/test_cache_output.pl\n\n-- \nJakub Narebski\nShadeHawk on #git\nPoland\n"},{"id":"158503","messageId":"20101222235459.7998.43333.stgit@localhost.localdomain","threadId":"26127","inReplyTo":"20101222234843.7998.87068.stgit@localhost.localdomain","subject":"[RFC PATCH v7 1/9] gitweb: Go to DONE_REQUEST rather than DONE_GITWEB in die_error","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:55:00Z","receivedAt":"2010-12-22T23:55:00Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\n\nEnd the request after die_error finishes, rather than exiting gitweb\ninstance (perhaps wrapped like in ModPerl::Registry or gitweb.psgi\ncase).\n\nSigned-off-by: Jakub Narebski \n---\n\n gitweb/gitweb.perl | 3 ++-\n 1 files changed, 2 insertions(+), 1 deletions(-)\n\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 4779618..724287b 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1169,6 +1169,7 @@ sub run {\n \n \t\trun_request();\n \n+\tDONE_REQUEST:\n \t\t$post_dispatch_hook->()\n \t\t\tif $post_dispatch_hook;\n \t\t$first_request = 0;\n@@ -3767,7 +3768,7 @@ EOF\n \tprint \"\\n\";\n \n \tgit_footer_html();\n-\tgoto DONE_GITWEB\n+\tgoto DONE_REQUEST\n \t\tunless ($opts{'-error_handler'});\n }\n \n"},{"id":"158504","messageId":"20101222235525.7998.99816.stgit@localhost.localdomain","threadId":"26127","inReplyTo":"20101222234843.7998.87068.stgit@localhost.localdomain","subject":"[RFC PATCH v7 2/9] gitweb: use eval + die for error (exception) handling","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:55:26Z","receivedAt":"2010-12-22T23:55:26Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\n\nIn gitweb code it is assumed that calling die_error() subroutine would\nend request, just like running \"die\" would. Up till now it was done by\nhaving die_error() jump to DONE_REQUEST (earlier DONE_GITWEB), or in\nearlier version just 'exit' (for mod_perl via ModPerl::Registry it ends\nrequest instead of exiting worker).\n\nInstead of using 'goto DONE_REQUEST' for longjmp-like nonlocal jump, or\nusing 'exit', gitweb uses now native for Perl exception handlingin the\nform of eval / die pair (\"eval BLOCK\" to trap exceptions, \"die LIST\" to\nraise/throw them).\n\nUp till now the \"goto DONE_REQUEST\" solution was enough, but with the\ncoming output caching support and it adding modular structure to gitweb,\nit would be difficult to continue to keep using this solution\n(e.g. interaction with capturing output).\n\n\nBecause gitweb now traps all exceptions occuring run_request(), the\nhandle_errors_html handler (set via set_message from CGI::Carp) is not\nneeded; gitweb can call die_error in -error_handler mode itself. This\nhas the advantage that we can now set correct HTTP header (handler from\nCGI::Carp::set_message was run after HTTP headers were already sent).\n\nGitweb assumes here that exceptions thrown by Perl would be simple\nstrings; die_error() throws hash reference (if not for minimal\nextrenal dependencies, it would be probable object of Class::Exception\nor Throwable class thrown).\n\nNote: in newer versions of CGI::Carp there is set_die_handler(), where\nhandler have to set HTTP headers to the browser itself, but we cannot\nrely on new enough CGI::Carp version to have been installed. Also\nset_die_handler interferes with fatalsToBrowser.\n\nSigned-off-by: Jakub Narebski \n---\n\n gitweb/gitweb.perl | 26 ++++++++------------------\n 1 files changed, 8 insertions(+), 18 deletions(-)\n\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 724287b..c7a1892 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -12,7 +12,7 @@ use strict;\n use warnings;\n use CGI qw(:standard :escapeHTML -nosticky);\n use CGI::Util qw(unescape);\n-use CGI::Carp qw(fatalsToBrowser set_message);\n+use CGI::Carp qw(fatalsToBrowser);\n use Encode;\n use Fcntl ':mode';\n use File::Find qw();\n@@ -1045,21 +1045,6 @@ sub configure_gitweb_features {\n \t}\n }\n \n-# custom error handler: 'die ' is Internal Server Error\n-sub handle_errors_html {\n-\tmy $msg = shift; # it is already HTML escaped\n-\n-\t# to avoid infinite loop where error occurs in die_error,\n-\t# change handler to default handler, disabling handle_errors_html\n-\tset_message(\"Error occured when inside die_error:\\n$msg\");\n-\n-\t# you cannot jump out of die_error when called as error handler;\n-\t# the subroutine set via CGI::Carp::set_message is called _after_\n-\t# HTTP headers are already written, so it cannot write them itself\n-\tdie_error(undef, undef, $msg, -error_handler => 1, -no_http_header => 1);\n-}\n-set_message(\\&handle_errors_html);\n-\n # dispatch\n sub dispatch {\n \tif (!defined $action) {\n@@ -1167,7 +1152,11 @@ sub run {\n \t\t$pre_dispatch_hook->()\n \t\t\tif $pre_dispatch_hook;\n \n-\t\trun_request();\n+\t\teval { run_request() };\n+\t\tif (defined $@ && !ref($@)) {\n+\t\t\t# some Perl error, but not one thrown by die_error\n+\t\t\tdie_error(undef, undef, $@, -error_handler => 1);\n+\t\t}\n \n \tDONE_REQUEST:\n \t\t$post_dispatch_hook->()\n@@ -3768,7 +3757,8 @@ EOF\n \tprint \"\\n\";\n \n \tgit_footer_html();\n-\tgoto DONE_REQUEST\n+\n+\tdie {'status' => $status, 'error' => $error}\n \t\tunless ($opts{'-error_handler'});\n }\n \n"},{"id":"158505","messageId":"20101222235552.7998.76918.stgit@localhost.localdomain","threadId":"26127","inReplyTo":"20101222234843.7998.87068.stgit@localhost.localdomain","subject":"[RFC PATCH v7 3/9] gitweb: Introduce %actions_info, gathering information about actions","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:55:52Z","receivedAt":"2010-12-22T23:55:52Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\n\nCurrently it only contains information about output format, and is not\nused anywhere. It will be used to check whether current action\nproduces HTML output, and therefore is displaying HTML-based progress\ninfo about (re)generating cache makes sense.\n\nIt can contain information about allowed extra options, whether to\ndisplay link to feed (Atom or RSS), etc. in easier and faster way than\nlisting all matching or all non-matching actions at appropriate place.\n\n\nCurrently not used; will be used in next commit, to check if action\nproduces HTML output and therefore we can use HTML-specific progress\nindicator.\n\nSigned-off-by: Jakub Narebski \n---\n\n gitweb/gitweb.perl | 57 ++++++++++++++++++++++++++++++++++++++++++++++++----\n 1 files changed, 53 insertions(+), 4 deletions(-)\n\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex c7a1892..e50654b 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -749,6 +749,54 @@ our %allowed_options = (\n \t\"--no-merges\" => [ qw(rss atom log shortlog history) ],\n );\n \n+# action => {\n+# \t# what is the output format (content-type) of action\n+# \t'output_format' => ('html' | 'text' | 'feed' | 'binary' | undef),\n+# \t# does action require $project parameter to work\n+# \t'needs_project' => (boolean | undef),\n+# \t# log-like action, can start with arbitrary ref or revision\n+# \t'log_like' => (boolean | undef),\n+# \t# has no specific feed, or should lik to OPML / generic project feed\n+# \t'no_feed' => (boolean | undef),\n+# \t# allowed options to be passed ussing 'opt' parameter\n+# \t'allowed_options' => { 'option_1' => 1 [, ... ] },\n+# }\n+our %actions_info = ();\n+sub evaluate_actions_info {\n+\tour %actions_info;\n+\tour (%actions);\n+\n+\t# unless explicitely stated otherwise, default output format is html\n+\t# most actions needs $project parameter\n+\tforeach my $action (keys %actions) {\n+\t\t$actions_info{$action}{'output_format'} = 'html';\n+\t\t$actions_info{$action}{'needs_project'} = 1;\n+\t}\n+\t# list all exceptions; undef means variable format (no definite format)\n+\t$actions_info{$_}{'output_format'} = 'text'\n+\t\tforeach qw(commitdiff_plain patch patches project_index blame_data);\n+\t$actions_info{$_}{'output_format'} = 'feed'\n+\t\tforeach qw(rss atom opml); # there are different types (document formats) of XML\n+\t$actions_info{$_}{'output_format'} = undef\n+\t\tforeach qw(blob_plain object);\n+\t$actions_info{'snapshot'}{'output_format'} = 'binary';\n+\n+\t$actions_info{$_}{'needs_project'} = 0\n+\t\tforeach qw(opml project_list project_index);\n+\n+\t$actions_info{$_}{'log_like'} = 1\n+\t\tforeach qw(log shortlog history);\n+\n+\t$actions_info{$_}{'no_feed'} = 1\n+\t\tforeach qw(tags heads forks tag search);\n+\n+\tforeach my $opt (keys %allowed_options) {\n+\t\tforeach my $act (@{$allowed_options{$opt}}) {\n+\t\t\t$actions_info{$act}{'allowed_options'}{$opt} = 1;\n+\t\t}\n+\t}\n+}\n+\n # fill %input_params with the CGI parameters. All values except for 'opt'\n # should be single values, but opt can be an array. We should probably\n # build an array of parameters that can be multi-valued, but since for the time\n@@ -980,7 +1028,7 @@ sub evaluate_and_validate_params {\n \t\tif (not exists $allowed_options{$opt}) {\n \t\t\tdie_error(400, \"Invalid option parameter\");\n \t\t}\n-\t\tif (not grep(/^$action$/, @{$allowed_options{$opt}})) {\n+\t\tif (!$actions_info{$action}{'allowed_options'}{$opt}) {\n \t\t\tdie_error(400, \"Invalid option parameter for this action\");\n \t\t}\n \t}\n@@ -1061,7 +1109,7 @@ sub dispatch {\n \tif (!defined($actions{$action})) {\n \t\tdie_error(400, \"Unknown action\");\n \t}\n-\tif ($action !~ m/^(?:opml|project_list|project_index)$/ &&\n+\tif ($actions_info{$action}{'needs_project'} &&\n \t !$project) {\n \t\tdie_error(400, \"Project needed\");\n \t}\n@@ -1142,6 +1190,7 @@ sub evaluate_argv {\n \n sub run {\n \tevaluate_argv();\n+\tevaluate_actions_info();\n \n \t$first_request = 1;\n \t$pre_listen_hook->()\n@@ -1803,7 +1852,7 @@ sub format_ref_marker {\n \n \t\t\tif ($indirect) {\n \t\t\t\t$dest_action = \"tag\" unless $action eq \"tag\";\n-\t\t\t} elsif ($action =~ /^(history|(short)?log)$/) {\n+\t\t\t} elsif ($actions_info{$action}{'log_like'}) {\n \t\t\t\t$dest_action = $action;\n \t\t\t}\n \n@@ -2277,7 +2326,7 @@ sub get_feed_info {\n \treturn unless (defined $project);\n \t# some views should link to OPML, or to generic project feed,\n \t# or don't have specific feed yet (so they should use generic)\n-\treturn if ($action =~ /^(?:tags|heads|forks|tag|search)$/x);\n+\treturn if ($actions_info{$action}{'no_feed'});\n \n \tmy $branch;\n \t# branches refs uses 'refs/heads/' prefix (fullname) to differentiate\n"},{"id":"158506","messageId":"20101222235618.7998.17447.stgit@localhost.localdomain","threadId":"26127","inReplyTo":"20101222234843.7998.87068.stgit@localhost.localdomain","subject":"[RFC PATCH v7 4/9] gitweb: Prepare for splitting gitweb","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:56:18Z","receivedAt":"2010-12-22T23:56:18Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\n\nPrepare gitweb for having been split into modules that are to be\ninstalled alongside gitweb in 'lib/' subdirectory, by adding\n\n use lib __DIR__.'/lib';\n\nto gitweb.perl (to main gitweb script), and preparing for putting\nmodules (relative path) in $(GITWEB_MODULES) in gitweb/Makefile.\n\nThis preparatory work allows to add new module to gitweb by simply\nadding\n\n GITWEB_MODULES += \n\nto gitweb/Makefile (assuming that the module is in 'gitweb/lib/'\ndirectory).\n\nWhile at it pass GITWEBLIBDIR in addition to GITWEB_TEST_INSTALLED to\nallow testing installed version of gitweb and installed version of\nmodules (for future tests which would check individual (sub)modules).\n\n\nUsing __DIR__ from Dir::Self module (not in core, that's why currently\ngitweb includes excerpt of code from Dir::Self defining __DIR__) was\nchosen over using FindBin-based solution (in core since perl 5.00307,\nwhile gitweb itself requires at least perl 5.8.0) because FindBin uses\nBEGIN block, which is a problem under mod_perl and other persistent\nPerl environments (thought there are workarounds).\n\nAt Pavan Kumar Sankara suggestion gitweb/Makefile uses\n\n install [OPTION]... SOURCE... DIRECTORY\n\nformat (2nd format) with single SOURCE rather than\n\n install [OPTION]... SOURCE DEST\n\nformat (1st format) because of security reasons (race conditions).\nModern GNU install has `-T' / `--no-target-directory' option, but we\ncannot rely that the $(INSTALL) we are using supports this option.\n\nThe install-modules target in gitweb/Makefile uses shell 'for' loop,\ninstead of make's $(foreach) function, to avoid possible problem with\ngenerating a command line that exceeded the maximum argument list\nlength.\n\nSigned-off-by: Jakub Narebski \n---\n\n gitweb/Makefile | 17 +++++++++++++++--\n gitweb/gitweb.perl | 8 ++++++++\n 2 files changed, 23 insertions(+), 2 deletions(-)\n\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex 0a6ac00..e6029e1 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -57,6 +57,7 @@ PERL_PATH ?= /usr/bin/perl\n bindir_SQ = $(subst ','\\'',$(bindir))#'\n gitwebdir_SQ = $(subst ','\\'',$(gitwebdir))#'\n gitwebstaticdir_SQ = $(subst ','\\'',$(gitwebdir)/static)#'\n+gitweblibdir_SQ = $(subst ','\\'',$(gitwebdir)/lib)#'\n SHELL_PATH_SQ = $(subst ','\\'',$(SHELL_PATH))#'\n PERL_PATH_SQ = $(subst ','\\'',$(PERL_PATH))#'\n DESTDIR_SQ = $(subst ','\\'',$(DESTDIR))#'\n@@ -153,20 +154,32 @@ test:\n \n test-installed:\n \tGITWEB_TEST_INSTALLED='$(DESTDIR_SQ)$(gitwebdir_SQ)' \\\n+\tGITWEBLIBDIR='$(DESTDIR_SQ)$(gitweblibdir_SQ)' \\\n \t\t$(MAKE) -C ../t gitweb-test\n \n ### Installation rules\n \n-install: all\n+install: all install-modules\n \t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebdir_SQ)'\n \t$(INSTALL) -m 755 $(GITWEB_PROGRAMS) '$(DESTDIR_SQ)$(gitwebdir_SQ)'\n \t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n \t$(INSTALL) -m 644 $(GITWEB_FILES) '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n \n+install-modules:\n+\tinstall_dirs=\"$(sort $(dir $(GITWEB_MODULES)))\" && \\\n+\tfor dir in $$install_dirs; do \\\n+\t\ttest -d '$(DESTDIR_SQ)$(gitweblibdir_SQ)'/\"$$dir\" || \\\n+\t\t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitweblibdir_SQ)'/\"$$dir\"; \\\n+\tdone\n+\tgitweb_modules=\"$(GITWEB_MODULES)\" && \\\n+\tfor mod in $$gitweb_modules; do \\\n+\t\t$(INSTALL) -m 644 \"lib/$$mod\" '$(DESTDIR_SQ)$(gitweblibdir_SQ)'/\"$$(dirname $$mod)\"; \\\n+\tdone\n+\n ### Cleaning rules\n \n clean:\n \t$(RM) gitweb.cgi static/gitweb.min.js static/gitweb.min.css GITWEB-BUILD-OPTIONS\n \n-.PHONY: all clean install test test-installed .FORCE-GIT-VERSION-FILE FORCE\n+.PHONY: all clean install install-modules test test-installed .FORCE-GIT-VERSION-FILE FORCE\n \ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex e50654b..880fdf2 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -10,6 +10,14 @@\n use 5.008;\n use strict;\n use warnings;\n+\n+use File::Spec;\n+# __DIR__ is taken from Dir::Self __DIR__ fragment\n+sub __DIR__ () {\n+\tFile::Spec->rel2abs(join '', (File::Spec->splitpath(__FILE__))[0, 1]);\n+}\n+use lib __DIR__ . '/lib';\n+\n use CGI qw(:standard :escapeHTML -nosticky);\n use CGI::Util qw(unescape);\n use CGI::Carp qw(fatalsToBrowser);\n"},{"id":"158507","messageId":"20101222235639.7998.10673.stgit@localhost.localdomain","threadId":"26127","inReplyTo":"20101222234843.7998.87068.stgit@localhost.localdomain","subject":"[RFC PATCH v7 5/9] t/test-lib.sh: Export also GIT_BUILD_DIR in test_external","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:56:39Z","receivedAt":"2010-12-22T23:56:39Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\n\nThis way we can use it in test scripts written in other languages\n(e.g. in Perl), and not rely on \"$TEST_DIRECTORY/..\"\n\nSigned-off-by: Jakub Narebski \n---\n\n t/test-lib.sh | 4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 48fa516..c077fa4 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -552,9 +552,9 @@ test_external () {\n \t\t# Announce the script to reduce confusion about the\n \t\t# test output that follows.\n \t\tsay_color \"\" \"# run $test_count: $descr ($*)\"\n-\t\t# Export TEST_DIRECTORY, TRASH_DIRECTORY and GIT_TEST_LONG\n+\t\t# Export TEST_DIRECTORY, GIT_BUILD_DIR, TRASH_DIRECTORY and GIT_TEST_LONG\n \t\t# to be able to use them in script\n-\t\texport TEST_DIRECTORY TRASH_DIRECTORY GIT_TEST_LONG\n+\t\texport TEST_DIRECTORY GIT_BUILD_DIR TRASH_DIRECTORY GIT_TEST_LONG\n \t\t# Run command; redirect its stderr to &4 as in\n \t\t# test_run_, but keep its stdout on our stdout even in\n \t\t# non-verbose mode.\n"},{"id":"158508","messageId":"20101222235705.7998.76695.stgit@localhost.localdomain","threadId":"26127","inReplyTo":"20101222234843.7998.87068.stgit@localhost.localdomain","subject":"[RFC PATCH v7 6/9] gitweb/lib - Simple output capture by redirecting STDOUT to file","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:57:05Z","receivedAt":"2010-12-22T23:57:05Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\n\nAdd GitwebCache::Capture::ToFile package, which captures output by\nredirecting STDOUT to given file (specified by filename, or given opened\nfilehandle), earlier saving original STDOUT to restore it when finished\ncapturing.\n\nGitwebCache::Capture::ToFile preserves PerlIO layers, both those set\nbefore started capturing output, and those set during capture.\n\nNo care was taken to handle the following special cases (prior to\nstarting capture): closed STDOUT, STDOUT reopened to scalar reference,\ntied STDOUT. You shouldn't modify STDOUT during capture.\n\nIncludes separate tests for capturing output in\nt9510/test_capture_interface.pl which is run as external test from\nt9510-gitweb-capture-interface.sh. It tests capturing of utf8 data\nprinted in :utf8 mode, and of binary data (containing invalid utf8) in\n:raw mode.\n\n\nThis patch was based on \"gitweb: add output buffering and associated\nfunctions\" patch by John 'Warthog9' Hawley (J.H.) in \"Gitweb caching v7\"\nseries, and on code of Capture::Tiny by David Golden (Apache License 2.0).\n\nBased-on-work-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\n\n gitweb/lib/GitwebCache/Capture/ToFile.pm | 109 +++++++++++++++++++++++++\n t/t9510-gitweb-capture-interface.sh | 34 ++++++++\n t/t9510/test_capture_interface.pl | 132 ++++++++++++++++++++++++++++++\n 3 files changed, 275 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/Capture/ToFile.pm\n create mode 100755 t/t9510-gitweb-capture-interface.sh\n create mode 100755 t/t9510/test_capture_interface.pl\n\n\ndiff --git a/gitweb/lib/GitwebCache/Capture/ToFile.pm b/gitweb/lib/GitwebCache/Capture/ToFile.pm\nnew file mode 100644\nindex 0000000..d2dbf0f\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/Capture/ToFile.pm\n@@ -0,0 +1,109 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Simple output capturing via redirecting STDOUT to given file.\n+#\n+\n+# This is the same mechanism that Capture::Tiny uses, only simpler;\n+# we don't capture STDERR at all, we don't tee, we capture to\n+# explicitely provided file (or filehandle).\n+\n+package GitwebCache::Capture::ToFile;\n+\n+use strict;\n+use warnings;\n+\n+use PerlIO;\n+use Symbol qw(qualify_to_ref);\n+\n+# Constructor\n+sub new {\n+\tmy $class = shift;\n+\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\treturn $self;\n+}\n+\n+sub capture {\n+\tmy $self = shift;\n+\tmy $code = shift;\n+\n+\t$self->capture_start(@_); # pass rest of params\n+\teval { $code->(); 1; };\n+\tmy $exit_code = $?; # save this for later\n+\tmy $error = $@; # save this for later\n+\n+\tmy $got_out = $self->capture_stop();\n+\t$? = $exit_code;\n+\tdie $error if $error;\n+\n+\treturn $got_out;\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+# Start capturing data (STDOUT)\n+sub capture_start {\n+\tmy $self = shift;\n+\tmy $to = shift;\n+\n+\t# save copy of real STDOUT via duplicating it\n+\tmy @layers = PerlIO::get_layers(\\*STDOUT);\n+\topen $self->{'orig_stdout'}, \">&\", \\*STDOUT\n+\t\tor die \"Couldn't dup STDOUT for capture: $!\";\n+\n+\t# close STDOUT, so that it isn't used anymode (to have it fd0)\n+\tclose STDOUT;\n+\n+\t$self->{'to'} = $to;\n+\tmy $fileno = fileno(qualify_to_ref($to)); \n+\tif (defined $fileno) {\n+\t\t# if $to is filehandle, redirect\n+\t\topen STDOUT, '>&', $fileno;\n+\t} elsif (! ref($to)) {\n+\t\t# if $to is name of file, open it\n+\t\topen STDOUT, '>', $to;\n+\t}\n+\t_relayer(\\*STDOUT, \\@layers);\n+\n+\t# started capturing\n+\t$self->{'capturing'} = 1;\n+}\n+\n+# Stop capturing data (required for die_error)\n+sub capture_stop {\n+\tmy $self = shift;\n+\n+\t# return if we didn't start capturing\n+\treturn unless delete $self->{'capturing'};\n+\n+\t# close capture file, and restore original STDOUT\n+\tmy @layers = PerlIO::get_layers(\\*STDOUT);\n+\tclose STDOUT;\n+\topen STDOUT, '>&', fileno($self->{'orig_stdout'});\n+\t_relayer(\\*STDOUT, \\@layers);\n+\n+\treturn exists $self->{'to'} ? $self->{'to'} : $self->{'data'};\n+}\n+\n+# taken from Capture::Tiny by David Golden, Apache License 2.0\n+# with debugging stripped out\n+sub _relayer {\n+\tmy ($fh, $layers) = @_;\n+\n+\tmy %seen = ( unix => 1, perlio => 1); # filter these out\n+\tmy @unique = grep { !$seen{$_}++ } @$layers;\n+\n+\tbinmode($fh, join(\":\", \":raw\", @unique));\n+}\n+\n+\n+1;\n+__END__\n+# end of package GitwebCache::Capture::ToFile\ndiff --git a/t/t9510-gitweb-capture-interface.sh b/t/t9510-gitweb-capture-interface.sh\nnew file mode 100755\nindex 0000000..9151454\n--- /dev/null\n+++ b/t/t9510-gitweb-capture-interface.sh\n@@ -0,0 +1,34 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Jakub Narebski\n+#\n+\n+test_description='gitweb capturing interface\n+\n+This test checks capturing interface used for capturing gitweb output\n+in gitweb caching (GitwebCache::Capture* modules).'\n+\n+# for now we are running only cache interface tests\n+. ./test-lib.sh\n+\n+# this test is present in gitweb-lib.sh\n+if ! test_have_prereq PERL; then\n+\tskip_all='perl not available, skipping test'\n+\ttest_done\n+fi\n+\n+\"$PERL_PATH\" -MTest::More -e 0 >/dev/null 2>&1 || {\n+\tskip_all='perl module Test::More unavailable, skipping test'\n+\ttest_done\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+# The external test will outputs its own plan\n+test_external_has_tap=1\n+\n+test_external \\\n+\t'GitwebCache::Capture* Perl API (in gitweb/lib/)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9510/test_capture_interface.pl\n+\n+test_done\ndiff --git a/t/t9510/test_capture_interface.pl b/t/t9510/test_capture_interface.pl\nnew file mode 100755\nindex 0000000..6d90497\n--- /dev/null\n+++ b/t/t9510/test_capture_interface.pl\n@@ -0,0 +1,132 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+use utf8;\n+\n+use Test::More;\n+\n+# test source version\n+use lib $ENV{GITWEBLIBDIR} || \"$ENV{GIT_BUILD_DIR}/gitweb/lib\";\n+\n+# ....................................................................\n+\n+use_ok('GitwebCache::Capture::ToFile');\n+note(\"Using lib '$INC[0]'\");\n+note(\"Testing '$INC{'GitwebCache/Capture/ToFile.pm'}'\");\n+\n+# Test setting up capture\n+#\n+my $capture = new_ok('GitwebCache::Capture::ToFile' => [], 'The $capture');\n+\n+\n+# Test capturing to file (given by filename) and to filehandle\n+#\n+sub capture_block (&;$) {\n+\t$capture->capture(shift, shift || 'actual');\n+\n+\topen my $fh, '<', 'actual' or return;\n+\tlocal $/ = undef;\n+\tmy $result = <$fh>;\n+\tclose $fh;\n+\treturn $result;\n+}\n+\n+diag('Should not print anything except test results and diagnostic');\n+my $test_data = 'Capture this';\n+my $captured = capture_block {\n+\tprint $test_data;\n+};\n+is($captured, $test_data, 'capture simple data: filename');\n+\n+open my $fh, '>', 'actual';\n+$captured = capture_block(sub {\n+\tprint $test_data;\n+}, $fh);\n+close $fh;\n+is($captured, $test_data, 'capture simple data: filehandle');\n+\n+\n+# Test capturing :utf8 and :raw data\n+#\n+binmode STDOUT, ':utf8';\n+$test_data = <<'EOF';\n+Zażółć gęsią jaźń\n+EOF\n+utf8::decode($test_data);\n+$captured = capture_block {\n+\tbinmode STDOUT, ':utf8';\n+\n+\tprint $test_data;\n+};\n+utf8::decode($captured);\n+is($captured, $test_data, 'capture utf8 data');\n+\n+$test_data = '|\\x{fe}\\x{ff}|\\x{9F}|\\000|'; # invalid utf-8\n+$captured = capture_block {\n+\tbinmode STDOUT, ':raw';\n+\n+\tprint $test_data;\n+};\n+is($captured, $test_data, 'capture raw data');\n+\n+\n+# Test nested capturing, useful for future GitwebCache::CacheOutput tests\n+#\n+sub read_file {\n+\tmy $filename = shift;\n+\n+\topen my $fh, '<', $filename or return;\n+\tlocal $/ = undef;\n+\tmy $result = <$fh>;\n+\tclose $fh;\n+\n+\treturn $result;\n+}\n+\n+my $outer_capture = GitwebCache::Capture::ToFile->new();\n+$captured = $outer_capture->capture(sub {\n+\tprint \"pre|\";\n+\tmy $captured = $capture->capture(sub {\n+\t\tprint \"INNER\";\n+\t}, 'inner_actual');\n+\tprint \"|post\";\n+}, 'outer_actual');\n+\n+my $inner = read_file('inner_actual');\n+my $outer = read_file('outer_actual');\n+\n+is($inner, \"INNER\", 'nested capture: inner');\n+is($outer, \"pre||post\", 'nested capture: outer');\n+\n+\n+# Testing capture when code dies\n+#\n+$captured = $outer_capture->capture(sub {\n+\tprint \"pre|\";\n+\teval {\n+\t\tmy $captured = $capture->capture(sub {\n+\t\t\tprint \"INNER:pre|\";\n+\t\t\tdie \"die from inner\\n\";\n+\t\t\tprint \"INNER:post|\"\n+\t\t}, 'inner_actual');\n+\t};\n+\tprint \"@=$@\" if $@;\n+\tprint \"|post\";\n+}, 'outer_actual');\n+\n+my $inner = read_file('inner_actual');\n+my $outer = read_file('outer_actual');\n+\n+is($inner, \"INNER:pre|\",\n+ 'nested capture with die: inner output captured up to die');\n+is($outer, \"pre|@=die from inner\\n|post\",\n+ 'nested capture with die: outer caught rethrown exception from inner');\n+\n+\n+done_testing();\n+\n+# Local Variables:\n+# coding: utf-8\n+# End:\n"},{"id":"158509","messageId":"20101222235731.7998.3214.stgit@localhost.localdomain","threadId":"26127","inReplyTo":"20101222234843.7998.87068.stgit@localhost.localdomain","subject":"[RFC PATCH v7 7/9] gitweb/lib - Very simple file based cache","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:57:32Z","receivedAt":"2010-12-22T23:57:32Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\n\nThis is first step towards implementing file based output (response)\ncaching layer that is used on such large sites as kernel.org.\n\nThis patch introduces GitwebCaching::SimpleFileCache package, which\nfollows Cache::Cache / CHI interface, although do not implement it\nfully. The intent of following established convention for cache\ninterface is to be able to replace our simple file based cache,\ne.g. by the one using memcached.\n\nThe data is stored in the cache as-is, without adding metadata (like\nexpiration date), and without serialization (which means that one can\nstore only scalar data). At this point there is no support for\nexpiring cache entries.\n\n\nThe code of GitwebCaching::SimpleFileCache package in gitweb/lib\nwas heavily based on file-based cache in Cache::Cache package, i.e.\non Cache::FileCache, Cache::FileBackend and Cache::BaseCache, and on\nfile-based cache in CHI, i.e. on CHI::Driver::File and CHI::Driver\n(including implementing atomic write, something that original patch\nlacks). It tries to follow more modern CHI architecture, but without\nrequiring Moose. It is much simplified compared to both interfaces\nand their file-based drivers.\n\nThis patch does not yet enable output caching in gitweb (it doesn't\nhave all required features yet); on the other hand it includes tests\nof cache Perl API in t/t9503-gitweb-caching-interface.sh.\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\n\n gitweb/lib/GitwebCache/FileCacheWithLocking.pm | 452 ++++++++++++++++++++++++\n t/t9511-gitweb-caching-interface.sh | 34 ++\n t/t9511/test_cache_interface.pl | 381 ++++++++++++++++++++\n 3 files changed, 867 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm\n create mode 100755 t/t9511-gitweb-caching-interface.sh\n create mode 100755 t/t9511/test_cache_interface.pl\n\n\ndiff --git a/gitweb/lib/GitwebCache/FileCacheWithLocking.pm b/gitweb/lib/GitwebCache/FileCacheWithLocking.pm\nnew file mode 100644\nindex 0000000..ecd0e18\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/FileCacheWithLocking.pm\n@@ -0,0 +1,452 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2006, John 'Warthog9' Hawley \n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Gitweb caching engine, file-based cache with flock-based entry locking\n+#\n+\n+# Minimalistic cache that stores data in the filesystem, without serialization.\n+# It uses file locks (flock) to have only one process generating data and\n+# writing to cache, when using CHI-like interface ->compute_fh() method.\n+\n+package GitwebCache::FileCacheWithLocking;\n+\n+use strict;\n+use warnings;\n+\n+use Carp;\n+use File::Path qw(mkpath);\n+use Digest::MD5 qw(md5_hex);\n+use Fcntl qw(:flock);\n+use POSIX qw(setsid);\n+\n+# by default, the cache nests all entries on the filesystem single\n+# directory deep, i.e. '60/b725f10c9c85c70d97880dfe8191b3' for\n+# key name (key digest) 60b725f10c9c85c70d97880dfe8191b3.\n+#\n+our $DEFAULT_CACHE_DEPTH = 1;\n+\n+# by default, the root of the cache is located in 'cache'.\n+#\n+our $DEFAULT_CACHE_ROOT = \"cache\";\n+\n+# by default we don't use cache namespace (empty namespace);\n+# empty namespace does not allow for simple implementation of clear() method.\n+#\n+our $DEFAULT_NAMESPACE = '';\n+\n+# anything less than 0 means to not expire\n+#\n+our $NEVER_EXPIRE = -1;\n+\n+# cache expiration of 0 means that entry is expired\n+#\n+our $EXPIRE_NOW = 0;\n+\n+# ......................................................................\n+# constructor\n+\n+# The options are set by passing in hash or a reference to a hash containing\n+# any of the following keys:\n+# * 'namespace'\n+# The namespace associated with this cache. This allows easy separation of\n+# multiple, distinct caches without worrying about key collision. Defaults\n+# to $DEFAULT_NAMESPACE. Might be empty string.\n+# * 'cache_root' (Cache::FileCache compatibile),\n+# 'root_dir' (CHI::Driver::File compatibile),\n+# The location in the filesystem that will hold the root of the cache.\n+# Defaults to $DEFAULT_CACHE_ROOT.\n+# * 'cache_depth' (Cache::FileCache compatibile),\n+# 'depth' (CHI::Driver::File compatibile),\n+# The number of subdirectories deep to cache object item. This should be\n+# large enough that no cache directory has more than a few hundred objects.\n+# Defaults to $DEFAULT_CACHE_DEPTH unless explicitly set.\n+# * 'default_expires_in' (Cache::Cache compatibile),\n+# 'expires_in' (CHI compatibile) [seconds]\n+# The expiration time for objects place in the cache.\n+# Defaults to -1 (never expire) if not explicitly set.\n+# * 'max_lifetime' [seconds]\n+# If it is greater than 0, and cache entry is expired but not older\n+# than it, serve stale data when waiting for cache entry to be \n+# regenerated (refreshed). Non-adaptive.\n+# * 'on_error' (similar to CHI 'on_get_error'/'on_set_error')\n+# How to handle runtime errors occurring during cache gets and cache\n+# sets, which may or may not be considered fatal in your application.\n+# Options are:\n+# * \"die\" (the default) - call die() with an appropriate message\n+# * \"warn\" - call warn() with an appropriate message\n+# * \"ignore\" - do nothing\n+# * - call this code reference with an appropriate message\n+sub new {\n+\tmy $class = shift;\n+\tmy %opts = ref $_[0] ? %{ $_[0] } : @_;\n+\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\t$self->{'root'} =\n+\t\texists $opts{'cache_root'} ? $opts{'cache_root'} :\n+\t\texists $opts{'root_dir'} ? $opts{'root_dir'} :\n+\t\t$DEFAULT_CACHE_ROOT;\n+\t$self->{'depth'} =\n+\t\texists $opts{'cache_depth'} ? $opts{'cache_depth'} :\n+\t\texists $opts{'depth'} ? $opts{'depth'} :\n+\t\t$DEFAULT_CACHE_DEPTH;\n+\t$self->{'namespace'} =\n+\t\texists $opts{'namespace'} ? $opts{'namespace'} :\n+\t\t$DEFAULT_NAMESPACE;\n+\t$self->{'expires_in'} =\n+\t\texists $opts{'default_expires_in'} ? $opts{'default_expires_in'} :\n+\t\texists $opts{'expires_in'} ? $opts{'expires_in'} :\n+\t\t$NEVER_EXPIRE;\n+\t$self->{'max_lifetime'} =\n+\t\texists $opts{'max_lifetime'} ? $opts{'max_lifetime'} :\n+\t\texists $opts{'max_cache_lifetime'} ? $opts{'max_cache_lifetime'} :\n+\t\t$NEVER_EXPIRE;\n+\t$self->{'on_error'} =\n+\t\texists $opts{'on_error'} ? $opts{'on_error'} :\n+\t\texists $opts{'on_get_error'} ? $opts{'on_get_error'} :\n+\t\texists $opts{'on_set_error'} ? $opts{'on_set_error'} :\n+\t\texists $opts{'error_handler'} ? $opts{'error_handler'} :\n+\t\t'die';\n+\n+\t# validation could be put here\n+\n+\treturn $self;\n+}\n+\n+\n+# ......................................................................\n+# accessors\n+\n+# http://perldesignpatterns.com/perldesignpatterns.html#AccessorPattern\n+\n+# creates get_depth() and set_depth($depth) etc. methods\n+foreach my $i (qw(depth root namespace expires_in max_lifetime\n+ on_error)) {\n+\tmy $field = $i;\n+\tno strict 'refs';\n+\t*{\"get_$field\"} = sub {\n+\t\tmy $self = shift;\n+\t\treturn $self->{$field};\n+\t};\n+\t*{\"set_$field\"} = sub {\n+\t\tmy ($self, $value) = @_;\n+\t\t$self->{$field} = $value;\n+\t};\n+}\n+\n+\n+# ----------------------------------------------------------------------\n+# utility functions and methods\n+\n+# $path = $self->path_to_namespace();\n+#\n+# Return root dir for namespace (lazily built, cached)\n+sub path_to_namespace {\n+\tmy ($self) = @_;\n+\n+\tif (!exists $self->{'path_to_namespace'}) {\n+\t\tif (defined $self->{'namespace'} &&\n+\t\t $self->{'namespace'} ne '') {\n+\t\t\t$self->{'path_to_namespace'} = \"$self->{'root'}/$self->{'namespace'}\";\n+\t\t} else {\n+\t\t\t$self->{'path_to_namespace'} = $self->{'root'};\n+\t\t}\n+\t}\n+\treturn $self->{'path_to_namespace'};\n+}\n+\n+# $path = $cache->path_to_key($key);\n+# $path = $cache->path_to_key($key, \\$dir);\n+#\n+# Take an human readable key, and return file path.\n+# Puts dirname of file path in second argument, if it is provided.\n+sub path_to_key {\n+\tmy ($self, $key, $dir_ref) = @_;\n+\n+\tmy @paths = ( $self->path_to_namespace() );\n+\n+\t# Create a unique (hashed) key from human readable key\n+\tmy $filename = md5_hex($key); # or $digester->add($key)->hexdigest();\n+\n+\t# Split filename so that it have DEPTH subdirectories,\n+\t# where each subdirectory has a two-letter name\n+\tpush @paths, unpack(\"(a2)[$self->{'depth'}] a*\", $filename);\n+\t$filename = pop @paths;\n+\n+\t# Join paths together, computing dir separately if $dir_ref was passed.\n+\tmy $filepath;\n+\tif (defined $dir_ref && ref($dir_ref)) {\n+\t\tmy $dir = join('/', @paths);\n+\t\t$filepath = \"$dir/$filename\";\n+\t\t$$dir_ref = $dir;\n+\t} else {\n+\t\t$filepath = join('/', @paths, $filename);\n+\t}\n+\n+\treturn $filepath;\n+}\n+\n+# $self->ensure_path($dir);\n+#\n+# create $dir (directory) if it not exists, thus ensuring that path exists\n+sub ensure_path {\n+\tmy $self = shift;\n+\tmy $dir = shift || return;\n+\n+\tif (!-d $dir) {\n+\t\t# mkpath will croak()/die() if there is an error\n+\t\tmkpath($dir, 0, 0777);\n+\t}\n+}\n+\n+# $filename = $self->get_lockname($key);\n+#\n+# Take an human readable key, and return path to be used for lockfile\n+# Ensures that file can be created, if needed.\n+sub get_lockname {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $lockfile = $self->path_to_key($key, \\my $dir) . '.lock';\n+\n+\t# ensure that directory leading to lockfile exists\n+\t$self->ensure_path($dir);\n+\n+\treturn $lockfile;\n+}\n+\n+# ----------------------------------------------------------------------\n+# \"private\" utility functions and methods\n+\n+# ($fh, $filename) = $self->_tempfile_to_path($path_for_key, $dir_for_key);\n+#\n+# take a file path to cache entry, and its directory\n+# return filehandle and filename of open temporary file,\n+# like File::Temp::tempfile\n+sub _tempfile_to_path {\n+\tmy ($self, $file, $dir) = @_;\n+\n+\tmy $tempname = \"$file.tmp\";\n+\topen my $temp_fh, '>', $tempname\n+\t\tor die \"Couldn't open temporary file '$tempname' for writing: $!\";\n+\n+\treturn ($temp_fh, $tempname);\n+}\n+\n+# ($fh, $filename) = $self->_wait_for_data($key, $code);\n+#\n+# Wait for data to be available using (blocking) $code,\n+# then return filehandle and filename to read from for $key.\n+sub _wait_for_data {\n+\tmy ($self, $key, $sync_coderef) = @_;\n+\tmy @result;\n+\n+\t# wait for data to be available\n+\t$sync_coderef->();\n+\t# fetch data\n+\t@result = $self->fetch_fh($key);\n+\n+\treturn @result;\n+}\n+\n+# $self->_handle_error($raw_error)\n+#\n+# based on _handle_get_error and _dispatch_error_msg from CHI::Driver\n+sub _handle_error {\n+\tmy ($self, $error) = @_;\n+\n+\tfor ($self->get_on_error()) {\n+\t\t(ref($_) eq 'CODE') && do { $_->($error) };\n+\t\t/^ignore$/ && do { };\n+\t\t/^warn$/ && do { carp $error };\n+\t\t/^die$/ && do { croak $error };\n+\t}\n+}\n+\n+# ----------------------------------------------------------------------\n+# nonstandard worker and semi-interface methods\n+\n+# ($fh, $filename) = $self->fetch_fh($key);\n+#\n+# Get filehandle to read from for given $key, and filename of cache file.\n+# Doesn't check if entry expired.\n+sub fetch_fh {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $path = $self->path_to_key($key);\n+\treturn unless (defined $path);\n+\n+\topen my $fh, '<', $path or return;\n+\treturn ($fh, $path);\n+}\n+\n+# ($fh, $filename) = $self->get_fh($key, [option => value, ...])\n+#\n+# Returns filehandle to read from for given $key, and filename of cache file.\n+# Returns empty list if entry expired.\n+#\n+# $key may be followed by one or more name/value parameters:\n+# * expires_in [DURATION] - override global expiration time\n+sub get_fh {\n+\tmy ($self, $key, %opts) = @_;\n+\n+\treturn unless ($self->is_valid($key, $opts{'expires_in'}));\n+\n+\treturn $self->fetch_fh($key);\n+}\n+\n+# [($fh, $filename) =] $self->set_coderef_fh($key, $code_fh);\n+#\n+# Runs $code_fh, passing to it $fh and $filename of file to write to;\n+# the contents of this file would be contents of cache entry.\n+# Returns what $self->fetch_fh($key) would return.\n+sub set_coderef_fh {\n+\tmy ($self, $key, $code) = @_;\n+\n+\tmy $path = $self->path_to_key($key, \\my $dir);\n+\treturn unless (defined $path && defined $dir);\n+\n+\t# ensure that directory leading to cache file exists\n+\t$self->ensure_path($dir);\n+\n+\t# generate a temporary file / file to write to\n+\tmy ($fh, $tempfile) = $self->_tempfile_to_path($path, $dir);\n+\n+\t# code writes to filehandle or file\n+\t$code->($fh, $tempfile);\n+\n+\tclose $fh;\n+\trename($tempfile, $path)\n+\t\tor die \"Couldn't rename temporary file '$tempfile' to '$path': $!\";\n+\n+\topen $fh, '<', $path or return;\n+\treturn ($fh, $path);\n+}\n+\n+# ======================================================================\n+# ......................................................................\n+# interface methods\n+#\n+# note that only those methods use 'on_error' handler;\n+# all the rest just use \"die\"\n+\n+# Removing and expiring\n+\n+# $cache->remove($key)\n+#\n+# Remove the data associated with the $key from the cache.\n+sub remove {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $file = $self->path_to_key($key)\n+\t\tor return;\n+\treturn unless -f $file;\n+\tunlink($file)\n+\t\tor $self->_handle_error(\"Couldn't remove cache entry file '$file' for key '$key': $!\");\n+}\n+\n+# $cache->is_valid($key[, $expires_in])\n+#\n+# Returns a boolean indicating whether $key exists in the cache\n+# and has not expired. Uses global per-cache expires time, unless\n+# passed optional $expires_in argument.\n+sub is_valid {\n+\tmy ($self, $key, $expires_in) = @_;\n+\n+\tmy $path = $self->path_to_key($key);\n+\n+\t# does file exists in cache?\n+\treturn 0 unless -f $path;\n+\t# get its modification time\n+\tmy $mtime = (stat(_))[9] # _ to reuse stat structure used in -f test\n+\t\tor $self->_handle_error(\"Couldn't stat file '$path' for key '$key': $!\");\n+\n+\t# expire time can be set to never\n+\t$expires_in = defined $expires_in ? $expires_in : $self->get_expires_in();\n+\treturn 1 unless (defined $expires_in && $expires_in >= 0);\n+\n+\t# is file expired?\n+\tmy $now = time();\n+\n+\treturn (($now - $mtime) < $expires_in);\n+}\n+\n+# Getting and setting\n+\n+# ($fh, $filename) = $cache->compute_fh($key, $code);\n+#\n+# Combines the get and set operations in a single call. Attempts to\n+# get $key; if successful, returns the filehandle it can be read from.\n+# Otherwise, calls $code passing filehandle to write to as a\n+# parameter; contents of this file is then used as the new value for\n+# $key; returns filehandle from which one can read newly generated data.\n+#\n+# Uses file locking to have only one process updating value for $key\n+# to avoid 'cache miss stampede' (aka 'stampeding herd') problem.\n+sub compute_fh {\n+\tmy ($self, $key, $code_fh) = @_;\n+\n+\tmy @result = eval { $self->get_fh($key) };\n+\treturn @result if @result;\n+\t$self->_handle_error($@) if $@;\n+\n+\tmy $lockfile = $self->get_lockname($key);\n+\n+\t# this loop is to protect against situation where process that\n+\t# acquired exclusive lock (writer) dies or exits\n+\t# before writing data to cache\n+\tmy $lock_state; # needed for loop condition\n+\tdo {\n+\t\topen my $lock_fh, '+>', $lockfile\n+\t\t\tor $self->_handle_error(\"Could't open lockfile '$lockfile': $!\");\n+\n+\t\t$lock_state = flock($lock_fh, LOCK_EX | LOCK_NB);\n+\t\tif ($lock_state) {\n+\t\t\t## acquired writers lock, have to generate data\n+\t\t\t@result = eval { $self->set_coderef_fh($key, $code_fh) };\n+\t\t\t$self->_handle_error($@) if $@;\n+\n+\t\t\t# closing lockfile releases writer lock\n+\t\t\tflock($lock_fh, LOCK_UN);\n+\t\t\tclose $lock_fh\n+\t\t\t\tor $self->_handle_error(\"Could't close lockfile '$lockfile': $!\");\n+\n+\t\t} else {\n+\t\t\t## didn't acquire writers lock, get stale data or wait for regeneration\n+\n+\t\t\t# try to retrieve stale data\n+\t\t\teval {\n+\t\t\t\t@result = $self->get_fh($key,\n+\t\t\t\t\t'expires_in' => $self->get_max_lifetime());\n+\t\t\t};\n+\t\t\treturn @result if @result;\n+\t\t\t$self->_handle_error($@) if $@;\n+\n+\t\t\t# wait for regeneration if no stale data to serve,\n+\t\t\t# using shared / readers lock to sync (wait for data)\n+\t\t\t@result = eval {\n+\t\t\t\t$self->_wait_for_data($key, sub {\n+\t\t\t\t\tflock($lock_fh, LOCK_SH);\n+\t\t\t\t});\n+\t\t\t};\n+\t\t\t$self->_handle_error($@) if $@;\n+\t\t\t# closing lockfile releases readers lock\n+\t\t\tflock($lock_fh, LOCK_UN);\n+\t\t\tclose $lock_fh\n+\t\t\t\tor $self->_handle_error(\"Could't close lockfile '$lockfile': $!\");\n+\n+\t\t}\n+\t} until (@result || $lock_state);\n+\t# repeat until we have data, or we tried generating data oneself and failed\n+\treturn @result;\n+}\n+\n+\n+1;\n+__END__\n+# end of package GitwebCache::FileCacheWithLocking;\ndiff --git a/t/t9511-gitweb-caching-interface.sh b/t/t9511-gitweb-caching-interface.sh\nnew file mode 100755\nindex 0000000..d8fc946\n--- /dev/null\n+++ b/t/t9511-gitweb-caching-interface.sh\n@@ -0,0 +1,34 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Jakub Narebski\n+#\n+\n+test_description='gitweb caching interface\n+\n+This test checks caching interface used in gitweb caching, and caching\n+infrastructure (GitwebCache::* modules).'\n+\n+# for now we are running only cache interface tests\n+. ./test-lib.sh\n+\n+# this test is present in gitweb-lib.sh\n+if ! test_have_prereq PERL; then\n+\tskip_all='perl not available, skipping test'\n+\ttest_done\n+fi\n+\n+\"$PERL_PATH\" -MTest::More -e 0 >/dev/null 2>&1 || {\n+\tskip_all='perl module Test::More unavailable, skipping test'\n+\ttest_done\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+# The external test will outputs its own plan\n+test_external_has_tap=1\n+\n+test_external \\\n+\t'GitwebCache::*Cache* Perl API (in gitweb/lib/)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9511/test_cache_interface.pl\n+\n+test_done\ndiff --git a/t/t9511/test_cache_interface.pl b/t/t9511/test_cache_interface.pl\nnew file mode 100755\nindex 0000000..a2b006c\n--- /dev/null\n+++ b/t/t9511/test_cache_interface.pl\n@@ -0,0 +1,381 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use POSIX qw(dup2);\n+use Fcntl qw(:DEFAULT);\n+use IO::Handle;\n+use IO::Select;\n+use IO::Pipe;\n+use File::Basename;\n+\n+use Test::More;\n+\n+# test installed version or source version\n+use lib $ENV{GITWEBLIBDIR} || \"$ENV{GIT_BUILD_DIR}/gitweb/lib\";\n+\n+\n+# Test creating a cache\n+#\n+BEGIN { use_ok('GitwebCache::FileCacheWithLocking'); }\n+note(\"Using lib '$INC[0]'\");\n+note(\"Testing '$INC{'GitwebCache/FileCacheWithLocking.pm'}'\");\n+\n+my $cache = new_ok('GitwebCache::FileCacheWithLocking');\n+\n+# Test that default values are defined\n+#\n+ok(defined $GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_ROOT,\n+\t'$GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_ROOT defined');\n+ok(defined $GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_DEPTH,\n+\t'$GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_DEPTH defined');\n+\n+# Test some accessors and some default values for cache\n+#\n+SKIP: {\n+\tskip 'default values not defined', 2\n+\t\tunless ($GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_ROOT &&\n+\t\t $GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_DEPTH);\n+\n+\tcmp_ok($cache->get_root(), 'eq', $GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_ROOT,\n+\t\t\"default cache root is '$GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_ROOT'\");\n+\tcmp_ok($cache->get_depth(), '==', $GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_DEPTH,\n+\t\t\"default cache depth is $GitwebCache::FileCacheWithLocking::DEFAULT_CACHE_DEPTH\");\n+}\n+\n+# Test the getting and setting of a cached value,\n+# and removal of a cached value\n+#\n+my $key = 'Test Key';\n+my $value = 'Test Value';\n+\n+my $call_count = 0;\n+sub get_value_fh {\n+\tmy $fh = shift;\n+\t$call_count++;\n+\tprint {$fh} $value;\n+}\n+\n+# use ->compute_fh($key, $code_fh) interface\n+sub cache_compute_fh {\n+\tmy ($cache, $key, $code_fh) = @_;\n+\n+\tmy ($fh, $filename) = $cache->compute_fh($key, $code_fh);\n+\treturn unless $fh;\n+\n+\tlocal $/ = undef;\n+\treturn <$fh>;\n+}\n+\n+# use ->get_fh($key) interface\n+sub cache_get_fh {\n+\tmy ($cache, $key) = @_;\n+\n+\tmy ($fh, $filename) = $cache->get_fh($key);\n+\treturn unless $fh;\n+\n+\tlocal $/ = undef;\n+\treturn <$fh>;\n+}\n+\n+# use ->set_coderef_fh($key, $code_fh) to set $key to $value\n+sub cache_set_fh {\n+\tmy ($cache, $key, $value) = @_;\n+\n+\t$cache->set_coderef_fh($key, sub { print {$_[0]} $value });\n+\treturn $value;\n+}\n+\n+subtest 'compute_fh interface' => sub {\n+\tforeach my $method (qw(remove compute_fh)) {\n+\t\tcan_ok($cache, $method);\n+\t}\n+\n+\teval { $cache->remove('Not-Existent Key'); };\n+\tok(!$@, 'remove on non-existent key doesn\\'t die');\n+\tdiag($@) if $@;\n+\n+\t$cache->remove($key); # just in case\n+\tis(cache_compute_fh($cache, $key, \\&get_value_fh), $value,\n+\t \"compute_fh 1st time (set) returns '$value'\");\n+\tis(cache_compute_fh($cache, $key, \\&get_value_fh), $value,\n+\t \"compute_fh 2nd time (get) returns '$value'\");\n+\tis(cache_compute_fh($cache, $key, \\&get_value_fh), $value,\n+\t \"compute_fh 3rd time (get) returns '$value'\");\n+\tcmp_ok($call_count, '==', 1, 'get_value_fh() is called once from compute_fh');\n+\n+\tdone_testing();\n+};\n+\n+\n+# Test cache expiration\n+#\n+subtest 'cache expiration' => sub {\n+\t$cache->set_expires_in(60*60*24); # set expire time to 1 day\n+\tcmp_ok($cache->get_expires_in(), '>', 0, '\"expires in\" is greater than 0 (set to 1d)');\n+\t$call_count = 0;\n+\tcache_compute_fh($cache, $key, \\&get_value_fh);\n+\tcmp_ok($call_count, '==', 0, 'compute_fh didn\\'t need to compute data (not expired in 1d)');\n+\tis(cache_get_fh($cache, $key), $value, 'get_fh returns cached value (not expired in 1d)');\n+\n+\t$cache->set_expires_in(-1); # set expire time to never expire\n+\tis($cache->get_expires_in(), -1, '\"expires in\" is set to never (-1)');\n+\tis(cache_get_fh($cache, $key), $value, 'get returns cached value (not expired)');\n+\n+\t$cache->set_expires_in(0);\n+\tis($cache->get_expires_in(), 0, '\"expires in\" is set to now (0)');\n+\tok(!defined(cache_get_fh($cache, $key)), 'cache is expired, get_fh returns undef');\n+\tcache_compute_fh($cache, $key, \\&get_value_fh);\n+\tcmp_ok($call_count, '==', 1, 'compute_fh computed and set data');\n+\n+\tdone_testing();\n+};\n+\n+\n+# ----------------------------------------------------------------------\n+# CONCURRENT ACCESS\n+sub parallel_run (&); # forward declaration of prototype\n+\n+# Test 'stampeding herd' / 'cache miss stampede' problem\n+#\n+my $slow_time = 1; # how many seconds to sleep in mockup of slow generation\n+sub get_value_slow_fh {\n+\tmy $fh = shift;\n+\n+\t$call_count++;\n+\tsleep $slow_time;\n+\tprint {$fh} $value;\n+}\n+sub get_value_die {\n+\t$call_count++;\n+\tdie \"get_value_die\\n\";\n+}\n+my $lock_file = \"$0.$$.lock\"; # if exists then get_value_die_once_fh was already called\n+sub get_value_die_once_fh {\n+\tif (sysopen my $lock_fh, $lock_file, (O_WRONLY | O_CREAT | O_EXCL)) {\n+\t\tclose $lock_fh;\n+\t\tdie \"get_value_die_once_fh\\n\";\n+\t} else {\n+\t\tget_value_slow_fh(@_);\n+\t}\n+}\n+\n+my @output; # gathers output from concurrent invocations\n+my $sep = '|'; # separate different parts of data for tests\n+my $total_count = 0; # number of calls around all concurrent invocations\n+\n+note(\"Following tests contain artifical delay of $slow_time seconds\");\n+subtest 'parallel access' => sub {\n+\n+\t$cache->remove($key);\n+\t@output = parallel_run {\n+\t\t$call_count = 0;\n+\t\tmy $data = cache_compute_fh($cache, $key, \\&get_value_slow_fh);\n+\t\tprint $data if defined $data;\n+\t\tprint \"$sep$call_count\";\n+\t};\n+\t$total_count = 0;\n+\tforeach (@output) {\n+\t\tmy ($child_out, $child_count) = split(quotemeta $sep, $_);\n+\t\t$total_count += $child_count;\n+\t}\n+\tcmp_ok($total_count, '==', 1, 'parallel compute_fh: get_value_slow_fh() called only once');\n+\t# extract only data, without child count\n+\t@output = map { s/\\Q$sep\\E.*$//; $_ } @output;\n+\tis_deeply(\n+\t\t\\@output,\n+\t\t[ ($value) x 2 ],\n+\t\t\"parallel compute_fh: both returned '$value'\"\n+\t);\n+\n+\t$cache->set_on_error(sub { die @_; });\n+\teval {\n+\t\tlocal $SIG{ALRM} = sub { die \"alarm\\n\"; };\n+\t\talarm 4*$slow_time;\n+\n+\t\t@output = parallel_run {\n+\t\t\t$call_count = 0;\n+\t\t\tmy $data = eval { cache_compute_fh($cache, 'No Key', \\&get_value_die); };\n+\t\t\tmy $eval_error = $@;\n+\t\t\tprint \"$data\" if defined $data;\n+\t\t\tprint \"$sep\";\n+\t\t\tprint \"$eval_error\" if $eval_error;\n+\t\t};\n+\t\tis_deeply(\n+\t\t\t\\@output,\n+\t\t\t[ ( \"${sep}get_value_die\\n\" ) x 2 ],\n+\t\t\t'parallel compute_fh: get_value_die() died in both'\n+\t\t);\n+\n+\t\talarm 0;\n+\t};\n+\tok(!$@, 'parallel compute_fh: no alarm call (neither process hung)');\n+\tdiag($@) if $@;\n+\n+\t$cache->remove($key);\n+\tunlink($lock_file);\n+\t@output = parallel_run {\n+\t\tmy $data = eval { cache_compute_fh($cache, $key, \\&get_value_die_once_fh); };\n+\t\tmy $eval_error = $@;\n+\t\tprint \"$data\" if defined $data;\n+\t\tprint \"$sep\";\n+\t\tprint \"$eval_error\" if $eval_error;\n+\t};\n+\tis_deeply(\n+\t\t[sort @output],\n+\t\t[sort (\"$value$sep\", \"${sep}get_value_die_once_fh\\n\")],\n+\t\t'parallel compute_fh: return correct value even if other process died'\n+\t);\n+\tunlink($lock_file);\n+\n+\tdone_testing();\n+};\n+\n+\n+# Test that cache returns stale data in existing but expired cache situation\n+#\n+my $stale_value = 'Stale Value';\n+\n+subtest 'serving stale data when regenerating' => sub {\n+\tcache_set_fh($cache, $key, $stale_value);\n+\t$cache->set_expires_in(-1); # never expire, for next check\n+\tis(cache_get_fh($cache, $key), $stale_value,\n+\t 'stale value set (prepared) correctly');\n+\n+\t$call_count = 0;\n+\t$cache->set_expires_in(0); # expire now (so there are no fresh data)\n+\t$cache->set_max_lifetime(-1); # forever (always serve stale data)\n+\n+\t@output = parallel_run {\n+\t\tmy $data = cache_compute_fh($cache, $key, \\&get_value_slow_fh);\n+\t\tprint \"$call_count$sep\";\n+\t\tprint $data if defined $data;\n+\t};\n+\t# returning stale data works\n+\tis_deeply(\n+\t\t[sort @output],\n+\t\t[sort (\"0$sep$stale_value\", \"1$sep$value\")],\n+\t\t'no background: stale data returned by one process (the one not generating data)'\n+\t);\n+\t$cache->set_expires_in(-1); # never expire for next ->get\n+\tis(cache_get_fh($cache, $key), $value,\n+\t 'no background: value got set correctly, even if stale data returned');\n+\n+\n+\tcache_set_fh($cache, $key, $stale_value);\n+\t$cache->set_expires_in(0); # expire now\n+\t$cache->set_max_lifetime(0); # don't serve stale data\n+\n+\t@output = parallel_run {\n+\t\tmy $data = cache_compute_fh($cache, $key, \\&get_value_slow_fh);\n+\t\tprint $data;\n+\t};\n+\t# no returning stale data\n+\tok(!scalar(grep { $_ eq $stale_value } @output),\n+\t 'no stale data if configured');\n+\n+\n+\tdone_testing();\n+};\n+$cache->set_expires_in(-1);\n+\n+\n+done_testing();\n+\n+\n+#######################################################################\n+#######################################################################\n+#######################################################################\n+\n+# from http://aaroncrane.co.uk/talks/pipes_and_processes/\n+sub fork_child (&) {\n+\tmy ($child_process_code) = @_;\n+\n+\tmy $pid = fork();\n+\tdie \"Failed to fork: $!\\n\" if !defined $pid;\n+\n+\treturn $pid if $pid != 0;\n+\n+\t# Now we're in the new child process\n+\t$child_process_code->();\n+\texit;\n+}\n+\n+sub parallel_run (&) {\n+\tmy $child_code = shift;\n+\tmy $nchildren = 2;\n+\n+\tmy %children;\n+\tmy (%pid_for_child, %fd_for_child);\n+\tmy $sel = IO::Select->new();\n+\tforeach my $child_idx (1..$nchildren) {\n+\t\tmy $pipe = IO::Pipe->new()\n+\t\t\tor die \"Failed to create pipe: $!\\n\";\n+\n+\t\tmy $pid = fork_child {\n+\t\t\t$pipe->writer()\n+\t\t\t\tor die \"$$: Child \\$pipe->writer(): $!\\n\";\n+\t\t\tdup2(fileno($pipe), fileno(STDOUT))\n+\t\t\t\tor die \"$$: Child $child_idx failed to reopen stdout to pipe: $!\\n\";\n+\t\t\tclose $pipe\n+\t\t\t\tor die \"$$: Child $child_idx failed to close pipe: $!\\n\";\n+\n+\t\t\t# From Test-Simple-0.96/t/subtest/fork.t\n+\t\t\t#\n+\t\t\t# Force all T::B output into the pipe (redirected to STDOUT),\n+\t\t\t# for the parent builder as well as the current subtest builder.\n+\t\t\t{\n+\t\t\t\tno warnings 'redefine';\n+\t\t\t\t*Test::Builder::output = sub { *STDOUT };\n+\t\t\t\t*Test::Builder::failure_output = sub { *STDOUT };\n+\t\t\t\t*Test::Builder::todo_output = sub { *STDOUT };\n+\t\t\t}\n+\n+\t\t\t$child_code->();\n+\n+\t\t\t*STDOUT->flush();\n+\t\t\tclose(STDOUT);\n+\t\t};\n+\n+\t\t$pid_for_child{$pid} = $child_idx;\n+\t\t$pipe->reader()\n+\t\t\tor die \"Failed to \\$pipe->reader(): $!\\n\";\n+\t\t$fd_for_child{$pipe} = $child_idx;\n+\t\t$sel->add($pipe);\n+\n+\t\t$children{$child_idx} = {\n+\t\t\t'pid' => $pid,\n+\t\t\t'stdout' => $pipe,\n+\t\t\t'output' => '',\n+\t\t};\n+\t}\n+\n+\twhile (my @ready = $sel->can_read()) {\n+\t\tforeach my $fh (@ready) {\n+\t\t\tmy $buf = '';\n+\t\t\tmy $nread = sysread($fh, $buf, 1024);\n+\n+\t\t\texists $fd_for_child{$fh}\n+\t\t\t\tor die \"Cannot find child for fd: $fh\\n\";\n+\n+\t\t\tif ($nread > 0) {\n+\t\t\t\t$children{$fd_for_child{$fh}}{'output'} .= $buf;\n+\t\t\t} else {\n+\t\t\t\t$sel->remove($fh);\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\twhile (%pid_for_child) {\n+\t\tmy $pid = waitpid -1, 0;\n+\t\twarn \"Child $pid_for_child{$pid} ($pid) failed with status: $?\\n\"\n+\t\t\tif $? != 0;\n+\t\tdelete $pid_for_child{$pid};\n+\t}\n+\n+\treturn map { $children{$_}{'output'} } keys %children;\n+}\n+\n+__END__\n"},{"id":"158510","messageId":"20101222235757.7998.65738.stgit@localhost.localdomain","threadId":"26127","inReplyTo":"20101222234843.7998.87068.stgit@localhost.localdomain","subject":"[RFC PATCH v7 8/9] gitweb/lib - Cache captured output (using compute_fh)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:57:58Z","receivedAt":"2010-12-22T23:57:58Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\n\nAdd GitwebCache::CacheOutput package, which introduces cache_output\nsubroutine. If data for given key is present in cache, then\ncache_output gets data from cache and prints it. If data is not present\nin cache, then cache_output runs provided subroutine (code reference),\ncaptures its output, saves this output in cache, and prints it.\n\nIt requires that provided $cache supports ->capture_fh method, like\nGitwebCache::FileCacheWithLocking introduced in earlier commit, and that\nprovided $capture supports capturing to file or filehandle via\n->capture($code, $file) method, like GitwebCache::Capture::ToFile\nintroduced in some earlier commit.\n\nExceptions in $code should be thrown using 'die' (Perl exception\nmechanism); one can choose whether error output (output printed when\nexception is raised, before raising it) should be saved to cache or not.\nBy default error output is not cached.\n\nGitweb would use cache_output to get page from cache, or to generate\npage and save it to cache. The die_error subroutine throws exception,\nwhich will be caught and by default rethrown; error pages would not be\ncached.\n\nIt is assumed that data is saved to cache _converted_, and should\ntherefore be read from cache and printed to STDOUT in ':raw' (binary)\nmode.\n\n\nAdd t9512/test_cache_output.pl test, run as external test in\nt9512-gitweb-cache. It checks that cache_output behaves correctly,\nnamely that it saves and restores action output in cache, and that it\nprints generated output or cached output, depending on whether there\nexist data in cache.\n\nSigned-off-by: Jakub Narebski \n---\n\n gitweb/lib/GitwebCache/CacheOutput.pm | 84 ++++++++++++++++\n t/t9512-gitweb-cache-output-interface.sh | 34 ++++++\n t/t9512/test_cache_output.pl | 162 ++++++++++++++++++++++++++++++\n 3 files changed, 280 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm\n create mode 100755 t/t9512-gitweb-cache-output-interface.sh\n create mode 100755 t/t9512/test_cache_output.pl\n\n\ndiff --git a/gitweb/lib/GitwebCache/CacheOutput.pm b/gitweb/lib/GitwebCache/CacheOutput.pm\nnew file mode 100644\nindex 0000000..4a75a7f\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/CacheOutput.pm\n@@ -0,0 +1,84 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+# (C) 2006, John 'Warthog9' Hawley \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Capturing and caching (gitweb) output\n+#\n+\n+# Capture output, save it in cache and print it, or retrieve it from\n+# cache and print it.\n+\n+package GitwebCache::CacheOutput;\n+\n+use strict;\n+use warnings;\n+\n+use File::Copy qw();\n+use Symbol qw(qualify_to_ref);\n+\n+use Exporter qw(import);\n+our @EXPORT = qw(cache_output);\n+our %EXPORT_TAGS = (all => [ @EXPORT ]);\n+\n+# cache_output($cache, $capture, $key, $action_code, [ option => value ]);\n+#\n+# Attempts to get $key from $cache; if successful, prints the value.\n+# Otherwise, calls $action_code, capture its output using $capture,\n+# and use the captured output as the new value for $key in $cache,\n+# then print captured output.\n+#\n+# It is assumed that captured data is already converted and it is\n+# in ':raw' format (and thus restored in ':raw' from cache)\n+#\n+# Supported options:\n+# * -cache_errors => 0|1 - whether error output should be cached\n+sub cache_output {\n+\tmy ($cache, $capture, $key, $code, %opts) = @_;\n+\n+\tmy ($fh, $filename);\n+\tmy ($capture_fh, $capture_filename);\n+\teval { # this `eval` is to catch rethrown error, so we can print captured output\n+\t\t($fh, $filename) = $cache->compute_fh($key, sub {\n+\t\t\t($capture_fh, $capture_filename) = @_;\n+\n+\t\t\t# this `eval` is to be able to cache error output (up till 'die')\n+\t\t\teval { $capture->capture($code, $capture_fh); };\n+\n+\t\t\t# note that $cache can catch this error itself (like e.g. CHI);\n+\t\t\t# use \"die\"-ing error handler to rethrow this exception to outside\n+\t\t\tdie $@ if ($@ && ! $opts{'-cache_errors'});\n+\t\t});\n+\t};\n+\tmy $error = $@;\n+\n+\t# if an exception was rethrown, and not caught by caching engine (by $cache)\n+\t# then ->compute_fh will not set $fh nor $filename; use those used for capture\n+\tif (!defined $fh) {\n+\t\t$filename ||= $capture_filename;\n+\t}\n+\n+\tif (defined $fh || defined $filename) {\n+\t\t# set binmode only if $fh is defined (is a filehandle)\n+\t\t# File::Copy::copy opens files given by filename in binary mode\n+\t\tbinmode $fh, ':raw' if (defined $h);\n+\t\tbinmode STDOUT, ':raw';\n+\t\tFile::Copy::copy($fh || $filename, \\*STDOUT);\n+\t}\n+\n+\t# rethrow error if captured in outer `eval` (i.e. no -cache_errors),\n+\t# removing temporary file (exception thrown out of cache)\n+\tif ($error) {\n+\t\tunlink $capture_filename\n+\t\t\tif (defined $capture_filename && -e $capture_filename);\n+\t\tdie $error;\n+\t}\n+\treturn;\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::CacheOutput\ndiff --git a/t/t9512-gitweb-cache-output-interface.sh b/t/t9512-gitweb-cache-output-interface.sh\nnew file mode 100755\nindex 0000000..fb9525a\n--- /dev/null\n+++ b/t/t9512-gitweb-cache-output-interface.sh\n@@ -0,0 +1,34 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Jakub Narebski\n+#\n+\n+test_description='gitweb cache\n+\n+This test checks GitwebCache::CacheOutput Perl module that is\n+responsible for capturing and caching gitweb output.'\n+\n+# for now we are running only cache interface tests\n+. ./test-lib.sh\n+\n+# this test is present in gitweb-lib.sh\n+if ! test_have_prereq PERL; then\n+\tskip_all='perl not available, skipping test'\n+\ttest_done\n+fi\n+\n+\"$PERL_PATH\" -MTest::More -e 0 >/dev/null 2>&1 || {\n+\tskip_all='perl module Test::More unavailable, skipping test'\n+\ttest_done\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+# The external test will outputs its own plan\n+test_external_has_tap=1\n+\n+test_external \\\n+\t'GitwebCache::CacheOutput Perl API (in gitweb/lib/)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9512/test_cache_output.pl\n+\n+test_done\ndiff --git a/t/t9512/test_cache_output.pl b/t/t9512/test_cache_output.pl\nnew file mode 100755\nindex 0000000..758848c\n--- /dev/null\n+++ b/t/t9512/test_cache_output.pl\n@@ -0,0 +1,162 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use Test::More;\n+\n+# test source version\n+use lib $ENV{GITWEBLIBDIR} || \"$ENV{GIT_BUILD_DIR}/gitweb/lib\";\n+\n+# ....................................................................\n+\n+# prototypes must be known at compile time, otherwise they do not work\n+BEGIN { use_ok('GitwebCache::CacheOutput'); }\n+\n+require_ok('GitwebCache::FileCacheWithLocking');\n+require_ok('GitwebCache::Capture::ToFile');\n+\n+note(\"Using lib '$INC[0]'\");\n+note(\"Testing '$INC{'GitwebCache/CacheOutput.pm'}'\");\n+note(\"Testing '$INC{'GitwebCache/FileCacheWithLocking.pm'}'\");\n+note(\"Testing '$INC{'GitwebCache/Capture/ToFile.pm'}'\");\n+\n+\n+# Test setting up $cache and $capture\n+my ($cache, $capture);\n+subtest 'setup' => sub {\n+\t$cache = new_ok('GitwebCache::FileCacheWithLocking' => [], 'The $cache ');\n+\t$capture = new_ok('GitwebCache::Capture::ToFile' => [], 'The $capture');\n+\n+\tdone_testing();\n+};\n+\n+# ......................................................................\n+\n+# Prepare for testing cache_output\n+my $key = 'Key';\n+my $action_output = <<'EOF';\n+# This is data to be cached and shown\n+EOF\n+my $cached_output = <<\"EOF\";\n+$action_output# (version recovered from cache)\n+EOF\n+my $call_count = 0;\n+sub action {\n+\t$call_count++;\n+\tprint $action_output;\n+}\n+\n+my $die_output = <<\"EOF\";\n+$action_output# (died)\n+EOF\n+sub die_action {\n+\tprint $die_output;\n+\tdie \"die_action\\n\";\n+}\n+\n+# Catch output printed by cache_output\n+sub capture_output_of_cache_output {\n+\tmy ($code, @args) = @_;\n+\n+\tGitwebCache::Capture::ToFile->new()->capture(sub {\n+\t\tcache_output($cache, $capture, $key, $code, @args);\n+\t}, 'actual');\n+\n+\treturn get_actual();\n+}\n+\n+sub get_actual {\n+\topen my $fh, '<', 'actual' or return;\n+\tlocal $/ = undef;\n+\tmy $result = <$fh>;\n+\tclose $fh;\n+\treturn $result;\n+}\n+\n+# use ->get_fh($key) interface\n+sub cache_get_fh {\n+\tmy ($cache, $key) = @_;\n+\n+\tmy ($fh, $filename) = $cache->get_fh($key);\n+\treturn unless $fh;\n+\n+\tlocal $/ = undef;\n+\treturn <$fh>;\n+}\n+\n+# use ->set_coderef_fh($key, $code_fh) to set $key to $value\n+sub cache_set_fh {\n+\tmy ($cache, $key, $value) = @_;\n+\n+\t$cache->set_coderef_fh($key, sub { print {$_[0]} $value });\n+\treturn $value;\n+}\n+\n+\n+# ......................................................................\n+\n+# clean state\n+$cache->set_expires_in(-1);\n+$cache->remove($key);\n+my $test_data;\n+\n+# first time (if there is no cache) generates cache entry\n+subtest '1st time (generate data)' => sub {\n+\t$call_count = 0;\n+\t$test_data = capture_output_of_cache_output(\\&action);\n+\tis($test_data, $action_output, 'action() output is printed');\n+\tis(cache_get_fh($cache, $key), $action_output, 'action() output is saved in cache');\n+\tcmp_ok($call_count, '==', 1, 'action() was called to generate data');\n+\n+\tdone_testing();\n+};\n+\n+# second time (if cache is set/valid) reads from cache\n+subtest '2nd time (retreve from cache)' => sub {\n+\tcache_set_fh($cache, $key, $cached_output);\n+\t$call_count = 0;\n+\t$test_data = capture_output_of_cache_output(\\&action);\n+\tis(cache_get_fh($cache, $key), $cached_output, 'correct value is prepared in cache');\n+\tis($test_data, $cached_output, 'output is printed from cache');\n+\tcmp_ok($call_count, '==', 0, 'action() was not called');\n+\n+\tdone_testing();\n+};\n+\n+# caching output and error handling\n+subtest 'errors (exceptions) are not cached by default' => sub {\n+\t$cache->remove($key);\n+\tok(!defined cache_get_fh($cache, $key), 'cache is prepared correctly (no data in cache)');\n+\teval {\n+\t\t$test_data = capture_output_of_cache_output(\\&die_action);\n+\t};\n+\tmy $error = $@;\n+\t$test_data = get_actual();\n+\tis($test_data, $die_output, 'output of an error is printed');\n+\tok(!defined cache_get_fh($cache, $key), 'output is not captured and not cached');\n+\tlike($error, qr/^die_action\\n/m, 'exception made it to outside, correctly');\n+\n+\tdone_testing();\n+};\n+\n+subtest 'errors are cached with -cache_errors => 1' => sub {\n+\t$cache->remove($key);\n+\tok(!defined cache_get_fh($cache, $key), 'cache is prepared correctly (no data in cache)');\n+\teval {\n+\t\t$test_data = capture_output_of_cache_output(\\&die_action, -cache_errors => 1);\n+\t};\n+\tmy $error = $@;\n+\t$test_data = get_actual();\n+\tis($test_data, $die_output, 'output of an error is printed');\n+\tis(cache_get_fh($cache, $key), $die_output, 'output is captured and cached');\n+\tok(! $error, 'exception didn\\'t made it to outside');\n+\tdiag($error) if $error;\n+\n+\tdone_testing();\n+};\n+\n+\n+done_testing();\n+__END__\n"},{"id":"158511","messageId":"20101222235823.7998.15358.stgit@localhost.localdomain","threadId":"26127","inReplyTo":"20101222234843.7998.87068.stgit@localhost.localdomain","subject":"[RFC PATCH v7 9/9] gitweb: Add optional output caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-22T23:58:24Z","receivedAt":"2010-12-22T23:58:24Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\n\nThis commit actually adds output caching to gitweb, as we have now\nminimal features required for it in GitwebCache::FileCacheWithLocking\n(a 'dumb' but fast file-based cache engine). To enable cache you need\n(at least) set $caching_enabled to true in gitweb config, and copy\nrequired modules alongside generated gitweb.cgi - this is described\nin more detail in the new \"Gitweb caching\" section in gitweb/README.\n\"make install-gitweb\" would install all modules alongside gitweb\nitself.\n\nCapturing and caching is designed in such way that there is no\nbehaviour change if $caching_enabled is false. If caching is not\nenabled, then capturing is also turned off.\n\nEnabling caching causes the following additional changes to gitweb\noutput:\n* Disables content-type negotiation (choosing between 'text/html'\n mimetype and 'application/xhtml+xml') when caching, as there is no\n content-type negotiation done when retrieving page from cache.\n Use lowest common denominator of 'text/html' mimetype which can\n be used by all browsers. This may change in the future.\n* Disable optional timing info (how much time it took to generate the\n original page, and how many git commands it took), and in its place show\n unconditionally when page was originally generated (in GMT / UTC\n timezone).\n* Disable 'blame_incremental' view, as it doesn't make sense without\n printing data as soon as it is generated (which would require tee-ing\n when capturing output for caching)... and it doesn't work currently\n anyway. Alternate solution would be to run 'blame_incremental' view\n with caching disabled.\n\n\nAdd basic tests of caching support to t9500-gitweb-standalone-no-errors\ntest: set $caching_enabled to true and check for errors for first time\nrun (generating cache) and second time run (retrieving from cache) for a\nsingle view - summary view for a project.\n\nCheck in the t9501-gitweb-standalone-http-status test that gitweb at\nleast correctly handles \"404 Not Found\" error pages also in the case\nwhen gitweb caching is enabled.\n\nCheck in the t9502-gitweb-standalone-parse-output test that gitweb\nproduces the same output with and without caching, for first and\nsecond run, with binary or text output.\n\nAll those tests make use of new gitweb_enable_caching subroutine added\nto gitweb-lib.sh\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\n\n gitweb/Makefile | 5 +\n gitweb/README | 46 +++++++\n gitweb/gitweb.perl | 190 ++++++++++++++++++++++++++---\n gitweb/lib/GitwebCache/CacheOutput.pm | 2 \n t/gitweb-lib.sh | 11 ++\n t/t9500-gitweb-standalone-no-errors.sh | 20 +++\n t/t9501-gitweb-standalone-http-status.sh | 13 ++\n t/t9502-gitweb-standalone-parse-output.sh | 33 +++++\n 8 files changed, 299 insertions(+), 21 deletions(-)\n mode change 100644 => 100755 t/gitweb-lib.sh\n\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex e6029e1..d67c138 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -113,6 +113,11 @@ endif\n \n GITWEB_FILES += static/git-logo.png static/git-favicon.png\n \n+# gitweb output caching\n+GITWEB_MODULES += GitwebCache/CacheOutput.pm\n+GITWEB_MODULES += GitwebCache/SimpleFileCache.pm\n+GITWEB_MODULES += GitwebCache/Capture/Simple.pm\n+\n GITWEB_REPLACE = \\\n \t-e 's|++GIT_VERSION++|$(GIT_VERSION)|g' \\\n \t-e 's|++GIT_BINDIR++|$(bindir)|g' \\\ndiff --git a/gitweb/README b/gitweb/README\nindex 4a67393..efe3b2c 100644\n--- a/gitweb/README\n+++ b/gitweb/README\n@@ -258,6 +258,12 @@ not include variables usually directly set during build):\n their default values before every request, so if you want to change\n them, be sure to set this variable to true or a code reference effecting\n the desired changes. The default is true.\n+ * $caching_enabled\n+ If true, gitweb would use caching to speed up generating response.\n+ Currently supported is only output (response) caching. See \"Gitweb caching\"\n+ section below for details on how to configure and customize caching.\n+ The default is false (caching is disabled).\n+\n \n Projects list file format\n ~~~~~~~~~~~~~~~~~~~~~~~~~\n@@ -329,6 +335,46 @@ You can use the following files in repository:\n descriptions.\n \n \n+Gitweb caching\n+~~~~~~~~~~~~~~\n+\n+Currently gitweb supports only output (HTTP response) caching, similar\n+to the one used on http://git.kernel.org. To turn it on, set \n+$caching_enabled variable to true value in gitweb config file, i.e.:\n+\n+ our $caching_enabled = 1;\n+\n+You can choose which caching engine should gitweb use by setting\n+$cache variable to _initialized_ instance of cache interface, or to\n+the name of cache class.\n+\n+Currenly though only cache which implements non-standard ->compute_fh()\n+method is supported. Provided GitwebCache::FileCacheWithLocking implements\n+this method; it is the default caching engine used if $cache is not defined.\n+\n+The GitwebCache::FileCacheWithLocking is 'dumb' (but fast) file based\n+caching engine, currently without any support for cache size limiting, or\n+even removing expired / grossly expired entries. It has therefore the\n+downside of requiring a huge amount of disk space if there are a number of\n+repositories involved. It is not uncommon for git.kernel.org to have on the\n+order of 80G - 120G accumulate over the course of a few months. It is\n+therefore recommended that the cache directory be periodically completely\n+deleted; this operation is safe to perform. Suggested mechanism (substitute\n+$cachedir for actual path to gitweb cache):\n+\n+ # mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush\n+\n+Site-wide cache options are defined in %cache_options hash. Those options\n+apply only when $cache is unset (GitwebCache::FileCacheWithLocking is used),\n+or if $cache is name of cache class. You can override cache options in\n+gitweb config, e.g.:\n+\n+ $cache_options{'expires_in'} = 60; # 60 seconds = 1 minute\n+\n+Please read comments for %cache_options entries in gitweb/gitweb.perl for\n+description of available cache options.\n+\n+\n Webserver configuration\n -----------------------\n \ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 880fdf2..eb02b6b 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -268,6 +268,71 @@ our %highlight_ext = (\n \tmap { $_ => 'xml' } qw(xhtml html htm),\n );\n \n+\n+# This enables/disables the caching layer in gitweb. Currently supported\n+# is only output (response) caching, similar to the one used on git.kernel.org.\n+our $caching_enabled = 0;\n+# Set to _initialized_ instance of cache interface implementing (for now)\n+# compute_fh($key, $code) method (non-standard CHI-inspired interface),\n+# or to name of class of cache interface implementing said method.\n+# If unset, GitwebCache::FileCacheWithLocking would be used, which is 'dumb'\n+# (but fast) file based caching layer, currently without any support for\n+# cache size limiting. It is therefore recommended that the cache directory\n+# be periodically completely deleted; this operation is safe to perform.\n+#\n+# Suggested mechanism to clear cache:\n+# mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush\n+# where $cachedir is directory where cache is, i.e. $cache_options{'cache_root'}\n+our $cache;\n+# You define site-wide cache options defaults here; override them with\n+# $GITWEB_CONFIG as necessary.\n+our %cache_options = (\n+\t# The location in the filesystem that will hold the root of the cache.\n+\t# This directory will be created as needed (if possible) on the first\n+\t# cache set. Note that either this directory must exists and web server\n+\t# has to have write permissions to it, or web server must be able to\n+\t# create this directory.\n+\t# Possible values:\n+\t# * 'cache' (relative to gitweb),\n+\t# * File::Spec->catdir(File::Spec->tmpdir(), 'gitweb-cache'),\n+\t# * '/var/cache/gitweb' (FHS compliant, requires being set up),\n+\t'cache_root' => 'cache',\n+\n+\t# The number of subdirectories deep to cache object item. This should be\n+\t# large enough that no cache directory has more than a few hundred\n+\t# objects. Each non-leaf directory contains up to 256 subdirectories\n+\t# (00-ff). Must be larger than 0.\n+\t'cache_depth' => 1,\n+\n+\t# The (global) expiration time for objects placed in the cache, in seconds.\n+\t'expires_in' => 20,\n+\n+\t# How to handle runtime errors occurring during cache gets and cache\n+\t# sets. Options are:\n+\t# * \"die\" (the default) - call die() with an appropriate message\n+\t# * \"warn\" - call warn() with an appropriate message\n+\t# * \"ignore\" - do nothing\n+\t# * - call this code reference with an appropriate message\n+\t# Note that gitweb catches 'die ' via custom handle_errors_html\n+\t# handler, set via set_message() from CGI::Carp. 'warn ' are\n+\t# written to web server logs.\n+\t#\n+\t# The default is to use cache_error_handler, which wraps die_error.\n+\t# Only first argument passed to cache_error_handler is used (c.f. CHI)\n+\t'on_error' => \\&cache_error_handler,\n+\n+\t# Extra options passed to GitwebCache::CacheOutput::cache_output subroutine\n+\t'cache_output' => {\n+\t\t# Enable caching of error pages (boolean). Default is false.\n+\t\t'-cache_errors' => 0,\n+\t},\n+);\n+# Set to _initialized_ instance of GitwebCache::Capture::ToFile\n+# compatibile capturing engine, i.e. one implementing ->new()\n+# constructor, and ->capture($code, $file) method. If unset\n+# (default), the GitwebCache::Capture::ToFile would be used.\n+our $capture;\n+\n # You define site-wide feature defaults here; override them with\n # $GITWEB_CONFIG as necessary.\n our %feature = (\n@@ -1121,7 +1186,16 @@ sub dispatch {\n \t !$project) {\n \t\tdie_error(400, \"Project needed\");\n \t}\n-\t$actions{$action}->();\n+\n+\tif ($caching_enabled) {\n+\t\t# human readable key identifying gitweb output\n+\t\tmy $output_key = href(-replay => 1, -full => 1, -path_info => 0);\n+\n+\t\tcache_output($cache, $capture, $output_key, $actions{$action},\n+\t\t\t%{$cache_options{'cache_output'}});\n+\t} else {\n+\t\t$actions{$action}->();\n+\t}\n }\n \n sub reset_timer {\n@@ -1147,6 +1221,8 @@ sub run_request {\n \t\t}\n \t}\n \tcheck_loadavg();\n+\tconfigure_caching()\n+\t\tif ($caching_enabled);\n \n \t# $projectroot and $projects_list might be set in gitweb config file\n \t$projects_list ||= $projectroot;\n@@ -1210,7 +1286,7 @@ sub run {\n \t\t\tif $pre_dispatch_hook;\n \n \t\teval { run_request() };\n-\t\tif (defined $@ && !ref($@)) {\n+\t\tif ($@ && !ref($@)) {\n \t\t\t# some Perl error, but not one thrown by die_error\n \t\t\tdie_error(undef, undef, $@, -error_handler => 1);\n \t\t}\n@@ -1227,6 +1303,49 @@ sub run {\n \t1;\n }\n \n+sub configure_caching {\n+\tif (!eval { require GitwebCache::CacheOutput; 1; }) {\n+\t\tdie_error(500,\n+\t\t\t\"Caching enabled and error loading GitwebCache::CacheOutput\",\n+\t\t\tesc_html($@));\n+\n+\t\t# turn off caching and warn instead\n+\t\t#$caching_enabled = 0;\n+\t\t#warn \"Caching enabled and GitwebCache::CacheOutput not found\";\n+\t}\n+\tGitwebCache::CacheOutput->import();\n+\n+\t# $cache might be initialized (instantiated) cache, i.e. cache object,\n+\t# or it might be name of class, or it might be undefined\n+\tunless (defined $cache && ref($cache)) {\n+\t\t$cache ||= 'GitwebCache::FileCacheWithLocking';\n+\t\teval \"require $cache\";\n+\t\tif ($@) {\n+\t\t\tdie_error(500,\n+\t\t\t\t\"Error loading $cache\",\n+\t\t\t\tesc_html($@));\n+\t\t}\n+\n+\t\t$cache = $cache->new({\n+\t\t\t%cache_options,\n+\t\t\t#'cache_root' => '/tmp/cache',\n+\t\t\t#'cache_depth' => 2,\n+\t\t\t#'expires_in' => 20, # in seconds (CHI compatibile)\n+\t\t\t# (Cache::Cache compatibile initialization)\n+\t\t\t'default_expires_in' => $cache_options{'expires_in'},\n+\t\t\t# (CHI compatibile initialization)\n+\t\t\t'root_dir' => $cache_options{'cache_root'},\n+\t\t\t'depth' => $cache_options{'cache_depth'},\n+\t\t\t'on_get_error' => $cache_options{'on_error'},\n+\t\t\t'on_set_error' => $cache_options{'on_error'},\n+\t\t});\n+\t}\n+\tunless (defined $capture && ref($capture)) {\n+\t\trequire GitwebCache::Capture::ToFile;\n+\t\t$capture = GitwebCache::Capture::ToFile->new();\n+\t}\n+}\n+\n run();\n \n if (defined caller) {\n@@ -3597,7 +3716,9 @@ sub git_header_html {\n \t# 'application/xhtml+xml', otherwise send it as plain old 'text/html'.\n \t# we have to do this because MSIE sometimes globs '*/*', pretending to\n \t# support xhtml+xml but choking when it gets what it asked for.\n-\tif (defined $cgi->http('HTTP_ACCEPT') &&\n+\t# Disable content-type negotiation when caching (use mimetype good for all).\n+\tif (!$caching_enabled &&\n+\t defined $cgi->http('HTTP_ACCEPT') &&\n \t $cgi->http('HTTP_ACCEPT') =~ m/(,|;|\\s|^)application\\/xhtml\\+xml(,|;|\\s|$)/ &&\n \t $cgi->Accept('application/xhtml+xml') != 0) {\n \t\t$content_type = 'application/xhtml+xml';\n@@ -3622,7 +3743,9 @@ sub git_header_html {\n EOF\n \t# the stylesheet, favicon etc urls won't work correctly with path_info\n \t# unless we set the appropriate base URL\n-\tif ($ENV{'PATH_INFO'}) {\n+\t# if caching is enabled we can get it from cache for path_info when it\n+\t# is generated without path_info\n+\tif ($ENV{'PATH_INFO'} || $caching_enabled) {\n \t\tprint \"\\n\";\n \t}\n \t# print out each stylesheet that exist, providing backwards capability\n@@ -3739,17 +3862,25 @@ sub git_footer_html {\n \t}\n \tprint \"\\n\"; # class=\"page_footer\"\n \n-\tif (defined $t0 && gitweb_check_feature('timed')) {\n+\t# timing info doesn't make much sense with output (response) caching,\n+\t# so when caching is enabled gitweb prints the time of page generation\n+\tif ((defined $t0 || $caching_enabled) &&\n+\t gitweb_check_feature('timed')) {\n \t\tprint \"
\\n\";\n-\t\tprint 'This page took '.\n-\t\t ''.\n-\t\t tv_interval($t0, [ gettimeofday() ]).\n-\t\t ' seconds '.\n-\t\t ' and '.\n-\t\t ''.\n-\t\t $number_of_git_cmds.\n-\t\t ' git commands '.\n-\t\t \" to generate.\\n\";\n+\t\tif ($caching_enabled) {\n+\t\t\tprint 'This page was generated at '.\n+\t\t\t gmtime( time() ).\" GMT\\n\";\n+\t\t} else {\n+\t\t\tprint 'This page took '.\n+\t\t\t ''.\n+\t\t\t tv_interval($t0, [ gettimeofday() ]).\n+\t\t\t ' seconds '.\n+\t\t\t ' and '.\n+\t\t\t ''.\n+\t\t\t $number_of_git_cmds.\n+\t\t\t ' git commands '.\n+\t\t\t \" to generate.\\n\";\n+\t\t}\n \t\tprint \"
\\n\"; # class=\"page_footer\"\n \t}\n \n@@ -3758,8 +3889,8 @@ sub git_footer_html {\n \t}\n \n \tprint qq!\\n!;\n-\tif (defined $action &&\n-\t $action eq 'blame_incremental') {\n+\tif (!$caching_enabled &&\n+\t defined $action && $action eq 'blame_incremental') {\n \t\tprint qq!