{"thread":{"id":"24107","subject":"[RFC PATCHv4 00/17] gitweb: Simple file based output caching","startedAt":"2010-06-14T16:08:12Z","lastAt":"2010-06-24T07:56:53Z","messageCount":21,"participants":["Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"143661","messageId":"1276531710-22945-1-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":null,"subject":"[RFC PATCHv4 00/17] gitweb: Simple file based output caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:12Z","receivedAt":"2010-06-14T16:08:12Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This 17 patches long patch series is intended (yet again) as preliminary\nversion for splitting large 'gitweb: File based caching layer (from\ngit.kernel.org)' mega-patch by John 'Warthog9' Hawley aka J.H., by starting\nsmall and adding features piece by piece.\n\nThis is fourth version (fourth release) of this series, and is available on\nhttp://repo.or.cz/w/git/jnareb-git.git as 'gitweb/cache-kernel-v4' branch.\nEarlier versions are available there as branches 'gitweb/cache-kernel',\n'gitweb/cache-kernel-v2' and 'gitweb/cache-kernel-v3'. Previous version of\nthis series was sent to git mailing list as:\n\n* [RFC PATCHv3 00/10] gitweb: Simple file based output caching\n Message-Id: <1266349005-15393-1-git-send-email-jnareb@gmail.com>\n http://thread.gmane.org/gmane.comp.version-control.git/140138\n\nThis version is based on top of c0cb4ed (git-instaweb: Configure it to work\nwith new gitweb structure, 2010-05-28), which was merged as 'ps/gitweb-soc'\ninto next in 5db4adf, on 2010-06-03. Next version would be rebased on top\nof 'next', which would include \"sub run\" code in 'jn/gitweb-fastcgi', which\nwould require resolving conflict with this series.\n\nThis series of patches does not include comments on individual patches,\ncomparing them to previous version, and to original series by J.H. The\nnext release of this series, hopefully not as RFC, would include them.\n\n\nThe MAIN DIFFERENCE from previous version is that, inspired by splitting\ngitweb part of \"Splitting gitweb and developing write functionalities\n(Integrated web client for git)\" Google Summer of Code 2010 project by\nPavan Kumar Sankara, the gitweb output caching code now resides in separate\nmodules under 'lib/' subdirectory, instead of being all in 'cache.pm' file.\n\nThe 'gitweb: href(..., -path_info => 0|1)' which began v3 series is already\nmerged in, together with other patches from 'jn/gitweb-caching-prep'.\n\nThe following changes are available in the git repository at:\n\n git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v4\n\n\nAll comments welcome!\n\nShortlog:\n~~~~~~~~~\n 1. gitweb: Return or exit after done serving request\n\n This patch was generated in response to noticing the following kind of\n errors in /var/log/httpd/error_log:\n\n [Sun Jun 13 11:58:02 2010] gitweb.cgi: Subroutine git_atom redefined\n at /var/www/perl/gitweb/gitweb.cgi line 6804.\n\n when running gitweb under mod_perl (with ModPerl::Registry handler).\n After this patch I didn't see such errors... but I failed to reproduce\n this error without this patch.\n\n This patch was sent separately to git mailing list as\n http://thread.gmane.org/gmane.comp.version-control.git/149040\n but should probably be skipped.\n\n 2. gitweb: Fix typo in hash key name in %opts in git_header_html\n\n This patch was sent separately to git mailing list and is already\n present in 'master' as ad709ea (gitweb: Fix typo in hash key name in\n %opts in git_header_html, 2010-06-13).\n\n 3. gitweb: Prepare for splitting gitweb\n\n IMPORTANT!!!\n\n This patch is the main reason this series is CC-ed to Pavan Kumar\n Sankara (student) and Christian Couder (mentor) of \"Integrated web\n client for git\" GSoC2010 project, which includes splitting gitweb.\n\n As you can see later in series it makes it very easy to add new gitweb\n modules which are to be installed. Just add\n\n GITWEB_MODULES += lib/Module/Foo.pm\n\n to gitweb/Makefile for Module::Foo.\n\n 4. gitweb/lib - Very simple file based cache\n 5. gitweb/lib - Stat-based cache expiration\n\n The major change is that the code is now in GitwebCache::SimpleFileCache\n module in gitweb/lib/GitwebCache/SimpleFileCache.pm file, instead of\n being in gitweb/cache.pm file.\n\n Other change is removing using File::Spec to manipulate pathnames,\n using faster concatenation instead. Using File::Spec do not bring us\n extra portability, as gitweb itself assumes '/' as path separator.\n It should be tiny little bit faster without File::Spec::catfile etc.\n\n 6. gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)\n\n This is new patch, not present in previous series. Note that you have\n to run benchmark manually; it is not run by t9503 test.\n\n The t/t9503/benchmark_caching_interface.pl includes example benchmark\n results at the end of the file.\n\n 7. gitweb/lib - Simple select(FH) based output capture\n\n The major change is that the code is now in GitwebCache::Capture and\n GitwebCache::Capture::SelectFH modules which are in *separate* files\n (one file per module), instead of being all in gitweb/cache.pm\n\n 8. gitweb/lib - Alternate ways of capturing output\n\n This is new patch, not present in previous series. It includes new\n test (not run by t9503) and new benchmark.\n\n 9. gitweb/lib - Cache captured output (using get/set)\n\n The major change is that the code is now in GitwebCache::CacheOutput\n module in gitweb/lib/GitwebCache/CacheOutput.pm file.\n\n10. gitweb: Add optional output caching\n\n \"use lib __DIR__.'/lib';\" is now added by earlier patch in series.\n It uses new infrastructure instead of 'bag of modules' cache.pm file.\n\n11. gitweb/lib - Adaptive cache expiration time\n12. gitweb/lib - Use CHI compatible (compute method) caching interface\n\n Similar to what was in previous version of this series.\n\n13. gitweb/lib - Use locking to avoid 'cache miss stampede' problem\n14. gitweb/lib - Serve stale data when waiting for filling cache\n15. gitweb/lib - Regenerate (refresh) cache in background\n\n\n The major change is that the code is now in new (not present in\n previous version) GitwebCache::FileCacheWithLocking module (in its own\n file), which is derived from GitwebCache::SimpleFileCache. This allow\n to choose whether to use locking (and other features that require\n locking) or not.\n\n16. gitweb: Show appropriate \"Generating...\" page when regenerating cache\n17. gitweb: Add startup delay to activity indicator for cache\n\n No major changes, I think (besides using new infrastructure).\n\n gitweb/Makefile | 10 +\n gitweb/README | 62 ++++\n gitweb/gitweb.perl | 333 +++++++++++++++++--\n gitweb/lib/GitwebCache/CacheOutput.pm | 93 ++++++\n gitweb/lib/GitwebCache/Capture.pm | 66 ++++\n gitweb/lib/GitwebCache/Capture/PerlIO.pm | 79 +++++\n gitweb/lib/GitwebCache/Capture/SelectFH.pm | 82 +++++\n gitweb/lib/GitwebCache/Capture/TiedCapture.pm | 149 +++++++++\n gitweb/lib/GitwebCache/FileCacheWithLocking.pm | 264 +++++++++++++++\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 417 ++++++++++++++++++++++++\n t/t9500-gitweb-standalone-no-errors.sh | 19 +\n t/t9503-gitweb-caching.sh | 38 +++\n t/t9503/benchmark_caching_interface.pl | 132 ++++++++\n t/t9503/benchmark_capture_implementations.pl | 198 +++++++++++\n t/t9503/test_cache_interface.pl | 409 +++++++++++++++++++++++\n t/t9503/test_cache_output.pl | 66 ++++\n t/t9503/test_capture_interface.pl | 76 +++++\n 17 files changed, 2468 insertions(+), 25 deletions(-)\n\n create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/PerlIO.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/TiedCapture.pm\n create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm\n create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm\n create mode 100755 t/t9503-gitweb-caching.sh\n create mode 100755 t/t9503/benchmark_caching_interface.pl\n create mode 100755 t/t9503/benchmark_capture_implementations.pl\n create mode 100755 t/t9503/test_cache_interface.pl\n create mode 100755 t/t9503/test_cache_output.pl\n create mode 100755 t/t9503/test_capture_interface.pl\n"},{"id":"143662","messageId":"1276531710-22945-2-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 01/17] gitweb: Return or exit after done serving request","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:13Z","receivedAt":"2010-06-14T16:08:13Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Check if there is a caller in top frame of gitweb, and either 'return'\nif gitweb code is wrapped in subroutine, or 'exit' if it is not.\n\nThis should avoid\n\n gitweb.cgi: Subroutine git_SOMETHING redefined at gitweb.cgi line NNN\n\nwarnings in error_log when running gitweb with mod_perl (using\nModPerl::Registry handler)\n\nSigned-off-by: Jakub Narebski \n---\n gitweb/gitweb.perl | 10 ++++++++++\n 1 files changed, 10 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 934aacb..c54a8a8 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -987,7 +987,17 @@ if ($action !~ m/^(?:opml|project_list|project_index)$/ &&\n \tdie_error(400, \"Project needed\");\n }\n $actions{$action}->();\n+\n DONE_GITWEB:\n+if (defined caller) {\n+\t# wrapped in a subroutine processing requests,\n+\t# e.g. mod_perl with ModPerl::Registry, or PSGI with Plack::App::WrapCGI\n+\treturn;\n+} else {\n+\t# pure CGI script, serving single request\n+\t# or 'exit' hijacked approprietly\n+\texit;\n+}\n 1;\n \n ## ======================================================================\n-- \n1.7.0.1\n"},{"id":"143669","messageId":"1276531710-22945-3-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 02/17] gitweb: Fix typo in hash key name in %opts in git_header_html","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:14Z","receivedAt":"2010-06-14T16:08:14Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"The name of the key has to be the same in call site handle_errors_html\nand in called subroutine that uses it, i.e. git_header_html.\n\nSigned-off-by: Jakub Narebski \n---\n gitweb/gitweb.perl | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex c54a8a8..74be92b 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -3230,7 +3230,7 @@ sub git_header_html {\n \t}\n \tprint $cgi->header(-type=>$content_type, -charset => 'utf-8',\n \t -status=> $status, -expires => $expires)\n-\t\tunless ($opts{'-no_http_headers'});\n+\t\tunless ($opts{'-no_http_header'});\n \tmy $mod_perl_version = $ENV{'MOD_PERL'} ? \" $ENV{'MOD_PERL'}\" : '';\n \tprint <\n-- \n1.7.0.1\n"},{"id":"143668","messageId":"1276531710-22945-4-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 03/17] gitweb: Prepare for splitting gitweb","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:15Z","receivedAt":"2010-06-14T16:08:15Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Prepare gitweb for having been split into modules that are to be\ninstalled alongside gitweb in 'lib/' subdirectory, by adding\n\n use lib __DIR__.'/lib';\n\nto gitweb.perl (to main gitweb script), and preparing for putting\nmodules (relative path) in $(GITWEB_MODULES) in gitweb/Makefile.\n\nSigned-off-by: Jakub Narebski \n---\n gitweb/Makefile | 3 +++\n gitweb/gitweb.perl | 9 +++++++++\n 2 files changed, 12 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex d2584fe..d2401e1 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -55,6 +55,7 @@ PERL_PATH ?= /usr/bin/perl\n bindir_SQ = $(subst ','\\'',$(bindir))#'\n gitwebdir_SQ = $(subst ','\\'',$(gitwebdir))#'\n gitwebstaticdir_SQ = $(subst ','\\'',$(gitwebdir)/static)#'\n+gitweblibdir_SQ = $(subst ','\\'',$(gitwebdir)/lib)#'\n SHELL_PATH_SQ = $(subst ','\\'',$(SHELL_PATH))#'\n PERL_PATH_SQ = $(subst ','\\'',$(PERL_PATH))#'\n DESTDIR_SQ = $(subst ','\\'',$(DESTDIR))#'\n@@ -150,6 +151,8 @@ install: all\n \t$(INSTALL) -m 755 $(GITWEB_PROGRAMS) '$(DESTDIR_SQ)$(gitwebdir_SQ)'\n \t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n \t$(INSTALL) -m 644 $(GITWEB_FILES) '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n+\t$(foreach dir,$(sort $(dir $(GITWEB_MODULES))),test -d '$(DESTDIR_SQ)$(gitwebdir_SQ)/$(dir)' || $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebdir_SQ)/$(dir)';)\n+\t$(foreach mod,$(GITWEB_MODULES),$(INSTALL) -m 644 $(mod) '$(DESTDIR_SQ)$(gitwebdir_SQ)/$(mod)';)\n \n ### Cleaning rules\n \ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 74be92b..bb81e14 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -9,6 +9,14 @@\n \n use strict;\n use warnings;\n+\n+use File::Spec;\n+# __DIR__ is taken from Dir::Self __DIR__ fragment\n+sub __DIR__ () {\n+\tFile::Spec->rel2abs(join '', (File::Spec->splitpath(__FILE__))[0, 1]);\n+}\n+use lib __DIR__ . '/lib';\n+\n use CGI qw(:standard :escapeHTML -nosticky);\n use CGI::Util qw(unescape);\n use CGI::Carp qw(fatalsToBrowser set_message);\n@@ -16,6 +24,7 @@ use Encode;\n use Fcntl ':mode';\n use File::Find qw();\n use File::Basename qw(basename);\n+\n binmode STDOUT, ':utf8';\n \n our $t0;\n-- \n1.7.0.1\n"},{"id":"143670","messageId":"1276531710-22945-5-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 04/17] gitweb/lib - Very simple file based cache","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:16Z","receivedAt":"2010-06-14T16:08:16Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This is first step towards implementing file based output (response)\ncaching layer that is used on such large sites as kernel.org.\n\nThis patch introduces GitwebCaching::SimpleFileCache package, which\nfollows Cache::Cache / CHI interface, although do not implement it\nfully. The intent of following established convention for cache\ninterface is to be able to replace our simple file based cache,\ne.g. by the one using memcached.\n\nLike in original patch by John 'Warthog9' Hawley (J.H.) (the one this\ncommit intends to be incremental step to), the data is stored in the\ncase as-is, without adding metadata (like expiration date), and\nwithout serialization (which means that one can store only scalar\ndata).\n\nTo be implemented (from original patch by J.H.):\n* cache expiration (based on file stats, current time and global\n expiration time); currently elements in cache do not expire at all\n* actually using this cache in gitweb, with exception of error pages\n* adaptive cache expiration, based on average system load\n* optional locking interface, where only one process can update cache\n (using flock)\n* server-side progress indicator when waiting for filling cache,\n which in turn requires separating situations (like snapshots and\n other non-HTML responses) where we should not show 'please wait'\n message\n\nPossible extensions (beyound what was in original patch):\n* (optionally) show information about cache utilization\n* AJAX (JavaScript-based) progress indicator\n* JavaScript code to update relative dates in cached output\n* make cache size-aware (try to not exceed specified maximum size)\n* utilize X-Sendfile header (or equivalent) to show cached data\n (optional, as it makes sense only if web server supports sendfile\n feature and have it enabled)\n* variable expiration feature from CHI, allowing items to expire a bit\n earlier than the stated expiration time to prevent cache miss\n stampedes (although locking, if available, should take care of\n this)\n\nThe code of GitwebCaching::SimpleFileCache package in gitweb/lib\nwas heavily based on file-based cache in Cache::Cache package, i.e.\non Cache::FileCache, Cache::FileBackend and Cache::BaseCache, and on\nfile-based cache in CHI, i.e. on CHI::Driver::File and CHI::Driver\n(including implementing atomic write, something that original patch\nlacks). It tries to follow more modern CHI architecture, but without\nrequiring Moose. It is much simplified compared to both interfaces\nand their file-based drivers.\n\nThis patch does not yet enable output caching in gitweb (it doesn't\nhave all required features yet); on the other hand it includes tests\nof cache Perl API in t/t9503-gitweb-caching.sh.\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 310 +++++++++++++++++++++++++++++\n t/t9503-gitweb-caching.sh | 32 +++\n t/t9503/test_cache_interface.pl | 81 ++++++++\n 3 files changed, 423 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm\n create mode 100755 t/t9503-gitweb-caching.sh\n create mode 100755 t/t9503/test_cache_interface.pl\n\ndiff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm\nnew file mode 100644\nindex 0000000..b51a124\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm\n@@ -0,0 +1,310 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2006, John 'Warthog9' Hawley \n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Gitweb caching engine, simple file-based cache\n+#\n+\n+# Minimalistic cache that stores data in the filesystem, without serialization\n+# and currently without any kind of cache expiration (all keys last forever till\n+# they got explicitely removed).\n+#\n+# It follows Cache::Cache and CHI interfaces (but does not implement it fully)\n+\n+package GitwebCache::SimpleFileCache;\n+\n+use strict;\n+use warnings;\n+\n+use File::Path qw(mkpath);\n+use File::Temp qw(tempfile);\n+use Digest::MD5 qw(md5_hex);\n+\n+# by default, the cache nests all entries on the filesystem single\n+# directory deep, i.e. '60/b725f10c9c85c70d97880dfe8191b3' for\n+# key name (key digest) 60b725f10c9c85c70d97880dfe8191b3.\n+#\n+our $DEFAULT_CACHE_DEPTH = 1;\n+\n+# by default, the root of the cache is located in 'cache'.\n+#\n+our $DEFAULT_CACHE_ROOT = \"cache\";\n+\n+# by default we don't use cache namespace (empty namespace);\n+# empty namespace does not allow for simple implementation of clear() method.\n+#\n+our $DEFAULT_NAMESPACE = '';\n+\n+# ......................................................................\n+# constructor\n+\n+# The options are set by passing in a reference to a hash containing\n+# any of the following keys:\n+# * 'namespace'\n+# The namespace associated with this cache. This allows easy separation of\n+# multiple, distinct caches without worrying about key collision. Defaults\n+# to $DEFAULT_NAMESPACE.\n+# * 'cache_root' (Cache::FileCache compatibile),\n+# 'root_dir' (CHI::Driver::File compatibile),\n+# The location in the filesystem that will hold the root of the cache.\n+# Defaults to $DEFAULT_CACHE_ROOT.\n+# * 'cache_depth' (Cache::FileCache compatibile),\n+# 'depth' (CHI::Driver::File compatibile),\n+# The number of subdirectories deep to cache object item. This should be\n+# large enough that no cache directory has more than a few hundred objects.\n+# Defaults to $DEFAULT_CACHE_DEPTH unless explicitly set.\n+sub new {\n+\tmy ($proto, $p_options_hash_ref) = @_;\n+\n+\tmy $class = ref($proto) || $proto;\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\tmy ($root, $depth, $ns);\n+\tif (defined $p_options_hash_ref) {\n+\t\t$root =\n+\t\t\t$p_options_hash_ref->{'cache_root'} ||\n+\t\t\t$p_options_hash_ref->{'root_dir'};\n+\t\t$depth =\n+\t\t\t$p_options_hash_ref->{'cache_depth'} ||\n+\t\t\t$p_options_hash_ref->{'depth'};\n+\t\t$ns = $p_options_hash_ref->{'namespace'};\n+\t}\n+\t$root = $DEFAULT_CACHE_ROOT unless defined($root);\n+\t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n+\t$ns = $DEFAULT_NAMESPACE unless defined($ns);\n+\n+\t$self->set_root($root);\n+\t$self->set_depth($depth);\n+\t$self->set_namespace($ns);\n+\n+\treturn $self;\n+}\n+\n+# ......................................................................\n+# accessors\n+\n+# http://perldesignpatterns.com/perldesignpatterns.html#AccessorPattern\n+\n+# creates get_depth() and set_depth($depth) etc. methods\n+foreach my $i (qw(depth root namespace)) {\n+\tmy $field = $i;\n+\tno strict 'refs';\n+\t*{\"get_$field\"} = sub {\n+\t\tmy $self = shift;\n+\t\treturn $self->{$field};\n+\t};\n+\t*{\"set_$field\"} = sub {\n+\t\tmy ($self, $value) = @_;\n+\t\t$self->{$field} = $value;\n+\t};\n+}\n+\n+# ----------------------------------------------------------------------\n+# utility functions and methods\n+\n+# Return root dir for namespace (lazily built, cached)\n+sub path_to_namespace {\n+\tmy ($self) = @_;\n+\n+\tif (!exists $self->{'path_to_namespace'}) {\n+\t\tif (defined $self->{'namespace'} &&\n+\t\t $self->{'namespace'} ne '') {\n+\t\t\t$self->{'path_to_namespace'} = \"$self->{'root'}/$self->{'namespace'}\";\n+\t\t} else {\n+\t\t\t$self->{'path_to_namespace'} = $self->{'root'};\n+\t\t}\n+\t}\n+\treturn $self->{'path_to_namespace'};\n+}\n+\n+# Take an human readable key, and return file path.\n+# Puts dirname of file path in second argument, if it is provided.\n+sub path_to_key {\n+\tmy ($self, $key, $dir_ref) = @_;\n+\n+\tmy @paths = ( $self->path_to_namespace() );\n+\n+\t# Create a unique (hashed) key from human readable key\n+\tmy $filename = md5_hex($key); # or $digester->add($key)->hexdigest();\n+\n+\t# Split filename so that it have DEPTH subdirectories,\n+\t# where each subdirectory has a two-letter name\n+\tpush @paths, unpack(\"(a2)\".($self->{'depth'} || 1).\" a*\", $filename);\n+\t$filename = pop @paths;\n+\n+\t# Join paths together, computing dir separately if $dir_ref was passed.\n+\tmy $filepath;\n+\tif (defined $dir_ref && ref($dir_ref)) {\n+\t\tmy $dir = join('/', @paths);\n+\t\t$filepath = \"$dir/$filename\";\n+\t\t$$dir_ref = $dir;\n+\t} else {\n+\t\t$filepath = join('/', @paths, $filename);\n+\t}\n+\n+\treturn $filepath;\n+}\n+\n+\n+# ----------------------------------------------------------------------\n+# worker methods\n+\n+sub fetch {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $file = $self->path_to_key($key);\n+\treturn undef unless (defined $file && -f $file);\n+\n+\t# Fast slurp, adapted from File::Slurp::read, with unnecessary options removed\n+\t# via CHI::Driver::File (from CHI-0.33)\n+\tmy $buf = '';\n+\topen my $read_fh, '<', $file\n+\t\tor return undef;\n+\tbinmode $read_fh, ':raw';\n+\n+\tmy $size_left = -s $read_fh;\n+\n+\twhile ($size_left > 0) {\n+\t\tmy $read_cnt = sysread($read_fh, $buf, $size_left, length($buf));\n+\t\treturn undef unless defined $read_cnt;\n+\n+\t\tlast if $read_cnt == 0;\n+\t\t$size_left -= $read_cnt;\n+\t\t#last if $size_left <= 0;\n+\t}\n+\n+\tclose $read_fh\n+\t\tor die \"Couldn't close file '$file' opened for reading: $!\";\n+\treturn $buf;\n+}\n+\n+sub store {\n+\tmy ($self, $key, $data) = @_;\n+\n+\tmy $dir;\n+\tmy $file = $self->path_to_key($key, \\$dir);\n+\treturn undef unless (defined $file && defined $dir);\n+\n+\t# ensure that directory leading to cache file exists\n+\tif (!-d $dir) {\n+\t\tmkpath($dir, 0, 0777)\n+\t\t\tor die \"Couldn't create leading directory '$dir' (mkpath): $!\";\n+\t}\n+\n+\t# generate a temporary file\n+\tmy $temp = File::Temp->new(\n+\t\t#DIR => $dir,\n+\t\tTEMPLATE => \"${file}_XXXXX\",\n+\t\tSUFFIX => \".tmp\"\n+\t) or die \"Couldn't create temporary file with '${file}_XXXXX' template: $!\";\n+\tchmod 0666, $temp\n+\t\tor warn \"Couldn't change permissions to 0666 for '$temp': $!\";\n+\n+\t# Fast spew, adapted from File::Slurp::write, with unnecessary options removed\n+\t# via CHI::Driver::File (from CHI-0.33)\n+\tmy $write_fh = $temp;\n+\tbinmode $write_fh, ':raw';\n+\n+\tmy $size_left = length($data);\n+\tmy $offset = 0;\n+\n+\twhile ($size_left > 0) {\n+\t\tmy $write_cnt = syswrite($write_fh, $data, $size_left, $offset);\n+\t\treturn undef unless defined $write_cnt;\n+\t\t\n+\t\t$size_left -= $write_cnt;\n+\t\t$offset += $write_cnt; # == length($data);\n+\t}\n+\n+\tclose $temp\n+\t\tor die \"Couldn't close temporary file '$temp' opened for writing: $!\";\n+\trename($temp, $file)\n+\t\tor die \"Couldn't rename temporary file '$temp' to '$file': $!\";\n+}\n+\n+# get size of an element associated with the $key (not the size of whole cache)\n+sub get_size {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $path = $self->path_to_key($key)\n+\t\tor return undef;\n+\tif (-f $path) {\n+\t\treturn -s $path;\n+\t}\n+\treturn 0;\n+}\n+\n+# ......................................................................\n+# interface methods\n+\n+# Removing and expiring\n+\n+# $cache->remove($key)\n+#\n+# Remove the data associated with the $key from the cache.\n+sub remove {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $file = $self->path_to_key($key)\n+\t\tor return undef;\n+\treturn undef unless -f $file;\n+\tunlink($file)\n+\t\tor die \"Couldn't remove file '$file': $!\";\n+}\n+\n+# Getting and setting\n+\n+# $cache->set($key, $data);\n+#\n+# Associates $data with $key in the cache, overwriting any existing entry.\n+# Returns $data.\n+sub set {\n+\tmy ($self, $key, $data) = @_;\n+\n+\treturn unless (defined $key && defined $data);\n+\n+\t$self->store($key, $data);\n+\n+\treturn $data;\n+}\n+\n+# $data = $cache->get($key);\n+#\n+# Returns the data associated with $key. If $key does not exist\n+# or has expired, returns undef.\n+sub get {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $data = $self->fetch($key)\n+\t\tor return undef;\n+\n+\treturn $data;\n+}\n+\n+# $data = $cache->compute($key, $code);\n+#\n+# Combines the get and set operations in a single call. Attempts to\n+# get $key; if successful, returns the value. Otherwise, calls $code\n+# and uses the return value as the new value for $key, which is then\n+# returned.\n+sub compute {\n+\tmy ($self, $key, $code) = @_;\n+\n+\tmy $data = $self->get($key);\n+\tif (!defined $data) {\n+\t\t$data = $code->($self, $key);\n+\t\t$self->set($key, $data);\n+\t}\n+\n+\treturn $data;\n+}\n+\n+# end of package GitwebCache::SimpleFileCache;\n+1;\n+\n+\ndiff --git a/t/t9503-gitweb-caching.sh b/t/t9503-gitweb-caching.sh\nnew file mode 100755\nindex 0000000..768080c\n--- /dev/null\n+++ b/t/t9503-gitweb-caching.sh\n@@ -0,0 +1,32 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Jakub Narebski\n+#\n+\n+test_description='caching interface to be used in gitweb'\n+#test_description='caching interface used in gitweb, gitweb caching\n+#\n+#This test checks cache (interface) used in gitweb caching, caching\n+#infrastructure and gitweb response (output) caching (the last by\n+#running gitweb as CGI script from commandline).'\n+\n+# for now we are running only cache interface tests\n+. ./test-lib.sh\n+\n+# this test is present in gitweb-lib.sh\n+if ! test_have_prereq PERL; then\n+\tsay 'perl not available, skipping test'\n+\ttest_done\n+fi\n+\n+\"$PERL_PATH\" -MTest::More -e 0 >/dev/null 2>&1 || {\n+\tsay 'perl module Test::More unavailable, skipping test'\n+\ttest_done\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+test_external 'GitwebCache::* Perl API (in gitweb/cache.pm)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_cache_interface.pl\n+\n+test_done\ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nnew file mode 100755\nindex 0000000..9242129\n--- /dev/null\n+++ b/t/t9503/test_cache_interface.pl\n@@ -0,0 +1,81 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use Test::More;\n+\n+# test source version\n+use lib \"$ENV{TEST_DIRECTORY}/../gitweb/lib\";\n+\n+# Test creating a cache\n+#\n+BEGIN { use_ok('GitwebCache::SimpleFileCache'); }\n+my $cache = new_ok('GitwebCache::SimpleFileCache');\n+\n+# Test that default values are defined\n+#\n+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,\n+\t'$DEFAULT_CACHE_ROOT defined');\n+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,\n+\t'$DEFAULT_CACHE_DEPTH defined');\n+\n+# Test accessors and default values for cache\n+#\n+SKIP: {\n+\tskip 'default values not defined', 3\n+\t\tunless ($GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT &&\n+\t\t $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH);\n+\n+\tis($cache->get_namespace(), '', \"default namespace is ''\");\n+\tcmp_ok($cache->get_root(), 'eq', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,\n+\t\t\"default cache root is '$GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT'\");\n+\tcmp_ok($cache->get_depth(), '==', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,\n+\t\t\"default cache depth is $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH\");\n+}\n+\n+# Test the getting, setting, and removal of a cached value\n+# (Cache::Cache interface)\n+#\n+my $key = 'Test Key';\n+my $value = 'Test Value';\n+\n+subtest 'Cache::Cache interface' => sub {\n+\tforeach my $method (qw(get set remove)) {\n+\t\tcan_ok($cache, $method);\n+\t}\n+\n+\t$cache->set($key, $value);\n+\tcmp_ok($cache->get_size($key), '>', 0, 'get_size after set, is greater than 0');\n+\tis($cache->get($key), $value, 'get after set, returns cached value');\n+\t$cache->remove($key);\n+\tok(!defined($cache->get($key)), 'get after remove, is undefined');\n+\n+\teval { $cache->remove('Not-Existent Key'); };\n+\tok(!$@, 'remove on non-existent key doesn\\'t die');\n+\tdiag($@) if $@;\n+\n+\tdone_testing();\n+};\n+\n+# Test the getting and setting of a cached value\n+# (CHI interface)\n+#\n+my $call_count = 0;\n+sub get_value {\n+\t$call_count++;\n+\treturn $value;\n+}\n+subtest 'CHI interface' => sub {\n+\tcan_ok($cache, qw(compute));\n+\n+\tis($cache->compute($key, \\&get_value), $value, \"compute 1st time (set) returns '$value'\");\n+\tis($cache->compute($key, \\&get_value), $value, \"compute 2nd time (get) returns '$value'\");\n+\tis($cache->compute($key, \\&get_value), $value, \"compute 3rd time (get) returns '$value'\");\n+\tcmp_ok($call_count, '==', 1, 'get_value() is called once from compute');\n+\n+\tdone_testing();\n+};\n+\n+done_testing();\n-- \n1.7.0.1\n"},{"id":"143664","messageId":"1276531710-22945-6-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 05/17] gitweb/lib - Stat-based cache expiration","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:17Z","receivedAt":"2010-06-14T16:08:17Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add stat-based cache expiration to file-based GitwebCache::SimpleFileCache.\nContrary to the way other caching interfaces such as Cache::Cache and CHI\ndo it, the time cache element expires in is _global_ value associated with\ncache instance, and is not local property of cache entry. (Currently cache\nentry does not store any metadata associated with entry... which means that\nthere is no need for serialization / marshalling / freezing and thawing.)\nDefault expire time is -1, which means never expire.\n\nTo check if cache entry is expired, GitwebCache::SimpleFileCache compares\ndifference between mtime (last modify time) of a cache file and current time\nwith (global) time to expire. It is done using CHI-compatibile is_valid()\nmethod.\n\nAdd some tests checking that expiring works correctly (on the level of API).\n\nTo be implemented (from original patch by J.H.):\n* actually using this cache in gitweb, except error pages\n* adaptive cache expiration, based on average system load\n* optional locking interface, where only one process can update cache\n (using flock)\n* server-side progress indicator when waiting for filling cache,\n which in turn requires separating situations (like snapshots and\n other non-HTML responses) where we should not show 'please wait'\n message\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 42 ++++++++++++++++++++++++++--\n t/t9503/test_cache_interface.pl | 19 +++++++++++++\n 2 files changed, 58 insertions(+), 3 deletions(-)\n\ndiff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm\nindex b51a124..91b3373 100644\n--- a/gitweb/lib/GitwebCache/SimpleFileCache.pm\n+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm\n@@ -57,6 +57,10 @@ our $DEFAULT_NAMESPACE = '';\n # The number of subdirectories deep to cache object item. This should be\n # large enough that no cache directory has more than a few hundred objects.\n # Defaults to $DEFAULT_CACHE_DEPTH unless explicitly set.\n+# * 'default_expires_in' (Cache::Cache compatibile),\n+# 'expires_in' (CHI compatibile) [seconds]\n+# The expiration time for objects place in the cache.\n+# Defaults to -1 (never expire) if not explicitly set.\n sub new {\n \tmy ($proto, $p_options_hash_ref) = @_;\n \n@@ -64,7 +68,7 @@ sub new {\n \tmy $self = {};\n \t$self = bless($self, $class);\n \n-\tmy ($root, $depth, $ns);\n+\tmy ($root, $depth, $ns, $expires_in);\n \tif (defined $p_options_hash_ref) {\n \t\t$root =\n \t\t\t$p_options_hash_ref->{'cache_root'} ||\n@@ -73,14 +77,19 @@ sub new {\n \t\t\t$p_options_hash_ref->{'cache_depth'} ||\n \t\t\t$p_options_hash_ref->{'depth'};\n \t\t$ns = $p_options_hash_ref->{'namespace'};\n+\t\t$expires_in =\n+\t\t\t$p_options_hash_ref->{'default_expires_in'} ||\n+\t\t\t$p_options_hash_ref->{'expires_in'};\n \t}\n \t$root = $DEFAULT_CACHE_ROOT unless defined($root);\n \t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n \t$ns = $DEFAULT_NAMESPACE unless defined($ns);\n+\t$expires_in = -1 unless defined($expires_in); # <0 means never\n \n \t$self->set_root($root);\n \t$self->set_depth($depth);\n \t$self->set_namespace($ns);\n+\t$self->set_expires_in($expires_in);\n \n \treturn $self;\n }\n@@ -91,7 +100,7 @@ sub new {\n # http://perldesignpatterns.com/perldesignpatterns.html#AccessorPattern\n \n # creates get_depth() and set_depth($depth) etc. methods\n-foreach my $i (qw(depth root namespace)) {\n+foreach my $i (qw(depth root namespace expires_in)) {\n \tmy $field = $i;\n \tno strict 'refs';\n \t*{\"get_$field\"} = sub {\n@@ -257,6 +266,31 @@ sub remove {\n \t\tor die \"Couldn't remove file '$file': $!\";\n }\n \n+# $cache->is_valid($key)\n+#\n+# Returns a boolean indicating whether $key exists in the cache\n+# and has not expired (global per-cache 'expires_in').\n+sub is_valid {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $path = $self->path_to_key($key);\n+\n+\t# does file exists in cache?\n+\treturn 0 unless -f $path;\n+\t# get its modification time\n+\tmy $mtime = (stat(_))[9] # _ to reuse stat structure used in -f test\n+\t\tor die \"Couldn't stat file '$path': $!\";\n+\n+\t# expire time can be set to never\n+\tmy $expires_in = $self->get_expires_in();\n+\treturn 1 unless (defined $expires_in && $expires_in >= 0);\n+\n+\t# is file expired?\n+\tmy $now = time();\n+\n+\treturn (($now - $mtime) < $expires_in);\n+}\n+\n # Getting and setting\n \n # $cache->set($key, $data);\n@@ -280,6 +314,7 @@ sub set {\n sub get {\n \tmy ($self, $key) = @_;\n \n+\treturn undef unless $self->is_valid($key);\n \tmy $data = $self->fetch($key)\n \t\tor return undef;\n \n@@ -304,7 +339,8 @@ sub compute {\n \treturn $data;\n }\n \n-# end of package GitwebCache::SimpleFileCache;\n 1;\n+__END__\n+# end of package GitwebCache::SimpleFileCache;\n \n \ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nindex 9242129..b1e9036 100755\n--- a/t/t9503/test_cache_interface.pl\n+++ b/t/t9503/test_cache_interface.pl\n@@ -78,4 +78,23 @@ subtest 'CHI interface' => sub {\n \tdone_testing();\n };\n \n+# Test cache expiration\n+#\n+subtest 'cache expiration' => sub {\n+\t$cache->set_expires_in(60*60*24); # set expire time to 1 day\n+\tcmp_ok($cache->get_expires_in(), '>', 0, '\"expires in\" is greater than 0');\n+\tis($cache->get($key), $value, 'get returns cached value (not expired in 1d)');\n+\n+\t$cache->set_expires_in(-1); # set expire time to never expire\n+\tis($cache->get_expires_in(), -1, '\"expires in\" is set to never (-1)');\n+\tis($cache->get($key), $value, 'get returns cached value (not expired)');\n+\n+\t$cache->set_expires_in(0);\n+\tis($cache->get_expires_in(), 0, '\"expires in\" is set to now (0)');\n+\t$cache->set($key, $value);\n+\tok(!defined($cache->get($key)), 'cache is expired');\n+\n+\tdone_testing();\n+};\n+\n done_testing();\n-- \n1.7.0.1\n"},{"id":"143663","messageId":"1276531710-22945-7-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 06/17] gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:18Z","receivedAt":"2010-06-14T16:08:18Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add t/t9503/benchmark_caching_interface.pl, which benchmarks 'set' and\n'get' methods from GitwebCache::SimpleFileCache, and compares them\nagainst Cache::FileCache, Cache::MemoryCache, CHI with 'File' driver,\nCache::FastMmap if they are available.\n\nIncludes example benchmark results.\n\nSigned-off-by: Jakub Narebski \n---\n t/t9503/benchmark_caching_interface.pl | 132 ++++++++++++++++++++++++++++++++\n 1 files changed, 132 insertions(+), 0 deletions(-)\n create mode 100755 t/t9503/benchmark_caching_interface.pl\n\ndiff --git a/t/t9503/benchmark_caching_interface.pl b/t/t9503/benchmark_caching_interface.pl\nnew file mode 100755\nindex 0000000..5a46077\n--- /dev/null\n+++ b/t/t9503/benchmark_caching_interface.pl\n@@ -0,0 +1,132 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use File::Spec;\n+use File::Path;\n+use Benchmark qw(:all);\n+\n+# benchmark source version\n+sub __DIR__ () {\n+\tFile::Spec->rel2abs(join '', (File::Spec->splitpath(__FILE__))[0, 1]);\n+}\n+use lib __DIR__.\"/../../gitweb/lib\";\n+use GitwebCache::SimpleFileCache;\n+\n+my $key = 'http://localhost/cgi-bin/gitweb.cgi?p=git/git.git;a=blob_plain;f=lorem.txt;hb=HEAD';\n+my $value = <<'EOF';\n+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do\n+eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad\n+minim veniam, quis nostrud exercitation ullamco laboris nisi ut\n+aliquip ex ea commodo consequat. Duis aute irure dolor in\n+reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla\n+pariatur. Excepteur sint occaecat cupidatat non proident, sunt in\n+culpa qui officia deserunt mollit anim id est laborum.\n+EOF\n+\n+my $cache_root = __DIR__.'/cache';\n+my %cache_options = (\n+\t'cache_root' => $cache_root, # Cache::FileCache compatibile\n+\t'root_dir' => $cache_root, # CHI::Driver::File compatibile\n+\t'cache_depth' => 1, # Cache::FileCache compatibile\n+\t'depth' => 1, # CHI::Driver::File compatibile\n+\t'default_expires_in' => 24*60*60, # Cache::Cache compatibile\n+\t'expires_in' => 24*60*60, # CHI compatibile\n+\t'expire_time' => 24*60*60, # Cache::FastMmap compatibile\n+);\n+\n+mkdir($cache_root);\n+\n+my @caches;\n+push @caches, GitwebCache::SimpleFileCache->new({\n+\t'namespace' => 'GitwebCache-SimpleFileCache',\n+\t%cache_options,\n+});\n+\n+if (eval { require Cache::FileCache; 1; }) {\n+\tpush @caches, Cache::FileCache->new({\n+\t\t'namespace' => 'Cache-FileCache',\n+\t\t%cache_options,\n+\t});\n+}\n+\n+if (eval { require Cache::MemoryCache; 1; }) {\n+\tpush @caches, Cache::MemoryCache->new({\n+\t\t'namespace' => 'Cache-MemoryCache',\n+\t\t%cache_options,\n+\t});\n+}\n+if (eval { require CHI; 1; }) {\n+\tpush @caches, CHI->new(\n+\t\t'driver' => 'File',\n+\t\t'namespace' => 'CHI-Driver-File',\n+\t\t%cache_options,\n+\t);\n+}\n+if (eval { require Cache::FastMmap; 1 }) {\n+\tpush @caches, Cache::FastMmap->new(\n+\t\t'share_file' => $cache_root.'/Cache-FastMmap',\n+\t\t'init_file' => 1,\n+\t\t'raw_values' => 1,\n+\t\t%cache_options,\n+\t);\n+}\n+\n+my %caches;\n+my $count = -10;\n+\n+print '$cache->set($key, $value)'.\"\\n\";\n+for my $cache (@caches) {\n+\tmy $name = ref($cache);\n+\t$name = 'GitwebCache' if ($name =~ /^GitwebCache/);\n+\t$caches{$name} = sub { $cache->set($key, $value); };\n+}\n+\n+my $result_set = timethese($count, { %caches });\n+cmpthese($result_set);\n+\n+print \"\\n\";\n+\n+print '$cache->get($key)'.\"\\n\";\n+for my $cache (@caches) {\n+\tmy $name = ref($cache);\n+\t$name = 'GitwebCache' if ($name =~ /^GitwebCache/);\n+\t$caches{$name} = sub { $cache->get($key); };\n+}\n+\n+my $result_get = timethese($count, { %caches });\n+cmpthese($result_get);\n+\n+rmtree($cache_root);\n+\n+1;\n+__END__\n+## EXAMPLE OUTPUT ##\n+#\n+# $cache->set($key, $value)\n+# Benchmark: running Cache::FastMmap, Cache::FileCache, Cache::MemoryCache, GitwebCache\n+# for at least 10 CPU seconds...\n+# Cache::FastMmap: 11 wallclock secs ( 8.46 usr + 2.17 sys = 10.63 CPU) @ 15710.25/s (n=167000)\n+# Cache::FileCache: 15 wallclock secs ( 8.43 usr + 2.25 sys = 10.68 CPU) @ 356.27/s (n=3805)\n+# Cache::MemoryCache: 13 wallclock secs ( 9.91 usr + 0.09 sys = 10.00 CPU) @ 3306.20/s (n=33062)\n+# GitwebCache: 29 wallclock secs ( 7.02 usr + 3.47 sys = 10.49 CPU) @ 605.91/s (n=6356)\n+# Rate Cache::FileCache GitwebCache Cache::MemoryCache Cache::FastMmap\n+# Cache::FileCache 356/s -- -41% -89% -98%\n+# GitwebCache 606/s 70% -- -82% -96%\n+# Cache::MemoryCache 3306/s 828% 446% -- -79%\n+# Cache::FastMmap 15710/s 4310% 2493% 375% --\n+#\n+# $cache->get($key)\n+# Benchmark: running Cache::FastMmap, Cache::FileCache, Cache::MemoryCache, GitwebCache\n+# for at least 10 CPU seconds...\n+# Cache::FastMmap: 13 wallclock secs ( 7.32 usr + 2.83 sys = 10.15 CPU) @ 24260.59/s (n=246245)\n+# Cache::FileCache: 12 wallclock secs ( 9.22 usr + 1.30 sys = 10.52 CPU) @ 972.62/s (n=10232)\n+# Cache::MemoryCache: 14 wallclock secs ( 9.89 usr + 0.12 sys = 10.01 CPU) @ 3679.52/s (n=36832)\n+# GitwebCache: 20 wallclock secs ( 8.16 usr + 2.36 sys = 10.52 CPU) @ 4401.05/s (n=46299)\n+# Rate Cache::FileCache Cache::MemoryCache GitwebCache Cache::FastMmap\n+# Cache::FileCache 973/s -- -74% -78% -96%\n+# Cache::MemoryCache 3680/s 278% -- -16% -85%\n+# GitwebCache 4401/s 352% 20% -- -82%\n+# Cache::FastMmap 24261/s 2394% 559% 451% --\n-- \n1.7.0.1\n"},{"id":"143665","messageId":"1276531710-22945-8-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 07/17] gitweb/lib - Simple select(FH) based output capture","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:19Z","receivedAt":"2010-06-14T16:08:19Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add two packages: GitwebCache::Capture, which defines interface, and\nGitwebCache::Capture::SelectFH, which is actually implements simple\ncapturing. GitwebCache::Capture::SelectFH captures output by using\nselect(FILEHANDLE) to change default filehandle for output. This\nmeans that output of a \"print\" or a \"printf\" (or a \"write\") without\na filehandle would be captured.\n\nTo change mode of filehandle used for capturing correctly,\n binmode select(), ;\nneeds to be used in place of\n binmode STDOUT, ;\n\nCapturing is done using in-memory file held in Perl scalar.\n\nUsing select(FILEHANDLE) is a bit fragile as a method of capturing\noutput, as it assumes that we always use \"print\" or \"printf\" without\nfilehandle, and use select() which returns default filehandle for\noutput in place of explicit STDOUT. On the other hand it has the\nadvantage of being simple. Alternate solutions include using tie\n(like in CGI::Cache), or using PerlIO layers - but the last requires\nnon-standard PerlIO::Util module.\n\n\nIncludes separate tests for capturing output, in t9503/test_capture.pl\nwhich is run as external test from t9503-gitweb-caching.sh\n\nSigned-off-by: Jakub Narebski \n---\n gitweb/lib/GitwebCache/Capture.pm | 66 ++++++++++++++++++++++\n gitweb/lib/GitwebCache/Capture/SelectFH.pm | 82 ++++++++++++++++++++++++++++\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 7 +--\n t/t9503-gitweb-caching.sh | 5 ++-\n t/t9503/test_capture_interface.pl | 76 ++++++++++++++++++++++++++\n 5 files changed, 231 insertions(+), 5 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/Capture.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm\n create mode 100755 t/t9503/test_capture_interface.pl\n\ndiff --git a/gitweb/lib/GitwebCache/Capture.pm b/gitweb/lib/GitwebCache/Capture.pm\nnew file mode 100644\nindex 0000000..3e9fe81\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/Capture.pm\n@@ -0,0 +1,66 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Output capturing for gitweb caching engine\n+#\n+\n+# It is base abstract class (a role) for capturing output of gitweb\n+# actions for gitweb caching engine.\n+# \n+# Child (derived) concrete classes, which actually implement some method\n+# of capturing STDOUT output, must implement the following methods:\n+# * ->new(), to create new object of a capturing class\n+# * ->start(), to start capturing output\n+# * ->stop(), to stop capturing output and return it\n+#\n+# Before starting capture by using capture_block etc. subroutines,\n+# one has to run ->setup().\n+\n+package GitwebCache::Capture;\n+\n+use strict;\n+use warnings;\n+\n+use Exporter qw(import);\n+our @EXPORT = qw(capture_start capture_stop capture_block);\n+our @EXPORT_OK = qw(setup_capture);\n+our %EXPORT_TAGS = (all => [ @EXPORT, @EXPORT_OK ]);\n+\n+# Holds object used for capture (of child class)\n+my $capture;\n+\n+sub setup_capture {\n+\tmy $self = shift || __PACKAGE__;\n+\n+\t$capture = $self->new(@_);\n+}\n+\n+sub capture {\n+\tmy ($self, $code) = @_;\n+\n+\t$self->start();\n+\t$code->();\n+\treturn $self->stop();\n+}\n+\n+# Wrap caching data; capture only STDOUT\n+sub capture_block (&) {\n+\tmy $code = shift;\n+\treturn $capture->capture($code);\n+}\n+\n+sub capture_start {\n+\t$capture->start(@_);\n+}\n+\n+sub capture_stop {\n+\treturn $capture->stop(@_);\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::Capture;\ndiff --git a/gitweb/lib/GitwebCache/Capture/SelectFH.pm b/gitweb/lib/GitwebCache/Capture/SelectFH.pm\nnew file mode 100644\nindex 0000000..2b904d3\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/Capture/SelectFH.pm\n@@ -0,0 +1,82 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Simple output capturing using select(FH);\n+#\n+\n+# This module (class) captures output of 'print ', 'printf '\n+# and 'write ' (without a filehandle) by using select(FILEHANDLE)\n+# to change default filehandle for output, changing it to in-memory\n+# file (saving output to scalar).\n+#\n+# Note that when using this simplest way of capturing, to change mode of\n+# filehandle using for capturing correctly, \"binmode STDOUT, ;\"\n+# has to be changed to \"binmode select(), ;\". This has no change\n+# if we are not capturing output using GitwebCache::Capture::SelectFH.\n+\n+package GitwebCache::Capture::SelectFH;\n+\n+use PerlIO;\n+\n+use strict;\n+use warnings;\n+\n+use base qw(GitwebCache::Capture);\n+use GitwebCache::Capture qw(:all);\n+\n+use Exporter qw(import);\n+our @EXPORT = @GitwebCache::Capture::EXPORT;\n+our @EXPORT_OK = @GitwebCache::Capture::EXPORT_OK;\n+our %EXPORT_TAGS = %GitwebCache::Capture::EXPORT_TAGS;\n+\n+# Constructor\n+sub new {\n+\tmy $proto = shift;\n+\n+\tmy $class = ref($proto) || $proto;\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\t$self->{'oldfh'} = select();\n+\t$self->{'data'} = '';\n+\n+\treturn $self;\n+}\n+\n+# Start capturing data (STDOUT)\n+# (printed using 'print ' or 'printf ')\n+sub start {\n+\tmy $self = shift;\n+\t\n+\t$self->{'data'} = '';\n+\t$self->{'data_fh'} = undef;\n+\t\n+\topen $self->{'data_fh'}, '>', \\$self->{'data'}\n+\t\tor die \"Couldn't open in-memory file for capture: $!\";\n+\t$self->{'oldfh'} = select($self->{'data_fh'});\n+\n+\t# note: this does not cover all cases\n+\tbinmode select(), ':utf8'\n+\t\tif ((PerlIO::get_layers($self->{'oldfh'}))[-1] eq 'utf8');\n+}\n+\n+# Stop capturing data (required for die_error)\n+sub stop {\n+\tmy $self = shift;\n+\n+\t# return if we didn't start capturing\n+\treturn unless defined $self->{'data_fh'};\n+\t\n+\tselect($self->{'oldfh'});\n+\tclose $self->{'data_fh'}\n+\t\tor die \"Couldn't close in-memory file for capture: $!\";\n+\treturn $self->{'data'};\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::Capture::SelectFH;\ndiff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm\nindex 91b3373..7c90350 100644\n--- a/gitweb/lib/GitwebCache/SimpleFileCache.pm\n+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm\n@@ -201,16 +201,17 @@ sub store {\n \n \t# ensure that directory leading to cache file exists\n \tif (!-d $dir) {\n-\t\tmkpath($dir, 0, 0777)\n+\t\teval { mkpath($dir, 0, 0777); 1 }\n \t\t\tor die \"Couldn't create leading directory '$dir' (mkpath): $!\";\n \t}\n \n \t# generate a temporary file\n+\t# (Temp::File dies itself if it cannot create file)\n \tmy $temp = File::Temp->new(\n \t\t#DIR => $dir,\n \t\tTEMPLATE => \"${file}_XXXXX\",\n \t\tSUFFIX => \".tmp\"\n-\t) or die \"Couldn't create temporary file with '${file}_XXXXX' template: $!\";\n+\t);# or die \"Couldn't create temporary file with '${file}_XXXXX' template: $!\";\n \tchmod 0666, $temp\n \t\tor warn \"Couldn't change permissions to 0666 for '$temp': $!\";\n \n@@ -342,5 +343,3 @@ sub compute {\n 1;\n __END__\n # end of package GitwebCache::SimpleFileCache;\n-\n-\ndiff --git a/t/t9503-gitweb-caching.sh b/t/t9503-gitweb-caching.sh\nindex 768080c..73b3f5a 100755\n--- a/t/t9503-gitweb-caching.sh\n+++ b/t/t9503-gitweb-caching.sh\n@@ -26,7 +26,10 @@ fi\n \n # ----------------------------------------------------------------------\n \n-test_external 'GitwebCache::* Perl API (in gitweb/cache.pm)' \\\n+test_external 'GitwebCache::SimpleFileCache Perl API (in gitweb/cache.pm)' \\\n \t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_cache_interface.pl\n \n+test_external 'GitwebCache::Capture Perl API (in gitweb/cache.pm)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_capture_interface.pl\n+\n test_done\ndiff --git a/t/t9503/test_capture_interface.pl b/t/t9503/test_capture_interface.pl\nnew file mode 100755\nindex 0000000..d7aa480\n--- /dev/null\n+++ b/t/t9503/test_capture_interface.pl\n@@ -0,0 +1,76 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+use utf8;\n+\n+use Test::More;\n+\n+# test source version\n+use lib \"$ENV{TEST_DIRECTORY}/../gitweb/lib\";\n+\n+# ....................................................................\n+\n+# prototypes must be known at compile time, otherwise they do not work\n+BEGIN { use_ok('GitwebCache::Capture::SelectFH', qw(:all)); }\n+\n+# Test setting up capture\n+#\n+my $capture = new_ok('GitwebCache::Capture::SelectFH' => [], 'The $capture');\n+isa_ok($capture, 'GitwebCache::Capture', 'The $capture');\n+ok(setup_capture('GitwebCache::Capture::SelectFH'),\n+ 'setup_capture with package name: GitwebCache::Capture::SelectFH');\n+ok(setup_capture($capture),\n+ 'setup_capture with subclass object: $capture');\n+\n+# Test properties of capture_block\n+#\n+is(prototype('capture_block'), '&', 'capture_block has (&) prototype');\n+\n+# Test capturing\n+#\n+diag('Should not print anything except test results and diagnostic');\n+my $test_data = 'Capture this';\n+my $captured = capture_block {\n+\tprint $test_data;\n+};\n+is($captured, $test_data, 'capture_block captures simple data');\n+\n+binmode STDOUT, ':utf8';\n+$test_data = <<'EOF';\n+Áéí óú\n+ÄËÑÏÖ\n+Ábçdèfg\n+Zażółć gęsią jaźń\n+山田 太郎\n+ブレームのテストです。\n+\n+はれひほふ\n+\n+しているのが、いるので。\n+濱浜ほれぷりぽれまびぐりろへ。\n+EOF\n+utf8::decode($test_data);\n+$captured = capture_block {\n+\tbinmode select(), ':utf8';\n+\n+\tprint $test_data;\n+};\n+utf8::decode($captured);\n+is($captured, $test_data, 'capture_block captures utf8 data');\n+\n+$test_data = '|\\x{fe}\\x{ff}|\\x{9F}|\\000|'; # invalid utf-8\n+$captured = capture_block {\n+\tbinmode select(), ':raw';\n+\n+\tprint $test_data;\n+};\n+is($captured, $test_data, 'capture_block captures raw data');\n+\n+\n+done_testing();\n+\n+# Local Variables:\n+# encoding: utf-8\n+# End:\n-- \n1.7.0.1\n"},{"id":"143667","messageId":"1276531710-22945-9-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 08/17] gitweb/lib - Alternate ways of capturing output","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:20Z","receivedAt":"2010-06-14T16:08:20Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Besides GitwebCache::Capture::SelectFH, which uses select(FH) to\nredirect 'print LIST' and 'printf FORMAT, LIST' to in-memory file to\ncapture output, add GitwebCache::Capture::TiedCapture which uses\ntie-ing filehandle to capture output, and GitwebCache::Capture::PerlIO\nwhich uses push_layer method from non-core PerlIO::Util module to\ncapture output.\n\nAdd test (which canbe run standalone) for all those implementations,\nchecking ':utf8' and ':raw' output, and benchmark comparing them\n(includes example benchmark tests). Please note that the test for\nalternate implementations is not run from t/t9503-gitweb-caching.sh\ntest.\n\nSigned-off-by: Jakub Narebski \n---\n gitweb/lib/GitwebCache/Capture/PerlIO.pm | 79 ++++++++++\n gitweb/lib/GitwebCache/Capture/TiedCapture.pm | 149 +++++++++++++++++++\n t/t9503/benchmark_capture_implementations.pl | 198 +++++++++++++++++++++++++\n 3 files changed, 426 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/Capture/PerlIO.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/TiedCapture.pm\n create mode 100755 t/t9503/benchmark_capture_implementations.pl\n\ndiff --git a/gitweb/lib/GitwebCache/Capture/PerlIO.pm b/gitweb/lib/GitwebCache/Capture/PerlIO.pm\nnew file mode 100644\nindex 0000000..199aeed\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/Capture/PerlIO.pm\n@@ -0,0 +1,79 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Output capturing using PerlIO layers\n+#\n+\n+# This module requires PaerlIO::Util installed.\n+\n+package GitwebCache::Capture::PerlIO;\n+\n+use PerlIO::Util;\n+\n+use strict;\n+use warnings;\n+\n+use base qw(GitwebCache::Capture);\n+use GitwebCache::Capture qw(:all);\n+\n+use Exporter qw(import);\n+our @EXPORT = @GitwebCache::Capture::EXPORT;\n+our @EXPORT_OK = @GitwebCache::Capture::EXPORT_OK;\n+our %EXPORT_TAGS = %GitwebCache::Capture::EXPORT_TAGS;\n+\n+# Constructor\n+sub new {\n+\tmy $proto = shift;\n+\n+\tmy $class = ref($proto) || $proto;\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\t$self->{'data'} = '';\n+\n+\treturn $self;\n+}\n+\n+# Start capturing data (STDOUT)\n+# (printed using 'print ' or 'printf ')\n+sub start {\n+\tmy $self = shift;\n+\n+\t$self->{'data'} = '';\n+\t*STDOUT->push_layer('scalar' => \\$self->{'data'});\n+\n+\t# push ':utf8' on top, if it was on top\n+\t*STDOUT->push_layer(':utf8')\n+\t\tif ((*STDOUT->get_layers())[-2] eq 'utf8');\n+}\n+\n+# Stop capturing data (required for die_error)\n+sub stop {\n+\tmy $self = shift;\n+\n+\t# return if we didn't start capturing\n+\tmy @layers = *STDOUT->get_layers();\n+\treturn unless grep { $_ eq 'scalar' } @layers;\n+\n+\tmy $was_utf8 = $layers[-1] eq 'utf8';\n+\t# stop saving to scalar, i.e. remove topmost 'scalar' layer,\n+\t# but remember that 'utf8' layer might be on top of it\n+\twhile ((my $layer = *STDOUT->pop_layer())) {\n+\t\tpop @layers;\n+\t\tlast if $layer eq 'scalar';\n+\t}\n+\t# restore ':utf8' mode, if needed\n+\tif ($was_utf8 && $layers[-1] ne 'utf8') {\n+\t\t*STDOUT->push_layer('utf8');\n+\t}\n+\n+\treturn $self->{'data'};\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::Capture::PerlIO;\ndiff --git a/gitweb/lib/GitwebCache/Capture/TiedCapture.pm b/gitweb/lib/GitwebCache/Capture/TiedCapture.pm\nnew file mode 100644\nindex 0000000..e7cc2e3\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/Capture/TiedCapture.pm\n@@ -0,0 +1,149 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Simple output capturing by tie-ing filehandle\n+#\n+\n+package GitwebCache::Capture::TiedCapture;\n+\n+use PerlIO;\n+\n+use strict;\n+use warnings;\n+\n+use base qw(GitwebCache::Capture);\n+use GitwebCache::Capture qw(:all);\n+\n+use Exporter qw(import);\n+our @EXPORT = @GitwebCache::Capture::EXPORT;\n+our @EXPORT_OK = @GitwebCache::Capture::EXPORT_OK;\n+our %EXPORT_TAGS = %GitwebCache::Capture::EXPORT_TAGS;\n+\n+# Constructor\n+sub new {\n+\tmy $proto = shift;\n+\n+\tmy $class = ref($proto) || $proto;\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\t$self->{'data'} = '';\n+\t$self->{'tied'} = undef;\n+\n+\treturn $self;\n+}\n+\n+# Start capturing data (STDOUT)\n+# (printed using 'print ' or 'printf ')\n+sub start {\n+\tmy $self = shift;\n+\n+\t# savie tie\n+\t$self->{'tied'} = tied *STDOUT;\n+\n+\t$self->{'data'} = '';\n+\ttie *STDOUT, 'GitwebCache::TiedCapture', \\$self->{'data'};\n+\n+\t# re-binmode, so that tied class would pick it up\n+\tbinmode STDOUT,\n+\t\t(PerlIO::get_layers(*STDOUT))[-1] eq 'utf8' ? ':utf8' : ':raw';\n+}\n+\n+# Stop capturing data (required for die_error)\n+sub stop {\n+\tmy $self = shift;\n+\n+\t# return if we didn't start capturing\n+\treturn unless tied(*STDOUT)->isa('GitwebCache::TiedCapture');\n+\n+\t# restore ties, if there were any\n+\tuntie *STDOUT;\n+\tif ($self->{'tied'}) {\n+\t\ttie *STDOUT, 'Tie::Restore', $self->{'tied'};\n+\t}\n+\n+\treturn $self->{'data'};\n+}\n+\n+1;\n+\n+########################################################################\n+\n+package GitwebCache::TiedCapture;\n+\n+our $VERSION = '0.001';\n+$VERSION = eval $VERSION;\n+\n+use utf8; # not strictly necessary: utf8::* is in core\n+use strict;\n+use warnings;\n+\n+sub TIEHANDLE {\n+\tmy ($proto, $dataref) = @_;\n+\tmy $class = ref($proto) || $proto;\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\t$self->{'scalar'} = $dataref;\n+\t$self->{'binmode'} = ':utf8';\n+\treturn $self;\n+}\n+\n+sub append_str {\n+\tmy ($self, $str) = @_;\n+\tutf8::encode($str) if ($self->{'binmode'} eq ':utf8');\n+\t${$self->{'scalar'}} .= $str;\n+}\n+\n+sub WRITE {\n+\tmy ($self, $buffer, $length, $offset) = @_;\n+\t$self->append_str(substr($buffer, $offset, $length));\n+}\n+\n+sub PRINT {\n+\tmy $self = shift;\n+\t$self->append_str(join('',@_));\n+}\n+\n+sub PRINTF {\n+\tmy $self = shift;\n+\t$self->append_str(sprintf(@_));\n+}\n+\n+sub BINMODE {\n+\tmy $self = shift;\n+\t$self->{'mode'} = shift || ':raw';\n+}\n+\n+#sub UNTIE {\n+#\tlocal $^W = 0;\n+#}\n+\n+1;\n+\n+########################################################################\n+\n+package Tie::Restore;\n+# Written by Robby Walker ( webmaster@pointwriter.com )\n+# for Point Writer ( http://www.pointwriter.com/ ).\n+\n+our $VERSION = '0.11';\n+$VERSION = eval $VERSION;\n+\n+# $object = tied %hash;\n+# tie %hash, 'Tie::Restore', $object;\n+\n+sub TIESCALAR { $_[1] }\n+sub TIEARRAY { $_[1] }\n+sub TIEHASH { $_[1] }\n+sub TIEHANDLE { $_[1] }\n+\n+1;\n+\n+\n+1;\n+__END__\n+# end of package GitwebCache::Capture::TiedCapture;\ndiff --git a/t/t9503/benchmark_capture_implementations.pl b/t/t9503/benchmark_capture_implementations.pl\nnew file mode 100755\nindex 0000000..73806a3\n--- /dev/null\n+++ b/t/t9503/benchmark_capture_implementations.pl\n@@ -0,0 +1,198 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use File::Spec;\n+use File::Path;\n+use Benchmark qw(:all);\n+\n+use PerlIO;\n+\n+# benchmark source version\n+sub __DIR__ () {\n+\tFile::Spec->rel2abs(join '', (File::Spec->splitpath(__FILE__))[0, 1]);\n+}\n+use lib __DIR__.\"/../../gitweb/lib\";\n+\n+# ....................................................................\n+\n+# Load modules (without importing)\n+#\n+my @modules =\n+\tmap { \"GitwebCache::Capture::$_\" }\n+\tqw(SelectFH TiedCapture);\n+foreach my $mod (@modules) {\n+\teval \"require $mod\";\n+}\n+if (eval { require PerlIO::Util; 1 }) {\n+\trequire GitwebCache::Capture::PerlIO;\n+\tpush @modules, 'GitwebCache::Capture::PerlIO';\n+}\n+\n+# Set up capturing, for each module\n+#\n+my @captures = map { $_->new() } @modules;\n+\n+my $test_data = <<'EOF';\n+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do\n+eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad\n+minim veniam, quis nostrud exercitation ullamco laboris nisi ut\n+aliquip ex ea commodo consequat. Duis aute irure dolor in\n+reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla\n+pariatur. Excepteur sint occaecat cupidatat non proident, sunt in\n+culpa qui officia deserunt mollit anim id est laborum.\n+\n+Sed ut perspiciatis unde omnis iste natus error sit voluptatem\n+accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae\n+ab illo inventore veritatis et quasi architecto beatae vitae dicta\n+sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit\n+aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos\n+qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui\n+dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed\n+quia non numquam eius modi tempora incidunt ut labore et dolore magnam\n+aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum\n+exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex\n+ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in\n+ea voluptate velit esse quam nihil molestiae consequatur, vel illum\n+qui dolorem eum fugiat quo voluptas nulla pariatur?\n+EOF\n+\n+my @captured_output;\n+my $repeat = 100;\n+sub capture_output {\n+\tmy ($class, $mode) = @_;\n+\n+\t$class->start();\n+\tbinmode select(), $mode if defined($mode);\n+\tprint $test_data for (1..$repeat);\n+\n+\treturn $class->stop();\n+}\n+\n+my %codehash;\n+for (my $i = 0; $i < @captures; $i++) {\n+\tmy $capture = $captures[$i];\n+\tmy $name = ref($capture);\n+\t$name =~ s/^.*:://;\n+\n+\t$codehash{$name} = sub { $captured_output[$i] = capture_output($capture) };\n+}\n+\n+# ....................................................................\n+\n+my $test_other_modules = 0;\n+\n+if ($test_other_modules) {\n+\n+\tif (eval { require Capture::Tiny; 1; }) {\n+\t\t$codehash{'Capture::Tiny'} = sub {\n+\t\t\tmy ($stdout, $stderr) = Capture::Tiny::capture(sub {\n+\t\t\t\tprint $test_data for (1..$repeat);\n+\t\t\t});\n+\t\t\tprint STDERR $stderr if defined($stderr);\n+\t\t};\n+\t}\n+\n+\tif (eval { require IO::CaptureOutput; 1; }) {\n+\t\t$codehash{'IO::CaptureOutput'} = sub {\n+\t\t\tmy ($stdout, $stderr);\n+\t\t\tIO::CaptureOutput::capture(sub {\n+\t\t\t\tprint $test_data for (1..$repeat);\n+\t\t\t}, \\$stdout, \\$stderr);\n+\t\t\tprint STDERR $stderr if defined($stderr);\n+\t\t};\n+\t\t# somehow it interferes with capturing in GitwebCache::Capture::PerlIO\n+\t\tdelete $codehash{'PerlIO'};\n+\t}\n+\n+\tif (eval { require IO::Capture::Stdout; 1; }) {\n+\t\t$codehash{'IO::Capture'} = sub {\n+\t\t\tmy $capture = IO::Capture::Stdout->new();\n+\n+\t\t\t$capture->start();\n+\t\t\tprint $test_data for (1..$repeat);\n+\t\t\t$capture->stop();\n+\n+\t\t\tmy $captured_output = join('', $capture->read());\n+\t\t};\n+\t}\n+} # end if ($test_other_modules)\n+\n+# ....................................................................\n+\n+print \"Capturing $repeat x \".length($test_data).\n+ \" = \".($repeat * length($test_data)).\" characters\\n\";\n+my $count = -10; # CPU seconds\n+my $result = timethese($count, \\%codehash);\n+cmpthese($result);\n+\n+#if (exists $codehash{PerlIO}) {\n+#\tcmpthese(-10, {\n+#\t\t'PerlIO::get_layers' => sub { PerlIO::get_layers(*STDOUT); },\n+#\t\t'PerlIO::Util method' => sub { *STDOUT->get_layers(); },\n+#\t});\n+#}\n+\n+1;\n+__END__\n+## EXAMPLE OUTPUT ##\n+#\n+## 1 x $test_data, PerlIO using *STDOUT->get_layers();\n+# Benchmark: running PerlIO, SelectFH, TiedCapture for at least 10 CPU seconds...\n+# PerlIO: 9 wallclock secs (10.38 usr + 0.13 sys = 10.51 CPU) @ 9676.31/s (n=101698)\n+# SelectFH: 12 wallclock secs (10.51 usr + 0.02 sys = 10.53 CPU) @ 12294.21/s (n=129458)\n+# TiedCapture: 10 wallclock secs (10.24 usr + 0.06 sys = 10.30 CPU) @ 9489.22/s (n=97739)\n+# Rate TiedCapture PerlIO SelectFH\n+# TiedCapture 9489/s -- -2% -23%\n+# PerlIO 9676/s 2% -- -21%\n+# SelectFH 12294/s 30% 27% --\n+#\n+## 10 x $test_data, PerlIO using *STDOUT->get_layers();\n+# Benchmark: running PerlIO, SelectFH, TiedCapture for at least 10 CPU seconds...\n+# PerlIO: 9 wallclock secs (10.47 usr + 0.07 sys = 10.54 CPU) @ 7558.35/s (n=79665)\n+# SelectFH: 11 wallclock secs (10.36 usr + 0.04 sys = 10.40 CPU) @ 8970.87/s (n=93297)\n+# TiedCapture: 11 wallclock secs (10.45 usr + 0.02 sys = 10.47 CPU) @ 2602.77/s (n=27251)\n+# Rate TiedCapture PerlIO SelectFH\n+# TiedCapture 2603/s -- -66% -71%\n+# PerlIO 7558/s 190% -- -16%\n+# SelectFH 8971/s 245% 19% --\n+#\n+## 100 x $test_data, PerlIO using *STDOUT->get_layers();\n+# Benchmark: running PerlIO, SelectFH, TiedCapture for at least 50 CPU seconds...\n+# PerlIO: 67 wallclock secs (35.28 usr + 17.82 sys = 53.10 CPU) @ 832.41/s (n=44201)\n+# SelectFH: 73 wallclock secs (33.83 usr + 18.63 sys = 52.46 CPU) @ 830.06/s (n=43545)\n+# TiedCapture: 71 wallclock secs (50.93 usr + 0.41 sys = 51.34 CPU) @ 95.31/s (n=4893)\n+# Rate TiedCapture SelectFH PerlIO\n+# TiedCapture 95.3/s -- -89% -89%\n+# SelectFH 830/s 771% -- -0%\n+# PerlIO 832/s 773% 0% --\n+#\n+## 100 x $test_data, PerlIO using mix of *STDOUT->get_layers() and PerlIO::get_layers(*STDOUT);\n+# Capturing 100 x 1314 = 131400 characters\n+# Benchmark: timing 25000 iterations of PerlIO, SelectFH, TiedCapture...\n+# PerlIO: 30 wallclock secs (19.05 usr + 10.29 sys = 29.34 CPU) @ 852.08/s (n=25000)\n+# SelectFH: 30 wallclock secs (18.95 usr + 10.26 sys = 29.21 CPU) @ 855.87/s (n=25000)\n+# TiedCapture: 307 wallclock secs (267.37 usr + 2.95 sys = 270.32 CPU) @ 92.48/s (n=25000)\n+# Rate TiedCapture PerlIO SelectFH\n+# TiedCapture 92.5/s -- -89% -89%\n+# PerlIO 852/s 821% -- -0%\n+# SelectFH 856/s 825% 0% --\n+#\n+## 100 x $test_data (IO::CaptureOutput interferes with GitwebCache::Capture::PerlIO)\n+# Capturing 100 x 1314 = 131400 characters\n+# Benchmark: running IO::CaptureOutput, SelectFH, TiedCapture for at least 10 CPU seconds...\n+# IO::CaptureOutput: 12 wallclock secs ( 5.12 usr + 5.63 sys = 10.75 CPU) @ 126.60/s (n=1361)\n+# SelectFH: 12 wallclock secs ( 6.93 usr + 3.45 sys = 10.38 CPU) @ 808.29/s (n=8390)\n+# TiedCapture: 11 wallclock secs (10.11 usr + 0.01 sys = 10.12 CPU) @ 103.26/s (n=1045)\n+# Rate TiedCapture IO::CaptureOutput SelectFH\n+# TiedCapture 103/s -- -18% -87%\n+# IO::CaptureOutput 127/s 23% -- -84%\n+# SelectFH 808/s 683% 538% --\n+#\n+## PerlIO::get_layers == PerlIO::get_layers(*STDOUT)\n+## PerlIU::Util method == *STDOUT->get_layers()\n+# Rate PerlIO::Util method PerlIO::get_layers\n+# PerlIO::Util method 54405/s -- -38%\n+# PerlIO::get_layers 87672/s 61% --\n-- \n1.7.0.1\n"},{"id":"143672","messageId":"1276531710-22945-10-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 09/17] gitweb/lib - Cache captured output (using get/set)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:21Z","receivedAt":"2010-06-14T16:08:21Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add GitwebCache::CacheOutput package, which contains cache_output\nsubroutine, and (re)exports capture_stop from GitwebCache::Capture.\nThe cache_output gets data from cache and prints it, or captures\noutput of provided subroutine (code reference), saves it to cache and\nprints it. It currently uses Cache::Cache compatibile (get, set)\ninterface to cache. Capturing is currently (not configurable) done\nusing GitwebCache::Capture::SelectFH introduced in previous commit,\nbut any class derived from GitwebCache::Capture (like provided example\nGitwebCache::Capture::TiedCapture and GitwebCache::Capture::PerlIO)\nwould work.\n\nGitweb would use cache_output to get page from cache, or to generate\npage and save it to cache. The capture_stop would be used in\ndie_error subroutine, because error pages would not be cached.\n\nIt is assumed that data is saved to cache _converted_, and should\ntherefore be read from cache and printed to STDOUT in ':raw' (binary)\nmode.\n\n\nAdd t9503/test_cache_output.pl test, run as external test in\nt9503-gitweb-caching. It checks that cache_output behaves correctly,\nnamely that it saves and restores action output in cache, and that it\nprints generated output or cached output.\n\nSigned-off-by: Jakub Narebski \n---\n gitweb/lib/GitwebCache/CacheOutput.pm | 62 +++++++++++++++++++++++++++++++\n t/t9503-gitweb-caching.sh | 3 +\n t/t9503/test_cache_output.pl | 66 +++++++++++++++++++++++++++++++++\n 3 files changed, 131 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm\n create mode 100755 t/t9503/test_cache_output.pl\n\ndiff --git a/gitweb/lib/GitwebCache/CacheOutput.pm b/gitweb/lib/GitwebCache/CacheOutput.pm\nnew file mode 100644\nindex 0000000..8195f0b\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/CacheOutput.pm\n@@ -0,0 +1,62 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+# (C) 2006, John 'Warthog9' Hawley \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Capturing and caching (gitweb) output\n+#\n+\n+# Capture output, save it in cache and print it, or retrieve it from\n+# cache and print it.\n+\n+package GitwebCache::CacheOutput;\n+\n+use strict;\n+use warnings;\n+\n+use GitwebCache::SimpleFileCache;\n+use GitwebCache::Capture::SelectFH qw(:all);\n+\n+use Exporter qw(import);\n+our @EXPORT = qw(cache_output capture_stop);\n+our %EXPORT_TAGS = (all => [ @EXPORT ]);\n+\n+# cache_output($cache, $key, $action_code);\n+#\n+# Attempts to get $key from $cache; if successful, prints the value.\n+# Otherwise, calls $action_code, capture its output and use\n+# the captured output as the new value for $key in $cache,\n+# then print captured output.\n+our $CAPTURE_CLASS = 'GitwebCache::Capture::SelectFH';\n+\n+sub cache_output {\n+\tmy ($cache, $key, $code) = @_;\n+\n+\tmy $data = $cache->get($key);\n+\n+\t# capture and cache output, if there was nothing in the cache\n+\tif (!defined $data) {\n+\t\tmy $capture = $CAPTURE_CLASS;\n+\t\tsetup_capture($capture);\n+\n+\t\t# do not use 'capture_block' prototype\n+\t\t$data = &capture_block($code);\n+\t\t$cache->set($key, $data) if defined $data;\n+\t}\n+\n+\t# print cached data\n+\tif (defined $data) {\n+\t\t# select() instead of STDOUT is here for t9503 test:\n+\t\tbinmode select(), ':raw';\n+\t\tprint $data;\n+\t}\n+\n+\treturn $data;\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::CacheOutput;\ndiff --git a/t/t9503-gitweb-caching.sh b/t/t9503-gitweb-caching.sh\nindex 73b3f5a..0afcc0c 100755\n--- a/t/t9503-gitweb-caching.sh\n+++ b/t/t9503-gitweb-caching.sh\n@@ -32,4 +32,7 @@ test_external 'GitwebCache::SimpleFileCache Perl API (in gitweb/cache.pm)' \\\n test_external 'GitwebCache::Capture Perl API (in gitweb/cache.pm)' \\\n \t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_capture_interface.pl\n \n+test_external 'GitwebCache::CacheOutput Perl API (in gitweb/cache.pm)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_cache_output.pl\n+\n test_done\ndiff --git a/t/t9503/test_cache_output.pl b/t/t9503/test_cache_output.pl\nnew file mode 100755\nindex 0000000..96eedb5\n--- /dev/null\n+++ b/t/t9503/test_cache_output.pl\n@@ -0,0 +1,66 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use Test::More;\n+\n+# test source version\n+use lib \"$ENV{TEST_DIRECTORY}/../gitweb/lib\";\n+\n+# ....................................................................\n+\n+# prototypes must be known at compile time, otherwise they do not work\n+BEGIN { use_ok('GitwebCache::CacheOutput'); }\n+\n+# Test setting up $cache and $capture\n+my $cache = new_ok('GitwebCache::SimpleFileCache' => [], 'The $cache ');\n+my $capture = new_ok('GitwebCache::Capture::SelectFH' => [], 'The $capture');\n+\n+# ......................................................................\n+\n+# Prepare for testing cache_output\n+my $key = 'Key';\n+my $action_output = <<'EOF';\n+# This is data to be cached and shown\n+EOF\n+my $cached_output = <<\"EOF\";\n+$action_output# (version recovered from cache)\n+EOF\n+sub action {\n+\tprint $action_output;\n+}\n+\n+# Catch output printed by cache_fetch\n+# (only for 'print ' and 'printf ')\n+sub capture_output_of_cache_output {\n+\tmy $test_data = '';\n+\n+\topen my $test_data_fh, '>', \\$test_data;\n+\tmy $oldfh = select($test_data_fh);\n+\n+\tcache_output($cache, $key, \\&action);\n+\n+\tselect($oldfh);\n+\tclose $test_data_fh;\n+\n+\treturn $test_data;\n+}\n+\n+# clean state\n+$cache->set_expires_in(-1);\n+$cache->remove($key);\n+my $test_data;\n+\n+# first time (if there is no cache) generates cache entry\n+$test_data = capture_output_of_cache_output();\n+is($test_data, $action_output, 'action output is printed (generated)');\n+is($cache->get($key), $action_output, 'action output is saved in cache (generated)');\n+\n+# second time (if cache is set/valid) reads from cache\n+$cache->set($key, $cached_output);\n+$test_data = capture_output_of_cache_output();\n+is($test_data, $cached_output, 'action output is printed (from cache)');\n+\n+done_testing();\n-- \n1.7.0.1\n"},{"id":"143674","messageId":"1276531710-22945-11-git-send-email-jnareb@gmail.com","threadId":"24107","inReplyTo":"1276531710-22945-1-git-send-email-jnareb@gmail.com","subject":"[RFC PATCHv4 10/17] gitweb: Add optional output caching (from gitweb/cache.pm)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-06-14T16:08:22Z","receivedAt":"2010-06-14T16:08:22Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This commit actually adds output caching to gitweb, as we have now\nminimal features required for it in GitwebCache::SimpleFileCache\n(a 'dumb' but fast file-based cache engine). To enable cache you need\n(at least) set $caching_enabled to true in gitweb config, and copy\ngitweb/cache.pm alongside generated gitweb.cgi - this is described\nin more detail in the new \"Gitweb caching\" section in gitweb/README.\n\nCaching in theory can be done using any Perl module that implements\nCache::Cache compatibile get/set (method) interface. The default is\nto use GitwebCache::SimpleFileCache from gitweb/cache.pm. Capturing\nand caching output is done via cache_output subroutine from\nGitwebCache::CacheOutput, also in gitweb/cache.pm.\n\nThe cache_output subroutine in GitwebCache::CacheOutput currently\ncaptures output using GitwebCache::Capture::SelectFH package, which in\nturn uses select(FILEHANDLE) to change default filehandle for output.\nThis means that a \"print\" or a \"printf\" (or a \"write\") in gitweb\nsource without a filehandle would be captured. To change mode of\nfilehandle used for capturing correctly, \"binmode STDOUT, \" had\nto be changed to \"binmode select(), \".\n\nCapturing and caching is designed in such way that there is no\nbehaviour change if $caching_enabled is false. If caching is not\nenabled, then capturing is also turned off.\n\nEnabling caching causes the following additional changes to gitweb\noutput:\n* Disables content-type negotiation (choosing between 'text/html'\n mimetype and 'application/xhtml+xml') when caching, as there is no\n content-type negotiation done when retrieving page from cache.\n Use lowest common denominator of 'text/html' mimetype which can\n be used by all browsers.\n* Disable optional timing info (how much time it took to generate the\n original page, and how many git commands it took), and in its place show\n unconditionally when page was originally generated (in GMT / UTC\n timezone).\n* Disable 'blame_incremental' view, as it doesn't make sense without\n printing data as soon as it is generated (which would require tee-ing\n when capturing output for caching)... and it doesn't work currently\n anyway.\n\nAdd basic tests of caching support to t9500-gitweb-standalone-no-errors\ntest: set $caching_enabled to true and check for errors for first time\nrun (generating cache) and second time run (retrieving from cache) for a\nsingle view - summary view for a project.\n\n\nTo be implemented (from original patch by J.H.):\n* adaptive cache expiration, based on average system load\n* optional locking interface, where only one process can update cache\n (using flock)\n* server-side progress indicator when waiting for filling cache,\n which in turn requires separating situations (like snapshots and\n other non-HTML responses) where we should not show 'please wait'\n message\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\n gitweb/Makefile | 6 ++\n gitweb/README | 58 +++++++++++++++\n gitweb/gitweb.perl | 127 ++++++++++++++++++++++++++------\n t/t9500-gitweb-standalone-no-errors.sh | 19 +++++\n 4 files changed, 187 insertions(+), 23 deletions(-)\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex d2401e1..025060b 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -111,6 +111,12 @@ endif\n \n GITWEB_FILES += static/git-logo.png static/git-favicon.png\n \n+# gitweb output caching\n+GITWEB_MODULES += lib/GitwebCache/CacheOutput.pm\n+GITWEB_MODULES += lib/GitwebCache/SimpleFileCache.pm\n+GITWEB_MODULES += lib/GitwebCache/Capture.pm\n+GITWEB_MODULES += lib/GitwebCache/Capture/SelectFH.pm\n+\n GITWEB_REPLACE = \\\n \t-e 's|++GIT_VERSION++|$(GIT_VERSION)|g' \\\n \t-e 's|++GIT_BINDIR++|$(bindir)|g' \\\ndiff --git a/gitweb/README b/gitweb/README\nindex 0e19be8..9b3e5d7 100644\n--- a/gitweb/README\n+++ b/gitweb/README\n@@ -236,6 +236,11 @@ not include variables usually directly set during build):\n If server load exceed this value then return \"503 Service Unavaliable\" error.\n Server load is taken to be 0 if gitweb cannot determine its value. Set it to\n undefined value to turn it off. The default is 300.\n+ * $caching_enabled\n+ If true, gitweb would use caching to speed up generating response.\n+ Currently supported is only output (response) caching. See \"Gitweb caching\"\n+ section below for details on how to configure and customize caching.\n+ The default is false (caching is disabled).\n \n \n Projects list file format\n@@ -308,6 +313,59 @@ You can use the following files in repository:\n descriptions.\n \n \n+Gitweb caching\n+~~~~~~~~~~~~~~\n+\n+Currently gitweb supports only output (HTTP response) caching, similar\n+to the one used on http://git.kernel.org. To turn it on, set \n+$caching_enabled variable to true value in gitweb config file, i.e.:\n+\n+ our $caching_enabled = 1;\n+\n+You can choose which caching engine should gitweb use by setting $cache\n+variable to _initialized_ instance of cache interface, e.g.:\n+\n+ use CHI;\n+ our $cache = CHI->new( driver => 'Memcached',\n+ \tservers => [ \"10.0.0.15:11211\", \"10.0.0.15:11212\" ],\n+ \tl1_cache => { driver => 'FastMmap', root_dir => '/var/cache/gitweb' }\n+ );\n+\n+Alternatively you can set $cache variable to the name of cache class,\n+e.g.:\n+\n+ use Cache::FileCache;\n+ our $cache = 'Cache::FileCache';\n+\n+In this case caching engine should support Cache::Cache or CHI names for\n+cache config (see below), and ignore unrecognized options. Such caching\n+engine should also implement (at least) ->get($key) and ->set($key, $data)\n+methods (Cache::Cache and CHI compatible interface).\n+\n+If $cache is left unset (if it is left undefined), then gitweb would use\n+GitwebCache::SimpleFileCache as caching engine. This engine is 'dumb' (but\n+fast) file based caching layer, currently without any support for cache size\n+limiting, or even removing expired / grossly expired entries. It has\n+therefore the downside of requiring a huge amount of disk space if there are\n+a number of repositories involved. It is not uncommon for git.kernel.org to\n+have on the order of 80G - 120G accumulate over the course of a few months.\n+It is therefore recommended that the cache directory be periodically\n+completely deleted; this operation is safe to perform. Suggested mechanism\n+(substitute $cachedir for actual path to gitweb cache):\n+\n+ # mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush\n+\n+Site-wide cache options are defined in %cache_options hash. Those options\n+apply only when $cache is unset (GitwebCache::SimpleFileCache is used), or\n+if $cache is name of cache class (e.g. $cache = 'Cache::FileCache'). You\n+can override cache options in gitweb config, e.g.:\n+\n+ $cache_options{'expires_in'} = 60; # 60 seconds = 1 minute\n+\n+Please read comments for %cache_options entries in gitweb/gitweb.perl for\n+description of available cache options.\n+\n+\n Webserver configuration\n -----------------------\n \ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex bb81e14..6772f6e 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -236,6 +236,41 @@ our %avatar_size = (\n # Leave it undefined (or set to 'undef') to turn off load checking.\n our $maxload = 300;\n \n+# This enables/disables the caching layer in gitweb. Currently supported\n+# is only output (response) caching, similar to the one used on git.kernel.org.\n+our $caching_enabled = 0;\n+# Set to _initialized_ instance of cache interface implementing (at least)\n+# get($key) and set($key, $data) methods (Cache::Cache and CHI interfaces).\n+# If unset, GitwebCache::SimpleFileCache would be used, which is 'dumb'\n+# (but fast) file based caching layer, currently without any support for\n+# cache size limiting. It is therefore recommended that the cache directory\n+# be periodically completely deleted; this operation is safe to perform.\n+# Suggested mechanism:\n+# mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush\n+our $cache;\n+# You define site-wide cache options defaults here; override them with\n+# $GITWEB_CONFIG as necessary.\n+our %cache_options = (\n+\t# The location in the filesystem that will hold the root of the cache.\n+\t# This directory will be created as needed (if possible) on the first\n+\t# cache set. Note that either this directory must exists and web server\n+\t# has to have write permissions to it, or web server must be able to\n+\t# create this directory.\n+\t# Possible values:\n+\t# * 'cache' (relative to gitweb),\n+\t# * File::Spec->catdir(File::Spec->tmpdir(), 'gitweb-cache'),\n+\t# * '/var/cache/gitweb' (FHS compliant, requires being set up),\n+\t'cache_root' => 'cache',\n+\t# The number of subdirectories deep to cache object item. This should be\n+\t# large enough that no cache directory has more than a few hundred\n+\t# objects. Each non-leaf directory contains up to 256 subdirectories\n+\t# (00-ff). Must be larger than 0.\n+\t'cache_depth' => 1,\n+\t# The (global) expiration time for objects placed in the cache, in seconds.\n+\t'expires_in' => 20,\n+);\n+\n+\n # You define site-wide feature defaults here; override them with\n # $GITWEB_CONFIG as necessary.\n our %feature = (\n@@ -995,7 +1030,37 @@ if ($action !~ m/^(?:opml|project_list|project_index)$/ &&\n !$project) {\n \tdie_error(400, \"Project needed\");\n }\n-$actions{$action}->();\n+\n+if ($caching_enabled) {\n+\tif (!eval { require GitwebCache::CacheOutput; 1; }) {\n+\t\tdie_error(500, \"Caching enabled and GitwebCache::CacheOutput not found\");\n+\t}\n+\tGitwebCache::CacheOutput->import();\n+\n+\t# $cache might be initialized (instantiated) cache, i.e. cache object,\n+\t# or it might be name of class, or it might be undefined\n+\tunless (defined $cache && ref($cache)) {\n+\t\t$cache ||= 'GitwebCache::SimpleFileCache';\n+\t\t$cache = $cache->new({\n+\t\t\t%cache_options,\n+\t\t\t#'cache_root' => '/tmp/cache',\n+\t\t\t#'cache_depth' => 2,\n+\t\t\t#'expires_in' => 20, # in seconds (CHI compatibile)\n+\t\t\t# (Cache::Cache compatibile initialization)\n+\t\t\t'default_expires_in' => $cache_options{'expires_in'},\n+\t\t\t# (CHI compatibile initialization)\n+\t\t\t'root_dir' => $cache_options{'cache_root'},\n+\t\t\t'depth' => $cache_options{'cache_depth'},\n+\t\t});\n+\t}\n+\n+\t# human readable key identifying gitweb output\n+\tmy $output_key = href(-replay => 1, -full => 1, -path_info => 0);\n+\n+\tcache_output($cache, $output_key, $actions{$action});\n+} else {\n+\t$actions{$action}->();\n+}\n \n DONE_GITWEB:\n if (defined caller) {\n@@ -3230,7 +3295,9 @@ sub git_header_html {\n \t# 'application/xhtml+xml', otherwise send it as plain old 'text/html'.\n \t# we have to do this because MSIE sometimes globs '*/*', pretending to\n \t# support xhtml+xml but choking when it gets what it asked for.\n-\tif (defined $cgi->http('HTTP_ACCEPT') &&\n+\t# Disable content-type negotiation when caching (use mimetype good for all).\n+\tif (!$caching_enabled &&\n+\t defined $cgi->http('HTTP_ACCEPT') &&\n \t $cgi->http('HTTP_ACCEPT') =~ m/(,|;|\\s|^)application\\/xhtml\\+xml(,|;|\\s|$)/ &&\n \t $cgi->Accept('application/xhtml+xml') != 0) {\n \t\t$content_type = 'application/xhtml+xml';\n@@ -3404,17 +3471,25 @@ sub git_footer_html {\n \t}\n \tprint \"\\n\"; # class=\"page_footer\"\n \n-\tif (defined $t0 && gitweb_check_feature('timed')) {\n+\t# timing info doesn't make much sense with output (response) caching,\n+\t# so when caching is enabled gitweb prints the time of page generation\n+\tif ((defined $t0 || $caching_enabled) &&\n+\t gitweb_check_feature('timed')) {\n \t\tprint \"
\\n\";\n-\t\tprint 'This page took '.\n-\t\t ''.\n-\t\t Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).\n-\t\t ' seconds '.\n-\t\t ' and '.\n-\t\t ''.\n-\t\t $number_of_git_cmds.\n-\t\t ' git commands '.\n-\t\t \" to generate.\\n\";\n+\t\tif ($caching_enabled) {\n+\t\t\tprint 'This page was generated at '.\n+\t\t\t gmtime( time() ).\" GMT\\n\";\n+\t\t} else {\n+\t\t\tprint 'This page took '.\n+\t\t\t ''.\n+\t\t\t Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).\n+\t\t\t ' seconds '.\n+\t\t\t ' and '.\n+\t\t\t ''.\n+\t\t\t $number_of_git_cmds.\n+\t\t\t ' git commands '.\n+\t\t\t \" to generate.\\n\";\n+\t\t}\n \t\tprint \"
\\n\"; # class=\"page_footer\"\n \t}\n \n@@ -3423,8 +3498,8 @@ sub git_footer_html {\n \t}\n \n \tprint qq!\\n!;\n-\tif (defined $action &&\n-\t $action eq 'blame_incremental') {\n+\tif (!$caching_enabled &&\n+\t defined $action && $action eq 'blame_incremental') {\n \t\tprint qq!