{"thread":{"id":"25371","subject":"[PATCHv5 01/17] t/test-lib.sh: Export also GIT_BUILD_DIR in test_external","startedAt":"2010-10-06T22:01:45Z","lastAt":"2010-10-24T21:34:37Z","messageCount":41,"participants":["Jakub Narebski","Thomas Adam","Ævar Arnfjörð Bjarmason","J.H."],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"152787","messageId":"1286402526-13143-1-git-send-email-jnareb@gmail.com","threadId":"25371","inReplyTo":null,"subject":"[PATCHv5 00/17] gitweb: Simple file based output caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-06T22:01:45Z","receivedAt":"2010-10-06T22:01:45Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This 17+ patches long patch series is intended as a replacement for\nlarge 'gitweb: File based caching layer (from git.kernel.org)'\nmega-patch by John 'Warthog9' Hawley aka J.H., by starting small and\nadding features piece by piece.\n\nThis is fourthfifth version (5th release) of this series, and is\navailable on http://repo.or.cz/w/git/jnareb-git.git as\n'gitweb/cache-kernel-v5' branch. Earlier versions are available there\nas branches 'gitweb/cache-kernel', 'gitweb/cache-kernel-v2',\n'gitweb/cache-kernel-v3' and 'gitweb/cache-kernel-v4'. Previous version\nof this series was sent to git mailing list as:\n\n* [RFC PATCHv4 00/17] gitweb: Simple file based output caching\n Message-Id: <1276531710-22945-1-git-send-email-jnareb@gmail.com>\n http://thread.gmane.org/gmane.comp.version-control.git/149101\n\nThis version is based on top of aad24ad (gitweb: Move call to\nevaluate_git_version after evaluate_gitweb_config, 2010-10-05) in\n'gitweb/web' branch of http://repo.or.cz/w/git/jnareb-git.git\nrepository.\n\nThe 'gitweb/web' branch consist of the following commits on top of\n1e63341 (Merge branch 'maint', 2010-09-30) in 'master' branch of main\ngit repository:\n gitweb/Makefile: Include gitweb/config.mak\n t/gitweb-lib.sh: Add support for GITWEB_TEST_INSTALLED\n gitweb/Makefile: Add 'test' and 'test-installed' targets\n gitweb: Move call to evaluate_git_version after evaluate_gitweb_config\n\nThe first three patches were send as 3-part series:\n* [PATCHv2 0/3] Testing installed gitweb\n Message-Id: <1285506146-8009-1-git-send-email-jnareb@gmail.com>\n http://thread.gmane.org/gmane.comp.version-control.git/157222\n\nThose are required to be able to test installed version of gitweb and\nmodules, to check if there are any problems with process of installing\ngitweb.\n\nThe last patch in 'gitweb/web' branch was send as:\n* [PATCH] gitweb: Move call to evaluate_git_version after evaluate_gitweb_config \n Message-ID: <20100926113431.28918.53550.stgit@localhost.localdomain>\n http://article.gmane.org/gmane.comp.version-control.git/157220\n\nThis was needed during development of this series, to avoid spurious\noutput of gitweb tests when using `--debug' option.\n\n\nThe MAIN DIFFERENCE from previous version is that it is based on never\nversion of gitweb; in particular putting all code in subroutines: gitweb\ncalls run() subroutine on startup, and run_request() subroutine on each\nrequest (per connection) from the 'jn/gitweb-fastcgi' branch.\n\nThe patches\n gitweb: Return or exit after done serving request\n gitweb: Fix typo in hash key name in %opts in git_header_html\n\npresent at beginning of previous (fourth) version of this series are\nalready merged in.\n\nThe patches\n gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)\n gitweb/lib - Alternate ways of capturing output\n\nwhich were interleaved with other patches in previous version of this\nseries got moved to the end of series, and marked proof of concept\n(\"PoC\").\n\n\nThe following changes are available in the git repository at:\n\n git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v5\n\n\nAll comments welcome!\n\nJ.H., could you comment on how this series relates to the gitweb code\n*currently* running on git.kernel.org wrt. code? If possible, could you\ntry to compare performance of those two implementations, the one\npresented here in this series, and the one used by git.kernel.org.\n\nPasky or other repo.or.cz admins, could you tell us if and what kind of\ncaching gitweb used by repo.or.cz uses, and how it compares (wrt code\nand if possible also performance) to solution implemented in this\nseries?\n\n\nTODO list and areas of possible improvements would be send in separate\nemail.\n\n\nJakub Narebski (21):\n t/test-lib.sh: Export also GIT_BUILD_DIR in test_external\n gitweb: Prepare for splitting gitweb\n gitweb/lib - Very simple file based cache\n gitweb/lib - Stat-based cache expiration\n gitweb/lib - Regenerate entry if the cache file has size of 0\n gitweb/lib - Simple select(FH) based output capture\n gitweb/lib - Cache captured output (using get/set)\n gitweb: Add optional output caching (from gitweb/cache.pm)\n gitweb/lib - Adaptive cache expiration time\n gitweb/lib - Use CHI compatibile (compute method) caching interface\n gitweb/lib - Use locking to avoid 'cache miss stampede' problem\n gitweb/lib - No need for File::Temp when locking\n gitweb/lib - Serve stale data when waiting for filling cache\n gitweb/lib - Regenerate (refresh) cache in background\n gitweb: Introduce %actions_info, gathering information about actions\n gitweb: Show appropriate \"Generating...\" page when regenerating cache\n gitweb: Add startup delay to activity indicator for cache\n gitweb/lib - Add clear() and size() methods to caching interface\n gitweb: Add beginnings of cache administration page\n gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)\n gitweb/lib - Alternate ways of capturing output\n\n gitweb/Makefile | 24 ++-\n gitweb/README | 61 +++\n gitweb/gitweb.perl | 459 ++++++++++++++++++++++--\n gitweb/lib/GitwebCache/CacheOutput.pm | 95 +++++\n gitweb/lib/GitwebCache/Capture.pm | 66 ++++\n gitweb/lib/GitwebCache/Capture/PerlIO.pm | 79 ++++\n gitweb/lib/GitwebCache/Capture/SelectFH.pm | 82 ++++\n gitweb/lib/GitwebCache/Capture/TiedCapture.pm | 76 ++++\n gitweb/lib/GitwebCache/FileCacheWithLocking.pm | 280 ++++++++++++++\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 472 ++++++++++++++++++++++++\n gitweb/lib/Tie/Restore.pm | 24 ++\n gitweb/lib/TiedCapture/PerlIO.pm | 56 +++\n gitweb/lib/TiedCapture/String.pm | 53 +++\n t/t9500-gitweb-standalone-no-errors.sh | 20 +\n t/t9503-gitweb-caching-interface.sh | 34 ++\n t/t9503/benchmark_caching_interface.pl | 209 +++++++++++\n t/t9503/test_cache_interface.pl | 407 ++++++++++++++++++++\n t/t9504-gitweb-capture-interface.sh | 34 ++\n t/t9504/benchmark_capture_implementations.pl | 226 +++++++++++\n t/t9504/test_capture_implementations.pl | 85 +++++\n t/t9504/test_capture_interface.pl | 76 ++++\n t/t9505-gitweb-cache.sh | 34 ++\n t/t9505/test_cache_output.pl | 70 ++++\n t/test-lib.sh | 4 +-\n 24 files changed, 2997 insertions(+), 29 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/PerlIO.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/TiedCapture.pm\n create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm\n create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm\n create mode 100644 gitweb/lib/Tie/Restore.pm\n create mode 100644 gitweb/lib/TiedCapture/PerlIO.pm\n create mode 100644 gitweb/lib/TiedCapture/String.pm\n create mode 100755 t/t9503-gitweb-caching-interface.sh\n create mode 100755 t/t9503/benchmark_caching_interface.pl\n create mode 100755 t/t9503/test_cache_interface.pl\n create mode 100755 t/t9504-gitweb-capture-interface.sh\n create mode 100755 t/t9504/benchmark_capture_implementations.pl\n create mode 100755 t/t9504/test_capture_implementations.pl\n create mode 100755 t/t9504/test_capture_interface.pl\n create mode 100755 t/t9505-gitweb-cache.sh\n create mode 100755 t/t9505/test_cache_output.pl\n\n-- \n1.7.3\n"},{"id":"152784","messageId":"1286402526-13143-2-git-send-email-jnareb@gmail.com","threadId":"25371","inReplyTo":"1286402526-13143-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv5 01/17] t/test-lib.sh: Export also GIT_BUILD_DIR in test_external","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-06T22:01:46Z","receivedAt":"2010-10-06T22:01:46Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This way we can use it in test scripts written in other languages\n(e.g. in Perl), and not rely on \"$TEST_DIRECTORY/..\"\n\nSigned-off-by: Jakub Narebski \n---\nThis is not strictly required, but if we modified t/gitweb-lib.sh to\nuse GIT_BUILD_DIR, we would want to do the same in the test scripts\nfor caching interface etc. included in this series.\n\nThis patch was not present in previous version of this series, and it\nhas no equivalent in original patch by J.H.\n\n t/test-lib.sh | 4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n mode change 100644 => 100755 t/test-lib.sh\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 830e5e7..968d240 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -510,9 +510,9 @@ test_external () {\n \t\t# Announce the script to reduce confusion about the\n \t\t# test output that follows.\n \t\tsay_color \"\" \"# run $test_count: $descr ($*)\"\n-\t\t# Export TEST_DIRECTORY, TRASH_DIRECTORY and GIT_TEST_LONG\n+\t\t# Export TEST_DIRECTORY, GIT_BUILD_DIR, TRASH_DIRECTORY and GIT_TEST_LONG\n \t\t# to be able to use them in script\n-\t\texport TEST_DIRECTORY TRASH_DIRECTORY GIT_TEST_LONG\n+\t\texport TEST_DIRECTORY GIT_BUILD_DIR TRASH_DIRECTORY GIT_TEST_LONG\n \t\t# Run command; redirect its stderr to &4 as in\n \t\t# test_run_, but keep its stdout on our stdout even in\n \t\t# non-verbose mode.\n-- \n1.7.3\n"},{"id":"152785","messageId":"1286402526-13143-3-git-send-email-jnareb@gmail.com","threadId":"25371","inReplyTo":"1286402526-13143-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv5 02/17] gitweb: Prepare for splitting gitweb","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-06T22:01:47Z","receivedAt":"2010-10-06T22:01:47Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Prepare gitweb for having been split into modules that are to be\ninstalled alongside gitweb in 'lib/' subdirectory, by adding\n\n use lib __DIR__.'/lib';\n\nto gitweb.perl (to main gitweb script), and preparing for putting\nmodules (relative path) in $(GITWEB_MODULES) in gitweb/Makefile.\n\nThis preparatory work allows to add new module to gitweb by simply\nadding\n\n GITWEB_MODULES += \n\nto gitweb/Makefile (assuming that the module is in 'gitweb/lib/'\ndirectory).\n\nWhile at it pass GITWEBLIBDIR in addition to GITWEB_TEST_INSTALLED\nto test instaleed version of gitweb and installed version of modules\n(for tests which check individual (sub)modules).\n\n\nUsing __DIR__ from Dir::Self module (not in core, that's why currently\ngitweb includes excerpt of code from Dir::Self defining __DIR__) was\nchosen over using FindBin-based solution (in core since perl 5.00307,\nwhile gitweb itself requires at least perl 5.8.0) because FindBin uses\nBEGIN block, which is a problem under mod_perl and other persistent\nPerl environments (thought there are workarounds).\n\nAt Pavan Kumar Sankara suggestion gitweb/Makefile uses\n\n install [OPTION]... SOURCE... DIRECTORY\n\nformat (2nd format) with single SOURCE rather than\n\n install [OPTION]... SOURCE DEST\n\nformat (1st format) because of security reasons (race conditions).\nModern GNU install has `-T' / `--no-target-directory' option, but we\ncannot rely that the $(INSTALL) we are using supports this option.\n\nThe install-modules target in gitweb/Makefile uses shell 'for' loop,\ninstead of make's $(foreach) function, to avoid possible problem with\ngenerating a command line that exceeded the maximum argument list\nlength.\n\nSigned-off-by: Jakub Narebski \n---\nThis patch can serve as a way to introduce infrastructure required for\nsplitting gitweb into modules for better maintability. This issue was\nto be first part (first milestone) for \"Integrated web client for git\"\nGSoC2010 project by Pavan Kumar Sankara (with Christian Couder as\nmentor, and Petr 'Pasky' Baudis as co-mentor)... which unfortunately\nfailed middterm evaluations.\n\nAs you can see later in series it makes it very easy to add new gitweb\nmodules which are to be installed. Just add\n\n GITWEB_MODULES += Module/Foo.pm\n\nto gitweb/Makefile for Module::Foo (present in gitweb/lib/Module/Foo.pm)\nand 'install-gitweb' target would automatically install it along with\ngitweb.\n\nDifferences from v4:\n* The 'install-modules' target in gitweb/Makefile uses shell 'for' loop,\n instead of make's $(foreach) function; see commit message\n\n* Modules in GITWEB_MODULES would not include the 'lib/' prefix, so\n for module in gitweb/lib/Module/Foo.pm it would be\n GITWEB_MODULES += Module/Foo.pm\n and not\n GITWEB_MODULES += lib/Module/Foo.pm\n\n* Installing modules is left for separate 'install-modules' target,\n which is prerequisite for 'install' target\n\n* Includes hunk about 'test-installed' target, making it possible to\n test installed modules and not only installed gitweb.\n\n* 'install' and 'install-modules' targets are marked as .PHONY\n\n* Does not contain spurious vertical whitespace change (adding empty\n line before 'binmode' invocation).\n\n* Commit message is much longer and more detailed.\n\n\nDifferences from relevant parts of J.H. patch (probably apply also to\nthe gitweb version running git.kernel.org):\n* It uses standard Perl way of naming and installing modules; the main\n caching module is named GitwebCache::CacheOutput and it is in the\n gitweb/lib/GitwebCache/CacheOutput.pm and is loaded using 'require'\n and found thanks to 'use lib', rather than being in gitweb/cache.pm\n and loaded using \"do 'cache.pm'\", and found by path.\n\n This has the advantage of automatically detecting errors in the\n caching module. (Note that this patch series up to and including v2\n also used 'do $cache_pm', but implemented error handling, while v3\n used 'require $cache_pm' i.e. also finding cache module by path.)\n\n* Instead of one 'cache.pm' file, there are separate Perl modules for\n handling different parts of work: file-based cache, capturing gitweb\n output, and caching captured output are all in separate modules.\n\n The code of those modules are made generic and disentangled from\n gitweb.perl code, which makes it possible to test those modules in\n isolation.\n\n* 'make install-gitweb' is newer than original patch by J.H.; this\n patch series makes it possible to install gitweb, its dependencies\n (static files) _and modules_, using this one simple command.\n\n gitweb/Makefile | 17 +++++++++++++++--\n gitweb/gitweb.perl | 8 ++++++++\n 2 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex df908a1..c2d72e4 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -56,6 +56,7 @@ PERL_PATH ?= /usr/bin/perl\n bindir_SQ = $(subst ','\\'',$(bindir))#'\n gitwebdir_SQ = $(subst ','\\'',$(gitwebdir))#'\n gitwebstaticdir_SQ = $(subst ','\\'',$(gitwebdir)/static)#'\n+gitweblibdir_SQ = $(subst ','\\'',$(gitwebdir)/lib)#'\n SHELL_PATH_SQ = $(subst ','\\'',$(SHELL_PATH))#'\n PERL_PATH_SQ = $(subst ','\\'',$(PERL_PATH))#'\n DESTDIR_SQ = $(subst ','\\'',$(DESTDIR))#'\n@@ -151,20 +152,32 @@ test:\n \n test-installed:\n \tGITWEB_TEST_INSTALLED='$(DESTDIR_SQ)$(gitwebdir_SQ)' \\\n+\tGITWEBLIBDIR='$(DESTDIR_SQ)$(gitweblibdir_SQ)' \\\n \t\t$(MAKE) -C ../t gitweb-test\n \n ### Installation rules\n \n-install: all\n+install: all install-modules\n \t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebdir_SQ)'\n \t$(INSTALL) -m 755 $(GITWEB_PROGRAMS) '$(DESTDIR_SQ)$(gitwebdir_SQ)'\n \t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n \t$(INSTALL) -m 644 $(GITWEB_FILES) '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n \n+install-modules:\n+\tinstall_dirs=\"$(sort $(dir $(GITWEB_MODULES)))\" && \\\n+\tfor dir in $$install_dirs; do \\\n+\t\ttest -d '$(DESTDIR_SQ)$(gitweblibdir_SQ)/$$dir' || \\\n+\t\t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitweblibdir_SQ)/$$dir'; \\\n+\tdone\n+\tgitweb_modules=\"$(GITWEB_MODULES)\" && \\\n+\tfor mod in $$gitweb_modules; do \\\n+\t\t$(INSTALL) -m 644 lib/$$mod '$(DESTDIR_SQ)$(gitweblibdir_SQ)/$$(dirname $$mod)'; \\\n+\tdone\n+\n ### Cleaning rules\n \n clean:\n \t$(RM) gitweb.cgi static/gitweb.min.js static/gitweb.min.css GITWEB-BUILD-OPTIONS\n \n-.PHONY: all clean install test test-installed .FORCE-GIT-VERSION-FILE FORCE\n+.PHONY: all clean install install-modules test test-installed .FORCE-GIT-VERSION-FILE FORCE\n \ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex d521b4c..e4c08ba 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -9,6 +9,14 @@\n \n use strict;\n use warnings;\n+\n+use File::Spec;\n+# __DIR__ is taken from Dir::Self __DIR__ fragment\n+sub __DIR__ () {\n+\tFile::Spec->rel2abs(join '', (File::Spec->splitpath(__FILE__))[0, 1]);\n+}\n+use lib __DIR__ . '/lib';\n+\n use CGI qw(:standard :escapeHTML -nosticky);\n use CGI::Util qw(unescape);\n use CGI::Carp qw(fatalsToBrowser set_message);\n-- \n1.7.3\n"},{"id":"152786","messageId":"1286402526-13143-4-git-send-email-jnareb@gmail.com","threadId":"25371","inReplyTo":"1286402526-13143-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv5 03/17] gitweb/lib - Very simple file based cache","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-06T22:01:48Z","receivedAt":"2010-10-06T22:01:48Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This is first step towards implementing file based output (response)\ncaching layer that is used on such large sites as kernel.org.\n\nThis patch introduces GitwebCaching::SimpleFileCache package, which\nfollows Cache::Cache / CHI interface, although do not implement it\nfully. The intent of following established convention for cache\ninterface is to be able to replace our simple file based cache,\ne.g. by the one using memcached.\n\nLike in original patch by John 'Warthog9' Hawley (J.H.) (the one this\ncommit intends to be incremental step to), the data is stored in the\ncase as-is, without adding metadata (like expiration date), and\nwithout serialization (which means that one can store only scalar\ndata).\n\nTo be implemented (from original patch by J.H.):\n* cache expiration (based on file stats, current time and global\n expiration time); currently elements in cache do not expire at all\n* actually using this cache in gitweb, with exception of error pages\n* adaptive cache expiration, based on average system load\n* optional locking interface, where only one process can update cache\n (using flock)\n* server-side progress indicator when waiting for filling cache,\n which in turn requires separating situations (like snapshots and\n other non-HTML responses) where we should not show 'please wait'\n message\n\nPossible extensions (beyound what was in original patch):\n* (optionally) show information about cache utilization\n* AJAX (JavaScript-based) progress indicator\n* JavaScript code to update relative dates in cached output\n* make cache size-aware (try to not exceed specified maximum size)\n* utilize X-Sendfile header (or equivalent) to show cached data\n (optional, as it makes sense only if web server supports sendfile\n feature and have it enabled)\n* variable expiration feature from CHI, allowing items to expire a bit\n earlier than the stated expiration time to prevent cache miss\n stampedes (although locking, if available, should take care of\n this)\n\nThe code of GitwebCaching::SimpleFileCache package in gitweb/lib\nwas heavily based on file-based cache in Cache::Cache package, i.e.\non Cache::FileCache, Cache::FileBackend and Cache::BaseCache, and on\nfile-based cache in CHI, i.e. on CHI::Driver::File and CHI::Driver\n(including implementing atomic write, something that original patch\nlacks). It tries to follow more modern CHI architecture, but without\nrequiring Moose. It is much simplified compared to both interfaces\nand their file-based drivers.\n\nThis patch does not yet enable output caching in gitweb (it doesn't\nhave all required features yet); on the other hand it includes tests\nof cache Perl API in t/t9503-gitweb-caching-interface.sh.\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\nDifferences from v4:\n* The test got renamed a bit, because separate external tests would be\n invoked by separate test scripts, for 'prove' to work correctly\n (as test_external doesn't yet work as subtest).\n\n The t/t9503-gitweb-caching-interface.sh gets a long description,\n and is updated to follow conventions used post-TAPification.\n\n* The t/t9503/test_cache_interface.pl uses GIT_BUILD_DIR rather than\n TEST_DIRECTORY, and respect GITWEBLIBDIR to make it possible to test\n installed version of module.\n\n* Generating a temporary file to write to is refactored into \"private\"\n method _tempfile_to_path\n\n* The call to 'mkdir' is enveloped in eval, as it dies on error.\n make_path would be better here, but is not available for File::Path\n before v2.00, which is in core only since 5.10\n\n\nDifferences from relevant parts of J.H. patch:\n* Does not use make_path, but older mkpath interface from File::Path;\n this way we do not require File::Path version >= 2.0.\n\n* Writes to cache file atomically, by writing to unique temporary file\n (from File::Temp at this point of series) and then renaming/moving\n this file to the cache file. This is to make caching more robust,\n by preventing readers from getting partial data. The issue this\n prevents should be very rare, as we write whole data (whole page) at\n once, after it is generated in full, so this situation can\n theoretically happen only in the case of heavy load, many clients,\n and very large page (over the size of (file)system cache).\n\n* Error reporting is done via \"die\", not via die_error from gitweb.\n This is to avoid entangling cache.pm and gitweb.perl code (making\n cache.pm dependent on gitweb). This can be improved further by\n passing die handler (wrapper around die_error in case of gitweb) to\n cache (see CHI source code for how it can be solved).\n\n* The depth of cache hierarchy is configurable, while J.H. patch uses\n hardcoded depth of 1 subdirectory. It uses unpack rather than substr.\n For 06b90e786e... key digest it uses 06/b90e786e... (with cache_depth = 1)\n rather than 06/06b90e786e... in J.H. patch. \n\n* GitwebCache::SimpleFileCache uses OO (object-oriented) interface,\n with constructor that takes param names that are compatibile with\n other cache engines (like CHI or Cache::Cache).\n\n GitwebCache::SimpleFileCache is a proper Perl module.\n\n* GitwebCache::SimpleFileCache uses standard ->get, ->set, ->compute\n methods, which should allow easy replacing it with CHI or Cache::Cache\n cache (see later commit, adding caching support to gitweb).\n\n* Support for namespaces (though default is to have no namespace, at\n least at this point of series).\n\n* ->fetch($key) and ->store($key, $data) use fast slurp / spew,\n adapted from File::Slurp read and write subroutines, via\n CHI::Driver::File\n\n* Tests of caching API in t/t9503-gitweb-caching-interface.sh\n\n* Better Perl style, more consistent with what gitweb itself uses.\n\n\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 324 +++++++++++++++++++++++++++++\n t/t9503-gitweb-caching-interface.sh | 34 +++\n t/t9503/test_cache_interface.pl | 81 +++++++\n 3 files changed, 439 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm\n create mode 100755 t/t9503-gitweb-caching-interface.sh\n create mode 100755 t/t9503/test_cache_interface.pl\n\ndiff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm\nnew file mode 100644\nindex 0000000..a54f78f\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm\n@@ -0,0 +1,324 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2006, John 'Warthog9' Hawley \n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Gitweb caching engine, simple file-based cache\n+#\n+\n+# Minimalistic cache that stores data in the filesystem, without serialization\n+# and currently without any kind of cache expiration (all keys last forever till\n+# they got explicitely removed).\n+#\n+# It follows Cache::Cache and CHI interfaces (but does not implement it fully)\n+\n+package GitwebCache::SimpleFileCache;\n+\n+use strict;\n+use warnings;\n+\n+use File::Path qw(mkpath);\n+use File::Temp qw(tempfile);\n+use Digest::MD5 qw(md5_hex);\n+\n+# by default, the cache nests all entries on the filesystem single\n+# directory deep, i.e. '60/b725f10c9c85c70d97880dfe8191b3' for\n+# key name (key digest) 60b725f10c9c85c70d97880dfe8191b3.\n+#\n+our $DEFAULT_CACHE_DEPTH = 1;\n+\n+# by default, the root of the cache is located in 'cache'.\n+#\n+our $DEFAULT_CACHE_ROOT = \"cache\";\n+\n+# by default we don't use cache namespace (empty namespace);\n+# empty namespace does not allow for simple implementation of clear() method.\n+#\n+our $DEFAULT_NAMESPACE = '';\n+\n+# ......................................................................\n+# constructor\n+\n+# The options are set by passing in a reference to a hash containing\n+# any of the following keys:\n+# * 'namespace'\n+# The namespace associated with this cache. This allows easy separation of\n+# multiple, distinct caches without worrying about key collision. Defaults\n+# to $DEFAULT_NAMESPACE.\n+# * 'cache_root' (Cache::FileCache compatibile),\n+# 'root_dir' (CHI::Driver::File compatibile),\n+# The location in the filesystem that will hold the root of the cache.\n+# Defaults to $DEFAULT_CACHE_ROOT.\n+# * 'cache_depth' (Cache::FileCache compatibile),\n+# 'depth' (CHI::Driver::File compatibile),\n+# The number of subdirectories deep to cache object item. This should be\n+# large enough that no cache directory has more than a few hundred objects.\n+# Defaults to $DEFAULT_CACHE_DEPTH unless explicitly set.\n+sub new {\n+\tmy ($proto, $p_options_hash_ref) = @_;\n+\n+\tmy $class = ref($proto) || $proto;\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\tmy ($root, $depth, $ns);\n+\tif (defined $p_options_hash_ref) {\n+\t\t$root =\n+\t\t\t$p_options_hash_ref->{'cache_root'} ||\n+\t\t\t$p_options_hash_ref->{'root_dir'};\n+\t\t$depth =\n+\t\t\t$p_options_hash_ref->{'cache_depth'} ||\n+\t\t\t$p_options_hash_ref->{'depth'};\n+\t\t$ns = $p_options_hash_ref->{'namespace'};\n+\t}\n+\t$root = $DEFAULT_CACHE_ROOT unless defined($root);\n+\t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n+\t$ns = $DEFAULT_NAMESPACE unless defined($ns);\n+\n+\t$self->set_root($root);\n+\t$self->set_depth($depth);\n+\t$self->set_namespace($ns);\n+\n+\treturn $self;\n+}\n+\n+# ......................................................................\n+# accessors\n+\n+# http://perldesignpatterns.com/perldesignpatterns.html#AccessorPattern\n+\n+# creates get_depth() and set_depth($depth) etc. methods\n+foreach my $i (qw(depth root namespace)) {\n+\tmy $field = $i;\n+\tno strict 'refs';\n+\t*{\"get_$field\"} = sub {\n+\t\tmy $self = shift;\n+\t\treturn $self->{$field};\n+\t};\n+\t*{\"set_$field\"} = sub {\n+\t\tmy ($self, $value) = @_;\n+\t\t$self->{$field} = $value;\n+\t};\n+}\n+\n+# ----------------------------------------------------------------------\n+# utility functions and methods\n+\n+# Return root dir for namespace (lazily built, cached)\n+sub path_to_namespace {\n+\tmy ($self) = @_;\n+\n+\tif (!exists $self->{'path_to_namespace'}) {\n+\t\tif (defined $self->{'namespace'} &&\n+\t\t $self->{'namespace'} ne '') {\n+\t\t\t$self->{'path_to_namespace'} = \"$self->{'root'}/$self->{'namespace'}\";\n+\t\t} else {\n+\t\t\t$self->{'path_to_namespace'} = $self->{'root'};\n+\t\t}\n+\t}\n+\treturn $self->{'path_to_namespace'};\n+}\n+\n+# $path = $cache->path_to_key($key);\n+# $path = $cache->path_to_key($key, \\$dir);\n+#\n+# Take an human readable key, and return file path.\n+# Puts dirname of file path in second argument, if it is provided.\n+sub path_to_key {\n+\tmy ($self, $key, $dir_ref) = @_;\n+\n+\tmy @paths = ( $self->path_to_namespace() );\n+\n+\t# Create a unique (hashed) key from human readable key\n+\tmy $filename = md5_hex($key); # or $digester->add($key)->hexdigest();\n+\n+\t# Split filename so that it have DEPTH subdirectories,\n+\t# where each subdirectory has a two-letter name\n+\tpush @paths, unpack(\"(a2)\".($self->{'depth'} || 1).\" a*\", $filename);\n+\t$filename = pop @paths;\n+\n+\t# Join paths together, computing dir separately if $dir_ref was passed.\n+\tmy $filepath;\n+\tif (defined $dir_ref && ref($dir_ref)) {\n+\t\tmy $dir = join('/', @paths);\n+\t\t$filepath = \"$dir/$filename\";\n+\t\t$$dir_ref = $dir;\n+\t} else {\n+\t\t$filepath = join('/', @paths, $filename);\n+\t}\n+\n+\treturn $filepath;\n+}\n+\n+# ----------------------------------------------------------------------\n+# \"private\" utility functions and methods\n+\n+# take a file path to cache entry, and its directory\n+# return filehandle and filename of open temporary file,\n+# like File::Temp::tempfile\n+sub _tempfile_to_path {\n+\tmy ($file, $dir) = @_;\n+\n+\t# tempfile will croak() if there is an error\n+\treturn tempfile(\"${file}_XXXXX\",\n+\t\t#DIR => $dir,\n+\t\t'UNLINK' => 0, # ensure that we don't unlink on close; file is renamed\n+\t\t'SUFFIX' => '.tmp');\n+}\n+\n+\n+# ----------------------------------------------------------------------\n+# worker methods\n+\n+sub fetch {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $file = $self->path_to_key($key);\n+\treturn undef unless (defined $file && -f $file);\n+\n+\t# Fast slurp, adapted from File::Slurp::read, with unnecessary options removed\n+\t# via CHI::Driver::File (from CHI-0.33)\n+\tmy $buf = '';\n+\topen my $read_fh, '<', $file\n+\t\tor return undef;\n+\tbinmode $read_fh, ':raw';\n+\n+\tmy $size_left = -s $read_fh;\n+\n+\twhile ($size_left > 0) {\n+\t\tmy $read_cnt = sysread($read_fh, $buf, $size_left, length($buf));\n+\t\treturn undef unless defined $read_cnt;\n+\n+\t\tlast if $read_cnt == 0;\n+\t\t$size_left -= $read_cnt;\n+\t\t#last if $size_left <= 0;\n+\t}\n+\n+\tclose $read_fh\n+\t\tor die \"Couldn't close file '$file' opened for reading: $!\";\n+\treturn $buf;\n+}\n+\n+sub store {\n+\tmy ($self, $key, $data) = @_;\n+\n+\tmy $dir;\n+\tmy $file = $self->path_to_key($key, \\$dir);\n+\treturn undef unless (defined $file && defined $dir);\n+\n+\t# ensure that directory leading to cache file exists\n+\tif (!-d $dir) {\n+\t\teval { mkpath($dir, 0, 0777); 1 }\n+\t\t\tor die \"Couldn't create leading directory '$dir' (mkpath): $!\";\n+\t}\n+\n+\t# generate a temporary file\n+\tmy ($temp_fh, $tempname) = $self->_tempfile_to_path($file, $dir);\n+\tchmod 0666, $tempname\n+\t\tor warn \"Couldn't change permissions to 0666 / -rw-rw-rw- for '$tempname': $!\";\n+\n+\t# Fast spew, adapted from File::Slurp::write, with unnecessary options removed\n+\t# via CHI::Driver::File (from CHI-0.33)\n+\tmy $write_fh = $temp_fh;\n+\tbinmode $write_fh, ':raw';\n+\n+\tmy $size_left = length($data);\n+\tmy $offset = 0;\n+\n+\twhile ($size_left > 0) {\n+\t\tmy $write_cnt = syswrite($write_fh, $data, $size_left, $offset);\n+\t\treturn undef unless defined $write_cnt;\n+\t\t\n+\t\t$size_left -= $write_cnt;\n+\t\t$offset += $write_cnt; # == length($data);\n+\t}\n+\n+\tclose $temp_fh\n+\t\tor die \"Couldn't close temporary file '$tempname' opened for writing: $!\";\n+\trename($tempname, $file)\n+\t\tor die \"Couldn't rename temporary file '$tempname' to '$file': $!\";\n+}\n+\n+# get size of an element associated with the $key (not the size of whole cache)\n+sub get_size {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $path = $self->path_to_key($key)\n+\t\tor return undef;\n+\tif (-f $path) {\n+\t\treturn -s $path;\n+\t}\n+\treturn 0;\n+}\n+\n+# ......................................................................\n+# interface methods\n+\n+# Removing and expiring\n+\n+# $cache->remove($key)\n+#\n+# Remove the data associated with the $key from the cache.\n+sub remove {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $file = $self->path_to_key($key)\n+\t\tor return undef;\n+\treturn undef unless -f $file;\n+\tunlink($file)\n+\t\tor die \"Couldn't remove file '$file': $!\";\n+}\n+\n+# Getting and setting\n+\n+# $cache->set($key, $data);\n+#\n+# Associates $data with $key in the cache, overwriting any existing entry.\n+# Returns $data.\n+sub set {\n+\tmy ($self, $key, $data) = @_;\n+\n+\treturn unless (defined $key && defined $data);\n+\n+\t$self->store($key, $data);\n+\n+\treturn $data;\n+}\n+\n+# $data = $cache->get($key);\n+#\n+# Returns the data associated with $key. If $key does not exist\n+# or has expired, returns undef.\n+sub get {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $data = $self->fetch($key)\n+\t\tor return undef;\n+\n+\treturn $data;\n+}\n+\n+# $data = $cache->compute($key, $code);\n+#\n+# Combines the get and set operations in a single call. Attempts to\n+# get $key; if successful, returns the value. Otherwise, calls $code\n+# and uses the return value as the new value for $key, which is then\n+# returned.\n+sub compute {\n+\tmy ($self, $key, $code) = @_;\n+\n+\tmy $data = $self->get($key);\n+\tif (!defined $data) {\n+\t\t$data = $code->($self, $key);\n+\t\t$self->set($key, $data);\n+\t}\n+\n+\treturn $data;\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::SimpleFileCache;\ndiff --git a/t/t9503-gitweb-caching-interface.sh b/t/t9503-gitweb-caching-interface.sh\nnew file mode 100755\nindex 0000000..819da1d\n--- /dev/null\n+++ b/t/t9503-gitweb-caching-interface.sh\n@@ -0,0 +1,34 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Jakub Narebski\n+#\n+\n+test_description='gitweb caching interface\n+\n+This test checks caching interface used in gitweb caching, and caching\n+infrastructure (GitwebCache::* modules).'\n+\n+# for now we are running only cache interface tests\n+. ./test-lib.sh\n+\n+# this test is present in gitweb-lib.sh\n+if ! test_have_prereq PERL; then\n+\tskip_all='perl not available, skipping test'\n+\ttest_done\n+fi\n+\n+\"$PERL_PATH\" -MTest::More -e 0 >/dev/null 2>&1 || {\n+\tskip_all='perl module Test::More unavailable, skipping test'\n+\ttest_done\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+# The external test will outputs its own plan\n+test_external_has_tap=1\n+\n+test_external \\\n+\t'GitwebCache::* Perl API (in gitweb/lib/)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_cache_interface.pl\n+\n+test_done\ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nnew file mode 100755\nindex 0000000..6a7b715\n--- /dev/null\n+++ b/t/t9503/test_cache_interface.pl\n@@ -0,0 +1,81 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use Test::More;\n+\n+# test source version\n+use lib $ENV{GITWEBLIBDIR} || \"$ENV{GIT_BUILD_DIR}/gitweb/lib\";\n+\n+# Test creating a cache\n+#\n+BEGIN { use_ok('GitwebCache::SimpleFileCache'); }\n+my $cache = new_ok('GitwebCache::SimpleFileCache');\n+\n+# Test that default values are defined\n+#\n+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,\n+\t'$DEFAULT_CACHE_ROOT defined');\n+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,\n+\t'$DEFAULT_CACHE_DEPTH defined');\n+\n+# Test accessors and default values for cache\n+#\n+SKIP: {\n+\tskip 'default values not defined', 3\n+\t\tunless ($GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT &&\n+\t\t $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH);\n+\n+\tis($cache->get_namespace(), '', \"default namespace is ''\");\n+\tcmp_ok($cache->get_root(), 'eq', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,\n+\t\t\"default cache root is '$GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT'\");\n+\tcmp_ok($cache->get_depth(), '==', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,\n+\t\t\"default cache depth is $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH\");\n+}\n+\n+# Test the getting, setting, and removal of a cached value\n+# (Cache::Cache interface)\n+#\n+my $key = 'Test Key';\n+my $value = 'Test Value';\n+\n+subtest 'Cache::Cache interface' => sub {\n+\tforeach my $method (qw(get set remove)) {\n+\t\tcan_ok($cache, $method);\n+\t}\n+\n+\t$cache->set($key, $value);\n+\tcmp_ok($cache->get_size($key), '>', 0, 'get_size after set, is greater than 0');\n+\tis($cache->get($key), $value, 'get after set, returns cached value');\n+\t$cache->remove($key);\n+\tok(!defined($cache->get($key)), 'get after remove, is undefined');\n+\n+\teval { $cache->remove('Not-Existent Key'); };\n+\tok(!$@, 'remove on non-existent key doesn\\'t die');\n+\tdiag($@) if $@;\n+\n+\tdone_testing();\n+};\n+\n+# Test the getting and setting of a cached value\n+# (CHI interface)\n+#\n+my $call_count = 0;\n+sub get_value {\n+\t$call_count++;\n+\treturn $value;\n+}\n+subtest 'CHI interface' => sub {\n+\tcan_ok($cache, qw(compute));\n+\n+\tis($cache->compute($key, \\&get_value), $value, \"compute 1st time (set) returns '$value'\");\n+\tis($cache->compute($key, \\&get_value), $value, \"compute 2nd time (get) returns '$value'\");\n+\tis($cache->compute($key, \\&get_value), $value, \"compute 3rd time (get) returns '$value'\");\n+\tcmp_ok($call_count, '==', 1, 'get_value() is called once from compute');\n+\n+\tdone_testing();\n+};\n+\n+done_testing();\n-- \n1.7.3\n"},{"id":"152788","messageId":"1286402526-13143-5-git-send-email-jnareb@gmail.com","threadId":"25371","inReplyTo":"1286402526-13143-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv5 04/17] gitweb/lib - Stat-based cache expiration","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-06T22:01:49Z","receivedAt":"2010-10-06T22:01:49Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add stat-based cache expiration to file-based GitwebCache::SimpleFileCache.\nContrary to the way other caching interfaces such as Cache::Cache and CHI\ndo it, the time cache element expires in is _global_ value associated with\ncache instance, and is not local property of cache entry. (Currently cache\nentry does not store any metadata associated with entry... which means that\nthere is no need for serialization / marshalling / freezing and thawing.)\nDefault expire time is -1, which means never expire.\n\nTo check if cache entry is expired, GitwebCache::SimpleFileCache compares\ndifference between mtime (last modify time) of a cache file and current time\nwith (global) time to expire. It is done using CHI-compatibile is_valid()\nmethod.\n\nAdd some tests checking that expiring works correctly (on the level of API).\n\nTo be implemented (from original patch by J.H.):\n* actually using this cache in gitweb, except error pages\n* adaptive cache expiration, based on average system load\n* optional locking interface, where only one process can update cache\n (using flock)\n* server-side progress indicator when waiting for filling cache,\n which in turn requires separating situations (like snapshots and\n other non-HTML responses) where we should not show 'please wait'\n message\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\nNow that caching engine supports cache expiration, we can add caching\nsupport to gitweb.\n\nNo significant differences from v4 version of this patch.\n\nDifferences from relevant parts of J.H. patch:\n* It simply uses stat on last accessed file (checked for existence),\n instead of opening file for reading (without error detection!), running\n stat on it, and then closing it.\n\n* One can use expire time of -1 (or to be more exact less than 0) to set\n expire time to never (cache is considered fresh forever, does not expire).\n\n* There are some tests in t9503 of cache expiration (one of those assume\n that expire time of one day would be not expired in get after set).\n\n* It reuses stat structure (stat(_)), and calculates current time only\n once.\n\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 39 +++++++++++++++++++++++++++-\n t/t9503/test_cache_interface.pl | 19 ++++++++++++++\n 2 files changed, 56 insertions(+), 2 deletions(-)\n\ndiff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm\nindex a54f78f..74d7246 100644\n--- a/gitweb/lib/GitwebCache/SimpleFileCache.pm\n+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm\n@@ -57,6 +57,10 @@ our $DEFAULT_NAMESPACE = '';\n # The number of subdirectories deep to cache object item. This should be\n # large enough that no cache directory has more than a few hundred objects.\n # Defaults to $DEFAULT_CACHE_DEPTH unless explicitly set.\n+# * 'default_expires_in' (Cache::Cache compatibile),\n+# 'expires_in' (CHI compatibile) [seconds]\n+# The expiration time for objects place in the cache.\n+# Defaults to -1 (never expire) if not explicitly set.\n sub new {\n \tmy ($proto, $p_options_hash_ref) = @_;\n \n@@ -64,7 +68,7 @@ sub new {\n \tmy $self = {};\n \t$self = bless($self, $class);\n \n-\tmy ($root, $depth, $ns);\n+\tmy ($root, $depth, $ns, $expires_in);\n \tif (defined $p_options_hash_ref) {\n \t\t$root =\n \t\t\t$p_options_hash_ref->{'cache_root'} ||\n@@ -73,14 +77,19 @@ sub new {\n \t\t\t$p_options_hash_ref->{'cache_depth'} ||\n \t\t\t$p_options_hash_ref->{'depth'};\n \t\t$ns = $p_options_hash_ref->{'namespace'};\n+\t\t$expires_in =\n+\t\t\t$p_options_hash_ref->{'default_expires_in'} ||\n+\t\t\t$p_options_hash_ref->{'expires_in'};\n \t}\n \t$root = $DEFAULT_CACHE_ROOT unless defined($root);\n \t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n \t$ns = $DEFAULT_NAMESPACE unless defined($ns);\n+\t$expires_in = -1 unless defined($expires_in); # <0 means never\n \n \t$self->set_root($root);\n \t$self->set_depth($depth);\n \t$self->set_namespace($ns);\n+\t$self->set_expires_in($expires_in);\n \n \treturn $self;\n }\n@@ -91,7 +100,7 @@ sub new {\n # http://perldesignpatterns.com/perldesignpatterns.html#AccessorPattern\n \n # creates get_depth() and set_depth($depth) etc. methods\n-foreach my $i (qw(depth root namespace)) {\n+foreach my $i (qw(depth root namespace expires_in)) {\n \tmy $field = $i;\n \tno strict 'refs';\n \t*{\"get_$field\"} = sub {\n@@ -272,6 +281,31 @@ sub remove {\n \t\tor die \"Couldn't remove file '$file': $!\";\n }\n \n+# $cache->is_valid($key)\n+#\n+# Returns a boolean indicating whether $key exists in the cache\n+# and has not expired (global per-cache 'expires_in').\n+sub is_valid {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $path = $self->path_to_key($key);\n+\n+\t# does file exists in cache?\n+\treturn 0 unless -f $path;\n+\t# get its modification time\n+\tmy $mtime = (stat(_))[9] # _ to reuse stat structure used in -f test\n+\t\tor die \"Couldn't stat file '$path': $!\";\n+\n+\t# expire time can be set to never\n+\tmy $expires_in = $self->get_expires_in();\n+\treturn 1 unless (defined $expires_in && $expires_in >= 0);\n+\n+\t# is file expired?\n+\tmy $now = time();\n+\n+\treturn (($now - $mtime) < $expires_in);\n+}\n+\n # Getting and setting\n \n # $cache->set($key, $data);\n@@ -295,6 +329,7 @@ sub set {\n sub get {\n \tmy ($self, $key) = @_;\n \n+\treturn undef unless $self->is_valid($key);\n \tmy $data = $self->fetch($key)\n \t\tor return undef;\n \ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nindex 6a7b715..adca88d 100755\n--- a/t/t9503/test_cache_interface.pl\n+++ b/t/t9503/test_cache_interface.pl\n@@ -78,4 +78,23 @@ subtest 'CHI interface' => sub {\n \tdone_testing();\n };\n \n+# Test cache expiration\n+#\n+subtest 'cache expiration' => sub {\n+\t$cache->set_expires_in(60*60*24); # set expire time to 1 day\n+\tcmp_ok($cache->get_expires_in(), '>', 0, '\"expires in\" is greater than 0');\n+\tis($cache->get($key), $value, 'get returns cached value (not expired in 1d)');\n+\n+\t$cache->set_expires_in(-1); # set expire time to never expire\n+\tis($cache->get_expires_in(), -1, '\"expires in\" is set to never (-1)');\n+\tis($cache->get($key), $value, 'get returns cached value (not expired)');\n+\n+\t$cache->set_expires_in(0);\n+\tis($cache->get_expires_in(), 0, '\"expires in\" is set to now (0)');\n+\t$cache->set($key, $value);\n+\tok(!defined($cache->get($key)), 'cache is expired');\n+\n+\tdone_testing();\n+};\n+\n done_testing();\n-- \n1.7.3\n"},{"id":"152789","messageId":"1286402526-13143-6-git-send-email-jnareb@gmail.com","threadId":"25371","inReplyTo":"1286402526-13143-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv5 05/17] gitweb/lib - Regenerate entry if the cache file has size of 0","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-06T22:01:50Z","receivedAt":"2010-10-06T22:01:50Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"If the file representing cache entry has size 0 (in bytes), the cache\nentry is considered invalid, regardless of its freshness, even if cache\nexpiration is turned off.\n\n[jh: This is a quick, and thankfully easy, check to regenerate the\ncache if the resulting file is of size 0. This *SHOULDN'T* happen,\nbut it is possible that it could and thus adding the check.]\n\nBased-on-commit-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\nThis patch is new and wasn't present in previous version of this\nseries.\n\nIt is inspired by commit 22eb7af (Gitweb - Regenerate the cache if the\nresulting file has size of 0, 2010-09-23) on 'master' branch of\ngit/warthog9/gitweb.git repository on git.kernel.org\n\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 2 ++\n 1 files changed, 2 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm\nindex 74d7246..6833699 100644\n--- a/gitweb/lib/GitwebCache/SimpleFileCache.pm\n+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm\n@@ -295,6 +295,8 @@ sub is_valid {\n \t# get its modification time\n \tmy $mtime = (stat(_))[9] # _ to reuse stat structure used in -f test\n \t\tor die \"Couldn't stat file '$path': $!\";\n+\t# cache entry is invalid if it is size 0 (in bytes)\n+\treturn 0 unless ((stat(_))[7] > 0);\n \n \t# expire time can be set to never\n \tmy $expires_in = $self->get_expires_in();\n-- \n1.7.3\n"},{"id":"152794","messageId":"1286402526-13143-7-git-send-email-jnareb@gmail.com","threadId":"25371","inReplyTo":"1286402526-13143-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv5 06/17] gitweb/lib - Simple select(FH) based output capture","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-06T22:01:51Z","receivedAt":"2010-10-06T22:01:51Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add two packages: GitwebCache::Capture, which defines interface, and\nGitwebCache::Capture::SelectFH, which is actually implements simple\ncapturing. GitwebCache::Capture::SelectFH captures output by using\nselect(FILEHANDLE) to change default filehandle for output. This\nmeans that output of a \"print\" or a \"printf\" (or a \"write\") without\na filehandle would be captured.\n\nTo change mode of filehandle used for capturing correctly,\n binmode select(), ;\nneeds to be used in place of\n binmode STDOUT, ;\n\nCapturing is done using in-memory file held in Perl scalar.\n\nUsing select(FILEHANDLE) is a bit fragile as a method of capturing\noutput, as it assumes that we always use \"print\" or \"printf\" without\nfilehandle, and use select() which returns default filehandle for\noutput in place of explicit STDOUT. On the other hand it has the\nadvantage of being simple. Alternate solutions include using tie\n(like in CGI::Cache), or using PerlIO layers - but the last requires\nnon-standard PerlIO::Util module.\n\n\nIncludes separate tests for capturing output.\n\nSigned-off-by: Jakub Narebski \n---\nYou can see alternate solutions for capturing output in 21/17 patch of\nthis series: \"gitweb/lib - Alternate ways of capturing output\" (an\nappendix to this series).\n\nDifferences from v4:\n* The capture interface tests are now invoked by a separate test\n script t/t9504-gitweb-capture-interface.sh, for 'prove' to work\n correctly (as test_external doesn't yet work as subtest).\n\n* The t/t9504/test_capture_interface.pl uses GIT_BUILD_DIR rather than\n TEST_DIRECTORY, and respect GITWEBLIBDIR to make it possible to test\n installed version of module.\n\n* Removed spurious changes and fixes to commits earlier in series\n (patch cleanup).\n\n\nDifferences from relevant parts of J.H. patch:\n* Capturing gitweb output will be done without need to modify gitweb\n to either save generated output into $output variable, and then\n print it or save it in cache after it is generated in full (original\n J.H. patch in \"Gitweb caching v2\"), or changing all print statements\n to print to explicit filehandle which points to STDOUT if caching is\n disabled and to in-memory file if caching is enabled (modified\n J.H. patch in \"Gitweb caching v5\").\n\n* Contrary to the '$output .= ' solution, and similar to the\n 'print {$out} ' or 'print $out ' (which can be thought of\n as explicit version of select($out)), this way of capturing output\n doesn't change gitweb behavior when caching is turned off; in\n particular it preserves streaming.\n\n Also the '$output .= ' solution can affect performance because\n of repeated string concatenation.\n\n* The most important issue is that I/O \"layers\" (PerlIO), like ':utf8'\n or ':raw', are *already applied* to the output that is captured.\n This means that captured output is *always* in binary (':raw') mode.\n In Perl 6 language it means that data returned by capturing engine\n is an equivalent of Buf, a collection of bytes, whether Buf or Str\n (a colection of logical characters) is printed.\n\n The overal result is that we would not need separate code path for\n caching binary output, and separate naming conventions for cache\n files for binary data.\n\n The t9504 test is about checking if both ':utf8' and ':raw' output\n is captured correctly.\n\n gitweb/lib/GitwebCache/Capture.pm | 66 ++++++++++++++++++++++\n gitweb/lib/GitwebCache/Capture/SelectFH.pm | 82 ++++++++++++++++++++++++++++\n t/t9504-gitweb-capture-interface.sh | 34 ++++++++++++\n t/t9504/test_capture_interface.pl | 76 ++++++++++++++++++++++++++\n 4 files changed, 258 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/Capture.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm\n create mode 100755 t/t9504-gitweb-capture-interface.sh\n create mode 100755 t/t9504/test_capture_interface.pl\n\ndiff --git a/gitweb/lib/GitwebCache/Capture.pm b/gitweb/lib/GitwebCache/Capture.pm\nnew file mode 100644\nindex 0000000..3e9fe81\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/Capture.pm\n@@ -0,0 +1,66 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Output capturing for gitweb caching engine\n+#\n+\n+# It is base abstract class (a role) for capturing output of gitweb\n+# actions for gitweb caching engine.\n+# \n+# Child (derived) concrete classes, which actually implement some method\n+# of capturing STDOUT output, must implement the following methods:\n+# * ->new(), to create new object of a capturing class\n+# * ->start(), to start capturing output\n+# * ->stop(), to stop capturing output and return it\n+#\n+# Before starting capture by using capture_block etc. subroutines,\n+# one has to run ->setup().\n+\n+package GitwebCache::Capture;\n+\n+use strict;\n+use warnings;\n+\n+use Exporter qw(import);\n+our @EXPORT = qw(capture_start capture_stop capture_block);\n+our @EXPORT_OK = qw(setup_capture);\n+our %EXPORT_TAGS = (all => [ @EXPORT, @EXPORT_OK ]);\n+\n+# Holds object used for capture (of child class)\n+my $capture;\n+\n+sub setup_capture {\n+\tmy $self = shift || __PACKAGE__;\n+\n+\t$capture = $self->new(@_);\n+}\n+\n+sub capture {\n+\tmy ($self, $code) = @_;\n+\n+\t$self->start();\n+\t$code->();\n+\treturn $self->stop();\n+}\n+\n+# Wrap caching data; capture only STDOUT\n+sub capture_block (&) {\n+\tmy $code = shift;\n+\treturn $capture->capture($code);\n+}\n+\n+sub capture_start {\n+\t$capture->start(@_);\n+}\n+\n+sub capture_stop {\n+\treturn $capture->stop(@_);\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::Capture;\ndiff --git a/gitweb/lib/GitwebCache/Capture/SelectFH.pm b/gitweb/lib/GitwebCache/Capture/SelectFH.pm\nnew file mode 100644\nindex 0000000..18ce5c3\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/Capture/SelectFH.pm\n@@ -0,0 +1,82 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Simple output capturing using select(FH);\n+#\n+\n+# This module (class) captures output of 'print ', 'printf '\n+# and 'write ' (without a filehandle) by using select(FILEHANDLE)\n+# to change default filehandle for output, changing it to in-memory\n+# file (saving output to scalar).\n+#\n+# Note that when using this simplest way of capturing, to change mode of\n+# filehandle using for capturing correctly, \"binmode STDOUT, ;\"\n+# has to be changed to \"binmode select(), ;\". This has no change\n+# if we are not capturing output using GitwebCache::Capture::SelectFH.\n+\n+package GitwebCache::Capture::SelectFH;\n+\n+use PerlIO;\n+\n+use strict;\n+use warnings;\n+\n+use base qw(GitwebCache::Capture);\n+use GitwebCache::Capture qw(:all);\n+\n+use Exporter qw(import);\n+our @EXPORT = @GitwebCache::Capture::EXPORT;\n+our @EXPORT_OK = @GitwebCache::Capture::EXPORT_OK;\n+our %EXPORT_TAGS = %GitwebCache::Capture::EXPORT_TAGS;\n+\n+# Constructor\n+sub new {\n+\tmy $proto = shift;\n+\n+\tmy $class = ref($proto) || $proto;\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\t$self->{'oldfh'} = select();\n+\t$self->{'data'} = '';\n+\n+\treturn $self;\n+}\n+\n+# Start capturing data (STDOUT)\n+# (printed using 'print ' or 'printf ')\n+sub start {\n+\tmy $self = shift;\n+\n+\t$self->{'data'} = '';\n+\t$self->{'data_fh'} = undef;\n+\t\n+\topen $self->{'data_fh'}, '>', \\$self->{'data'}\n+\t\tor die \"Couldn't open in-memory file for capture: $!\";\n+\t$self->{'oldfh'} = select($self->{'data_fh'});\n+\n+\t# note: this does not cover all cases\n+\tbinmode select(), ':utf8'\n+\t\tif ((PerlIO::get_layers($self->{'oldfh'}))[-1] eq 'utf8');\n+}\n+\n+# Stop capturing data (required for die_error)\n+sub stop {\n+\tmy $self = shift;\n+\n+\t# return if we didn't start capturing\n+\treturn unless defined $self->{'data_fh'};\n+\n+\tselect($self->{'oldfh'});\n+\tclose $self->{'data_fh'}\n+\t\tor die \"Couldn't close in-memory file for capture: $!\";\n+\treturn $self->{'data'};\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::Capture::SelectFH;\ndiff --git a/t/t9504-gitweb-capture-interface.sh b/t/t9504-gitweb-capture-interface.sh\nnew file mode 100755\nindex 0000000..82623f1\n--- /dev/null\n+++ b/t/t9504-gitweb-capture-interface.sh\n@@ -0,0 +1,34 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Jakub Narebski\n+#\n+\n+test_description='gitweb capturing interface\n+\n+This test checks capturing interface used for capturing gitweb output\n+in gitweb caching (GitwebCache::Capture* modules).'\n+\n+# for now we are running only cache interface tests\n+. ./test-lib.sh\n+\n+# this test is present in gitweb-lib.sh\n+if ! test_have_prereq PERL; then\n+\tskip_all='perl not available, skipping test'\n+\ttest_done\n+fi\n+\n+\"$PERL_PATH\" -MTest::More -e 0 >/dev/null 2>&1 || {\n+\tskip_all='perl module Test::More unavailable, skipping test'\n+\ttest_done\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+# The external test will outputs its own plan\n+test_external_has_tap=1\n+\n+test_external \\\n+\t'GitwebCache::Capture Perl API (in gitweb/lib/)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9504/test_capture_interface.pl\n+\n+test_done\ndiff --git a/t/t9504/test_capture_interface.pl b/t/t9504/test_capture_interface.pl\nnew file mode 100755\nindex 0000000..55c402a\n--- /dev/null\n+++ b/t/t9504/test_capture_interface.pl\n@@ -0,0 +1,76 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+use utf8;\n+\n+use Test::More;\n+\n+# test source version\n+use lib $ENV{GITWEBLIBDIR} || \"$ENV{GIT_BUILD_DIR}/gitweb/lib\";\n+\n+# ....................................................................\n+\n+# prototypes must be known at compile time, otherwise they do not work\n+BEGIN { use_ok('GitwebCache::Capture::SelectFH', qw(:all)); }\n+\n+# Test setting up capture\n+#\n+my $capture = new_ok('GitwebCache::Capture::SelectFH' => [], 'The $capture');\n+isa_ok($capture, 'GitwebCache::Capture', 'The $capture');\n+ok(setup_capture('GitwebCache::Capture::SelectFH'),\n+ 'setup_capture with package name: GitwebCache::Capture::SelectFH');\n+ok(setup_capture($capture),\n+ 'setup_capture with subclass object: $capture');\n+\n+# Test properties of capture_block\n+#\n+is(prototype('capture_block'), '&', 'capture_block has (&) prototype');\n+\n+# Test capturing\n+#\n+diag('Should not print anything except test results and diagnostic');\n+my $test_data = 'Capture this';\n+my $captured = capture_block {\n+\tprint $test_data;\n+};\n+is($captured, $test_data, 'capture_block captures simple data');\n+\n+binmode STDOUT, ':utf8';\n+$test_data = <<'EOF';\n+Áéí óú\n+ÄËÑÏÖ\n+Ábçdèfg\n+Zażółć gęsią jaźń\n+山田 太郎\n+ブレームのテストです。\n+\n+はれひほふ\n+\n+しているのが、いるので。\n+濱浜ほれぷりぽれまびぐりろへ。\n+EOF\n+utf8::decode($test_data);\n+$captured = capture_block {\n+\tbinmode select(), ':utf8';\n+\n+\tprint $test_data;\n+};\n+utf8::decode($captured);\n+is($captured, $test_data, 'capture_block captures utf8 data');\n+\n+$test_data = '|\\x{fe}\\x{ff}|\\x{9F}|\\000|'; # invalid utf-8\n+$captured = capture_block {\n+\tbinmode select(), ':raw';\n+\n+\tprint $test_data;\n+};\n+is($captured, $test_data, 'capture_block captures raw data');\n+\n+\n+done_testing();\n+\n+# Local Variables:\n+# encoding: utf-8\n+# End:\n-- \n1.7.3\n"},{"id":"152796","messageId":"1286402526-13143-8-git-send-email-jnareb@gmail.com","threadId":"25371","inReplyTo":"1286402526-13143-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv5 07/17] gitweb/lib - Cache captured output (using get/set)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-06T22:01:52Z","receivedAt":"2010-10-06T22:01:52Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add GitwebCache::CacheOutput package, which contains cache_output\nsubroutine, and (re)exports capture_stop from GitwebCache::Capture.\nThe cache_output gets data from cache and prints it, or captures\noutput of provided subroutine (code reference), saves it to cache and\nprints it. It currently uses Cache::Cache compatibile (get, set)\ninterface to cache. Capturing is currently (not configurable) done\nusing GitwebCache::Capture::SelectFH introduced in previous commit,\nbut any class derived from GitwebCache::Capture (like provided example\nGitwebCache::Capture::TiedCapture and GitwebCache::Capture::PerlIO)\nwould work.\n\nGitweb would use cache_output to get page from cache, or to generate\npage and save it to cache. The capture_stop would be used in\ndie_error subroutine, because error pages would not be cached.\n\nIt is assumed that data is saved to cache _converted_, and should\ntherefore be read from cache and printed to STDOUT in ':raw' (binary)\nmode.\n\n\nAdd t9505/test_cache_output.pl test, run as external test in\nt9505-gitweb-cache.sh. It checks that cache_output behaves correctly,\nnamely that it saves and restores action output in cache, and that it\nprints generated output or cached output.\n\nSigned-off-by: Jakub Narebski \n---\nDifferences from v4:\n* Similarly to other patches in this revision of gitweb caching\n series, the new test for this new module, written in Perl, is now\n invoked from a separate test script, to make it possible to run\n 'prove' on git test suite.\n\n Also new test uses GIT_BUILD_DIR rather than TEST_DIRECTORY/.., and\n it respects GITWEBLIBDIR to make it possible to test installed\n version of this module.\n\n* GitwebCache::CacheOutput no longer loads GitwebCache::SimpleFileCache;\n It is now assumed that cache_output() caller loads required packages\n itself, so that ->get() and ->set() methods $cache instance, passed\n as argument to cache_output(), work correctly.\n\n This required small change to t/t9505/test_cache_output.pl\n\n\nDifferences from relevant parts of J.H. patch:\n* cache_fetch() subroutine, now named cache_output(), is much, much\n simpler. Well, at this point in the patch series it lacks most of\n the features of original cache_fetch() by J.H.: it doesn't have\n adaptive cache lifetime, nor locking to prevent 'stampeding herd'\n problem, nor serving stale data when waiting for cache regeneration,\n nor background data generation, nor activity indicator... but the\n cache_output() itself doesn't change much in later commits, as those\n features are added mainly via methods and subroutines that\n cache_output() calls.\n\n* Capturing gitweb output would be done without need to extensively\n modify gitweb to either 1) save generated output into $output\n variable, i.e. replace 'print ' by '$output .= ', and print\n it or save in cache after it is generated in full (original J.H. patch\n in \"Gitweb caching v2\"), or 2) changing all print statements to print\n to explicit filehandle, i.e. replace 'print ' by \n 'print {$out} ', which filehandle points to STDOUT if caching is\n disabled and to in-memory file if caching is enabled (modified\n J.H. patch in \"Gitweb caching v5\").\n\n Currently capturing output is hardcoded to GitwebCache::Capture::SelectFH,\n but changing it to other compatibile capturing engine requires (in\n current version) change only only two lines in GitwebCache::CacheOutput\n module.\n\n* It is assumed that capture generates binary output (with I/O filters\n already applied), and that therefore we read from cache files in\n binary mode, and we print both captured output and data retrieved from\n cache in ':raw' mode (in binmode).\n\n* The key (human-readable unique id) for a page (for given capture) is\n passed explicitely to cache_output(). The cache_output() subroutine\n no longer depends of subroutines (like href()) and variables (like\n $cation or %actions) defined in gitweb.perl.\n\n This allows to simply use 'require ' instead of 'do $package_pm'\n (which also means that it can be installed as module in PERL5LIB etc.).\n\n* Added tests for cache_output() behavior.\n\n gitweb/lib/GitwebCache/CacheOutput.pm | 64 ++++++++++++++++++++++++++++++\n t/t9505-gitweb-cache.sh | 34 ++++++++++++++++\n t/t9505/test_cache_output.pl | 70 +++++++++++++++++++++++++++++++++\n 3 files changed, 168 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm\n create mode 100755 t/t9505-gitweb-cache.sh\n create mode 100755 t/t9505/test_cache_output.pl\n\ndiff --git a/gitweb/lib/GitwebCache/CacheOutput.pm b/gitweb/lib/GitwebCache/CacheOutput.pm\nnew file mode 100644\nindex 0000000..bba73ee\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/CacheOutput.pm\n@@ -0,0 +1,64 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+# (C) 2006, John 'Warthog9' Hawley \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Capturing and caching (gitweb) output\n+#\n+\n+# Capture output, save it in cache and print it, or retrieve it from\n+# cache and print it.\n+\n+package GitwebCache::CacheOutput;\n+\n+use strict;\n+use warnings;\n+\n+use GitwebCache::Capture::SelectFH qw(:all);\n+\n+use Exporter qw(import);\n+our @EXPORT = qw(cache_output capture_stop);\n+our %EXPORT_TAGS = (all => [ @EXPORT ]);\n+\n+# cache_output($cache, $key, $action_code);\n+#\n+# Attempts to get $key from $cache; if successful, prints the value.\n+# Otherwise, calls $action_code, capture its output and use\n+# the captured output as the new value for $key in $cache,\n+# then print captured output.\n+#\n+# It is assumed that captured data is already converted and it is\n+# in ':raw' format (and thus restored in ':raw' from cache)\n+our $CAPTURE_CLASS = 'GitwebCache::Capture::SelectFH';\n+\n+sub cache_output {\n+\tmy ($cache, $key, $code) = @_;\n+\n+\tmy $data = $cache->get($key);\n+\n+\t# capture and cache output, if there was nothing in the cache\n+\tif (!defined $data) {\n+\t\tmy $capture = $CAPTURE_CLASS;\n+\t\tsetup_capture($capture);\n+\n+\t\t# do not use 'capture_block' prototype\n+\t\t$data = &capture_block($code);\n+\t\t$cache->set($key, $data) if defined $data;\n+\t}\n+\n+\t# print cached data\n+\tif (defined $data) {\n+\t\t# select() instead of STDOUT is here for tests:\n+\t\tbinmode select(), ':raw';\n+\t\tprint $data;\n+\t}\n+\n+\treturn $data;\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::CacheOutput;\ndiff --git a/t/t9505-gitweb-cache.sh b/t/t9505-gitweb-cache.sh\nnew file mode 100755\nindex 0000000..55d3e17\n--- /dev/null\n+++ b/t/t9505-gitweb-cache.sh\n@@ -0,0 +1,34 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Jakub Narebski\n+#\n+\n+test_description='gitweb cache\n+\n+This test checks GitwebCache::CacheOutput Perl module that is\n+responsible for capturing and caching gitweb output.'\n+\n+# for now we are running only cache interface tests\n+. ./test-lib.sh\n+\n+# this test is present in gitweb-lib.sh\n+if ! test_have_prereq PERL; then\n+\tskip_all='perl not available, skipping test'\n+\ttest_done\n+fi\n+\n+\"$PERL_PATH\" -MTest::More -e 0 >/dev/null 2>&1 || {\n+\tskip_all='perl module Test::More unavailable, skipping test'\n+\ttest_done\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+# The external test will outputs its own plan\n+test_external_has_tap=1\n+\n+test_external \\\n+\t'GitwebCache::CacheOutput Perl API (in gitweb/lib/)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9505/test_cache_output.pl\n+\n+test_done\ndiff --git a/t/t9505/test_cache_output.pl b/t/t9505/test_cache_output.pl\nnew file mode 100755\nindex 0000000..0343591\n--- /dev/null\n+++ b/t/t9505/test_cache_output.pl\n@@ -0,0 +1,70 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use Test::More;\n+\n+# test source version\n+use lib $ENV{GITWEBLIBDIR} || \"$ENV{GIT_BUILD_DIR}/gitweb/lib\";\n+\n+# ....................................................................\n+\n+# prototypes must be known at compile time, otherwise they do not work\n+BEGIN { use_ok('GitwebCache::CacheOutput'); }\n+\n+# load GitwebCache::SimpleFileCache (gitweb.perl uses require too)\n+# GitwebCache::Capture::SelectFH is loaded by GitwebCache::CacheOutput\n+require_ok('GitwebCache::SimpleFileCache');\n+\n+# Test setting up $cache and $capture\n+my $cache = new_ok('GitwebCache::SimpleFileCache' => [], 'The $cache ');\n+my $capture = new_ok('GitwebCache::Capture::SelectFH' => [], 'The $capture');\n+\n+# ......................................................................\n+\n+# Prepare for testing cache_output\n+my $key = 'Key';\n+my $action_output = <<'EOF';\n+# This is data to be cached and shown\n+EOF\n+my $cached_output = <<\"EOF\";\n+$action_output# (version recovered from cache)\n+EOF\n+sub action {\n+\tprint $action_output;\n+}\n+\n+# Catch output printed by cache_fetch\n+# (only for 'print ' and 'printf ')\n+sub capture_output_of_cache_output {\n+\tmy $test_data = '';\n+\n+\topen my $test_data_fh, '>', \\$test_data;\n+\tmy $oldfh = select($test_data_fh);\n+\n+\tcache_output($cache, $key, \\&action);\n+\n+\tselect($oldfh);\n+\tclose $test_data_fh;\n+\n+\treturn $test_data;\n+}\n+\n+# clean state\n+$cache->set_expires_in(-1);\n+$cache->remove($key);\n+my $test_data;\n+\n+# first time (if there is no cache) generates cache entry\n+$test_data = capture_output_of_cache_output();\n+is($test_data, $action_output, 'action output is printed (generated)');\n+is($cache->get($key), $action_output, 'action output is saved in cache (generated)');\n+\n+# second time (if cache is set/valid) reads from cache\n+$cache->set($key, $cached_output);\n+$test_data = capture_output_of_cache_output();\n+is($test_data, $cached_output, 'action output is printed (from cache)');\n+\n+done_testing();\n-- \n1.7.3\n"},{"id":"152795","messageId":"1286402526-13143-9-git-send-email-jnareb@gmail.com","threadId":"25371","inReplyTo":"1286402526-13143-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv5 08/17] gitweb: Add optional output caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-06T22:01:53Z","receivedAt":"2010-10-06T22:01:53Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This commit actually adds output caching to gitweb, as we have now\nminimal features required for it in GitwebCache::SimpleFileCache (a\n'dumb' but fast file-based cache engine). To enable cache you need\n(at least) set $caching_enabled to true in gitweb config, and copy\ncaching modules alongside generated gitweb.cgi, e.g. by using 'make\ninstall-gitweb'. This is described in more detail in the new \"Gitweb\ncaching\" section in gitweb/README.\n\nCaching in theory can be done using any Perl module that implements\nCache::Cache compatibile get/set (method) interface. The default is\nto use GitwebCache::SimpleFileCache. Capturing and caching output is\ndone via cache_output subroutine from GitwebCache::CacheOutput.\n\nThe cache_output subroutine in GitwebCache::CacheOutput currently\ncaptures output using GitwebCache::Capture::SelectFH package, which in\nturn uses select(FILEHANDLE) to change default filehandle for output.\nThis means that a \"print\" or a \"printf\" (or a \"write\") in gitweb\nsource without a filehandle would be captured. To change mode of\nfilehandle used for capturing correctly, \"binmode STDOUT, \" had\nto be changed to \"binmode select(), \".\n\nCapturing and caching is designed in such way that there is no\nbehaviour change if $caching_enabled is false. If caching is not\nenabled, then capturing is also turned off.\n\nEnabling caching causes the following additional changes to gitweb\noutput:\n* Disables content-type negotiation (choosing between 'text/html'\n mimetype and 'application/xhtml+xml') when caching, as there is no\n content-type negotiation done when retrieving page from cache.\n Use lowest common denominator of 'text/html' mimetype which can\n be used by all browsers.\n* Disable optional timing info (how much time it took to generate the\n original page, and how many git commands it took), and in its place show\n unconditionally when page was originally generated (in GMT / UTC\n timezone).\n* Disable 'blame_incremental' view, as it doesn't make sense without\n printing data as soon as it is generated (which would require tee-ing\n when capturing output for caching)... and it doesn't work currently\n anyway.\n\nAdd basic tests of caching support to t9500-gitweb-standalone-no-errors\ntest: set $caching_enabled to true and check for errors for first time\nrun (generating cache) and second time run (retrieving from cache) for a\nsingle view - summary view for a project.\n\n\nTo be implemented (from original patch by J.H.):\n* adaptive cache expiration, based on average system load\n* optional locking interface, where only one process can update cache\n (using flock)\n* server-side progress indicator when waiting for filling cache,\n which in turn requires separating situations (like snapshots and\n other non-HTML responses) where we should not show 'please wait'\n message\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\nDifferences from v4:\n* Paths to modules to install gathered in GITWEB_MODULES variable in\n gitweb/Makefile do not include common 'lib/' prefix, following the\n change in \"gitweb: Prepare for splitting gitweb\"\n\n* Separate preparing cache for output caching into configure_caching()\n subroutine; cache_output() is now called from dispatch(), following\n update to gitweb.\n\n* Load needed module for cache (if $cache is set to name of module, it\n loads this module using 'eval \"require $cache\"'; note that if $cache\n is unset and $cache_enabled is true, then $cache is set to\n 'GitwebCache::SimpleFileCache' before this). \n\n This is connected with the fact that GitwebCache::CacheOutput no\n longer includes 'use GitwebCache::SimpleFileCache;'.\n\n* Do not use die_error when configuring cache, as the code might be\n called before $cgi is set (though I think it is not the case with\n current gitweb version).\n\n* Updated gitweb/README\n\n* Remove spurious changes\n\n\nDifferences from relevant parts of J.H. patch:\n* Capturing gitweb output is done without need to extensively modify\n gitweb; the only change related to capturing output is replacing \n 'binmode STDOUT, ' with 'binmode select(), ', required\n for currently used select($out) based capturing.\n\n* Instead of using \"binary\" (sic!) valued $cache_enable (which means 0 or 1\n valued $cache_enable), a set of two variables is used. The $cache\n variable can be used to select alternate caching engine / caching class.\n The $caching_enabled variable is used to actually enable/disable cache.\n\n* The information about the time when page was generated is shown only if\n 'timed' feature is enabled in gitweb config, and it is shown in place of\n usual time it took to generate page (shown when caching is not enabled).\n This means that change to gitweb/gitweb.css was not needed.\n\n* cache_output() is run only when $caching_enabled. Some of cache\n initializations, like creating cache instance, are in\n configure_cache() subroutine in gitweb.perl, and not at beginning of\n cache_output() (which was n.b. called cache_fetch() in J.H. patch).\n\n* Cache options are contained in %cache_options hash, instead of individual\n global variables (which were using non-Perlish camelCase notation).\n\n* There is information about caching in gitweb in gitweb/README\n\n* Some of new features, like incremental blame view, are turned off\n when caching is enabled.\n\n* There is simple test of output caching in gitweb in t9500, namely\n that gitweb runs without errors or warnings with caching enabled,\n both when saving and when restoring from cache.\n\n gitweb/Makefile | 6 ++\n gitweb/README | 57 +++++++++++++\n gitweb/gitweb.perl | 140 +++++++++++++++++++++++++++-----\n t/t9500-gitweb-standalone-no-errors.sh | 20 +++++\n 4 files changed, 201 insertions(+), 22 deletions(-)\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex c2d72e4..ec14cd1 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -112,6 +112,12 @@ endif\n \n GITWEB_FILES += static/git-logo.png static/git-favicon.png\n \n+# gitweb output caching\n+GITWEB_MODULES += GitwebCache/CacheOutput.pm\n+GITWEB_MODULES += GitwebCache/SimpleFileCache.pm\n+GITWEB_MODULES += GitwebCache/Capture.pm\n+GITWEB_MODULES += GitwebCache/Capture/SelectFH.pm\n+\n GITWEB_REPLACE = \\\n \t-e 's|++GIT_VERSION++|$(GIT_VERSION)|g' \\\n \t-e 's|++GIT_BINDIR++|$(bindir)|g' \\\ndiff --git a/gitweb/README b/gitweb/README\nindex d481198..5e3a0bc 100644\n--- a/gitweb/README\n+++ b/gitweb/README\n@@ -236,6 +236,11 @@ not include variables usually directly set during build):\n If server load exceed this value then return \"503 Service Unavailable\" error.\n Server load is taken to be 0 if gitweb cannot determine its value. Set it to\n undefined value to turn it off. The default is 300.\n+ * $caching_enabled\n+ If true, gitweb would use caching to speed up generating response.\n+ Currently supported is only output (response) caching. See \"Gitweb caching\"\n+ section below for details on how to configure and customize caching.\n+ The default is false (caching is disabled).\n \n \n Projects list file format\n@@ -308,6 +313,58 @@ You can use the following files in repository:\n descriptions.\n \n \n+Gitweb caching\n+~~~~~~~~~~~~~~\n+\n+Currently gitweb supports only output (HTTP response) caching, similar\n+to the one used on http://git.kernel.org. To turn it on, set \n+$caching_enabled variable to true value in gitweb config file, i.e.:\n+\n+ our $caching_enabled = 1;\n+\n+You can choose which caching engine should gitweb use by setting $cache\n+variable to _initialized_ instance of cache interface, e.g.:\n+\n+ use CHI;\n+ our $cache = CHI->new( driver => 'Memcached',\n+ \tservers => [ \"10.0.0.15:11211\", \"10.0.0.15:11212\" ],\n+ \tl1_cache => { driver => 'FastMmap', root_dir => '/var/cache/gitweb' }\n+ );\n+\n+Alternatively you can set $cache variable to the name of cache class,\n+e.g.:\n+\n+ our $cache = 'Cache::FileCache';\n+\n+In this case caching engine should support Cache::Cache or CHI names for\n+cache config (see below), and ignore unrecognized options. Such caching\n+engine should also implement (at least) ->get($key) and ->set($key, $data)\n+methods (Cache::Cache and CHI compatible interface).\n+\n+If $cache is left unset (if it is left undefined), then gitweb would use\n+GitwebCache::SimpleFileCache as caching engine. This engine is 'dumb' (but\n+fast) file based caching layer, currently without any support for cache size\n+limiting, or even removing expired / grossly expired entries. It has\n+therefore the downside of requiring a huge amount of disk space if there are\n+a number of repositories involved. It is not uncommon for git.kernel.org to\n+have on the order of 80G - 120G accumulate over the course of a few months.\n+It is therefore recommended that the cache directory be periodically\n+completely deleted; this operation is safe to perform. Suggested mechanism\n+(substitute $cachedir for actual path to gitweb cache):\n+\n+ # mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush\n+\n+Site-wide cache options are defined in %cache_options hash. Those options\n+apply only when $cache is unset (GitwebCache::SimpleFileCache is used), or\n+if $cache is name of cache class (e.g. $cache = 'Cache::FileCache'). You\n+can override cache options in gitweb config, e.g.:\n+\n+ $cache_options{'expires_in'} = 60; # 60 seconds = 1 minute\n+\n+Please read comments for %cache_options entries in gitweb/gitweb.perl for\n+description of available cache options.\n+\n+\n Webserver configuration\n -----------------------\n \ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex e4c08ba..f81a4a2 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -263,6 +263,44 @@ our %highlight_ext = (\n \tmap { $_ => 'xml' } qw(xhtml html htm),\n );\n \n+\n+# This enables/disables the caching layer in gitweb. Currently supported\n+# is only output (response) caching, similar to the one used on git.kernel.org.\n+our $caching_enabled = 0;\n+# Set to _initialized_ instance of cache interface implementing (at least)\n+# get($key) and set($key, $data) methods (Cache::Cache and CHI interfaces).\n+# If unset, GitwebCache::SimpleFileCache would be used, which is 'dumb'\n+# (but fast) file based caching layer, currently without any support for\n+# cache size limiting. It is therefore recommended that the cache directory\n+# be periodically completely deleted; this operation is safe to perform.\n+# Suggested mechanism:\n+# mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush\n+our $cache;\n+# You define site-wide cache options defaults here; override them with\n+# $GITWEB_CONFIG as necessary.\n+our %cache_options = (\n+\t# The location in the filesystem that will hold the root of the cache.\n+\t# This directory will be created as needed (if possible) on the first\n+\t# cache set. Note that either this directory must exists and web server\n+\t# has to have write permissions to it, or web server must be able to\n+\t# create this directory.\n+\t# Possible values:\n+\t# * 'cache' (relative to gitweb),\n+\t# * File::Spec->catdir(File::Spec->tmpdir(), 'gitweb-cache'),\n+\t# * '/var/cache/gitweb' (FHS compliant, requires being set up),\n+\t'cache_root' => 'cache',\n+\n+\t# The number of subdirectories deep to cache object item. This should be\n+\t# large enough that no cache directory has more than a few hundred\n+\t# objects. Each non-leaf directory contains up to 256 subdirectories\n+\t# (00-ff). Must be larger than 0.\n+\t'cache_depth' => 1,\n+\n+\t# The (global) expiration time for objects placed in the cache, in seconds.\n+\t'expires_in' => 20,\n+);\n+\n+\n # You define site-wide feature defaults here; override them with\n # $GITWEB_CONFIG as necessary.\n our %feature = (\n@@ -1055,7 +1093,15 @@ sub dispatch {\n \t !$project) {\n \t\tdie_error(400, \"Project needed\");\n \t}\n-\t$actions{$action}->();\n+\n+\tif ($caching_enabled) {\n+\t\t# human readable key identifying gitweb output\n+\t\tmy $output_key = href(-replay => 1, -full => 1, -path_info => 0);\n+\n+\t\tcache_output($cache, $output_key, $actions{$action});\n+\t} else {\n+\t\t$actions{$action}->();\n+\t}\n }\n \n sub reset_timer {\n@@ -1071,6 +1117,8 @@ sub run_request {\n \tevaluate_gitweb_config();\n \tevaluate_git_version();\n \tcheck_loadavg();\n+\tconfigure_caching()\n+\t\tif ($caching_enabled);\n \n \t# $projectroot and $projects_list might be set in gitweb config file\n \t$projects_list ||= $projectroot;\n@@ -1143,6 +1191,38 @@ sub run {\n \t1;\n }\n \n+sub configure_caching {\n+\tif (!eval { require GitwebCache::CacheOutput; 1; }) {\n+\t\t# cache is configured _before_ handling request, so $cgi is not defined,\n+\t\t# so we can't just \"die\" with sending error message to web browser\n+\t\t#die_error(500, \"Caching enabled and GitwebCache::CacheOutput not found\");\n+\n+\t\t# turn off caching and warn instead\n+\t\t$caching_enabled = 0;\n+\t\twarn \"Caching enabled and GitwebCache::CacheOutput not found\";\n+\t}\n+\tGitwebCache::CacheOutput->import();\n+\n+\t# $cache might be initialized (instantiated) cache, i.e. cache object,\n+\t# or it might be name of class, or it might be undefined\n+\tunless (defined $cache && ref($cache)) {\n+\t\t$cache ||= 'GitwebCache::SimpleFileCache';\n+\t\teval \"require $cache\";\n+\t\tdie $@ if $@;\n+\t\t$cache = $cache->new({\n+\t\t\t%cache_options,\n+\t\t\t#'cache_root' => '/tmp/cache',\n+\t\t\t#'cache_depth' => 2,\n+\t\t\t#'expires_in' => 20, # in seconds (CHI compatibile)\n+\t\t\t# (Cache::Cache compatibile initialization)\n+\t\t\t'default_expires_in' => $cache_options{'expires_in'},\n+\t\t\t# (CHI compatibile initialization)\n+\t\t\t'root_dir' => $cache_options{'cache_root'},\n+\t\t\t'depth' => $cache_options{'cache_depth'},\n+\t\t});\n+\t}\n+}\n+\n run();\n \n if (defined caller) {\n@@ -3405,7 +3485,9 @@ sub git_header_html {\n \t# 'application/xhtml+xml', otherwise send it as plain old 'text/html'.\n \t# we have to do this because MSIE sometimes globs '*/*', pretending to\n \t# support xhtml+xml but choking when it gets what it asked for.\n-\tif (defined $cgi->http('HTTP_ACCEPT') &&\n+\t# Disable content-type negotiation when caching (use mimetype good for all).\n+\tif (!$caching_enabled &&\n+\t defined $cgi->http('HTTP_ACCEPT') &&\n \t $cgi->http('HTTP_ACCEPT') =~ m/(,|;|\\s|^)application\\/xhtml\\+xml(,|;|\\s|$)/ &&\n \t $cgi->Accept('application/xhtml+xml') != 0) {\n \t\t$content_type = 'application/xhtml+xml';\n@@ -3579,17 +3661,25 @@ sub git_footer_html {\n \t}\n \tprint \"\\n\"; # class=\"page_footer\"\n \n-\tif (defined $t0 && gitweb_check_feature('timed')) {\n+\t# timing info doesn't make much sense with output (response) caching,\n+\t# so when caching is enabled gitweb prints the time of page generation\n+\tif ((defined $t0 || $caching_enabled) &&\n+\t gitweb_check_feature('timed')) {\n \t\tprint \"
\\n\";\n-\t\tprint 'This page took '.\n-\t\t ''.\n-\t\t Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).\n-\t\t ' seconds '.\n-\t\t ' and '.\n-\t\t ''.\n-\t\t $number_of_git_cmds.\n-\t\t ' git commands '.\n-\t\t \" to generate.\\n\";\n+\t\tif ($caching_enabled) {\n+\t\t\tprint 'This page was generated at '.\n+\t\t\t gmtime( time() ).\" GMT\\n\";\n+\t\t} else {\n+\t\t\tprint 'This page took '.\n+\t\t\t ''.\n+\t\t\t Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).\n+\t\t\t ' seconds '.\n+\t\t\t ' and '.\n+\t\t\t ''.\n+\t\t\t $number_of_git_cmds.\n+\t\t\t ' git commands '.\n+\t\t\t \" to generate.\\n\";\n+\t\t}\n \t\tprint \"
\\n\"; # class=\"page_footer\"\n \t}\n \n@@ -3598,8 +3688,8 @@ sub git_footer_html {\n \t}\n \n \tprint qq!\\n!;\n-\tif (defined $action &&\n-\t $action eq 'blame_incremental') {\n+\tif (!$caching_enabled &&\n+\t defined $action && $action eq 'blame_incremental') {\n \t\tprint qq!