{"thread":{"id":"25978","subject":"[PATCH 06/24] gitweb/lib - Simple output capture by redirecting STDOUT","startedAt":"2010-12-06T23:10:45Z","lastAt":"2010-12-06T23:11:09Z","messageCount":25,"participants":["Jakub Narebski"],"isPatch":true,"patchVersion":1,"patchTotal":24},"messages":[{"id":"157460","messageId":"1291677069-6559-1-git-send-email-jnareb@gmail.com","threadId":"25978","inReplyTo":null,"subject":"[PATCHv6/RFC 00/24] gitweb: Simple file based output caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-06T23:10:45Z","receivedAt":"2010-12-06T23:10:45Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"[I Cc-ed everybody who *might* be interested in this series. I am\n sorry if it included somebody by mistake]\n\nThis 22+ patches long series (2 last patches are proof of concept) is\nintended as replacement (rewrite) of \"Gitweb caching v7\" series from\nJohn 'Warthog9' Hawley (J.H.):\n\n http://thread.gmane.org/gmane.comp.version-control.git/160147\n\nThis is sixth version (6th release) of this series, and is available\nin the following repositories (links are to web interface):\n\n http://repo.or.cz/w/git/jnareb-git.git\n http://github.com/jnareb/git\n\nas 'gitweb/cache-kernel-v6' branch. Earlier versions are available at\nhttp://repo.or.cz/w/git/jnareb-git.git as 'gitweb/cache-kernel-v5'\n(previous version) to 'gitweb/cache-kernel' (first version).\n\nPrevious version of this series was sent to git mailing list as:\n\n [PATCHv5 00/17] gitweb: Simple file based output caching\n Message-Id: <1286402526-13143-1-git-send-email-jnareb@gmail.com>\n http://thread.gmane.org/gmane.comp.version-control.git/158313\n\nYou can find link to next to previous version, et cetera.\n\n\nThe main ideas in lifted from J.H. patches are the following\n(features in common with \"Gitweb caching v7\" series by John Hawley):\n\n* caching captured output of gitweb in flat files, without any\n serialization (caching raw data)\n\n* using global (per-cache, not per-entry) expiration time, and\n using difference between mtime of cache file and current time\n for expiration\n\n* using file locking (flock) to prevent 'cache miss stampede'\n problem, i.e. to ensure that only one process is (re)generating\n cache entry\n\n* serving stale but not too old version, and regenerating data\n in background, to avoid waiting for data to be regenerated\n\n* progress info indicator based on http-equiv refresh trick\n (described in more detail how it works in the commit message)\n\n* capturing gitweb output by redirecting STDOUT to cache entry file\n\n\nThe main differences between this patch series and \"Gitweb caching v7\"\n(and my minimal fixups in \"Gitweb caching v7.[1-3]\") are the following:\n\n* features are added piece by piece in multiple patches (22 patches\n covering v7 features vs 3-4 patches in v7/v7.x series), making it\n hopefully easier to review, as patches are smaller. OTOH this series\n is much longer...\n\n* In J.H. series subroutines responsible for capturing gitweb output are\n in gitweb.perl, and subroutines responsible for caching are in lib/cache.pl\n (cache.pm in original patch). cache.pl/cache.pm uses variables and\n subroutines from gitweb script, so it couldn't be made into Perl module;\n therefore we have to use 'do' rather than 'require' to load it.\n\n In this series GitwebCache::Capture::Simple module is responsible for\n capturing [gitweb] output, GitwebCache::SimpleFileCache and\n GitwebCache::FileCacheWithLocking are responsible for caching, and\n GitwebCache::CacheOutput is about caching captured output (ties them\n together). This allowed \"unit\" testing, i.e. testing each module\n in isolation (tests t9503 - t9505).\n\n* GitwebCache::CacheOutput::cache_output (equivalent of cache_fetch from\n cache.pm in J.H. patch) supports any cache supporting ->get / ->set or\n ->compute interface (e.g. Cache::FileCache from Cache::Cache, or CHI\n with 'File' driver, or Cache::FastMmap) - it is described in gitweb/README\n in \"Gitweb caching\" section.\n\n For this capturing engine (GitwebCache::Capture::Simple) supports returning\n captured output (via capturing to in-memory file).\n\n Tested once upon a time with Cache::FileCache $cache.\n\n* There is no difference between treating actions with binary output or\n possibly binary output like 'snapshot' or 'blob_plain' (which use binary\n or ':raw' mode) and other actions (which use text or ':utf8' mode).\n GitwebCache::Capture::Simple captures transformed output i.e. raw bytes,\n so data from cache is dumped to STDOUT (to web browser) in ':raw' mode.\n\n* Instead of disabling caching of 'blame_incremental' action (so it is\n used without caching), this alternate to plain 'blame' action is\n disabled if caching is turned off.\n\n In the future 'blame_interactive' would use cache for caching its\n initial output and for caching 'blame_data' it uses.\n\n* Configuring cache is done via %cache_options (and %generating_options)\n instead of via gitweb config variables. For example instead of \n $minCacheTime there is $cache_options{'expires_min'}.\n\n It is also more configurable than in J.H. patch; more parameters can be\n changed (like e.g. factor multiplying get_loadavg() in adaptive cache\n lifetime; 'check_load', 'generating_info', 'on_error' are configurable\n callbacks).\n\n \"gitweb: Support legacy options used by kernel.org caching engine\"\n patch in this series makes this rewrite support configuration variables\n used by \"Gitweb caching v7\" series.\n\n* This rewrite uses lexical filehandles, i.e.\n\n open my $fh, '>', $filename\n\n instead of globals that J.H. patch uses\n\n open FH, '>', $filename\n\n (though it hides it in \"open(cacheFile, '<', $filename)\"). J.H. is\n working on \"Gitweb caching v8\" and I think he would address that issue\n there.\n\n* When generating cache in background process, the background process\n daemonizes itself. Therefore it should be safe to enable / use\n 'background_cache' also for persistent environments, like mod_perl via\n ModPerl::Registry, FastCGI when run as gitweb.fcgi, PSGI via gitweb.psgi\n wrapper that git-instaweb generates.\n\n- Other changes might be mentioned in comments to individual patches\n\nTwo last patches in this series introduce proof of concept cache\nadministration page, where you can currently check how much file space is\nused by cache, and where you can also safely clean cache (remove all\nentries). Those two patches are slightly outside scope of \"gitweb output\ncaching\", and that is why I refer to this series as 22+ patches long\n(there are 24 patches in total).\n\nPrevious version of this series had\n gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)\n gitweb/lib - Alternate ways of capturing output\nas two last patche in the series. They are missing in this release.\n\n\nThe following changes since commit 0b0cd0e0a29a139f418991dd769ea4266ffec370:\n\n Merge branch 'jn/ignore-doc' (2010-12-03 16:13:06 -0800)\n\nare available in the git repository at:\n\n git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v6\n\nJakub Narebski (24):\n t/test-lib.sh: Export also GIT_BUILD_DIR in test_external\n gitweb: Prepare for splitting gitweb\n gitweb/lib - Very simple file based cache\n gitweb/lib - Stat-based cache expiration\n gitweb/lib - Regenerate entry if the cache file has size of 0\n gitweb/lib - Simple output capture by redirecting STDOUT\n gitweb/lib - Cache captured output (using get/set)\n gitweb: Add optional output caching\n gitweb/lib - Adaptive cache expiration time\n gitweb/lib - Use CHI compatibile (compute method) caching interface\n gitweb/lib - capture output directly to cache entry file\n gitweb/lib - Use locking to avoid 'cache miss stampede' problem\n gitweb/lib - No need for File::Temp when locking\n gitweb/lib - Serve stale data when waiting for filling cache\n gitweb/lib - Regenerate (refresh) cache in background\n gitweb: Introduce %actions_info, gathering information about actions\n gitweb: Show appropriate \"Generating...\" page when regenerating cache\n gitweb/lib - Configure running 'generating_info' when generating data\n gitweb: Add startup delay to activity indicator for cache\n gitweb/lib - Add support for setting error handler in cache\n gitweb: Wrap die_error to use as error handler for caching engine\n gitweb: Support legacy options used by kernel.org caching engine\n gitweb/lib - Add clear() and size() methods to caching interface\n gitweb: Add beginnings of cache administration page (proof of\n concept)\n\n gitweb/Makefile | 23 +-\n gitweb/README | 62 +++\n gitweb/gitweb.perl | 544 +++++++++++++++++++-\n gitweb/lib/GitwebCache/CacheOutput.pm | 131 +++++\n gitweb/lib/GitwebCache/Capture/Simple.pm | 110 ++++\n gitweb/lib/GitwebCache/FileCacheWithLocking.pm | 376 ++++++++++++++\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 592 ++++++++++++++++++++++\n t/gitweb-lib.sh | 12 +\n t/t9500-gitweb-standalone-no-errors.sh | 20 +\n t/t9501-gitweb-standalone-http-status.sh | 21 +\n t/t9502-gitweb-standalone-parse-output.sh | 33 ++\n t/t9503-gitweb-caching-interface.sh | 34 ++\n t/t9503/test_cache_interface.pl | 647 ++++++++++++++++++++++++\n t/t9504-gitweb-capture-interface.sh | 34 ++\n t/t9504/test_capture_interface.pl | 108 ++++\n t/t9505-gitweb-cache.sh | 39 ++\n t/t9505/test_cache_output.pl | 86 ++++\n t/test-lib.sh | 4 +-\n 18 files changed, 2850 insertions(+), 26 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm\n create mode 100644 gitweb/lib/GitwebCache/Capture/Simple.pm\n create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm\n create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm\n create mode 100755 t/t9503-gitweb-caching-interface.sh\n create mode 100755 t/t9503/test_cache_interface.pl\n create mode 100755 t/t9504-gitweb-capture-interface.sh\n create mode 100755 t/t9504/test_capture_interface.pl\n create mode 100755 t/t9505-gitweb-cache.sh\n create mode 100755 t/t9505/test_cache_output.pl\n\n-- \n1.7.3\n"},{"id":"157463","messageId":"1291677069-6559-2-git-send-email-jnareb@gmail.com","threadId":"25978","inReplyTo":"1291677069-6559-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 01/24] t/test-lib.sh: Export also GIT_BUILD_DIR in test_external","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-06T23:10:46Z","receivedAt":"2010-12-06T23:10:46Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This way we can use it in test scripts written in other languages\n(e.g. in Perl), and not rely on \"$TEST_DIRECTORY/..\"\n\nSigned-off-by: Jakub Narebski \n---\nThis is not strictly required, but if we modified t/gitweb-lib.sh to\nuse GIT_BUILD_DIR, we would want to do the same in the test scripts\nfor caching interface etc. included in this series.\n\nThis patch is the same as in previous version of this series. It has\nno equivalent in \"Gitweb caching v7\" by J.H.; it was not needed there\nbecause that series doesn't have separate \"unit\" tests for individual\ncomponents of gitweb output caching.\n\n t/test-lib.sh | 4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 48fa516..c077fa4 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -552,9 +552,9 @@ test_external () {\n \t\t# Announce the script to reduce confusion about the\n \t\t# test output that follows.\n \t\tsay_color \"\" \"# run $test_count: $descr ($*)\"\n-\t\t# Export TEST_DIRECTORY, TRASH_DIRECTORY and GIT_TEST_LONG\n+\t\t# Export TEST_DIRECTORY, GIT_BUILD_DIR, TRASH_DIRECTORY and GIT_TEST_LONG\n \t\t# to be able to use them in script\n-\t\texport TEST_DIRECTORY TRASH_DIRECTORY GIT_TEST_LONG\n+\t\texport TEST_DIRECTORY GIT_BUILD_DIR TRASH_DIRECTORY GIT_TEST_LONG\n \t\t# Run command; redirect its stderr to &4 as in\n \t\t# test_run_, but keep its stdout on our stdout even in\n \t\t# non-verbose mode.\n-- \n1.7.3\n"},{"id":"157457","messageId":"1291677069-6559-3-git-send-email-jnareb@gmail.com","threadId":"25978","inReplyTo":"1291677069-6559-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 02/24] gitweb: Prepare for splitting gitweb","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-06T23:10:47Z","receivedAt":"2010-12-06T23:10:47Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Prepare gitweb for having been split into modules that are to be\ninstalled alongside gitweb in 'lib/' subdirectory, by adding\n\n use lib __DIR__.'/lib';\n\nto gitweb.perl (to main gitweb script), and preparing for putting\nmodules (relative path) in $(GITWEB_MODULES) in gitweb/Makefile.\n\nThis preparatory work allows to add new module to gitweb by simply\nadding\n\n GITWEB_MODULES += \n\nto gitweb/Makefile (assuming that the module is in 'gitweb/lib/'\ndirectory).\n\nWhile at it pass GITWEBLIBDIR in addition to GITWEB_TEST_INSTALLED to\nallow testing installed version of gitweb and installed version of\nmodules (for future tests which would check individual (sub)modules).\n\n\nUsing __DIR__ from Dir::Self module (not in core, that's why currently\ngitweb includes excerpt of code from Dir::Self defining __DIR__) was\nchosen over using FindBin-based solution (in core since perl 5.00307,\nwhile gitweb itself requires at least perl 5.8.0) because FindBin uses\nBEGIN block, which is a problem under mod_perl and other persistent\nPerl environments (thought there are workarounds).\n\nAt Pavan Kumar Sankara suggestion gitweb/Makefile uses\n\n install [OPTION]... SOURCE... DIRECTORY\n\nformat (2nd format) with single SOURCE rather than\n\n install [OPTION]... SOURCE DEST\n\nformat (1st format) because of security reasons (race conditions).\nModern GNU install has `-T' / `--no-target-directory' option, but we\ncannot rely that the $(INSTALL) we are using supports this option.\n\nThe install-modules target in gitweb/Makefile uses shell 'for' loop,\ninstead of make's $(foreach) function, to avoid possible problem with\ngenerating a command line that exceeded the maximum argument list\nlength.\n\nSigned-off-by: Jakub Narebski \n---\nIn previous version there was a BUG: shell variables should not be in\nsingle quotes.\n\nThis is fixed version, the same as in\n\n [PATCHv7.4 0/4] Gitweb caching v7.4\n Message-ID: <1291404335-25541-1-git-send-email-jnareb@gmail.com>\n http://thread.gmane.org/gmane.comp.version-control.git/162830\n\nseries, which are a bit fixed up and a tiny bit cleaned up version of\n\"Gitweb caching v7\" series from John 'Warthog9' Hawley:\n\n http://thread.gmane.org/gmane.comp.version-control.git/160147\n\n\nSidenote: this patch can serve as a way to introduce infrastructure\nrequired for splitting gitweb into modules for better maintability.\nThis issue was to be first part (first milestone) for \"Integrated web\nclient for git\" GSoC2010 project by Pavan Kumar Sankara (with Christian\nCouder as mentor, and Petr 'Pasky' Baudis as co-mentor)... which\nunfortunately failed middterm evaluations.\n\nAs you can see later in series it makes it very easy to add new gitweb\nmodules which are to be installed. Just add\n\n GITWEB_MODULES += Module/Foo.pm\n\nto gitweb/Makefile for Module::Foo (present in gitweb/lib/Module/Foo.pm)\nand 'install-gitweb' target would automatically install it along with\ngitweb.\n\n\n\n gitweb/Makefile | 17 +++++++++++++++--\n gitweb/gitweb.perl | 8 ++++++++\n 2 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex 0a6ac00..e6029e1 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -57,6 +57,7 @@ PERL_PATH ?= /usr/bin/perl\n bindir_SQ = $(subst ','\\'',$(bindir))#'\n gitwebdir_SQ = $(subst ','\\'',$(gitwebdir))#'\n gitwebstaticdir_SQ = $(subst ','\\'',$(gitwebdir)/static)#'\n+gitweblibdir_SQ = $(subst ','\\'',$(gitwebdir)/lib)#'\n SHELL_PATH_SQ = $(subst ','\\'',$(SHELL_PATH))#'\n PERL_PATH_SQ = $(subst ','\\'',$(PERL_PATH))#'\n DESTDIR_SQ = $(subst ','\\'',$(DESTDIR))#'\n@@ -153,20 +154,32 @@ test:\n \n test-installed:\n \tGITWEB_TEST_INSTALLED='$(DESTDIR_SQ)$(gitwebdir_SQ)' \\\n+\tGITWEBLIBDIR='$(DESTDIR_SQ)$(gitweblibdir_SQ)' \\\n \t\t$(MAKE) -C ../t gitweb-test\n \n ### Installation rules\n \n-install: all\n+install: all install-modules\n \t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebdir_SQ)'\n \t$(INSTALL) -m 755 $(GITWEB_PROGRAMS) '$(DESTDIR_SQ)$(gitwebdir_SQ)'\n \t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n \t$(INSTALL) -m 644 $(GITWEB_FILES) '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n \n+install-modules:\n+\tinstall_dirs=\"$(sort $(dir $(GITWEB_MODULES)))\" && \\\n+\tfor dir in $$install_dirs; do \\\n+\t\ttest -d '$(DESTDIR_SQ)$(gitweblibdir_SQ)'/\"$$dir\" || \\\n+\t\t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitweblibdir_SQ)'/\"$$dir\"; \\\n+\tdone\n+\tgitweb_modules=\"$(GITWEB_MODULES)\" && \\\n+\tfor mod in $$gitweb_modules; do \\\n+\t\t$(INSTALL) -m 644 \"lib/$$mod\" '$(DESTDIR_SQ)$(gitweblibdir_SQ)'/\"$$(dirname $$mod)\"; \\\n+\tdone\n+\n ### Cleaning rules\n \n clean:\n \t$(RM) gitweb.cgi static/gitweb.min.js static/gitweb.min.css GITWEB-BUILD-OPTIONS\n \n-.PHONY: all clean install test test-installed .FORCE-GIT-VERSION-FILE FORCE\n+.PHONY: all clean install install-modules test test-installed .FORCE-GIT-VERSION-FILE FORCE\n \ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 679f2da..cfa511c 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -10,6 +10,14 @@\n use 5.008;\n use strict;\n use warnings;\n+\n+use File::Spec;\n+# __DIR__ is taken from Dir::Self __DIR__ fragment\n+sub __DIR__ () {\n+\tFile::Spec->rel2abs(join '', (File::Spec->splitpath(__FILE__))[0, 1]);\n+}\n+use lib __DIR__ . '/lib';\n+\n use CGI qw(:standard :escapeHTML -nosticky);\n use CGI::Util qw(unescape);\n use CGI::Carp qw(fatalsToBrowser set_message);\n-- \n1.7.3\n"},{"id":"157461","messageId":"1291677069-6559-4-git-send-email-jnareb@gmail.com","threadId":"25978","inReplyTo":"1291677069-6559-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 03/24] gitweb/lib - Very simple file based cache","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-06T23:10:48Z","receivedAt":"2010-12-06T23:10:48Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This is first step towards implementing file based output (response)\ncaching layer that is used on such large sites as kernel.org.\n\nThis patch introduces GitwebCaching::SimpleFileCache package, which\nfollows Cache::Cache / CHI interface, although do not implement it\nfully. The intent of following established convention for cache\ninterface is to be able to replace our simple file based cache,\ne.g. by the one using memcached.\n\nThe data is stored in the cache as-is, without adding metadata (like\nexpiration date), and without serialization (which means that one can\nstore only scalar data). At this point there is no support for\nexpiring cache entries.\n\n\nThe code of GitwebCaching::SimpleFileCache package in gitweb/lib\nwas heavily based on file-based cache in Cache::Cache package, i.e.\non Cache::FileCache, Cache::FileBackend and Cache::BaseCache, and on\nfile-based cache in CHI, i.e. on CHI::Driver::File and CHI::Driver\n(including implementing atomic write, something that original patch\nlacks). It tries to follow more modern CHI architecture, but without\nrequiring Moose. It is much simplified compared to both interfaces\nand their file-based drivers.\n\nThis patch does not yet enable output caching in gitweb (it doesn't\nhave all required features yet); on the other hand it includes tests\nof cache Perl API in t/t9503-gitweb-caching-interface.sh.\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\nThis version is cleaned up from remainders of old-style code inspired\nby code in Cach::Cache distribution, and of cargo-culted code from\nexamples in Perl documentation; it uses more modern Perl coding\nconventions. Therefore it has significantly less warnings from\nPerl::Critic (though they are not eliminated entirely).\n\nContrary to some earlier versions we do not try to be operating system\nagnostic by using File::Spec; concatenating paths with '/' taken as\ndirectory separator is faster, and that is what gitweb uses anyway.\n\nNote that writing to cache entry file is done atomically via File::Temp\nand renaming file to final version. This way we can ensure that gitweb\nwouldn't get halfway written file. Use of File::Temp (which gives\nsafety but unfortunately has negative impact on performance) will be\nmade not necessary with introduction of locking.\n\nBecause GitwebCache::SimpleFileCache (and its derivatives) use standard\n->new, ->get, ->set, ->compute methods / API / interface, then other\ncaching engines such as Cache::Cache, Cache, CHI, Cache::Memcached or\nCache::FastMmap can be used instead of this one.\n\nFast spew / fast slurp might be interesting, but they are obsoleted\n(unused) when redirecting output directly to cache entry file in later\npatch.\n\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 336 +++++++++++++++++++++++++++++\n t/t9503-gitweb-caching-interface.sh | 34 +++\n t/t9503/test_cache_interface.pl | 85 ++++++++\n 3 files changed, 455 insertions(+), 0 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm\n create mode 100755 t/t9503-gitweb-caching-interface.sh\n create mode 100755 t/t9503/test_cache_interface.pl\n\ndiff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm\nnew file mode 100644\nindex 0000000..2f26a6c\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm\n@@ -0,0 +1,336 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2006, John 'Warthog9' Hawley \n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Gitweb caching engine, simple file-based cache\n+#\n+\n+# Minimalistic cache that stores data in the filesystem, without serialization\n+# and currently without any kind of cache expiration (all keys last forever till\n+# they got explicitely removed).\n+#\n+# It follows Cache::Cache and CHI interfaces (but does not implement it fully)\n+\n+package GitwebCache::SimpleFileCache;\n+\n+use strict;\n+use warnings;\n+\n+use File::Path qw(mkpath);\n+use File::Temp qw(tempfile);\n+use Digest::MD5 qw(md5_hex);\n+\n+# by default, the cache nests all entries on the filesystem single\n+# directory deep, i.e. '60/b725f10c9c85c70d97880dfe8191b3' for\n+# key name (key digest) 60b725f10c9c85c70d97880dfe8191b3.\n+#\n+our $DEFAULT_CACHE_DEPTH = 1;\n+\n+# by default, the root of the cache is located in 'cache'.\n+#\n+our $DEFAULT_CACHE_ROOT = \"cache\";\n+\n+# by default we don't use cache namespace (empty namespace);\n+# empty namespace does not allow for simple implementation of clear() method.\n+#\n+our $DEFAULT_NAMESPACE = '';\n+\n+# ......................................................................\n+# constructor\n+\n+# The options are set by passing in a reference to a hash containing\n+# any of the following keys:\n+# * 'namespace'\n+# The namespace associated with this cache. This allows easy separation of\n+# multiple, distinct caches without worrying about key collision. Defaults\n+# to $DEFAULT_NAMESPACE.\n+# * 'cache_root' (Cache::FileCache compatibile),\n+# 'root_dir' (CHI::Driver::File compatibile),\n+# The location in the filesystem that will hold the root of the cache.\n+# Defaults to $DEFAULT_CACHE_ROOT.\n+# * 'cache_depth' (Cache::FileCache compatibile),\n+# 'depth' (CHI::Driver::File compatibile),\n+# The number of subdirectories deep to cache object item. This should be\n+# large enough that no cache directory has more than a few hundred objects.\n+# Defaults to $DEFAULT_CACHE_DEPTH unless explicitly set.\n+sub new {\n+\tmy $class = shift;\n+\tmy %opts = ref $_[0] ? %{ $_[0] } : @_;\n+\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\tmy ($root, $depth, $ns);\n+\tif (%opts) {\n+\t\t$root =\n+\t\t\t$opts{'cache_root'} ||\n+\t\t\t$opts{'root_dir'};\n+\t\t$depth =\n+\t\t\t$opts{'cache_depth'} ||\n+\t\t\t$opts{'depth'};\n+\t\t$ns = $opts{'namespace'};\n+\t}\n+\t$root = $DEFAULT_CACHE_ROOT unless defined($root);\n+\t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n+\t$ns = $DEFAULT_NAMESPACE unless defined($ns);\n+\n+\t$self->set_root($root);\n+\t$self->set_depth($depth);\n+\t$self->set_namespace($ns);\n+\n+\treturn $self;\n+}\n+\n+\n+# ......................................................................\n+# accessors\n+\n+# http://perldesignpatterns.com/perldesignpatterns.html#AccessorPattern\n+\n+# creates get_depth() and set_depth($depth) etc. methods\n+foreach my $i (qw(depth root namespace)) {\n+\tmy $field = $i;\n+\tno strict 'refs';\n+\t*{\"get_$field\"} = sub {\n+\t\tmy $self = shift;\n+\t\treturn $self->{$field};\n+\t};\n+\t*{\"set_$field\"} = sub {\n+\t\tmy ($self, $value) = @_;\n+\t\t$self->{$field} = $value;\n+\t};\n+}\n+\n+\n+# ----------------------------------------------------------------------\n+# utility functions and methods\n+\n+# Return root dir for namespace (lazily built, cached)\n+sub path_to_namespace {\n+\tmy ($self) = @_;\n+\n+\tif (!exists $self->{'path_to_namespace'}) {\n+\t\tif (defined $self->{'namespace'} &&\n+\t\t $self->{'namespace'} ne '') {\n+\t\t\t$self->{'path_to_namespace'} = \"$self->{'root'}/$self->{'namespace'}\";\n+\t\t} else {\n+\t\t\t$self->{'path_to_namespace'} = $self->{'root'};\n+\t\t}\n+\t}\n+\treturn $self->{'path_to_namespace'};\n+}\n+\n+# $path = $cache->path_to_key($key);\n+# $path = $cache->path_to_key($key, \\$dir);\n+#\n+# Take an human readable key, and return file path.\n+# Puts dirname of file path in second argument, if it is provided.\n+sub path_to_key {\n+\tmy ($self, $key, $dir_ref) = @_;\n+\n+\tmy @paths = ( $self->path_to_namespace() );\n+\n+\t# Create a unique (hashed) key from human readable key\n+\tmy $filename = md5_hex($key); # or $digester->add($key)->hexdigest();\n+\n+\t# Split filename so that it have DEPTH subdirectories,\n+\t# where each subdirectory has a two-letter name\n+\tpush @paths, unpack(\"(a2)[$self->{'depth'}] a*\", $filename);\n+\t$filename = pop @paths;\n+\n+\t# Join paths together, computing dir separately if $dir_ref was passed.\n+\tmy $filepath;\n+\tif (defined $dir_ref && ref($dir_ref)) {\n+\t\tmy $dir = join('/', @paths);\n+\t\t$filepath = \"$dir/$filename\";\n+\t\t$$dir_ref = $dir;\n+\t} else {\n+\t\t$filepath = join('/', @paths, $filename);\n+\t}\n+\n+\treturn $filepath;\n+}\n+\n+sub read_file {\n+\tmy $filename = shift;\n+\n+\t# Fast slurp, adapted from File::Slurp::read, with unnecessary options removed\n+\t# via CHI::Driver::File (from CHI-0.33)\n+\tmy $buf = '';\n+\topen my $read_fh, '<', $filename\n+\t\tor return;\n+\tbinmode $read_fh, ':raw';\n+\n+\tmy $size_left = -s $read_fh;\n+\n+\twhile ($size_left > 0) {\n+\t\tmy $read_cnt = sysread($read_fh, $buf, $size_left, length($buf));\n+\t\treturn unless defined $read_cnt;\n+\n+\t\tlast if $read_cnt == 0;\n+\t\t$size_left -= $read_cnt;\n+\t\t#last if $size_left <= 0;\n+\t}\n+\n+\tclose $read_fh\n+\t\tor die \"Couldn't close file '$filename' opened for reading: $!\";\n+\treturn $buf;\n+}\n+\n+sub write_fh {\n+\tmy ($write_fh, $filename, $data) = @_;\n+\n+\t# Fast spew, adapted from File::Slurp::write, with unnecessary options removed\n+\t# via CHI::Driver::File (from CHI-0.33)\n+\tbinmode $write_fh, ':raw';\n+\n+\tmy $size_left = length($data);\n+\tmy $offset = 0;\n+\n+\twhile ($size_left > 0) {\n+\t\tmy $write_cnt = syswrite($write_fh, $data, $size_left, $offset);\n+\t\treturn unless defined $write_cnt;\n+\n+\t\t$size_left -= $write_cnt;\n+\t\t$offset += $write_cnt; # == length($data);\n+\t}\n+\n+\tclose $write_fh\n+\t\tor die \"Couldn't close file '$filename' opened for writing: $!\";\n+}\n+\n+# ----------------------------------------------------------------------\n+# \"private\" utility functions and methods\n+\n+# take a file path to cache entry, and its directory\n+# return filehandle and filename of open temporary file,\n+# like File::Temp::tempfile\n+sub _tempfile_to_path {\n+\tmy ($file, $dir) = @_;\n+\n+\t# tempfile will croak() if there is an error\n+\treturn tempfile(\"${file}_XXXXX\",\n+\t\t#DIR => $dir,\n+\t\t'UNLINK' => 0, # ensure that we don't unlink on close; file is renamed\n+\t\t'SUFFIX' => '.tmp');\n+}\n+\n+\n+# ----------------------------------------------------------------------\n+# worker methods\n+\n+sub fetch {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $file = $self->path_to_key($key);\n+\treturn unless (defined $file && -f $file);\n+\n+\treturn read_file($file);\n+}\n+\n+sub store {\n+\tmy ($self, $key, $data) = @_;\n+\n+\tmy $dir;\n+\tmy $file = $self->path_to_key($key, \\$dir);\n+\treturn unless (defined $file && defined $dir);\n+\n+\t# ensure that directory leading to cache file exists\n+\tif (!-d $dir) {\n+\t\t# mkpath will croak()/die() if there is an error\n+\t\tmkpath($dir, 0, 0777);\n+\t}\n+\n+\t# generate a temporary file\n+\tmy ($temp_fh, $tempname) = _tempfile_to_path($file, $dir);\n+\tchmod 0666, $tempname\n+\t\tor warn \"Couldn't change permissions to 0666 / -rw-rw-rw- for '$tempname': $!\";\n+\n+\twrite_fh($temp_fh, $tempname, $data);\n+\n+\trename($tempname, $file)\n+\t\tor die \"Couldn't rename temporary file '$tempname' to '$file': $!\";\n+}\n+\n+# get size of an element associated with the $key (not the size of whole cache)\n+sub get_size {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $path = $self->path_to_key($key)\n+\t\tor return undef;\n+\tif (-f $path) {\n+\t\treturn -s $path;\n+\t}\n+\treturn 0;\n+}\n+\n+\n+# ......................................................................\n+# interface methods\n+\n+# Removing and expiring\n+\n+# $cache->remove($key)\n+#\n+# Remove the data associated with the $key from the cache.\n+sub remove {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $file = $self->path_to_key($key)\n+\t\tor return;\n+\treturn unless -f $file;\n+\tunlink($file)\n+\t\tor die \"Couldn't remove file '$file': $!\";\n+}\n+\n+# Getting and setting\n+\n+# $cache->set($key, $data);\n+#\n+# Associates $data with $key in the cache, overwriting any existing entry.\n+# Returns $data.\n+sub set {\n+\tmy ($self, $key, $data) = @_;\n+\n+\treturn unless (defined $key && defined $data);\n+\n+\t$self->store($key, $data);\n+\n+\treturn $data;\n+}\n+\n+# $data = $cache->get($key);\n+#\n+# Returns the data associated with $key. If $key does not exist\n+# or has expired, returns undef.\n+sub get {\n+\tmy ($self, $key) = @_;\n+\n+\treturn $self->fetch($key);;\n+}\n+\n+# $data = $cache->compute($key, $code);\n+#\n+# Combines the get and set operations in a single call. Attempts to\n+# get $key; if successful, returns the value. Otherwise, calls $code\n+# and uses the return value as the new value for $key, which is then\n+# returned.\n+sub compute {\n+\tmy ($self, $key, $code) = @_;\n+\n+\tmy $data = $self->get($key);\n+\tif (!defined $data) {\n+\t\t$data = $code->();\n+\t\t$self->set($key, $data);\n+\t}\n+\n+\treturn $data;\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::SimpleFileCache;\ndiff --git a/t/t9503-gitweb-caching-interface.sh b/t/t9503-gitweb-caching-interface.sh\nnew file mode 100755\nindex 0000000..819da1d\n--- /dev/null\n+++ b/t/t9503-gitweb-caching-interface.sh\n@@ -0,0 +1,34 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2010 Jakub Narebski\n+#\n+\n+test_description='gitweb caching interface\n+\n+This test checks caching interface used in gitweb caching, and caching\n+infrastructure (GitwebCache::* modules).'\n+\n+# for now we are running only cache interface tests\n+. ./test-lib.sh\n+\n+# this test is present in gitweb-lib.sh\n+if ! test_have_prereq PERL; then\n+\tskip_all='perl not available, skipping test'\n+\ttest_done\n+fi\n+\n+\"$PERL_PATH\" -MTest::More -e 0 >/dev/null 2>&1 || {\n+\tskip_all='perl module Test::More unavailable, skipping test'\n+\ttest_done\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+# The external test will outputs its own plan\n+test_external_has_tap=1\n+\n+test_external \\\n+\t'GitwebCache::* Perl API (in gitweb/lib/)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_cache_interface.pl\n+\n+test_done\ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nnew file mode 100755\nindex 0000000..1a3f374\n--- /dev/null\n+++ b/t/t9503/test_cache_interface.pl\n@@ -0,0 +1,85 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use Test::More;\n+\n+# test source version\n+use lib $ENV{GITWEBLIBDIR} || \"$ENV{GIT_BUILD_DIR}/gitweb/lib\";\n+\n+\n+# Test creating a cache\n+#\n+BEGIN { use_ok('GitwebCache::SimpleFileCache'); }\n+diag(\"Using lib '$INC[0]'\");\n+diag(\"Testing '$INC{'GitwebCache/SimpleFileCache.pm'}'\");\n+\n+my $cache = new_ok('GitwebCache::SimpleFileCache');\n+\n+# Test that default values are defined\n+#\n+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,\n+\t'$DEFAULT_CACHE_ROOT defined');\n+ok(defined $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,\n+\t'$DEFAULT_CACHE_DEPTH defined');\n+\n+# Test accessors and default values for cache\n+#\n+SKIP: {\n+\tskip 'default values not defined', 3\n+\t\tunless ($GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT &&\n+\t\t $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH);\n+\n+\tis($cache->get_namespace(), '', \"default namespace is ''\");\n+\tcmp_ok($cache->get_root(), 'eq', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,\n+\t\t\"default cache root is '$GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT'\");\n+\tcmp_ok($cache->get_depth(), '==', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,\n+\t\t\"default cache depth is $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH\");\n+}\n+\n+# Test the getting, setting, and removal of a cached value\n+# (Cache::Cache interface)\n+#\n+my $key = 'Test Key';\n+my $value = 'Test Value';\n+\n+subtest 'Cache::Cache interface' => sub {\n+\tforeach my $method (qw(get set remove)) {\n+\t\tcan_ok($cache, $method);\n+\t}\n+\n+\t$cache->set($key, $value);\n+\tcmp_ok($cache->get_size($key), '>', 0, 'get_size after set, is greater than 0');\n+\tis($cache->get($key), $value, 'get after set, returns cached value');\n+\t$cache->remove($key);\n+\tok(!defined($cache->get($key)), 'get after remove, is undefined');\n+\n+\teval { $cache->remove('Not-Existent Key'); };\n+\tok(!$@, 'remove on non-existent key doesn\\'t die');\n+\tdiag($@) if $@;\n+\n+\tdone_testing();\n+};\n+\n+# Test the getting and setting of a cached value\n+# (CHI interface)\n+#\n+my $call_count = 0;\n+sub get_value {\n+\t$call_count++;\n+\treturn $value;\n+}\n+subtest 'CHI interface' => sub {\n+\tcan_ok($cache, qw(compute));\n+\n+\tis($cache->compute($key, \\&get_value), $value, \"compute 1st time (set) returns '$value'\");\n+\tis($cache->compute($key, \\&get_value), $value, \"compute 2nd time (get) returns '$value'\");\n+\tis($cache->compute($key, \\&get_value), $value, \"compute 3rd time (get) returns '$value'\");\n+\tcmp_ok($call_count, '==', 1, 'get_value() is called once from compute');\n+\n+\tdone_testing();\n+};\n+\n+done_testing();\n-- \n1.7.3\n"},{"id":"157458","messageId":"1291677069-6559-5-git-send-email-jnareb@gmail.com","threadId":"25978","inReplyTo":"1291677069-6559-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 04/24] gitweb/lib - Stat-based cache expiration","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-06T23:10:49Z","receivedAt":"2010-12-06T23:10:49Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add stat-based cache expiration to file-based GitwebCache::SimpleFileCache.\nContrary to the way other caching interfaces such as Cache::Cache and CHI\ndo it, the time when cache element expires in is _global_ value associated\nwith cache instance, and is not local property of cache entry. (Currently\ncache entry does not store any metadata associated with entry... which\nmeans that there is no need for serialization / marshalling / freezing and\nthawing.) Default expire time is -1, which means never expire.\n\nTo check if cache entry is expired, GitwebCache::SimpleFileCache compares\ndifference between mtime (last modify time) of a cache file and current time\nwith (global) time to expire. It is done using CHI-compatible is_valid()\nmethod.\n\nIncludes some tests checking that expiring works correctly (on the level of\nAPI) in t9503.\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\nCompared to \"Gitweb caching v7\" series, checking if cache entry is\nexpired (based on mtime of cache file) is simpler: we reuse stat\nstructure from test if file even exists (via stat(_)).\n\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 40 +++++++++++++++++++++++++++-\n t/t9503/test_cache_interface.pl | 19 +++++++++++++\n 2 files changed, 57 insertions(+), 2 deletions(-)\n\ndiff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm\nindex 2f26a6c..790383d 100644\n--- a/gitweb/lib/GitwebCache/SimpleFileCache.pm\n+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm\n@@ -57,6 +57,10 @@ our $DEFAULT_NAMESPACE = '';\n # The number of subdirectories deep to cache object item. This should be\n # large enough that no cache directory has more than a few hundred objects.\n # Defaults to $DEFAULT_CACHE_DEPTH unless explicitly set.\n+# * 'default_expires_in' (Cache::Cache compatibile),\n+# 'expires_in' (CHI compatibile) [seconds]\n+# The expiration time for objects place in the cache.\n+# Defaults to -1 (never expire) if not explicitly set.\n sub new {\n \tmy $class = shift;\n \tmy %opts = ref $_[0] ? %{ $_[0] } : @_;\n@@ -64,7 +68,7 @@ sub new {\n \tmy $self = {};\n \t$self = bless($self, $class);\n \n-\tmy ($root, $depth, $ns);\n+\tmy ($root, $depth, $ns, $expires_in);\n \tif (%opts) {\n \t\t$root =\n \t\t\t$opts{'cache_root'} ||\n@@ -73,14 +77,19 @@ sub new {\n \t\t\t$opts{'cache_depth'} ||\n \t\t\t$opts{'depth'};\n \t\t$ns = $opts{'namespace'};\n+\t\t$expires_in =\n+\t\t\t$opts{'default_expires_in'} ||\n+\t\t\t$opts{'expires_in'};\n \t}\n \t$root = $DEFAULT_CACHE_ROOT unless defined($root);\n \t$depth = $DEFAULT_CACHE_DEPTH unless defined($depth);\n \t$ns = $DEFAULT_NAMESPACE unless defined($ns);\n+\t$expires_in = -1 unless defined($expires_in); # <0 means never\n \n \t$self->set_root($root);\n \t$self->set_depth($depth);\n \t$self->set_namespace($ns);\n+\t$self->set_expires_in($expires_in);\n \n \treturn $self;\n }\n@@ -92,7 +101,7 @@ sub new {\n # http://perldesignpatterns.com/perldesignpatterns.html#AccessorPattern\n \n # creates get_depth() and set_depth($depth) etc. methods\n-foreach my $i (qw(depth root namespace)) {\n+foreach my $i (qw(depth root namespace expires_in)) {\n \tmy $field = $i;\n \tno strict 'refs';\n \t*{\"get_$field\"} = sub {\n@@ -287,6 +296,31 @@ sub remove {\n \t\tor die \"Couldn't remove file '$file': $!\";\n }\n \n+# $cache->is_valid($key)\n+#\n+# Returns a boolean indicating whether $key exists in the cache\n+# and has not expired (global per-cache 'expires_in').\n+sub is_valid {\n+\tmy ($self, $key) = @_;\n+\n+\tmy $path = $self->path_to_key($key);\n+\n+\t# does file exists in cache?\n+\treturn 0 unless -f $path;\n+\t# get its modification time\n+\tmy $mtime = (stat(_))[9] # _ to reuse stat structure used in -f test\n+\t\tor die \"Couldn't stat file '$path': $!\";\n+\n+\t# expire time can be set to never\n+\tmy $expires_in = $self->get_expires_in();\n+\treturn 1 unless (defined $expires_in && $expires_in >= 0);\n+\n+\t# is file expired?\n+\tmy $now = time();\n+\n+\treturn (($now - $mtime) < $expires_in);\n+}\n+\n # Getting and setting\n \n # $cache->set($key, $data);\n@@ -310,6 +344,8 @@ sub set {\n sub get {\n \tmy ($self, $key) = @_;\n \n+\treturn unless $self->is_valid($key);\n+\n \treturn $self->fetch($key);;\n }\n \ndiff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl\nindex 1a3f374..1517fb6 100755\n--- a/t/t9503/test_cache_interface.pl\n+++ b/t/t9503/test_cache_interface.pl\n@@ -82,4 +82,23 @@ subtest 'CHI interface' => sub {\n \tdone_testing();\n };\n \n+# Test cache expiration\n+#\n+subtest 'cache expiration' => sub {\n+\t$cache->set_expires_in(60*60*24); # set expire time to 1 day\n+\tcmp_ok($cache->get_expires_in(), '>', 0, '\"expires in\" is greater than 0');\n+\tis($cache->get($key), $value, 'get returns cached value (not expired in 1d)');\n+\n+\t$cache->set_expires_in(-1); # set expire time to never expire\n+\tis($cache->get_expires_in(), -1, '\"expires in\" is set to never (-1)');\n+\tis($cache->get($key), $value, 'get returns cached value (not expired)');\n+\n+\t$cache->set_expires_in(0);\n+\tis($cache->get_expires_in(), 0, '\"expires in\" is set to now (0)');\n+\t$cache->set($key, $value);\n+\tok(!defined($cache->get($key)), 'cache is expired');\n+\n+\tdone_testing();\n+};\n+\n done_testing();\n-- \n1.7.3\n"},{"id":"157465","messageId":"1291677069-6559-6-git-send-email-jnareb@gmail.com","threadId":"25978","inReplyTo":"1291677069-6559-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 05/24] gitweb/lib - Regenerate entry if the cache file has size of 0","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-06T23:10:50Z","receivedAt":"2010-12-06T23:10:50Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"If the file representing cache entry has size 0 (in bytes), the cache\nentry is considered invalid, regardless of its freshness, even if cache\nexpiration is turned off.\n\n[jh: This is a quick, and thankfully easy, check to regenerate the\ncache if the resulting file is of size 0. This *SHOULDN'T* happen,\nbut it is possible that it could and thus adding the check.]\n\nBased-on-commit-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\nThis patch is unchanged from previos version of this series.\n\nIt is inspired by commit 22eb7af (Gitweb - Regenerate the cache if the\nresulting file has size of 0, 2010-09-23) on 'master' branch of\ngit/warthog9/gitweb.git repository on git.kernel.org\n\nThe \"gitweb: File based caching layer (from git.kernel.org)\" in\n\"Gitweb caching v7\" by J.H. incudes the same test.\n\n gitweb/lib/GitwebCache/SimpleFileCache.pm | 2 ++\n 1 files changed, 2 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm\nindex 790383d..bf74f7c 100644\n--- a/gitweb/lib/GitwebCache/SimpleFileCache.pm\n+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm\n@@ -310,6 +310,8 @@ sub is_valid {\n \t# get its modification time\n \tmy $mtime = (stat(_))[9] # _ to reuse stat structure used in -f test\n \t\tor die \"Couldn't stat file '$path': $!\";\n+\t# cache entry is invalid if it is size 0 (in bytes)\n+\treturn 0 unless ((stat(_))[7] > 0);\n \n \t# expire time can be set to never\n \tmy $expires_in = $self->get_expires_in();\n-- \n1.7.3\n"},{"id":"157442","messageId":"1291677069-6559-7-git-send-email-jnareb@gmail.com","threadId":"25978","inReplyTo":"1291677069-6559-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 06/24] gitweb/lib - Simple output capture by redirecting STDOUT","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-06T23:10:51Z","receivedAt":"2010-12-06T23:10:51Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add GitwebCache::Capture::Simple package, which captures output by\nredirecting STDOUT to in-memory file (saving what is printed to\nscalar), earlier saving original STDOUT to restore it when finished\ncapturing.\n\nGitwebCache::Capture::Simple preserves PerlIO layers, both those set\nbefore started capturing output, and those set during capture. The\nexceptions is the 'scalar' layer, which needs additional parameter,\nand which for proper handling needs non-core module PerlIO::Util.\n\nNo care was taken to handle the following special cases (prior to\nstarting capture): closed STDOUT, STDOUT reopened to scalar reference,\ntied STDOUT. You shouldn't modify STDOUT during capture.\n\nIncludes separate tests for capturing output in\nt9504/test_capture_interface.pl which is run as external test from\nt9504-gitweb-capture-interface.sh. It tests capturing of utf8 data\nprinted in :utf8 mode, and of binary data (containing invalid utf8) in\n:raw mode.\n\nNote that nested capturing doesn't work (and probably couldn't be made\nto work when capturing to in-memory file), but this feature wouldn't\nbe needed for capturing gitweb output (to cache it).\n\n\nThis patch was based on \"gitweb: add output buffering and associated\nfunctions\" patch by John 'Warthog9' Hawley (J.H.) in \"Gitweb caching v7\"\nseries, and on code of Capture::Tiny by David Golden (Apache License 2.0).\n\nBased-on-work-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\nIn previous version of this series the basic (default) capturing\nengine was GitwebCache::Capture::SelectFH, which made use of\nselect(FH) to change default filehandle used for print and printf\nwithout filehandle argument (print LIST). This required changing\n binmode STDOUT, \nto\n binmode select(), \nin gitweb.perl, which is now not necessary.\n\nTo simplify code this time we don't use GitwebCache::Capture as base\nclass providing common features, but we reserve space to add it if we\nfeel it necessary (e.g. when adding more capturing engines).\n\nEven re-layering is probably not necessary in the case of gitweb, as\nwe set ':utf8' mode at beginning, and if we change it to ':raw' it is\nalways after we started capture.\n\nOriginal \"Gitweb caching v7\" used capturing to in-memory files in the\ncase when caching was disabled; in my minimal fixup i.e. in the\n\"Gitweb caching v7.x\" threads capturing to in-memory file is not done\nat all; output is redirected straight to cache file (or equivalent).\nThe same would be done later in this series.\n\n gitweb/lib/GitwebCache/Capture/Simple.pm | 96 ++++++++++++++++++++\n ...erface.sh => t9504-gitweb-capture-interface.sh} | 10 +-\n t/t9504/test_capture_interface.pl | 91 +++++++++++++++++++\n 3 files changed, 192 insertions(+), 5 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/Capture/Simple.pm\n copy t/{t9503-gitweb-caching-interface.sh => t9504-gitweb-capture-interface.sh} (66%)\n create mode 100755 t/t9504/test_capture_interface.pl\n\ndiff --git a/gitweb/lib/GitwebCache/Capture/Simple.pm b/gitweb/lib/GitwebCache/Capture/Simple.pm\nnew file mode 100644\nindex 0000000..3585e58\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/Capture/Simple.pm\n@@ -0,0 +1,96 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Simple output capturing via redirecting STDOUT to in-memory file.\n+#\n+\n+# This is the same mechanism that Capture::Tiny uses, only simpler;\n+# we don't capture STDERR at all, we don't tee, we don't support\n+# capturing output of external commands.\n+\n+package GitwebCache::Capture::Simple;\n+\n+use strict;\n+use warnings;\n+\n+use PerlIO;\n+\n+# Constructor\n+sub new {\n+\tmy $class = shift;\n+\n+\tmy $self = {};\n+\t$self = bless($self, $class);\n+\n+\treturn $self;\n+}\n+\n+sub capture {\n+\tmy ($self, $code) = @_;\n+\n+\t$self->capture_start();\n+\t$code->();\n+\treturn $self->capture_stop();\n+}\n+\n+# ----------------------------------------------------------------------\n+\n+# Start capturing data (STDOUT)\n+sub capture_start {\n+\tmy $self = shift;\n+\n+\t# save copy of real STDOUT via duplicating it\n+\tmy @layers = PerlIO::get_layers(\\*STDOUT);\n+\topen $self->{'orig_stdout'}, \">&\", \\*STDOUT\n+\t\tor die \"Couldn't dup STDOUT for capture: $!\";\n+\n+\t# close STDOUT, so that it isn't used anymode (to have it fd0)\n+\tclose STDOUT;\n+\n+\t# reopen STDOUT as in-memory file\n+\t$self->{'data'} = '';\n+\tunless (open STDOUT, '>', \\$self->{'data'}) {\n+\t\topen STDOUT, '>&', fileno($self->{'orig_stdout'});\n+\t\tdie \"Couldn't reopen STDOUT as in-memory file for capture: $!\";\n+\t}\n+\t_relayer(\\*STDOUT, \\@layers);\n+\n+\t# started capturing\n+\t$self->{'capturing'} = 1;\n+}\n+\n+# Stop capturing data (required for die_error)\n+sub capture_stop {\n+\tmy $self = shift;\n+\n+\t# return if we didn't start capturing\n+\treturn unless delete $self->{'capturing'};\n+\n+\t# close in-memory file, and restore original STDOUT\n+\tmy @layers = PerlIO::get_layers(\\*STDOUT);\n+\tclose STDOUT;\n+\topen STDOUT, '>&', fileno($self->{'orig_stdout'});\n+\t_relayer(\\*STDOUT, \\@layers);\n+\n+\treturn $self->{'data'};\n+}\n+\n+# taken from Capture::Tiny by David Golden, Apache License 2.0\n+# with debugging stripped out, and added filtering out 'scalar' layer\n+sub _relayer {\n+\tmy ($fh, $layers) = @_;\n+\n+\tmy %seen = ( unix => 1, perlio => 1, scalar => 1 ); # filter these out\n+\tmy @unique = grep { !$seen{$_}++ } @$layers;\n+\n+\tbinmode($fh, join(\":\", \":raw\", @unique));\n+}\n+\n+\n+1;\n+__END__\n+# end of package GitwebCache::Capture::Simple\ndiff --git a/t/t9503-gitweb-caching-interface.sh b/t/t9504-gitweb-capture-interface.sh\nsimilarity index 66%\ncopy from t/t9503-gitweb-caching-interface.sh\ncopy to t/t9504-gitweb-capture-interface.sh\nindex 819da1d..82623f1 100755\n--- a/t/t9503-gitweb-caching-interface.sh\n+++ b/t/t9504-gitweb-capture-interface.sh\n@@ -3,10 +3,10 @@\n # Copyright (c) 2010 Jakub Narebski\n #\n \n-test_description='gitweb caching interface\n+test_description='gitweb capturing interface\n \n-This test checks caching interface used in gitweb caching, and caching\n-infrastructure (GitwebCache::* modules).'\n+This test checks capturing interface used for capturing gitweb output\n+in gitweb caching (GitwebCache::Capture* modules).'\n \n # for now we are running only cache interface tests\n . ./test-lib.sh\n@@ -28,7 +28,7 @@ fi\n test_external_has_tap=1\n \n test_external \\\n-\t'GitwebCache::* Perl API (in gitweb/lib/)' \\\n-\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_cache_interface.pl\n+\t'GitwebCache::Capture Perl API (in gitweb/lib/)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9504/test_capture_interface.pl\n \n test_done\ndiff --git a/t/t9504/test_capture_interface.pl b/t/t9504/test_capture_interface.pl\nnew file mode 100755\nindex 0000000..47ab804\n--- /dev/null\n+++ b/t/t9504/test_capture_interface.pl\n@@ -0,0 +1,91 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+use utf8;\n+\n+use Test::More;\n+\n+# test source version\n+use lib $ENV{GITWEBLIBDIR} || \"$ENV{GIT_BUILD_DIR}/gitweb/lib\";\n+\n+# ....................................................................\n+\n+use_ok('GitwebCache::Capture::Simple');\n+diag(\"Using lib '$INC[0]'\");\n+diag(\"Testing '$INC{'GitwebCache/Capture/Simple.pm'}'\");\n+\n+# Test setting up capture\n+#\n+my $capture = new_ok('GitwebCache::Capture::Simple' => [], 'The $capture');\n+\n+# Test capturing\n+#\n+sub capture_block (&) {\n+\treturn $capture->capture(shift);\n+}\n+\n+diag('Should not print anything except test results and diagnostic');\n+my $test_data = 'Capture this';\n+my $captured = capture_block {\n+\tprint $test_data;\n+};\n+is($captured, $test_data, 'capture simple data');\n+\n+binmode STDOUT, ':utf8';\n+$test_data = <<'EOF';\n+Zażółć gęsią jaźń\n+EOF\n+utf8::decode($test_data);\n+$captured = capture_block {\n+\tbinmode STDOUT, ':utf8';\n+\n+\tprint $test_data;\n+};\n+utf8::decode($captured);\n+is($captured, $test_data, 'capture utf8 data');\n+\n+$test_data = '|\\x{fe}\\x{ff}|\\x{9F}|\\000|'; # invalid utf-8\n+$captured = capture_block {\n+\tbinmode STDOUT, ':raw';\n+\n+\tprint $test_data;\n+};\n+is($captured, $test_data, 'capture raw data');\n+\n+# Test nested capturing\n+#\n+TODO: {\n+\tlocal $TODO = \"not required for capturing gitweb output\";\n+\tno warnings;\n+\n+\tmy $outer_capture = GitwebCache::Capture::Simple->new();\n+\t$captured = $outer_capture->capture(sub {\n+\t\tprint \"pre|\";\n+\t\tmy $captured = $capture->capture(sub {\n+\t\t\tprint \"INNER\";\n+\t\t});\n+\t\tprint lc($captured);\n+\t\tprint \"|post\";\n+\t});\n+\tis($captured, \"pre|inner|post\", 'nested capture');\n+}\n+\n+SKIP: {\n+\tskip \"Capture::Tiny not available\", 1\n+\t\tunless eval { require Capture::Tiny; };\n+\n+\t$captured = Capture::Tiny::capture(sub {\n+\t\tmy $inner = $capture->capture(sub {\n+\t\t\tprint \"INNER\";\n+\t\t});\n+\t});\n+\tis($captured, '', \"doesn't print while capturing\");\n+}\n+\n+done_testing();\n+\n+# Local Variables:\n+# coding: utf-8\n+# End:\n-- \n1.7.3\n"},{"id":"157443","messageId":"1291677069-6559-8-git-send-email-jnareb@gmail.com","threadId":"25978","inReplyTo":"1291677069-6559-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 07/24] gitweb/lib - Cache captured output (using get/set)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-06T23:10:52Z","receivedAt":"2010-12-06T23:10:52Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add GitwebCache::CacheOutput package, which contains cache_output\nsubroutine. The cache_output gets data from cache and prints it, or\ncaptures output of provided subroutine (code reference), saves it to\ncache and prints it. It currently uses Cache::Cache compatibile (get,\nset) interface to cache. The capture_stop currently simply runs\n$capture->capture_stop().\n\nCapturing output is done using GitwebCache::Capture::Simple compatibile\ncapture (->new(), ->capture($code)) passed as one of parameters; the\ndefault solution is to use GitwebCache::Capture::Simple.\n\nGitweb would use cache_output to get page from cache, or to generate\npage and save it to cache. The capture_stop would be used in die_error\nsubroutine, because error pages would not be cached.\n\nIt is assumed that data is saved to cache _converted_, and should\ntherefore be read from cache and printed to STDOUT in ':raw' (binary)\nmode.\n\n\nAdd t9505/test_cache_output.pl test, run as external test in\nt9505-gitweb-cache. It checks that cache_output behaves correctly,\nnamely that it saves and restores action output in cache, and that it\nprints generated output or cached output, depending on whether there\nexist data in cache. This test requires Capture::Tiny to be installed\n(because GitwebCache::Capture::Simple dosn't support recursive\ncapturing).\n\nSigned-off-by: Jakub Narebski \n---\nIn this version capture_stop is defined in GitwebCache::CacheOutput,\ninstead of being re-exported from GitwebCache::Capture (which no\nlonger exists). Compared to previous version cache_output now expects\n$capture (selecting capturing engine) as second argument, and \ncapture_stop expect $cache and $capture, while it was paramless before.\nt9505 now requires Capture::Tiny, because it needs reliable nested\ncapture.\n\ncapture_stop is here only in the case in the future we would accept\nwider range of possible $capture values, e.g. \\&Capture::Tiny::capture\n(reference to capturing subroutine).\n\n\ncache_output($cache, $capture, $key, $code) is roughly equivalent to\ncache_fetch($action) from cache.pm file in \"Gitweb caching v7\" series\nby J.H.... but nothing in GitwebCache::CacheOutput is gitweb specific.\nThis module is simply about caching captured output, using given caching\nengine, given capturing engine, given key to cache entry, and given \ncoderef output of which is to be captured and cached, and then printing\ngenerated data or data from cache.\n\nSimilarly to J.H. patch, and differently from CGI::Cache or\nPlack::Middleware::Cache accessing cache is/would be quite explicit.\n\n gitweb/lib/GitwebCache/CacheOutput.pm | 90 ++++++++++++++++++++\n ...-caching-interface.sh => t9505-gitweb-cache.sh} | 15 ++-\n t/t9505/test_cache_output.pl | 86 +++++++++++++++++++\n 3 files changed, 186 insertions(+), 5 deletions(-)\n create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm\n copy t/{t9503-gitweb-caching-interface.sh => t9505-gitweb-cache.sh} (58%)\n create mode 100755 t/t9505/test_cache_output.pl\n\ndiff --git a/gitweb/lib/GitwebCache/CacheOutput.pm b/gitweb/lib/GitwebCache/CacheOutput.pm\nnew file mode 100644\nindex 0000000..458e314\n--- /dev/null\n+++ b/gitweb/lib/GitwebCache/CacheOutput.pm\n@@ -0,0 +1,90 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2010, Jakub Narebski \n+# (C) 2006, John 'Warthog9' Hawley \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Capturing and caching (gitweb) output\n+#\n+\n+# Capture output, save it in cache and print it, or retrieve it from\n+# cache and print it.\n+\n+package GitwebCache::CacheOutput;\n+\n+use strict;\n+use warnings;\n+\n+use Exporter qw(import);\n+our @EXPORT = qw(cache_output capture_stop);\n+our %EXPORT_TAGS = (all => [ @EXPORT ]);\n+\n+# cache_output($cache, $capture, $key, $action_code);\n+#\n+# Attempts to get $key from $cache; if successful, prints the value.\n+# Otherwise, calls $action_code, capture its output using $capture,\n+# and use the captured output as the new value for $key in $cache,\n+# then print captured output.\n+#\n+# It is assumed that captured data is already converted and it is\n+# in ':raw' format (and thus restored in ':raw' from cache)\n+\n+# default capture class (engine), if none provided\n+our $DEFAULT_CAPTURE_CLASS = 'GitwebCache::Capture::Simple';\n+sub cache_output {\n+\tmy ($cache, $capture, $key, $code) = @_;\n+\n+\t$capture = setup_capture($capture);\n+\n+\t# check if data is in the cache\n+\tmy $data = $cache->get($key);\n+\n+\t# capture and cache output, if there was nothing in the cache\n+\tif (!defined $data) {\n+\t\t$data = $capture->capture($code);\n+\t\t$cache->set($key, $data) if defined $data;\n+\t}\n+\n+\t# print cached data\n+\tif (defined $data) {\n+\t\tbinmode STDOUT, ':raw';\n+\t\tprint $data;\n+\t}\n+\n+\treturn $data;\n+}\n+\n+# capture_stop($cache, $capture);\n+#\n+# Stops capturing output; to be used in die_error, so that error pages\n+# are not cached (not captured and cached).\n+sub capture_stop {\n+\tmy ($cache, $capture) = @_;\n+\n+\tif (defined $capture) {\n+\t\treturn $capture->capture_stop();\n+\t}\n+\treturn;\n+}\n+\n+# ......................................................................\n+# helper subroutines\n+\n+# setup capture engine\n+sub setup_capture {\n+\tmy $capture = shift;\n+\n+\t$capture ||= $DEFAULT_CAPTURE_CLASS;\n+\tif (!ref($capture)) {\n+\t\teval \"require $capture;\" or die $@;\n+\t\t$capture = $capture->new();\n+\t}\n+\n+\treturn $capture;\n+}\n+\n+1;\n+__END__\n+# end of package GitwebCache::CacheOutput\ndiff --git a/t/t9503-gitweb-caching-interface.sh b/t/t9505-gitweb-cache.sh\nsimilarity index 58%\ncopy from t/t9503-gitweb-caching-interface.sh\ncopy to t/t9505-gitweb-cache.sh\nindex 819da1d..181577f 100755\n--- a/t/t9503-gitweb-caching-interface.sh\n+++ b/t/t9505-gitweb-cache.sh\n@@ -3,10 +3,10 @@\n # Copyright (c) 2010 Jakub Narebski\n #\n \n-test_description='gitweb caching interface\n+test_description='gitweb cache\n \n-This test checks caching interface used in gitweb caching, and caching\n-infrastructure (GitwebCache::* modules).'\n+This test checks GitwebCache::CacheOutput Perl module that is\n+responsible for capturing and caching gitweb output.'\n \n # for now we are running only cache interface tests\n . ./test-lib.sh\n@@ -22,13 +22,18 @@ fi\n \ttest_done\n }\n \n+\"$PERL_PATH\" -MCapture::Tiny -e 0 >/dev/null 2>&1 || {\n+\tskip_all='perl module Capture::Tiny unavailable, skipping test'\n+\ttest_done\n+}\n+\n # ----------------------------------------------------------------------\n \n # The external test will outputs its own plan\n test_external_has_tap=1\n \n test_external \\\n-\t'GitwebCache::* Perl API (in gitweb/lib/)' \\\n-\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9503/test_cache_interface.pl\n+\t'GitwebCache::CacheOutput Perl API (in gitweb/lib/)' \\\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/t9505/test_cache_output.pl\n \n test_done\ndiff --git a/t/t9505/test_cache_output.pl b/t/t9505/test_cache_output.pl\nnew file mode 100755\nindex 0000000..167bb36\n--- /dev/null\n+++ b/t/t9505/test_cache_output.pl\n@@ -0,0 +1,86 @@\n+#!/usr/bin/perl\n+use lib (split(/:/, $ENV{GITPERLLIB}));\n+\n+use warnings;\n+use strict;\n+\n+use Test::More;\n+use Capture::Tiny qw(capture);\n+\n+# test source version\n+use lib $ENV{GITWEBLIBDIR} || \"$ENV{GIT_BUILD_DIR}/gitweb/lib\";\n+\n+# ....................................................................\n+\n+# prototypes must be known at compile time, otherwise they do not work\n+BEGIN { use_ok('GitwebCache::CacheOutput'); }\n+\n+require_ok('GitwebCache::SimpleFileCache');\n+require_ok('GitwebCache::Capture::Simple');\n+\n+diag(\"Using lib '$INC[0]'\");\n+diag(\"Testing '$INC{'GitwebCache/CacheOutput.pm'}'\");\n+diag(\"Testing '$INC{'GitwebCache/SimpleFileCache.pm'}'\");\n+diag(\"Testing '$INC{'GitwebCache/Capture/Simple.pm'}'\");\n+\n+\n+# Test setting up $cache and $capture\n+my $cache = new_ok('GitwebCache::SimpleFileCache' => [], 'The $cache ');\n+my $capture = new_ok('GitwebCache::Capture::Simple' => [], 'The $capture');\n+\n+# ......................................................................\n+\n+# Prepare for testing cache_output\n+my $key = 'Key';\n+my $action_output = <<'EOF';\n+# This is data to be cached and shown\n+EOF\n+my $cached_output = <<\"EOF\";\n+$action_output# (version recovered from cache)\n+EOF\n+sub action {\n+\tprint $action_output;\n+}\n+\n+my $no_capture_output = <<\"EOF\";\n+$action_output# (no capture)\n+EOF\n+sub no_capture {\n+\tcapture_stop($cache, $capture);\n+\tprint $no_capture_output;\n+}\n+\n+# Catch output printed by cache_fetch\n+# (only for 'print ' and 'printf ')\n+sub capture_output_of_cache_output {\n+\tmy $code = shift;\n+\n+\tmy ($stdout, $stderr) = capture {\n+\t\tcache_output($cache, $capture, $key, $code);\n+\t};\n+\tprint STDERR $stderr;\n+\treturn $stdout;\n+}\n+\n+# clean state\n+$cache->set_expires_in(-1);\n+$cache->remove($key);\n+my $test_data;\n+\n+# first time (if there is no cache) generates cache entry\n+$test_data = capture_output_of_cache_output(\\&action);\n+is($test_data, $action_output, 'action output is printed (generated)');\n+is($cache->get($key), $action_output, 'action output is saved in cache (generated)');\n+\n+# second time (if cache is set/valid) reads from cache\n+$cache->set($key, $cached_output);\n+$test_data = capture_output_of_cache_output(\\&action);\n+is($test_data, $cached_output, 'action output is printed (from cache)');\n+\n+# test using capture_stop\n+$cache->remove($key);\n+$test_data = capture_output_of_cache_output(\\&no_capture);\n+is($test_data, $no_capture_output, 'no_capture output is printed (generated)');\n+ok(! $cache->get($key), 'no_capture output is not captured and not cached');\n+\n+done_testing();\n-- \n1.7.3\n"},{"id":"157449","messageId":"1291677069-6559-9-git-send-email-jnareb@gmail.com","threadId":"25978","inReplyTo":"1291677069-6559-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 08/24] gitweb: Add optional output caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-06T23:10:53Z","receivedAt":"2010-12-06T23:10:53Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This commit actually adds output caching to gitweb, as we have now\nminimal features required for it in GitwebCache::SimpleFileCache\n(a 'dumb' but fast file-based cache engine). To enable cache you need\n(at least) set $caching_enabled to true in gitweb config, and copy\nrequired modules alongside generated gitweb.cgi - this is described\nin more detail in the new \"Gitweb caching\" section in gitweb/README.\n\"make install-gitweb\" would install all modules alongside gitweb\nitself.\n\nCaching in theory can be done using any Perl module that implements\nCache::Cache compatibile get/set (method) interface. The default is\nto use GitwebCache::SimpleFileCache. Capturing and caching output\nis done via cache_output subroutine from GitwebCache::CacheOutput.\n\nThe cache_output subroutine in GitwebCache::CacheOutput currently uses\nGitwebCache::Capture::Simple compatibile capturing engine passed as\none of parameters to cache_output subroutine. The default is to use\nGitwebCache::Capture::Simple package.\n\nCapturing and caching is designed in such way that there is no\nbehaviour change if $caching_enabled is false. If caching is not\nenabled, then capturing is also turned off.\n\nEnabling caching causes the following additional changes to gitweb\noutput:\n* Disables content-type negotiation (choosing between 'text/html'\n mimetype and 'application/xhtml+xml') when caching, as there is no\n content-type negotiation done when retrieving page from cache.\n Use lowest common denominator of 'text/html' mimetype which can\n be used by all browsers. This may change in the future.\n* Disable optional timing info (how much time it took to generate the\n original page, and how many git commands it took), and in its place show\n unconditionally when page was originally generated (in GMT / UTC\n timezone).\n* Disable 'blame_incremental' view, as it doesn't make sense without\n printing data as soon as it is generated (which would require tee-ing\n when capturing output for caching)... and it doesn't work currently\n anyway. Alternate solution would be to run 'blame_incremental' view\n with caching disabled.\n\n\nAdd basic tests of caching support to t9500-gitweb-standalone-no-errors\ntest: set $caching_enabled to true and check for errors for first time\nrun (generating cache) and second time run (retrieving from cache) for a\nsingle view - summary view for a project.\n\nCheck in the t9501-gitweb-standalone-http-status test that gitweb at\nleast correctly handles \"404 Not Found\" error pages also in the case\nwhen gitweb caching is enabled.\n\nCheck in the t9502-gitweb-standalone-parse-output test that gitweb\nproduces the same output with and without caching, for first and\nsecond run, with binary or text output.\n\nAll those tests make use of new gitweb_enable_caching subroutine added\nto gitweb-lib.sh\n\nInspired-by-code-by: John 'Warthog9' Hawley \nSigned-off-by: Jakub Narebski \n---\nGitweb caching tests are (almost) the same as in\n\n [PATCHv7.4 4/4] gitweb: Minimal testing of gitweb caching\n Message-Id: <1291404335-25541-4-git-send-email-jnareb@gmail.com>\n http://thread.gmane.org/gmane.comp.version-control.git/162830/focus=162834\n\nThe difference is of course in setting up tests, i.e. in the contents\nof gitweb_enable_caching function in t/gitweb-lib.sh.\n\nOnly the t9500 tests were present in previous version of this series.\n\n\nOne of differences from \"Gitweb caching v7\" by J.H. (and minimal fixup of\nthereof) is that in \"gitweb: File based caching layer (from git.kernel.org)\"\nhashed (internal) key is composed of $my_url ($cgi->url()) and\n$ENV{'QUERY_STRING'}... which meant that it ignores path_info (which is\nWRONG, as you can have different views that differ only in path_info part;\ndifferent views which shouldn't be lumped together under one cache key), and\nthat links which differ only in ordering of parameters (e.g. handcrafted\nURL) would result in different cache keys.\n\nI have chosen here to use href(-replay => 1, -full => 1, -path_info => 0) as\nhuman-readable cache key, which means that if different link gives use the\nsame action then it gives us the same cache key.\n\n\nNOTE: I had to put quick fix about base url, so sha1 of gitweb.perl preimage\nmight not match.\n\n gitweb/Makefile | 5 +\n gitweb/README | 58 ++++++++++++\n gitweb/gitweb.perl | 144 +++++++++++++++++++++++++----\n t/gitweb-lib.sh | 11 ++\n t/t9500-gitweb-standalone-no-errors.sh | 20 ++++\n t/t9501-gitweb-standalone-http-status.sh | 13 +++\n t/t9502-gitweb-standalone-parse-output.sh | 33 +++++++\n 7 files changed, 265 insertions(+), 19 deletions(-)\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex e6029e1..d67c138 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -113,6 +113,11 @@ endif\n \n GITWEB_FILES += static/git-logo.png static/git-favicon.png\n \n+# gitweb output caching\n+GITWEB_MODULES += GitwebCache/CacheOutput.pm\n+GITWEB_MODULES += GitwebCache/SimpleFileCache.pm\n+GITWEB_MODULES += GitwebCache/Capture/Simple.pm\n+\n GITWEB_REPLACE = \\\n \t-e 's|++GIT_VERSION++|$(GIT_VERSION)|g' \\\n \t-e 's|++GIT_BINDIR++|$(bindir)|g' \\\ndiff --git a/gitweb/README b/gitweb/README\nindex bf3664f..3dc01bd 100644\n--- a/gitweb/README\n+++ b/gitweb/README\n@@ -246,6 +246,12 @@ not include variables usually directly set during build):\n http://www.andre-simon.de due to assumptions about parameters and output).\n Useful if highlight is not installed on your webserver's PATH.\n [Default: highlight]\n+ * $caching_enabled\n+ If true, gitweb would use caching to speed up generating response.\n+ Currently supported is only output (response) caching. See \"Gitweb caching\"\n+ section below for details on how to configure and customize caching.\n+ The default is false (caching is disabled).\n+\n \n Projects list file format\n ~~~~~~~~~~~~~~~~~~~~~~~~~\n@@ -317,6 +323,58 @@ You can use the following files in repository:\n descriptions.\n \n \n+Gitweb caching\n+~~~~~~~~~~~~~~\n+\n+Currently gitweb supports only output (HTTP response) caching, similar\n+to the one used on http://git.kernel.org. To turn it on, set \n+$caching_enabled variable to true value in gitweb config file, i.e.:\n+\n+ our $caching_enabled = 1;\n+\n+You can choose which caching engine should gitweb use by setting $cache\n+variable to _initialized_ instance of cache interface, e.g.:\n+\n+ use CHI;\n+ our $cache = CHI->new( driver => 'Memcached',\n+ \tservers => [ \"10.0.0.15:11211\", \"10.0.0.15:11212\" ],\n+ \tl1_cache => { driver => 'FastMmap', root_dir => '/var/cache/gitweb' }\n+ );\n+\n+Alternatively you can set $cache variable to the name of cache class,\n+e.g.:\n+\n+ our $cache = 'Cache::FileCache';\n+\n+In this case caching engine should support Cache::Cache or CHI names for\n+cache config (see below), and ignore unrecognized options. Such caching\n+engine should also implement (at least) ->get($key) and ->set($key, $data)\n+methods (Cache::Cache and CHI compatible interface).\n+\n+If $cache is left unset (if it is left undefined), then gitweb would use\n+GitwebCache::SimpleFileCache as caching engine. This engine is 'dumb' (but\n+fast) file based caching layer, currently without any support for cache size\n+limiting, or even removing expired / grossly expired entries. It has\n+therefore the downside of requiring a huge amount of disk space if there are\n+a number of repositories involved. It is not uncommon for git.kernel.org to\n+have on the order of 80G - 120G accumulate over the course of a few months.\n+It is therefore recommended that the cache directory be periodically\n+completely deleted; this operation is safe to perform. Suggested mechanism\n+(substitute $cachedir for actual path to gitweb cache):\n+\n+ # mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush\n+\n+Site-wide cache options are defined in %cache_options hash. Those options\n+apply only when $cache is unset (GitwebCache::SimpleFileCache is used), or\n+if $cache is name of cache class (e.g. $cache = 'Cache::FileCache'). You\n+can override cache options in gitweb config, e.g.:\n+\n+ $cache_options{'expires_in'} = 60; # 60 seconds = 1 minute\n+\n+Please read comments for %cache_options entries in gitweb/gitweb.perl for\n+description of available cache options.\n+\n+\n Webserver configuration\n -----------------------\n \ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex cfa511c..3286709 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -270,6 +270,48 @@ our %highlight_ext = (\n \tmap { $_ => 'xml' } qw(xhtml html htm),\n );\n \n+\n+# This enables/disables the caching layer in gitweb. Currently supported\n+# is only output (response) caching, similar to the one used on git.kernel.org.\n+our $caching_enabled = 0;\n+# Set to _initialized_ instance of cache interface implementing (at least)\n+# get($key) and set($key, $data) methods (Cache::Cache and CHI interfaces),\n+# or to name of class of cache interface implementing said methods.\n+# If unset, GitwebCache::SimpleFileCache would be used, which is 'dumb'\n+# (but fast) file based caching layer, currently without any support for\n+# cache size limiting. It is therefore recommended that the cache directory\n+# be periodically completely deleted; this operation is safe to perform.\n+# Suggested mechanism:\n+# mv $cachedir $cachedir.flush && mkdir $cachedir && rm -rf $cachedir.flush\n+our $cache;\n+# You define site-wide cache options defaults here; override them with\n+# $GITWEB_CONFIG as necessary.\n+our %cache_options = (\n+\t# The location in the filesystem that will hold the root of the cache.\n+\t# This directory will be created as needed (if possible) on the first\n+\t# cache set. Note that either this directory must exists and web server\n+\t# has to have write permissions to it, or web server must be able to\n+\t# create this directory.\n+\t# Possible values:\n+\t# * 'cache' (relative to gitweb),\n+\t# * File::Spec->catdir(File::Spec->tmpdir(), 'gitweb-cache'),\n+\t# * '/var/cache/gitweb' (FHS compliant, requires being set up),\n+\t'cache_root' => 'cache',\n+\n+\t# The number of subdirectories deep to cache object item. This should be\n+\t# large enough that no cache directory has more than a few hundred\n+\t# objects. Each non-leaf directory contains up to 256 subdirectories\n+\t# (00-ff). Must be larger than 0.\n+\t'cache_depth' => 1,\n+\n+\t# The (global) expiration time for objects placed in the cache, in seconds.\n+\t'expires_in' => 20,\n+);\n+# Set to _initialized_ instance of GitwebCache::Capture compatibile capturing\n+# engine, i.e. one implementing ->new() constructor, and ->capture($code)\n+# method. If unset (default), the GitwebCache::Capture::Simple would be used.\n+our $capture;\n+\n # You define site-wide feature defaults here; override them with\n # $GITWEB_CONFIG as necessary.\n our %feature = (\n@@ -1069,7 +1111,15 @@ sub dispatch {\n \t !$project) {\n \t\tdie_error(400, \"Project needed\");\n \t}\n-\t$actions{$action}->();\n+\n+\tif ($caching_enabled) {\n+\t\t# human readable key identifying gitweb output\n+\t\tmy $output_key = href(-replay => 1, -full => 1, -path_info => 0);\n+\n+\t\tcache_output($cache, $capture, $output_key, $actions{$action});\n+\t} else {\n+\t\t$actions{$action}->();\n+\t}\n }\n \n sub reset_timer {\n@@ -1085,6 +1135,8 @@ sub run_request {\n \tevaluate_gitweb_config();\n \tevaluate_git_version();\n \tcheck_loadavg();\n+\tconfigure_caching()\n+\t\tif ($caching_enabled);\n \n \t# $projectroot and $projects_list might be set in gitweb config file\n \t$projects_list ||= $projectroot;\n@@ -1157,6 +1209,42 @@ sub run {\n \t1;\n }\n \n+sub configure_caching {\n+\tif (!eval { require GitwebCache::CacheOutput; 1; }) {\n+\t\t# cache is configured _before_ handling request, so $cgi is not defined,\n+\t\t# so we can't just \"die\" with sending error message to web browser\n+\t\t#die_error(500, \"Caching enabled and GitwebCache::CacheOutput not found\");\n+\n+\t\t# turn off caching and warn instead\n+\t\t$caching_enabled = 0;\n+\t\twarn \"Caching enabled and GitwebCache::CacheOutput not found\";\n+\t}\n+\tGitwebCache::CacheOutput->import();\n+\n+\t# $cache might be initialized (instantiated) cache, i.e. cache object,\n+\t# or it might be name of class, or it might be undefined\n+\tunless (defined $cache && ref($cache)) {\n+\t\t$cache ||= 'GitwebCache::SimpleFileCache';\n+\t\teval \"require $cache\";\n+\t\tdie $@ if $@;\n+\t\t$cache = $cache->new({\n+\t\t\t%cache_options,\n+\t\t\t#'cache_root' => '/tmp/cache',\n+\t\t\t#'cache_depth' => 2,\n+\t\t\t#'expires_in' => 20, # in seconds (CHI compatibile)\n+\t\t\t# (Cache::Cache compatibile initialization)\n+\t\t\t'default_expires_in' => $cache_options{'expires_in'},\n+\t\t\t# (CHI compatibile initialization)\n+\t\t\t'root_dir' => $cache_options{'cache_root'},\n+\t\t\t'depth' => $cache_options{'cache_depth'},\n+\t\t});\n+\t}\n+\tunless (defined $capture && ref($capture)) {\n+\t\trequire GitwebCache::Capture::Simple;\n+\t\t$capture = GitwebCache::Capture::Simple->new();\n+\t}\n+}\n+\n run();\n \n if (defined caller) {\n@@ -3420,7 +3508,9 @@ sub git_header_html {\n \t# 'application/xhtml+xml', otherwise send it as plain old 'text/html'.\n \t# we have to do this because MSIE sometimes globs '*/*', pretending to\n \t# support xhtml+xml but choking when it gets what it asked for.\n-\tif (defined $cgi->http('HTTP_ACCEPT') &&\n+\t# Disable content-type negotiation when caching (use mimetype good for all).\n+\tif (!$caching_enabled &&\n+\t defined $cgi->http('HTTP_ACCEPT') &&\n \t $cgi->http('HTTP_ACCEPT') =~ m/(,|;|\\s|^)application\\/xhtml\\+xml(,|;|\\s|$)/ &&\n \t $cgi->Accept('application/xhtml+xml') != 0) {\n \t\t$content_type = 'application/xhtml+xml';\n@@ -3445,7 +3535,9 @@ sub git_header_html {\n EOF\n \t# the stylesheet, favicon etc urls won't work correctly with path_info\n \t# unless we set the appropriate base URL\n-\tif ($ENV{'PATH_INFO'}) {\n+\t# if caching is enabled we can get it from cache for path_info when it\n+\t# is generated without path_info\n+\tif ($ENV{'PATH_INFO'} || $caching_enabled) {\n \t\tprint \"\\n\";\n \t}\n \t# print out each stylesheet that exist, providing backwards capability\n@@ -3594,17 +3686,25 @@ sub git_footer_html {\n \t}\n \tprint \"\\n\"; # class=\"page_footer\"\n \n-\tif (defined $t0 && gitweb_check_feature('timed')) {\n+\t# timing info doesn't make much sense with output (response) caching,\n+\t# so when caching is enabled gitweb prints the time of page generation\n+\tif ((defined $t0 || $caching_enabled) &&\n+\t gitweb_check_feature('timed')) {\n \t\tprint \"
\\n\";\n-\t\tprint 'This page took '.\n-\t\t ''.\n-\t\t Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).\n-\t\t ' seconds '.\n-\t\t ' and '.\n-\t\t ''.\n-\t\t $number_of_git_cmds.\n-\t\t ' git commands '.\n-\t\t \" to generate.\\n\";\n+\t\tif ($caching_enabled) {\n+\t\t\tprint 'This page was generated at '.\n+\t\t\t gmtime( time() ).\" GMT\\n\";\n+\t\t} else {\n+\t\t\tprint 'This page took '.\n+\t\t\t ''.\n+\t\t\t Time::HiRes::tv_interval($t0, [Time::HiRes::gettimeofday()]).\n+\t\t\t ' seconds '.\n+\t\t\t ' and '.\n+\t\t\t ''.\n+\t\t\t $number_of_git_cmds.\n+\t\t\t ' git commands '.\n+\t\t\t \" to generate.\\n\";\n+\t\t}\n \t\tprint \"
\\n\"; # class=\"page_footer\"\n \t}\n \n@@ -3613,8 +3713,8 @@ sub git_footer_html {\n \t}\n \n \tprint qq!\\n!;\n-\tif (defined $action &&\n-\t $action eq 'blame_incremental') {\n+\tif (!$caching_enabled &&\n+\t defined $action && $action eq 'blame_incremental') {\n \t\tprint qq!