[PATCHv5 07/17] gitweb/lib - Cache captured output (using get/set)

From

Jakub Narebski <jnareb@gmail.com>

Date

Oct 6, 2010, 22:01 UTC

Message-ID

<1286402526-13143-8-git-send-email-jnareb@gmail.com>

In-Reply-To

<1286402526-13143-1-git-send-email-jnareb@gmail.com>

Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
Differences from v4:
* Similarly to other patches in this revision of gitweb caching
  series, the new test for this new module, written in Perl, is now
  invoked from a separate test script, to make it possible to run
  'prove' on git test suite.

  Also new test uses GIT_BUILD_DIR rather than TEST_DIRECTORY/.., and
  it respects GITWEBLIBDIR to make it possible to test installed
  version of this module.

* GitwebCache::CacheOutput no longer loads GitwebCache::SimpleFileCache;
  It is now assumed that cache_output() caller loads required packages
  itself, so that ->get() and ->set() methods $cache instance, passed
  as argument to cache_output(), work correctly.

  This required small change to t/t9505/test_cache_output.pl

Differences from relevant parts of J.H. patch:
* cache_fetch() subroutine, now named cache_output(), is much, much
  simpler.  Well, at this point in the patch series it lacks most of
  the features of original cache_fetch() by J.H.: it doesn't have
  adaptive cache lifetime, nor locking to prevent 'stampeding herd'
  problem, nor serving stale data when waiting for cache regeneration,
  nor background data generation, nor activity indicator... but the
  cache_output() itself doesn't change much in later commits, as those
  features are added mainly via methods and subroutines that
  cache_output() calls.

* Capturing gitweb output would be done without need to extensively
  modify gitweb to either 1) save generated output into $output
  variable, i.e. replace 'print <sth>' by '$output .= <sth>', and print
  it or save in cache after it is generated in full (original J.H. patch
  in "Gitweb caching v2"), or 2) changing all print statements to print
  to explicit filehandle, i.e. replace 'print <sth>' by 
  'print {$out} <sth>', which filehandle points to STDOUT if caching is
  disabled and to in-memory file if caching is enabled (modified
  J.H. patch in "Gitweb caching v5").

  Currently capturing output is hardcoded to GitwebCache::Capture::SelectFH,
  but changing it to other compatibile capturing engine requires (in
  current version) change only only two lines in GitwebCache::CacheOutput
  module.

* It is assumed that capture generates binary output (with I/O filters
  already applied), and that therefore we read from cache files in
  binary mode, and we print both captured output and data retrieved from
  cache in ':raw' mode (in binmode).

* The key (human-readable unique id) for a page (for given capture) is
  passed explicitely to cache_output().  The cache_output() subroutine
  no longer depends of subroutines (like href()) and variables (like
  $cation or %actions) defined in gitweb.perl.

  This allows to simply use 'require <package>' instead of 'do $package_pm'
  (which also means that it can be installed as module in PERL5LIB etc.).

* Added tests for cache_output() behavior.

 gitweb/lib/GitwebCache/CacheOutput.pm |   64 ++++++++++++++++++++++++++++++
 t/t9505-gitweb-cache.sh               |   34 ++++++++++++++++
 t/t9505/test_cache_output.pl          |   70 +++++++++++++++++++++++++++++++++
 3 files changed, 168 insertions(+), 0 deletions(-)
 create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm
 create mode 100755 t/t9505-gitweb-cache.sh
 create mode 100755 t/t9505/test_cache_output.pl

diff --git a/gitweb/lib/GitwebCache/CacheOutput.pm b/gitweb/lib/GitwebCache/CacheOutput.pm
new file mode 100644
index 0000000..bba73ee
--- /dev/null
+++ b/gitweb/lib/GitwebCache/CacheOutput.pm
@@ -0,0 +1,64 @@
+# gitweb - simple web interface to track changes in git repositories
+#
+# (C) 2010, Jakub Narebski <jnareb@gmail.com>
+# (C) 2006, John 'Warthog9' Hawley <warthog19@eaglescrag.net>
+#
+# This program is licensed under the GPLv2
+
+#
+# Capturing and caching (gitweb) output
+#
+
+# Capture output, save it in cache and print it, or retrieve it from
+# cache and print it.
+
+package GitwebCache::CacheOutput;
+
+use strict;
+use warnings;
+
+use GitwebCache::Capture::SelectFH qw(:all);
+
+use Exporter qw(import);
+our @EXPORT      = qw(cache_output capture_stop);
+our %EXPORT_TAGS = (all => [ @EXPORT ]);
+
+# cache_output($cache, $key, $action_code);
+#
+# Attempts to get $key from $cache; if successful, prints the value.
+# Otherwise, calls $action_code, capture its output and use
+# the captured output as the new value for $key in $cache,
+# then print captured output.
+#
+# It is assumed that captured data is already converted and it is
+# in ':raw' format (and thus restored in ':raw' from cache)
+our $CAPTURE_CLASS = 'GitwebCache::Capture::SelectFH';
+
+sub cache_output {
+	my ($cache, $key, $code) = @_;
+
+	my $data = $cache->get($key);
+
+	# capture and cache output, if there was nothing in the cache
+	if (!defined $data) {
+		my $capture = $CAPTURE_CLASS;
+		setup_capture($capture);
+
+		# do not use 'capture_block' prototype
+		$data = &capture_block($code);
+		$cache->set($key, $data) if defined $data;
+	}
+
+	# print cached data
+	if (defined $data) {
+		# select() instead of STDOUT is here for tests:
+		binmode select(), ':raw';
+		print $data;
+	}
+
+	return $data;
+}
+
+1;
+__END__
+# end of package GitwebCache::CacheOutput;
diff --git a/t/t9505-gitweb-cache.sh b/t/t9505-gitweb-cache.sh
new file mode 100755
index 0000000..55d3e17
--- /dev/null
+++ b/t/t9505-gitweb-cache.sh
@@ -0,0 +1,34 @@
+#!/bin/sh
+#
+# Copyright (c) 2010 Jakub Narebski
+#
+
+test_description='gitweb cache
+
+This test checks GitwebCache::CacheOutput Perl module that is
+responsible for capturing and caching gitweb output.'
+
+# for now we are running only cache interface tests
+. ./test-lib.sh
+
+# this test is present in gitweb-lib.sh
+if ! test_have_prereq PERL; then
+	skip_all='perl not available, skipping test'
+	test_done
+fi
+
+"$PERL_PATH" -MTest::More -e 0 >/dev/null 2>&1 || {
+	skip_all='perl module Test::More unavailable, skipping test'
+	test_done
+}
+
+# ----------------------------------------------------------------------
+
+# The external test will outputs its own plan
+test_external_has_tap=1
+
+test_external \
+	'GitwebCache::CacheOutput Perl API (in gitweb/lib/)' \
+	"$PERL_PATH" "$TEST_DIRECTORY"/t9505/test_cache_output.pl
+
+test_done
diff --git a/t/t9505/test_cache_output.pl b/t/t9505/test_cache_output.pl
new file mode 100755
index 0000000..0343591
--- /dev/null
+++ b/t/t9505/test_cache_output.pl
@@ -0,0 +1,70 @@
+#!/usr/bin/perl
+use lib (split(/:/, $ENV{GITPERLLIB}));
+
+use warnings;
+use strict;
+
+use Test::More;
+
+# test source version
+use lib $ENV{GITWEBLIBDIR} || "$ENV{GIT_BUILD_DIR}/gitweb/lib";
+
+# ....................................................................
+
+# prototypes must be known at compile time, otherwise they do not work
+BEGIN { use_ok('GitwebCache::CacheOutput'); }
+
+# load GitwebCache::SimpleFileCache (gitweb.perl uses require too)
+# GitwebCache::Capture::SelectFH is loaded by GitwebCache::CacheOutput
+require_ok('GitwebCache::SimpleFileCache');
+
+# Test setting up $cache and $capture
+my $cache   = new_ok('GitwebCache::SimpleFileCache'   => [], 'The $cache  ');
+my $capture = new_ok('GitwebCache::Capture::SelectFH' => [], 'The $capture');
+
+# ......................................................................
+
+# Prepare for testing cache_output
+my $key = 'Key';
+my $action_output = <<'EOF';
+# This is data to be cached and shown
+EOF
+my $cached_output = <<"EOF";
+$action_output# (version recovered from cache)
+EOF
+sub action {
+	print $action_output;
+}
+
+# Catch output printed by cache_fetch
+# (only for 'print <sth>' and 'printf <sth>')
+sub capture_output_of_cache_output {
+	my $test_data = '';
+
+	open my $test_data_fh, '>', \$test_data;
+	my $oldfh = select($test_data_fh);
+
+	cache_output($cache, $key, \&action);
+
+	select($oldfh);
+	close $test_data_fh;
+
+	return $test_data;
+}
+
+# clean state
+$cache->set_expires_in(-1);
+$cache->remove($key);
+my $test_data;
+
+# first time (if there is no cache) generates cache entry
+$test_data = capture_output_of_cache_output();
+is($test_data, $action_output,        'action output is printed (generated)');
+is($cache->get($key), $action_output, 'action output is saved in cache (generated)');
+
+# second time (if cache is set/valid) reads from cache
+$cache->set($key, $cached_output);
+$test_data = capture_output_of_cache_output();
+is($test_data, $cached_output,        'action output is printed (from cache)');
+
+done_testing();

-- 
1.7.3