git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFC PATCHv4 07/17] gitweb/lib - Simple select(FH) based output capture

From
Jakub Narebski <jnareb@gmail.com>
Date
Jun 14, 2010, 16:08 UTC
Message-ID
<1276531710-22945-8-git-send-email-jnareb@gmail.com>
In-Reply-To
<1276531710-22945-1-git-send-email-jnareb@gmail.com>

Add two packages: GitwebCache::Capture, which defines interface, and GitwebCache::Capture::SelectFH, which is actually implements simple capturing. GitwebCache::Capture::SelectFH captures output by using select(FILEHANDLE) to change default filehandle for output. This means that output of a "print" or a "printf" (or a "write") without a filehandle would be captured.

To change mode of filehandle used for capturing correctly,
  binmode select(), <mode>;
needs to be used in place of
  binmode STDOUT, <mode>;
Capturing is done using in-memory file held in Perl scalar.

Using select(FILEHANDLE) is a bit fragile as a method of capturing output, as it assumes that we always use "print" or "printf" without filehandle, and use select() which returns default filehandle for output in place of explicit STDOUT. On the other hand it has the advantage of being simple. Alternate solutions include using tie (like in CGI::Cache), or using PerlIO layers - but the last requires non-standard PerlIO::Util module.

Includes separate tests for capturing output, in t9503/test_capture.pl which is run as external test from t9503-gitweb-caching.sh

Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
 gitweb/lib/GitwebCache/Capture.pm          |   66 ++++++++++++++++++++++
 gitweb/lib/GitwebCache/Capture/SelectFH.pm |   82 ++++++++++++++++++++++++++++
 gitweb/lib/GitwebCache/SimpleFileCache.pm  |    7 +--
 t/t9503-gitweb-caching.sh                  |    5 ++-
 t/t9503/test_capture_interface.pl          |   76 ++++++++++++++++++++++++++
 5 files changed, 231 insertions(+), 5 deletions(-)
 create mode 100644 gitweb/lib/GitwebCache/Capture.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm
 create mode 100755 t/t9503/test_capture_interface.pl
diff --git a/gitweb/lib/GitwebCache/Capture.pm b/gitweb/lib/GitwebCache/Capture.pm
new file mode 100644
index 0000000..3e9fe81
--- /dev/null
+++ b/gitweb/lib/GitwebCache/Capture.pm
@@ -0,0 +1,66 @@
+# gitweb - simple web interface to track changes in git repositories
+#
+# (C) 2010, Jakub Narebski <jnareb@gmail.com>
+#
+# This program is licensed under the GPLv2
+
+#
+# Output capturing for gitweb caching engine
+#
+
+# It is base abstract class (a role) for capturing output of gitweb
+# actions for gitweb caching engine.
+# 
+# Child (derived) concrete classes, which actually implement some method
+# of capturing STDOUT output, must implement the following methods:
+# * ->new(), to create new object of a capturing class
+# * ->start(), to start capturing output
+# * ->stop(), to stop capturing output and return it
+#
+# Before starting capture by using capture_block etc. subroutines,
+# one has to run <child class>->setup().
+
+package GitwebCache::Capture;
+
+use strict;
+use warnings;
+
+use Exporter qw(import);
+our @EXPORT    = qw(capture_start capture_stop capture_block);
+our @EXPORT_OK = qw(setup_capture);
+our %EXPORT_TAGS = (all => [ @EXPORT, @EXPORT_OK ]);
+
+# Holds object used for capture (of child class)
+my $capture;
+
+sub setup_capture {
+	my $self = shift || __PACKAGE__;
+
+	$capture = $self->new(@_);
+}
+
+sub capture {
+	my ($self, $code) = @_;
+
+	$self->start();
+	$code->();
+	return $self->stop();
+}
+
+# Wrap caching data; capture only STDOUT
+sub capture_block (&) {
+	my $code = shift;
+	return $capture->capture($code);
+}
+
+sub capture_start {
+	$capture->start(@_);
+}
+
+sub capture_stop {
+	return $capture->stop(@_);
+}
+
+1;
+__END__
+# end of package GitwebCache::Capture;
diff --git a/gitweb/lib/GitwebCache/Capture/SelectFH.pm b/gitweb/lib/GitwebCache/Capture/SelectFH.pm
new file mode 100644
index 0000000..2b904d3
--- /dev/null
+++ b/gitweb/lib/GitwebCache/Capture/SelectFH.pm
@@ -0,0 +1,82 @@
+# gitweb - simple web interface to track changes in git repositories
+#
+# (C) 2010, Jakub Narebski <jnareb@gmail.com>
+#
+# This program is licensed under the GPLv2
+
+#
+# Simple output capturing using select(FH);
+#
+
+# This module (class) captures output of 'print <sth>', 'printf <sth>'
+# and 'write <sth>' (without a filehandle) by using select(FILEHANDLE)
+# to change default filehandle for output, changing it to in-memory
+# file (saving output to scalar).
+#
+# Note that when using this simplest way of capturing, to change mode of
+# filehandle using for capturing correctly, "binmode STDOUT, <mode>;"
+# has to be changed to "binmode select(), <mode>;".  This has no change
+# if we are not capturing output using GitwebCache::Capture::SelectFH.
+
+package GitwebCache::Capture::SelectFH;
+
+use PerlIO;
+
+use strict;
+use warnings;
+
+use base qw(GitwebCache::Capture);
+use GitwebCache::Capture qw(:all);
+
+use Exporter qw(import);
+our @EXPORT      = @GitwebCache::Capture::EXPORT;
+our @EXPORT_OK   = @GitwebCache::Capture::EXPORT_OK;
+our %EXPORT_TAGS = %GitwebCache::Capture::EXPORT_TAGS;
+
+# Constructor
+sub new {
+	my $proto = shift;
+
+	my $class = ref($proto) || $proto;
+	my $self  = {};
+	$self = bless($self, $class);
+
+	$self->{'oldfh'} = select();
+	$self->{'data'} = '';
+
+	return $self;
+}
+
+# Start capturing data (STDOUT)
+# (printed using 'print <sth>' or 'printf <sth>')
+sub start {
+	my $self = shift;
+	
+	$self->{'data'}    = '';
+	$self->{'data_fh'} = undef;
+	
+	open $self->{'data_fh'}, '>', \$self->{'data'}
+		or die "Couldn't open in-memory file for capture: $!";
+	$self->{'oldfh'} = select($self->{'data_fh'});
+
+	# note: this does not cover all cases
+	binmode select(), ':utf8'
+		if ((PerlIO::get_layers($self->{'oldfh'}))[-1] eq 'utf8');
+}
+
+# Stop capturing data (required for die_error)
+sub stop {
+	my $self = shift;
+
+	# return if we didn't start capturing
+	return unless defined $self->{'data_fh'};
+	
+	select($self->{'oldfh'});
+	close $self->{'data_fh'}
+		or die "Couldn't close in-memory file for capture: $!";
+	return $self->{'data'};
+}
+
+1;
+__END__
+# end of package GitwebCache::Capture::SelectFH;
diff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm
index 91b3373..7c90350 100644
--- a/gitweb/lib/GitwebCache/SimpleFileCache.pm
+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm
@@ -201,16 +201,17 @@ sub store {
 
 	# ensure that directory leading to cache file exists
 	if (!-d $dir) {
-		mkpath($dir, 0, 0777)
+		eval { mkpath($dir, 0, 0777); 1 }
 			or die "Couldn't create leading directory '$dir' (mkpath): $!";
 	}
 
 	# generate a temporary file
+	# (Temp::File dies itself if it cannot create file)
 	my $temp = File::Temp->new(
 		#DIR => $dir,
 		TEMPLATE => "${file}_XXXXX",
 		SUFFIX => ".tmp"
-	) or die "Couldn't create temporary file with '${file}_XXXXX' template: $!";
+	);# or die "Couldn't create temporary file with '${file}_XXXXX' template: $!";
 	chmod 0666, $temp
 		or warn "Couldn't change permissions to 0666 for '$temp': $!";
 
@@ -342,5 +343,3 @@ sub compute {
 1;
 __END__
 # end of package GitwebCache::SimpleFileCache;
-
-
diff --git a/t/t9503-gitweb-caching.sh b/t/t9503-gitweb-caching.sh
index 768080c..73b3f5a 100755
--- a/t/t9503-gitweb-caching.sh
+++ b/t/t9503-gitweb-caching.sh
@@ -26,7 +26,10 @@ fi
 
 # ----------------------------------------------------------------------
 
-test_external 'GitwebCache::* Perl API (in gitweb/cache.pm)' \
+test_external 'GitwebCache::SimpleFileCache Perl API (in gitweb/cache.pm)' \
 	"$PERL_PATH" "$TEST_DIRECTORY"/t9503/test_cache_interface.pl
 
+test_external 'GitwebCache::Capture Perl API (in gitweb/cache.pm)' \
+	"$PERL_PATH" "$TEST_DIRECTORY"/t9503/test_capture_interface.pl
+
 test_done
diff --git a/t/t9503/test_capture_interface.pl b/t/t9503/test_capture_interface.pl
new file mode 100755
index 0000000..d7aa480
--- /dev/null
+++ b/t/t9503/test_capture_interface.pl
@@ -0,0 +1,76 @@
+#!/usr/bin/perl
+use lib (split(/:/, $ENV{GITPERLLIB}));
+
+use warnings;
+use strict;
+use utf8;
+
+use Test::More;
+
+# test source version
+use lib "$ENV{TEST_DIRECTORY}/../gitweb/lib";
+
+# ....................................................................
+
+# prototypes must be known at compile time, otherwise they do not work
+BEGIN { use_ok('GitwebCache::Capture::SelectFH', qw(:all)); }
+
+# Test setting up capture
+#
+my $capture = new_ok('GitwebCache::Capture::SelectFH' => [], 'The $capture');
+isa_ok($capture, 'GitwebCache::Capture', 'The $capture');
+ok(setup_capture('GitwebCache::Capture::SelectFH'),
+   'setup_capture with package name: GitwebCache::Capture::SelectFH');
+ok(setup_capture($capture),
+   'setup_capture with subclass object: $capture');
+
+# Test properties of capture_block
+#
+is(prototype('capture_block'), '&', 'capture_block has (&) prototype');
+
+# Test capturing
+#
+diag('Should not print anything except test results and diagnostic');
+my $test_data = 'Capture this';
+my $captured = capture_block {
+	print $test_data;
+};
+is($captured, $test_data, 'capture_block captures simple data');
+
+binmode STDOUT, ':utf8';
+$test_data = <<'EOF';
+Áéí óú
+ÄËÑÏÖ
+Ábçdèfg
+Zażółć gęsią jaźń
+山田 太郎
+ブレームのテストです。
+
+はれひほふ
+
+しているのが、いるので。
+濱浜ほれぷりぽれまびぐりろへ。
+EOF
+utf8::decode($test_data);
+$captured = capture_block {
+	binmode select(), ':utf8';
+
+	print $test_data;
+};
+utf8::decode($captured);
+is($captured, $test_data, 'capture_block captures utf8 data');
+
+$test_data = '|\x{fe}\x{ff}|\x{9F}|\000|'; # invalid utf-8
+$captured = capture_block {
+	binmode select(), ':raw';
+
+	print $test_data;
+};
+is($captured, $test_data, 'capture_block captures raw data');
+
+
+done_testing();
+
+# Local Variables:
+# encoding: utf-8
+# End:

-- 
1.7.0.1
Previous: Jakub NarebskiNext: Jakub Narebski
Message 8 of 21 in “[RFC PATCHv4 00/17] gitweb: Simple file based output caching”
  1. Jakub NarebskiJun 14, 2010
  2. 01/17 gitweb: Return or exit after done serving requestJakub Narebski, Jun 14, 2010
  3. 02/17 gitweb: Fix typo in hash key name in %opts in git_header_htmlJakub Narebski, Jun 14, 2010
  4. 03/17 gitweb: Prepare for splitting gitwebJakub Narebski, Jun 14, 2010
  5. 04/17 gitweb/lib - Very simple file based cacheJakub Narebski, Jun 14, 2010
  6. 05/17 gitweb/lib - Stat-based cache expirationJakub Narebski, Jun 14, 2010
  7. 06/17 gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)Jakub Narebski, Jun 14, 2010
  8. 07/17 gitweb/lib - Simple select(FH) based output captureJakub Narebski, Jun 14, 2010
  9. 08/17 gitweb/lib - Alternate ways of capturing outputJakub Narebski, Jun 14, 2010
  10. 09/17 gitweb/lib - Cache captured output (using get/set)Jakub Narebski, Jun 14, 2010
  11. 10/17 gitweb: Add optional output caching (from gitweb/cache.pm)Jakub Narebski, Jun 14, 2010
  12. 10/18 gitweb/lib - Cache captured output (using get/set)Jakub Narebski, Jun 14, 2010
  13. 11/17 gitweb/lib - Adaptive cache expiration timeJakub Narebski, Jun 14, 2010
  14. 12/17 gitweb/lib - Use CHI compatibile (compute method) caching interfaceJakub Narebski, Jun 14, 2010
  15. 13/17 gitweb/lib - Use locking to avoid 'cache miss stampede' problemJakub Narebski, Jun 14, 2010
  16. 14/17 gitweb/lib - Serve stale data when waiting for filling cacheJakub Narebski, Jun 14, 2010
  17. 15/17 gitweb/lib - Regenerate (refresh) cache in backgroundJakub Narebski, Jun 14, 2010
  18. 16/17 gitweb: Show appropriate "Generating..." page when regenerating cacheJakub Narebski, Jun 14, 2010
  19. 17/17 gitweb: Add startup delay to activity indicator for cacheJakub Narebski, Jun 14, 2010
  20. 18/17 gitweb/lib - Add clear() and size() methods to caching interfaceJakub Narebski, Jun 24, 2010
  21. 19/17 gitweb: Add beginnings of cache administration page (WIP)Jakub Narebski, Jun 24, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.