git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFC PATCHv4 18/17] gitweb/lib - Add clear() and size() methods to caching interface

From
Jakub Narebski <jnareb@gmail.com>
Date
Jun 24, 2010, 07:56 UTC
Message-ID
<1277366213-28381-1-git-send-email-jnareb@gmail.com>
In-Reply-To
<1276531710-22945-1-git-send-email-jnareb@gmail.com>

Add ->size() method, which following Cache::Cache interface returns estimated total size of all entries in whole cache (in the namsepace assiciated with give cache instance). Note that ->get_size($key) returns size of a single entry!

Add ->clear() method, which removes all entries from the namespace associated with given cache instance. For safety it requires namespace to be set to true value, which means that it cannot be empty; therefore default namespace is changed to 'gitweb'.

The ->clear() method should be fairly safe, because it first renames directory (which should be atomic), and only then removes it (following code from CGI::Driver::File).

Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This is new patch, implementing methods that allow for rudimentary
gitweb cache administration.  For now it is only safe removing whole
cache, in the future it could include removing only stale (expired)
entries.
This is "backend" patch.
 gitweb/lib/GitwebCache/SimpleFileCache.pm |   51 ++++++++++++++++++++++++++--
 t/t9503/test_cache_interface.pl           |    1 -
 2 files changed, 47 insertions(+), 5 deletions(-)
diff --git a/gitweb/lib/GitwebCache/SimpleFileCache.pm b/gitweb/lib/GitwebCache/SimpleFileCache.pm
index 9eb4896..457bbe7 100644
--- a/gitweb/lib/GitwebCache/SimpleFileCache.pm
+++ b/gitweb/lib/GitwebCache/SimpleFileCache.pm
@@ -20,8 +20,9 @@ package GitwebCache::SimpleFileCache;
 use strict;
 use warnings;
 
-use File::Path qw(mkpath);
-use File::Temp qw(tempfile);
+use File::Path qw(mkpath rmtree);
+use File::Temp qw(tempfile mktemp);
+use File::Find qw(find);
 use Digest::MD5 qw(md5_hex);
 
 # by default, the cache nests all entries on the filesystem single
@@ -37,7 +38,7 @@ our $DEFAULT_CACHE_ROOT = "cache";
 # by default we don't use cache namespace (empty namespace);
 # empty namespace does not allow for simple implementation of clear() method.
 #
-our $DEFAULT_NAMESPACE = '';
+our $DEFAULT_NAMESPACE = "gitweb";
 
 # ......................................................................
 # constructor
@@ -202,6 +203,9 @@ sub path_to_namespace {
 	return $self->{'path_to_namespace'};
 }
 
+# $path = $cache->path_to_key($key);
+# $path = $cache->path_to_key($key, \$dir);
+#
 # Take an human readable key, and return file path.
 # Puts dirname of file path in second argument, if it is provided.
 sub path_to_key {
@@ -321,7 +325,7 @@ sub get_size {
 }
 
 # ......................................................................
-# interface methods
+# interface methods dealing with single item
 
 # Removing and expiring
 
@@ -412,6 +416,45 @@ sub compute {
 	return $data;
 }
 
+# ......................................................................
+# interface methods dealing with whole namespace
+
+# $cache->clear();
+#
+# Remove all entries from the namespace.
+# Namespace must be defined and not empty.
+sub clear {
+	my $self = shift;
+
+	return unless $self->get_namespace();
+
+	my $namespace_dir = $self->path_to_namespace();
+	return if !-d $namespace_dir;
+
+	my $renamed_dir = mktemp($namespace_dir . '.XXXX');
+	rename($namespace_dir, $renamed_dir);
+	rmtree($renamed_dir);
+	die "Couldn't remove '$renamed_dir' directory"
+		if -d $renamed_dir;
+}
+
+# $size = $cache->size();
+#
+# Size of whole names (or whole cache if namespace empty)
+sub size {
+	my $self = shift;
+
+	my $namespace_dir = $self->path_to_namespace();
+	return if !-d $namespace_dir;
+
+	my $total_size = 0;
+	my $add_size = sub { $total_size += -s $_ };
+
+	File::Find::find({ wanted => $add_size, no_chdir => 1 }, $namespace_dir);
+
+	return $total_size;
+}
+
 1;
 __END__
 # end of package GitwebCache::SimpleFileCache;
diff --git a/t/t9503/test_cache_interface.pl b/t/t9503/test_cache_interface.pl
index a84faf9..0a18f77 100755
--- a/t/t9503/test_cache_interface.pl
+++ b/t/t9503/test_cache_interface.pl
@@ -35,7 +35,6 @@ SKIP: {
 		unless ($GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT &&
 		        $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH);
 
-	is($cache->get_namespace(), '', "default namespace is ''");
 	cmp_ok($cache->get_root(),  'eq', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT,
 		"default cache root is '$GitwebCache::SimpleFileCache::DEFAULT_CACHE_ROOT'");
 	cmp_ok($cache->get_depth(), '==', $GitwebCache::SimpleFileCache::DEFAULT_CACHE_DEPTH,
-- 
1.7.0.1
Previous: Jakub NarebskiNext: Jakub Narebski
Message 20 of 21 in “[RFC PATCHv4 00/17] gitweb: Simple file based output caching”
  1. Jakub NarebskiJun 14, 2010
  2. 01/17 gitweb: Return or exit after done serving requestJakub Narebski, Jun 14, 2010
  3. 02/17 gitweb: Fix typo in hash key name in %opts in git_header_htmlJakub Narebski, Jun 14, 2010
  4. 03/17 gitweb: Prepare for splitting gitwebJakub Narebski, Jun 14, 2010
  5. 04/17 gitweb/lib - Very simple file based cacheJakub Narebski, Jun 14, 2010
  6. 05/17 gitweb/lib - Stat-based cache expirationJakub Narebski, Jun 14, 2010
  7. 06/17 gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)Jakub Narebski, Jun 14, 2010
  8. 07/17 gitweb/lib - Simple select(FH) based output captureJakub Narebski, Jun 14, 2010
  9. 08/17 gitweb/lib - Alternate ways of capturing outputJakub Narebski, Jun 14, 2010
  10. 09/17 gitweb/lib - Cache captured output (using get/set)Jakub Narebski, Jun 14, 2010
  11. 10/17 gitweb: Add optional output caching (from gitweb/cache.pm)Jakub Narebski, Jun 14, 2010
  12. 10/18 gitweb/lib - Cache captured output (using get/set)Jakub Narebski, Jun 14, 2010
  13. 11/17 gitweb/lib - Adaptive cache expiration timeJakub Narebski, Jun 14, 2010
  14. 12/17 gitweb/lib - Use CHI compatibile (compute method) caching interfaceJakub Narebski, Jun 14, 2010
  15. 13/17 gitweb/lib - Use locking to avoid 'cache miss stampede' problemJakub Narebski, Jun 14, 2010
  16. 14/17 gitweb/lib - Serve stale data when waiting for filling cacheJakub Narebski, Jun 14, 2010
  17. 15/17 gitweb/lib - Regenerate (refresh) cache in backgroundJakub Narebski, Jun 14, 2010
  18. 16/17 gitweb: Show appropriate "Generating..." page when regenerating cacheJakub Narebski, Jun 14, 2010
  19. 17/17 gitweb: Add startup delay to activity indicator for cacheJakub Narebski, Jun 14, 2010
  20. 18/17 gitweb/lib - Add clear() and size() methods to caching interfaceJakub Narebski, Jun 24, 2010
  21. 19/17 gitweb: Add beginnings of cache administration page (WIP)Jakub Narebski, Jun 24, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.