Re: [PATCH] gitweb: Support caching projects list

From: Jakub Narebski <jnareb@gmail.com>
Date: Mar 15, 2008, 21:44 UTC
Message-ID: <m3ve3nwtl3.fsf@localhost.localdomain>
In-Reply-To: <20080313231413.27966.3383.stgit@rover>

Petr Baudis <pasky@suse.cz> writes:

> On repo.or.cz (permanently I/O overloaded and hosting 1050 project +
> forks),

It looks like repo.or.cz is overwhelmed by its success. I hope that now that there are other software hosting sites with git hosting (Savannah, GitHub, Gitorious,...) the number of projects wouldn't grow as rapidly.

> the projects list (the default gitweb page) can take more than
> a minute to generate. This naive patch adds simple support for caching
> the projects list data structure so that all the projects do not need
> to get rescanned at every page access.

Another solution would be to divide projects list page into pages, perhaps adding search box for searching for a project (by name, by description and by owner).

Nevertheless even with pagination, if we want to have "sort by last update" we do need caching.

[...]

Show 8 quoted lines

> +# projects list cache for busy sites with many projects;
> +# if you set this to non-zero, it will be used as the cached
> +# index lifetime in minutes
> +# the cached list version is stored in /tmp and can be tweaked
> +# by other scripts running with the same uid as gitweb - use this
> +# only at secure installations; only single gitweb project root per
> +# system is supported!
> +our $projlist_cache_lifetime = 0;

[...]

> +sub git_project_list_body {

[...]

Show 30 quoted lines

> +	my $cache_file = '/tmp/gitweb.index.cache';
> +	use File::stat;
> +
> +	my @projects;
> +	my $stale = 0;
> +	if ($cache_lifetime and -f $cache_file
> +	    and stat($cache_file)->mtime + $cache_lifetime * 60 > time()
> +	    and open (my $fd, $cache_file)) {
> +		$stale = time() - stat($cache_file)->mtime;
> +		my @dump = <$fd>;
> +		close $fd;
> +		# Hack zone start
> +		my $VAR1;
> +		eval join("\n", @dump);
> +		@projects = @$VAR1;
> +		# Hack zone end
> +	} else {
> +		if ($cache_lifetime and -f $cache_file) {
> +			# Postpone timeout by two minutes so that we get
> +			# enough time to do our job.
> +			my $time = time() - $cache_lifetime + 120;
> +			utime $time, $time, $cache_file;
> +		}
> +		@projects = git_get_projects_details($projlist, $check_forks);
> +		if ($cache_lifetime and open (my $fd, '>'.$cache_file)) {
> +			use Data::Dumper;
> +			print $fd Dumper(\@projects);
> +			close $fd;
> +		}
> +	}

This could be much simplified with perl-cache (perl-Cache-Cache). Unfortunately this is non-standard module, not distributed (yet?) with Perl.

Warning: not tested in gitweb!

+ use Cache::FileCache; + + my $cache; + my $projects; + + if ($cache_lifetime) { + $cache = new Cache::FileCache( + { namespace => 'gitweb', + default_expires_in => $cache_lifetime + }); + $projects = $cache->get('projects_list'); + } + if (!defined $projects) { + $projects = [ git_get_projects_details($projlist, $check_forks); ]; + $cache->set('projects_list', $projects) + if defined $cache; + }

-- 
Jakub Narebski
Poland
ShadeHawk on #git