{"thread":{"id":"25942","subject":"[PATCHv7.4 0/4] Gitweb caching v7.2","startedAt":"2010-12-03T19:25:31Z","lastAt":"2010-12-03T19:30:22Z","messageCount":6,"participants":["Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"157239","messageId":"1291404335-25541-1-git-send-email-jnareb@gmail.com","threadId":"25942","inReplyTo":null,"subject":"[PATCHv7.4 0/4] Gitweb caching v7.2","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-03T19:25:31Z","receivedAt":"2010-12-03T19:25:31Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"This series is a bit fixed up and a tiny bit cleaned up version of\n\"Gitweb caching v7\" series from John 'Warthog9' Hawley:\n http://thread.gmane.org/gmane.comp.version-control.git/160147\n\nThis version of this series is based on top of 'master'\n\n\nCompared to initial v7.1 fixup of \"Gitweb caching v7\" series it\ncontains fix to \"gitweb: Prepare for splitting gitweb\" preventing\ninstallation and fix to \"gitweb: File based caching layer (from\ngit.kernel.org)\" that generated problems with t9501 test, as noticed\nby Thomas Rast in\n http://article.gmane.org/gmane.comp.version-control.git/162787\n http://marc.info/?l=git&m=129137257619613&w=2\n\nIt also includes trivial fixup in \"gitweb: Minimal testing of gitweb\ncaching\"... but still those new tests don't pass, at least for me.\nJohn Hawley is (according to information from #git IRC channel) is\nworking on \"Gitweb caching v8\"; I hope that it is fixed there.\n\nShortlog:\n~~~~~~~~~\nJakub Narebski (2):\n gitweb: Prepare for splitting gitweb\n gitweb: Minimal testing of gitweb caching\n\nJohn 'Warthog9' Hawley (2):\n gitweb: add output buffering and associated functions\n gitweb: File based caching layer (from git.kernel.org)\n\nTable of contents:\n~~~~~~~~~~~~~~~~~~\n [PATCHv7.4 1/4] gitweb: Prepare for splitting gitweb\n [PATCHv7.4 2/4] gitweb: add output buffering and associated functions\n [PATCHv7.4 3/4] gitweb: File based caching layer (from git.kernel.org)\n [PATCHv7.4 4/4] gitweb: Minimal testing of gitweb caching\n\nDiffstat:\n~~~~~~~~~\n gitweb/Makefile | 21 ++-\n gitweb/gitweb.perl | 136 +++++++++++-\n gitweb/lib/cache.pl | 348 +++++++++++++++++++++++++++++\n gitweb/static/gitweb.css | 6 +\n t/gitweb-lib.sh | 15 ++\n t/t9500-gitweb-standalone-no-errors.sh | 20 ++\n t/t9501-gitweb-standalone-http-status.sh | 13 +\n t/t9502-gitweb-standalone-parse-output.sh | 33 +++\n 8 files changed, 581 insertions(+), 11 deletions(-)\n create mode 100644 gitweb/lib/cache.pl\n\n-- \n1.7.3\n"},{"id":"157241","messageId":"1291404335-25541-2-git-send-email-jnareb@gmail.com","threadId":"25942","inReplyTo":"1291404335-25541-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv7.4 1/4] gitweb: Prepare for splitting gitweb","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-03T19:25:32Z","receivedAt":"2010-12-03T19:25:32Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Prepare gitweb for having been split into modules that are to be\ninstalled alongside gitweb in 'lib/' subdirectory, by adding\n\n use lib __DIR__.'/lib';\n\nto gitweb.perl (to main gitweb script), and preparing for putting\nmodules (relative path) in $(GITWEB_MODULES) in gitweb/Makefile.\n\nThis preparatory work allows to add new module to gitweb by simply\nadding\n\n GITWEB_MODULES += \n\nto gitweb/Makefile (assuming that the module is in 'gitweb/lib/'\ndirectory).\n\nWhile at it pass GITWEBLIBDIR in addition to GITWEB_TEST_INSTALLED to\nallow testing installed version of gitweb and installed version of\nmodules (for future tests which would check individual (sub)modules).\n\n\nUsing __DIR__ from Dir::Self module (not in core, that's why currently\ngitweb includes excerpt of code from Dir::Self defining __DIR__) was\nchosen over using FindBin-based solution (in core since perl 5.00307,\nwhile gitweb itself requires at least perl 5.8.0) because FindBin uses\nBEGIN block, which is a problem under mod_perl and other persistent\nPerl environments (thought there are workarounds).\n\nAt Pavan Kumar Sankara suggestion gitweb/Makefile uses\n\n install [OPTION]... SOURCE... DIRECTORY\n\nformat (2nd format) with single SOURCE rather than\n\n install [OPTION]... SOURCE DEST\n\nformat (1st format) because of security reasons (race conditions).\nModern GNU install has `-T' / `--no-target-directory' option, but we\ncannot rely that the $(INSTALL) we are using supports this option.\n\nThe install-modules target in gitweb/Makefile uses shell 'for' loop,\ninstead of make's $(foreach) function, to avoid possible problem with\ngenerating a command line that exceeded the maximum argument list\nlength.\n\nSigned-off-by: Jakub Narebski \n---\nBUGFIX: shell variables should not be in single quotes\n\nI am very sorry about this bug...\n\n\nThis version is identical to \n [PATCHv7.3 1/4 (bugfix)] gitweb: Prepare for splitting gitweb\n Message-Id: <201012021117.16183.jnareb@gmail.com>\n http://article.gmane.org/gmane.comp.version-control.git/162690\n\nsent as a separate patch.\n\n gitweb/Makefile | 18 +++++++++++++++---\n gitweb/gitweb.perl | 8 ++++++++\n 2 files changed, 23 insertions(+), 3 deletions(-)\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex 0a6ac00..ad5a282 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -57,6 +57,7 @@ PERL_PATH ?= /usr/bin/perl\n bindir_SQ = $(subst ','\\'',$(bindir))#'\n gitwebdir_SQ = $(subst ','\\'',$(gitwebdir))#'\n gitwebstaticdir_SQ = $(subst ','\\'',$(gitwebdir)/static)#'\n+gitweblibdir_SQ = $(subst ','\\'',$(gitwebdir)/lib)#'\n SHELL_PATH_SQ = $(subst ','\\'',$(SHELL_PATH))#'\n PERL_PATH_SQ = $(subst ','\\'',$(PERL_PATH))#'\n DESTDIR_SQ = $(subst ','\\'',$(DESTDIR))#'\n@@ -153,20 +154,31 @@ test:\n \n test-installed:\n \tGITWEB_TEST_INSTALLED='$(DESTDIR_SQ)$(gitwebdir_SQ)' \\\n+\tGITWEBLIBDIR='$(DESTDIR_SQ)$(gitweblibdir_SQ)' \\\n \t\t$(MAKE) -C ../t gitweb-test\n \n ### Installation rules\n \n-install: all\n+install: all install-modules\n \t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebdir_SQ)'\n \t$(INSTALL) -m 755 $(GITWEB_PROGRAMS) '$(DESTDIR_SQ)$(gitwebdir_SQ)'\n \t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n \t$(INSTALL) -m 644 $(GITWEB_FILES) '$(DESTDIR_SQ)$(gitwebstaticdir_SQ)'\n \n+install-modules:\n+\tinstall_dirs=\"$(sort $(dir $(GITWEB_MODULES)))\" && \\\n+\tfor dir in $$install_dirs; do \\\n+\t\ttest -d '$(DESTDIR_SQ)$(gitweblibdir_SQ)'/\"$$dir\" || \\\n+\t\t$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(gitweblibdir_SQ)'/\"$$dir\"; \\\n+\tdone\n+\tgitweb_modules=\"$(GITWEB_MODULES)\" && \\\n+\tfor mod in $$gitweb_modules; do \\\n+\t\t$(INSTALL) -m 644 \"lib/$$mod\" '$(DESTDIR_SQ)$(gitweblibdir_SQ)'/\"$$(dirname $$mod)\"; \\\n+\tdone\n+\n ### Cleaning rules\n \n clean:\n \t$(RM) gitweb.cgi static/gitweb.min.js static/gitweb.min.css GITWEB-BUILD-OPTIONS\n \n-.PHONY: all clean install test test-installed .FORCE-GIT-VERSION-FILE FORCE\n-\n+.PHONY: all clean install install-modules test test-installed .FORCE-GIT-VERSION-FILE FORCE\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 679f2da..cfa511c 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -10,6 +10,14 @@\n use 5.008;\n use strict;\n use warnings;\n+\n+use File::Spec;\n+# __DIR__ is taken from Dir::Self __DIR__ fragment\n+sub __DIR__ () {\n+\tFile::Spec->rel2abs(join '', (File::Spec->splitpath(__FILE__))[0, 1]);\n+}\n+use lib __DIR__ . '/lib';\n+\n use CGI qw(:standard :escapeHTML -nosticky);\n use CGI::Util qw(unescape);\n use CGI::Carp qw(fatalsToBrowser set_message);\n-- \n1.7.3\n"},{"id":"157240","messageId":"1291404335-25541-3-git-send-email-jnareb@gmail.com","threadId":"25942","inReplyTo":"1291404335-25541-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv7.4 2/4] gitweb: add output buffering and associated functions","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-03T19:25:33Z","receivedAt":"2010-12-03T19:25:33Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"From: John 'Warthog9' Hawley \n\nThis adds output buffering for gitweb, mainly in preparation for\ncaching support. This is a dramatic change to how caching was being\ndone, mainly in passing around the variable manually and such.\n\nThis centrally flips the entire STDOUT to a variable, which after the\ncompletion of the run, flips it back and does a print on the resulting\ndata.\n\nThis should save on the previous 10K line patch (or so) that adds more\nexplicit output passing.\n\n[jn: modified reset_output to silence 'gitweb.perl: Name \"main::STDOUT_REAL\"\n used only once: possible typo at ../gitweb/gitweb.perl line 1130.' warning]\n\nSigned-off-by: Junio C Hamano \n---\nReminder: it uses\n\n open STDOUT, \">&\", \\*STDOUT_REAL;\n\nrather than\n\n open(STDOUT,\">&STDOUT_REAL\");\n\nto silence spurious (invalid) warning\n\n gitweb.perl: Name \"main::STDOUT_REAL\" used only once: possible typo\n at ../gitweb/gitweb.perl line 1130.\n\n\nThis version is identical to \n [PATCHv7.1 2/4] gitweb: add output buffering and associated functions\n Message-Id: <1288607092-31458-3-git-send-email-jnareb@gmail.com>\n http://article.gmane.org/gmane.comp.version-control.git/160475\n\n gitweb/gitweb.perl | 29 +++++++++++++++++++++++++++++\n 1 files changed, 29 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex cfa511c..cae0e34 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -39,6 +39,9 @@ BEGIN {\n \n our $version = \"++GIT_VERSION++\";\n \n+# Output buffer variable\n+our $output = \"\";\n+\n our ($my_url, $my_uri, $base_url, $path_info, $home_link);\n sub evaluate_uri {\n \tour $cgi;\n@@ -1134,6 +1137,25 @@ sub evaluate_argv {\n \t);\n }\n \n+sub change_output {\n+\tour $output;\n+\n+\t# Trap the 'proper' STDOUT to STDOUT_REAL for things like error messages and such\n+\topen(STDOUT_REAL,\">&STDOUT\") or die \"Unable to capture STDOUT $!\\n\";\n+\n+\t# Close STDOUT, so that it isn't being used anymore.\n+\tclose STDOUT;\n+\n+\t# Trap STDOUT to the $output variable, which is what I was using in the original\n+\t# patch anyway.\n+\topen(STDOUT,\">\", \\$output) || die \"Unable to open STDOUT: $!\"; #open STDOUT handle to use $var\n+}\n+\n+sub reset_output {\n+\t# This basically takes STDOUT_REAL and puts it back as STDOUT\n+\topen STDOUT, \">&\", \\*STDOUT_REAL;\n+}\n+\n sub run {\n \tevaluate_argv();\n \n@@ -1145,7 +1167,10 @@ sub run {\n \t\t$pre_dispatch_hook->()\n \t\t\tif $pre_dispatch_hook;\n \n+\t\tchange_output();\n \t\trun_request();\n+\t\treset_output();\n+\t\tprint $output;\n \n \t\t$post_dispatch_hook->()\n \t\t\tif $post_dispatch_hook;\n@@ -3655,6 +3680,10 @@ sub die_error {\n \t\t500 => '500 Internal Server Error',\n \t\t503 => '503 Service Unavailable',\n \t);\n+\t# Reset the output so that we are actually going to STDOUT as opposed\n+\t# to buffering the output.\n+\treset_output();\n+\n \tgit_header_html($http_responses{$status}, undef, %opts);\n \tprint <\n-- \n1.7.3\n"},{"id":"157243","messageId":"1291404335-25541-4-git-send-email-jnareb@gmail.com","threadId":"25942","inReplyTo":"1291404335-25541-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv7.4 3/4] gitweb: File based caching layer (from git.kernel.org)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-03T19:25:34Z","receivedAt":"2010-12-03T19:25:34Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"From: John 'Warthog9' Hawley \n\nThis is a relatively large patch that implements the file based\ncaching layer that is quite similar to the one used on such large\nsites as kernel.org and soon git.fedoraproject.org. This provides\na simple, and straight forward caching mechanism that scales\ndramatically better than Gitweb by itself.\n\nThe caching layer basically buffers the output that Gitweb would\nnormally return, and saves that output to a cache file on the local\ndisk. When the file is requested it attempts to gain a shared lock\non the cache file and cat it out to the client. Should an exclusive\nlock be on a file (it's being updated) the code has a choice to either\nupdate in the background and go ahead and show the stale page while\nupdate is being performed, or stall the client(s) until the page\nis generated.\n\nThere are two forms of stalling involved here, background building\nand non-background building, both of which are discussed in the\nconfiguration page.\n\nThere are still a few known \"issues\" with respect to this:\n- Code needs to be added to be \"browser\" aware so\n that clients like wget that are trying to get a\n binary blob don't obtain a \"Generating...\" page\n\nCaching is disabled by default. You can turn it on by setting\n$caching_enabled variable to true to enable file based caching.\n\n[jn: added error checking to loading 'cache.pl'; moved check\n for $caching_enabled outside out of cache_fetch, which required\n update to die_error()]\n\nSigned-off-by: Junio C Hamano \n---\nThis fixes a bug that generated problems with t9501 test, as noticed\nby Thomas Rast in\n http://article.gmane.org/gmane.comp.version-control.git/162787\n http://marc.info/?l=git&m=129137257619613&w=2\n\nFixup was sent as interdiff in\n http://marc.info/?l=git&m=129138558404482&w=2\n\nand it was tested that it fixes problem by Thomas Rast... so perhaps\nit should be\n\nNoticed-and-tested-by: Thomas Rast \n\n gitweb/Makefile | 3 +\n gitweb/gitweb.perl | 107 +++++++++++++--\n gitweb/lib/cache.pl | 348 ++++++++++++++++++++++++++++++++++++++++++++++\n gitweb/static/gitweb.css | 6 +\n 4 files changed, 452 insertions(+), 12 deletions(-)\n create mode 100644 gitweb/lib/cache.pl\n\ndiff --git a/gitweb/Makefile b/gitweb/Makefile\nindex ad5a282..1f892c3 100644\n--- a/gitweb/Makefile\n+++ b/gitweb/Makefile\n@@ -113,6 +113,9 @@ endif\n \n GITWEB_FILES += static/git-logo.png static/git-favicon.png\n \n+# Gitweb caching\n+GITWEB_MODULES += cache.pl\n+\n GITWEB_REPLACE = \\\n \t-e 's|++GIT_VERSION++|$(GIT_VERSION)|g' \\\n \t-e 's|++GIT_BINDIR++|$(bindir)|g' \\\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex cae0e34..c465aa6 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -250,6 +250,53 @@ our %avatar_size = (\n # Leave it undefined (or set to 'undef') to turn off load checking.\n our $maxload = 300;\n \n+# This enables/disables the caching layer in gitweb. This currently only supports the\n+# 'dumb' file based caching layer, primarily used on git.kernel.org. this is reasonably\n+# effective but it has the downside of requiring a huge amount of disk space if there\n+# are a number of repositories involved. It is not uncommon for git.kernel.org to have\n+# on the order of 80G - 120G accumulate over the course of a few months. It is recommended\n+# that the cache directory be periodically completely deleted, and this is safe to perform.\n+# Suggested mechanism\n+# mv $cacheidr $cachedir.flush;mkdir $cachedir;rm -rf $cachedir.flush\n+our $caching_enabled = 0;\n+\n+# Used to set the minimum cache timeout for the dynamic caching algorithm. Basically\n+# if we calculate the cache to be under this number of seconds we set the cache timeout\n+# to this minimum.\n+# Value is in seconds. 1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour\n+our $minCacheTime = 20;\n+\n+# Used to set the maximum cache timeout for the dynamic caching algorithm. Basically\n+# if we calculate the cache to exceed this number of seconds we set the cache timeout\n+# to this maximum.\n+# Value is in seconds. 1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour\n+our $maxCacheTime = 1200;\n+\n+# If you need to change the location of the caching directory, override this\n+# otherwise this will probably do fine for you\n+our $cachedir = 'cache';\n+\n+# If this is set (to 1) cache will do it's best to always display something instead\n+# of making someone wait for the cache to update. This will launch the cacheUpdate\n+# into the background and it will lock a .bg file and will only lock the\n+# actual cache file when it needs to write into it. In theory this will make\n+# gitweb seem more responsive at the price of possibly stale data.\n+our $backgroundCache = 1;\n+\n+# Used to set the maximum cache file life. If a cache files last modify time exceeds\n+# this value, it will assume that the data is just too old, and HAS to be regenerated\n+# instead of trying to display the existing cache data.\n+# Value is in seconds. 1 = 1 seconds, 60 = 1 minute, 600 = 10 minutes, 3600 = 1 hour\n+# 18000 = 5 hours\n+our $maxCacheLife = 18000;\n+\n+# Used to enable or disable background forking of the gitweb caching. Mainly here for debugging purposes\n+our $cacheDoFork = 1;\n+\n+our $fullhashpath = *STDOUT;\n+our $fullhashbinpath = *STDOUT;\n+our $fullhashbinpathfinal = *STDOUT;\n+\n # configuration for 'highlight' (http://www.andre-simon.de/)\n # match by basename\n our %highlight_basename = (\n@@ -506,6 +553,15 @@ our %feature = (\n \t\t'default' => [0]},\n );\n \n+#\n+# Includes\n+#\n+if (!exists $INC{'cache.pl'}) {\n+\tmy $return = do 'cache.pl';\n+\tdie $@ if $@;\n+\tdie \"Couldn't read 'cache.pl': $!\" if (!defined $return);\n+}\n+\n sub gitweb_get_feature {\n \tmy ($name) = @_;\n \treturn unless exists $feature{$name};\n@@ -734,6 +790,10 @@ our %actions = (\n \t\"project_list\" => \\&git_project_list,\n \t\"project_index\" => \\&git_project_index,\n );\n+sub is_cacheable {\n+\tmy $action = shift;\n+\treturn !($action eq 'blame_data' || $action eq 'blame_incremental');\n+}\n \n # finally, we have the hash of allowed extra_options for the commands that\n # allow them\n@@ -1072,7 +1132,11 @@ sub dispatch {\n \t !$project) {\n \t\tdie_error(400, \"Project needed\");\n \t}\n-\t$actions{$action}->();\n+\tif ($caching_enabled && is_cacheable($action)) {\n+\t\tcache_fetch($action);\n+\t} else {\n+\t\t$actions{$action}->();\n+\t}\n }\n \n sub reset_timer {\n@@ -1142,6 +1206,7 @@ sub change_output {\n \n \t# Trap the 'proper' STDOUT to STDOUT_REAL for things like error messages and such\n \topen(STDOUT_REAL,\">&STDOUT\") or die \"Unable to capture STDOUT $!\\n\";\n+\tprint STDOUT_REAL \"\";\n \n \t# Close STDOUT, so that it isn't being used anymore.\n \tclose STDOUT;\n@@ -1167,10 +1232,7 @@ sub run {\n \t\t$pre_dispatch_hook->()\n \t\t\tif $pre_dispatch_hook;\n \n-\t\tchange_output();\n \t\trun_request();\n-\t\treset_output();\n-\t\tprint $output;\n \n \t\t$post_dispatch_hook->()\n \t\t\tif $post_dispatch_hook;\n@@ -3447,7 +3509,8 @@ sub git_header_html {\n \t# support xhtml+xml but choking when it gets what it asked for.\n \tif (defined $cgi->http('HTTP_ACCEPT') &&\n \t $cgi->http('HTTP_ACCEPT') =~ m/(,|;|\\s|^)application\\/xhtml\\+xml(,|;|\\s|$)/ &&\n-\t $cgi->Accept('application/xhtml+xml') != 0) {\n+\t $cgi->Accept('application/xhtml+xml') != 0 &&\n+\t !$caching_enabled) {\n \t\t$content_type = 'application/xhtml+xml';\n \t} else {\n \t\t$content_type = 'text/html';\n@@ -3592,6 +3655,7 @@ sub git_footer_html {\n \tmy $feed_class = 'rss_logo';\n \n \tprint \"
\\n\";\n+\tprint \"
Cache Last Updated: \". gmtime( time ) .\" GMT
\\n\";\n \tif (defined $project) {\n \t\tmy $descr = git_get_project_description($project);\n \t\tif (defined $descr) {\n@@ -3680,9 +3744,14 @@ sub die_error {\n \t\t500 => '500 Internal Server Error',\n \t\t503 => '503 Service Unavailable',\n \t);\n+\t# The output handlers for die_error need to be reset to STDOUT\n+\t# so that half the message isn't being output to random and\n+\t# half to STDOUT as expected. This is mainly for the benefit\n+\t# of using git_header_html() and git_footer_html() since\n+\t#\n \t# Reset the output so that we are actually going to STDOUT as opposed\n \t# to buffering the output.\n-\treset_output();\n+\treset_output() if ($caching_enabled);\n \n \tgit_header_html($http_responses{$status}, undef, %opts);\n \tprint <;\n-\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n+\tif ($caching_enabled) {\n+\t\topen BINOUT, '>', $fullhashbinpath or die_error(500, \"Could not open bin dump file\");\n+\t\tbinmode BINOUT, ':raw';\n+\t\tprint BINOUT <$fd>;\n+\t\tclose BINOUT;\n+\t} else {\n+\t\tbinmode STDOUT, ':raw';\n+\t\tprint <$fd>;\n+\t\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n+\t}\n \tclose $fd;\n }\n \n@@ -5879,9 +5955,16 @@ sub git_snapshot {\n \n \topen my $fd, \"-|\", $cmd\n \t\tor die_error(500, \"Execute git-archive failed\");\n-\tbinmode STDOUT, ':raw';\n-\tprint <$fd>;\n-\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n+\tif ($caching_enabled) {\n+\t\topen BINOUT, '>', $fullhashbinpath or die_error(500, \"Could not open bin dump file\");\n+\t\tbinmode BINOUT, ':raw';\n+\t\tprint BINOUT <$fd>;\n+\t\tclose BINOUT;\n+\t} else {\n+\t\tbinmode STDOUT, ':raw';\n+\t\tprint <$fd>;\n+\t\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n+\t}\n \tclose $fd;\n }\n \ndiff --git a/gitweb/lib/cache.pl b/gitweb/lib/cache.pl\nnew file mode 100644\nindex 0000000..4360b3d\n--- /dev/null\n+++ b/gitweb/lib/cache.pl\n@@ -0,0 +1,348 @@\n+# gitweb - simple web interface to track changes in git repositories\n+#\n+# (C) 2006, John 'Warthog9' Hawley \n+#\n+# This program is licensed under the GPLv2\n+\n+#\n+# Gitweb caching engine\n+#\n+\n+#use File::Path qw(make_path remove_tree);\n+use File::Path qw(mkpath rmtree); # Used for compatability reasons\n+use Digest::MD5 qw(md5 md5_hex md5_base64);\n+use Fcntl ':flock';\n+use File::Copy;\n+\n+sub cache_fetch {\n+\tmy ($action) = @_;\n+\tmy $cacheTime = 0;\n+\n+\tif(! -d $cachedir){\n+\t\tprint \"*** Warning ***: Caching enabled but cache directory does not exsist. ($cachedir)\\n\";\n+\t\tmkdir (\"cache\", 0755) || die \"Cannot create cache dir - you will need to manually create\";\n+\t\tprint \"Cache directory created successfully\\n\";\n+\t}\n+\n+\tour $full_url = \"$my_url?\". $ENV{'QUERY_STRING'};\n+\tour $urlhash = md5_hex($full_url);\n+\tour $fullhashdir = \"$cachedir/\". substr( $urlhash, 0, 2) .\"/\";\n+\n+\teval { mkpath( $fullhashdir, 0, 0777 ) };\n+\tif ($@) {\n+\t\tdie_error(500, \"Internal Server Error\", \"Could not create cache directory: $@\");\n+\t}\n+\t$fullhashpath = \"$fullhashdir/\". substr( $urlhash, 2 );\n+\t$fullhashbinpath = \"$fullhashpath.bin.wt\";\n+\t$fullhashbinpathfinal = \"$fullhashpath.bin\";\n+\n+\tif(! -e \"$fullhashpath\" ){\n+\t\tif(! $cacheDoFork || ! defined(my $childPid = fork()) ){\n+\t\t\tcacheUpdate($action,0);\n+\t\t\tcacheDisplay($action);\n+\t\t} elsif ( $childPid == 0 ){\n+\t\t\t#run the updater\n+\t\t\tcacheUpdate($action,1);\n+\t\t}else{\n+\t\t\tcacheWaitForUpdate($action);\n+\t\t}\n+\t}else{\n+\t\t#if cache is out dated, update\n+\t\t#else displayCache();\n+\t\topen(cacheFile, '<', \"$fullhashpath\");\n+\t\tstat(cacheFile);\n+\t\tclose(cacheFile);\n+\t\tmy $stat_time = (stat(_))[9];\n+\t\tmy $stat_size = (stat(_))[7];\n+\n+\t\t$cacheTime = get_loadavg() * 60;\n+\t\tif( $cacheTime > $maxCacheTime ){\n+\t\t\t$cacheTime = $maxCacheTime;\n+\t\t}\n+\t\tif( $cacheTime < $minCacheTime ){\n+\t\t\t$cacheTime = $minCacheTime;\n+\t\t}\n+\t\tif( $stat_time < (time - $cacheTime) || $stat_size == 0 ){\n+\t\t\tif( ! $cacheDoFork || ! defined(my $childPid = fork()) ){\n+\t\t\t\tcacheUpdate($action,0);\n+\t\t\t\tcacheDisplay($action);\n+\t\t\t} elsif ( $childPid == 0 ){\n+\t\t\t\t#run the updater\n+\t\t\t\t#print \"Running updater\\n\";\n+\t\t\t\tcacheUpdate($action,1);\n+\t\t\t}else{\n+\t\t\t\t#print \"Waiting for update\\n\";\n+\t\t\t\tcacheWaitForUpdate($action);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tcacheDisplay($action);\n+\t\t}\n+\n+\n+\t}\n+\n+\t#\n+\t# If all of the caching failes - lets go ahead and press on without it and fall back to 'default'\n+\t# non-caching behavior. This is the softest of the failure conditions.\n+\t#\n+\t#$actions{$action}->();\n+}\n+\n+sub cacheUpdate {\n+\tmy ($action,$areForked) = @_;\n+\tmy $lockingStatus;\n+\tmy $fileData = \"\";\n+\n+\tif($backgroundCache){\n+\t\topen(cacheFileBG, '>:utf8', \"$fullhashpath.bg\");\n+\t\tmy $lockStatBG = flock(cacheFileBG,LOCK_EX|LOCK_NB);\n+\n+\t\t$lockStatus = $lockStatBG;\n+\t}else{\n+\t\topen(cacheFile, '>:utf8', $fullhashpath);\n+\t\tmy $lockStat = flock(cacheFile,LOCK_EX|LOCK_NB);\n+\n+\t\t$lockStatus = $lockStat;\n+\t}\n+\t#print \"lock status: $lockStat\\n\";\n+\n+\n+\tif (! $lockStatus ){\n+\t\tif ( $areForked ){\n+\t\t\texit(0);\n+\t\t}else{\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tif(\n+\t\t$action eq \"snapshot\"\n+\t\t||\n+\t\t$action eq \"blob_plain\"\n+\t){\n+\t\tmy $openstat = open(cacheFileBinWT, '>>:utf8', \"$fullhashbinpath\");\n+\t\tmy $lockStatBin = flock(cacheFileBinWT,LOCK_EX|LOCK_NB);\n+\t}\n+\n+\t# Trap all output from the action\n+\tchange_output();\n+\n+\t$actions{$action}->();\n+\n+\t# Reset the outputs as we should be fine now\n+\treset_output();\n+\n+\n+\tif($backgroundCache){\n+\t\topen(cacheFile, '>:utf8', \"$fullhashpath\");\n+\t\t$lockStat = flock(cacheFile,LOCK_EX);\n+\n+\t\tif (! $lockStat ){\n+\t\t\tif ( $areForked ){\n+\t\t\t\texit(0);\n+\t\t\t}else{\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif(\n+\t\t$action eq \"snapshot\"\n+\t\t||\n+\t\t$action eq \"blob_plain\"\n+\t){\n+\t\tmy $openstat = open(cacheFileBinFINAL, '>:utf8', \"$fullhashbinpathfinal\");\n+\t\t$lockStatBIN = flock(cacheFileBinFINAL,LOCK_EX);\n+\n+\t\tif (! $lockStatBIN ){\n+\t\t\tif ( $areForked ){\n+\t\t\t\texit(0);\n+\t\t\t}else{\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\t# Actually dump the output to the proper file handler\n+\tlocal $/ = undef;\n+\t$|++;\n+\tprint cacheFile \"$output\";\n+\t$|--;\n+\tif(\n+\t\t$action eq \"snapshot\"\n+\t\t||\n+\t\t$action eq \"blob_plain\"\n+\t){\n+\t\tmove(\"$fullhashbinpath\", \"$fullhashbinpathfinal\") or die \"Binary Cache file could not be updated: $!\";\n+\n+\t\tflock(cacheFileBinFINAL,LOCK_UN);\n+\t\tclose(cacheFileBinFINAL);\n+\n+\t\tflock(cacheFileBinWT,LOCK_UN);\n+\t\tclose(cacheFileBinWT);\n+\t}\n+\n+\tflock(cacheFile,LOCK_UN);\n+\tclose(cacheFile);\n+\n+\tif($backgroundCache){\n+\t\tflock(cacheFileBG,LOCK_UN);\n+\t\tclose(cacheFileBG);\n+\t}\n+\n+\tif ( $areForked ){\n+\t\texit(0);\n+\t} else {\n+\t\treturn;\n+\t}\n+}\n+\n+\n+sub cacheWaitForUpdate {\n+\tmy ($action) = @_;\n+\tmy $x = 0;\n+\tmy $max = 10;\n+\tmy $lockStat = 0;\n+\n+\tif( $backgroundCache ){\n+\t\tif( -e \"$fullhashpath\" ){\n+\t\t\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n+\t\t\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n+\t\t\tstat(cacheFile);\n+\t\t\tclose(cacheFile);\n+\n+\t\t\tif( $lockStat && ( (stat(_))[9] > (time - $maxCacheLife) ) ){\n+\t\t\t\tcacheDisplay($action);\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif(\n+\t\t$action eq \"atom\"\n+\t\t||\n+\t\t$action eq \"rss\"\n+\t\t||\n+\t\t$action eq \"opml\"\n+\t){\n+\t\tdo {\n+\t\t\tsleep 2 if $x > 0;\n+\t\t\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n+\t\t\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n+\t\t\tclose(cacheFile);\n+\t\t\t$x++;\n+\t\t\t$combinedLockStat = $lockStat;\n+\t\t} while ((! $combinedLockStat) && ($x < $max));\n+\n+\t\tif( $x != $max ){\n+\t\t\tcacheDisplay($action);\n+\t\t}\n+\t\treturn;\n+\t}\n+\n+\t$| = 1;\n+\n+\tprint $::cgi->header(\n+\t\t\t\t-type=>'text/html',\n+\t\t\t\t-charset => 'utf-8',\n+\t\t\t\t-status=> 200,\n+\t\t\t\t-expires => 'now',\n+\t\t\t\t# HTTP/1.0\n+\t\t\t\t-Pragma => 'no-cache',\n+\t\t\t\t# HTTP/1.1\n+\t\t\t\t-Cache_Control => join(\n+\t\t\t\t\t\t\t', ',\n+\t\t\t\t\t\t\tqw(\n+\t\t\t\t\t\t\t\tprivate\n+\t\t\t\t\t\t\t\tno-cache\n+\t\t\t\t\t\t\t\tno-store\n+\t\t\t\t\t\t\t\tmust-revalidate\n+\t\t\t\t\t\t\t\tmax-age=0\n+\t\t\t\t\t\t\t\tpre-check=0\n+\t\t\t\t\t\t\t\tpost-check=0\n+\t\t\t\t\t\t\t)\n+\t\t\t\t\t\t)\n+\t\t\t\t);\n+\n+\tprint <\n+\n+\n+\n+\n+\n+\n+\n+$title\n+\n+\n+EOF\n+\n+\tprint \"Generating..\";\n+\tdo {\n+\t\tprint \".\";\n+\t\tsleep 2 if $x > 0;\n+\t\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n+\t\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n+\t\tclose(cacheFile);\n+\t\t$x++;\n+\t\t$combinedLockStat = $lockStat;\n+\t} while ((! $combinedLockStat) && ($x < $max));\n+\tprint <\n+\n+EOF\n+\treturn;\n+}\n+\n+sub cacheDisplay {\n+\tlocal $/ = undef;\n+\t$|++;\n+\n+\tmy ($action) = @_;\n+\topen(cacheFile, '<:utf8', \"$fullhashpath\");\n+\t$lockStat = flock(cacheFile,LOCK_SH|LOCK_NB);\n+\n+\tif (! $lockStat ){\n+\t\tclose(cacheFile);\n+\t\tcacheWaitForUpdate($action);\n+\t}\n+\n+\tif(\n+\t\t(\n+\t\t\t$action eq \"snapshot\"\n+\t\t\t||\n+\t\t\t$action eq \"blob_plain\"\n+\t\t)\n+\t){\n+\t\tmy $openstat = open(cacheFileBin, '<', \"$fullhashbinpathfinal\");\n+\t\t$lockStatBIN = flock(cacheFileBin,LOCK_SH|LOCK_NB);\n+\t\tif (! $lockStatBIN ){\n+\t\t\tsystem (\"echo 'cacheDisplay - bailing due to binary lock failure' >> /tmp/gitweb.log\");\n+\t\t\tclose(cacheFile);\n+\t\t\tclose(cacheFileBin);\n+\t\t\tcacheWaitForUpdate($action);\n+\t\t}\n+\n+\t\tmy $binfilesize = -s \"$fullhashbinpathfinal\";\n+\t\tprint \"Content-Length: $binfilesize\";\n+\t}\n+\twhile( ){\n+\t\tprint $_;\n+\t}\n+\tif(\n+\t\t$action eq \"snapshot\"\n+\t\t||\n+\t\t$action eq \"blob_plain\"\n+\t){\n+\t\tbinmode STDOUT, ':raw';\n+\t\tprint ;\n+\t\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n+\t\tclose(cacheFileBin);\n+\t}\n+\tclose(cacheFile);\n+\t$|--;\n+}\n+\n+1;\n+__END__\ndiff --git a/gitweb/static/gitweb.css b/gitweb/static/gitweb.css\nindex 4132aab..972d32e 100644\n--- a/gitweb/static/gitweb.css\n+++ b/gitweb/static/gitweb.css\n@@ -67,6 +67,12 @@ div.page_path {\n \tborder-width: 0px 0px 1px;\n }\n \n+div.cachetime {\n+\tfloat: left;\n+\tmargin-right: 10px;\n+\tcolor: #555555;\n+}\n+\n div.page_footer {\n \theight: 17px;\n \tpadding: 4px 8px;\n-- \n1.7.3\n"},{"id":"157242","messageId":"1291404335-25541-5-git-send-email-jnareb@gmail.com","threadId":"25942","inReplyTo":"1291404335-25541-1-git-send-email-jnareb@gmail.com","subject":"[PATCHv7.4 4/4] gitweb: Minimal testing of gitweb caching","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-03T19:25:35Z","receivedAt":"2010-12-03T19:25:35Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Add basic tests of caching support to t9500-gitweb-standalone-no-errors\ntest: set $caching_enabled to true and check for errors for first time\nrun (generating cache) and second time run (retrieving from cache) for a\nsingle view - summary view for a project. Check also that request for\nnon-existent object (which results in die_error() codepath to be called)\ndoesn't produce errors.\n\nCheck in t9501-gitweb-standalone-http-status that request for\nnon-existent object produces correct output (HTTP headers and HTML\noutput) also when caching is enabled.\n\nCheck in the t9502-gitweb-standalone-parse-output test that gitweb\nproduces the same output with and without caching, for first and\nsecond run, with binary (raw) or plain text (utf8) output.\n\nThe common routine that enables cache, gitweb_enable_caching, is\ndefined in t/gitweb-lib.sh\n\nSigned-off-by: Jakub Narebski \n---\nThe minor bug in value for $xxxCacheTime value in t/gitweb-lib.sh\nnoticed by Junio C Hamano in\n http://thread.gmane.org/gmane.comp.version-control.git/160147/focus=161635\n\nis fixed here, but it doesn't make gitweb pass new tests.\n\n t/gitweb-lib.sh | 15 +++++++++++++\n t/t9500-gitweb-standalone-no-errors.sh | 20 +++++++++++++++++\n t/t9501-gitweb-standalone-http-status.sh | 13 +++++++++++\n t/t9502-gitweb-standalone-parse-output.sh | 33 +++++++++++++++++++++++++++++\n 4 files changed, 81 insertions(+), 0 deletions(-)\n\ndiff --git a/t/gitweb-lib.sh b/t/gitweb-lib.sh\nindex b9bb95f..2c2ac90 100644\n--- a/t/gitweb-lib.sh\n+++ b/t/gitweb-lib.sh\n@@ -52,6 +52,21 @@ EOF\n \texport SCRIPT_NAME\n }\n \n+gitweb_enable_caching () {\n+\ttest_expect_success 'enable caching' '\n+\t\tcat >>gitweb_config.perl <<-\\EOF &&\n+\t\tour $caching_enabled = 1;\n+\t\tour $minCacheTime = 60*60*24*7; # very long expiration time for tests (a week)\n+\t\tour $maxCacheTime = 60*60*24*30; # upper bound for dynamic (adaptive) caching\n+\t\tour $cachedir = \"cache\"; # for testsuite to clear the right thing\n+\t\t# required, because otherwise some tests might intermittently not pass\n+\t\tour $backgroundCache = 0; # should turn off cacheWaitForUpdate() / \"Generating...\"\n+\t\t#our $cacheDoFork = 0;\n+\t\tEOF\n+\t\trm -rf cache/\n+\t'\n+}\n+\n gitweb_run () {\n \tGATEWAY_INTERFACE='CGI/1.1'\n \tHTTP_ACCEPT='*/*'\ndiff --git a/t/t9500-gitweb-standalone-no-errors.sh b/t/t9500-gitweb-standalone-no-errors.sh\nindex 21cd286..86c1b50 100755\n--- a/t/t9500-gitweb-standalone-no-errors.sh\n+++ b/t/t9500-gitweb-standalone-no-errors.sh\n@@ -677,4 +677,24 @@ test_expect_success HIGHLIGHT \\\n \t gitweb_run \"p=.git;a=blob;f=test.sh\"'\n test_debug 'cat gitweb.log'\n \n+# ----------------------------------------------------------------------\n+# caching\n+\n+gitweb_enable_caching\n+\n+test_expect_success \\\n+\t'caching enabled (project summary, first run, generating cache)' \\\n+\t'gitweb_run \"p=.git;a=summary\"'\n+test_debug 'cat gitweb.log'\n+\n+test_expect_success \\\n+\t'caching enabled (project summary, second run, cached version)' \\\n+\t'gitweb_run \"p=.git;a=summary\"'\n+test_debug 'cat gitweb.log'\n+\n+test_expect_success \\\n+\t'caching enabled (non-existent commit, non-cache error page)' \\\n+\t'gitweb_run \"p=.git;a=commit;h=non-existent\"'\n+test_debug 'cat gitweb.log'\n+\n test_done\ndiff --git a/t/t9501-gitweb-standalone-http-status.sh b/t/t9501-gitweb-standalone-http-status.sh\nindex 2487da1..5b1df3f 100755\n--- a/t/t9501-gitweb-standalone-http-status.sh\n+++ b/t/t9501-gitweb-standalone-http-status.sh\n@@ -134,5 +134,18 @@ cat >>gitweb_config.perl <<\\EOF\n our $maxload = undef;\n EOF\n \n+# ----------------------------------------------------------------------\n+# output caching\n+\n+gitweb_enable_caching\n+\n+test_expect_success 'caching enabled (non-existent commit, 404 error)' '\n+\tgitweb_run \"p=.git;a=commit;h=non-existent\" &&\n+\tgrep \"Status: 404 Not Found\" gitweb.headers &&\n+\tgrep \"404 - Unknown commit object\" gitweb.body\n+'\n+test_debug 'echo \"log\" && cat gitweb.log'\n+test_debug 'echo \"headers\" && cat gitweb.headers'\n+test_debug 'echo \"body\" && cat gitweb.body'\n \n test_done\ndiff --git a/t/t9502-gitweb-standalone-parse-output.sh b/t/t9502-gitweb-standalone-parse-output.sh\nindex dd83890..bc8eb01 100755\n--- a/t/t9502-gitweb-standalone-parse-output.sh\n+++ b/t/t9502-gitweb-standalone-parse-output.sh\n@@ -112,4 +112,37 @@ test_expect_success 'snapshot: hierarchical branch name (xx/test)' '\n '\n test_debug 'cat gitweb.headers'\n \n+\n+# ----------------------------------------------------------------------\n+# whether gitweb with caching enabled produces the same output\n+\n+test_expect_success 'setup for caching tests (utf8 patch, binary file)' '\n+\t. \"$TEST_DIRECTORY\"/t3901-utf8.txt &&\n+\tcp \"$TEST_DIRECTORY\"/test9200a.png image.png &&\n+\tgit add image.png &&\n+\tgit commit -F \"$TEST_DIRECTORY\"/t3900/1-UTF-8.txt &&\n+\tgitweb_run \"p=.git;a=patch\" &&\n+\tmv gitweb.body no_cache.txt &&\n+\tgitweb_run \"p=.git;a=blob_plain;f=image.png\" &&\n+\tmv gitweb.body no_cache.png\n+'\n+\n+gitweb_enable_caching\n+\n+for desc in 'generating cache' 'cached version'; do\n+\ttest_expect_success \"caching enabled, plain text (utf8) output, $desc\" '\n+\t\tgitweb_run \"p=.git;a=patch\" &&\n+\t\tmv gitweb.body cache.txt &&\n+\t\ttest_cmp no_cache.txt cache.txt\n+\t'\n+done\n+\n+for desc in 'generating cache' 'cached version'; do\n+\ttest_expect_success \"caching enabled, binary output (raw), $desc\" '\n+\t\tgitweb_run \"p=.git;a=blob_plain;f=image.png\" &&\n+\t\tmv gitweb.body cache.png &&\n+\t\tcmp no_cache.png cache.png\n+\t'\n+done\n+\n test_done\n-- \n1.7.3\n"},{"id":"157245","messageId":"201012032030.24639.jnareb@gmail.com","threadId":"25942","inReplyTo":"1291404335-25541-1-git-send-email-jnareb@gmail.com","subject":"Re: [PATCHv7.4 0/4] Gitweb caching v7.2","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-03T19:30:22Z","receivedAt":"2010-12-03T19:30:22Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Sorry, of course subject should read:\n\n [PATCHv7.4 0/4] Gitweb caching v7.4\n\n-- \nJakub Narebski\nPoland\n"}]}