{"thread":{"id":"23422","subject":"gitweb: Simple file based output caching - current state of series","startedAt":"2010-04-11T21:17:54Z","lastAt":"2010-04-11T21:17:54Z","messageCount":1,"participants":["Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"139289","messageId":"201004112317.55240.jnareb@gmail.com","threadId":"23422","inReplyTo":null,"subject":"gitweb: Simple file based output caching - current state of series","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-04-11T21:17:54Z","receivedAt":"2010-04-11T21:17:54Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"I am still working on adding support for output (response) caching to\ngitweb, based originally on patch series by John 'Warthog9' Hawley aka J.H.\nThe idea was to start small, and add features piece by piece for easier\nreview.  As you can see the idea snowballed a bit, with 16 patches in\nthe series now (4 of those are more or less independent of caching support).\n\nCurrent version of my work on simple file based output caching in\ngitweb can be found at:\n  git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-pu\n\n  http://repo.or.cz/w/git/jnareb-git.git/log/refs/heads/gitweb/cache-kernel-pu\n\nNote that 'gitweb/cache-kernel-pu' will be undergoing rewriting!\nPrevious versions of my take on series can be found as branches\n'gitweb/cache-kernel', 'gitweb/cache-kernel-v2', 'gitweb/cache-kernel-v3'.\nJ.H. series can be found in git://git.kernel.org/pub/scm/git/warthog9/gitweb.git\nas branches 'gitweb-ml-v2', 'gitweb-ml-v3', 'gitweb-ml-v4'.\n\nShortlog:\n~~~~~~~~~\n   1. Export more test-related variables when running external tests\n\n      This patch is actually independent of caching.  I needed \n      TEST_DIRECTORY to find gitweb/cache.pm for later tests of\n      caching (which are written in Perl).\n\n   2. gitweb: href(..., -path_info => 0|1)\n\n      This is needed to have unique key for a gitweb page, independently\n      on whether we used query string or path_info URL to arrive at it.\n      In other words it is used to implement canonical URL for a gitweb\n      page, which would not depend on gitweb configuration.\n\n   3. gitweb: Add custom error handler using die_error (WIP)\n\n      This patch was created for gitweb to provide similar output for its\n      errors (thrown using die_error() subroutine), and errors from the\n      caching layer (thrown using simply \"die\"), and errors from Perl\n      (like Perl errors, or mkdir from File::Path, or tempfile from\n      File::Temp.  The idea was to use die_error as error handler for\n      the CGI layer via set_message from CGI::Carp.\n\n      Unfortunately I have not realized until later that error message\n      handler set using set_message from CGI::Carp is responsible only\n      for the HTTP response body part: you can't set HTTP headers in\n      CGI::Carp error handler (fatalsToBrowser case).\n\n      Therefore this patch, if it is to stay, should be fixed so that\n      die_error does not produce HTTP headers when invoked as error handler,\n      and not explicitely.\n\n      Note also that due to the fact that die_error uses exit, errors\n      handled by die_error as error handler would not be logged.  Fixing\n      *this* would require additional patch, removing 'exit' from gitweb\n      and replacing it by non-local jump (non-local goto)... which I guess\n      would also improve gitweb performance in persistent environments\n      such as mod_perl, mod_perlite (WIP), or (if gitweb would in the\n      future support it) FastCGI.\n\n   4. gitweb/cache.pm - Very simple file based cache\n   5. gitweb/cache.pm - Stat-based cache expiration\n\n      Those two patches didn't change much from previous versions.\n      The idea is to provide Cache::Cache and CHI caching interface\n      compatibile file-based cache package, so that one would be able\n      to use CHI or Cache::Cache instead of it.\n\n      I have this time actually tested that Cache::FileCache seems to\n      work as gitweb caching engine.\n\n   6. gitweb/cache.pm - Simple select(FH) based output capture\n\n      Compared to previous versions, this time capturing is separated\n      into two packages.  One implements capturing interface, second\n      is select(FH)-based driver that actually does \"capturing\".\n\n      The idea is to be able to replace simplistic select(FH)-based\n      \"capturing\" with capturing using tied filehandle, or capturing\n      using push_layer method from non-standard PerlIO::Util module.\n\n   7. gitweb/cache.pm - Cache captured output (using get/set)\n   8. gitweb: Add optional output caching (from gitweb/cache.pm)\n\n      Compared to previous version of gitweb caching series, adding\n      caching captured output (as yet another package in gitweb/cache.pm)\n      is separated from adding output caching to gitweb itself.\n\n      In this point of series out simple file-based cache has enough\n      features to be useful for caching gitweb output.\n\n   9. gitweb/cache.pm - Adaptive cache expiration time\n\n      Not changed much from previous version.  The idea is to lengthen\n      cache expiration time when there is heavy load on server.  More\n      configurable that in original patch by J.H.  Compare Cache::Adaptive.\n\n  10. gitweb/cache.pm - Use CHI compatibile (compute method) caching\n                        interface\n\n      Just updated to newer way of capturing output.  It is needed because\n      locking requires ->compute API, and would not work with only \n      ->get / ->set API.\n\n  11. gitweb/cache.pm - Use locking to avoid 'cache miss stampede' problem\n  12. gitweb/cache.pm - Serve stale data when waiting for filling cache\n  13. gitweb/cache.pm - Regenerate (refresh) cache in background\n\n      Those three patches didn't change much from previous versions.  \n\n      Note that compared to original J.H. patches caching engine now\n      daemonizes process that does bacground cache generation; this should\n      make it work correctly also in persistent environments like mod_perl\n      (and perhaps in the future also FastCGI).\n\n  14. gitweb: Move generating page title to separate subroutine\n\n      This patch is actually independent of caching, although without\n      \"progress info\" support for cache in subsequent patches the new\n      subroutine would be used only in one place, in git_header_html().\n      Nevertheless refactoring title generation allows to reduce indent\n      level.\n\n  15. gitweb: Show appropriate \"Generating...\" page when regenerating cache\n\n      It should actually work now.  The problem was that flock can be\n      interrupted by SIGALRM from timer used to print progress info.\n\n      Might need extra tests not that of generic support for\n      generating_info subroutine in caching engine, but that subroutine\n      similar to git_generating_html in gitweb works correctly.\n\n  16. gitweb: Add startup delay to activity indicator for cache\n\n      In previous version of series it was lumped together in previous\n      patch, even though it is a new feature, not present in original\n      patch series by J.H.\n\n      This feature was inspired (semi-requested) by comments from Petr\n      'Pasky' Baudis during discussion on some previous version of my\n      series, or J.H. series about gitweb caching.\n\nDiffstat:\n~~~~~~~~~\n gitweb/INSTALL                         |    6 +\n gitweb/README                          |   70 +++\n gitweb/cache.pm                        |  779 ++++++++++++++++++++++++++++++++\n gitweb/gitweb.perl                     |  391 ++++++++++++++--\n t/gitweb-lib.sh                        |    2 +\n t/t9500-gitweb-standalone-no-errors.sh |   19 +\n t/t9503-gitweb-caching.sh              |   38 ++\n t/t9503/test_cache_interface.pl        |  339 ++++++++++++++\n t/t9503/test_cache_output.pl           |   76 +++\n t/t9503/test_capture.pl                |   74 +++\n t/test-lib.sh                          |    3 +\n 11 files changed, 1756 insertions(+), 41 deletions(-)\n create mode 100644 gitweb/cache.pm\n create mode 100755 t/t9503-gitweb-caching.sh\n create mode 100755 t/t9503/test_cache_interface.pl\n create mode 100755 t/t9503/test_cache_output.pl\n create mode 100755 t/t9503/test_capture.pl\n\n-- \nJakub Narebski\n"}]}