git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFC PATCHv4 00/17] gitweb: Simple file based output caching

From
Jakub Narebski <jnareb@gmail.com>
Date
Jun 14, 2010, 16:08 UTC
Message-ID
<1276531710-22945-1-git-send-email-jnareb@gmail.com>

This 17 patches long patch series is intended (yet again) as preliminary version for splitting large 'gitweb: File based caching layer (from git.kernel.org)' mega-patch by John 'Warthog9' Hawley aka J.H., by starting small and adding features piece by piece.

This is fourth version (fourth release) of this series, and is available on http://repo.or.cz/w/git/jnareb-git.git as 'gitweb/cache-kernel-v4' branch. Earlier versions are available there as branches 'gitweb/cache-kernel', 'gitweb/cache-kernel-v2' and 'gitweb/cache-kernel-v3'. Previous version of this series was sent to git mailing list as:

* [RFC PATCHv3 00/10] gitweb: Simple file based output caching
  Message-Id: <1266349005-15393-1-git-send-email-jnareb@gmail.com>
  http://thread.gmane.org/gmane.comp.version-control.git/140138

This version is based on top of c0cb4ed (git-instaweb: Configure it to work with new gitweb structure, 2010-05-28), which was merged as 'ps/gitweb-soc' into next in 5db4adf, on 2010-06-03. Next version would be rebased on top of 'next', which would include "sub run" code in 'jn/gitweb-fastcgi', which would require resolving conflict with this series.

This series of patches does not include comments on individual patches, comparing them to previous version, and to original series by J.H. The next release of this series, hopefully not as RFC, would include them.

The MAIN DIFFERENCE from previous version is that, inspired by splitting gitweb part of "Splitting gitweb and developing write functionalities (Integrated web client for git)" Google Summer of Code 2010 project by Pavan Kumar Sankara, the gitweb output caching code now resides in separate modules under 'lib/' subdirectory, instead of being all in 'cache.pm' file.

The 'gitweb: href(..., -path_info => 0|1)' which began v3 series is already merged in, together with other patches from 'jn/gitweb-caching-prep'.

The following changes are available in the git repository at:
  git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v4
All comments welcome!
Shortlog:
~~~~~~~~~
 1. gitweb: Return or exit after done serving request
    This patch was generated in response to noticing the following kind of
    errors in /var/log/httpd/error_log:
      [Sun Jun 13 11:58:02 2010] gitweb.cgi: Subroutine git_atom redefined
      at /var/www/perl/gitweb/gitweb.cgi line 6804.
    when running gitweb under mod_perl (with ModPerl::Registry handler).
    After this patch I didn't see such errors... but I failed to reproduce
    this error without this patch.
    This patch was sent separately to git mailing list as
      http://thread.gmane.org/gmane.comp.version-control.git/149040
    but should probably be skipped.
 2. gitweb: Fix typo in hash key name in %opts in git_header_html
    This patch was sent separately to git mailing list and is already
    present in 'master' as ad709ea (gitweb: Fix typo in hash key name in
    %opts in git_header_html, 2010-06-13).
 3. gitweb: Prepare for splitting gitweb
    IMPORTANT!!!
    This patch is the main reason this series is CC-ed to Pavan Kumar
    Sankara (student) and Christian Couder (mentor) of "Integrated web
    client for git" GSoC2010 project, which includes splitting gitweb.
    As you can see later in series it makes it very easy to add new gitweb
    modules which are to be installed.  Just add
      GITWEB_MODULES += lib/Module/Foo.pm
    to gitweb/Makefile for Module::Foo.
 4. gitweb/lib - Very simple file based cache
 5. gitweb/lib - Stat-based cache expiration
    The major change is that the code is now in GitwebCache::SimpleFileCache
    module in gitweb/lib/GitwebCache/SimpleFileCache.pm file, instead of
    being in gitweb/cache.pm file.
    Other change is removing using File::Spec to manipulate pathnames,
    using faster concatenation instead.  Using File::Spec do not bring us
    extra portability, as gitweb itself assumes '/' as path separator.
    It should be tiny little bit faster without File::Spec::catfile etc.
 6. gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)
    This is new patch, not present in previous series.  Note that you have
    to run benchmark manually; it is not run by t9503 test.
    The t/t9503/benchmark_caching_interface.pl includes example benchmark
    results at the end of the file.
 7. gitweb/lib - Simple select(FH) based output capture
    The major change is that the code is now in GitwebCache::Capture and
    GitwebCache::Capture::SelectFH modules which are in *separate* files
    (one file per module), instead of being all in gitweb/cache.pm
 8. gitweb/lib - Alternate ways of capturing output
    This is new patch, not present in previous series.  It includes new
    test (not run by t9503) and new benchmark.
 9. gitweb/lib - Cache captured output (using get/set)
    The major change is that the code is now in GitwebCache::CacheOutput
    module in gitweb/lib/GitwebCache/CacheOutput.pm file.
10. gitweb: Add optional output caching
    "use lib __DIR__.'/lib';" is now added by earlier patch in series.
    It uses new infrastructure instead of 'bag of modules' cache.pm file.
11. gitweb/lib - Adaptive cache expiration time
12. gitweb/lib - Use CHI compatible (compute method) caching interface
    Similar to what was in previous version of this series.
13. gitweb/lib - Use locking to avoid 'cache miss stampede' problem
14. gitweb/lib - Serve stale data when waiting for filling cache
15. gitweb/lib - Regenerate (refresh) cache in background
    The major change is that the code is now in new (not present in
    previous version) GitwebCache::FileCacheWithLocking module (in its own
    file), which is derived from GitwebCache::SimpleFileCache.  This allow
    to choose whether to use locking (and other features that require
    locking) or not.
16. gitweb: Show appropriate "Generating..." page when regenerating cache
17. gitweb: Add startup delay to activity indicator for cache
    No major changes, I think (besides using new infrastructure).
 gitweb/Makefile                                |   10 +
 gitweb/README                                  |   62 ++++
 gitweb/gitweb.perl                             |  333 +++++++++++++++++--
 gitweb/lib/GitwebCache/CacheOutput.pm          |   93 ++++++
 gitweb/lib/GitwebCache/Capture.pm              |   66 ++++
 gitweb/lib/GitwebCache/Capture/PerlIO.pm       |   79 +++++
 gitweb/lib/GitwebCache/Capture/SelectFH.pm     |   82 +++++
 gitweb/lib/GitwebCache/Capture/TiedCapture.pm  |  149 +++++++++
 gitweb/lib/GitwebCache/FileCacheWithLocking.pm |  264 +++++++++++++++
 gitweb/lib/GitwebCache/SimpleFileCache.pm      |  417 ++++++++++++++++++++++++
 t/t9500-gitweb-standalone-no-errors.sh         |   19 +
 t/t9503-gitweb-caching.sh                      |   38 +++
 t/t9503/benchmark_caching_interface.pl         |  132 ++++++++
 t/t9503/benchmark_capture_implementations.pl   |  198 +++++++++++
 t/t9503/test_cache_interface.pl                |  409 +++++++++++++++++++++++
 t/t9503/test_cache_output.pl                   |   66 ++++
 t/t9503/test_capture_interface.pl              |   76 +++++
 17 files changed, 2468 insertions(+), 25 deletions(-)
 create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/PerlIO.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/TiedCapture.pm
 create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm
 create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm
 create mode 100755 t/t9503-gitweb-caching.sh
 create mode 100755 t/t9503/benchmark_caching_interface.pl
 create mode 100755 t/t9503/benchmark_capture_implementations.pl
 create mode 100755 t/t9503/test_cache_interface.pl
 create mode 100755 t/t9503/test_cache_output.pl
 create mode 100755 t/t9503/test_capture_interface.pl
Next: Jakub Narebski
Message 1 of 21 in “[RFC PATCHv4 00/17] gitweb: Simple file based output caching”
  1. Jakub NarebskiJun 14, 2010
  2. 01/17 gitweb: Return or exit after done serving requestJakub Narebski, Jun 14, 2010
  3. 02/17 gitweb: Fix typo in hash key name in %opts in git_header_htmlJakub Narebski, Jun 14, 2010
  4. 03/17 gitweb: Prepare for splitting gitwebJakub Narebski, Jun 14, 2010
  5. 04/17 gitweb/lib - Very simple file based cacheJakub Narebski, Jun 14, 2010
  6. 05/17 gitweb/lib - Stat-based cache expirationJakub Narebski, Jun 14, 2010
  7. 06/17 gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)Jakub Narebski, Jun 14, 2010
  8. 07/17 gitweb/lib - Simple select(FH) based output captureJakub Narebski, Jun 14, 2010
  9. 08/17 gitweb/lib - Alternate ways of capturing outputJakub Narebski, Jun 14, 2010
  10. 09/17 gitweb/lib - Cache captured output (using get/set)Jakub Narebski, Jun 14, 2010
  11. 10/17 gitweb: Add optional output caching (from gitweb/cache.pm)Jakub Narebski, Jun 14, 2010
  12. 10/18 gitweb/lib - Cache captured output (using get/set)Jakub Narebski, Jun 14, 2010
  13. 11/17 gitweb/lib - Adaptive cache expiration timeJakub Narebski, Jun 14, 2010
  14. 12/17 gitweb/lib - Use CHI compatibile (compute method) caching interfaceJakub Narebski, Jun 14, 2010
  15. 13/17 gitweb/lib - Use locking to avoid 'cache miss stampede' problemJakub Narebski, Jun 14, 2010
  16. 14/17 gitweb/lib - Serve stale data when waiting for filling cacheJakub Narebski, Jun 14, 2010
  17. 15/17 gitweb/lib - Regenerate (refresh) cache in backgroundJakub Narebski, Jun 14, 2010
  18. 16/17 gitweb: Show appropriate "Generating..." page when regenerating cacheJakub Narebski, Jun 14, 2010
  19. 17/17 gitweb: Add startup delay to activity indicator for cacheJakub Narebski, Jun 14, 2010
  20. 18/17 gitweb/lib - Add clear() and size() methods to caching interfaceJakub Narebski, Jun 24, 2010
  21. 19/17 gitweb: Add beginnings of cache administration page (WIP)Jakub Narebski, Jun 24, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.