git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCHv5 00/17] gitweb: Simple file based output caching

From
Jakub Narebski <jnareb@gmail.com>
Date
Oct 6, 2010, 22:01 UTC
Message-ID
<1286402526-13143-1-git-send-email-jnareb@gmail.com>

This 17+ patches long patch series is intended as a replacement for large 'gitweb: File based caching layer (from git.kernel.org)' mega-patch by John 'Warthog9' Hawley aka J.H., by starting small and adding features piece by piece.

This is fourthfifth version (5th release) of this series, and is available on http://repo.or.cz/w/git/jnareb-git.git as 'gitweb/cache-kernel-v5' branch. Earlier versions are available there as branches 'gitweb/cache-kernel', 'gitweb/cache-kernel-v2', 'gitweb/cache-kernel-v3' and 'gitweb/cache-kernel-v4'. Previous version of this series was sent to git mailing list as:

* [RFC PATCHv4 00/17] gitweb: Simple file based output caching
  Message-Id: <1276531710-22945-1-git-send-email-jnareb@gmail.com>
  http://thread.gmane.org/gmane.comp.version-control.git/149101

This version is based on top of aad24ad (gitweb: Move call to evaluate_git_version after evaluate_gitweb_config, 2010-10-05) in 'gitweb/web' branch of http://repo.or.cz/w/git/jnareb-git.git repository.

The 'gitweb/web' branch consist of the following commits on top of
1e63341 (Merge branch 'maint', 2010-09-30) in 'master' branch of main
git repository:
  gitweb/Makefile: Include gitweb/config.mak
  t/gitweb-lib.sh: Add support for GITWEB_TEST_INSTALLED
  gitweb/Makefile: Add 'test' and 'test-installed' targets
  gitweb: Move call to evaluate_git_version after evaluate_gitweb_config
The first three patches were send as 3-part series:
* [PATCHv2 0/3] Testing installed gitweb
  Message-Id: <1285506146-8009-1-git-send-email-jnareb@gmail.com>
  http://thread.gmane.org/gmane.comp.version-control.git/157222

Those are required to be able to test installed version of gitweb and modules, to check if there are any problems with process of installing gitweb.

The last patch in 'gitweb/web' branch was send as:
* [PATCH] gitweb: Move call to evaluate_git_version after evaluate_gitweb_config 
  Message-ID: <20100926113431.28918.53550.stgit@localhost.localdomain>
  http://article.gmane.org/gmane.comp.version-control.git/157220

This was needed during development of this series, to avoid spurious output of gitweb tests when using `--debug' option.

The MAIN DIFFERENCE from previous version is that it is based on never version of gitweb; in particular putting all code in subroutines: gitweb calls run() subroutine on startup, and run_request() subroutine on each request (per connection) from the 'jn/gitweb-fastcgi' branch.

The patches
  gitweb: Return or exit after done serving request
  gitweb: Fix typo in hash key name in %opts in git_header_html

present at beginning of previous (fourth) version of this series are already merged in.

The patches
  gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)
  gitweb/lib - Alternate ways of capturing output

which were interleaved with other patches in previous version of this series got moved to the end of series, and marked proof of concept ("PoC").

The following changes are available in the git repository at:
  git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v5
All comments welcome!

J.H., could you comment on how this series relates to the gitweb code *currently* running on git.kernel.org wrt. code? If possible, could you try to compare performance of those two implementations, the one presented here in this series, and the one used by git.kernel.org.

Pasky or other repo.or.cz admins, could you tell us if and what kind of caching gitweb used by repo.or.cz uses, and how it compares (wrt code and if possible also performance) to solution implemented in this series?

TODO list and areas of possible improvements would be send in separate email.

Jakub Narebski (21):
  t/test-lib.sh: Export also GIT_BUILD_DIR in test_external
  gitweb: Prepare for splitting gitweb
  gitweb/lib - Very simple file based cache
  gitweb/lib - Stat-based cache expiration
  gitweb/lib - Regenerate entry if the cache file has size of 0
  gitweb/lib - Simple select(FH) based output capture
  gitweb/lib - Cache captured output (using get/set)
  gitweb: Add optional output caching (from gitweb/cache.pm)
  gitweb/lib - Adaptive cache expiration time
  gitweb/lib - Use CHI compatibile (compute method) caching interface
  gitweb/lib - Use locking to avoid 'cache miss stampede' problem
  gitweb/lib - No need for File::Temp when locking
  gitweb/lib - Serve stale data when waiting for filling cache
  gitweb/lib - Regenerate (refresh) cache in background
  gitweb: Introduce %actions_info, gathering information about actions
  gitweb: Show appropriate "Generating..." page when regenerating cache
  gitweb: Add startup delay to activity indicator for cache
  gitweb/lib - Add clear() and size() methods to caching interface
  gitweb: Add beginnings of cache administration page
  gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)
  gitweb/lib - Alternate ways of capturing output
 gitweb/Makefile                                |   24 ++-
 gitweb/README                                  |   61 +++
 gitweb/gitweb.perl                             |  459 ++++++++++++++++++++++--
 gitweb/lib/GitwebCache/CacheOutput.pm          |   95 +++++
 gitweb/lib/GitwebCache/Capture.pm              |   66 ++++
 gitweb/lib/GitwebCache/Capture/PerlIO.pm       |   79 ++++
 gitweb/lib/GitwebCache/Capture/SelectFH.pm     |   82 ++++
 gitweb/lib/GitwebCache/Capture/TiedCapture.pm  |   76 ++++
 gitweb/lib/GitwebCache/FileCacheWithLocking.pm |  280 ++++++++++++++
 gitweb/lib/GitwebCache/SimpleFileCache.pm      |  472 ++++++++++++++++++++++++
 gitweb/lib/Tie/Restore.pm                      |   24 ++
 gitweb/lib/TiedCapture/PerlIO.pm               |   56 +++
 gitweb/lib/TiedCapture/String.pm               |   53 +++
 t/t9500-gitweb-standalone-no-errors.sh         |   20 +
 t/t9503-gitweb-caching-interface.sh            |   34 ++
 t/t9503/benchmark_caching_interface.pl         |  209 +++++++++++
 t/t9503/test_cache_interface.pl                |  407 ++++++++++++++++++++
 t/t9504-gitweb-capture-interface.sh            |   34 ++
 t/t9504/benchmark_capture_implementations.pl   |  226 +++++++++++
 t/t9504/test_capture_implementations.pl        |   85 +++++
 t/t9504/test_capture_interface.pl              |   76 ++++
 t/t9505-gitweb-cache.sh                        |   34 ++
 t/t9505/test_cache_output.pl                   |   70 ++++
 t/test-lib.sh                                  |    4 +-
 24 files changed, 2997 insertions(+), 29 deletions(-)
 create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/PerlIO.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/TiedCapture.pm
 create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm
 create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm
 create mode 100644 gitweb/lib/Tie/Restore.pm
 create mode 100644 gitweb/lib/TiedCapture/PerlIO.pm
 create mode 100644 gitweb/lib/TiedCapture/String.pm
 create mode 100755 t/t9503-gitweb-caching-interface.sh
 create mode 100755 t/t9503/benchmark_caching_interface.pl
 create mode 100755 t/t9503/test_cache_interface.pl
 create mode 100755 t/t9504-gitweb-capture-interface.sh
 create mode 100755 t/t9504/benchmark_capture_implementations.pl
 create mode 100755 t/t9504/test_capture_implementations.pl
 create mode 100755 t/t9504/test_capture_interface.pl
 create mode 100755 t/t9505-gitweb-cache.sh
 create mode 100755 t/t9505/test_cache_output.pl
-- 
1.7.3
Next: Jakub Narebski
Message 1 of 41 in “[PATCHv5 00/17] gitweb: Simple file based output caching”
  1. Jakub NarebskiOct 6, 2010
  2. 01/17 t/test-lib.sh: Export also GIT_BUILD_DIR in test_externalJakub Narebski, Oct 6, 2010
  3. 02/17 gitweb: Prepare for splitting gitwebJakub Narebski, Oct 6, 2010
  4. 03/17 gitweb/lib - Very simple file based cacheJakub Narebski, Oct 6, 2010
  5. Thomas AdamOct 6, 2010
  6. Ævar Arnfjörð BjarmasonOct 6, 2010
  7. Thomas AdamOct 6, 2010
  8. Ævar Arnfjörð BjarmasonOct 6, 2010
  9. Jakub NarebskiOct 6, 2010
  10. Thomas AdamOct 6, 2010
  11. Jakub NarebskiOct 6, 2010
  12. Ævar Arnfjörð BjarmasonOct 6, 2010
  13. Jakub NarebskiOct 6, 2010
  14. 04/17 gitweb/lib - Stat-based cache expirationJakub Narebski, Oct 6, 2010
  15. 05/17 gitweb/lib - Regenerate entry if the cache file has size of 0Jakub Narebski, Oct 6, 2010
  16. 06/17 gitweb/lib - Simple select(FH) based output captureJakub Narebski, Oct 6, 2010
  17. Thomas AdamOct 6, 2010
  18. Jakub NarebskiOct 6, 2010
  19. Ævar Arnfjörð BjarmasonOct 6, 2010
  20. Jakub NarebskiOct 6, 2010
  21. 07/17 gitweb/lib - Cache captured output (using get/set)Jakub Narebski, Oct 6, 2010
  22. 08/17 gitweb: Add optional output cachingJakub Narebski, Oct 6, 2010
  23. Ævar Arnfjörð BjarmasonOct 6, 2010
  24. Jakub NarebskiOct 6, 2010
  25. Ævar Arnfjörð BjarmasonOct 6, 2010
  26. 09/17 gitweb/lib - Adaptive cache expiration timeJakub Narebski, Oct 6, 2010
  27. 10/21 gitweb/lib - Use CHI compatibile (compute method) caching interfaceJakub Narebski, Oct 6, 2010
  28. 11/17 gitweb/lib - Use locking to avoid 'cache miss stampede' problemJakub Narebski, Oct 6, 2010
  29. 12/17 gitweb/lib - No need for File::Temp when lockingJakub Narebski, Oct 6, 2010
  30. 13/17 gitweb/lib - Serve stale data when waiting for filling cacheJakub Narebski, Oct 6, 2010
  31. 14/17 gitweb/lib - Regenerate (refresh) cache in backgroundJakub Narebski, Oct 6, 2010
  32. 15/17 gitweb: Introduce %actions_info, gathering information about actionsJakub Narebski, Oct 6, 2010
  33. 16/17 gitweb: Show appropriate "Generating..." page when regenerating cacheJakub Narebski, Oct 6, 2010
  34. 17/17 gitweb: Add startup delay to activity indicator for cacheJakub Narebski, Oct 6, 2010
  35. 18/17 gitweb/lib - Add clear() and size() methods to caching interfaceJakub Narebski, Oct 6, 2010
  36. Thomas AdamOct 6, 2010
  37. 19/17 gitweb: Add beginnings of cache administration pageJakub Narebski, Oct 6, 2010
  38. 20/17 gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)Jakub Narebski, Oct 6, 2010
  39. 21/17 gitweb/lib - Alternate ways of capturing outputJakub Narebski, Oct 6, 2010
  40. [RFD] Possible improvements for output caching in gitwebJakub Narebski, Oct 10, 2010
  41. J.H.Oct 24, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.