{"thread":{"id":"25643","subject":"[RFC] Implementing gitweb output caching - issues to solve","startedAt":"2010-11-04T16:21:51Z","lastAt":"2010-12-10T04:46:55Z","messageCount":9,"participants":["Jakub Narebski","J.H.","Junio C Hamano","Jonathan Nieder","Olaf Alders"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"155175","messageId":"201011041721.53371.jnareb@gmail.com","threadId":"25643","inReplyTo":null,"subject":"[RFC] Implementing gitweb output caching - issues to solve","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-11-04T16:21:51Z","receivedAt":"2010-11-04T16:21:51Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"First, what is the status of gitweb output caching?  I have send minimal\nfixup for latest John 'Warthog9' Hawley series as Gitweb caching v7.1[1].\nJohn said that he can work on fixes on November 2 at earliest.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/160147/focus=160473\n\n\nDuring working on my rewrite of gitweb output caching, based on ideas\n(and some code) from John 'Warthog9' Hawley (J.H.) series I have met\nwith the following issues:\n\n1. Error handling\n\n   There are three separate places where we can encounter error during\n   caching.  They are:\n\n    A. setting up cache (initialization)\n    B. retrieving data from cache\n    C. storing (saving) data in cache.\n\n\n   There are two possible choices for errors during cache initialization\n   (case A):\n     * turn off caching\n     * show 500 Internal Server Error\n\n   The problem with turning off caching is that it hides an error, and\n   admin can not notice that caching does not work.  Therefore I think\n   that better solution would be to generate error, especially that such\n   errors are not, usually, incidental.\n\n   Note that in my opinion cache should not be initialized if caching is\n   disabled.\n  \n\n   The modern CHI caching interface provides a way to change how runtime\n   errors are handled via 'on_get_error' and 'on_set_error' parameters to\n   constructor.  The default for CHI is to log error, or ignore if no\n   logger is set.  You can provide a coderef (callback).\n\n\n   For errors during getting data from cache (case B), we can ignore error\n   which means generating data as if cache was disabled/entry did not\n   exist in cache, or erroring out and showing error to user.  Such errors\n   should be rare, so erroring out might be appropriate reaction.  Note\n   that we have special case of zero sized cache entry file, in which case\n   we treat it as non-existent entry, already.\n\n\n   For errors during setting (saving) data to cache (case C), we can\n   ignore error and not re-generate cache (not even calling callback if\n   error is detected early), or error out.  Such errors can happen for\n   example if filesystem gets full.  It might be better to ignore such\n   errore, at least in the case of ENOSPC / ENOMEM.\n\n\n2. Progress info and not caching error pages\n\n   J.H. find out or invented very nice hack to generate progress info\n   page without Ajax (and therefore without need for JavaScript), to make\n   user aware that data is being regenerated, and that he/she should be\n   patient instead of refreshing or complaining.\n\n   This trick uses the fact that the internal redirect using meta\n   refresh[2]\n\n      <meta http-equiv=\"refresh\" content=\"0\" />\n\n   does not occur untill the web browser gets full page (which means that\n   connection get closed / end of request).\n\n   [2] http://en.wikipedia.org/wiki/Meta_refresh\n\n   Besides the fact that, as J.H. wrote\n\n    JH> There are still a few known \"issues\" with respect to this:\n    JH> - Code needs to be added to be \"browser\" aware so\n    JH>   that clients like wget that are trying to get a\n    JH>   binary blob don't obtain a \"Generating...\" page\n\n   there is additional problem, at least in my rewrite.  (I didn't follow\n   the code flow in J.H. v7 patch series to check if it also affected).\n\n\n   Let's assume that 'progress_info' feature is turned on, and that cache\n   is generated in background.  Let's assume that error pages (die_error)\n   are not cached.\n\n   Now client tries to access page which would/will result in an error.\n   With caching it acquires exclusive (writers) lock, check that there\n   are no stale data (error pages are never cached), checks that it is\n   provided with 'generating_info' subroutine, so it forks a process to\n   generate data in background.\n\n   Background process detaches, tries to generate data to cache,\n   die_error is called.  die_error turns off capturing, and prints to\n   STDOUT.  At the end of die_error there is jump to outer scope to\n   DONE_GITWEB; therefore $lock_fh goes out of scope and lockfile is\n   closed, and lock released.  Background process finishes.\n\n   Parent process runs 'generating_info' subroutine.  Now if it waits a\n   bit like in my rewrite before employing trick mentioned above, and\n   does not print anything if lockfile is released before startup delay,\n   _and_ die_error finishes within this delay, then everything is all\n   right: the error message is sent to client.\n\n   If die_error doesn't finish before git_generating_data_html prints\n   meta refresh, or there is no startup delay, then error pages would get\n   infinite redirects (remember: there is never anything in cache for\n   error pages).  This is a bad thing.\n\n\n   One possible solution for this problem (beside employing startup\n   delay) is to have tee output, i.e. print it as it is being captured.\n   Streamed (partial) response would serve as progress indicator for\n   process (re)generating cache; only parallel processes waiting for\n   cache would show 'generating_info'.\n\n   I think that in current implementation of capturing it would be as\n   simple as not closing STDOUT, but I'd have to check that.\n\n\n3. Using generic cache engine and memory consumption\n\n   Most Perl caching interfaces support only $cache->set($key, $data),\n   where $data is a Perl variable, and $data = $cache->get($key), or\n   their equivalents.  Even for file-based cache drivers you save from\n   memory and read into memory.  \n\n   The only exception I know of is Cache interface with Cache::File\n   driver, that provides $cache->handle($key [, $mode]), where optional\n   $mode argument can be any of mode strings that can be used in 'open'\n   function, or one of fopen(3) modes.  (Of course for memory-based or\n   network-based (like memcached) caches it might not make sense to\n   provide such interface).\n\n   The most often recommended capture module, Capture::Tiny, allows only\n   capturing into scalar (into memory) \n\n     $stdout = capture \\&code;\n\n   or (using its prototype)\n     \n     $stdout = capture { <code> };\n\n   Well, it is *::Tiny, so it supports minimal API.  From the list of\n   different capture modules, that allow capturing of Perl code output,\n   with different interfaces (API), in \"See also\" section of\n   Capture::Tiny documentation[3], only IO::CaptureOutput allow capturing\n   into specified file:\n\n     capture \\&code, \\$stdout, undef, $outfile;\n\n   [3] http://p3rl.org/Capture::Tiny\n\n\n   J.H.'s gitweb output caching v7 captures output directly ito cache\n   files.  The problem with doing it in my rewrite is to allow capturing\n   directly into cache entry file without losing ability to select\n   different caching engine, which might be not file-based (like\n   e.g. memcached-based).\n\n\nP.S. You can always check out my latest work on gitweb output caching\nrewrite here (warning: rebased!)\n\n  git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-pu\n\n(it is also available in git://github.com/jnareb/git.git)\n\n-- \nJakub Narębski\nPoland\n"},{"id":"157657","messageId":"4D00316F.9000305@eaglescrag.net","threadId":"25643","inReplyTo":"201011041721.53371.jnareb@gmail.com","subject":"Re: [RFC] Implementing gitweb output caching - issues to solve","fromName":"J.H.","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-12-09T01:31:27Z","receivedAt":"2010-12-09T01:31:27Z","isPatch":false,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"> 1. Error handling\n[...]\n>    Note that in my opinion cache should not be initialized if caching is\n>    disabled.\n\nWell it wasn't getting initialized in my code if caching was turned off.\n It might have been buffering, but it wasn't mucking with the directory\ncreation and such...\n\n>    The modern CHI caching interface provides a way to change how runtime\n>    errors are handled via 'on_get_error' and 'on_set_error' parameters to\n>    constructor.  The default for CHI is to log error, or ignore if no\n>    logger is set.  You can provide a coderef (callback).\n> \n> \n>    For errors during getting data from cache (case B), we can ignore error\n>    which means generating data as if cache was disabled/entry did not\n>    exist in cache, or erroring out and showing error to user.  Such errors\n>    should be rare, so erroring out might be appropriate reaction.  Note\n>    that we have special case of zero sized cache entry file, in which case\n>    we treat it as non-existent entry, already.\n> \n> \n>    For errors during setting (saving) data to cache (case C), we can\n>    ignore error and not re-generate cache (not even calling callback if\n>    error is detected early), or error out.  Such errors can happen for\n>    example if filesystem gets full.  It might be better to ignore such\n>    errore, at least in the case of ENOSPC / ENOMEM.\n\nThe potential, at least for the deployed sites who are already using the\ngitweb caching, for disaster if caching isn't running is pretty high.  I\nwould agree that erroring out with a 500 that explains the 'why' behind\nthe failure is the only real option right now.\n\nThe callback mechanism in CHI is nice, but I think for the time being is\na little overkill for what we've got right now.  I also don't see that\nmany, typical, sysadmins adding in that many error handling hooks and\nsuch into gitweb and/or their infrastructure.  I could be wrong, I just\ndon't see that as being a common occurrence.\n\n> 2. Progress info and not caching error pages\n> \n>    J.H. find out or invented very nice hack to generate progress info\n>    page without Ajax (and therefore without need for JavaScript), to make\n>    user aware that data is being regenerated, and that he/she should be\n>    patient instead of refreshing or complaining.\n\nNot so much found or invented, it's something that's been around since I\nstarted doing any web development back in 1995.  It's a bit old-school,\nI'll admit, but it gets the job done still.\n\n[...]\n>     JH> There are still a few known \"issues\" with respect to this:\n>     JH> - Code needs to be added to be \"browser\" aware so\n>     JH>   that clients like wget that are trying to get a\n>     JH>   binary blob don't obtain a \"Generating...\" page\n\nI've solved this in v8, there is now an explicit blacklist for clients\nthat we assume are only there for whatever is getting generated.  Right\nnow the list is only wget and curl as those are the only two I can think\nof immediately that wouldn't be able to parse the meta-equiv refresh.\n\nThere is already a special case for feeds to not get this, so those\nshouldn't be affected as it is already.\n\nDownside is it's a blacklist, and inherits all the problems of a\nblacklist.  Though there really isn't a better way to handle that, given\nthere's no way to really query a browser and ask what it's capabilities\nreally are.\n\n>    there is additional problem, at least in my rewrite.  (I didn't follow\n>    the code flow in J.H. v7 patch series to check if it also affected).\n> \n> \n>    Let's assume that 'progress_info' feature is turned on, and that cache\n>    is generated in background.  Let's assume that error pages (die_error)\n>    are not cached.\n> \n>    Now client tries to access page which would/will result in an error.\n>    With caching it acquires exclusive (writers) lock, check that there\n>    are no stale data (error pages are never cached), checks that it is\n>    provided with 'generating_info' subroutine, so it forks a process to\n>    generate data in background.\n> \n>    Background process detaches, tries to generate data to cache,\n>    die_error is called.  die_error turns off capturing, and prints to\n>    STDOUT.  At the end of die_error there is jump to outer scope to\n>    DONE_GITWEB; therefore $lock_fh goes out of scope and lockfile is\n>    closed, and lock released.  Background process finishes.\n> \n>    Parent process runs 'generating_info' subroutine.  Now if it waits a\n>    bit like in my rewrite before employing trick mentioned above, and\n>    does not print anything if lockfile is released before startup delay,\n>    _and_ die_error finishes within this delay, then everything is all\n>    right: the error message is sent to client.\n> \n>    If die_error doesn't finish before git_generating_data_html prints\n>    meta refresh, or there is no startup delay, then error pages would get\n>    infinite redirects (remember: there is never anything in cache for\n>    error pages).  This is a bad thing.\n> \n> \n>    One possible solution for this problem (beside employing startup\n>    delay) is to have tee output, i.e. print it as it is being captured.\n>    Streamed (partial) response would serve as progress indicator for\n>    process (re)generating cache; only parallel processes waiting for\n>    cache would show 'generating_info'.\n> \n>    I think that in current implementation of capturing it would be as\n>    simple as not closing STDOUT, but I'd have to check that.\n\nActually in reading through this I thought of a better way to get the\nerror message cleanly passed from the backend process to the frontend\nwaiting processes.\n\nIt's implemented in v8, but it basically boils down to, strangely\nenough, caching the error. (I've got a hammer, it clearly solves all\nproblems!).  It's not a full cache (like what I'm doing with the rest of\nthe cache) but it basically generates a file that can be used to short\ncircuit things a bit.\n\n> 3. Using generic cache engine and memory consumption\n> \n>    Most Perl caching interfaces support only $cache->set($key, $data),\n>    where $data is a Perl variable, and $data = $cache->get($key), or\n>    their equivalents.  Even for file-based cache drivers you save from\n>    memory and read into memory.  \n> \n>    The only exception I know of is Cache interface with Cache::File\n>    driver, that provides $cache->handle($key [, $mode]), where optional\n>    $mode argument can be any of mode strings that can be used in 'open'\n>    function, or one of fopen(3) modes.  (Of course for memory-based or\n>    network-based (like memcached) caches it might not make sense to\n>    provide such interface).\n> \n>    The most often recommended capture module, Capture::Tiny, allows only\n>    capturing into scalar (into memory) \n> \n>      $stdout = capture \\&code;\n> \n>    or (using its prototype)\n>      \n>      $stdout = capture { <code> };\n> \n>    Well, it is *::Tiny, so it supports minimal API.  From the list of\n>    different capture modules, that allow capturing of Perl code output,\n>    with different interfaces (API), in \"See also\" section of\n>    Capture::Tiny documentation[3], only IO::CaptureOutput allow capturing\n>    into specified file:\n> \n>      capture \\&code, \\$stdout, undef, $outfile;\n> \n>    [3] http://p3rl.org/Capture::Tiny\n> \n> \n>    J.H.'s gitweb output caching v7 captures output directly ito cache\n>    files.  The problem with doing it in my rewrite is to allow capturing\n>    directly into cache entry file without losing ability to select\n>    different caching engine, which might be not file-based (like\n>    e.g. memcached-based).\n\nOne of the big things that my caching engine now stops is the stampeding\nherd problem, basically by locking everyone out of the cache for that\nentry till it's prepared.  Assuming that can be preserved with any\narbitrary caching engine, then capturing directly to the file isn't that\nbig of a deal.\n\nIf that can be preserved in your full re-write, then it doesn't really\nmatter how or when the data makes it into the caching system be it\nmemory, disk, /dev/random or collapsing quantum wave form\n\nI'm going to try and get v8 out tomorrow morning, and I'm going to let\nthis stew on kernel.org overnight.  It's looking like 18 patches (which\nincludes Jakub's v7.2, since I was building straight on top of that).\n\nIn poking around, I'll admit, I keep finding things I'd love to fix /\ncleanup throughout the entirety of the code base.  I'm kinda hoping to\nfinally get caching dealt with and merged so I can start scratching\nthose other itches.  Mind you this has been 4+ years in the making\nalready...\n\n- John 'Warthog9' Hawley\n\nP.S. Most of this e-mail was written a couple of weeks ago, I found it\nin my drafts folder and wanted to get it out along with a note that I've\ngot v8 percolating. -JH\n"},{"id":"157669","messageId":"7v62v3ikqj.fsf@alter.siamese.dyndns.org","threadId":"25643","inReplyTo":"4D00316F.9000305@eaglescrag.net","subject":"Re: [RFC] Implementing gitweb output caching - issues to solve","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-12-09T05:22:12Z","receivedAt":"2010-12-09T05:22:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"J.H.\" <warthog9@eaglescrag.net> writes:\n\n> P.S. Most of this e-mail was written a couple of weeks ago, I found it\n> in my drafts folder and wanted to get it out along with a note that I've\n> got v8 percolating. -JH\n\nThanks.  Is \"Tomorrow morning\" relative to \"a couple of weeks ago\"? ;-)\n"},{"id":"157670","messageId":"4D0068ED.8070409@eaglescrag.net","threadId":"25643","inReplyTo":"7v62v3ikqj.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC] Implementing gitweb output caching - issues to solve","fromName":"J.H.","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-12-09T05:28:13Z","receivedAt":"2010-12-09T05:28:13Z","isPatch":false,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":"On 12/08/2010 09:22 PM, Junio C Hamano wrote:\n> \"J.H.\" <warthog9@eaglescrag.net> writes:\n> \n>> P.S. Most of this e-mail was written a couple of weeks ago, I found it\n>> in my drafts folder and wanted to get it out along with a note that I've\n>> got v8 percolating. -JH\n> \n> Thanks.  Is \"Tomorrow morning\" relative to \"a couple of weeks ago\"? ;-)\n\nNo tomorrow morning is relative to today - so ~12hrs from now, ish,\nmaybe big ISH ;-)\n\nI finished up the patch series today.  The series itself is already\npushed out to my repo on kernel.org\n\nhttp://git.kernel.org/?p=git/warthog9/gitweb.git;a=shortlog;h=refs/heads/gitweb-ml-v8\n\njust wanted to let that stew a bit before pushing it out to the mailing\nlist, since it is now an 18 part patch :-/\n\n- John 'Warthog9' Hawley\n"},{"id":"157759","messageId":"201012092330.06688.jnareb@gmail.com","threadId":"25643","inReplyTo":"4D00316F.9000305@eaglescrag.net","subject":"Re: [RFC] Implementing gitweb output caching - issues to solve","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-12-09T22:30:05Z","receivedAt":"2010-12-09T22:30:05Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 9 Dec 2010, J.H. wrote:\n> > 1. Error handling\n> [...]\n> >    Note that in my opinion cache should not be initialized if caching is\n> >    disabled.\n> \n> Well it wasn't getting initialized in my code if caching was turned off.\n> It might have been buffering, but it wasn't mucking with the directory\n> creation and such...\n\nWhat I wanted to say here was that, in my opinion, adding support for\ncaching to gitweb shouldn't change gitweb behavior in the case when\ncaching is enabled (and should modify gitweb code only minimally).  \n\nThe original v7 series ran capture_fetch also when caching was disabled,\nand in such case captured gitweb to in-memory file to print it at end.\nThis was fixed (well, fixed if we consider this a bug/misfeature, which\nI do) in v7.1, and finally done without errors in v7.4\n\n> >    The modern CHI caching interface provides a way to change how runtime\n> >    errors are handled via 'on_get_error' and 'on_set_error' parameters to\n> >    constructor.  The default for CHI is to log error, or ignore if no\n> >    logger is set.  You can provide a coderef (callback).\n> > \n> > \n> >    For errors during getting data from cache (case B), we can ignore error\n> >    which means generating data as if cache was disabled/entry did not\n> >    exist in cache, or erroring out and showing error to user.  Such errors\n> >    should be rare, so erroring out might be appropriate reaction.  Note\n> >    that we have special case of zero sized cache entry file, in which case\n> >    we treat it as non-existent entry, already.\n> > \n> > \n> >    For errors during setting (saving) data to cache (case C), we can\n> >    ignore error and not re-generate cache (not even calling callback if\n> >    error is detected early), or error out.  Such errors can happen for\n> >    example if filesystem gets full.  It might be better to ignore such\n> >    errore, at least in the case of ENOSPC / ENOMEM.\n> \n> The potential, at least for the deployed sites who are already using the\n> gitweb caching, for disaster if caching isn't running is pretty high.  I\n> would agree that erroring out with a 500 that explains the 'why' behind\n> the failure is the only real option right now.\n> \n> The callback mechanism in CHI is nice, but I think for the time being is\n> a little overkill for what we've got right now.  I also don't see that\n> many, typical, sysadmins adding in that many error handling hooks and\n> such into gitweb and/or their infrastructure.  I could be wrong, I just\n> don't see that as being a common occurrence.\n\nWell, we would of course provide default callback that uses die_error,\nsee for example\n\n  [PATCHv6/RFC 21/24] gitweb: Wrap die_error to use as error handler for caching engine\n  http://thread.gmane.org/gmane.comp.version-control.git/163052/focus=163043\n  http://repo.or.cz/w/git/jnareb-git.git/commitdiff/9e8e92f318c6b776486ab19509326f0b470d491d\n\nWhat I want to avoid is using die_error in caching module; this entangles\ncaching module code with gitweb code, and makes it much more difficult to\ndo unit tests / test caching code in isolation (die_error would have to be\nmocked).  This would also prevent making caching engine a proper Perl \nmodule, I think...\n\n\nThere is also another issue: with error handler passed as callback we\ncan for example use CHI as caching engine (with any of its drivers:\nFile, FastMmap, Memcached) with our own/gitweb error reporting.\n\n\n> > 2. Progress info and not caching error pages\n> > \n> >    J.H. find out or invented very nice hack to generate progress info\n> >    page without Ajax (and therefore without need for JavaScript), to make\n> >    user aware that data is being regenerated, and that he/she should be\n> >    patient instead of refreshing or complaining.\n> \n> Not so much found or invented, it's something that's been around since I\n> started doing any web development back in 1995.  It's a bit old-school,\n> I'll admit, but it gets the job done still.\n\nNice idea, anyway.  I do wonder though if all web browsers follow this\nbehavior, and is this behavior documented / required...\n\n> [...]\n> >     JH> There are still a few known \"issues\" with respect to this:\n> >     JH> - Code needs to be added to be \"browser\" aware so\n> >     JH>   that clients like wget that are trying to get a\n> >     JH>   binary blob don't obtain a \"Generating...\" page\n> \n> I've solved this in v8, there is now an explicit blacklist for clients\n> that we assume are only there for whatever is getting generated.  Right\n> now the list is only wget and curl as those are the only two I can think\n> of immediately that wouldn't be able to parse the meta-equiv refresh.\n\nIn my rewrite\n\n  [PATCHv6 17/24] gitweb: Show appropriate \"Generating...\" page when regenerating cache\n  http://thread.gmane.org/gmane.comp.version-control.git/163052/focus=163040\n  http://repo.or.cz/w/git/jnareb-git.git/commitdiff/48679f7985ccda16dc54fda97790841bab4a0ba2#patch1\n\n(see the browser_is_robot() subroutine:\n\n  http://repo.or.cz/w/git/jnareb-git.git/blob/48679f7985ccda16dc54fda97790841bab4a0ba2:/gitweb/gitweb.perl#l870\n\nI use HTTP::BrowserDetect package if available and it's ->robot() method.\n\nThe fallback is to use *whitelist*, assuming that it would be better to\nnot show \"Generating...\" page rather than download the wrong thing.\nI also guess that most (all?) web browsers use \"Mozilla compatibile\"\nsomewhere in their User-Agent string, thus matching 'Mozilla'.\n\n> \n> There is already a special case for feeds to not get this, so those\n> shouldn't be affected as it is already.\n\nYes, it is true that you limit actions that are affected.  I also\nuse whitelist in my rewrite, allowing only actions which output HTML\nto use \"Generating...\" page.  Perhaps that's too strict.\n \n> Downside is it's a blacklist, and inherits all the problems of a\n> blacklist.  Though there really isn't a better way to handle that, given\n> there's no way to really query a browser and ask what it's capabilities\n> really are.\n\nWell, we can always use whitelist instead, see above.\n\n> >    there is additional problem, at least in my rewrite.  (I didn't follow\n> >    the code flow in J.H. v7 patch series to check if it also affected).\n> > \n> > \n> >    Let's assume that 'progress_info' feature is turned on, and that cache\n> >    is generated in background.  Let's assume that error pages (die_error)\n> >    are not cached.\n> > \n> >    Now client tries to access page which would/will result in an error.\n> >    With caching it acquires exclusive (writers) lock, check that there\n> >    are no stale data (error pages are never cached), checks that it is\n> >    provided with 'generating_info' subroutine, so it forks a process to\n> >    generate data in background.\n> > \n> >    Background process detaches, tries to generate data to cache,\n> >    die_error is called.  die_error turns off capturing, and prints to\n> >    STDOUT.  At the end of die_error there is jump to outer scope to\n> >    DONE_GITWEB; therefore $lock_fh goes out of scope and lockfile is\n> >    closed, and lock released.  Background process finishes.\n> > \n> >    Parent process runs 'generating_info' subroutine.  Now if it waits a\n> >    bit like in my rewrite before employing trick mentioned above, and\n> >    does not print anything if lockfile is released before startup delay,\n> >    _and_ die_error finishes within this delay, then everything is all\n> >    right: the error message is sent to client.\n> > \n> >    If die_error doesn't finish before git_generating_data_html prints\n> >    meta refresh, or there is no startup delay, then error pages would get\n> >    infinite redirects (remember: there is never anything in cache for\n> >    error pages).  This is a bad thing.\n> > \n> > \n> >    One possible solution for this problem (beside employing startup\n> >    delay) is to have tee output, i.e. print it as it is being captured.\n> >    Streamed (partial) response would serve as progress indicator for\n> >    process (re)generating cache; only parallel processes waiting for\n> >    cache would show 'generating_info'.\n> > \n> >    I think that in current implementation of capturing it would be as\n> >    simple as not closing STDOUT, but I'd have to check that.\n> \n> Actually in reading through this I thought of a better way to get the\n> error message cleanly passed from the backend process to the frontend\n> waiting processes.\n> \n> It's implemented in v8, but it basically boils down to, strangely\n> enough, caching the error. (I've got a hammer, it clearly solves all\n> problems!).  It's not a full cache (like what I'm doing with the rest of\n> the cache) but it basically generates a file that can be used to short\n> circuit things a bit.\n\nHmmm... I'll take a look at your solution then.  Looks like it can be\ninteresting.  I can think of capturing die_error output to in-memory\nfile (doing caching of its output in-memory), but I don't see how you\ncan change \"goto DONE_GITWEB\" (or in patch[1] sent to git mailing list\n\"goto DONE_REQUEST\") to exit action subroutine but not exit cache_fetch\nso gitweb would be able to display catched output.\n\n[1]: http://permalink.gmane.org/gmane.comp.version-control.git/162156\n     \"[PATCH/RFC] gitweb: Go to DONE_REQUEST rather than DONE_GITWEB in die_error\"\n\n> > 3. Using generic cache engine and memory consumption\n> > \n> >    Most Perl caching interfaces support only $cache->set($key, $data),\n> >    where $data is a Perl variable, and $data = $cache->get($key), or\n> >    their equivalents.  Even for file-based cache drivers you save from\n> >    memory and read into memory.  \n> > \n> >    The only exception I know of is Cache interface with Cache::File\n> >    driver, that provides $cache->handle($key [, $mode]), where optional\n> >    $mode argument can be any of mode strings that can be used in 'open'\n> >    function, or one of fopen(3) modes.  (Of course for memory-based or\n> >    network-based (like memcached) caches it might not make sense to\n> >    provide such interface).\n> > \n> >    The most often recommended capture module, Capture::Tiny, allows only\n> >    capturing into scalar (into memory) \n> > \n> >      $stdout = capture \\&code;\n> > \n> >    or (using its prototype)\n> >      \n> >      $stdout = capture { <code> };\n> > \n> >    Well, it is *::Tiny, so it supports minimal API.  From the list of\n> >    different capture modules, that allow capturing of Perl code output,\n> >    with different interfaces (API), in \"See also\" section of\n> >    Capture::Tiny documentation[3], only IO::CaptureOutput allow capturing\n> >    into specified file:\n> > \n> >      capture \\&code, \\$stdout, undef, $outfile;\n> > \n> >    [3] http://p3rl.org/Capture::Tiny\n> > \n> > \n> >    J.H.'s gitweb output caching v7 captures output directly ito cache\n> >    files.  The problem with doing it in my rewrite is to allow capturing\n> >    directly into cache entry file without losing ability to select\n> >    different caching engine, which might be not file-based (like\n> >    e.g. memcached-based).\n> \n> One of the big things that my caching engine now stops is the stampeding\n> herd problem, basically by locking everyone out of the cache for that\n> entry till it's prepared.  Assuming that can be preserved with any\n> arbitrary caching engine, then capturing directly to the file isn't that\n> big of a deal.\n\nWell, the recommended mechanism that one can use to prevent 'cache miss\nstampede' problem in CHI[2] caching interface (though not the more generic\n'stampeding herd' problem) is to use \"expires_variance\" option.  From\nCHI manpage:\n\n  expires_variance [FLOAT]\n\n     Controls the variable expiration feature, which allows items to expire\n     a little earlier than the stated expiration time to help prevent cache\n     miss stampedes.\n\n     Value is between 0.0 and 1.0, with 0.0 meaning that items expire\n     exactly when specified (feature is disabled), and 1.0 meaning that\n     items might expire anytime from now til the stated expiration time.\n     The default is 0.0. A setting of 0.10 to 0.25 would introduce a small\n     amount of variation without interfering too much with intended\n     expiration times.\n\n     The probability of expiration increases as a function of how far along\n     we are in the potential expiration window, with the probability being\n     near 0 at the beginning of the window and approaching 1 at the end.\n\n     [...]\n\n[2]: http://p3rl.org/CHI\n\n\nThough with locking we can for example lock whole cache instead of single\nentry when average load gets too high (but lower than $maxload, when gitweb\nreturns '503 - The load average on the server is too high' (custom version\nof '503 Service Unavailable' HTTP error code/status)).\n\n\nI think that CHI can be extended (via roles perhaps) to use locking:\nflock for 'File' driver, IPC::Lock::Memcached or similar for 'Memcached'\ndriver, get_and_set from Cache::FastMmap which IIUC uses fcntl to lock\nfragments of mmapped file for 'FastMmap' driver.\n\nCache::File driver from Cache package (http://p3rl.org/Cache) supports\nlocking by itself:\n\n  lock_level\n\n     Specify the level of locking to be used. There are three different\n     levels available:\n\n     Cache::File::LOCK_NONE()\n\n         No locking is performed. Useful when you can guarantee only one\n         process will be accessing the cache at a time.\n\n    Cache::File::LOCK_LOCAL()\n\n        Locking is performed, but it is not suitable for use over NFS\n        filesystems. However it is more efficient.\n\n    Cache::File::LOCK_NFS()\n\n        Locking is performed in a way that is suitable for use on NFS\n        filesystems.\n\n\n  [...]\n\n  locking efficiency\n\n    Currently LOCK_LOCAL is not implemented (if uses the same code as\n    LOCK_NFS).\n\n    There are two points of locking in Cache::File, index locking and\n    entry locking. The index locking is always exclusive and the lock\n    is required briefly during most operations. The entry locking is\n    either shared or exclusive and is also required during most operations.\n    When locking is enabled, File::NFSLock is used to provide the locking\n    for both situations. This is not overly efficient, especially as the\n    entry lock is only ever grabbed whilst the index lock is held.\n\n> If that can be preserved in your full re-write, then it doesn't really\n> matter how or when the data makes it into the caching system be it\n> memory, disk, /dev/random or collapsing quantum wave form\n\nIn my rewrite I use the same flock-based locking mechanism... just\nrewritten.  From the cover letter:\n\nJN> The main ideas in lifted from J.H. patches are the following\nJN> (features in common with \"Gitweb caching v7\" series by John Hawley):\nJN>\nJN> * caching captured output of gitweb in flat files, without any\nJN>   serialization (caching raw data)\nJN>\nJN> * using global (per-cache, not per-entry) expiration time, and\nJN>   using difference between mtime of cache file and current time\nJN>   for expiration\nJN>\nJN> * using file locking (flock) to prevent 'cache miss stampede'\nJN>   problem, i.e. to ensure that only one process is (re)generating\nJN>   cache entry\nJN>\nJN> * serving stale but not too old version, and regenerating data\nJN>   in background, to avoid waiting for data to be regenerated\nJN>\nJN> * progress info indicator based on http-equiv refresh trick\nJN>   (described in more detail how it works in the commit message)\nJN>\nJN> * capturing gitweb output by redirecting STDOUT to cache entry file\n\n\n> I'm going to try and get v8 out tomorrow morning, and I'm going to let\n> this stew on kernel.org overnight.  It's looking like 18 patches (which\n> includes Jakub's v7.2, since I was building straight on top of that).\n\nHmmm... I was thinking about doing rewrite of rewrite, keeping only\nparts which are actually used.  Instead of 22 patches it would be\n4-9 patches:\n\n  * capturing output to file or filehandle\n  * caching with locking, using 'compute_fh' interface\n  * module implementing cache_output subroutine, joining those two\n  * adding caching support to gitweb\n\nwhere the following patches might be extracted as additional patches\n\n  * adaptive cache lifetime\n  * zero-size file check\n  * \"Generating...\" page for cache and for gitweb\n\nperhaps also with separating\n\n  * serving stale data\n  * background cache generation\n \n> In poking around, I'll admit, I keep finding things I'd love to fix /\n> cleanup throughout the entirety of the code base.  I'm kinda hoping to\n> finally get caching dealt with and merged so I can start scratching\n> those other itches.  Mind you this has been 4+ years in the making\n> already...\n\nI am thinking about adding support for \"tee\" in addition to \"capture\",\ni.e. printing when capturing, and using it in cache...\n \n> - John 'Warthog9' Hawley\n> \n> P.S. Most of this e-mail was written a couple of weeks ago, I found it\n> in my drafts folder and wanted to get it out along with a note that I've\n> got v8 percolating. -JH\n\nThis email was written before v8 appeared on git mailing list...\n\n\nP.S. Did you had a chance to read mine rerwite of gitweb output caching\n\"[PATCHv6/RFC 00/24] gitweb: Simple file based output caching\"?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"157760","messageId":"20101209225211.GA20426@burratino","threadId":"25643","inReplyTo":"201012092330.06688.jnareb@gmail.com","subject":"Re: [RFC] Implementing gitweb output caching - issues to solve","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-09T22:52:11Z","receivedAt":"2010-12-09T22:52:11Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jakub Narebski wrote:\n\n> In my rewrite\n>\n>   [PATCHv6 17/24] gitweb: Show appropriate \"Generating...\" page when regenerating cache\n>   http://thread.gmane.org/gmane.comp.version-control.git/163052/focus=163040\n>   http://repo.or.cz/w/git/jnareb-git.git/commitdiff/48679f7985ccda16dc54fda97790841bab4a0ba2#patch1\n>\n> (see the browser_is_robot() subroutine:\n>\n>   http://repo.or.cz/w/git/jnareb-git.git/blob/48679f7985ccda16dc54fda97790841bab4a0ba2:/gitweb/gitweb.perl#l870\n>\n> I use HTTP::BrowserDetect package if available and it's ->robot() method.\n>\n> The fallback is to use *whitelist*, assuming that it would be better to\n> not show \"Generating...\" page rather than download the wrong thing.\n> I also guess that most (all?) web browsers use \"Mozilla compatibile\"\n> somewhere in their User-Agent string, thus matching 'Mozilla'.\n\nInteresting.  http://www.user-agents.org/ seems to suggest that many\nrobots do use Mozilla (though I don't think it's worth bending over\nbackwards to help them see the page correctly).\n\nHTTP::BrowserDetect uses a blacklist as far as I can tell.  Maybe in\nthe long term it would be nice to add a whitelist ->human() method.\n\nCc-ing Olaf Alders for ideas.\n"},{"id":"157790","messageId":"88CF82F1-0363-47B4-8C6F-AE4A2DA1714B@wundersolutions.com","threadId":"25643","inReplyTo":"20101209225211.GA20426@burratino","subject":"Re: [RFC] Implementing gitweb output caching - issues to solve","fromName":"Olaf Alders","fromEmail":"olaf@wundersolutions.com","sentAt":"2010-12-10T03:17:51Z","receivedAt":"2010-12-10T03:17:51Z","isPatch":false,"sender":{"key":"olaf@wundersolutions.com","avatar":null},"body":"\nOn 2010-12-09, at 5:52 PM, Jonathan Nieder wrote:\n\n> Jakub Narebski wrote:\n> \n>> In my rewrite\n>> \n>>  [PATCHv6 17/24] gitweb: Show appropriate \"Generating...\" page when regenerating cache\n>>  http://thread.gmane.org/gmane.comp.version-control.git/163052/focus=163040\n>>  http://repo.or.cz/w/git/jnareb-git.git/commitdiff/48679f7985ccda16dc54fda97790841bab4a0ba2#patch1\n>> \n>> (see the browser_is_robot() subroutine:\n>> \n>>  http://repo.or.cz/w/git/jnareb-git.git/blob/48679f7985ccda16dc54fda97790841bab4a0ba2:/gitweb/gitweb.perl#l870\n>> \n>> I use HTTP::BrowserDetect package if available and it's ->robot() method.\n>> \n>> The fallback is to use *whitelist*, assuming that it would be better to\n>> not show \"Generating...\" page rather than download the wrong thing.\n>> I also guess that most (all?) web browsers use \"Mozilla compatibile\"\n>> somewhere in their User-Agent string, thus matching 'Mozilla'.\n> \n> Interesting.  http://www.user-agents.org/ seems to suggest that many\n> robots do use Mozilla (though I don't think it's worth bending over\n> backwards to help them see the page correctly).\n> \n> HTTP::BrowserDetect uses a blacklist as far as I can tell.  Maybe in\n> the long term it would be nice to add a whitelist ->human() method.\n> \n> Cc-ing Olaf Alders for ideas.\n\nThanks for including me in this.  :)  I'm certainly open to patching the module, but I'm not 100% clear on how  you would want to implement this.  How is ->is_human different from !->is_robot?  To clarify, I should say that from the snippet above, I'm not 100% clear on what the problem is which needs to be solved.\n\nOlaf\n\n--\nOlaf Alders\nolaf@wundersolutions.com\n\nhttp://www.wundersolutions.com\nhttp://twitter.com/wundercounter\n\n866 503 2204 (Toll free - North America)\n416 944 8306 (direct)\n"},{"id":"157793","messageId":"20101210041144.GA28166@burratino","threadId":"25643","inReplyTo":"88CF82F1-0363-47B4-8C6F-AE4A2DA1714B@wundersolutions.com","subject":"Re: [RFC] Implementing gitweb output caching - issues to solve","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-10T04:11:44Z","receivedAt":"2010-12-10T04:11:44Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Olaf Alders wrote:\n> On 2010-12-09, at 5:52 PM, Jonathan Nieder wrote:\n\n>> HTTP::BrowserDetect uses a blacklist as far as I can tell.  Maybe in\n>> the long term it would be nice to add a whitelist ->human() method.\n>>\n>> Cc-ing Olaf Alders for ideas.\n>\n> Thanks for including me in this.  :)  I'm certainly open to patching\n> the module, but I'm not 100% clear on how  you would want to\n> implement this.  How is ->is_human different from !->is_robot?  To\n> clarify, I should say that from the snippet above, I'm not 100%\n> clear on what the problem is which needs to be solved.\n\nContext (sorry I did not include this in the first place):\n\nThe caching code (in development) for git's web interface uses a page\nthat says \"Generating...\" for cache misses, with an http refresh\nredirecting to the generated content.  The big downside is that if\ndone naively this breaks wget, curl, and similar user agents that are\nnot patient enough to grab the actual content instead of the redirect\npage.\n\nThe first solution tried was to explicitly special case wget and curl.\nBut in this case it is better to be more inclusive[2]; when in doubt,\nleave out the nice \"Generating...\" page and just serve the actual\ncontent slowly just in case.\n\nIn other words, the idea was that user agents fall into three\ncategories:\n\n A. definitely will not replace content with target of HTTP refresh\n B. definitely will replace content with target of HTTP refresh\n C. unknown\n\nand maybe ->is_robot could return true for A and ->is_human return\ntrue for B (leaving C as !->is_human && !->is_robot).  In this case,\nwe should show the \"Generating...\" page only in the ->is_human (B)\ncase.\n\nThat said, I know almost nothing on this subject, so it is likely\nthis analysis misses something.  J.H. or Jakub can likely say more.\n\nThanks,\nJonathan\n"},{"id":"157794","messageId":"4D01B0BF.1010609@eaglescrag.net","threadId":"25643","inReplyTo":"88CF82F1-0363-47B4-8C6F-AE4A2DA1714B@wundersolutions.com","subject":"Re: [RFC] Implementing gitweb output caching - issues to solve","fromName":"J.H.","fromEmail":"warthog9@eaglescrag.net","sentAt":"2010-12-10T04:46:55Z","receivedAt":"2010-12-10T04:46:55Z","isPatch":false,"sender":{"key":"warthog9@kernel.org","avatar":"https://avatars.githubusercontent.com/u/2334704?v=4"},"body":">> Interesting.  http://www.user-agents.org/ seems to suggest that many\n>> robots do use Mozilla (though I don't think it's worth bending over\n>> backwards to help them see the page correctly).\n\nIf a robot reports itself and we don't know about it, I'm fine with\ngiving it the 'Generating...' page as opposed to what it's expecting.\nThe number of robots and things of that nature that won't handle the\nmeta refresh are fewer than the number of people who will be clicking\nwith eyeballs on a screen.\n\n>> HTTP::BrowserDetect uses a blacklist as far as I can tell.  Maybe in\n>> the long term it would be nice to add a whitelist ->human() method.\n>>\n>> Cc-ing Olaf Alders for ideas.\n> \n> Thanks for including me in this.  :)  I'm certainly open to patching the module, but I'm not 100% clear on how  you would want to implement this.  How is ->is_human different from !->is_robot?  To clarify, I should say that from the snippet above, I'm not 100% clear on what the problem is which needs to be solved.\n\nAt this point I don't really see an issue with HTTP::BrowserDetect's\nrobot() function, and I agree with human = !->is_robot.\n\nOne thing I would like to see is the ability to do some sort of an add\nto the list of things to check for.  As you are probably aware there are\nmore agents that exist than what you have setup, I'm moving forward and\nhandling it with the following:\n\nsub is_dumb_client {\n        my($user_agent) = lc $ENV{'HTTP_USER_AGENT'};\n\n        my $browser_detect = HTTP::BrowserDetect->new($user_agent);\n\n        return 1 if ( $browser_detect->robot() );\n\n        foreach my $adc ( @additional_dumb_clients ) {\n                return 1 if ( index( $user_agent, lc $adc ) != -1 );\n        }\n\n        return 0;\n}\n\nwhich could be simplified if there was just some way to do\n\n        my($user_agent) = lc $ENV{'HTTP_USER_AGENT'};\n\n        my $browser_detect = HTTP::BrowserDetect->new($user_agent);\n\n        $browser_detect->add_robots( @array );\n\n        return 1 if ( $browser_detect->robot() );\n\nNot sure that particularly generalizes, and honestly it's only 4 lines\nof code to do add additional checks.\n\n- John 'Warthog9' Hawley\n"}]}