Re: [RFC] Implementing gitweb output caching - issues to solve

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Dec 10, 2010, 04:11 UTC
Message-ID: <20101210041144.GA28166@burratino>
In-Reply-To: <88CF82F1-0363-47B4-8C6F-AE4A2DA1714B@wundersolutions.com>

Olaf Alders wrote:

> On 2010-12-09, at 5:52 PM, Jonathan Nieder wrote:

Show 10 quoted lines

>> HTTP::BrowserDetect uses a blacklist as far as I can tell.  Maybe in
>> the long term it would be nice to add a whitelist ->human() method.
>>
>> Cc-ing Olaf Alders for ideas.
>
> Thanks for including me in this.  :)  I'm certainly open to patching
> the module, but I'm not 100% clear on how  you would want to
> implement this.  How is ->is_human different from !->is_robot?  To
> clarify, I should say that from the snippet above, I'm not 100%
> clear on what the problem is which needs to be solved.

Context (sorry I did not include this in the first place):

The caching code (in development) for git's web interface uses a page that says "Generating..." for cache misses, with an http refresh redirecting to the generated content. The big downside is that if done naively this breaks wget, curl, and similar user agents that are not patient enough to grab the actual content instead of the redirect page.

The first solution tried was to explicitly special case wget and curl. But in this case it is better to be more inclusive[2]; when in doubt, leave out the nice "Generating..." page and just serve the actual content slowly just in case.

In other words, the idea was that user agents fall into three categories:

 A. definitely will not replace content with target of HTTP refresh
 B. definitely will replace content with target of HTTP refresh
 C. unknown

and maybe ->is_robot could return true for A and ->is_human return true for B (leaving C as !->is_human && !->is_robot). In this case, we should show the "Generating..." page only in the ->is_human (B) case.

That said, I know almost nothing on this subject, so it is likely this analysis misses something. J.H. or Jakub can likely say more.

Thanks, Jonathan