Re: [RFC] Implementing gitweb output caching - issues to solve

From: J.H. <warthog9@eaglescrag.net>
Date: Dec 10, 2010, 04:46 UTC
Message-ID: <4D01B0BF.1010609@eaglescrag.net>
In-Reply-To: <88CF82F1-0363-47B4-8C6F-AE4A2DA1714B@wundersolutions.com>

>> Interesting.  http://www.user-agents.org/ seems to suggest that many
>> robots do use Mozilla (though I don't think it's worth bending over
>> backwards to help them see the page correctly).

If a robot reports itself and we don't know about it, I'm fine with giving it the 'Generating...' page as opposed to what it's expecting. The number of robots and things of that nature that won't handle the meta refresh are fewer than the number of people who will be clicking with eyeballs on a screen.

Show 6 quoted lines

>> HTTP::BrowserDetect uses a blacklist as far as I can tell.  Maybe in
>> the long term it would be nice to add a whitelist ->human() method.
>>
>> Cc-ing Olaf Alders for ideas.
> 
> Thanks for including me in this.  :)  I'm certainly open to patching the module, but I'm not 100% clear on how  you would want to implement this.  How is ->is_human different from !->is_robot?  To clarify, I should say that from the snippet above, I'm not 100% clear on what the problem is which needs to be solved.

At this point I don't really see an issue with HTTP::BrowserDetect's robot() function, and I agree with human = !->is_robot.

One thing I would like to see is the ability to do some sort of an add to the list of things to check for. As you are probably aware there are more agents that exist than what you have setup, I'm moving forward and handling it with the following:

sub is_dumb_client {
        my($user_agent) = lc $ENV{'HTTP_USER_AGENT'};

        my $browser_detect = HTTP::BrowserDetect->new($user_agent);

        return 1 if ( $browser_detect->robot() );

        foreach my $adc ( @additional_dumb_clients ) {
                return 1 if ( index( $user_agent, lc $adc ) != -1 );
        }

        return 0;
}

which could be simplified if there was just some way to do

        my($user_agent) = lc $ENV{'HTTP_USER_AGENT'};

        my $browser_detect = HTTP::BrowserDetect->new($user_agent);

        $browser_detect->add_robots( @array );

        return 1 if ( $browser_detect->robot() );

Not sure that particularly generalizes, and honestly it's only 4 lines of code to do add additional checks.

- John 'Warthog9' Hawley