From: Martin Langhoff <martin.langhoff@gmail.com>
Date: Sat, 09 Dec 2006 02:52:09 GMT
Subject: Re: kernel.org mirroring (Re: [GIT PULL] MMC update)
Message-ID: <46a038f90612081852u63e05da1qe57504636f3578fd@mail.gmail.com>
In-Reply-To: <457A1962.6000401@zytor.com>

On 12/9/06, H. Peter Anvin <hpa@zytor.com> wrote:
> Martin Langhoff wrote:
> > On 12/9/06, Jakub Narebski <jnareb@gmail.com> wrote:
> >> Perhaps gitweb should generate it's own ETag instead of messing with
> >> 'expires' header?
> >
> > That'll be the winning solution.
>
> Doesn't solve the thundering herd problem or the timeout problem at all,
> though.

I posted separately about those. And I've been mulling about whether
the thundering herd is really such a big problem that we need to
address it head-on. If we doHTTP  caching headers right (that is, a
bit better than now) then the fact that web caches are distributed
means that even a cache restart or cache invalidation won't trigger a
thundering herd.

And gitweb rarely has a "new" URL that gets a ton of hits immediately.
Our real problem is the summary page, and the fact that we aren't
setting an effecting ETag there. If we do, a front-end cache plus the
ability to revalidate the ETag cheaply will get us through.

We get 99% of the benefit from ETags and cheap revalidations,
specially if they are coupled with a reverse caching proxy,. The
remaining 1% of dealing with the highly infrequent thundering herd can
be addressed with the scheme I've posted 5 minutes ago.

cheers