From: H. Peter Anvin Date: Sat, 09 Dec 2006 16:26:15 GMT Subject: Re: kernel.org mirroring (Re: [GIT PULL] MMC update) Message-ID: <457AE3A7.4080802@zytor.com> In-Reply-To: <46a038f90612082134x38be9c8dgca6fe60c087bf100@mail.gmail.com> Martin Langhoff wrote: > On 12/9/06, H. Peter Anvin wrote: >> Martin Langhoff wrote: >> > I posted separately about those. And I've been mulling about whether >> > the thundering herd is really such a big problem that we need to >> > address it head-on. >> >> Uhm... yes it is. > > Got some more info, discussion points or links to stuff I should read > to appreciate why that is? I am trying to articulate why I consider it > is not a high-payoff task, as well as describing how to tackle it. > > To recap, the reasons it is not high payoff is that: > > - the main benefit comes from being cacheable and able to revalidate > the cache cheaply (with the ETags-based strategy discussed above) > - highly distributed caches/proxies means we'll seldom see a true > cold cache situation > - we have a huge set of URLs which are seldom hit, and will never see > a thundering anything > - we have a tiny set of very popular URLs that are the key target for > the thundering herd - (projects page, summary page, shortlog, fulllog) > - but those are in the clear as soon as the caches are populated > > Why do we have to take it head-on? :-) > Because the primary failure scenario is timeout on the common queries due to excess parallel invocations under high I/O load resulting in catastrophic failure.