{"thread":{"id":"43694","subject":"Re: kernel.org mirroring (Re: [GIT PULL] MMC update)","startedAt":"2006-12-11T03:40:18Z","lastAt":"2006-12-11T09:30:04Z","messageCount":2,"participants":["linux@horizon.com","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"296776","messageId":"20061211034018.16937.qmail@science.horizon.com","threadId":"43694","inReplyTo":null,"subject":"Re: kernel.org mirroring (Re: [GIT PULL] MMC update)","fromName":"","fromEmail":"linux@horizon.com","sentAt":"2006-12-11T03:40:18Z","receivedAt":"2006-12-11T03:40:18Z","isPatch":false,"sender":{"key":"linux@horizon.com","avatar":null},"body":">>> I posted separately about those. And I've been mulling about whether\n>>> the thundering herd is really such a big problem that we need to\n>>> address it head-on.\n>>\n>> Uhm... yes it is.\n> \n> Got some more info, discussion points or links to stuff I should read\n> to appreciate why that is? I am trying to articulate why I consider it\n> is not a high-payoff task, as well as describing how to tackle it.\n> \n> To recap, the reasons it is not high payoff is that:\n>\n>  - the main benefit comes from being cacheable and able to revalidate\n>    the cache cheaply (with the ETags-based strategy discussed above)\n>  - highly distributed caches/proxies means we'll seldom see a true\n>    cold cache situation\n>  - we have a huge set of URLs which are seldom hit, and will never see\n>    a thundering anything\n>  - we have a tiny set of very popular URLs that are the key target for\n>    the thundering herd - (projects page, summary page, shortlog, fulllog)\n>  - but those are in the clear as soon as the caches are populated\n> \n> Why do we have to take it head-on? :-)\n\nI think I agree with you, but not as strongly.  Certainly, having any\nkind of effective cacheing (heck, just comparing the timestamp of the\nrelevant ref(s) with the If-Modified-Since: header) will help kernel.org\nenormously.\n\nBut as soon as there's a push, particularly a release push, that\ninvalidates *all* of the popular pages *and* the thindering herd arrives.\n\nThe result is that all of the popular \"what's new?\" summary pages get\nfetched 15 times in parallel and, because the front end doesn't serialize\nthem, populating the caches can be a painful process involving a lot of\nrepeated work.\n\nI tend to agree that for the basic project summary pages, generating them\npreemptively as static pages out of the push script seems best.\n(\"find /usr/src/linux -type d -print | wc -l\" is 1492.  Dear me.\nOh!  There is no per-directory shortlog page; that simplifies things.\nBut there *should* be.)\n\nThe only tricky thing is the \"n minutes/hours/days ago\" timestamps.\nBasically, you want to generate a half-formatted, indefinitely-cacheable\npage that contains them as absolute timestamps, and a have system for\nregenerating the fully-formatted page from that (and the current time).\n\nThe ideas that people have been posting seem excellent.  Give a page\ntwo timeouts.  If a GET arrives before the first timestamp, and no\nprerequisites have changes, it's served directly from cache.  If it\narrives after the second timeout, or the prerequisites have changed,\nit blocks until the page is regenerated.  But if it arrives between\nthose two times, it serves the stale data and starts generating fresh\ndata in the background.\n\nSo for the fully-formed timestamps, the first timeout is when the next\nhuman-readable timestamp on the page ticks over.  But the second timeout\ncan be past that by, say, 5% of the timeout value.  It's okay to display\n\"3 hours ago\" until 12 minutes past the 4 hour mark.\n\nIt might be okay to allow even the prerequisites to be slightly stale when\nserving old data; it's okay if it takes 30 seconds for the kernel.org\nweb page to notice that Linus pushed.  But on my office gitweb, I'm not\nsure that it's okay to take 30 seconds to notice that *I* just pushed.\n(I'm also not sure about consistency issues.  If I link from one page\nthat shows the new release to another, it would be a bit disconcerting\nif it disappeared.)\n\n\nThe nasty problem with built-in cacheing is that you need a whole cache\nreclaim infrastructure; it would be so much nicer to let Squid deal\nwith that whole mess.  But it can't deal with anything other than fully\n"},{"id":"297768","messageId":"elj8af$32n$1@sea.gmane.org","threadId":"43694","inReplyTo":"20061211034018.16937.qmail@science.horizon.com","subject":"Re: kernel.org mirroring (Re: [GIT PULL] MMC update)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-11T09:30:04Z","receivedAt":"2006-12-11T09:30:04Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Side note: I wonder why this mail is not attached to the rest of thread in\nthe gmane.comp.version-control.git news/Usenet GMane mirrot of git mailing\nlist.\n\nlinux@horizon.com wrote:\n\n> Oh!  There is no per-directory shortlog page; that simplifies things.\n> But there *should* be.)\n\nThere is. It is called \"history\" (and currently we have only shortlog-like\nview for history and no log-like view).\n\n> The only tricky thing is the \"n minutes/hours/days ago\" timestamps.\n> Basically, you want to generate a half-formatted, indefinitely-cacheable\n> page that contains them as absolute timestamps, and a have system for\n> regenerating the fully-formatted page from that (and the current time).\n\nOr use ECMAScript (JavaScript) for that. I plan (if this feature is\nrequested) to make it a %feature.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"}]}