{"thread":{"id":"2128","subject":"gitweb.cgi","startedAt":"2005-10-18T02:57:22Z","lastAt":"2005-10-19T16:16:44Z","messageCount":15,"participants":["H. Peter Anvin","Kay Sievers","Brian Gerst","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"10197","messageId":"43546492.3020401@zytor.com","threadId":"2128","inReplyTo":null,"subject":"gitweb.cgi","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-18T02:57:22Z","receivedAt":"2005-10-18T02:57:22Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"It is increasingly clear that gitweb.cgi is producing an unacceptable \nload on the kernel.org servers.  Most of the hits we get are either the \ngitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth \nlike crazy.\n\nThis has become particularly painful during the current one-server outage.\n\nKay, gitweb really needs to be able to do caching, or be run behind a \ncaching proxy.  Otherwise I will have to turn it off until we can come \nup with a dedicated piece of server hardware for it.\n\n\t-hpa\n"},{"id":"10203","messageId":"20051018110725.GB6929@vrfy.org","threadId":"2128","inReplyTo":"43546492.3020401@zytor.com","subject":"Re: gitweb.cgi","fromName":"Kay Sievers","fromEmail":"kay.sievers@vrfy.org","sentAt":"2005-10-18T11:07:25Z","receivedAt":"2005-10-18T11:07:25Z","isPatch":false,"sender":{"key":"kay.sievers@vrfy.org","avatar":null},"body":"On Mon, Oct 17, 2005 at 07:57:22PM -0700, H. Peter Anvin wrote:\n> It is increasingly clear that gitweb.cgi is producing an unacceptable \n> load on the kernel.org servers.\n\nSure, sorry, was on 3 conferences in a row the last weeks.\n\n> Most of the hits we get are either the \n> gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth \n> like crazy.\n\nI tested some stuff on these boxes and 30 stat() calls alone take app. 2 seconds\non these boxes cause of I/O load ... :)\n\n> This has become particularly painful during the current one-server outage.\n> \n> Kay, gitweb really needs to be able to do caching, or be run behind a \n> caching proxy.  Otherwise I will have to turn it off until we can come \n> up with a dedicated piece of server hardware for it.\n\nHow about Apache's mod_cache? Worked nicely for me several times in other\nsetups.\n\nKay\n"},{"id":"10208","messageId":"4355283D.2000908@zytor.com","threadId":"2128","inReplyTo":"20051018110725.GB6929@vrfy.org","subject":"Re: gitweb.cgi","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-18T16:52:13Z","receivedAt":"2005-10-18T16:52:13Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Kay Sievers wrote:\n> \n>>This has become particularly painful during the current one-server outage.\n>>\n>>Kay, gitweb really needs to be able to do caching, or be run behind a \n>>caching proxy.  Otherwise I will have to turn it off until we can come \n>>up with a dedicated piece of server hardware for it.\n> \n> How about Apache's mod_cache? Worked nicely for me several times in other\n> setups.\n> \n\nI will look at it and see if I can make it work properly.\n\n\t-hpa\n"},{"id":"10209","messageId":"43552FC2.3000000@zytor.com","threadId":"2128","inReplyTo":"20051018110725.GB6929@vrfy.org","subject":"Re: gitweb.cgi","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-18T17:24:18Z","receivedAt":"2005-10-18T17:24:18Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Kay Sievers wrote:\n> \n>>Most of the hits we get are either the \n>>gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth \n>>like crazy.\n> \n> I tested some stuff on these boxes and 30 stat() calls alone take app. 2 seconds\n> on these boxes cause of I/O load ... :)\n> \n\nWelcome to my hell :)\n\nI set up mod_cache (which I didn't know about, silly me) and so far it \nseems to work and has produced a tremendous decrease in load and \nimprovement in response time.  I do, have, however, a request.  There \nare some gitweb pages which are more likely to change than others; in \nparticular, some gitweb pages will *never* change (because they directly \nreflect immutable git data.)\n\nIf gitweb could produce Last-Modified and Expires headers where \nappropriate, it should improve caching performance.\n\n\t-hpa\n"},{"id":"10221","messageId":"43555E99.1010902@didntduck.org","threadId":"2128","inReplyTo":"43552FC2.3000000@zytor.com","subject":"Re: gitweb.cgi","fromName":"Brian Gerst","fromEmail":"bgerst@didntduck.org","sentAt":"2005-10-18T20:44:09Z","receivedAt":"2005-10-18T20:44:09Z","isPatch":false,"sender":{"key":"bgerst@didntduck.org","avatar":null},"body":"H. Peter Anvin wrote:\n> Kay Sievers wrote:\n> \n>>\n>>> Most of the hits we get are either the gitweb front page or the \n>>> gitweb rss feeds, and it's eating I/O bandwidth like crazy.\n>>\n>>\n>> I tested some stuff on these boxes and 30 stat() calls alone take app. \n>> 2 seconds\n>> on these boxes cause of I/O load ... :)\n>>\n> \n> Welcome to my hell :)\n> \n> I set up mod_cache (which I didn't know about, silly me) and so far it \n> seems to work and has produced a tremendous decrease in load and \n> improvement in response time.  I do, have, however, a request.  There \n> are some gitweb pages which are more likely to change than others; in \n> particular, some gitweb pages will *never* change (because they directly \n> reflect immutable git data.)\n> \n> If gitweb could produce Last-Modified and Expires headers where \n> appropriate, it should improve caching performance.\n> \n>     -hpa\n\nSome other areas for improvement would be to seperate out the git icon \nand the style sheet into seperate static files.\n\n--\n\t\t\t\tBrian Gerst\n"},{"id":"10235","messageId":"Pine.LNX.4.64.0510181645200.3369@g5.osdl.org","threadId":"2128","inReplyTo":"43552FC2.3000000@zytor.com","subject":"Re: gitweb.cgi","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-19T00:27:53Z","receivedAt":"2005-10-19T00:27:53Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 18 Oct 2005, H. Peter Anvin wrote:\n> \n> I set up mod_cache (which I didn't know about, silly me) and so far it seems\n> to work and has produced a tremendous decrease in load and improvement in\n> response time.\n\nIt really doesn't work very well with the \"front page\" though..\n\nDoing a \"Save page as..\" shows that it's not a huge page: it's roughly 700 \nlines long and 57kB in size, but pressing the reload button (or just going \nsomewhere else and coming back immediately) takes 45 seconds to reload for \nme.\n\nTrying again shows that it _is_ cached if you press the reload button \nimmediately again, but I haven't quite figured out how long the cache \ntimeout is. It seems to be around one minute (from some very preliminary \ntests it's more than 25 seconds, but less than a minute and a half).\n\nConsidering that apparently the load is enough that it takes 45 seconds to \ngenerate (scary in itself), is should clearly be cached for more than one \nminute. More like ten minutes or half an hour, especially since mirroring \nany content changes takes longer than that anyway.\n\nNow, I suspect all the content on kernel.org could easily be cached for \nten minutes.\n\nAs far as I can tell, mod_cache without any expiry information uses\n\n\tCacheDefaultExpire \n\nwhich should default to one hour according to the docs. Have you changed \nthat to one minute? Maybe making it 10 minutes would be better?\n\nThat said, I tried to figure out how the front page is generated, but \nhaven't quite. Can somebody (Kay?) please say what it does most, and I can \ntry to make sure git does that efficiently.. \n\n\t\t\tLinus\n"},{"id":"10237","messageId":"43559399.2030903@zytor.com","threadId":"2128","inReplyTo":"Pine.LNX.4.64.0510181645200.3369@g5.osdl.org","subject":"Re: gitweb.cgi","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-19T00:30:17Z","receivedAt":"2005-10-19T00:30:17Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Linus Torvalds wrote:\n> \n> It really doesn't work very well with the \"front page\" though..\n> \n> Doing a \"Save page as..\" shows that it's not a huge page: it's roughly 700 \n> lines long and 57kB in size, but pressing the reload button (or just going \n> somewhere else and coming back immediately) takes 45 seconds to reload for \n> me.\n> \n> Trying again shows that it _is_ cached if you press the reload button \n> immediately again, but I haven't quite figured out how long the cache \n> timeout is. It seems to be around one minute (from some very preliminary \n> tests it's more than 25 seconds, but less than a minute and a half).\n> \n\nThe cache timeout is set to 300 seconds, however, that's per server, of \ncourse.\n\n> Considering that apparently the load is enough that it takes 45 seconds to \n> generate (scary in itself), is should clearly be cached for more than one \n> minute. More like ten minutes or half an hour, especially since mirroring \n> any content changes takes longer than that anyway.\n\nThe latency for an I/O operation on the kernel.org servers is positively \nscary.\n\n\t-hpa\n"},{"id":"10238","messageId":"43559575.1060902@zytor.com","threadId":"2128","inReplyTo":"Pine.LNX.4.64.0510181645200.3369@g5.osdl.org","subject":"Re: gitweb.cgi","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-19T00:38:13Z","receivedAt":"2005-10-19T00:38:13Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Linus Torvalds wrote:\n> \n> It really doesn't work very well with the \"front page\" though..\n> \n> Doing a \"Save page as..\" shows that it's not a huge page: it's roughly 700 \n> lines long and 57kB in size, but pressing the reload button (or just going \n> somewhere else and coming back immediately) takes 45 seconds to reload for \n> me.\n> \n> Trying again shows that it _is_ cached if you press the reload button \n> immediately again, but I haven't quite figured out how long the cache \n> timeout is. It seems to be around one minute (from some very preliminary \n> tests it's more than 25 seconds, but less than a minute and a half).\n> \n\nIt turns out that the default CacheSize is only 256K.  D'oh!  Fixed.\n\nI also changed the CacheDefaultExpire to 600 seconds.\n\n> That said, I tried to figure out how the front page is generated, but \n> haven't quite. Can somebody (Kay?) please say what it does most, and I can \n> try to make sure git does that efficiently.. \n\nThe only thing the front page really should need is to know when the \nlast change to the tree was, which presumably means looking at each head \nof each tree and follow the chain until there is a datable object.\n\n\t-hpa\n"},{"id":"10241","messageId":"Pine.LNX.4.64.0510181742070.3369@g5.osdl.org","threadId":"2128","inReplyTo":"43559399.2030903@zytor.com","subject":"Optimize common case of git-rev-list (was Re: gitweb.cgi)","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-19T00:53:11Z","receivedAt":"2005-10-19T00:53:11Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 18 Oct 2005, H. Peter Anvin wrote:\n> \n> > Considering that apparently the load is enough that it takes 45 seconds to\n> > generate (scary in itself), is should clearly be cached for more than one\n> > minute. More like ten minutes or half an hour, especially since mirroring\n> > any content changes takes longer than that anyway.\n> \n> The latency for an I/O operation on the kernel.org servers is positively\n> scary.\n\nI took a look at webgit, and it looks like at least for the \"projects\" \npage, the most common operation ends up being basically\n\n\tgit-rev-list --header --parents --max-count=1 HEAD\n\nNow, the thing is, the way \"git-rev-list\" works, it always keeps on \npopping the parents and parsing them in order to build the list of \nparents, and it turns out that even though we just want a single commit, \ngit-rev-list will invariably look up _three_ generations of commits.\n\nIt will parse:\n - the commit we want (it obviously needs this)\n - it's parent(s) as part of the \"pop_most_recent_commit()\" logic\n - it will then pop one of the parents before it notices that it doesn't \n   need any more\n - and as part of popping the parent, it will parse the grandparent (again \n   due to \"pop_most_recent_commit()\".\n\nNow, I've strace'd it, and it really is pretty efficient on the whole, but \nif things aren't nicely cached, and with long-latency IO, doing those two \nextra objects (at a minimum - if the parent is a merge it will be more) is \njust wasted time, and potentially a lot of it.\n\nSo here's a quick special-case for the trivial case of \"just one commit, \nand no date-limits or other special rules\".\n\nSigned-off-by: Linus Torvalds <torvalds@osdl.org>\n---\n\nI've actually tried to test it (but hey, \"exhaustive\" is hard with all the \ndifferent options), and I tried to be very careful to only do the \nspecial-case when really nothing can go wrong, and this all looks obvious.\n\nBut buyer beware.\n\nBtw, doing an \"strace\" on git-rev-list shows that most of the system calls \nby far are the dynamic loader. Now, those accesses _should_ all be cached, \nso they should be fast and low-latency, but it's entirely possible that \nfor a server configuration you might want to actually link things \nstatically. Or not. Just a thought.\n\ndiff --git a/rev-list.c b/rev-list.c\nindex c60aa72..d4da1bd 100644\n--- a/rev-list.c\n+++ b/rev-list.c\n@@ -624,6 +624,10 @@ int main(int argc, char **argv)\n \n \tif (!merge_order) {\t\t\n \t\tsort_by_date(&list);\n+\t\tif (list && !limited && max_count == 1) {\n+\t\t\tshow_commit(list->item);\n+\t\t\treturn 0;\n+\t\t}\n \t        if (limited)\n \t\t\tlist = limit_list(list);\n \t\tif (topo_order)\n"},{"id":"10243","messageId":"Pine.LNX.4.64.0510181753340.3369@g5.osdl.org","threadId":"2128","inReplyTo":"43559575.1060902@zytor.com","subject":"Re: gitweb.cgi","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-19T01:02:29Z","receivedAt":"2005-10-19T01:02:29Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 18 Oct 2005, H. Peter Anvin wrote:\n> \n> It turns out that the default CacheSize is only 256K.  D'oh!  Fixed.\n> \n> I also changed the CacheDefaultExpire to 600 seconds.\n\nOk, that sounds like it should improve things. My quick tests didn't seem \nto show any difference, though. Do you need to re-load the apache module \nor something?\n\n> The only thing the front page really should need is to know when the last\n> change to the tree was, which presumably means looking at each head of each\n> tree and follow the chain until there is a datable object.\n\nYeah. I tried to follow gitweb.cgi, but I'm neither http- nor \nperl-literate, so I'm not sure I caught everything.\n\nBut it does seem to basically end up doing a \"git_read_commit()\" for each \nproject, and that in turn was doing the \"git-rev-list --max-count=1\" thing \nthat I just sent out a suggested improvement for.\n\nIt effectively removes two or more copies of\n\n\tstat64(\"/objects/xy/zzy\", {...})\n\tfd = open(\"objects/xy/zzy\", O_RDONLY|O_NOATIME)\n\taddr = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0)\n\tclose(fd)\n\tmunmap(addr, size)\n\nwhich really should be very cheap operations, but hey, if the disk head is \nsomewhere else (and busy) and it's not cached, it can be quite expensive. \nEspecially since we don't end up usign the result.\n\nI'm sure there's room for improvement inside gitweb itself too, but maybe \nthe git-rev-list optimization will help.\n\n\t\tLinus\n"},{"id":"10246","messageId":"43559DFE.7060503@zytor.com","threadId":"2128","inReplyTo":"Pine.LNX.4.64.0510181753340.3369@g5.osdl.org","subject":"Re: gitweb.cgi","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-19T01:14:38Z","receivedAt":"2005-10-19T01:14:38Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Linus Torvalds wrote:\n> \n> On Tue, 18 Oct 2005, H. Peter Anvin wrote:\n> \n>>It turns out that the default CacheSize is only 256K.  D'oh!  Fixed.\n>>\n>>I also changed the CacheDefaultExpire to 600 seconds.\n> \n> \n> Ok, that sounds like it should improve things. My quick tests didn't seem \n> to show any difference, though. Do you need to re-load the apache module \n> or something?\n> \n\nYes, but I did that.  It seems very strange when something hits the \ncache.  A cgi script can apparently be run quite a few number of times \nbefore mod_cache sees it globally.\n\n\t-hpa\n"},{"id":"10249","messageId":"20051019012341.GA15256@vrfy.org","threadId":"2128","inReplyTo":"Pine.LNX.4.64.0510181753340.3369@g5.osdl.org","subject":"Re: gitweb.cgi","fromName":"Kay Sievers","fromEmail":"kay.sievers@vrfy.org","sentAt":"2005-10-19T01:23:41Z","receivedAt":"2005-10-19T01:23:41Z","isPatch":false,"sender":{"key":"kay.sievers@vrfy.org","avatar":null},"body":"On Tue, Oct 18, 2005 at 06:02:29PM -0700, Linus Torvalds wrote:\n> \n> \n> On Tue, 18 Oct 2005, H. Peter Anvin wrote:\n> > \n> > It turns out that the default CacheSize is only 256K.  D'oh!  Fixed.\n> > \n> > I also changed the CacheDefaultExpire to 600 seconds.\n> \n> Ok, that sounds like it should improve things. My quick tests didn't seem \n> to show any difference, though. Do you need to re-load the apache module \n> or something?\n> \n> > The only thing the front page really should need is to know when the last\n> > change to the tree was, which presumably means looking at each head of each\n> > tree and follow the chain until there is a datable object.\n> \n> Yeah. I tried to follow gitweb.cgi, but I'm neither http- nor \n> perl-literate, so I'm not sure I caught everything.\n> \n> But it does seem to basically end up doing a \"git_read_commit()\" for each \n> project, and that in turn was doing the \"git-rev-list --max-count=1\" thing \n> that I just sent out a suggested improvement for.\n> \n> It effectively removes two or more copies of\n> \n> \tstat64(\"/objects/xy/zzy\", {...})\n> \tfd = open(\"objects/xy/zzy\", O_RDONLY|O_NOATIME)\n> \taddr = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0)\n> \tclose(fd)\n> \tmunmap(addr, size)\n> \n> which really should be very cheap operations, but hey, if the disk head is \n> somewhere else (and busy) and it's not cached, it can be quite expensive. \n> Especially since we don't end up usign the result.\n> \n> I'm sure there's room for improvement inside gitweb itself too, but maybe \n> the git-rev-list optimization will help.\n\nThere definitely is! But I tried a single \"stat() all HEAD files\" with a\nsimple script and it took more than 3 seconds for the 80 trees. Then I\ngave up \"optimizing\" and was sure we want to have a single-file cached front\npage instead. :)\n\nKay\n"},{"id":"10250","messageId":"20051019013321.GA10331@vrfy.org","threadId":"2128","inReplyTo":"43552FC2.3000000@zytor.com","subject":"Re: gitweb.cgi","fromName":"Kay Sievers","fromEmail":"kay.sievers@vrfy.org","sentAt":"2005-10-19T01:33:21Z","receivedAt":"2005-10-19T01:33:21Z","isPatch":false,"sender":{"key":"kay.sievers@vrfy.org","avatar":null},"body":"On Tue, Oct 18, 2005 at 10:24:18AM -0700, H. Peter Anvin wrote:\n> Kay Sievers wrote:\n> >\n> >>Most of the hits we get are either the \n> >>gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth \n> >>like crazy.\n> >\n> >I tested some stuff on these boxes and 30 stat() calls alone take app. 2 \n> >seconds\n> >on these boxes cause of I/O load ... :)\n> >\n> \n> Welcome to my hell :)\n\nYeah, I get an idea now :)\n\n> I set up mod_cache (which I didn't know about, silly me) and so far it \n> seems to work and has produced a tremendous decrease in load and \n> improvement in response time.\n\nGreat! Hope that will work.\n\n> I do, have, however, a request.  There \n> are some gitweb pages which are more likely to change than others; in \n> particular, some gitweb pages will *never* change (because they directly \n> reflect immutable git data.)\n\nYes, makes sense.\n\n> If gitweb could produce Last-Modified and Expires headers where \n> appropriate, it should improve caching performance.\n\nI've added the Expires: header to the commit and commitdiff pages with\none whole day ahead. Let's see if that will help...\n\nKay\n"},{"id":"10281","messageId":"Pine.LNX.4.64.0510190655230.3369@g5.osdl.org","threadId":"2128","inReplyTo":"43559DFE.7060503@zytor.com","subject":"Re: gitweb.cgi","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-19T13:59:28Z","receivedAt":"2005-10-19T13:59:28Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 18 Oct 2005, H. Peter Anvin wrote:\n> \n> Yes, but I did that.  It seems very strange when something hits the cache.  A\n> cgi script can apparently be run quite a few number of times before mod_cache\n> sees it globally.\n\nWell, I tried again this morning, and _boy_ is it better. The \"projects\" \nthing came up immediately, no waiting. \n\nMaybe it just took a while for the mod_cache thing to take effect as \nApache swarm members time out? \n\nOr maybe it's just that the west coast is really quiet at 7AM.\n\n\t\tLinus\n"},{"id":"10286","messageId":"4356716C.7080003@zytor.com","threadId":"2128","inReplyTo":"Pine.LNX.4.64.0510190655230.3369@g5.osdl.org","subject":"Re: gitweb.cgi","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2005-10-19T16:16:44Z","receivedAt":"2005-10-19T16:16:44Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Linus Torvalds wrote:\n> \n> Well, I tried again this morning, and _boy_ is it better. The \"projects\" \n> thing came up immediately, no waiting. \n> \n> Maybe it just took a while for the mod_cache thing to take effect as \n> Apache swarm members time out? \n> \n\nNo, that's not it.  It takes killing httpd and restarting it.  Rather, \nit looks like mod_cache in different swarm members doesn't always \ncommunicate instantly, which is also underscored by the fact that cache \nfiles don't appear into the filesystem until after a while.\n\n\t-hpa\n"}]}