gitweb.cgi

It is increasingly clear that gitweb.cgi is producing an unacceptable load on the kernel.org servers. Most of the hits we get are either the gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth like crazy.

This has become particularly painful during the current one-server outage.

Kay, gitweb really needs to be able to do caching, or be run behind a caching proxy. Otherwise I will have to turn it off until we can come up with a dedicated piece of server hardware for it.

	-hpa

On Mon, Oct 17, 2005 at 07:57:22PM -0700, H. Peter Anvin wrote:

Sure, sorry, was on 3 conferences in a row the last weeks.

I tested some stuff on these boxes and 30 stat() calls alone take app. 2 seconds on these boxes cause of I/O load ... :)

How about Apache's mod_cache? Worked nicely for me several times in other setups.

Kay

Kay Sievers wrote:

I will look at it and see if I can make it work properly.

	-hpa

Kay Sievers wrote:

Welcome to my hell :)

I set up mod_cache (which I didn't know about, silly me) and so far it seems to work and has produced a tremendous decrease in load and improvement in response time. I do, have, however, a request. There are some gitweb pages which are more likely to change than others; in particular, some gitweb pages will *never* change (because they directly reflect immutable git data.)

If gitweb could produce Last-Modified and Expires headers where appropriate, it should improve caching performance.

	-hpa

H. Peter Anvin wrote:

Some other areas for improvement would be to seperate out the git icon and the style sheet into seperate static files.

--
				Brian Gerst

On Tue, 18 Oct 2005, H. Peter Anvin wrote:

It really doesn't work very well with the "front page" though..

Doing a "Save page as.." shows that it's not a huge page: it's roughly 700 lines long and 57kB in size, but pressing the reload button (or just going somewhere else and coming back immediately) takes 45 seconds to reload for me.

Trying again shows that it _is_ cached if you press the reload button immediately again, but I haven't quite figured out how long the cache timeout is. It seems to be around one minute (from some very preliminary tests it's more than 25 seconds, but less than a minute and a half).

Considering that apparently the load is enough that it takes 45 seconds to generate (scary in itself), is should clearly be cached for more than one minute. More like ten minutes or half an hour, especially since mirroring any content changes takes longer than that anyway.

Now, I suspect all the content on kernel.org could easily be cached for ten minutes.

As far as I can tell, mod_cache without any expiry information uses

	CacheDefaultExpire 

which should default to one hour according to the docs. Have you changed that to one minute? Maybe making it 10 minutes would be better?

That said, I tried to figure out how the front page is generated, but haven't quite. Can somebody (Kay?) please say what it does most, and I can try to make sure git does that efficiently..

			Linus

Linus Torvalds wrote:

The cache timeout is set to 300 seconds, however, that's per server, of course.

The latency for an I/O operation on the kernel.org servers is positively scary.

	-hpa

On Tue, 18 Oct 2005, H. Peter Anvin wrote:

I took a look at webgit, and it looks like at least for the "projects" page, the most common operation ends up being basically

	git-rev-list --header --parents --max-count=1 HEAD

Now, the thing is, the way "git-rev-list" works, it always keeps on popping the parents and parsing them in order to build the list of parents, and it turns out that even though we just want a single commit, git-rev-list will invariably look up _three_ generations of commits.

It will parse:
 - the commit we want (it obviously needs this)
 - it's parent(s) as part of the "pop_most_recent_commit()" logic
 - it will then pop one of the parents before it notices that it doesn't 
   need any more
 - and as part of popping the parent, it will parse the grandparent (again 
   due to "pop_most_recent_commit()".

Now, I've strace'd it, and it really is pretty efficient on the whole, but if things aren't nicely cached, and with long-latency IO, doing those two extra objects (at a minimum - if the parent is a merge it will be more) is just wasted time, and potentially a lot of it.

So here's a quick special-case for the trivial case of "just one commit, and no date-limits or other special rules".

Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---

I've actually tried to test it (but hey, "exhaustive" is hard with all the different options), and I tried to be very careful to only do the special-case when really nothing can go wrong, and this all looks obvious.

But buyer beware.

Btw, doing an "strace" on git-rev-list shows that most of the system calls by far are the dynamic loader. Now, those accesses _should_ all be cached, so they should be fast and low-latency, but it's entirely possible that for a server configuration you might want to actually link things statically. Or not. Just a thought.

Linus Torvalds wrote:

It turns out that the default CacheSize is only 256K.  D'oh!  Fixed.

I also changed the CacheDefaultExpire to 600 seconds.

The only thing the front page really should need is to know when the last change to the tree was, which presumably means looking at each head of each tree and follow the chain until there is a datable object.

	-hpa

On Tue, 18 Oct 2005, H. Peter Anvin wrote:

Ok, that sounds like it should improve things. My quick tests didn't seem to show any difference, though. Do you need to re-load the apache module or something?

Yeah. I tried to follow gitweb.cgi, but I'm neither http- nor perl-literate, so I'm not sure I caught everything.

But it does seem to basically end up doing a "git_read_commit()" for each project, and that in turn was doing the "git-rev-list --max-count=1" thing that I just sent out a suggested improvement for.

It effectively removes two or more copies of

	stat64("/objects/xy/zzy", {...})
	fd = open("objects/xy/zzy", O_RDONLY|O_NOATIME)
	addr = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0)
	close(fd)
	munmap(addr, size)

which really should be very cheap operations, but hey, if the disk head is somewhere else (and busy) and it's not cached, it can be quite expensive. Especially since we don't end up usign the result.

I'm sure there's room for improvement inside gitweb itself too, but maybe the git-rev-list optimization will help.

		Linus

Linus Torvalds wrote:

Yes, but I did that. It seems very strange when something hits the cache. A cgi script can apparently be run quite a few number of times before mod_cache sees it globally.

	-hpa

On Tue, 18 Oct 2005, H. Peter Anvin wrote:

Well, I tried again this morning, and _boy_ is it better. The "projects" thing came up immediately, no waiting.

Maybe it just took a while for the mod_cache thing to take effect as Apache swarm members time out?

Or maybe it's just that the west coast is really quiet at 7AM.

		Linus

Linus Torvalds wrote:

No, that's not it. It takes killing httpd and restarting it. Rather, it looks like mod_cache in different swarm members doesn't always communicate instantly, which is also underscored by the fact that cache files don't appear into the filesystem until after a while.

	-hpa

On Tue, Oct 18, 2005 at 06:02:29PM -0700, Linus Torvalds wrote:

There definitely is! But I tried a single "stat() all HEAD files" with a simple script and it took more than 3 seconds for the 80 trees. Then I gave up "optimizing" and was sure we want to have a single-file cached front page instead. :)

Kay

On Tue, Oct 18, 2005 at 10:24:18AM -0700, H. Peter Anvin wrote:

Yeah, I get an idea now :)

Great! Hope that will work.

Yes, makes sense.

I've added the Expires: header to the commit and commitdiff pages with one whole day ahead. Let's see if that will help...

Kay