Re: Gitweb caching: Google Summer of Code project

From: Jakub Narebski <jnareb@gmail.com>
Date: May 27, 2008, 21:53 UTC
Message-ID: <200805272353.34319.jnareb@gmail.com>
In-Reply-To: <483C4CFF.2070101@gmail.com>

On Tue, 27 May 2008, Lea Wiemann wrote:

Show 7 quoted lines

> I just wanted to let everyone know that I'm currently getting started on 
> a Google Summer of Code project to improve the caching mechanism in gitweb.
> 
> Sorry for not posting about this earlier...  Anyways, some key data: 
> John 'warthog9' Hawley (who wrote the current caching system for 
> kernel.org) is my mentor, and GSoC is from May 26 to Aug 18, minus a 
> vacation from Jul 19 to Aug 9.

Thanks for the info.

> While I'm planning to keep much of it on the list, if anyone else is 
> particularly interested in helping or providing input, please notify me. 
>   (Looking at the logs, Jakub maybe?  Cc'ing him just in case.)

I'm certainly interested, at least from theoretical point of view, and I think I can help (as one of main gitweb contributors).

I guess that Petr Baudis would also be interested, because he maintains
repo.or.cz, a public Git hosting site.  Lately he posted a patch
implementing projects list caching, in a bit different way from how it
is done on kernel.org, namely by caching data and not final output:
  http://thread.gmane.org/gmane.comp.version-control.git/77151
AFAIK it is implemented in repo.or.cz gitweb: 
  http://repo.or.cz/w/git/repo.git

This indirectly lead to a bit of research on caching in Perl by yours
truly:
  http://thread.gmane.org/gmane.comp.version-control.git/77529
(mentioned in http://git.or.cz/gitwiki/SoC2008Projects#gitweb-caching).

I think that you can also get some help on caching from Lars Hjemli, author of cgit, which is caching git web interface written in C.

(I have added both Petr Baudis and Lars Hjemli to Cc:)

Show 7 quoted lines

> The current plan is basically to get the gitweb caching fork that's been 
> implemented for kernel.org back to the gitweb mainline, and then 
> optimize it (probably move to memcached).  I'm not yet sure how to 
> approach this (e.g. whether to merge from the fork to the mainline or 
> vice versa), but I'll probably figure this out together with John and 
> might post separately about that later.  In any case, expect patches and 
> messages from me on the list. :)

>From what I remember correcly from the discussion surrounding

implementing caching for kernel.org gitweb, the main culprit of having it remain separate from mainline was splitting gitweb into many, many files. While it helped John in understanding gitweb, it made it difficult to merge changes back to mainline.

Note also that it is easier to make a site-specific changes, than to make generic, closs-platform and cross-operating system change.

-- 
Jakub Narebski
Poland