git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Gitweb caching: Google Summer of Code project

From
LWLea Wiemann <lewiemann@gmail.com>
Date
May 27, 2008, 22:54 UTC
Message-ID
<483C912F.6010802@gmail.com>
In-Reply-To
<200805272353.34319.jnareb@gmail.com>
Jakub Narebski wrote:
> Lately he posted a patch
> implementing projects list caching, in a bit different way from how it
> is done on kernel.org, namely by caching data and not final output:
Thanks for this and all the other pointers.

Caching data and not final output is actually what I'm about to try next. If I'm not mistaken, the HTML output is significantly larger than the source (repository) data; however, kernel.org still seems to benefit from caching the HTML, rather than letting Linux' page cache cache the source data. That leads me to think that the page cache somehow fails to cache the source data properly -- I'm not sure why (wild speculation: perhaps because of the pack format). Anyway, I'd hope that I can encapsulate the 30-40 git_cmd calls in gitweb.perl and somehow cache their results (or, to save memory, the parts of their results that are actually used) and cache them using memcached. If that works well, we can stop bothering about frontend (HTML) caching, unless CPU becomes an issue, since all HTML pages are generated from cacheable source data.

I'm *kindof* hoping that in the end there will be only few issues with cache expiry, since most calls are uniquely identified through hashes. (And the ones that are not, like getting the hash of the most recent commit, can perhaps be cached with some fairly low expiry time.)

So that's what I'll try next. If you have any comments or warnings off the top of your heads, feel free to send email of course. :)

> the main culprit of [the fork] was splitting gitweb into many, many
> files.  While it helped John in understanding gitweb, it made it
> difficult to merge changes back to mainline.

Interesting point, thanks for letting me know. (I might have gone ahead and tried to split the mainline gitweb myself... ^^) I think it would be nice if gitweb.perl could be split at some point, but I assume there are too many patches out there for that to be worth the merge problems, right?

-- Lea
Previous: Jakub NarebskiNext: Jakub Narebski
Message 3 of 18 in “Gitweb caching: Google Summer of Code project”
  1. Lea WiemannMay 27, 2008
  2. Jakub NarebskiMay 27, 2008
  3. Lea WiemannMay 27, 2008
  4. Jakub NarebskiMay 28, 2008
  5. Lea WiemannMay 28, 2008
  6. Jakub NarebskiMay 29, 2008
  7. Lea WiemannMay 30, 2008
  8. Jakub NarebskiMay 30, 2008
  9. Lea WiemannMay 30, 2008
  10. Petr BaudisMay 30, 2008
  11. Lea WiemannMay 30, 2008
  12. Petr BaudisMay 30, 2008
  13. Rafael Garcia-SuarezMay 30, 2008
  14. J.H.May 30, 2008
  15. Junio C HamanoMay 30, 2008
  16. Lea WiemannMay 30, 2008
  17. Lea WiemannMay 30, 2008
  18. Jakub NarebskiMay 31, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.