Re: kernel.org mirroring (Re: [GIT PULL] MMC update)

From: Jakub Narebski <jnareb@gmail.com>
Date: Dec 8, 2006, 09:43 UTC
Message-ID: <200612081043.07192.jnareb@gmail.com>
In-Reply-To: <Pine.LNX.4.64.0612071052560.3615@woody.osdl.org>

Linus Torvalds wrote: [...]

Show 14 quoted lines

> For example, if the git "refs/heads/" (or tags) directory hasn't changed 
> in the last two months, we should probably set any ref-relative gitweb 
> pages to have a caching timeout of a day or two. In contrast, if it's 
> changed in the last hour, maybe we should only cache it for five minutes.
> 
> Jakub: any way to make gitweb set the "expires" fields _much_ more 
> aggressively. I think we should at least have the ability to set a basic 
> rules like
> 
>  - a _minimum_ of five minutes regardless of anything else
> 
>    We might even tweak this based on loadaverage, and it might be 
>    worthwhile to add a randomization, to make sure that you don't get into 
>    situations where everything webpage needs to be recalculated at once.

I think the minimum expires (or minimum _additional_ expires: as of now
giweb only does expires +1d for explicit hash requests) should depend on
how often project changes. How often there are pushes to kernel.org?

Show 7 quoted lines

>  - if refs/ directories are old, raise the minimum by the age of the refs
> 
>    If it's more than an hour old, raise it to ten minutes. If it's more 
>    than a day, raise it to an hour. If it's more than a month old, raise 
>    it to a day. And if it's more than half a year, it's some historical 
>    archive like linux-history, and should probably default to a week or 
>    more.

What about packed refs?

We can certainly raise expires for tags (tags objects), as they should not
usually change.

>  - infinite for stuff that isn't ref-related.

As sha1 is not changeable, everything that is accessed by explicit sha1 (hash), or by explicit sha1 (hash_base) plus pathname (file_name) should have effectively infinite expires.

Every caching would need some temporary memory, or temporary disk space. And perhaps mod_perl specific caching would be useful here...

P.S. I have added Pasky to Cc:, as he manages http://repo.or.cz public git repository hosting (much smaller than kernel.org and I think under less load: but also I think withour kernel.org resources).

-- 
Jakub Narebski