Re: Smart fetch via HTTP?

From: Shawn O. Pearce <spearce@spearce.org>
Date: May 17, 2007, 01:45 UTC
Message-ID: <20070517014542.GW3141@spearce.org>
In-Reply-To: <20070517012602.GV3141@spearce.org>

"Shawn O. Pearce" <spearce@spearce.org> wrote:

Show 12 quoted lines

> david@lang.hm wrote:
> > On Wed, 16 May 2007, Shawn O. Pearce wrote:
> > >
> > >The other thing is kernel.org should really try to encourage the
> > >folks with repositories there to try and share against one master
> > >repository, so the poor OS has a better chance at holding the bulk
> > >of linux-2.6.git in buffer cache.
> > 
> > do you mean more precisely share against one object store or do you really 
> > mean repository?
> 
> Sorry, I did mean "object store".  ;-)

And even there, I don't mean symlink objects to a shared database, I mean use the objects/info/alternates file to point to the shared, read-only location.

Its not perfect. The hotter parts of the object database is almost always the recent stuff, as that's what people are actively trying to fetch, or are using as a base when they are trying to fetch from someone else. The hotter parts are also probably too new to be in the shared store offered by kernel.org admins, which means you cannot get good IO buffering. Back to the current set of problems.

A single shared object directory that everyone can write new files into, but cannot modify or delete from, would help that problem quite a bit. But it opens up huge problems about pruning, as there is no way to perform garbage collection on that database without scanning every ref on the system, and that's just not simply possible on a busy system like kernel.org.

-- 
Shawn.