Re: Smart fetch via HTTP?

From: Shawn O. Pearce <spearce@spearce.org>
Date: May 17, 2007, 01:03 UTC
Message-ID: <20070517010335.GU3141@spearce.org>
In-Reply-To: <Pine.LNX.4.64.0705170152470.6410@racer.site>

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

> Don't forget that those 10% probably do not do you the favour to be in 
> large chunks. Chances are that _every_ _single_ wanted object is separate 
> from the others.

That's completely possible.  Assuming the objects even are packed
in the first place.  Its very unlikely that you would be able to
fetch very large of a range from an existing packfile, you would be
submitting most of your range requests for very very small sections.

Show 7 quoted lines

> > And for services like SF.net it'd be a safe low-cpu way of serving git
> > files. 'cause the git protocol is quite expensive server-side (io+cpu)
> > as we've seen with kernel.org. Being really smart with a cgi is
> > probably going to be expensive too.
> 
> It's probably better and faster than relying on a feature which does not 
> exactly help.

Yes.  Packing more often and pack v4 may help a lot there.

The other thing is kernel.org should really try to encourage the folks with repositories there to try and share against one master repository, so the poor OS has a better chance at holding the bulk of linux-2.6.git in buffer cache.

I'm not suggesting they share specifically against Linus' repository; maybe hpa and the other admins can host one seperately from Linus and enourage users to use that repository when on a system they maintain.

In an SF.net type case this doesn't help however. Most of SF.net is tiny projects with very few, if any, developers. Hence most of that is going to be unsharable, infrequently accessed, and uh, not needed to be stored in buffer cache. For the few projects that are hosted there that have a large developer base they could use a shared repository approach as I just suggested for kernel.org.

aka the "forks" thing in gitweb, and on repo.or.cz...

-- 
Shawn.