Re: Efficiency of initial clone from server

From: Jon Smirl <jonsmirl@gmail.com>
Date: Feb 12, 2007, 14:31 UTC
Message-ID: <9e4733910702120631o1be92037ue89b069b0354c8cb@mail.gmail.com>
In-Reply-To: <Pine.LNX.4.63.0702121241130.22628@wbgn013.biozentrum.uni-wuerzburg.de>

On 2/12/07, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 19 quoted lines

> Hi,
>
> On Sun, 11 Feb 2007, Junio C Hamano wrote:
>
> > You are assuming everybody does initial clone all the time.  I do not
> > think that holds true in practice.
>
> It depends how you interpret "all the time". What you (Junio) are
> suggesting is that the count of initial clones is relatively small as
> compared to the total number of fetches.
>
> However, you can interpret "all the time" in terms of "time". Most fetches
> are really small. They even often end up in no objects pulled at all.
> These are cheap for the server. The initial clones take a long time. They
> are expensive.
>
> I'd be interested to learn how much of the CPU time is actually spent in
> initial clones, rather than other types of fetches. It might make sense
> yet to optimize initial clones.

I don't think CPU is a problem at kernel.org, but disk IO defnitely is. The initial clones cause several minutes (sometimes 10 min or more when there kernel.org is loaded) worth of disk IO. They also totally thrash the kernel.org cache. The alternative of using a clone to trigger a repack would go through this once, and then use sendfile (is gitd that smart?) to send the packs. Sendfile uses the smallest cache required.

Why doesn't clone copy the existing packs down first with sendfile, then build a small pack for what is left and avoid the initial step of making a giant pack. Isn't clone going to break when the repo exceeds 2GB?

Show 5 quoted lines

>
> Ciao,
> Dscho
>
>

-- 
Jon Smirl
jonsmirl@gmail.com