Re: Efficiency of initial clone from server

From: Shawn O. Pearce <spearce@spearce.org>
Date: Feb 12, 2007, 05:11 UTC
Message-ID: <20070212051108.GB699@spearce.org>
In-Reply-To: <9e4733910702112101h256b1468j5de65c6e89109a22@mail.gmail.com>

Jon Smirl <jonsmirl@gmail.com> wrote:

Show 5 quoted lines

> But pack to the original point, can't the server check and see if it
> has write access so that it can keep the fully packed tree? I've just
> caused kernel.org to needlessly repack the wireless-dev tree a dozen
> times playing with this clone command. If it didn't have to keep
> repacking for the clone, clone would be a lot faster.

We probably could.

I have actually been thinking about another problem that is somewhat related. We cannot put more than 4 GiB of data into a single packfile, due to the current index size limitation, or more than 2^32-1 objects into one packfile, due to the header nr_objects field size.

Right now we are sending a single packfile down to the client, even if the remote server end has the repository broken down into a couple of packfiles (such as "really old historical stuff" and "active stuff from this year"). If we could send more than one packfile to the client in a single stream, we could still keep the file size limitations.

We can also avoid this huge repack case on the server. Because it could just send all of the packfiles that it already has, followed by whatever is loose which wasn't in a prior packfile. And no write access required.

Of course, we still could do the optimization of caching the packfile, but I'm not sure how well that would work on kernel.org, as I understand the trees are owned by the devs which created them while the git daemon is probably not running as their UNIX user.

-- 
Shawn.