Re: git-daemon on NSLU2

From: Jon Smirl <jonsmirl@gmail.com>
Date: Aug 24, 2007, 23:46 UTC
Message-ID: <9e4733910708241646x7b285574t94c3d7eb32bb60c9@mail.gmail.com>
In-Reply-To: <fanmmk$f5q$1@sea.gmane.org>

On 8/24/07, Jakub Narebski <jnareb@gmail.com> wrote:

Show 5 quoted lines

> There was idea to special case clone (just concatenate the packs, the
> receiving side as someone told there can detect pack boundaries; do not
> forget to pack loose objects, first), instead of using generic fetch --all
> for clone, bnut no code. Code speaks louder than words (although if someone
> would provide details of pack boundary detection...)

A related concept, initial clone of a repository does the equivalent of repack -a on the repo before transmitting it. Why aren't we saving those results by switching the repo onto the new pack file? Then the next clone that comes along won't have to do anything but send the file.

But this logic can be flipped around, if the remote needs any object from the pack file, just send them the whole pack file and let the remote sort it out. Using this logic you can still minimize the IO statistically.

When a remote does a fetch you have to pack all of the loose objects. When the loose object pile reaches 20MB or so, the fetch can trigger a repack of the oldest half into a pack that is kept by the tree and replaces those older loose objects. For future fetches simply apply the rule of sending the whole pack if any object is needed.

The repack of the 10MB of older objects can be kicked out to another process and copied into the tree when it is finished. At that point the loose objects can be deleted. The git db can tolerate a process copying in a new packfile and deleting the old objects while other processes may be using the database, right?

This model shouldn't statistically change the amount of data very much. If you haven't synced your tree in a month a few too many objects may get sent to you. However, it should dramatically reduce the IO load on the server cause by git protocol initial clones.

-- 
Jon Smirl
jonsmirl@gmail.com