Re: Performance issue: initial git clone causes massive repack

From: Shawn O. Pearce <spearce@spearce.org>
Date: Apr 6, 2009, 14:48 UTC
Message-ID: <20090406144829.GF23604@spearce.org>
In-Reply-To: <9e4733910904060737k3d1c082fk785cd98cdeb6d73d@mail.gmail.com>

Jon Smirl <jonsmirl@gmail.com> wrote:

Show 6 quoted lines

> 
> How do you deal with dense history packs? These packs take many hours
> to make (on a server class machine) and can be half the size of a
> regular pack. Shouldn't there be a way to copy these packs intact on
> an initial clone? It's ok if these packs are specially marked as being
> ok to copy.

These should be copied as-is.

Basically, object enumeration lists every reachable object, which should include every object in this pack if its a "dense history pack". We then start to write out each object. As each object is written we look to see if it already exists in a pack. It does (in your dense history pack), so we then look to see if its delta base is also in the output list (it is), so we send the data as-is.

One of the bigger costs with such clones is building that huge list of objects needed to send. The primary cost appears to be unpacking the trees from the "dense history pack", where delta chains are usually quite long. The GSoC 2009 pack caching project idea is based on the theory that we should be able to save a list of objects that are reachable from some fixed point (e.g. a very well known, stable tag), and avoid needing to read these ancient trees.

But its just a theory. Caching always costs you management overheads. And it may not save us that much time . And most of the theory here is based on JGit's performance during packing, *not* git-core.

I came up with the object list caching idea because JGit's object enumeration is just pitiful. (Its Java, what do you want, if you wanted fast, you'd use portable assembler... like git-core does.) Whether or not its worth applying to git-core is another story entirely.

-- 
Shawn.