Re: git pull is slow

From: Shawn O. Pearce <spearce@spearce.org>
Date: Jul 13, 2008, 22:11 UTC
Message-ID: <20080713221141.GC12708@spearce.org>
In-Reply-To: <alpine.DEB.1.00.0807131555560.4816@eeepc-johanness>

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 18 quoted lines

> On Sun, 13 Jul 2008, Shawn O. Pearce wrote:
> > Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > ...
> > > I expect this to touch the resolve_delta() function of index-pack.c in a 
> > > major way, though.
> > 
> > Yea, that's going to be ugly.  The "cache" you speak of above is held on 
> > the call stack as resolv_delta() recurses through the delta chain to 
> > reconstruct objects and generate their SHA-1s.  There isn't a way to 
> > release these objects when memory gets low so your worst case scenario 
> > is a 100M+ blob with a delta chain of 50 or more - that will take you 5G 
> > of memory to pass through index-pack.
> 
> Actually, there is...
> 
> You would only need to tap into the "release_pack_memory()" mechanism, 
> adding a sort of a "smart pointer" that knows how to reconstruct its 
> contents.

I guess you don't really know how index-pack is organized than. It is not quite that simple.

I think we want to use the core.deltabasecachelimit parameter inside of index-pack to limit the size of the cache, but evict based on the callstack depth. Objects deeper back in the callstack should evict before objects closer to the top of the stack.

Reconstruction is a bit complex as we don't have the machinary in sha1_file.c available to us, as the index is not available, as we are still in the process of creating the damn thing.

I'm working up a patch series to resolve this. I'm going to shutup now and just try to post working code sometime this evening.

-- 
Shawn.