Re: git-fetching from a big repository is slow

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Dec 14, 2006, 23:26 UTC
Message-ID: <Pine.LNX.4.63.0612150015330.3635@wbgn013.biozentrum.uni-wuerzburg.de>
In-Reply-To: <7v8xha2osd.fsf@assigned-by-dhcp.cox.net>

Hi,

On Thu, 14 Dec 2006, Junio C Hamano wrote:

Show 14 quoted lines

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > git-show-ref traverses every single _local_ tag when called. This is to 
> > overcome the problem that tags can be packed now, so a simple file 
> > existence check is not sufficient.
> 
> Is "traverses every single _local_ tag" a fact?  It might go
> through every single _local_ (possibly stale) packed tag in
> memory but it should not traverse $GIT_DIR/refs/tags.
> 
> If I recall correctly, show-ref (1) first checks the filesystem
> "$GIT_DIR/$named_ref" and says Ok if found and valid; otherwise
> (2) checks packed refs (reads $GIT_DIR/packed-refs if not
> already).

If I read builtin-show-ref.c correctly, it _always_ calls for_each_ref(show_ref, NULL);

The only reason that the loop in for_each_ref can stop early is if show_ref returns something different than 0. But it does not! Every single return in show_ref() returns 0. It does not matter, though (see below).

Show 5 quoted lines

> So that would be at most one open (which may fail in (1)) and one 
> open+read (in (2)).  Unless we are talking about fork+exec overhead, 
> that "traverse" should be reasonably fast.
> 
> Where is the bottleneck?

The problem is that so many stat()s _do_ take time. Again, if I read the code correctly, it not only stat()s every loose ref, but also resolves the refs in get_ref_dir(), which is called from get_loose_refs(), which is unconditionally called in for_each_ref().

Even if the refs are packed, it takes quite _long_ (I confirmed this). And it is not at all necessary! Instead of a O(n^2) we can easily reduce this to O(n*log(n)), and we can reduce the n fork()&exec()s of git-show-ref by a single one.

Ciao, Dscho