Re: q: git-fetch a tad slow?

From: Shawn O. Pearce <spearce@spearce.org>
Date: Jul 30, 2008, 04:48 UTC
Message-ID: <20080730044855.GA7225@spearce.org>
In-Reply-To: <20080729090802.GA11373@elte.hu>

Ingo Molnar <mingo@elte.hu> wrote:

Show 11 quoted lines

> * Shawn O. Pearce <spearce@spearce.org> wrote:
> > Ingo Molnar <mingo@elte.hu> wrote:
> > > 
> > > Setup/background: distributed kernel testing cluster, [...]
> > > 
> > > Problem: i noticed that git-fetch is a tad slow:
> > > 
> > >   titan:~/tip> time git-fetch
> > >   real    0m2.372s
>
> note that titan is a very beefy box, almost 3 GHz Core2Duo:

That isn't going to matter if you have a quadratic algorithm and a
large dataset.  Especially when the inner loops are doing multiple
system calls per item in a long list of items.  :-|   Linux is fast,
but it isn't magic pixie dust.  It cannot fix broken applications.

> [...] So if we have a quadratic overhead on number of 
> branches, that's going to be quite a PITA.

Right.

Show 6 quoted lines

> > I wonder if git-pack-refs + fetching only a single branch will get you 
> > closer to the tip-fetch time.
> 
> should i pack on both repos? I dont explicitly pack anything, but on the 
> server it goes into regular gc runs. (which will pack most stuff, 
> right?)

git-gc automatically runs `git pack-refs --all --prune` like I recommended, unless you disabled it with config gc.packrefs = false. So its probably already packed.

What does `find .git/refs -type f | wc -l` give for the repository on the central server? If its more than a handful (~20) I would suggest running git-gc before testing again.

But I'm really suspecting that this is just our quadratic matching algorithm running up against a large number of branches, causing it to suck.

jgit at least uses an O(N) algorithm here, but since it is written in Java its of course slow compared to C Git. Takes a while to get that JVM running.

I'll try to find some time to reproduce the issue and look at the bottleneck here. I'm two days into a new job so my git time has been really quite short this week. :-|

-- 
Shawn.