Re: Git and GCC

From: Nicolas Pitre <nico@cam.org>
Date: Dec 6, 2007, 22:08 UTC
Message-ID: <alpine.LFD.0.99999.0712061645120.555@xanadu.home>
In-Reply-To: <9e4733910712061339n3aef023r22e5b73aac120c8a@mail.gmail.com>

On Thu, 6 Dec 2007, Jon Smirl wrote:

Show 22 quoted lines

> On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> > > When I lasted looked at the code, the problem was in evenly dividing
> > > the work. I was using a four core machine and most of the time one
> > > core would end up with 3-5x the work of the lightest loaded core.
> > > Setting pack.threads up to 20 fixed the problem. With a high number of
> > > threads I was able to get a 4hr pack to finished in something like
> > > 1:15.
> >
> > But as far as I know you didn't try my latest incarnation which has been
> > available in Git's master branch for a few months already.
> 
> I've deleted all my giant packs. Using the kernel pack:
> 4GB Q6600
> 
> Using the current thread pack code I get these results.
> 
> The interesting case is the last one. I set it to 15 threads and
> monitored with 'top'.
> For 0-60% compression I was at 300% CPU, 60-74% was 200% CPU and
> 74-100% was 100% CPU. It never used all for cores. The only other
> things running were top and my desktop. This is the same load
> balancing problem I observed earlier.

Well, that's possible with a window 25 times larger than the default.

The load balancing is solved with a master thread serving relatively small object list segments to any work thread that finished with its previous segment. But the size for those segments is currently fixed to window * 1000 which is way too large when window == 250.

I have to find a way to auto-tune that segment size somehow.

But with the default window size there should not be any such noticeable load balancing problem.

Note that threading only happens in the compression phase. The count and write phase are hardly paralleled.

Nicolas