Re: Better value for chunk_size when threaded

From: Nicolas Pitre <nico@cam.org>
Date: Dec 7, 2007, 03:25 UTC
Message-ID: <alpine.LFD.0.99999.0712062139170.555@xanadu.home>
In-Reply-To: <9e4733910712061737o50a9a5f1ldccdf943bb19319f@mail.gmail.com>

On Thu, 6 Dec 2007, Jon Smirl wrote:

Show 14 quoted lines

> On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> > On Thu, 6 Dec 2007, Jon Smirl wrote:
> >
> > > I tried some various ideas out for chunk_size and the best strategy I
> > > found was to simply set it to a constant. How does 20,000 work on
> > > other CPUs?
> >
> > That depends on the object size.  If you have a repo with big objects
> > but only 1000 of them for example, then the constant doesn't work.
> 
> How about defaulting it to 20,000 and allowing an override? It's not
> fatal if we guess wrong, we just want to most common cases to work out
> of the box. 20,000 is definitely better than the current window *
> 1000.

Sure.

... But I think this can be made much better than that with no guessing at all.

Say you have 4 threads. then let's divide the whole object list into 4 big segments and feed those to each thread.

One thread will always finish before the others. The idea is to find the active thread with the largest amount of remaining objects to process at that point, and steal half of them and give that to the thread that just finished. Repeat for each thread that completes its segment until everything is done.

Nicolas