Re: Better value for chunk_size when threaded

From: Nicolas Pitre <nico@cam.org>
Date: Dec 7, 2007, 01:27 UTC
Message-ID: <alpine.LFD.0.99999.0712062014060.555@xanadu.home>
In-Reply-To: <9e4733910712061558k19fbc864ia1fb7a3431fd2603@mail.gmail.com>

On Thu, 6 Dec 2007, Jon Smirl wrote:

> I tried some various ideas out for chunk_size and the best strategy I
> found was to simply set it to a constant. How does 20,000 work on
> other CPUs?

That depends on the object size. If you have a repo with big objects but only 1000 of them for example, then the constant doesn't work.

Ideally I'd opt for a value that tend towards around 5 seconds worth of work per segment, or something like that. Maybe using the actual objects size could be another way.

> I'd turn on default threaded support with this change. With threads=1
> versus non-threaded there is no appreciable difference in the time.

Would need a way to determine pthreads availability from Makefile.

> Is there an API to ask how many CPUs are in the system? It would be
> nice to default the number of threads equal to the number of CPUs and
> only use pack.threads=X to override.

If there is one besides futzing with /proc/cpuinfo I'd like to know about it. Bonus points if it is portable.

Nicolas