Re: Git and GCC

From: David Miller <davem@davemloft.net>
Date: Dec 7, 2007, 12:53 UTC
Message-ID: <20071207.045329.204650714.davem@davemloft.net>
In-Reply-To: <9e4733910712062310s30153afibc44a5550fd9ea99@mail.gmail.com>

From: "Jon Smirl" <jonsmirl@gmail.com>
Date: Fri, 7 Dec 2007 02:10:49 -0500

Show 9 quoted lines

> On 12/7/07, Jeff King <peff@peff.net> wrote:
> > On Thu, Dec 06, 2007 at 07:31:21PM -0800, David Miller wrote:
> >
> > # and test multithreaded large depth/window repacking
> > cd test
> > git config pack.threads 4
> 
> 64 threads with 64 CPUs, if they are multicore you want even more.
> you need to adjust chunk_size as mentioned in the other mail.

It's an 8 core system with 64 cpu threads.

> > time git repack -a -d -f --window=250 --depth=250

Didn't work very well, even with the one-liner patch for chunk_size it died. I think I need to build 64-bit binaries.

davem@huronp11:~/src/GCC/git/test$ time git repack -a -d -f --window=250 --depth=250 Counting objects: 1190671, done. fatal: Out of memory? mmap failed: Cannot allocate memory

real 58m36.447s user 289m8.270s sys 4m40.680s davem@huronp11:~/src/GCC/git/test$

While it did run the load was anywhere between 5 and 9, although it did create 64 threads, and the size of the process was about 3.2GB This may be in part why it wasn't able to use all 64 thread effectively. Like I said it seemed to have 9 active at best, at any one time, most of the time only 4 or 5 were busy doing anything.

Also I could end up being performance limited by SHA, it's not very well tuned on Sparc. It's been on my TODO list to code up the crypto unit support for Niagara-2 in the kernel, then work with Herbert Xu on the userland interfaces to take advantage of that in things like libssl. Even a better C/asm version would probably improve GIT performance a bit.

Is SHA a significant portion of the compute during these repacks? I should run oprofile...