Re: [PATCH] Teach "git add" and friends to be paranoid

From: Dmitry Potapov <dpotapov@gmail.com>
Date: Feb 22, 2010, 03:35 UTC
Message-ID: <20100222033553.GA10191@dpotapov.dyndns.org>
In-Reply-To: <7vhbpas7ut.fsf@alter.siamese.dyndns.org>

On Sun, Feb 21, 2010 at 11:32:10AM -0800, Junio C Hamano wrote:

> 
> 32*1024 sounds like a better cut-off to me.  After that, doubling the size
> does not get comparable gain, and numbers get unstable (notice the glitch
> around 256kB).

The reduction of speed-up after 32Kb is most likely due to L1 cache size, which is 32Kb data per core on Core 2, and L2 cache is shared among cores and is considerably slow. I have run my test a few more times, and here are results:

   1 - 39.25%
   2 - 30.00%
   4 - 17.79%
   8 - 11.76%
  16 - 7.58%
  32 - 5.38%
  64 - 3.89%
 128 - 2.87%
 256 - 2.31%
 512 - 2.92%
1024 - 1.14%

and here is one more re-run starting with 32Kb:

  32 - 5.38%
  64 - 3.89%
 128 - 2.29%
 256 - 2.91%
 512 - 2.92%
1024 - 1.14%

If you look at speed-up numbers, you can think that the numbers are unstable, but in fact, the best time in 5 runs does not differ more than 0.01s between those trials. But because difference for >=128Kb is 0.05s or less, the accuracy of the above numbers is less than 25%. But overall the outcome is clear -- read() is always a winner.

It would be interesting to see what difference Nehalem, which has a smaller but much faster L2 cache than Core 2. It may perform better at larger sizes up to 256Kb.

Anyway, based on above data, I believe that the proper cut-off should be at least 64Kb, because additional 32Kb (from 32Kb to 64Kb) is about of 2.5% of total memory that git consumes anyway, and it gives you speed-up around 3.5%...

Dmitry