From: Pieter de Bie <pdebie@ai.rug.nl>
Date: Sat, 19 Apr 2008 23:56:19 GMT
Subject: Re: Git performance on OS X
Message-ID: <2F8F3BF2-66F9-473C-BE82-8F784E1FF9A4@ai.rug.nl>
In-Reply-To: <alpine.LFD.1.10.0804191619240.2779@woody.linux-foundation.org>


On 20 apr 2008, at 01:26, Linus Torvalds wrote:
>
> It may be that the problem with OS X is a sucky pathname cache  
> mechanism.
>
> The trivial patch cut down the number of stat() calls by a fair  
> amount,
> but the calls that got removed were all of the "do two 'lstat()'  
> calls on
> the exact same pathname consecutively" type.
>
> Maybe OS X has some very limited pathname caching that catches that,  
> or
> even if not, it just ends up being very nice in the D$, so it's not  
> a big
> deal. And then the real suckiness happens only with bigger workloads.
>

Yes, I just tested this.

	for (int i = 0; i < 50000; i++) {
		sprintf(s, "/Users/pieter/test/perf/%i", i);
		int ret = lstat(s, a);
	}

This loop needs about 3 seconds to  run. Replacing the i with 10 in  
the sprintf reduces it to 0.24seconds.

>> Command                                            Mean     Std
>> git status .                                       13.970  1.298
>> /Users/pieter/projects/External/git/git-status .   13.759  0.321
>
> This is the WebKit archive, right?
>
> For me, doing a "time git status ." on the WebKit thing I just  
> cloned from
> git://git.webkit.org/WebKit.git is much faster: 1.264s (and it goes  
> down
> by maybe 5-10% with my lstat-avoidance patch).
>
> Is there any system-level profiler for OS X to get a clue where that  
> cost
> is, in case it's not the lstat() at all?

Yes, that was the webkit repo (the test above was in a dir with 50k  
files).

Alas, I tried to create a nice profiling for the "git status .". In  
the Instruments application I can create a sampler, but I see no way  
to export it. The option to export the script as a dtrace script is  
greyed out in the menu.

 From the sampler, it appears that the lstat calls still account for  
most of the time. I have uploaded a screenshot to http://ss.frim.nl/==759.png 
. It actually shows quite nicely when the lstats are being done --  
it's when the CPU is idle. Next to the lstats, the read_tree_recursive  
is also called often.

- Pieter