Re: cloning the kernel - why long time in "Resolving 313037 deltas"

From: Linus Torvalds <torvalds@osdl.org>
Date: Dec 18, 2006, 23:28 UTC
Message-ID: <Pine.LNX.4.64.0612181511260.3479@woody.osdl.org>
In-Reply-To: <8664c896xv.fsf@blue.stonehenge.com>

On Mon, 18 Dec 2006, Randal L. Schwartz wrote:

Show 5 quoted lines

> 
> Linus> You're running this under OS X, aren't you? It's a pig of an OS, but 
> Linus> "almost one hour" vs "25 seconds" is still unreasonable.
> 
> I agree!

The only thing that comes to mind is

 - some actual difference in how the pack looks. Although I don't really 
   see that. I re-pack both my own private kernel repos and the kernel.org 
   one fairly often, and usually with a pretty recent git. I don't think 
   any of the packing heuristics have changed in a long time. And I just 
   checked: the kernel.org pack-file (the big one) unpacks quickly too.

 - SHA1 library differences. Maybe OpenSSL on OS X sucks? I still don't 
   see the difference being _that_ big, but whatever..

 - OS X "mmap()" performance really really sucks.

   We do end up doing a _log_ of small mmaps in the "Resolving .." stage. 
   On Linux, the strace looks like

	write(2, "Resolving 316071 deltas.\n", 25) = 25
	mmap(NULL, 2619, PROT_READ, MAP_PRIVATE, 3, 0x4c000) = 0x2aaf13be2000
	munmap(0x2aaf13be2000, 2619)            = 0
	mmap(NULL, 2848, PROT_READ, MAP_PRIVATE, 3, 0x4c000) = 0x2aaf13be2000
	munmap(0x2aaf13be2000, 2848)            = 0
	mmap(NULL, 2988, PROT_READ, MAP_PRIVATE, 3, 0x4c000) = 0x2aaf13be2000
	munmap(0x2aaf13be2000, 2988)            = 0
	write(2, "   0% (2/316071) done\r", 22) = 22
	mmap(NULL, 3336, PROT_READ, MAP_PRIVATE, 3, 0x4c000) = 0x2aaf13be2000
	munmap(0x2aaf13be2000, 3336)            = 0
	mmap(NULL, 3471, PROT_READ, MAP_PRIVATE, 3, 0x4c000) = 0x2aaf13be2000
	munmap(0x2aaf13be2000, 3471)            = 0
	mmap(NULL, 3611, PROT_READ, MAP_PRIVATE, 3, 0x4c000) = 0x2aaf13be2000
	munmap(0x2aaf13be2000, 3611)            = 0
	...

   and it's entirely possible that mmap() really does suck *ss on OS X. 
   Those small mmap's go on for a _loong_ time.

I've heard of other operations being slow on OS X - and two orders of magnitude really isn't unthinkable. I don't think people always seem to really understand how _good_ Linux is, and how much faster it can be. It's not just "Windows XP" sucks. Quite often it's literally "Linux is just damn fast".

Sadly, that causes problems when the main developers don't even see any issues, just because the Linux kernel environment makes things look really really cheap. Even when it isn't always cheap on other platforms.

Nico - have you looked at perhaps making the index-pack.c "mmap()" usage do chunking? Or just mmap the whole damn thing once? Linux is fast, but even Linux will be faster if you just mmap it once ;)