Re: Efficient way to import snapshots?

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Jul 31, 2007, 01:44 UTC
Message-ID: <alpine.LFD.0.999.0707301825130.4161@woody.linux-foundation.org>
In-Reply-To: <20070731011707.GA91930@nowhere>

On Mon, 30 Jul 2007, Craig Boston wrote:

Show 8 quoted lines

> > 
> > So I _seriously_ hope that about 65 of those 67 seconds was the "cvs 
> > update -d" or something like that. 
> 
> No, the only thing included in that is
> 
> git ls-files -o | git update-index --add --stdin
> git commit -a -m "${COMMITMSG}"

Ouch.

Show 13 quoted lines

> > Anything that takes a minute in git is way way *way* too slow. Any 
> > half-way normal git operations should take less than a second.
> 
> That said, I don't think it's git's fault.  I think most of the time is
> spent calling stat() on all the files.  The machine that took 60 seconds
> isn't what I'd call top-of-the-line:
> 
> 1st or maybe 2nd-gen Willamette CPU
> 512MB memory (stupid motherboard that won't accept more)
> Slow disks in RAID-5 configuration
> Running ZFS with less than half of the recommended minimum memory, to
> the point where I had to reduce the number of vnodes that the kernel is
> allowed to cache to avoid running out of KVA

Oh, ok. Solaris.

With slow pathname lookup, and hard limits on the inode cache sizes.

Git really normally avoids reading the data, so even in 512M you should _easily_ be able to cache the metadata (directory and inodes), which is all you need. But yeah, Linux will probably do that a whole lot more aggressively than Solaris does.

[ And to be honest, any CVS update would probably have blown the caches on 
  Linux too - I don't know what all CVS ends up doing, but from past
  experience, I'll bet it's not good ]

But just for comparison, on a lowly 480MB mac-mini (running Linux, not OS X, of course - and the 480MB is because the graphics is UMA and takes part of the 512MB total), and the kernel archive (which is just 22k files, not 37k), with a laptop drive:

 - cold-cache "git status":
	real    0m17.975s
	user    0m1.098s
	sys     0m0.539s

 - rerunning it immediately afterwards:
	real    0m1.079s
	user    0m0.896s
	sys     0m0.183s

so the target really _should_ generally be one second.

But yeah, in order to hit that target, you definitely do want to keep the metadata cached, and I guess that means more than 512M on Solaris.

			Linus