Re:

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: May 9, 2009, 16:44 UTC
Message-ID: <alpine.LFD.2.01.0905090926300.3586@localhost.localdomain>
In-Reply-To: <twBG1KnSrgPNk7NoVey4mgig1BeAk7e1GHOT90PSV9ZGTs-zCWYdtA@cipher.nrlssc.navy.mil>

On Fri, 8 May 2009, Brandon Casey wrote:

> 
> btw, I've since done some more testing on some centos5.3 boxes we have.
> I get similar results (less ancient kernel 2.6.18).

Yes, 2.6.18 is still much too old to matter from a locking standpoint.

When people initially worried about scalability, the issues were more about server side stuff and the cached cases. NFS (as a client) is certainly used on the server side too, but it tends to be a somewhat secondary worry where only specific parts really matter. So people worked a lot more on the core kernel, and on local high-performance filesystem scaling.

Only lately have we been pretty aggressive about finally really getting rid of the old "single big lock" (BKL) model entirely, or moving outwards from the core.

And while we removed the BKL from the normal NFS read/write paths long long ago, all the name lookup and directory handling code still had it until a year ago.

That, btw, is directly explained by perceived scalability issues: NFS is fairly often used as the backing store for a database and scaling thus matters there. But databases tend to keep their few big files open and use pread/pwrite - so pathname lookup is not nearly as significant for server ops as plain read/write.

(Pathname lookup is important for things like web servers etc, but they rely heavily on caching for that, and the cached case scales fine).

> I've also scanned through the errata announcements that RedHat has 
> released for their kernel updates.  A few of them involve NFS.  
> Possibly, whatever RedHat modified in the 5.X kernel was also backported 
> to the 4.X kernel.

That is very possibly the case. Expanding the BKL usage in some case could easily trigger the lock getting contention - and the way lock contention works, once you get a just even a small _hint_ of contention, things often fall off a cliff. The contention slows locking down, which in turn causes more CPU usage, which in turn causes _more_ contention.

So even a small amount of extra locking - or even just slowing down some code that was inside the lock - can have catastrophic behavioural changes when the lock is close to being a problem. You do not get a nice gradual slowdown at all - you just hit a hard wall.

I guess I should really try to set up some fileserver here at home to improve my test coverage. And to do better backups (or the little private data I have that I can't just mirror out to the world by turning it into an open-source project ;^)

				Linus