From: Paolo Bonzini <bonzini@gnu.org>
Date: Sun, 14 Feb 2010 11:55:24 GMT
Subject: Re: [PATCH] don't use mmap() to hash files
Message-ID: <4B77E4AC.6030600@gnu.org>
In-Reply-To: <20100214024259.GB9704@dpotapov.dyndns.org>

On 02/14/2010 03:42 AM, Dmitry Potapov wrote:
> In fact, we know mmap() also may fail for huge files, so can
> strbuf_read().

On a 64-bit machine mmap should fail pretty much never, as it is limited
by address space and not available memory.

That said I can reproduce the result here too, and it's actually quite 
understandable from your "time" results: mmap has a minor page fault for 
every 4k you read (or at least that's the order of magnitude), read has 
basically none.  Furthermore, it's true that with read you touch the 
whole memory twice from beginning to end, while with mmap you touch a 
page twice and move on; however, a file's contents will almost always 
fit in the L2 cache, so it's not too expensive.

I tried madvise(buf, size, MADV_SEQUENTIAL) and MADV_WILLNEED but it has 
no effect.

I suspect mmap will be faster only if the data is not in cache (so the 
cost of a page fault is negligible compared to going to disk) and the 
average file size is a few megabytes.

Paolo