Re: cloning the kernel - why long time in "Resolving 313037 deltas"

From: Linus Torvalds <torvalds@osdl.org>
Date: Dec 19, 2006, 06:47 UTC
Message-ID: <Pine.LNX.4.64.0612182234260.3479@woody.osdl.org>
In-Reply-To: <20061219051108.GA29405@thunk.org>

On Tue, 19 Dec 2006, Theodore Tso wrote:

> 
> Actually, depending on the size of the chunk, even on Linux
> malloc/read/free can be faster than the mmap/munmap

Yes.

In general, mmap/munmap is faster only if:
 - you access the same data multiple times within one page (ie a single 
   page-fault will actually result in more than one access)
OR
 - you can use it to avoid management overhead (ie you know your data is 
   going to accessed very sparsely, but you don't know the patterns, and 
   trying to keep track of it is painful as hell)

That said, under Linux, mmap is almost never really _slower_ either, which is why this issue never made any real difference. The overhead of doing page table manipulation is pretty much balanced out by the overhead of doing a memcpy.

But that "mmap is fast" is _not_ true on many other operating systems, which is why it might be worthwhile to try something like the appended on OS X, which uses pread() instead of mmap().

This is _not_ very much tested. It seems to work. Caveat emptor. It would be interesting to hear if many small "pread()" calls are faster than many mmap/munmap calls on OS X. I bet they are. Under Linux, there should be almost no difference.

		Linus
---

diff --git a/index-pack.c b/index-pack.c
index 6d6c92b..094f8b2 100644
--- a/index-pack.c
+++ b/index-pack.c
@@ -8,6 +8,7 @@
 #include "tree.h"
 #include <sys/time.h>
 #include <signal.h>
+#include <unistd.h>
 
 static const char index_pack_usage[] =
 "git-index-pack [-v] [-o <index-file>] [{ ---keep | --keep=<msg> }] { <pack-file> | --stdin [--fix-thin] [<pack-file>] }";
@@ -279,27 +280,25 @@ static void *get_data_from_pack(struct object_entry *obj)
 {
 	unsigned long from = obj[0].offset + obj[0].hdr_size;
 	unsigned long len = obj[1].offset - from;
-	unsigned pg_offset = from % getpagesize();
-	unsigned char *map, *data;
+	unsigned char *src, *data;
 	z_stream stream;
 	int st;
 
-	map = mmap(NULL, len + pg_offset, PROT_READ, MAP_PRIVATE,
-		   mmap_fd, from - pg_offset);
-	if (map == MAP_FAILED)
-		die("cannot mmap pack file: %s", strerror(errno));
+	src = xmalloc(len);
+	if (pread(mmap_fd, src, len, from) != len)
+		die("cannot pread pack file: %s", strerror(errno));
 	data = xmalloc(obj->size);
 	memset(&stream, 0, sizeof(stream));
 	stream.next_out = data;
 	stream.avail_out = obj->size;
-	stream.next_in = map + pg_offset;
+	stream.next_in = src;
 	stream.avail_in = len;
 	inflateInit(&stream);
 	while ((st = inflate(&stream, Z_FINISH)) == Z_OK);
 	inflateEnd(&stream);
 	if (st != Z_STREAM_END || stream.total_out != obj->size)
 		die("serious inflate inconsistency");
-	munmap(map, len + pg_offset);
+	free(src);
 	return data;
 }