Re: [PATCH] write-tree performance problems

From: Chris Mason <mason@suse.com>
Date: Apr 20, 2005, 16:37 UTC
Message-ID: <200504201237.38374.mason@suse.com>
In-Reply-To: <Pine.LNX.4.58.0504200833580.6467@ppc970.osdl.org>

On Wednesday 20 April 2005 11:40, Linus Torvalds wrote:

Show 6 quoted lines

> On Wed, 20 Apr 2005, Chris Mason wrote:
> > Thanks for looking at this.  Your new tree is faster, it gets the commit
> > 100 patches time down from 1m5s to 50s.
>
> It really _shouldn't_ be faster. It still does the compression, and throws
> the end result away.

Well, that's a little odd. I had thought about making sure you did this change and forgotten. 1 minute benchmarks are a horrible idea since they run into noise with cache writebacks. I should know better...

At any rate, the time for a single write-tree is pretty consistent. Before it was around .5 seconds, and with this change it goes down to .128s. My patch was .024.

The 100 patch time is down to 32s (3 run average). This is close enough that I don't think my patch is worth it if no other part of git can benefit from having trees in the index.

>
> To actually go faster, it _should_ need this patch. Untested. See if it
> works..

Thanks. This one missed the filling in the returnsha1.  New patch attached.

-chris

diff -u linus.back/sha1_file.c linus/sha1_file.c --- linus.back/sha1_file.c 2005-04-20 12:31:00.240181016 -0400 +++ linus/sha1_file.c 2005-04-20 12:13:56.339837528 -0400

@@ -173,12 +173,27 @@
 	z_stream stream;
 	unsigned char sha1[20];
 	SHA_CTX c;
+	char *filename;
+	int fd;
 
 	/* Sha1.. */
 	SHA1_Init(&c);
 	SHA1_Update(&c, buf, len);
 	SHA1_Final(sha1, &c);
 
+	filename = sha1_file_name(sha1);
+	fd = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
+	if (fd < 0) {
+		if (errno != EEXIST)
+			return -1;
+
+		/*
+		 * We might do collision checking here, but we'd need to
+		 * uncompress the old file and check it. Later.
+		 */
+		goto out;
+	}
+
 	/* Set it up */
 	memset(&stream, 0, sizeof(stream));
 	deflateInit(&stream, Z_BEST_COMPRESSION);
@@ -195,8 +210,10 @@
 	deflateEnd(&stream);
 	size = stream.total_out;
 
-	if (write_sha1_buffer(sha1, compressed, size) < 0)
-		return -1;
+	if (write(fd, compressed, size) != size)
+		die("unable to write file");
+	close(fd);
+out:		
 	if (returnsha1)
 		memcpy(returnsha1, sha1, 20);
 	return 0;