Re: [PATCH] write-tree performance problems

On Wed, 20 Apr 2005, Chris Mason wrote:

If you ever have a cache_entry bigger than 16384, your code will write things out in the wrong order (write the new cache without flushing the old buffer).

You also don't free the buffer.

Finally, if you really want to go fast, you should really try to make your writes powers-of-two, ie fill up the buffer entirely rather than saying "if I were to overflow, flush it now". It doesn't matter that much for some filesystems (especially local and append-only like the patterns are here), but it can definitely matter for the stupid ones.

But yeah, we could obviously chunk things out properly. You might want to just use stdio and "fwrite()", though, which does all of that for you, and hopefully does it right.

(I'm not a big fan of stdio for something like this, so if you want to create a little helper function that just does the chunking, go wild. Something like

	#define BUFSIZ 8192
	static char buffer[BUFSIZ];
	static unsigned long buflen;

	int ce_write(int fd, void *data, unsigned int len)
	{
		while (len) {
			unsigned int buffered = buflen;
			unsigned int partial = BUFSIZ - buflen;
			if (partial > len)
				partial = len;
			memcpy(buffer + buflen, data, partial);
			buffered += partial;
			if (buffered == BUFSIZ) {
				if (write(fd, buffer, BUFSIZ) != BUFSIZ)
					die("unable to write");
				buffered = 0;
			}
			buflen = buffered;
			len -= partial;
			data += partial;
		}
	}

	int ce_flush(int fd)
	{
		unsigned int left = buflen;
		if (left) {
			buflen = 0;
			if (write(fd, buffer, left) != left)
				die("unable to write");
		}
	}

which should be ok, and cheesily avoids the allocation overhread issues by just having a nice static buffer.

"If you want to go fast, do it right".

Untested, as usual.

		Linus