Re: [RFC \ WISH] Add -o option to git-rev-list

From: Linus Torvalds <torvalds@osdl.org>
Date: Dec 10, 2006, 22:53 UTC
Message-ID: <Pine.LNX.4.64.0612101439540.12500@woody.osdl.org>
In-Reply-To: <e5bfff550612101435o6bc938acmac28ad6adf0c8844@mail.gmail.com>

On Sun, 10 Dec 2006, Marco Costalba wrote:

> 
> I consistently have len == 65536 bytes until the last fread() where
> it's less.

That's because fread() will block until it gets all data. Did you actually ever try this with a uncached tree and did you compare what you got with a plain "read()".

On older kernels, I guarantee that you get 4kB at a time for reads, even for a blocking pipe. Because we have bigger pipe buffers these days, it _may_ return 64kB at a time every time, but only if the writer is much faster than the reader.

Based on the fact that you say that "read()" was ten times slower than fread(), I very much suspect you got 4kB at a time, and then slept 100ms each time or something.

Anyway, you should seriously also check the case when "git rev-list" is slow because you have cold caches and unpacked objects. You can't wait for it to synchronously write a thousand lines or so - that could take seconds.

> How can open in nonblocking mode with popen() ?
> 
> FILE *popen(const char *command, const char *type);

something like

	fcntl(fileno(file), F_SETFL, O_NONBLOCK);

will do it, and then your loop should look something like

	for (;;) {
		int count = read(fileno(file), buf, BUFSIZE);
		if (!count) {
			/* All done, no more to read */
			return 0;
		}
		if (count < 0) {
			if (errno == EAGAIN)
				break;
			if (errno == EINTR)
				continue;
			/* Anything else is fatal - report error */
			return -1;
		}
		... handle 'count' new bytes here ...
	}
	.. this is the EAGAIN case ..
	either set polling on the file descriptor, or use a
	timer here yo get back to this loop eventually.

looks about right.