Re: [RFC \ WISH] Add -o option to git-rev-list

From: Linus Torvalds <torvalds@osdl.org>
Date: Dec 10, 2006, 20:08 UTC
Message-ID: <Pine.LNX.4.64.0612101159530.12500@woody.osdl.org>
In-Reply-To: <e5bfff550612101151r5e77ddc0w8f61f36fc1f2920e@mail.gmail.com>

On Sun, 10 Dec 2006, Marco Costalba wrote:

Show 6 quoted lines

> 
> - memory use: the next natural step with files is, instead of loading
> the file content in memory and *keep it there*, we could load one
> chunk at a time, index the chunk and discard. At the end we keep in
> memory only indexing info to quickly get to the data when needed, but
> the big part of data stay on the file.

Well, that's still just going slower than swapping.

The reason loading things into memory is nice is that:
 - in the common case, you don't need to do anything else.
 - if  the machine is low on memory, it can page things out just about as 
   easily as you could write things to a file anyway.

So don't worry too much about low-memory situations. Yes, there are cases where it's better to keep things in files and simply not have a big working set AT ALL, but if you keep something in a file and the file data is still part of the working set (ie you read it several times, but at the beginning and the end), that really isn't any better than having it in memory.

So the time to try to optimize memory usage is really only for "streaming behaviour" - where you need to touch something only once. Then the best option is to actually use a pipe and re-use the memory, but if you have file data, you can use things like fadvise(DONTNEED).

> - This is probably my ignorance, but experimenting with popen() I
> found I could not know *when* git-rev-list ends because both feof()
> and ferror() give 0 after a fread() with git-rev-list already defunct.

I suspect you had a bug somewhere. It could be a bug in stdio, but I doubt it.

You do realize that the correct way to check "feof()" is only _after_ fread() returns 0?

Stdio ferror/feof is horrible to use corrrectly, and few people get it right. Mostly because it's such a crap interface thanks to being "easy" to use and thus hiding all the error handling details on purpose.