Re: being nice to patch(1)

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Jul 6, 2007, 18:08 UTC
Message-ID: <alpine.LFD.0.98.0707061102280.9434@woody.linux-foundation.org>
In-Reply-To: <86644xd7wr.fsf@lola.quinscape.zz>

On Fri, 6 Jul 2007, David Kastrup wrote:

Show 6 quoted lines

> 
> Well, hmph!  I just rewrote my git-diff-using script to not check
> stuff into a throw-away git repository, and guess what: with real-life
> use cases (diffing trees of about 500MB size), git-diff runs out of
> memory (the machine probably has something like 1.5GB of virtual memory
> size) when operating outside of a git repository.

Ok, that's probably some huge memory leak that just doesn't show up with any normal git operations, likely simply because all the normal git operations will have thrown out the case of "identical files" without ever even looking at the file.

I'd guess that when using the diff logic on outside files, we'll read them all in, compare them, and keep them all in memory even though they are identical.

Generally, though, "git diff" has a much higher memory footprint than any normal file-by-file recursive diff, exactly because of the rename logic. An external "diff" won't ever have any reason to keep more than two files in memory at a time, but because git diff does rename and copy detection, it wants to keep the file data in memory over much longer times.

But I bet there is some stupid bug where we just make it much much worse for the "no git tree/index" case, and keep the whole tree in memory or something.

(The same is true of "git apply", btw, for a different reason: because git-apply will refuse to write out partial results in case some later patch fails, git-apply will keep the whole result in memory until the very end, and then do the write-out in one go. Again, that obviously means that it will potentially use a lot more memory than the "one patch at a time" approach that regular "patch" does)

			Linus