Re: git annotate runs out of memory

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Dec 11, 2007, 21:14 UTC
Message-ID: <alpine.LFD.0.9999.0712111300440.25032@woody.linux-foundation.org>
In-Reply-To: <alpine.LFD.0.9999.0712111122400.25032@woody.linux-foundation.org>

On Tue, 11 Dec 2007, Linus Torvalds wrote:

> 
> PS. I also do agree that we seem to use an excessive amount of memory 
> there. As to whether it's the same issue or not, I'd not go as far as Nico 
> and say "yes" yet. But it's interesting.

I think the answer here is that git-annotate is a totally different issue.

The blame machinery keeps around all the blobs it has ever needed to do a diff, which explains why something like gcc/ChangeLog blows up badly.

Try this trivial patch.

It will cause us to potentially re-generate some blobs much more, but that's a reasonably cheap operation, and our delta base cache will get the expensive cases.

It's still not a free operation, but I get

	[torvalds@woody gcc]$ /usr/bin/time ~/git/git-blame gcc/ChangeLog > /dev/null
	20.68user 1.25system 0:21.94elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
	0inputs+0outputs (0major+599833minor)pagefaults 0swaps

so it took 22s and I never saw it grow very large either (it grew to 72M resident, but I don't know how much of that was the mmap of the pack-file, so that number is pretty meaningless). Valgrind reports that it used a maximum heap of about 24M, and almost all of that seems to have been in the delta cache (which is all good).

		Linus

----
 builtin-blame.c |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/builtin-blame.c b/builtin-blame.c
index c158d31..18f9924 100644
--- a/builtin-blame.c
+++ b/builtin-blame.c
@@ -87,6 +87,14 @@ struct origin {
 	char path[FLEX_ARRAY];
 };
 
+static void drop_origin_blob(struct origin *o)
+{
+	if (o->file.ptr) {
+		free(o->file.ptr);
+		o->file.ptr = NULL;
+	}
+}
+
 /*
  * Given an origin, prepare mmfile_t structure to be used by the
  * diff machinery
@@ -558,6 +566,8 @@ static struct patch *get_patch(struct origin *parent, struct origin *origin)
 	if (!file_p.ptr || !file_o.ptr)
 		return NULL;
 	patch = compare_buffer(&file_p, &file_o, 0);
+	drop_origin_blob(parent);
+	drop_origin_blob(origin);
 	num_get_patch++;
 	return patch;
 }