Re: [PATCH] Speedup recursive by flushing index only once for all entries

From: Junio C Hamano <junkio@cox.net>
Date: Jan 11, 2007, 20:23 UTC
Message-ID: <7vfyah48j2.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <Pine.LNX.4.64.0701110823300.3594@woody.osdl.org>

Linus Torvalds <torvalds@osdl.org> writes:

Show 11 quoted lines

> That said, I think we actually have another problem entirely:
>
> Look at "write_cache()", Junio: isn't it leaking memory like mad?
>
> Shouldn't we have something like this?
>
> It's entirely possible that the _real_ problem with the "flush the index 
> all the time" was that it just caused this bug: tons and tons of lost 
> memory, causing git-merge-recursive to grow explosively (~6MB per 
> cache flush, and a _lot_ of cache flushes), which on a 384MB machine 
> quickly uses up memory and causes totally unnecessary swapping.

You are right -- there is absolutely no reason to retain this memory. It is a serialized representation of cache-tree data only to be stored in the index, and no other user of this data exists. Thanks for spotting this.

Writing out 6MB per every path changed in a merge would still be an unnecessary overhead over the one in 'next', so there is no reason to replace 'next' with this single liner of yours, but I am interested in seeing how much of the 20-minute vs 1-minute difference is attributable to this leak, just out of curiosity.

Alex, if you have a chance, could you apply Linus's single-liner on top of 'master', without either of the merge-recursive patches in 'next', and see what kind of numbers you would get?