Re: [PATCH] Speedup recursive by flushing index only once for all entries

From: Junio C Hamano <junkio@cox.net>
Date: Jan 11, 2007, 23:53 UTC
Message-ID: <7vk5zt15ot.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <Pine.LNX.4.64.0701111424400.3594@woody.osdl.org>

Linus Torvalds <torvalds@osdl.org> writes:

Show 9 quoted lines

> On Thu, 11 Jan 2007, Alex Riesen wrote:
>> 
>> It must have been large leak, as I really have seen the memory usage
>> dropping down significantly.
>
> I really think it was about 6MB (or whatever your index file size was) per 
> every single resolved file. I think merge-recursive used to flush the 
> index file every time it resolved something, and every flush would 
> basically leak the whole buffer used to write the index.

This does not change the conclusion "leaking is bad", but it was not as bad as "the whole buffer used to write the index".

There are 10 4-byte ints, 20-bye SHA1, and a short plus pathname and padding stored per a file and Alex has 44k files. They are not included in that *data the code was leaking, I would suspect maybe a meg or so per path.

The version of merge-recursive in 'next' would not write out the index at all until the very end once, so that makes this leak somewhat irrelevant for that particular program ;-) but thanks for the fix.

Here is the list of what I have queued for 'master' (I am sending the list because it will be some time before I can push them out):

Eric Wong (1):
      Avoid errors and warnings when attempting to do I/O on zero bytes

Junio C Hamano (5):
      Document git-init
      index-pack: write-or-die instead of unchecked write-in-full.
      config-set: check write-in-full returns in set_multivar
      git-rm: do not fail on already removed file.
      git-status: wording update to deal with deleted files.

Linus Torvalds (3):
      write-cache: do not leak the serialized cache-tree data.
      write_in_full: really write in full or return error on disk full.
      Better error messages for corrupt databases