Re: git annotate runs out of memory

From: Davide Libenzi <davidel@xmailserver.org>
Date: Dec 12, 2007, 00:22 UTC
Message-ID: <Pine.LNX.4.64.0712111611570.1671@alien.or.mcafeemobile.com>
In-Reply-To: <alpine.LFD.0.9999.0712111548200.25032@woody.linux-foundation.org>

On Tue, 11 Dec 2007, Linus Torvalds wrote:

Show 25 quoted lines

> On Tue, 11 Dec 2007, Linus Torvalds wrote:
> > 
> > and while I suspect xdiff could be optimized a bit more for the cases 
> > where we have no changes at the end, that's beyond my skills.
> 
> Ok, I lied.
> 
> Nothing is beyond my skills. My mad k0der skillz are unbeatable.
> 
> This speeds up git-blame on ChangeLog-style files by a big amount, by just 
> ignoring the common end that we don't care about, since we don't want any 
> context anyway at that point. So I now get:
> 
> 	[torvalds@woody gcc]$ time git blame gcc/ChangeLog > /dev/null
> 
> 	real    0m7.031s
> 	user    0m6.852s
> 	sys     0m0.180s
> 
> which seems quite reasonable, and is about three times faster than trying 
> to diff those big files.
> 
> Davide: this really _does_ make a huge difference. Maybe xdiff itself 
> should do this optimization on its own, rather than have the caller hack 
> around the fact that xdiff doesn't handle this common case all that well?

I didn't follow the thread, but I can guess from the subject that this is about memory, isn't it? Libxdiff already has a xdl_trim_ends() that strips all the common beginning and ending records, but at that point files are already loaded. Since libxdiff works with memory files in order to keep any sort of system dependency out of the window, so the optimization would be useless on libxdiff side. This because the user would have to have already the file loaded in memory, to pass it to libxdiff. If this is really about memory, this better be kept on the libxdiff caller side, so that it can avoid loading the terminal file sections altogether. About your code, you may want to have an extend-till-next-eol code after the trimming part, since the last line may be used for context in the diffs.

- Davide