Re: git annotate runs out of memory

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Dec 11, 2007, 19:42 UTC
Message-ID: <alpine.LFD.0.9999.0712111122400.25032@woody.linux-foundation.org>
In-Reply-To: <4aca3dc20712111109y5d74a292rf29be6308932393c@mail.gmail.com>

On Tue, 11 Dec 2007, Daniel Berlin wrote:

>
> I understand this, and completely agree with you.
> However, I cannot force GCC people to adopt completely new workflow in
> this regard.

Oh, I agree. It's why we do have "git blame" these days, and it's why I've tried to make people use the nicer incremental mode, which is not at all faster, but it's a hell of a lot more pleasant to use because you get some output immediately.

In other words,

	git blame gcc/ChangeLog

is virtually useless because it's too expensive, but try doing

	git gui blame gcc ChangeLog

instead, and doesn't that just seem nicer? (*)

The difference is that the GUI one does it incrementally, and doesn't have to get _all_ the results before it can start reporting blame.

Not that I claim that the gui blame is perfect either (I dunno why it delays the nice coloring so long, for example), but it was something I pushed - and others made the gui for - exactly to help people with the fact that git interally really does it that incremental way.

> SVN had the same problem (the file retrieval was the most expensive op
> on FSFS). One of the things i did to speed it up tremendously was to
> do the annotate from newest to oldest (IE in reverse), and stop
> annotating when we had come up with annotate info for all the lines.

We do that. The expense for git is that we don't do the revisions as a single file at all. We'll look through each commit, check whether the "gcc" directory changed, if it did, we'll go into it, and check whether the "ChangeLog" file changed - and if it did, we'll actually diff it against the previous version.

> In GCC history, it is likely you will be able to cut off at least 30%
> of the time if you do this, because files often have changed entirely
> multiple times.

Not gcc/ChangeLog, though (apart from the renames that happen occasionally).

Btw, an example of something git *should* do right, but is just too damn expensive, is doing

	git gui blame gcc/ChangeLog-2000

and have it actually be able to track the original source of each of those annotations across that "ChangeLog split from hell".

I bet it would eventually get it right, but that's a large file, way back in history, and it will try to do a non-whitespace blame with copy detection.

That's *expensive*, although it is an amusing thing to try to do ;)

			Linus

PS. I also do agree that we seem to use an excessive amount of memory there. As to whether it's the same issue or not, I'd not go as far as Nico and say "yes" yet. But it's interesting.

It's not entirely surprising that we see multiple issues with the gcc repo, simply because it's not the kind of repo that people have ever really worked on. So I don't think it's necessarily related at all, except in the sense of it being a different load and showing issues.