Re: git-diff-tree inordinately (O(M*N)) slow on files with many changes

From: Linus Torvalds <torvalds@osdl.org>
Date: Oct 16, 2006, 16:54 UTC
Message-ID: <Pine.LNX.4.64.0610160948450.3962@g5.osdl.org>
In-Reply-To: <87mz7wp6ek.fsf@rho.meyering.net>

On Mon, 16 Oct 2006, Jim Meyering wrote:

Show 12 quoted lines

> Linus Torvalds <torvalds@osdl.org> wrote:
> > On Mon, 16 Oct 2006, Linus Torvalds wrote:
> ...
> > So I think xdiff has chosen too small a hash. Can you try what happens if
> > you change xdl_hashbits() (in xdiff/xutil.c) instead? Try making it return
> > a bigger value (for example, by initializing "bits" to 2 instead of 0),
> > and see if that makes a difference.
> 
> It makes no difference.
> 
> Bear in mind that there are a *lot* of duplicate lines in the files
> being compared: filtering each through "sort -u" removes 40-50k lines.

It can't be due to duplicate lines. If the lines are truly duplicate, then they'd get the same 32-bit hash value, and then the first conditional in the expression would always be true, and then it wouldn't _matter_ if it's a "&&" or a "||".

See?

So as far as I can tell it has to be some kind of collission on the hash queue with _different_ hash values being queued on the same hash queue.

Now, it could be that there's a bad hash algorithm somewhere (eg if XDL_HASHLONG() just does horribly badly in distributing the hash values onto the hash queues, you'd see this _regardless_ of how many bits you have, just because it clumps).

Or there could be something else that I'm just missing..

It would probably be nice to just get a sampling of what the hash-queue looks like for the bad case? Maybe it would be obvious that certain different hash values then get the same XDL_HASHLONG() thing..

		Linus