Re: git diff too slow for a file

From: SungHyun Nam <goweol@gmail.com>
Date: Apr 19, 2010, 00:43 UTC
Message-ID: <4BCBA725.2020307@gmail.com>
In-Reply-To: <4BC9D928.50909@lsrfire.ath.cx>

René Scharfe wrote:

Show 49 quoted lines

> Am 29.03.2010 03:42, schrieb SungHyun Nam:
>> Hello,
>>
>> If I run a attached script for bunzipped attached files, I get:
>> (To reduce size, I removed many lines and bzipped.)
>>
>>      $ ./mk.sh
>>      time diff -u x3 x4>/dev/null 2>&1
>>
>>      real    0m0.011s
>>      user    0m0.000s
>>      sys    0m0.010s
>>
>>      time git diff>/dev/null 2>&1
>>
>>      real    0m0.193s
>>      user    0m0.190s
>>      sys    0m0.000s
>>
>>      $ git version
>>      git version 1.7.0.2.273.gc2413
>>
>>      $ diff --version
>>      diff (GNU diffutils) 2.8.1
>>      ...
>>
>> Well, though the files are ascii file, they includes a random
>> hexa-decimal datas, so that I don't interest the diff result at
>> all.  But the real problem is 'rebasing took so long if the file
>> was changed'.  Because the git tree includes several such a file,
>> if they changed, rebase took some miniutes for every branch.
>> Such a branch includes a few lines of changes for a C source file,
>> though.  Now I'm waiting an hour to finish rebasing all the
>> branches and yet a rebasing script is running... :-(
>
> I can reproduce it; I concatenated your example files five times to get
> meaningful timings (x1 = five times x3, x2 = five times x4).
>
> The difference between GNU diff and git diff is that the latter is trying
> hard to minimize the size of the diff.  Each user of the xdiff library in
> git turns on the XDF_NEED_MINIMAL flag, which makes it very expensive
> (specifically the function xdl_split()).
>
> The following patch is not meant for inclusion, but rather to start a
> dicussion.  Is XDF_NEED_MINIMAL a good default to have?
>
> The patch removes XDF_NEED_MINIMAL and replaces it with XDF_QUICK, with
> reversed meaning.  XDF_QUICK is only set if the new option --quick is
> given, so without it the old behaviour is retained.  Some numbers:

The patch is great for me.  Thanks!

Added 'time git diff --quick' to the mk.sh and ran with a original file (about 180000 lines):

     $ ./mk.sh
     time diff -u x3 x4 >/dev/null 2>&1

     real	0m0.794s
     user	0m0.720s
     sys	0m0.010s

     time git diff >/dev/null 2>&1

     real	0m44.687s
     user	0m44.670s
     sys	0m0.020s

     time git diff --quick >/dev/null 2>&1

     real	0m1.853s
     user	0m1.840s
     sys	0m0.010s

Thanks! namsh