Re: Bug? Error during commit

From: Jeff King <peff@peff.net>
Date: Feb 7, 2018, 20:45 UTC
Message-ID: <20180207204520.GA31757@sigill.intra.peff.net>
In-Reply-To: <CACsJy8CGQ4ynYFT0mY1DfcGGdzwP36eonMvr-kEZazX_82ag2Q@mail.gmail.com>

On Mon, Feb 05, 2018 at 08:59:52PM +0700, Duy Nguyen wrote:

Show 19 quoted lines

> On Mon, Feb 5, 2018 at 8:48 PM, Andreas Kalz <andreas-kalz@gmx.de> wrote:
> > Hello,
> >
> > I am using git frequently and usually it is running great.
> >
> > I read to write to this eMail address regarding problems and possible bugs.
> > I am using git version 2.16.1.windows.2 / 64 Bit and during commit the following error message comes up:
> > e:\Internet>git commit -m 2018-01-27
> > fatal: unable to generate diffstat for Thunderbird/andreas-kalz.de/Mail/pop.gmx.net/Inbox
> > [master f74cf30] 2018-01-27
> >
> > I also tried this before with an older git version with same problem.
> >
> > Can you help me with this problem please? Thanks in advance.
> 
> I think if you add -q to that "git commit" command, diffstat is not
> generated and you can get past that. If that particular commit can be
> published in public, it'll help us find out why diffstat could not be
> generated.

I think that's the first time I've seen that particular error. :)

I think the only reason that xdiff would report failure is if malloc() failed, or if one of the files exceeds MAX_XDIFF_SIZE, which is ~1GB. I think we'd usually avoid doing a text diff on anything over core.bigFileThreshold, though.

But it doesn't seem to work:

  $ yes | head -c $((1024*1024*1024 - 10*1024*1024)) >file
  $ git add file
  $ git commit -m one
  $ yes | head -c $((1024*1024*1024)) >file
  $ git commit -am two
  fatal: unable to generate diffstat for file

What's weird is that if I run "git show --stat" on the same commit, it works! So there's something about how commit invokes the diff that doesn't let the big-file check kick in.

It looks like the logic in diff_filespec_is_binary() will only check big_file_threshold if we haven't already loaded the contents into RAM. So "commit" does that, but "diff" is more careful about not loading the file contents.

I think we probably ought to consider anything over big_file_threshold to be binary, no matter what. Possibly even if the user gave us a .gitattribute that says "no really, this is text". Because that 1GB limit is a hard limit that the code can't cope with; our options are either to generate a binary diff or to die.

-Peff