Re: [git-for-windows] How is detected binary files?

Johannes Schindelin· Nov 27, 2015, 14:14 UTC · lore

Hi Andrzej,

On Wed, 25 Nov 2015, Andrzej Borucki wrote:

Show 5 quoted lines

> How git detects that file is binary? This must be safe because it not 
> allowed to change line breaks in binary files. 
> Binary files can contain byte 0 (zero), but:
> - 16 bit UTF also can contain zero
> - short binary files can not contain zero

It would probably be better to direct this question to the general Git mailing list (you reached the Git for Windows one, and this issue is not specific to Windows).

To answer your question, a NUL byte within the first 8000 bytes is indeed considered as an indicator for binary files.

If you use UTF-16, you will need to mark your files as such explicitly (Git does not handle UTF-16 internally).

As to short binary files, you will also have to mark those explicitly.

Ciao, Johannes

Jeff King· Dec 2, 2015, 00:49 UTC · re: Johannes Schindelin · lore

On Fri, Nov 27, 2015 at 03:14:58PM +0100, Johannes Schindelin wrote:

Show 17 quoted lines

> On Wed, 25 Nov 2015, Andrzej Borucki wrote:
> 
> > How git detects that file is binary? This must be safe because it not 
> > allowed to change line breaks in binary files. 
> > Binary files can contain byte 0 (zero), but:
> > - 16 bit UTF also can contain zero
> > - short binary files can not contain zero
> 
> It would probably be better to direct this question to the general Git
> mailing list (you reached the Git for Windows one, and this issue is not
> specific to Windows).
> 
> To answer your question, a NUL byte within the first 8000 bytes is indeed
> considered as an indicator for binary files.
> 
> If you use UTF-16, you will need to mark your files as such explicitly
> (Git does not handle UTF-16 internally).

I'm not sure if it is a good idea to treat UTF-16 as text. The rest of the diff (headers, etc) will all be in ASCII, so one or the other is going to be mojibake.

You can get readable diffs by textconv-ing them to an ASCII-superset encoding like UTF-8. Something like:

    echo 'myfile diff=utf16' >.gitattributes
    git config diff.utf16.textconv 'iconv -f utf16 -t utf8'

but of course the resulting patches cannot be applied, and you may miss any changes that do not make it through the encoding (e.g., using different bytes to represent the same code point).

-Peff