Re: utf8 BOM

From: Eyvind Bernhardsen <eyvind.bernhardsen@gmail.com>
Date: May 16, 2010, 10:37 UTC
Message-ID: <00E0B9AC-2A2E-4F95-9B35-F3F63EBC3CF3@gmail.com>
In-Reply-To: <20100516051927.GA17200@dpotapov.dyndns.org>

On 16. mai 2010, at 07.19, Dmitry Potapov wrote:

Show 8 quoted lines

> On Sat, May 15, 2010 at 10:23:52PM +0200, Eyvind Bernhardsen wrote:
>> (which hopefully works no matter what your code
>> page is?  I don't know much about Windows i18n).
> 
> Yes, it does. I am not an expert on Windows either, but as far as I
> know, BOM are used to mark unicode files, which could be either UTF-8
> or UTF-16. BTW, UTF-16 are treated by Git as "binary" now, which may
> not always convenient, because impossible to do "merge" or "diff".

Okay, so something that checks text files to see if they're utf16 (maybe just accept anything with a utf16 BOM as utf16?) and converts them to utf8 might be useful on any platform.  Stripping utf8 BOMs and optionally re-adding them on output would be a natural extension.  "core.autoutf", anyone?

Show 9 quoted lines

>> Adding this to convert.c would be more difficult, at least
>> politically, since I assume it would be Windows-specific code.
> 
> I don't think it needs any Windows-specific code. We already have some
> functions to convert text from different charsets, which could be used.
> But this feature should be developed and tested by people who work on
> Windows regularly and need this feature, because there is no substitute
> for testing and experience of how well it works in practice. Currently,
> I rarely use Windows and can get by clean/smudge filters.

Yeah, the problem is finding someone who needs the feature _and_ is able/willing to implement it.  I try to keep a Unix-like experience on Windows, so I don't usually run into utf8 BOMs.

-- 
Eyvind