Re: [PATCH 0/2] Improve documentation on UTF-16

From: Johannes Sixt <j6t@kdbg.org>
Date: Dec 28, 2018, 08:59 UTC
Message-ID: <34d4f912-2ec3-9dd1-f5fb-aad6a26e1464@kdbg.org>
In-Reply-To: <20181227234535.GD423984@genre.crustytoothpaste.net>

Am 28.12.18 um 00:45 schrieb brian m. carlson:

Show 11 quoted lines

> On Thu, Dec 27, 2018 at 08:55:27PM +0100, Johannes Sixt wrote:
>> But why do you add another U+FEFF on the way to UTF-8? There is one in the
>> incoming UTF-16 data, and only *that* one must be converted. If there is no
>> U+FEFF in the UTF-16 data, the should not be one in UTF-8, either.
>> Puzzled...
> 
> So for UTF-16, there must be a BOM. For UTF-16LE and UTF-16BE, there
> must not be a BOM. So if we do this:
> 
>    $ printf '\xfe\xff\x00\x0a' | iconv -f UTF-16BE -t UTF-16 | xxd -g1
>    00000000: ff fe ff fe 0a 00                                ......

What sort of braindamage is this? Fix iconv.

But as I said, I'm not an expert. I just vented my worries that widespread existing practice would be ignored under the excuse "you are the outlier".

-- Hannes