RE: git-rebase is ignoring working-tree-encoding
- From
- Alexandre Grigoriev <alegrigoriev@gmail.com>
- Date
- Dec 26, 2018, 00:56 UTC
- Message-ID
- <002201d49cb5$cc554160$64ffc420$@gmail.com>
- In-Reply-To
- <20181108170230.GA6652@tor.lan>
Show 19 quoted lines
> -----Original Message----- > From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On > Behalf Of Torsten Bogershausen > Sent: Thursday, November 8, 2018 9:03 AM > To: Adrián Gimeno Balaguer > Cc: git@vger.kernel.org > Subject: Re: git-rebase is ignoring working-tree-encoding > > On Wed, Nov 07, 2018 at 05:38:18AM +0100, Adrián Gimeno Balaguer wrote: > > Hello Torsten, > > > > Thanks for answering. > > > > Answering to your question, I removed the comments with "rebase" since > > my reported encoding issue happens on more simpler operations > > (described in the PR), and the problem is not directly related to > > rebasing, so I considered it better in order to avoid unrelated > > confusions. > >
Show 8 quoted lines
> OK, I think I understand your problem now. > The file format which you ask for could be named "UTF-16-BOM-LE", > but that does not exist in reality. > If you use UTF-16, then there must be a BOM, and if there is a BOM, > then a Unicode-aware application -should- be able to handle it. > > Why does your project require such a format ? >
Many tools in Windows still do not understand UTF-8, although it's getting better. I think Windows is about the only OS where tools still require UTF-16 for full internationalization. Many tools written in C use MSVC RTL, where fopen(), unfortunately, doesn't understand UTF-16BE (though such a rudimentary program as Notepad does).
For this reason, it's very reasonable to ask that the programming tools produce UTF-16 files with particular endianness, natural for the platform they're running on.
The iconv programmers' boneheaded decision to always produce UTF-16BE with BOM for UTF-16 output doesn't make sense. Again, git and iconv/libiconv in Centos on x86 do the right thing and produce UTF-16LE with BOM in this case.
Also, iconv/libiconv should not be rejecting files with BOM for input encoding UTF-16BE or UTF-16LE. The BOM is not some magic tag. It's just a zero-width space, with unique property that its 8 and 16 bit encoding variants can be recognized one from another. It can appear anywhere in a file. If it's a first character in the file, then the file encoding can be reliably detected. But it's just a character, and iconv should be accepting such files as valid.