# Timur Sufiev: Re: [PATCH I18N filenames v2 3/3] Provide compatibility with MinGW 2 messages from 2009-11-06 to 2009-11-06. Participants: Timur Sufiev, Peter Krefting. Thread: https://gitlist.dev/t/21495 ## Timur Sufiev, 2009-11-06 10:00 Subject: Timur Sufiev: Re: [PATCH I18N filenames v2 3/3] Provide compatibility with MinGW Message-ID: <4af3f3cb.0305560a.7a34.0f43@mx.google.com> URL: https://gitlist.dev/e/4af3f3cb.0305560a.7a34.0f43%40mx.google.com ``` Hello, Peter > Hi! > > Instead of calling the open_i18n() which converts from UTF-8 to a local > 8-bit character set, this should probably call a version that converts from > UTF-8 to UTF-16 and uses _wopen(). > > Same thing for fopen_i18n() and _wfopen(). > > I created a small RFC patch for that that changed parts of the system > earlier this year - http://kerneltrap.org/mailarchive/git/2009/3/2/5350814 > > I did not address readdir() and friends, I'm not sure if they are available > in UTF-16 form or if they need to be rewritten using findfirst()/findnext(). > > -- > \\// Peter - http://www.softwolves.pp.se/ I've decided to stick to local 8-bit encoding for now having considered the following issues: 1. Many git front-ends, e.g. TortoiseGit, use 8-bit set, not UTF-16: they call git plumbing commands and pass filenames to command line (in local 8-bit encoding). So, using [UTF-8] <-> [UTF-16] approach, I had to deal with 3 different encodings: UTF-8, UTF-16 and local 8-bit one (CP1251 in my case). Moreover, Windows itself uses both UTF-16 and CP1251, so one had to deal with reencoding between them (if he plans to support UTF-16). Too much confusion. 2. UTF-16 is a proper solution for Windows, but my patch is useful for other OSes with locales different from UTF-8 (e.g. Linux with KOI8-R locale). Still there is a possibility that one day we'll stumble upon some UTF-8 symbol which cannot not be correctly mapped into 8-bit encoding. UTF-16 would be a remedy in this case, but what if don't have it (see 2)? -- Timur Sufiev -- Timur Sufiev ``` ## Peter Krefting, 2009-11-06 14:55 Subject: Re: Timur Sufiev: Re: [PATCH I18N filenames v2 3/3] Provide compatibility with MinGW Message-ID: URL: https://gitlist.dev/e/alpine.DEB.2.00.0911061549380.31816%40perkele.intern.softwolves.pp.se In-Reply-To: <4af3f3cb.0305560a.7a34.0f43@mx.google.com> ``` Timur Sufiev: > 1. Many git front-ends, e.g. TortoiseGit, use 8-bit set, not UTF-16: All of them do, that is because the output is using 8-bit. That is why the internal encoding need to remain eight-bit, for instance UTF-8. > they call git plumbing commands and pass filenames to command line (in > local 8-bit encoding). Well, yes. On Windows, however, there is the complication that the command line is available in two versions. There is a eight-bit and a UTF-16 version of it. Which one is constructed from which depends on how the application was launched. We can read the UTF-16 version and hope that it contains proper names (possibly looking at the eight-bit version as UTF-8 if necessary). > 2. UTF-16 is a proper solution for Windows, but my patch is useful for > other OSes with locales different from UTF-8 (e.g. Linux with KOI8-R > locale). Well, your patch re-implements the fopen() calls, converting the file name at that point (as well as readdir() and friends). I would do that on Windows as well, with the modification that on Windows, I would convert to UTF-16 and use _wfopen() instead. On systems that have it, you could also make it convert to UTF-32 and use their wfopen() (I'm not aware of many other OSes having those functions, though). > Still there is a possibility that one day we'll stumble upon some UTF-8 > symbol which cannot not be correctly mapped into 8-bit encoding. UTF-16 > would be a remedy in this case, but what if don't have it (see 2)? That is of course an issue. There are several approaches to that: - Fail with an error. - Convert to a place-holder character. - QP encode the file name, perhaps? -- \\// Peter - http://www.softwolves.pp.se/ ```