{"thread":{"id":"21495","subject":"Timur Sufiev: Re: [PATCH I18N filenames v2 3/3] Provide compatibility with MinGW","startedAt":"2009-11-06T10:00:42Z","lastAt":"2009-11-06T14:55:20Z","messageCount":2,"participants":["Timur Sufiev","Peter Krefting"],"isPatch":true,"patchVersion":2,"patchTotal":3},"messages":[{"id":"126973","messageId":"4af3f3cb.0305560a.7a34.0f43@mx.google.com","threadId":"21495","inReplyTo":null,"subject":"Timur Sufiev: Re: [PATCH I18N filenames v2 3/3] Provide compatibility with MinGW","fromName":"Timur Sufiev","fromEmail":"tsufiev@gmail.com","sentAt":"2009-11-06T10:00:42Z","receivedAt":"2009-11-06T10:00:42Z","isPatch":true,"sender":{"key":"tsufiev@gmail.com","avatar":"https://gravatar.com/avatar/e4a068ddcf68fc9b207d8be5cd98db47642d7c3d465191cdab7237538371144d?d=mp&s=160"},"body":"Hello, Peter\n> Hi!\n> \n> Instead of calling the open_i18n() which converts from UTF-8 to a local \n> 8-bit character set, this should probably call a version that converts from \n> UTF-8 to UTF-16 and uses _wopen().\n> \n> Same thing for fopen_i18n() and _wfopen().\n> \n> I created a small RFC patch for that that changed parts of the system \n> earlier this year - http://kerneltrap.org/mailarchive/git/2009/3/2/5350814\n> \n> I did not address readdir() and friends, I'm not sure if they are available \n> in UTF-16 form or if they need to be rewritten using findfirst()/findnext().\n> \n> -- \n> \\\\// Peter - http://www.softwolves.pp.se/\n\nI've decided to stick to local 8-bit encoding for now having considered\nthe following issues:\n\n1. Many git front-ends, e.g. TortoiseGit, use 8-bit set, not UTF-16:\nthey call git plumbing commands and pass filenames to command line (in\nlocal 8-bit encoding). So, using [UTF-8] <-> [UTF-16] approach, I had to\ndeal with 3 different encodings: UTF-8, UTF-16 and local 8-bit one\n(CP1251 in my case). Moreover, Windows itself uses both UTF-16 and\nCP1251, so one had to deal with reencoding between them (if he plans to\nsupport UTF-16). Too much confusion.\n \n2. UTF-16 is a proper solution for Windows, but my patch is useful for\nother OSes with locales different from UTF-8 (e.g. Linux with KOI8-R\nlocale).\n\nStill there is a possibility that one day we'll stumble upon some UTF-8\nsymbol which cannot not be correctly mapped into 8-bit encoding. UTF-16\nwould be a remedy in this case, but what if don't have it (see 2)?\n-- \nTimur Sufiev\n\n\n-- \nTimur Sufiev\n"},{"id":"126987","messageId":"alpine.DEB.2.00.0911061549380.31816@perkele.intern.softwolves.pp.se","threadId":"21495","inReplyTo":"4af3f3cb.0305560a.7a34.0f43@mx.google.com","subject":"Re: Timur Sufiev: Re: [PATCH I18N filenames v2 3/3] Provide compatibility with MinGW","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-11-06T14:55:20Z","receivedAt":"2009-11-06T14:55:20Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Timur Sufiev:\n\n> 1. Many git front-ends, e.g. TortoiseGit, use 8-bit set, not UTF-16:\n\nAll of them do, that is because the output is using 8-bit. That is why the \ninternal encoding need to remain eight-bit, for instance UTF-8.\n\n> they call git plumbing commands and pass filenames to command line (in \n> local 8-bit encoding).\n\nWell, yes. On Windows, however, there is the complication that the command \nline is available in two versions. There is a eight-bit and a UTF-16 version \nof it. Which one is constructed from which depends on how the application \nwas launched. We can read the UTF-16 version and hope that it contains \nproper names (possibly looking at the eight-bit version as UTF-8 if \nnecessary).\n\n> 2. UTF-16 is a proper solution for Windows, but my patch is useful for \n> other OSes with locales different from UTF-8 (e.g. Linux with KOI8-R \n> locale).\n\nWell, your patch re-implements the fopen() calls, converting the file name \nat that point (as well as readdir() and friends). I would do that on Windows \nas well, with the modification that on Windows, I would convert to UTF-16 \nand use _wfopen() instead. On systems that have it, you could also make it \nconvert to UTF-32 and use their wfopen() (I'm not aware of many other OSes \nhaving those functions, though).\n\n> Still there is a possibility that one day we'll stumble upon some UTF-8 \n> symbol which cannot not be correctly mapped into 8-bit encoding. UTF-16 \n> would be a remedy in this case, but what if don't have it (see 2)?\n\nThat is of course an issue. There are several approaches to that:\n\n- Fail with an error.\n- Convert to a place-holder character.\n- QP encode the file name, perhaps?\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"}]}