{"thread":{"id":"18122","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","startedAt":"2009-03-03T20:39:07Z","lastAt":"2009-03-03T21:56:44Z","messageCount":3,"participants":["John Dlugosz","Dmitry Potapov"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"106928","messageId":"450196A1AAAE4B42A00A8B27A59278E709F077AC@EXCHANGE.trad.tradestation.com","threadId":"18122","inReplyTo":null,"subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"John Dlugosz","fromEmail":"jdlugosz@tradestation.com","sentAt":"2009-03-03T20:39:07Z","receivedAt":"2009-03-03T20:39:07Z","isPatch":true,"sender":{"key":"jdlugosz@tradestation.com","avatar":null},"body":"Re: AFAIK, Microsoft C runtime library does not support UTF-8,\n\nActually, here is a clip from the runtime library source code:\n\n        tmode = _textmode(fh);\n\n        switch(tmode) {\n            case __IOINFO_TM_UTF8 :\n                /* For a UTF-8 file, we need 2 buffers, because after\nreading we\n                   need to convert it into UNICODE - MultiByteToWideChar\ndoesn't do\n                   in-place conversions. */\n\n                /* MultiByte To WideChar conversion may double the size\nof the\n                   buffer required & hence we divide cnt by 2 */\n\n                /*\n                 * Since we are reading UTF8 stream, cnt bytes read may\nvary\n                 * from cnt wchar_t characters to cnt/4 wchar_t\ncharacters. For\n                 * this reason if we need to read cnt characters, we\nwill\n                 * allocate MBCS buffer of cnt. In case cnt is 0, we\nwill\n                 * have 4 as minimum value. This will make sure we don't\n                 * overflow for reading from pipe case.\n                 *\n                 *\n                 * In this case the numbers of wchar_t characters that\nwe can\n                 * read is cnt/2. This means that the buffer size that\nwe will\n                 * require is cnt/2.\n                 */\n\n                /* For UTF8 we want the count to be an even number */\n\nThis is in the _read(fd, buffer, count) function, and shows that it will\nin fact read UTF-8 and automatically transform it to UTF-16LE\ntransparently.  The documentation for _open explains this feature.\n\nMeanwhile, a quick look at _mbslen() etc. shows that they are\nimplemented, and will handle UTF-8 encoded text as variable-length char*\njust fine as long as suitable tables are loaded in its locale.  An\ninternal header shows macros for generating the lead-byte information as\nneeded by that table.\n\nNow, the default when a program starts is to use the \"C\" locale.  The\nlocale argument to setlocale can take a form \".code_page\", so calling\n\n\tsetlocale (LC_CTYPE, \".65001\");\n\nshould do the trick.  Assuming, that is, that you don't hit macros that\nassume that characters are never multibyte.  So define the preprocessor\nsymbol _MBCS when you compile.\n\nOlder versions might not work right because MBCS (multibyte character\nstrings) was only actually implemented to DBCS (double-byte).  That is,\na single lead byte would be followed by a second byte, and no other\ncases are provided for.  But, GB18030 has up to 4 bytes in a single\ncharacter.  It might still not be completely \"clean\" though because\nGB18030 has a \"double double\" nature to it.  Just like assuming 16-bit\ncharacters period mostly works with surrogate pairs even if you didn't\ncode full UTF-16 support, DBCS code will see a 4-byte GB18030 character\nas two double byte characters.  So it gets the len (in characters)\nwrong, and might still break up what is supposed to be a single\ncharacter.  So it really needs some improvement from the historical\nDBCS-only code to work properly.  \n\nAnyway, if UTF-8 really doesn't work with MBCS functions acceptably\nwell, and the goal is to allow passage of all characters through the\nprogram, then set the program to use Chinese.  GB18030 is =fully=\nsupported and is just another (albeit strange) encoding for Unicode.\n\nAs for what\n\tfprintf (stderr, \"unable to open %s\", path);\nwill do, it will have no problem copying the contents of path to the\noutput stream no matter how it is encoded.  The result will be sent to\nstderr, which may be autotranslating the local code page to UTF-16 or\nUTF-8, but by default just feeds the stream of bytes to the console\nwindow's 8-bit API, which has its own code page setting.\n\nPersonally, I have printf'ed UTF-8 encoded text to standard output.  It\nlooks OK if the console is also set to UTF-8.\n\n--John\n(please excuse the footer; it's not my idea)\n\n\n\nTradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD) of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE, FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and subscription company, and TradeStation Europe Limited, a United Kingdom, FSA-authorized introducing brokerage firm. None of these companies provides trading or investment advice, recommendations or endorsements of any kind. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited.\n  If you received this in error, please contact the sender and delete the material from any computer.\n"},{"id":"106930","messageId":"37fcd2780903031302m5f98fe71u8bdb23f90a8df82a@mail.gmail.com","threadId":"18122","inReplyTo":"450196A1AAAE4B42A00A8B27A59278E709F077AC@EXCHANGE.trad.tradestation.com","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-03-03T21:02:40Z","receivedAt":"2009-03-03T21:02:40Z","isPatch":true,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Tue, Mar 3, 2009 at 11:39 PM, John Dlugosz <JDlugosz@tradestation.com> wrote:\n>\n> Now, the default when a program starts is to use the \"C\" locale.  The\n> locale argument to setlocale can take a form \".code_page\", so calling\n>\n>        setlocale (LC_CTYPE, \".65001\");\n>\n> should do the trick.  Assuming, that is, that you don't hit macros that\n> assume that characters are never multibyte.  So define the preprocessor\n> symbol _MBCS when you compile.\n\nIf Microsoft fixed the problem with UTF-8 support in C runtime, it is\na really good\nnews, because setlocale did not work not so long time ago:\n\nhttp://blogs.msdn.com/michkap/archive/2006/03/13/550191.aspx\n\nAs to Win32 API, it has always worked correctly with UTF-8... In fact, the\ndocumentation of GetOEMCP function goes as far as recommending\nto use UTF-8 or UTF-16: \"For the most consistent results, applications should\nuse Unicode, such as UTF-8 or UTF-16, instead of a specific code page.\n\nSo it would be great if Git supported UTF-8 on Windows (as an option), but it\nis not my itch right now....\n\nDmitry\n"},{"id":"106933","messageId":"450196A1AAAE4B42A00A8B27A59278E709F07826@EXCHANGE.trad.tradestation.com","threadId":"18122","inReplyTo":"37fcd2780903031302m5f98fe71u8bdb23f90a8df82a@mail.gmail.com","subject":"RE: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"John Dlugosz","fromEmail":"jdlugosz@tradestation.com","sentAt":"2009-03-03T21:56:44Z","receivedAt":"2009-03-03T21:56:44Z","isPatch":true,"sender":{"key":"jdlugosz@tradestation.com","avatar":null},"body":"===Re:===\nIf Microsoft fixed the problem with UTF-8 support in C runtime, it is\na really good\nnews, because setlocale did not work not so long time ago:\n===end===\n\nThey totally replaced it with one written by P.J.Plauger.  I'm not sure\nwhen, but I would guess around VC++7.1, which was a \"sea change\" and\nfelt more like a different brand than a simple update.  That's when\ntemplates started following the standard.\n\nRe:\nhttp://blogs.msdn.com/michkap/archive/2006/03/13/550191.aspx\n\nInteresting.  So it sort-of worked, as per my overlong muse as I looked\nat the source code, but they started explicitly preventing it because it\ndoesn't always work for everything.\n\n    //  verify codepage validity\n    if (!iCodePage || iCodePage == CP_UTF7 || iCodePage == CP_UTF8 ||\n        !IsValidCodePage((WORD)iCodePage))\n        return FALSE;\n\n\n===Re:===\nAs to Win32 API, it has always worked correctly with UTF-8... In fact,\nthe\ndocumentation of GetOEMCP function goes as far as recommending\nto use UTF-8 or UTF-16: \"For the most consistent results, applications\nshould\nuse Unicode, such as UTF-8 or UTF-16, instead of a specific code page.\n===end===\n\nI remember a time when it did not.  I don't recall if it was NT (as\nopposed to consumer windows) or some version of NT beyond 3.5 (starting\nin 4?) that it became available.  But I had to supply code with the\nprogram because it could not count on it.\n\n===Re:===\nSo it would be great if Git supported UTF-8 on Windows (as an option),\nbut it\nis not my itch right now....\n===end===\n\nsomeone else mentioned \"most people use ASCII file names\", and I would\ntake that to be true only if \"most people\" == \"developers\".  If you look\nat my wife's \"explorer\" view, it's all Chinese.  Files are downloaded\nwith Asian file names.  Most people =in= China are used to seemless\nsupport within Windows.  It's only with Chinese MUI on English Windows\nthat the \"ANSI\" stuff doesn't match and programs that use 8-bit API\ncalls suddenly croak as they see \"?????\" for input.\n\n--John\n\n\nTradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD) of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE, FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and subscription company, and TradeStation Europe Limited, a United Kingdom, FSA-authorized introducing brokerage firm. None of these companies provides trading or investment advice, recommendations or endorsements of any kind. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited.\n  If you received this in error, please contact the sender and delete the material from any computer.\n"}]}