{"thread":{"id":"18115","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","startedAt":"2009-03-03T18:25:14Z","lastAt":"2009-03-04T19:34:33Z","messageCount":3,"participants":["John Dlugosz","Peter Krefting"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"106905","messageId":"450196A1AAAE4B42A00A8B27A59278E709F076D2@EXCHANGE.trad.tradestation.com","threadId":"18115","inReplyTo":null,"subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"John Dlugosz","fromEmail":"jdlugosz@tradestation.com","sentAt":"2009-03-03T18:25:14Z","receivedAt":"2009-03-03T18:25:14Z","isPatch":true,"sender":{"key":"jdlugosz@tradestation.com","avatar":null},"body":"===Re:===\nThe other way would be to keep the char* APIs but convert to the Windows\n\nlocale encoding (\"ANSI codepage\"), but that will break horribly as not\nall \nfile names that can be used on a file system can be represented as such.\n===end===\n\nActually, UTF-8 is a valid code page on Windows.  The code page ID is\n65001.  So, if you set the process code page to that, =and= set the file\nsystem API's code page to follow rather than using the OEM code page\n(the default), it should work just fine.\n\nAlso, there is a national code page that =will= represent all file names\non the systems and is supported:  That is the Chinese GB18030, code page\n54936.  That has every character that Unicode does, just encoded\ndifferently to be forward compatible with GBK.  That is fully supported\nby windows, as it is required by law to sell in Chinese markets.\n\nLet me know if I can be of help.  I know character set stuff and Win32\nfairly well.\n\n--John\n\n\n\nTradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD) of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE, FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and subscription company, and TradeStation Europe Limited, a United Kingdom, FSA-authorized introducing brokerage firm. None of these companies provides trading or investment advice, recommendations or endorsements of any kind. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited.\n  If you received this in error, please contact the sender and delete the material from any computer.\n"},{"id":"106967","messageId":"alpine.DEB.2.00.0903041151440.8926@perkele.intern.softwolves.pp.se","threadId":"18115","inReplyTo":"450196A1AAAE4B42A00A8B27A59278E709F076D2@EXCHANGE.trad.tradestation.com","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-04T10:53:14Z","receivedAt":"2009-03-04T10:53:14Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"John Dlugosz:\n\n> Actually, UTF-8 is a valid code page on Windows.\n\nYes, but I am unsure whether it can be set as a thread locale for the sake \nof file APIs.\n\n> Also, there is a national code page that =will= represent all file names \n> on the systems and is supported: That is the Chinese GB18030, code page \n> 54936.\n\nYeah, but unfortunately it is explicitly documented that it is only \nsupported in MultiByteToWideChar, WideCharToMultiByte and some text painting \nAPIs in Windows, i.e the stdio functions and others may break horribly.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"107001","messageId":"450196A1AAAE4B42A00A8B27A59278E709F07A1C@EXCHANGE.trad.tradestation.com","threadId":"18115","inReplyTo":"alpine.DEB.2.00.0903041151440.8926@perkele.intern.softwolves.pp.se","subject":"RE: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"John Dlugosz","fromEmail":"jdlugosz@tradestation.com","sentAt":"2009-03-04T19:34:33Z","receivedAt":"2009-03-04T19:34:33Z","isPatch":true,"sender":{"key":"jdlugosz@tradestation.com","avatar":null},"body":"===Re:===\nYes, but I am unsure whether it [UTF-8] can be set as a thread locale\nfor the sake \nof file APIs.\n===end===\n\nWhy wouldn't it?  If the ANSI forms simply allocate buffers and call\nWideCharToMultiByte and MultiByteToWideChar, it should work with\nanything those functions handles.  My only concern would be with buffer\nlength when converting to MultiByte, if it assumes a limit based on 2\nbytes max per character.  But, it works with GB18030, which can have\n4-byte characters.\n\nIt's certainly easy enough to try.\n\n===Re:===\nYeah, but unfortunately it [GB18030] is explicitly documented that it is\nonly \nsupported in MultiByteToWideChar, WideCharToMultiByte and some text\npainting \nAPIs in Windows, i.e the stdio functions and others may break horribly.\n===end===\n\nCode that works with the other multi-byte \"ANSI\" character sets, and GBK\nin particular, will handle GB18030 \"reasonably well\" with no changes.\nFor example, printf (\"xxx%sxxx\", name), where each 'x' may actually be\nany character, will work without problems -- it won't mis-identify the %\nin the middle of a 4-byte character.  But printf (\"%5s\",name) will count\nsome of the characters in 'name' as two, and print less than 5 of them;\nor worse yet, break a character in half.\n\nI can't think of anything that breaks horribly.  Only situations that\ninvolve counting them will have issues.\n\nAs empirical evidence, lots of Windows software works fine in China.\nYou need full GB18030 support to read a newspaper on the web, because\nthe 4-byte characters are mostly obscure and regional words, but also\nproper nouns including the names of some prominent people (Prime\nMinister or something like that; I don't remember exactly).  But mostly\nyou don't encounter them and chug along with GBK and the occasional '?'\nwhere some character did not work.\n\n--John\n\nTradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD) of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE, FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and subscription company, and TradeStation Europe Limited, a United Kingdom, FSA-authorized introducing brokerage firm. None of these companies provides trading or investment advice, recommendations or endorsements of any kind. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited.\n  If you received this in error, please contact the sender and delete the material from any computer.\n"}]}