{"thread":{"id":"21776","subject":"non-US-ASCII file names (e.g. Hiragana) on Windows","startedAt":"2009-11-28T18:15:48Z","lastAt":"2010-10-30T09:52:54Z","messageCount":27,"participants":["Thomas Singer","Johannes Sixt","Maximilien Noal","Reece Dunn","Erik Faye-Lund","Shawn O. Pearce","Jakub Narebski","Robin Rosenberg","Shawn Pearce","brad12","demerphq"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"128631","messageId":"4B1168D4.5010902@syntevo.com","threadId":"21776","inReplyTo":null,"subject":"non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-11-28T18:15:48Z","receivedAt":"2009-11-28T18:15:48Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"I've created a file with unicode characters in its name (using Java):\n\n new File(dir, \"\\u3041\\u3042\\u3043\\u3044\").createNewFile();\n\nThe file name is stored correctly on disk, because if invoking a\n\n dir.list()\n\nthe name is listed correctly.\n\nWhen opening this directory in the Windows Explorer (German Windows XP SP3),\nit shows 4 boxes - which most likely is a problem of the font not supporting\nthese characters.\n\nWhen launching 'git status' from the git shell (msys 1.6.5.1.1367.gcd48 from\n7zip-bundle) it only shows me 4 question marks. I would have expected to see\nthe non-displayable characters escaped like it did with the umlauts on OS X.\n\nEven adding fails:\n\n$ git add .\nfatal: unable to stat '????': No such file or directory\n\nWhat should I do to make Git recognize these characters?\n\n-- \nThanks in advance,\nTom\n"},{"id":"128648","messageId":"200911282100.23000.j6t@kdbg.org","threadId":"21776","inReplyTo":"4B1168D4.5010902@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2009-11-28T20:00:22Z","receivedAt":"2009-11-28T20:00:22Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Samstag, 28. November 2009, Thomas Singer wrote:\n> I've created a file with unicode characters in its name (using Java):\n>\n>  new File(dir, \"\\u3041\\u3042\\u3043\\u3044\").createNewFile();\n>...\n> $ git add .\n> fatal: unable to stat '????': No such file or directory\n>\n> What should I do to make Git recognize these characters?\n\nYou cannot on a German Windows.\n\nYou can switch your Windows to Japanese (not the UI, just the codepage \naka \"locale\"; yes, that's possible, I have such a setup), but even then the \ncharacters of the file name will be recorded in Shift-JIS encoding, not UTF-8 \nor Unicode. When you later switch back to German, these bytes will be \ninterpreted as cp850 or cp1252 text and displayed accordingly.\n\n-- Hannes\n"},{"id":"128650","messageId":"4B11AD43.3070307@gmail.com","threadId":"21776","inReplyTo":"4B1168D4.5010902@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Maximilien Noal","fromEmail":"noal.maximilien@gmail.com","sentAt":"2009-11-28T23:07:47Z","receivedAt":"2009-11-28T23:07:47Z","isPatch":false,"sender":{"key":"noal.maximilien@gmail.com","avatar":null},"body":"Thomas Singer a écrit :\n> I've created a file with unicode characters in its name (using Java):\n> \n>  new File(dir, \"\\u3041\\u3042\\u3043\\u3044\").createNewFile();\n> \n> The file name is stored correctly on disk, because if invoking a\n> \n>  dir.list()\n> \n> the name is listed correctly.\n> \n> When opening this directory in the Windows Explorer (German Windows XP SP3),\n> it shows 4 boxes - which most likely is a problem of the font not supporting\n> these characters.\n> \n> When launching 'git status' from the git shell (msys 1.6.5.1.1367.gcd48 from\n> 7zip-bundle) it only shows me 4 question marks. I would have expected to see\n> the non-displayable characters escaped like it did with the umlauts on OS X.\n> \n> Even adding fails:\n> \n> $ git add .\n> fatal: unable to stat '????': No such file or directory\n> \n> What should I do to make Git recognize these characters?\n> \nHi\n\nAbout the 'boxes' :\n\nThe thing is, Windows' files for Asian languages are _not_ installed by \ndefault.\n\nThey can be installed (even while installing Windows), by checking the \ntwo checkboxes under the \"Supplemtal languages support\" groupbox in the \n\"Languages\" tab of the \"Regional and language options\" control panel. \n*re-take some breath ;-) *\n\nIt will remove the \"boxes\" in Explorer and display nice Asian characters.\n\nBut that will only fix Windows' files' names display, surely not git \n(unless I'm mistaken).\n"},{"id":"128651","messageId":"3f4fd2640911281537m3a3d7f5dl7836a613b48128f2@mail.gmail.com","threadId":"21776","inReplyTo":"4B1168D4.5010902@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Reece Dunn","fromEmail":"msclrhd@googlemail.com","sentAt":"2009-11-28T23:37:40Z","receivedAt":"2009-11-28T23:37:40Z","isPatch":false,"sender":{"key":"msclrhd@googlemail.com","avatar":null},"body":"2009/11/28 Thomas Singer <thomas.singer@syntevo.com>:\n>\n> When launching 'git status' from the git shell (msys 1.6.5.1.1367.gcd48 from\n> 7zip-bundle) it only shows me 4 question marks. I would have expected to see\n> the non-displayable characters escaped like it did with the umlauts on OS X.\n>\n> Even adding fails:\n>\n> $ git add .\n> fatal: unable to stat '????': No such file or directory\n>\n> What should I do to make Git recognize these characters?\n\nThis is a bug in git's character encoding/conversion logic. It looks\nlike git is taking the source string and converting it to ascii to be\ndisplayed on the console output (e.g. by using the WideCharToMultiByte\nconversion API) -- these APIs will use a '?' character for characters\nthat it cannot map to the target character encoding (like the Hiragana\ncharacters that you are using).\n\nSetConsoleOutputCP can be used to change the console output codepage\n[http://msdn.microsoft.com/en-us/library/ms686036%28VS.85%29.aspx] and\nSetConsoleCP is the equivalent for input\n[http://msdn.microsoft.com/en-us/library/ms686013%28VS.85%29.aspx].\ne.g.\n\n    SetConsoleCP(CP_UTF8);\n    SetConsoleOutputCP(CP_UTF8);\n\nshould make the console process UTF-8 characters, so git shouldn't\nneed to do any character conversions on Windows when reading/writing\nit's data.\n\nNOTE: I have not tested this, just noting what I have found via Google.\n\n- Reece\n"},{"id":"128674","messageId":"4B123C80.30607@syntevo.com","threadId":"21776","inReplyTo":"4B11AD43.3070307@gmail.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-11-29T09:18:56Z","receivedAt":"2009-11-29T09:18:56Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Maximilien Noal wrote:\n> About the 'boxes' :\n> \n> The thing is, Windows' files for Asian languages are _not_ installed by\n> default.\n> \n> They can be installed (even while installing Windows), by checking the\n> two checkboxes under the \"Supplemtal languages support\" groupbox in the\n> \"Languages\" tab of the \"Regional and language options\" control panel.\n> *re-take some breath ;-) *\n> \n> It will remove the \"boxes\" in Explorer and display nice Asian characters.\n\nThanks, now the characters are showing up fine in the Explorer.\n\nReece Dunn wrote:\n> This is a bug in git's character encoding/conversion logic. It looks\n> like git is taking the source string and converting it to ascii to be\n> displayed on the console output (e.g. by using the WideCharToMultiByte\n> conversion API) -- these APIs will use a '?' character for characters\n> that it cannot map to the target character encoding (like the Hiragana\n> characters that you are using).\n\nI have a screenshot from a SmartGit user where 1) the console can show the\nfar-east-characters and 2) Git *can* show the characters escaped. Are there\ntwo versions of Git available or does Gits behaviour depends somehow on the\nsystem locale?\n\n-- \nTom\n"},{"id":"128841","messageId":"4B14CA79.6040408@syntevo.com","threadId":"21776","inReplyTo":"4B123C80.30607@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-12-01T07:49:13Z","receivedAt":"2009-12-01T07:49:13Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Thomas Singer wrote:\n> Reece Dunn wrote:\n>> This is a bug in git's character encoding/conversion logic. It looks\n>> like git is taking the source string and converting it to ascii to be\n>> displayed on the console output (e.g. by using the WideCharToMultiByte\n>> conversion API) -- these APIs will use a '?' character for characters\n>> that it cannot map to the target character encoding (like the Hiragana\n>> characters that you are using).\n> \n> I have a screenshot from a SmartGit user where 1) the console can show the\n> far-east-characters and 2) Git *can* show the characters escaped. Are there\n> two versions of Git available or does Gits behaviour depends somehow on the\n> system locale?\n\nDoes no Git expert know what to do to get it working?\n\n-- \nTom\n"},{"id":"128845","messageId":"4B14D381.3010706@viscovery.net","threadId":"21776","inReplyTo":"4B14CA79.6040408@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-12-01T08:27:45Z","receivedAt":"2009-12-01T08:27:45Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Thomas Singer schrieb:\n> Thomas Singer wrote:\n>> Reece Dunn wrote:\n>>> This is a bug in git's character encoding/conversion logic. It looks\n>>> like git is taking the source string and converting it to ascii to be\n>>> displayed on the console output (e.g. by using the WideCharToMultiByte\n>>> conversion API) -- these APIs will use a '?' character for characters\n>>> that it cannot map to the target character encoding (like the Hiragana\n>>> characters that you are using).\n>> I have a screenshot from a SmartGit user where 1) the console can show the\n>> far-east-characters and 2) Git *can* show the characters escaped. Are there\n>> two versions of Git available or does Gits behaviour depends somehow on the\n>> system locale?\n> \n> Does no Git expert know what to do to get it working?\n\nhttp://article.gmane.org/gmane.comp.version-control.git/133980 [*]\n\nThe possible reason why some one else is seeing correct glyphs with\nSmartGit is because it is a Unicode application and the Windows box has\nsuitable fonts installed and the console is configured with a suitable\nfont as well.\n\n-- Hannes\n\n[*] I had a botch email infrastructure when I sent this message, and the\ncopy intended for you went to the waste bin, but I thought I had re-sent\nto you in a private mail.\n"},{"id":"128848","messageId":"4B14DA1A.4060505@syntevo.com","threadId":"21776","inReplyTo":"4B14D381.3010706@viscovery.net","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-12-01T08:55:54Z","receivedAt":"2009-12-01T08:55:54Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Johannes Sixt wrote:\n> Thomas Singer schrieb:\n>> Thomas Singer wrote:\n>>> Reece Dunn wrote:\n>>>> This is a bug in git's character encoding/conversion logic. It looks\n>>>> like git is taking the source string and converting it to ascii to be\n>>>> displayed on the console output (e.g. by using the WideCharToMultiByte\n>>>> conversion API) -- these APIs will use a '?' character for characters\n>>>> that it cannot map to the target character encoding (like the Hiragana\n>>>> characters that you are using).\n>>> I have a screenshot from a SmartGit user where 1) the console can show the\n>>> far-east-characters and 2) Git *can* show the characters escaped. Are there\n>>> two versions of Git available or does Gits behaviour depends somehow on the\n>>> system locale?\n>> Does no Git expert know what to do to get it working?\n> \n> http://article.gmane.org/gmane.comp.version-control.git/133980 [*]\n> \n> The possible reason why some one else is seeing correct glyphs with\n> SmartGit is because it is a Unicode application and the Windows box has\n> suitable fonts installed and the console is configured with a suitable\n> font as well.\n\nI wasn't talking about SmartGit, but msysgit on the Windows console. Sorry,\nif that wasn't clear.\n\nIs it a German Windows limitation, that far-east characters are not\nsupported on it (but work fine on a Japanese Windows), are there different\n(mysys)Git versions available or is this a configuration issue?\n\n-- \nTom\n"},{"id":"128849","messageId":"4B14DA78.70906@syntevo.com","threadId":"21776","inReplyTo":"200911282100.23000.j6t@kdbg.org","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-12-01T08:57:28Z","receivedAt":"2009-12-01T08:57:28Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Johannes Sixt wrote:\n> On Samstag, 28. November 2009, Thomas Singer wrote:\n>> I've created a file with unicode characters in its name (using Java):\n>>\n>>  new File(dir, \"\\u3041\\u3042\\u3043\\u3044\").createNewFile();\n>> ...\n>> $ git add .\n>> fatal: unable to stat '????': No such file or directory\n>>\n>> What should I do to make Git recognize these characters?\n> \n> You cannot on a German Windows.\n> \n> You can switch your Windows to Japanese (not the UI, just the codepage \n> aka \"locale\"; yes, that's possible, I have such a setup), but even then the \n> characters of the file name will be recorded in Shift-JIS encoding, not UTF-8 \n> or Unicode. When you later switch back to German, these bytes will be \n> interpreted as cp850 or cp1252 text and displayed accordingly.\n\nWho is interpreting the file names? Windows or Git or Java?\n\n-- \nTom\n"},{"id":"128850","messageId":"4B14DC20.6040808@syntevo.com","threadId":"21776","inReplyTo":"4B14DA78.70906@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-12-01T09:04:32Z","receivedAt":"2009-12-01T09:04:32Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Thomas Singer wrote:\n> Johannes Sixt wrote:\n>> You can switch your Windows to Japanese (not the UI, just the codepage \n>> aka \"locale\"; yes, that's possible, I have such a setup), but even then the \n>> characters of the file name will be recorded in Shift-JIS encoding, not UTF-8 \n>> or Unicode. When you later switch back to German, these bytes will be \n>> interpreted as cp850 or cp1252 text and displayed accordingly.\n> \n> Who is interpreting the file names? Windows or Git or Java?\n\nTo be more precise: Who is interpreting the bytes in the file names as\ncharacters? Windows, Git or Java?\n\n-- \nTom\n"},{"id":"128853","messageId":"40aa078e0912010112u4205452as3627ba019544c5fe@mail.gmail.com","threadId":"21776","inReplyTo":"4B123C80.30607@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@googlemail.com","sentAt":"2009-12-01T09:12:36Z","receivedAt":"2009-12-01T09:12:36Z","isPatch":false,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Sun, Nov 29, 2009 at 10:18 AM, Thomas Singer\n<thomas.singer@syntevo.com> wrote:\n> Maximilien Noal wrote:\n>> About the 'boxes' :\n>>\n>> The thing is, Windows' files for Asian languages are _not_ installed by\n>> default.\n>>\n>> They can be installed (even while installing Windows), by checking the\n>> two checkboxes under the \"Supplemtal languages support\" groupbox in the\n>> \"Languages\" tab of the \"Regional and language options\" control panel.\n>> *re-take some breath ;-) *\n>>\n>> It will remove the \"boxes\" in Explorer and display nice Asian characters.\n>\n> Thanks, now the characters are showing up fine in the Explorer.\n>\n> Reece Dunn wrote:\n>> This is a bug in git's character encoding/conversion logic. It looks\n>> like git is taking the source string and converting it to ascii to be\n>> displayed on the console output (e.g. by using the WideCharToMultiByte\n>> conversion API) -- these APIs will use a '?' character for characters\n>> that it cannot map to the target character encoding (like the Hiragana\n>> characters that you are using).\n>\n> I have a screenshot from a SmartGit user where 1) the console can show the\n> far-east-characters and 2) Git *can* show the characters escaped. Are there\n> two versions of Git available or does Gits behaviour depends somehow on the\n> system locale?\n\nDid you try to make sure your console window used a Unicode font on\nyour German Windows installation? Asian Windows installations might do\nthis by default, something at least neither English nor Norwegian\nWindows installations seems to do...\n\nYou can change the console window font through the properties-menu\nthat appears when you right click the title-bar.\n\n-- \nErik \"kusma\" Faye-Lund\n"},{"id":"128860","messageId":"4B14E934.9090304@viscovery.net","threadId":"21776","inReplyTo":"4B14DA1A.4060505@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-12-01T10:00:20Z","receivedAt":"2009-12-01T10:00:20Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Thomas Singer schrieb:\n> Is it a German Windows limitation, that far-east characters are not\n> supported on it (but work fine on a Japanese Windows), are there different\n> (mysys)Git versions available or is this a configuration issue?\n\nIt is a matter of configuration.\n\nSince 8 bits are not sufficient to support Japanese alphabet in addition\nto the German alphabet, programs that are not Unicode aware -- such as git\n-- have to make a decision which alphabet they support. The decision is\nmade by picking a \"codepage\".\n\nOn German Windows, you are in codepage 850 (in the console). The filenames\n (that actually are in Unicode) are converted to bytes according to\ncodepage 850 *before* git sees them. If your filenames contain Hiragana,\nthey are substituted by the \"unknown character\" marker because there is no\nplace for them in codepage 850.\n\nHowever, you can install Japanese language support on German Windows. Then\nyou can change your console to codepage 932:\n\n  chcp 932\n\nWhen you run git from *this* console, Hiragana in the filenames are\nconverted to cp932 before git sees them. The resulting byte sequence is\ndifferent from the one in cp850, but git will be able to see that the file\nexists and was modified, and you can 'git add' it.\n\nBut if you have files with umlauts, they will not be recognized anymore\nbecause umlauts have no place in cp932.\n\nIn neither case can you exchange the repository with Linux if you have\nyour locale set to UTF-8 on Linux, because neither byte sequence (umlauts\nfrom cp850 or Hiragana from cp932) are valid UTF-8 sequences, let alone\nresult in the expected glyphs.\n\nCorollary: Stick to ASCII file names.\n\nThere have been suggestions to switch the console to codepage 65001\n(UTF-8), but I have never heard of success reports. I'm not saying it does\nnot work, though.\n\n-- Hannes\n"},{"id":"128861","messageId":"4B14EB2E.9020906@viscovery.net","threadId":"21776","inReplyTo":"4B14DC20.6040808@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-12-01T10:08:46Z","receivedAt":"2009-12-01T10:08:46Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Thomas Singer schrieb:\n> To be more precise: Who is interpreting the bytes in the file names as\n> characters? Windows, Git or Java?\n\nIn the case of git: Windows does it, using the console's codepage to\nconvert between bytes and Unicode.\n\nI don't know about Java, but I guess that no conversion is necessary\nbecause Java is Unicode-aware.\n\n-- Hannes\n"},{"id":"128868","messageId":"4B150747.2030900@syntevo.com","threadId":"21776","inReplyTo":"4B14E934.9090304@viscovery.net","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-12-01T12:08:39Z","receivedAt":"2009-12-01T12:08:39Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Johannes Sixt wrote:\n> Thomas Singer schrieb:\n>> Is it a German Windows limitation, that far-east characters are not\n>> supported on it (but work fine on a Japanese Windows), are there different\n>> (mysys)Git versions available or is this a configuration issue?\n> \n> It is a matter of configuration.\n> \n> Since 8 bits are not sufficient to support Japanese alphabet in addition\n> to the German alphabet, programs that are not Unicode aware -- such as git\n> -- have to make a decision which alphabet they support. The decision is\n> made by picking a \"codepage\".\n> \n> On German Windows, you are in codepage 850 (in the console). The filenames\n>  (that actually are in Unicode) are converted to bytes according to\n> codepage 850 *before* git sees them. If your filenames contain Hiragana,\n> they are substituted by the \"unknown character\" marker because there is no\n> place for them in codepage 850.\n> \n> However, you can install Japanese language support on German Windows. Then\n> you can change your console to codepage 932:\n> \n>   chcp 932\n> \n> When you run git from *this* console, Hiragana in the filenames are\n> converted to cp932 before git sees them. The resulting byte sequence is\n> different from the one in cp850, but git will be able to see that the file\n> exists and was modified, and you can 'git add' it.\n> \n> But if you have files with umlauts, they will not be recognized anymore\n> because umlauts have no place in cp932.\n> \n> In neither case can you exchange the repository with Linux if you have\n> your locale set to UTF-8 on Linux, because neither byte sequence (umlauts\n> from cp850 or Hiragana from cp932) are valid UTF-8 sequences, let alone\n> result in the expected glyphs.\n> \n> Corollary: Stick to ASCII file names.\n> \n> There have been suggestions to switch the console to codepage 65001\n> (UTF-8), but I have never heard of success reports. I'm not saying it does\n> not work, though.\n\nThanks for the detailed explanation. I know the differences between bytes\nand characters and the needed *encoding* to convert from one to another, but\nI did not know how Git handles it. I'm quite surprised, that -- as I\nunderstand you -- msys-Git (or Git at all?) is not able to handle all\ncharacters (aka unicode) at the same time. I expected it would be better\nthan older tools, e.g. SVN.\n\nBTW, we are invoking the Git executable from Java. Is there automatically a\nconsole \"around\" Git? Should we invoke a shell-script (which sets the\nconsole's code page) instead of the Git executable directly?\n\n-- \nTom\n"},{"id":"128869","messageId":"4B1507E2.1050809@syntevo.com","threadId":"21776","inReplyTo":"40aa078e0912010112u4205452as3627ba019544c5fe@mail.gmail.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-12-01T12:11:14Z","receivedAt":"2009-12-01T12:11:14Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Erik Faye-Lund wrote:\n> Did you try to make sure your console window used a Unicode font on\n> your German Windows installation? Asian Windows installations might do\n> this by default, something at least neither English nor Norwegian\n> Windows installations seems to do...\n> \n> You can change the console window font through the properties-menu\n> that appears when you right click the title-bar.\n\nI've tried to change the console font (there is just one alternative), but\nwithout any change.\n\n-- \nTom\n"},{"id":"128870","messageId":"4B151782.8050309@viscovery.net","threadId":"21776","inReplyTo":"4B150747.2030900@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-12-01T13:17:54Z","receivedAt":"2009-12-01T13:17:54Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Thomas Singer schrieb:\n> I'm quite surprised, that -- as I\n> understand you -- msys-Git (or Git at all?) is not able to handle all\n> characters (aka unicode) at the same time. I expected it would be better\n> than older tools, e.g. SVN.\n\nThis has been discussed at length here and in the msysgit mailing list.\nGit expects that the file system returns file names with the same byte\nsequence that git used to create a file. On Windows, this works only as\nlong as you do not switch the codepage.\n\n> BTW, we are invoking the Git executable from Java. Is there automatically a\n> console \"around\" Git?\n\nI don't think so. In this case, the codepage that Java has set up will\napply. I guess that Java doesn't mess with the codepage at all, and then\non German Windows git would operate in cp1252.\n\n> Should we invoke a shell-script (which sets the\n> console's code page) instead of the Git executable directly?\n\nI don't think that is necessary.\n\n-- Hannes\n"},{"id":"128881","messageId":"4B15391C.5090302@syntevo.com","threadId":"21776","inReplyTo":"4B151782.8050309@viscovery.net","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-12-01T15:41:16Z","receivedAt":"2009-12-01T15:41:16Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Johannes Sixt wrote:\n> Thomas Singer schrieb:\n>> I'm quite surprised, that -- as I\n>> understand you -- msys-Git (or Git at all?) is not able to handle all\n>> characters (aka unicode) at the same time. I expected it would be better\n>> than older tools, e.g. SVN.\n> \n> This has been discussed at length here and in the msysgit mailing list.\n> Git expects that the file system returns file names with the same byte\n> sequence that git used to create a file. On Windows, this works only as\n> long as you do not switch the codepage.\n\nNow you confuse me: is this a problem of Windows, Git using a less capable\nWindows-API call or is there no unicode-capable API call to list file names\non Windows? I ask myself how Java does it in its internals, finally it\n(also) consists of a C-base, I guess.\n\n-- \nTom\n"},{"id":"128882","messageId":"40aa078e0912010750m4518f068s9c955042307bf9fe@mail.gmail.com","threadId":"21776","inReplyTo":"4B15391C.5090302@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@googlemail.com","sentAt":"2009-12-01T15:50:11Z","receivedAt":"2009-12-01T15:50:11Z","isPatch":false,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, Dec 1, 2009 at 4:41 PM, Thomas Singer <thomas.singer@syntevo.com> wrote:\n> Johannes Sixt wrote:\n>> Thomas Singer schrieb:\n>>> I'm quite surprised, that -- as I\n>>> understand you -- msys-Git (or Git at all?) is not able to handle all\n>>> characters (aka unicode) at the same time. I expected it would be better\n>>> than older tools, e.g. SVN.\n>>\n>> This has been discussed at length here and in the msysgit mailing list.\n>> Git expects that the file system returns file names with the same byte\n>> sequence that git used to create a file. On Windows, this works only as\n>> long as you do not switch the codepage.\n>\n> Now you confuse me: is this a problem of Windows, Git using a less capable\n> Windows-API call or is there no unicode-capable API call to list file names\n> on Windows? I ask myself how Java does it in its internals, finally it\n> (also) consists of a C-base, I guess.\n>\n\nGit uses the 8-bit file APIs, and Windows doesn't support setting\nUTF-8 as the locale. Some work have been done in msysGit to use\n_wopen() and friends instead, but AFAIK it's not completed. See the\nbranch called \"work/utf-filenames\" in\ngit://repo.or.cz/git/mingw/4msysgit.git if you are interested in\nhelping to complete it.\n\n-- \nErik \"kusma\" Faye-Lund\n"},{"id":"128891","messageId":"20091201162627.GE21299@spearce.org","threadId":"21776","inReplyTo":"4B14EB2E.9020906@viscovery.net","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-12-01T16:26:27Z","receivedAt":"2009-12-01T16:26:27Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Sixt <j.sixt@viscovery.net> wrote:\n> Thomas Singer schrieb:\n> > To be more precise: Who is interpreting the bytes in the file names as\n> > characters? Windows, Git or Java?\n> \n> In the case of git: Windows does it, using the console's codepage to\n> convert between bytes and Unicode.\n> \n> I don't know about Java, but I guess that no conversion is necessary\n> because Java is Unicode-aware.\n\nActually, conversion is necessary, and its something that is proving\nto be really painful within JGit.\n\nThe Java IO APIs use UTF-16 for file names.  However we are reading\na stream of unknown bytes from the index file and tree objects.\nThus JGit must convert a stream of bytes into UTF-16 just to get\nto the OS.\n\nThe JVM then turns around and converts from UTF-16 to some other\nencoding for the filesystem.\n\nOn Win32 I suspect the JVM uses the native UTF-16 file APIs, so\nthis translation is lossless.\n\nOn POSIX, I suspect the JVM uses $LANG or some other related\nenvironment variable to guess the user's preferred encoding, and\nthen converts from UTF-16 to bytes in that encoding.  And I have\nno idea how they handle normalization of composed code points.\n\nAll of these layers make for a *very* confusing situation for us\nwithin JGit:\n\n  git tree\n  +---------+\n  | bytes   | -+\n  +---------+   \\\n                 \\             +--------+            +---------+\n                  +-- JGit --> | UTF-16 | -- JVM --> | OS call |\n  .git/index     /             +--------+            +---------+\n  +---------+   /\n  | bytes   | -+\n  +---------+\n\nIts impossible for us to do what C git does, which is just use the\nbytes used by the OS call within the git datastructure.  Which of\ncourse also isn't always portable, e.g. the Mac OS X HFS+ mess.\n\n:-)\n\n-- \nShawn.\n"},{"id":"128892","messageId":"4B154560.1020806@syntevo.com","threadId":"21776","inReplyTo":"40aa078e0912010750m4518f068s9c955042307bf9fe@mail.gmail.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-12-01T16:33:36Z","receivedAt":"2009-12-01T16:33:36Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Erik Faye-Lund wrote:\n> Git uses the 8-bit file APIs, and Windows doesn't support setting\n> UTF-8 as the locale. Some work have been done in msysGit to use\n> _wopen() and friends instead, but AFAIK it's not completed. See the\n> branch called \"work/utf-filenames\" in\n> git://repo.or.cz/git/mingw/4msysgit.git if you are interested in\n> helping to complete it.\n\nThanks, now I understand.\n\n-- \nTom\n"},{"id":"128903","messageId":"m3k4x6na81.fsf@localhost.localdomain","threadId":"21776","inReplyTo":"4B150747.2030900@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-12-01T17:24:58Z","receivedAt":"2009-12-01T17:24:58Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Thomas Singer <thomas.singer@syntevo.com> writes:\n\n> Johannes Sixt wrote:\n>> Thomas Singer schrieb:\n>>>\n>>> Is it a German Windows limitation, that far-east characters are not\n>>> supported on it (but work fine on a Japanese Windows), are there different\n>>> (mysys)Git versions available or is this a configuration issue?\n>> \n>> It is a matter of configuration.\n>> \n>> Since 8 bits are not sufficient to support Japanese alphabet in addition\n>> to the German alphabet, programs that are not Unicode aware -- such as git\n>> -- have to make a decision which alphabet they support. The decision is\n>> made by picking a \"codepage\".\n>> \n>> On German Windows, you are in codepage 850 (in the console). The filenames\n>>  (that actually are in Unicode) are converted to bytes according to\n>> codepage 850 *before* git sees them. If your filenames contain Hiragana,\n>> they are substituted by the \"unknown character\" marker because there is no\n>> place for them in codepage 850.\n[...]\n\n>> Corollary: Stick to ASCII file names.\n>> \n>> There have been suggestions to switch the console to codepage 65001\n>> (UTF-8), but I have never heard of success reports. I'm not saying it does\n>> not work, though.\n> \n> Thanks for the detailed explanation. I know the differences between bytes\n> and characters and the needed *encoding* to convert from one to another, but\n> I did not know how Git handles it. I'm quite surprised, that -- as I\n> understand you -- msys-Git (or Git at all?) is not able to handle all\n> characters (aka unicode) at the same time. I expected it would be better\n> than older tools, e.g. SVN.\n\nThe problem is not with Git, as Git is (currently) agnostic with\nrespect to filename encoding; for Git filenames are opaque NUL ('\\0)\nterminated binary data.  There is some infrastructure to convert\nbetween filename encodings and other filename quirks (like\ncase-insensivity), though...\n\nThe problem is with MS Windows *console*, from which you invoke git\ncommands, and which does translation from filename encoding used by\nthe filesystem to encoding / codepage used by console.\n\n> BTW, we are invoking the Git executable from Java. Is there automatically a\n> console \"around\" Git? Should we invoke a shell-script (which sets the\n> console's code page) instead of the Git executable directly?\n\nIf you use Git from Java, why don't you just use JGit (www.jgit.org),\nwhich is Git implementation in Java?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"128915","messageId":"4B15668A.5050209@syntevo.com","threadId":"21776","inReplyTo":"m3k4x6na81.fsf@localhost.localdomain","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Thomas Singer","fromEmail":"thomas.singer@syntevo.com","sentAt":"2009-12-01T18:55:06Z","receivedAt":"2009-12-01T18:55:06Z","isPatch":false,"sender":{"key":"thomas.singer@syntevo.com","avatar":null},"body":"Jakub Narebski wrote:\n> If you use Git from Java, why don't you just use JGit (www.jgit.org),\n> which is Git implementation in Java?\n\nWe are using JGit for the read-only stuff and the Git command line\nexecutable for all writing commands. We very much appreciate Shawn O.\nPearce' (and the other JGit developers') effort, but Git is a fast moving\ntarget and (much) more complex than CVS or SVN, for which we use Java\nlibraries communicating with the corresponding server which adds another\nsanity layer to the repository making repository corruption less likely than\ndirect access.\n\n-- \nBest regards,\nThomas Singer\n=============\nsyntevo GmbH\nhttp://www.syntevo.com\nhttp://blog.syntevo.com\n"},{"id":"128928","messageId":"200912012311.14321.robin.rosenberg@dewire.com","threadId":"21776","inReplyTo":"20091201162627.GE21299@spearce.org","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-12-01T22:11:13Z","receivedAt":"2009-12-01T22:11:13Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"tisdag 01 december 2009 17:26:27 skrev du:\n> Johannes Sixt <j.sixt@viscovery.net> wrote:\n> > Thomas Singer schrieb:\n> > > To be more precise: Who is interpreting the bytes in the file names as\n> > > characters? Windows, Git or Java?\n> >\n> > In the case of git: Windows does it, using the console's codepage to\n> > convert between bytes and Unicode.\n> >\n> > I don't know about Java, but I guess that no conversion is necessary\n> > because Java is Unicode-aware.\n>\n> Actually, conversion is necessary, and its something that is proving\n> to be really painful within JGit.\n>\n> The Java IO APIs use UTF-16 for file names.  However we are reading\n> a stream of unknown bytes from the index file and tree objects.\n> Thus JGit must convert a stream of bytes into UTF-16 just to get\n> to the OS.\n>\n> The JVM then turns around and converts from UTF-16 to some other\n> encoding for the filesystem.\n>\n> On Win32 I suspect the JVM uses the native UTF-16 file APIs, so\n> this translation is lossless.\n>\n> On POSIX, I suspect the JVM uses $LANG or some other related\n> environment variable to guess the user's preferred encoding, and\n> then converts from UTF-16 to bytes in that encoding.  And I have\n> no idea how they handle normalization of composed code points.\n>\n> All of these layers make for a *very* confusing situation for us\n> within JGit:\n>\n>   git tree\n>   +---------+\n>\n>   | bytes   | -+\n>\n>   +---------+   \\\n>                  \\             +--------+            +---------+\n>                   +-- JGit --> | UTF-16 | -- JVM --> | OS call |\n>   .git/index     /             +--------+            +---------+\n>   +---------+   /\n>\n>   | bytes   | -+\n>\n>   +---------+\n>\n> Its impossible for us to do what C git does, which is just use the\n> bytes used by the OS call within the git datastructure.  Which of\n> course also isn't always portable, e.g. the Mac OS X HFS+ mess.\n\nWe can decode the index anyway we like but not file names coming from\nthe file system. On Windows, any sane name (it does allow invalid UTF-16 too, \nbut...) will be readable by JGit, but on a UTF-8 posix that may not be so, if \nthe filename is actually Latin.-1 encoded. In that case the Java runtime will \nreturn a decoded filename containing an \"invalid\" code point and any attempt to \naccess the file from java will fail. I can see some horribly expensive ways to \nwork around that but...\n\nAs for the more sane cases I have a compare routine that works on mixed \nencodings that may help to solve some of the problems. Ideally it would not\nonly be able to compare filenames with unknown encodings to handling case \nfolding and composing characters in one go too. I guess one could make it\nfall back to another encoding than Latin-1, but with lesser certainty, but\nit will not (for sure) work with any arbitrary set of encodings. You'll have \nto choose, so it's only a legacy workaround, as opposed to a solution. \n\n-- robin\n"},{"id":"129012","messageId":"ec874dac0912020822t58477e12y3e6d1ab37da4f7e5@mail.gmail.com","threadId":"21776","inReplyTo":"4B15668A.5050209@syntevo.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-12-02T16:22:58Z","receivedAt":"2009-12-02T16:22:58Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Tue, Dec 1, 2009 at 10:55 AM, Thomas Singer\n<thomas.singer@syntevo.com> wrote:\n>\n> Jakub Narebski wrote:\n> > If you use Git from Java, why don't you just use JGit (www.jgit.org),\n> > which is Git implementation in Java?\n>\n> We are using JGit for the read-only stuff and the Git command line\n> executable for all writing commands. We very much appreciate Shawn O.\n> Pearce' (and the other JGit developers') effort, but Git is a fast moving\n> target and (much) more complex than CVS or SVN, for which we use Java\n> libraries communicating with the corresponding server which adds another\n> sanity layer to the repository making repository corruption less likely than\n> direct access.\n\nUhm.  I'm sorry, but this is just plain FUD.\n\nJGit implements the current on disk formats and network protocols\ncompletely[1].  In the area of disk formats and network protocols, Git\n*IS NOT* a fast moving target.  This area of Git hasn't changed much\nsince pack files were first introduced.  As a community, we have been\nvery careful to avoid changes which break compatibility with older\nimplementations.\n\nGit is also a lot less complex than CVS or SVN.  Its data model is\nsimpler on disk.  Its network protocol is *vastly* more simple than\nSVN's WebDAV protocol.  And unlike SVN we haven't had to break the\nnetwork protocol on every 1.x release we make.\n\n\n[1]  Actually, JGit lacks --depth support for shallow clones, but\notherwise is complete.\n"},{"id":"154823","messageId":"1288411330615-5688741.post@n2.nabble.com","threadId":"21776","inReplyTo":"4B154560.1020806@syntevo.com","subject":"non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"brad12","fromEmail":"brad.john75@gmail.com","sentAt":"2010-10-30T04:02:10Z","receivedAt":"2010-10-30T04:02:10Z","isPatch":false,"sender":{"key":"brad.john75@gmail.com","avatar":null},"body":"\nactually I am also working Japanese language site , I have same problem , I\nhave also some japanese sentences which is to be written in japanese , but\nthey did not appear on browser....so what should I do\n\n-----\nhttp://www.learnjapanesefree.com/ Learning japanese language  | \nhttp://www.learnjapanesefree.com/japanese-hiragana.html Hiragana \n-- \nView this message in context: http://git.661346.n2.nabble.com/non-US-ASCII-file-names-e-g-Hiragana-on-Windows-tp4080246p5688741.html\nSent from the git mailing list archive at Nabble.com.\n"},{"id":"154844","messageId":"m3fwvogiqj.fsf@localhost.localdomain","threadId":"21776","inReplyTo":"1288411330615-5688741.post@n2.nabble.com","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-30T08:58:50Z","receivedAt":"2010-10-30T08:58:50Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"brad12 <brad.john75@gmail.com> writes:\n\n> actually I am also working Japanese language site , I have same problem , I\n> have also some japanese sentences which is to be written in japanese , but\n> they did not appear on browser....so what should I do\n\nWhy are you sending this question to *this* mailing list (this\nnewsgroup)?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"154847","messageId":"AANLkTi=EPN+NrcTFP_wjmE_-VQk=bEo2UdMJOJ1uVNwB@mail.gmail.com","threadId":"21776","inReplyTo":"m3k4x6na81.fsf@localhost.localdomain","subject":"Re: non-US-ASCII file names (e.g. Hiragana) on Windows","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2010-10-30T09:52:54Z","receivedAt":"2010-10-30T09:52:54Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"On 1 December 2009 18:24, Jakub Narebski <jnareb@gmail.com> wrote:\n> Thomas Singer <thomas.singer@syntevo.com> writes:\n>\n>> Johannes Sixt wrote:\n>>> Thomas Singer schrieb:\n>>>>\n>>>> Is it a German Windows limitation, that far-east characters are not\n>>>> supported on it (but work fine on a Japanese Windows), are there different\n>>>> (mysys)Git versions available or is this a configuration issue?\n>>>\n>>> It is a matter of configuration.\n>>>\n>>> Since 8 bits are not sufficient to support Japanese alphabet in addition\n>>> to the German alphabet, programs that are not Unicode aware -- such as git\n>>> -- have to make a decision which alphabet they support. The decision is\n>>> made by picking a \"codepage\".\n>>>\n>>> On German Windows, you are in codepage 850 (in the console). The filenames\n>>>  (that actually are in Unicode) are converted to bytes according to\n>>> codepage 850 *before* git sees them. If your filenames contain Hiragana,\n>>> they are substituted by the \"unknown character\" marker because there is no\n>>> place for them in codepage 850.\n> [...]\n>\n>>> Corollary: Stick to ASCII file names.\n>>>\n>>> There have been suggestions to switch the console to codepage 65001\n>>> (UTF-8), but I have never heard of success reports. I'm not saying it does\n>>> not work, though.\n>>\n>> Thanks for the detailed explanation. I know the differences between bytes\n>> and characters and the needed *encoding* to convert from one to another, but\n>> I did not know how Git handles it. I'm quite surprised, that -- as I\n>> understand you -- msys-Git (or Git at all?) is not able to handle all\n>> characters (aka unicode) at the same time. I expected it would be better\n>> than older tools, e.g. SVN.\n>\n> The problem is not with Git, as Git is (currently) agnostic with\n> respect to filename encoding; for Git filenames are opaque NUL ('\\0)\n> terminated binary data.  There is some infrastructure to convert\n> between filename encodings and other filename quirks (like\n> case-insensivity), though...\n\n\"You can use whatever encoding you want. So long as it looks like a\nstandard UNIX filename.\"\n\n\n\n\n\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"}]}