git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: non-US-ASCII file names (e.g. Hiragana) on Windows

From
TSThomas Singer <thomas.singer@syntevo.com>
Date
Dec 1, 2009, 12:08 UTC
Message-ID
<4B150747.2030900@syntevo.com>
In-Reply-To
<4B14E934.9090304@viscovery.net>
Johannes Sixt wrote:
Show 41 quoted lines
> Thomas Singer schrieb:
>> Is it a German Windows limitation, that far-east characters are not
>> supported on it (but work fine on a Japanese Windows), are there different
>> (mysys)Git versions available or is this a configuration issue?
> 
> It is a matter of configuration.
> 
> Since 8 bits are not sufficient to support Japanese alphabet in addition
> to the German alphabet, programs that are not Unicode aware -- such as git
> -- have to make a decision which alphabet they support. The decision is
> made by picking a "codepage".
> 
> On German Windows, you are in codepage 850 (in the console). The filenames
>  (that actually are in Unicode) are converted to bytes according to
> codepage 850 *before* git sees them. If your filenames contain Hiragana,
> they are substituted by the "unknown character" marker because there is no
> place for them in codepage 850.
> 
> However, you can install Japanese language support on German Windows. Then
> you can change your console to codepage 932:
> 
>   chcp 932
> 
> When you run git from *this* console, Hiragana in the filenames are
> converted to cp932 before git sees them. The resulting byte sequence is
> different from the one in cp850, but git will be able to see that the file
> exists and was modified, and you can 'git add' it.
> 
> But if you have files with umlauts, they will not be recognized anymore
> because umlauts have no place in cp932.
> 
> In neither case can you exchange the repository with Linux if you have
> your locale set to UTF-8 on Linux, because neither byte sequence (umlauts
> from cp850 or Hiragana from cp932) are valid UTF-8 sequences, let alone
> result in the expected glyphs.
> 
> Corollary: Stick to ASCII file names.
> 
> There have been suggestions to switch the console to codepage 65001
> (UTF-8), but I have never heard of success reports. I'm not saying it does
> not work, though.

Thanks for the detailed explanation. I know the differences between bytes and characters and the needed *encoding* to convert from one to another, but I did not know how Git handles it. I'm quite surprised, that -- as I understand you -- msys-Git (or Git at all?) is not able to handle all characters (aka unicode) at the same time. I expected it would be better than older tools, e.g. SVN.

BTW, we are invoking the Git executable from Java. Is there automatically a console "around" Git? Should we invoke a shell-script (which sets the console's code page) instead of the Git executable directly?

-- 
Tom
Previous: Johannes SixtNext: Johannes Sixt
Message 14 of 27 in “non-US-ASCII file names (e.g. Hiragana) on Windows”
  1. Thomas SingerNov 28, 2009
  2. Johannes SixtNov 28, 2009
  3. Thomas SingerDec 1, 2009
  4. Thomas SingerDec 1, 2009
  5. Johannes SixtDec 1, 2009
  6. Shawn O. PearceDec 1, 2009
  7. Robin RosenbergDec 1, 2009
  8. Maximilien NoalNov 28, 2009
  9. Thomas SingerNov 29, 2009
  10. Thomas SingerDec 1, 2009
  11. Johannes SixtDec 1, 2009
  12. Thomas SingerDec 1, 2009
  13. Johannes SixtDec 1, 2009
  14. Thomas SingerDec 1, 2009
  15. Johannes SixtDec 1, 2009
  16. Thomas SingerDec 1, 2009
  17. Erik Faye-LundDec 1, 2009
  18. Thomas SingerDec 1, 2009
  19. brad12Oct 30, 2010
  20. Jakub NarebskiOct 30, 2010
  21. Jakub NarebskiDec 1, 2009
  22. Thomas SingerDec 1, 2009
  23. Shawn PearceDec 2, 2009
  24. demerphqOct 30, 2010
  25. Erik Faye-LundDec 1, 2009
  26. Thomas SingerDec 1, 2009
  27. Reece DunnNov 28, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.