git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.

From
Robin Rosenberg <robin.rosenberg.lists@dewire.com>
Date
Mar 2, 2009, 19:58 UTC
Message-ID
<200903022058.33392.robin.rosenberg.lists@dewire.com>
In-Reply-To
<alpine.DEB.2.00.0903021404000.17877@perkele.intern.softwolves.pp.se>
Show 8 quoted lines
> Johannes Sixt:
> 
> > Can you set the local codepage per program? (I don't know.)
> 
> The locale is set per thread, and gets reset when the program exits. So 
> setting the codepage to UTF-8 before outputting should work. That should 
> also work for displaying the log to the terminal if you have UTF-8 log 
> messages.

Messing with locale is probably going to break subtly. An explicit approach is better, respecting the user's locale when necessary.

Show 14 quoted lines
> Converting it to wchar_t and using wprintf and similar should be safer, 
> though (and I have no idea what happens if you try to pipe the output to 
> something else).
> 
> > - What encoding should 'ls-files' use for its output? Certainly not always 
> > UTF-8: stdout should use the local code page so that the file names are 
> > interpreted correctly by the terminal window (it expects the local code 
> > page).
> 
> That is exactly why trying to mix "protocol" data ("plumbing" in Git's case) 
> and user output will always come back and bite you, one way or another. I 
> haven't really the faintest how pipes work with Unicode on Windows. 
> Somewhere along the line there will probably be some conversions, which 
> would cause interesting issues.

Pipes are just bytes so you have to know what you're piping by convention or protocol. You can ask for the console output page, which may be set to a multibyte locale or unicode and maybe trust that.... (just guessing, really).

> Better not use pipes, then. Heh. I sense that there is a slight problem with 
> the architecture of Git and trying to get it to behave on Windows... :-)

architecture? Like the "architecture" of species? No, it's evolution. If that applies to the linux kernel, it's not so strange it applies to git too.

Show 5 quoted lines
> > - What encoding should 'update-index' expect from its input? Can you be 
> > sure that other programs generate UTF-8 output?
> 
> Theoretically, if all the internal stuff is hacked around to output Unicode, 
> and the thread codepage is set up to use UTF-8, it should "just work". And

msys doesn't seem to understand UTF-8 at all, so depending on that to work seems futile. Simply bypassing the locale for any internal work is probably the most sane thing. That also won't depend of the quality of the locale support in the runtime. Start by making the git commands working without msys bash, and figure a way to fix msys later, unless someone has a very good idea on how to fix msys.

Show 9 quoted lines
> if run directly from the shell, it should still be converted to whatever the 
> system is set up to emit. That would mean, however, that a Git program that 
> internally runs
> 
>    git-foo | git-bar | git-gazonk
> 
> might behave differently compared to if a user would enter it on the 
> command-line.
> 
You might also want to check out my work in the area. See 
http://www.jgit.org/cgi-bin/gitweb/gitweb.cgi?p=GIT.git;a=shortlog;h=i18n

The goal is locale neutrality yielding the "expected", in the users eyes, result regardless of locale as much as possible. Junio didn't want to have it for five years, so I guess there's still three and half to go. Hopefully he can change his mind. That branch is heavily outdated by now, as some of functionality have been introduced by other means like logoutputencoding and other parts of git have been rewritten.

Related to this, JGit assumes UTF-8 on reading. If it's not valid UTF-8 we try the user's locale (rougly) and on writing object meta data, including any sort of identifier, we always write UTF-8 when have to be explicit. We let the runtime decide on how to encode file names in the file system using the user's locale.

I'd be almost happy with a solution that works when people are interacting using the subset that is convertible between the character sets in use.

-- robin
Previous: Peter KreftingNext: Peter Krefting
Message 8 of 26 in “Windows: Assume all file names to be UTF-8 encoded.”
  1. Windows: Assume all file names to be UTF-8 encoded.Peter Krefting, Mar 2, 2009
  2. Johannes SixtMar 2, 2009
  3. Peter KreftingMar 2, 2009
  4. Johannes SchindelinMar 2, 2009
  5. Peter KreftingMar 2, 2009
  6. Johannes SixtMar 2, 2009
  7. Peter KreftingMar 2, 2009
  8. Robin RosenbergMar 2, 2009
  9. Peter KreftingMar 2, 2009
  10. Robin RosenbergMar 2, 2009
  11. Peter KreftingMar 3, 2009
  12. Dmitry PotapovMar 3, 2009
  13. Peter KreftingMar 3, 2009
  14. Robin RosenbergMar 7, 2009
  15. Peter KreftingMar 2, 2009
  16. Thomas RastMar 2, 2009
  17. Peter KreftingMar 2, 2009
  18. Lars NoschinskiMar 3, 2009
  19. Peter KreftingMar 3, 2009
  20. Lars NoschinskiMar 3, 2009
  21. Robin RosenbergMar 3, 2009
  22. Dmitry PotapovMar 3, 2009
  23. Peter KreftingMar 3, 2009
  24. Dmitry PotapovMar 3, 2009
  25. Peter KreftingMar 4, 2009
  26. Dmitry PotapovMar 4, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.