{"thread":{"id":"18088","subject":"[RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","startedAt":"2009-03-02T08:47:22Z","lastAt":"2009-03-07T10:38:14Z","messageCount":26,"participants":["Peter Krefting","Johannes Sixt","Johannes Schindelin","Thomas Rast","Robin Rosenberg","Lars Noschinski","Dmitry Potapov"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"106687","messageId":"alpine.DEB.2.00.0903020941120.17877@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":null,"subject":"[RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-02T08:47:22Z","receivedAt":"2009-03-02T08:47:22Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"When opening a file through open() or fopen(), the path passed is\nUTF-8 encoded. To handle this on Windows, we need to convert the\npath string to UTF-16 and use the Unicode-based interface.\n---\nWindows does support file names using arbitrary Unicode characters, you just \nneed to use its wchar_t interfaces instead of the char ones (the char ones \njust gets converted into wchar_t on the API level anyway, for the same \nreasons). This is the beginnings of support for UTF-8 file names on Git on \nWindows.\n\nSince there is no real file system abstraction beyond using stdio (AFAIK), I \nneed to hack it by replacing fopen (and open). Probably opendir/readdir as \nwell (might be trickier), and possibly even hack around main() to parse the \nwchar_t command-line instead of the char copy.\n\nThis will lose all chances of Windows 9x compatibility, but I don't know if \nthere are any attempts of supporting it anyway?\n\nPlease note that MultiByteToWideChar() will reject any invalid UTF-8 \nstrings, perhaps it should just fall back to a regular open()/fopen() in \nthat case?\n\nNo Signed-Off line since this is unfinished, just presenting rough sketches \nof an idea.\n\n  compat/mingw.c |   60 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n  compat/mingw.h |    3 ++\n  2 files changed, 62 insertions(+), 1 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex e25cb4f..8b19b80 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -9,13 +9,30 @@ int mingw_open (const char *filename, int oflags, ...)\n  {\n  \tva_list args;\n  \tunsigned mode;\n+\twchar_t *unicode_filename;\n+\tint unicode_filename_len;\n  \tva_start(args, oflags);\n  \tmode = va_arg(args, int);\n  \tva_end(args);\n\n  \tif (!strcmp(filename, \"/dev/null\"))\n  \t\tfilename = \"nul\";\n-\tint fd = open(filename, oflags, mode);\n+\n+\tunicode_filename_len = MultiByteToWideChar(CP_UTF8, 0, filename, -1, NULL, 0);\n+\tif (0 == unicode_filename_len) {\n+\t\terrno = EINVAL;\n+\t\treturn -1;\n+\t};\n+\n+\tunicode_filename = xmalloc(unicode_filename_len * sizeof (wchar_t));\n+\tif (NULL == unicode_filename) {\n+\t\terrno = ENOMEM;\n+\t\treturn -1;\n+\t}\n+\tMultiByteToWideChar(CP_UTF8, 0, filename, -1, unicode_filename, unicode_filename_len);\n+\tint fd = _wopen(unicode_filename, oflags, mode);\n+\tfree(unicode_filename);\n+\n  \tif (fd < 0 && (oflags & O_CREAT) && errno == EACCES) {\n  \t\tDWORD attrs = GetFileAttributes(filename);\n  \t\tif (attrs != INVALID_FILE_ATTRIBUTES && (attrs & FILE_ATTRIBUTE_DIRECTORY))\n@@ -24,6 +41,47 @@ int mingw_open (const char *filename, int oflags, ...)\n  \treturn fd;\n  }\n\n+FILE *mingw_fopen (const char *filename, const char *mode)\n+{\n+\twchar_t *unicode_filename, *unicode_mode;\n+\tint unicode_filename_len, unicode_mode_len;\n+\tFILE *fh;\n+\n+\tunicode_filename_len = MultiByteToWideChar(CP_UTF8, 0, filename, -1, NULL, 0);\n+\tif (0 == unicode_filename_len) {\n+\t\terrno = EINVAL;\n+\t\treturn NULL;\n+\t};\n+\n+\tunicode_filename = xmalloc(unicode_filename_len * sizeof (wchar_t));\n+\tif (NULL == unicode_filename) {\n+\t\terrno = ENOMEM;\n+\t\treturn NULL;\n+\t}\n+\tMultiByteToWideChar(CP_UTF8, 0, filename, -1, unicode_filename, unicode_filename_len);\n+\n+\tunicode_mode_len = MultiByteToWideChar(CP_UTF8, 0, mode, -1, NULL, 0);\n+\tif (0 == unicode_mode_len) {\n+\t\tfree(unicode_filename);\n+\t\terrno = EINVAL;\n+\t\treturn NULL;\n+\t};\n+\n+\tunicode_mode = xmalloc(unicode_mode_len * sizeof (wchar_t));\n+\tif (NULL == unicode_mode) {\n+\t\tfree(unicode_mode);\n+\t\terrno = ENOMEM;\n+\t\treturn NULL;\n+\t}\n+\tMultiByteToWideChar(CP_UTF8, 0, mode, -1, unicode_mode, unicode_mode_len);\n+\n+\tfh = _wfopen(unicode_filename, unicode_mode);\n+\tfree(unicode_filename);\n+\tfree(unicode_mode);\n+\n+\treturn fh;\n+}\n+\n  static inline time_t filetime_to_time_t(const FILETIME *ft)\n  {\n  \tlong long winTime = ((long long)ft->dwHighDateTime << 32) + ft->dwLowDateTime;\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 4f275cb..235df0a 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -142,6 +142,9 @@ int sigaction(int sig, struct sigaction *in, struct sigaction *out);\n  int mingw_open (const char *filename, int oflags, ...);\n  #define open mingw_open\n\n+FILE *mingw_fopen (const char *filename, const char *mode);\n+#define fopen mingw_fopen\n+\n  char *mingw_getcwd(char *pointer, int len);\n  #define getcwd mingw_getcwd\n\n-- \n1.6.0.2.1172.ga5ed0\n"},{"id":"106689","messageId":"49ABB529.1080500@viscovery.net","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903020941120.17877@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-03-02T10:30:01Z","receivedAt":"2009-03-02T10:30:01Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Peter Krefting schrieb:\n> When opening a file through open() or fopen(), the path passed is\n> UTF-8 encoded.\n\nI don't think that this assumption is valid. Whenever the Windows API has\nto convert between Unicode strings and char* strings, it uses the current\n\"ANSI code page\". As far as I know, the UTF-8 codepage (65001) cannot be\nused as the \"current ANSI code page\". Users will always have some code\npage set that is not UTF-8.\n\nFor example, if the user specifies a file name on the command line, than\nit will not enter git in UTF-8, but in the current \"ANSI\" or \"OEM code\npage\" encoding. If git prints a file name under the assumption that it is\nUTF-8 encoded, then it will be displayed incorrectly because the system\nuses a different encoding.\n\n> Since there is no real file system abstraction beyond using stdio\n> (AFAIK), I need to hack it by replacing fopen (and open). Probably\n> opendir/readdir as well (might be trickier), and possibly even hack\n> around main() to parse the wchar_t command-line instead of the char copy.\n\nI think you are grossly underestimating the venture that you want to\nundertake here.\n\nPlease come up with a plan how you are going to deal with the various\nissues. File names enter and leave the system through different channels:\n\n- the command line and terminal window\n- object database (tree objects)\n- opendir/readdir; opening files or directories for reading or writing\n\nAnd there is probably some more... How do you treat encodings in these\nchannels? What if the file names are not valid UTF-8? Etc.\n\nThe biggest obstacle will be that git does not have a notion of \"file name\nencoding\" - it simply treats a file name as a stream of bytes. There is no\nplace to write an encoding. If the byte streams are regarded as having an\nencoding, then you can have ambiguities, mixed encodings, or invalid\ncharacters. You would have to deal with this in some way.\n\n> This will lose all chances of Windows 9x compatibility, but I don't know\n> if there are any attempts of supporting it anyway?\n\nWindows 9x is already out of the loop. We use GetFileInformationByHandle()\nthat is only available since Windows 2000.\n\n-- Hannes\n"},{"id":"106691","messageId":"alpine.DEB.2.00.0903021137110.17877@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"49ABB529.1080500@viscovery.net","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-02T10:46:47Z","receivedAt":"2009-03-02T10:46:47Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Johannes Sixt:\n\n> I don't think that this assumption is valid.\n\nDepends on where you are coming from. For the files stored in the Git \nrepositories, I believe all file names are supposed to be UTF-8 encoded \n(just like commit messages and user names are). That's the assumption I \nstarted working from.\n\n> Users will always have some code page set that is not UTF-8.\n\nIndeed. And as long as the char-pointer interfaces in stdio and elsewhere \nwork on that assumption, we have a problem.\n\n> For example, if the user specifies a file name on the command line, than\n> it will not enter git in UTF-8, but in the current \"ANSI\" or \"OEM code\n> page\" encoding.\n\nThat problem is already solved as we do have a wchar_t command line \navailable. If you pass a file name that is not representable in the current \n\"ANSI\" codepage on the command line, it will come out as garbage in the \nchar* version, but will be correct in the wchar_t* version. Thus we need to \nconvert that to utf-8 and use that instead.\n\n> If git prints a file name under the assumption that it is UTF-8 encoded, \n> then it will be displayed incorrectly because the system uses a different \n> encoding.\n\nHere setting the local codepage to UTF-8 *might* work, although I haven't \ntested that. Or always use the wchar_t versions of printf and friends.\n\n> I think you are grossly underestimating the venture that you want to \n> undertake here.\n\nI've done this before with other software, so, yes, I know it is quite a big \nundertaking. That is also why I started out with a minimal RFC patch to see \nif there was any interest in working with this.\n\n> Please come up with a plan how you are going to deal with the various\n> issues. File names enter and leave the system through different channels:\n>\n> - the command line and terminal window\n\nGetCommandLineW() as decribed above.\n\n> - object database (tree objects)\n\nThose file names are supposedly always UTF-8.\n\n> - opendir/readdir; opening files or directories for reading or writing\n\nWrap file open and directory read to use the wchar_t versions, converting \nthat to UTF-8 strings at the API level.\n\n> And there is probably some more... How do you treat encodings in these \n> channels? What if the file names are not valid UTF-8? Etc.\n\nIll-formed UTF-8 should just be rejected. Invalid UTF-8 is worse. I'm not \nsure what the Linux version does, when running in a UTF-8 locale. Does it \nallow ill-formed or illegal UTF-8 sequences?\n\nNTFS allows almost any sequence of wchar_t's, it doesn't even have to be \nvalid UTF-16.\n\n> The biggest obstacle will be that git does not have a notion of \"file name \n> encoding\" - it simply treats a file name as a stream of bytes.\n\nYeah, that is one of the major bugs in its design, IMHO. But almost everyone \nseems to assume that file names are UTF-8 strings anyway, so in the absence \nof any other information, it's a good assumption as any to make.\n\n> If the byte streams are regarded as having an encoding, then you can have \n> ambiguities, mixed encodings, or invalid characters. You would have to \n> deal with this in some way.\n\nConsidering we already see problems with file names that cannot properly be \nrepresented on some file systems (case-only differences in the Linux kernel \nwhen checked out on Windows; Mac OS' built-in Unicode normalization of file \nnames, etc.)\n\n> Windows 9x is already out of the loop.\n\nGood.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106692","messageId":"alpine.DEB.1.00.0903021153520.10279@pacific.mpi-cbg.de","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903021137110.17877@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-02T10:56:40Z","receivedAt":"2009-03-02T10:56:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 2 Mar 2009, Peter Krefting wrote:\n\n> Johannes Sixt:\n> \n> > I don't think that this assumption is valid.\n> \n> Depends on where you are coming from. For the files stored in the Git \n> repositories, I believe all file names are supposed to be UTF-8 encoded \n> (just like commit messages and user names are). That's the assumption I \n> started working from.\n\nNo.  As far as Git is concerned, the file names are just as much blobs as \nthe file contents.\n\nThe fact that Windows messes with this notion just as it messes with the \nfile contents (think the endless story whose name is CR/LF) shows only how \n\"well\" designed the concepts in Windows are.\n\nAnd as it stands, we have at least two issues on the msysGit issue tracker \nthat complain that Git does not work with localized file names properly.\n\nSo no, file names are not UTF-8 at all, especially not on Windows.\n\nDo not get me wrong, I really welcome you taking care of the issue, but I \ndo not think that forcing UTF-8 is a solution.\n\nThanks & sorry,\nDscho\n"},{"id":"106693","messageId":"alpine.DEB.2.00.0903021256200.17877@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"alpine.DEB.1.00.0903021153520.10279@pacific.mpi-cbg.de","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-02T12:03:58Z","receivedAt":"2009-03-02T12:03:58Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Johannes Schindelin:\n\n> No.  As far as Git is concerned, the file names are just as much blobs as \n> the file contents.\n\nI've struggled with the same problems on Linux before, since its file \nsystems doesn't have the concept of characters, either. I guess it's just \ndesign principles, but as far as I am concerned, having file names be \nconstructed from characters makes a lot more sense than having them \nconstructed from bytes.\n\nGit does the right thing in assuming commit messages and user names be UTF-8 \ncharacters, though, it would have been nice to have file names covered by \nthe same constraints.\n\n> The fact that Windows messes with this notion just as it messes with the \n> file contents (think the endless story whose name is CR/LF) shows only how \n> \"well\" designed the concepts in Windows are.\n\nIn this case, yes, Windows' way of doing does make more sense, at least to \nme. And as far as text files are concerned, treating text as sequences of \nbytes are in most cases not a very smart thing to do, either, but it's hard \nnot to given how most computers are constructed.\n\n> And as it stands, we have at least two issues on the msysGit issue tracker \n> that complain that Git does not work with localized file names properly.\n>\n> So no, file names are not UTF-8 at all, especially not on Windows.\n\nI am not trying to make file names *on Windows* to be UTF-8. I am trying to \nmake file names on Windows be Windows file names, i.e UTF-16 Unicode. It's \njust that since Git internally uses the char* APIs, and from what I have \nseen in most other cases assume that char* text is UTF-8, I am trying to \nconvert from Windows' view of path names to Git's (UTF-16 to UTF-8) and back.\n\nThe other way would be to keep the char* APIs but convert to the Windows \nlocale encoding (\"ANSI codepage\"), but that will break horribly as not all \nfile names that can be used on a file system can be represented as such. \nPlus, all calls to a Windows API using a char* path name *is* converted into \nUTF-16 anyway, since that is what is used internally in the Windows NT \nsubsystems.\n\n> Do not get me wrong, I really welcome you taking care of the issue, but I a\n> do not think that forcing UTF-8 is a solution.\n\nSome kind of handling of Git repositories where file names are not UTF-8 \nwould probably need to be added, yes.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106696","messageId":"49ABD24B.5060005@viscovery.net","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903021137110.17877@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-03-02T12:34:19Z","receivedAt":"2009-03-02T12:34:19Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Peter Krefting schrieb:\n> Johannes Sixt:\n>> If git prints a file name under the assumption that it is UTF-8\n>> encoded, then it will be displayed incorrectly because the system uses\n>> a different encoding.\n> \n> Here setting the local codepage to UTF-8 *might* work, although I\n> haven't tested that. Or always use the wchar_t versions of printf and\n> friends.\n\nYou cannot expect users to switch the locale. For example, I have to test\nour software with Japanese settings: I *cannot* switch to UTF-8 just\nbecause of git.\n\nCan you set the local codepage per program? (I don't know.) It might help\nhere, but it doesn't help in all cases, particularly in certain pipelines:\n\n  git ls-files -o\n  git ls-files -o | git update-index --add --stdin\n  find . -name \\*.jpg | git update-index --add --stdin\n\n- What encoding should 'ls-files' use for its output? Certainly not always\nUTF-8: stdout should use the local code page so that the file names are\ninterpreted correctly by the terminal window (it expects the local code page).\n\n- What encoding should 'update-index' expect from its input? Can you be\nsure that other programs generate UTF-8 output?\n\nHow do you solve that?\n\n-- Hannes\n"},{"id":"106700","messageId":"alpine.DEB.2.00.0903021404000.17877@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"49ABD24B.5060005@viscovery.net","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-02T13:12:32Z","receivedAt":"2009-03-02T13:12:32Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Johannes Sixt:\n\n> Can you set the local codepage per program? (I don't know.)\n\nThe locale is set per thread, and gets reset when the program exits. So \nsetting the codepage to UTF-8 before outputting should work. That should \nalso work for displaying the log to the terminal if you have UTF-8 log \nmessages.\n\nConverting it to wchar_t and using wprintf and similar should be safer, \nthough (and I have no idea what happens if you try to pipe the output to \nsomething else).\n\n> - What encoding should 'ls-files' use for its output? Certainly not always \n> UTF-8: stdout should use the local code page so that the file names are \n> interpreted correctly by the terminal window (it expects the local code \n> page).\n\nThat is exactly why trying to mix \"protocol\" data (\"plumbing\" in Git's case) \nand user output will always come back and bite you, one way or another. I \nhaven't really the faintest how pipes work with Unicode on Windows. \nSomewhere along the line there will probably be some conversions, which \nwould cause interesting issues.\n\nBetter not use pipes, then. Heh. I sense that there is a slight problem with \nthe architecture of Git and trying to get it to behave on Windows... :-)\n\n> - What encoding should 'update-index' expect from its input? Can you be \n> sure that other programs generate UTF-8 output?\n\nTheoretically, if all the internal stuff is hacked around to output Unicode, \nand the thread codepage is set up to use UTF-8, it should \"just work\". And \nif run directly from the shell, it should still be converted to whatever the \nsystem is set up to emit. That would mean, however, that a Git program that \ninternally runs\n\n   git-foo | git-bar | git-gazonk\n\nmight behave differently compared to if a user would enter it on the \ncommand-line.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106703","messageId":"alpine.DEB.2.00.0903021452010.17877@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"a2633edd0903020512u5682e9am203f0faccd0acf6a@mail.gmail.com","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-02T13:57:52Z","receivedAt":"2009-03-02T13:57:52Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Hi!\n\n> Makes sense too. I think the whole API would have to be changed to use \n> TCHAR*.\n\nI'd rather just say wchar_t explicitely. I'm not particularly fond of macros \nthat change under your feet just because you fail to define a symbol \nsomewhere...\n\n> Then you need to do the right conversion at the right places, this will be \n> quite tricky, painful work, but there is probably no way around that.\n\nIn the other project I worked on we ended up wrapping all file-related calls \nin our own porting interface, and then let each platform we compiled for \nimplement their own methods for handling Unicode paths. For Windows it's \ntrivial since all APIs are Unicode. For Unix-like OSes it's tricky as you \nhave to take the locale settings into account, but fortunately the world is \nslowly moving towards UTF-8 locales, which eases the pain a bit.\n\n> Note that not only conversions will be needed but you'll also need to \n> adjust all routines handling filenames to use the proper Unicode version. \n> (strchr -> _tstrchr, open -> _topen, strcpy -> _tstrcpy, strlen -> \n> _tcslen, ...).\n\nNot necessarily. If the code can be set up to use UTF-8 char* internally, \nnot everything needs to be rewritten (I've done that too, only took a \ncouple of years to move the codebase over to all-Unicode).\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106704","messageId":"200903021530.01165.trast@student.ethz.ch","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903021452010.17877@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2009-03-02T14:29:54Z","receivedAt":"2009-03-02T14:29:54Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Peter Krefting wrote:\n> In the other project I worked on we ended up wrapping all file-related calls \n> in our own porting interface, and then let each platform we compiled for \n> implement their own methods for handling Unicode paths. For Windows it's \n> trivial since all APIs are Unicode. For Unix-like OSes it's tricky as you \n> have to take the locale settings into account, but fortunately the world is \n> slowly moving towards UTF-8 locales, which eases the pain a bit.\n\nHave you thought about all the consequences this would have for the\n*nix people here? [*]\n\nEven if you pretend that Git did always enforce UTF-8 paths in its\ntrees, so that there's no backward compatibility to be cared for,\nyou're still in a world of hurt when trying to check out such paths\nunder a locale (or whatever setting might control this new encoding\nlogic) that does not support the whole range of UTF-8.\n\nLike, say, the C locale.\n\nNext you get to see to it that the users can spell all filenames even\nif their locale doesn't let them, since they'll want to do things like\n'git show $rev:$file' with them.\n\nWith backwards compatibility it's even worse as you're suddenly\nimposing extra restrictions on what a valid filename in the repository\nmust look like.\n\n\n[*] I'm _extremely_ tempted to write \"people using non-broken OSes\",\nbut let's pretend to be neutral for a second.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"106722","messageId":"200903022058.33392.robin.rosenberg.lists@dewire.com","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903021404000.17877@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-03-02T19:58:33Z","receivedAt":"2009-03-02T19:58:33Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"> Johannes Sixt:\n> \n> > Can you set the local codepage per program? (I don't know.)\n> \n> The locale is set per thread, and gets reset when the program exits. So \n> setting the codepage to UTF-8 before outputting should work. That should \n> also work for displaying the log to the terminal if you have UTF-8 log \n> messages.\n\nMessing with locale is probably going to break subtly. An explicit approach\nis better, respecting the user's locale when necessary.\n\n> Converting it to wchar_t and using wprintf and similar should be safer, \n> though (and I have no idea what happens if you try to pipe the output to \n> something else).\n> \n> > - What encoding should 'ls-files' use for its output? Certainly not always \n> > UTF-8: stdout should use the local code page so that the file names are \n> > interpreted correctly by the terminal window (it expects the local code \n> > page).\n> \n> That is exactly why trying to mix \"protocol\" data (\"plumbing\" in Git's case) \n> and user output will always come back and bite you, one way or another. I \n> haven't really the faintest how pipes work with Unicode on Windows. \n> Somewhere along the line there will probably be some conversions, which \n> would cause interesting issues.\n\nPipes are just bytes so you have to know what you're piping by convention\nor protocol. You can ask for the console output page, which may be set to\na multibyte locale or unicode and maybe trust that.... (just guessing, really).\n\n> Better not use pipes, then. Heh. I sense that there is a slight problem with \n> the architecture of Git and trying to get it to behave on Windows... :-)\n\narchitecture? Like the \"architecture\" of species? No, it's evolution.\nIf that applies to the linux kernel, it's not so strange it applies to git too.\n\n> > - What encoding should 'update-index' expect from its input? Can you be \n> > sure that other programs generate UTF-8 output?\n> \n> Theoretically, if all the internal stuff is hacked around to output Unicode, \n> and the thread codepage is set up to use UTF-8, it should \"just work\". And \n\nmsys doesn't seem to understand UTF-8 at all, so depending on that to work\nseems futile. Simply bypassing the locale for any internal work is probably the \nmost sane thing. That also won't depend of the quality of the locale support in \nthe runtime. Start by making the git commands working without msys bash,\nand figure a way to fix msys later, unless someone has a very good idea on\nhow to fix msys.\n\n> if run directly from the shell, it should still be converted to whatever the \n> system is set up to emit. That would mean, however, that a Git program that \n> internally runs\n> \n>    git-foo | git-bar | git-gazonk\n> \n> might behave differently compared to if a user would enter it on the \n> command-line.\n> \n\nYou might also want to check out my work in the area. See \n\nhttp://www.jgit.org/cgi-bin/gitweb/gitweb.cgi?p=GIT.git;a=shortlog;h=i18n\n\nThe goal is locale neutrality yielding the \"expected\", in the users eyes, result regardless\nof locale as much as possible. Junio didn't want to have it for five years, so I\nguess there's still three and half to go. Hopefully he can change his mind. That branch\nis heavily outdated by now, as some of functionality have been introduced by other\nmeans like logoutputencoding and other parts of git have been rewritten.\n\nRelated to this, JGit assumes UTF-8 on reading. If it's not valid UTF-8 we try the user's \nlocale (rougly) and on writing object meta data, including any sort of identifier, \nwe always write UTF-8 when have to be explicit. We let the runtime decide on how\nto encode file names in the file system using the user's locale.\n\nI'd be almost happy with a solution that works when people are interacting using\nthe subset that is convertible between the character sets in use.\n\n-- robin\n"},{"id":"106725","messageId":"alpine.DEB.2.00.0903022135360.20047@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"200903021530.01165.trast@student.ethz.ch","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-02T20:41:57Z","receivedAt":"2009-03-02T20:41:57Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Thomas Rast:\n\n> Have you thought about all the consequences this would have for the *nix \n> people here? [*]\n\nYeah. It will fix problems trying to check out a Git repository created by \nme in a iso8859-1 locale on a machine using a utf-8 locale, where both ends \nwould like to have a file named \"Ü\".\n\nOr, hopefully, a careful adoption of this on Windows won't affect Unixes and \nother systems with pre-Unicode APIs at all, since the Windows code would be \nin the \"compat\" directory.\n\n> you're still in a world of hurt when trying to check out such paths under \n> a locale (or whatever setting might control this new encoding logic) that \n> does not support the whole range of UTF-8.\n\nYeah. That would be a case similar to the casing problem on Windows.\n\n> With backwards compatibility it's even worse as you're suddenly imposing \n> extra restrictions on what a valid filename in the repository must look \n> like.\n\nIndeed. It is unfortunate that this wasn't properly specified to start with. \nIt's mostly a minor issue since *most* people will not use non-ASCII file \nnames. At least for most of the kind of projects that Git have attracted so \nfar, so the problem is not that big. The problem is if Git is to attract \n\"the masses\". Especially on Windows, where file names using non-ASCII are \ncommon, this needs to be addressed eventually.\n\n> [*] I'm _extremely_ tempted to write \"people using non-broken OSes\", but \n> let's pretend to be neutral for a second.\n\nIn most cases, I would most definitely agree with you on calling it that, \nbut when it comes to Unicode support, Windows is one of the least broken \nOSes (with Symbian being my favourite).\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106727","messageId":"alpine.DEB.2.00.0903022146510.20047@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"200903022058.33392.robin.rosenberg.lists@dewire.com","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-02T20:52:41Z","receivedAt":"2009-03-02T20:52:41Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Robin Rosenberg:\n\n> Pipes are just bytes so you have to know what you're piping by convention \n> or protocol. You can ask for the console output page, which may be set to \n> a multibyte locale or unicode and maybe trust that.... (just guessing, \n> really).\n\nYou can get cmd.exe to write data to pipes and redirections as UTF-16 \nUnicode (cmd.exe /u), perhaps there is a way to capitalise on that? \n\"Unfortunately\", the Git stuff is mostly called from a bash shell inside \nmsys, so it requires a \"bit\" more work...\n\n> architecture? Like the \"architecture\" of species? No, it's evolution.\n\nThere's still an architecture there, somewhere. Perhaps not intended or \nspecified, but there definitely is one :-)\n\n> http://www.jgit.org/cgi-bin/gitweb/gitweb.cgi?p=GIT.git;a=shortlog;h=i18n\n>\n> The goal is locale neutrality yielding the \"expected\", in the users eyes, \n> result regardless of locale as much as possible.\n\nAh, yes, that looks like an interesting starting point. I already assumed \nthat Git on Linux would use UTF-8 for everything already, since it already \ndoes that for the commit messages despite me using an iso8859-1 locale. \nApparently I haven't done my homework.\n\n> We let the runtime decide on how to encode file names in the file system \n> using the user's locale.\n\nThat's good. That's what I'm trying to achieve. Or, rather, avoid the user \nlocale altogether (which is easy on Windows since the file names are always \nstored in Unicode, and the user locale can be bypassed).\n\n> I'd be almost happy with a solution that works when people are interacting \n> using the subset that is convertible between the character sets in use.\n\nYou mean like the \"invariant\" character set? :-) Using Unicode internally \n(in whatever encoding) is nice, the problem is when you have to interact \nwith the world around you.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106728","messageId":"200903022221.10254.robin.rosenberg.lists@dewire.com","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903022146510.20047@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-03-02T21:21:09Z","receivedAt":"2009-03-02T21:21:09Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"måndag 02 mars 2009 21:52:41 skrev Peter Krefting <peter@softwolves.pp.se>:\n> Robin Rosenberg:\n> \n> > I'd be almost happy with a solution that works when people are interacting \n> > using the subset that is convertible between the character sets in use.\n> \n> You mean like the \"invariant\" character set? :-) Using Unicode internally \n> (in whatever encoding) is nice, the problem is when you have to interact \n> with the world around you.\n\nNot sure what that is. I mean that in a local nordic, setting people can use iso-8859-1|15/windows-1252/UTF-8 for their needs be means of converting the characters as-needed without loss, with very few practial restrictions. \n\nFor a larger setting that won't do, but then the need is typically less since people tend to use ASCII only, or you jump to all unicode.\n\nJust because I use UTF-8 doesn't mean I use start using more characters in practice.\n\n-- robin\n"},{"id":"106764","messageId":"alpine.DEB.2.00.0903030649231.2012@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"200903022221.10254.robin.rosenberg.lists@dewire.com","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-03T05:51:59Z","receivedAt":"2009-03-03T05:51:59Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Robin Rosenberg:\n\n> Not sure what that is.\n\n\"Invariant\" is defined in an old RFC as the common subset of several \nASCII-like and ASCII-based encodings. This was back before the MIME days, \nIIANM.\n\n> I mean that in a local nordic, setting people can use \n> iso-8859-1|15/windows-1252/UTF-8 for their needs be means of converting \n> the characters as-needed without loss, with very few practial \n> restrictions.\n\nIndeed. The trick is to have the storage (in this case, Git and it's tree \nobjects) storing the file name data in a commonly agreed-upon way. Then it \nis simple to convert at the end-points.\n\n> Just because I use UTF-8 doesn't mean I use start using more characters in \n> practice.\n\nMost people do not, no. But using a Unicode encoding means that they at \nleast have the option. Sometimes, having to mangle stuff down to ASCII is a \npain.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106781","messageId":"20090303075655.GB9875@lars.home.noschinski.de","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903022135360.20047@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Lars Noschinski","fromEmail":"lars-2008-2@usenet.noschinski.de","sentAt":"2009-03-03T07:56:55Z","receivedAt":"2009-03-03T07:56:55Z","isPatch":true,"sender":{"key":"lars-2008-2@usenet.noschinski.de","avatar":null},"body":"* Peter Krefting <peter@softwolves.pp.se> [09-03-02 21:41]:\n> Indeed. It is unfortunate that this wasn't properly specified to start with. \n> It's mostly a minor issue since *most* people will not use non-ASCII file \n> names. At least for most of the kind of projects that Git have attracted so \n> far, so the problem is not that big. The problem is if Git is to attract \"the \n> masses\". Especially on Windows, where file names using non-ASCII are common, \n> this needs to be addressed eventually.\n\nUsing no encoding for filenames was the obvious (and I would argue)\ncorrect choice. Unix filenames are specified to be a sequence of bytes,\nexcluding '/' and '\\0'. A lot of these sequences are not valid UTF-8.\nFurther, the encoding needed for filenames depends on the encoding used\nin the source code for referencing these files. Again, for the unix file\nhandling functions, this means no encoding.\n\nChanging the filename (on checkout), so that the user sees an Ü\nregardless of his or her locale (instead of an \\0xDC, which only\nresolves to an Ü on latin-1) would be an absolutely broken concept here.\n\n> >[*] I'm _extremely_ tempted to write \"people using non-broken OSes\", but let's \n> >pretend to be neutral for a second.\n> \n> In most cases, I would most definitely agree with you on calling it that, but \n> when it comes to Unicode support, Windows is one of the least broken OSes (with \n> Symbian being my favourite).\n\nIMHO having encoding specific open functions is begging for problems.\n\n - Lars.\n"},{"id":"106806","messageId":"37fcd2780903030143t7abe33d5sb7d8163c3c9bf505@mail.gmail.com","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903020941120.17877@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-03-03T09:43:55Z","receivedAt":"2009-03-03T09:43:55Z","isPatch":true,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Mon, Mar 02, 2009 at 09:47:22AM +0100, Peter Krefting wrote:\n> When opening a file through open() or fopen(), the path passed is\n> UTF-8 encoded. To handle this on Windows, we need to convert the\n> path string to UTF-16 and use the Unicode-based interface.\n\nIMHO, you grossly underestimate what is needed to enable UTF-8 encoding\nin Windows. AFAIK, Microsoft C runtime library does not support UTF-8,\nso you have to wrap all C functions taking 'char*' as an input parameter.\nFor example, think about what is going to happen if Git tries to print\na simple error message:\n  fprintf (stderr, \"unable to open %s\", path);\n\n> Since there is no real file system abstraction beyond using stdio_\n> (AFAIK), I need to hack it by replacing fopen (and open). Probably_\n> opendir/readdir as well (might be trickier), and possibly even hack_\n> around main() to parse the wchar_t command-line instead of the char copy.\n\nAnd the command-line is not the only source of file names. Some Git\ncommands read list of files from stdin usually though the pipe. In\nwhat encoding are they going to be?\n\nDmitry\n"},{"id":"106807","messageId":"37fcd2780903030147q7062ee47w7ce524c28a6aa347@mail.gmail.com","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903022135360.20047@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-03-03T09:47:31Z","receivedAt":"2009-03-03T09:47:31Z","isPatch":true,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Mon, Mar 02, 2009 at 09:41:57PM +0100, Peter Krefting wrote:\n>\n> In most cases, I would most definitely agree with you on calling it that,_\n> but when it comes to Unicode support, Windows is one of the least broken__\n> OSes (with Symbian being my favourite).\n\nThe C Standard requires that the type wchar_t is capable of representing\nany character in the current locale. If Windows uses UTF-16 as internal\nencoding (so, it can work with symbols outside of the BMP), it means you\ncannot have 16-bit wchar_t and be compliant with the C standard...\n\nDmitry\n"},{"id":"106822","messageId":"alpine.DEB.2.00.0903031246420.3702@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"37fcd2780903030147q7062ee47w7ce524c28a6aa347@mail.gmail.com","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-03T11:48:02Z","receivedAt":"2009-03-03T11:48:02Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Dmitry Potapov:\n\n> The C Standard requires that the type wchar_t is capable of representing \n> any character in the current locale. If Windows uses UTF-16 as internal \n> encoding (so, it can work with symbols outside of the BMP), it means you \n> cannot have 16-bit wchar_t and be compliant with the C standard...\n\nNo, that's not quite correct. wchar_t is defined to be \"an integer type whose \nrange of values can represent distinct codes for all members of \nthe largest extended character set specified among the supported locales\". \nSince Windows defines all local character sets as Unicode-based, having \nwchar_t defined as Unicode means that it can represent everything.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106824","messageId":"alpine.DEB.2.00.0903031248040.3702@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"20090303075655.GB9875@lars.home.noschinski.de","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-03T11:54:31Z","receivedAt":"2009-03-03T11:54:31Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Lars Noschinski:\n\n> Using no encoding for filenames was the obvious (and I would argue) \n> correct choice. Unix filenames are specified to be a sequence of bytes, \n> excluding '/' and '\\0'.\n\nI know the Unix way of thinking lends itself to such a design. This is one \nof the few cases where I personally think Unix has got it wrong, and Windows \n(NT) has got it right. But then again, Unix' design pre-dates the locale \nissue by quite some time, so it is not difficult to see where it comes from.\n\n> Changing the filename (on checkout), so that the user sees an Ü regardless \n> of his or her locale (instead of an \\0xDC, which only resolves to an Ü on \n> latin-1) would be an absolutely broken concept here.\n\nWhy would it? It is my view as a user on my files that define how file names \nare looked upon. If I have three machines, one Linux box using a iso8859-1 \nlocale, an OS X box (where, I would believe, file APIs use UTF-8, someone \nplease correct me if I'm wrong), and a Windows box (which uses UTF-16 on the \nfile system layer, but does provide compatibility functions that use char \npointers), and create a file on each of these called \"Ü.txt\" (which would be \nthe sequence \"DC 2E 74 78 74\" on the Linux box, \"C3 9C 2E 74 78 74\" (or \nprobably something else since I believe OS X decomposes the string) on the \nOS X box and \"00DC 002E 0074 0078 0074\" on the Windows box, I see these \nthree file names as equal.\n\nIf I would create a Git repo on each of the three machines and put the file \nname in it, and then clone that on one of the other machines. *I* would \nassume that the file names were converted to fit the host operating system.\n\n> IMHO having encoding specific open functions is begging for problems.\n\nIndeed. That's why I like Windows' wchar_t APIs, and dislike Unix' and \nLinux' char APIs that, in some ways, depend on the user locale.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106826","messageId":"alpine.DEB.2.00.0903031254440.3702@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"37fcd2780903030143t7abe33d5sb7d8163c3c9bf505@mail.gmail.com","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-03T11:56:47Z","receivedAt":"2009-03-03T11:56:47Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Dmitry Potapov:\n\n> IMHO, you grossly underestimate what is needed to enable UTF-8 encoding in \n> Windows. AFAIK, Microsoft C runtime library does not support UTF-8, so you \n> have to wrap all C functions taking 'char*' as an input parameter.\n\nI have to wrap all file-related functions, at least.\n\n> For example, think about what is going to happen if Git tries to print a \n> simple error message: fprintf (stderr, \"unable to open %s\", path);\n\nYeah. That's a problem. That might be solvable by setting the thread locale \nto something UTF-8 based and have the console window convert to the output \ncodepage (that is what it does when you use wprintf and friends).\n\n> And the command-line is not the only source of file names. Some Git \n> commands read list of files from stdin usually though the pipe. In what \n> encoding are they going to be?\n\nIndeed. Pipes are a problem.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106875","messageId":"20090303162917.GA8096@lars.home.noschinski.de","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903031248040.3702@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Lars Noschinski","fromEmail":"lars-2008-2@usenet.noschinski.de","sentAt":"2009-03-03T16:29:25Z","receivedAt":"2009-03-03T16:29:25Z","isPatch":true,"sender":{"key":"lars-2008-2@usenet.noschinski.de","avatar":null},"body":"* Peter Krefting <peter@softwolves.pp.se> [09-03-03 12:54]:\n> Lars Noschinski:\n> >Changing the filename (on checkout), so that the user sees an Ü regardless of \n> >his or her locale (instead of an \\0xDC, which only resolves to an Ü on \n> >latin-1) would be an absolutely broken concept here.\n> \n> Why would it? It is my view as a user on my files that define how file names \n> are looked upon. If I have three machines, one Linux box using a iso8859-1 \n> locale, an OS X box (where, I would believe, file APIs use UTF-8, someone \n> please correct me if I'm wrong), and a Windows box (which uses UTF-16 on the \n> file system layer, but does provide compatibility functions that use char \n> pointers), and create a file on each of these called \"Ü.txt\" (which would be \n> the sequence \"DC 2E 74 78 74\" on the Linux box, \"C3 9C 2E 74 78 74\" (or \n> probably something else since I believe OS X decomposes the string) on the OS X \n> box and \"00DC 002E 0074 0078 0074\" on the Windows box, I see these three file \n> names as equal.\n\nBecause a function in the source code refers to (e.g.) \"DC 2E 74 78 74\",\nnot \"C3 9C 2E 74 78 74\" nor \"00DC 0024 0074 0078 0074\". And it does so\nregardless of the locale.\n\nThe file name may look funny depending on your locale, but if you rename\nthe file to fit your local enconding, it would not work.\n"},{"id":"106887","messageId":"37fcd2780903030913q4ee0e5d0t45dc3b124285f748@mail.gmail.com","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903031246420.3702@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-03-03T17:13:58Z","receivedAt":"2009-03-03T17:13:58Z","isPatch":true,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Tue, Mar 3, 2009 at 2:48 PM, Peter Krefting <peter@softwolves.pp.se> wrote:\n> Dmitry Potapov:\n>\n>> The C Standard requires that the type wchar_t is capable of representing\n>> any character in the current locale. If Windows uses UTF-16 as internal\n>> encoding (so, it can work with symbols outside of the BMP), it means you\n>> cannot have 16-bit wchar_t and be compliant with the C standard...\n>\n> No, that's not quite correct. wchar_t is defined to be \"an integer type\n> whose range of values can represent distinct codes for all members of the\n> largest extended character set specified among the supported locales\". Since\n> Windows defines all local character sets as Unicode-based, having wchar_t\n> defined as Unicode means that it can represent everything.\n\nNo, it does not, if you have wchar_t that is only 16-bit wide, because\ncharacters\noutside of the BMP have integer values in Unicode greater than 65535...\n\nDmitry\n"},{"id":"106929","messageId":"200903032159.25461.robin.rosenberg.lists@dewire.com","threadId":"18088","inReplyTo":"20090303162917.GA8096@lars.home.noschinski.de","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-03-03T20:59:25Z","receivedAt":"2009-03-03T20:59:25Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Lars Noschinski <lars-2008-2@usenet.noschinski.de> writes:\n> * Peter Krefting <peter@softwolves.pp.se> [09-03-03 12:54]:\n> > Lars Noschinski:\n> > >Changing the filename (on checkout), so that the user sees an Ü regardless of \n> > >his or her locale (instead of an \\0xDC, which only resolves to an Ü on \n> > >latin-1) would be an absolutely broken concept here.\n> > \n> > Why would it? It is my view as a user on my files that define how file names \n> > are looked upon. If I have three machines, one Linux box using a iso8859-1 \n> > locale, an OS X box (where, I would believe, file APIs use UTF-8, someone \n> > please correct me if I'm wrong), and a Windows box (which uses UTF-16 on the \n> > file system layer, but does provide compatibility functions that use char \n> > pointers), and create a file on each of these called \"Ü.txt\" (which would be \n> > the sequence \"DC 2E 74 78 74\" on the Linux box, \"C3 9C 2E 74 78 74\" (or \n> > probably something else since I believe OS X decomposes the string) on the OS X \n> > box and \"00DC 002E 0074 0078 0074\" on the Windows box, I see these three file \n> > names as equal.\n> \n> Because a function in the source code refers to (e.g.) \"DC 2E 74 78 74\",\n> not \"C3 9C 2E 74 78 74\" nor \"00DC 0024 0074 0078 0074\". And it does so\n> regardless of the locale.\n\nThe only actual language I know where I've seen people use non-ascii names for\nreferenced files, i.e. classes, is Java and there you specify the encoding to\nthe compiler. Class names are not byte sequences there. XML files are another\ncase where references files are defined in unicode. I assume this applies to\nC# and other modern languages too.\n\n> The file name may look funny depending on your locale, but if you rename\n> the file to fit your local enconding, it would not work.\n\nIn the Java case, you /have/ to \"rename\" or the build will break. Build systems like Ant\nor Maven require you to \"rename\" too regardless of what you build. A C Git clone\nwill produce unbuildable code, but JGit will produce a working one for unicode\naware systems and documentation, the case where unicode filenames are more common\nthan in source, will look good.\n\n-- robin\n\nPS. I readded the people you forgot to Cc\n"},{"id":"106966","messageId":"alpine.DEB.2.00.0903041149250.8926@perkele.intern.softwolves.pp.se","threadId":"18088","inReplyTo":"37fcd2780903030913q4ee0e5d0t45dc3b124285f748@mail.gmail.com","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-03-04T10:51:15Z","receivedAt":"2009-03-04T10:51:15Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Dmitry Potapov:\n\n> No, it does not, if you have wchar_t that is only 16-bit wide, because \n> characters outside of the BMP have integer values in Unicode greater than \n> 65535...\n\nUTF-16 allows you to reference all of Unicode (i.e up to U+10FFFF) using \nsurrogate pairs. That means that not all characters can be represented as a \nsingle wchar_t, that is true. The problem with changing wchar_t is that it \nwas defined to use 16-bit values at a time where Unicode was defined to use \n16-bit code points (but they soon figured out that was not enough).\n\nAnyway, this is getting off-topic. Please feel free reply in private.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"106978","messageId":"37fcd2780903040618q24457d96m7a9b181a3e202c7d@mail.gmail.com","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903041149250.8926@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-03-04T14:18:41Z","receivedAt":"2009-03-04T14:18:41Z","isPatch":true,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Wed, Mar 04, 2009 at 11:51:15AM +0100, Peter Krefting wrote:\n\n> The problem with changing wchar_t is that_\n> it was defined to use 16-bit values at a time where Unicode was defined_\n> to use 16-bit code points (but they soon figured out that was not_\n> enough).\n\nI do realize that is a problem, and unfortunately there is no easy and\nquick fix to it. But you brought Windows as an example of good Unicode\nsupport... Well, to my mind, it is not, at least, not for C programs.\nYou have two serious problems here:\n1. wchar_t is too small to hold all Unicode characters as it is required\n   by C standard.\n2. UTF-8 support is broken in C runtime library.\n\nIn fact, if UTF-8 were supported by C runtime, we would not have this thread\nin the first place... Now, it is possible to wrap all C functions used by Git to\nmake them work with UTF-8, but it is a lot of work...\n\nDmitry\n"},{"id":"107285","messageId":"200903071138.14683.robin.rosenberg.lists@dewire.com","threadId":"18088","inReplyTo":"alpine.DEB.2.00.0903020941120.17877@perkele.intern.softwolves.pp.se","subject":"Re: [RFC PATCH] Windows: Assume all file names to be UTF-8 encoded.","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-03-07T10:38:14Z","receivedAt":"2009-03-07T10:38:14Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"\nSlightly related; A new cygwin (not msysgit-related) version with UTF-8 support was announced. Most notably:\n\n- New setlocale implementation allows to specify POSIX locale strings.\n  You can now use, for instance in bash, `export LC_ALL=en_US.UTF-8'.\n  The language and territory will be ignored for now, the charset\n  will be used by multibyte-releated functions.\n\n- UTF-8 filenames are supported now. \n\n- Support UTF-8 in console window.\n\nThis certainly makes it more feasable to interoperate with *nix repos that has non-ascii metadata and file names.\n\n-- robin\n"}]}