{"thread":{"id":"15935","subject":"Weird filename encoding issue","startedAt":"2008-10-17T07:59:20Z","lastAt":"2008-10-27T08:08:39Z","messageCount":10,"participants":["Alexander Gladysh","Johannes Sixt","Jakub Narebski","Robin Rosenberg","Peter Krefting"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"93274","messageId":"c6c947f60810170059s3859d906r9935b5e80d8a41c6@mail.gmail.com","threadId":"15935","inReplyTo":null,"subject":"Weird filename encoding issue","fromName":"Alexander Gladysh","fromEmail":"agladysh@gmail.com","sentAt":"2008-10-17T07:59:20Z","receivedAt":"2008-10-17T07:59:20Z","isPatch":false,"sender":{"key":"agladysh@gmail.com","avatar":"https://avatars.githubusercontent.com/u/38239?v=4"},"body":"Hi, all!\n\nOS X 10.5.5 (English locale), Git 1.5.5.1\nWindows XP SP2 (Russian), msysGit 1.5.6.1.1071.g76fb\n"},{"id":"93275","messageId":"c6c947f60810170108q3038b66bj2ad89bdcb7105903@mail.gmail.com","threadId":"15935","inReplyTo":"c6c947f60810170059s3859d906r9935b5e80d8a41c6@mail.gmail.com","subject":"Re: Weird filename encoding issue","fromName":"Alexander Gladysh","fromEmail":"agladysh@gmail.com","sentAt":"2008-10-17T08:08:04Z","receivedAt":"2008-10-17T08:08:04Z","isPatch":false,"sender":{"key":"agladysh@gmail.com","avatar":"https://avatars.githubusercontent.com/u/38239?v=4"},"body":"On Fri, Oct 17, 2008 at 11:59 AM, Alexander Gladysh <agladysh@gmail.com> wrote:\n> Hi, all!\n>\n> OS X 10.5.5 (English locale), Git 1.5.5.1\n> Windows XP SP2 (Russian), msysGit 1.5.6.1.1071.g76fb\n\nMy apologies, Gmail eaten part of my message.\n\nSorry for such old Git versions, I can not upgrade right now. Perhaps\nit was fixed in 1.6.x?\n\nSteps to reproduce:\n\n1. Create git repo on Windows machine, add there file, named\n\"Копия.txt\" (Russian for Copy.txt) in CP1251 (NOT UTF-8). Just in\ncase, hopefully correctly named file is attached.\n\nFile must be added and committed from git gui, not from command line.\n\n$ mkdir test && cd test\n$ git init\n<put file there>\n$ git gui\n\n2. Share the repo somewhere, so it would be accessible from OS X (repo\nshould still be on Windows machine).\n\n3. Clone repo to OS X machine and see the weird behaviour. Git status\ncan't decide how file should be named, \"%CA%EE%EF%E8%FF.txt\" or\n\"\\312\\356\\357\\350\\377.txt\".\n\n$ git clone /Volumes/Exchange/test/\nInitialized empty Git repository in /Users/agladysh/test/.git/\n$ cd test\n$ git status\n# On branch master\n# Untracked files:\n#   (use \"git add <file>...\" to include in what will be committed)\n#\n#\t%CA%EE%EF%E8%FF.txt\nnothing added to commit but untracked files present (use \"git add\" to track)\n$ ls\n%CA%EE%EF%E8%FF.txt\n$ git clean -fd\nRemoving %CA%EE%EF%E8%FF.txt\n$ git status\n# On branch master\n# Changed but not updated:\n#   (use \"git add/rm <file>...\" to update what will be committed)\n#\n#\tdeleted:    \"\\312\\356\\357\\350\\377.txt\"\n#\nno changes added to commit (use \"git add\" and/or \"git commit -a\")\n\nAlexander.\n"},{"id":"93278","messageId":"48F852FC.2070506@viscovery.net","threadId":"15935","inReplyTo":"c6c947f60810170108q3038b66bj2ad89bdcb7105903@mail.gmail.com","subject":"Re: Weird filename encoding issue","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-10-17T08:55:24Z","receivedAt":"2008-10-17T08:55:24Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Alexander Gladysh schrieb:\n> 3. Clone repo to OS X machine and see the weird behaviour. Git status\n> can't decide how file should be named, \"%CA%EE%EF%E8%FF.txt\" or\n> \"\\312\\356\\357\\350\\377.txt\".\n\nIt's not 'git status' that cannot decide how to name the file. It's OS X.\n\n> $ git status\n> # On branch master\n> # Untracked files:\n> #   (use \"git add <file>...\" to include in what will be committed)\n> #\n> #\t%CA%EE%EF%E8%FF.txt\n> nothing added to commit but untracked files present (use \"git add\" to track)\n> $ ls\n> %CA%EE%EF%E8%FF.txt\n\ngit asked OS X:\n\n   Does the file \"\\312\\356\\357\\350\\377.txt\" exist?\n\nand OS X said: \"Yes\". Because otherwise, you would have seen the file\nlisted as \"deleted:\" in the 'git status' call above.\n\nBut then git also requested a file listing from OS X in order to list the\nuntracked files. And now OS X returned the name \"%CA%EE%EF%E8%FF.txt\". How\ncould you expect git to tell that this is the same file when OS X cannot\ndecide how to name it?\n\nThe solution: Do not use file names with some local (let alone Windows\nspecific) encoding if you have to use the files on OS X, too.\n\n-- Hannes\n"},{"id":"93279","messageId":"m3hc7bk39g.fsf@localhost.localdomain","threadId":"15935","inReplyTo":"c6c947f60810170108q3038b66bj2ad89bdcb7105903@mail.gmail.com","subject":"Re: Weird filename encoding issue","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-10-17T08:57:36Z","receivedAt":"2008-10-17T08:57:36Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Alexander Gladysh\" <agladysh@gmail.com> writes:\n\n> Steps to reproduce:\n> \n> 1. Create git repo on Windows machine, add there file, named\n> \"Копия.txt\" (Russian for Copy.txt) in CP1251 (NOT UTF-8). Just in\n> case, hopefully correctly named file is attached.\n> \n> File must be added and committed from git gui, not from command line.\n> \n> $ mkdir test && cd test\n> $ git init\n> <put file there>\n> $ git gui\n> \n> 2. Share the repo somewhere, so it would be accessible from OS X (repo\n> should still be on Windows machine).\n> \n> 3. Clone repo to OS X machine and see the weird behaviour. Git status\n> can't decide how file should be named, \"%CA%EE%EF%E8%FF.txt\" or\n> \"\\312\\356\\357\\350\\377.txt\".\n\nI _suspect_ that it is caused by known issue with MacOS X filesystem\nin default settings.  Git assumes that what he puts in filesystem is\nthe same that it reads from filesystem; there is ongoing (although\nbit stalled) effort to make Git work around this.  Unfortunately HFS+\ndoes not, I think, fullfil this requirement.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"93336","messageId":"200810180458.53853.robin.rosenberg@gmail.com","threadId":"15935","inReplyTo":"48F852FC.2070506@viscovery.net","subject":"Re: Weird filename encoding issue","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@gmail.com","sentAt":"2008-10-18T02:58:53Z","receivedAt":"2008-10-18T02:58:53Z","isPatch":false,"sender":{"key":"robin.rosenberg@gmail.com","avatar":null},"body":"\nfredagen den 17 oktober 2008 10.55.24 skrev Johannes Sixt:\n> \n> Alexander Gladysh schrieb:\n> > 3. Clone repo to OS X machine and see the weird behaviour. Git status\n> > can't decide how file should be named, \"%CA%EE%EF%E8%FF.txt\" or\n> > \"\\312\\356\\357\\350\\377.txt\".\n> \n> It's not 'git status' that cannot decide how to name the file. It's OS X.\n> \n> > $ git status\n> > # On branch master\n> > # Untracked files:\n> > #   (use \"git add <file>...\" to include in what will be committed)\n> > #\n> > #\t%CA%EE%EF%E8%FF.txt\n> > nothing added to commit but untracked files present (use \"git add\" to track)\n> > $ ls\n> > %CA%EE%EF%E8%FF.txt\n> \n> git asked OS X:\n> \n>    Does the file \"\\312\\356\\357\\350\\377.txt\" exist?\n> \n> and OS X said: \"Yes\". Because otherwise, you would have seen the file\n> listed as \"deleted:\" in the 'git status' call above.\n> \n> But then git also requested a file listing from OS X in order to list the\n> untracked files. And now OS X returned the name \"%CA%EE%EF%E8%FF.txt\". How\n> could you expect git to tell that this is the same file when OS X cannot\n> decide how to name it?\n> \n> The solution: Do not use file names with some local (let alone Windows\n> specific) encoding if you have to use the files on OS X, too.\n\nThat seems to be OS-X's way of saying the is a string of garbage, lets handle it as such.\n\nGit (or msys) under windows is somewhat stupid here as it involves the eight-bit locale\ndespite running in a unicode OS. To get UTF-8 on XP you can set the codepage to UTF-8 \n(called 65001 in windows). That might still cause problems with OS-X for some chararacters\nas it wants to \"decompose\" the UTF-8 encoding for accented characters, which perhaps is\nnot a problem for you.\n\nI have no idea how well codepage 65001 plays with msys, but it would be nice it would work.\nI'm not sure how to switch codepage in msys, but if you set it from cmd and then start msys\nbash from there, I'm pretty sure it will pick it up.\n\n-- robin\n"},{"id":"93341","messageId":"c6c947f60810172322o12beeb80xfd39b551b0db7c99@mail.gmail.com","threadId":"15935","inReplyTo":"48F852FC.2070506@viscovery.net","subject":"Re: Weird filename encoding issue","fromName":"Alexander Gladysh","fromEmail":"agladysh@gmail.com","sentAt":"2008-10-18T06:22:24Z","receivedAt":"2008-10-18T06:22:24Z","isPatch":false,"sender":{"key":"agladysh@gmail.com","avatar":"https://avatars.githubusercontent.com/u/38239?v=4"},"body":"\nThanks for answers, guys.\n\n<...>\n\n> git asked OS X:\n>\n>   Does the file \"\\312\\356\\357\\350\\377.txt\" exist?\n>\n> and OS X said: \"Yes\". Because otherwise, you would have seen the file\n> listed as \"deleted:\" in the 'git status' call above.\n>\n> But then git also requested a file listing from OS X in order to list the\n> untracked files. And now OS X returned the name \"%CA%EE%EF%E8%FF.txt\". How\n> could you expect git to tell that this is the same file when OS X cannot\n> decide how to name it?\n>\n> The solution: Do not use file names with some local (let alone Windows\n> specific) encoding if you have to use the files on OS X, too.\n\nThank you, but this is not the solution, this is a workaround.\n\n1. Git hooks do not work under msysgit. Thus I'm not able to prevent\npeople on Windows from committing weird filenames to the repo. (Please\ndon't start elitist discussion that people using git must do\neverything conciously.)\n\nNote also, that even if hooks would work, such files still may appear\nin the repo in theory (somebody may disable hooks after all).\n\n2. Now I have one such file. I'm managing my repo on OS X, and have no\naccess to other machines right now. How can I create commit that\nrenames the file? Git GUI manages to create commit that adds the\nrenamed file, but does not delete old one.\n\nAlexander.\n"},{"id":"93352","messageId":"200810181135.52270.robin.rosenberg@gmail.com","threadId":"15935","inReplyTo":"c6c947f60810172322o12beeb80xfd39b551b0db7c99@mail.gmail.com","subject":"Re: [msysGit] Re: Weird filename encoding issue","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@gmail.com","sentAt":"2008-10-18T09:35:52Z","receivedAt":"2008-10-18T09:35:52Z","isPatch":false,"sender":{"key":"robin.rosenberg@gmail.com","avatar":null},"body":"lördagen den 18 oktober 2008 08.22.24 skrev Alexander Gladysh:\n> 2. Now I have one such file. I'm managing my repo on OS X, and have no\n> access to other machines right now. How can I create commit that\n> renames the file? Git GUI manages to create commit that adds the\n> renamed file, but does not delete old one.\n\nRename the file and add it. Then git add -u to remove the old name and commit.\n\n-- tobin\n"},{"id":"93379","messageId":"200810181951.53962.johannes.sixt@telecom.at","threadId":"15935","inReplyTo":"c6c947f60810172322o12beeb80xfd39b551b0db7c99@mail.gmail.com","subject":"Re: Weird filename encoding issue","fromName":"Johannes Sixt","fromEmail":"johannes.sixt@telecom.at","sentAt":"2008-10-18T17:51:53Z","receivedAt":"2008-10-18T17:51:53Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"\nOn Samstag, 18. Oktober 2008, Alexander Gladysh wrote:\n> 1. Git hooks do not work under msysgit.\n\nThis is new to me. Why are you saying that? (IOW, please be more precise with \nyour description. We can't help you if all you tell us is \"Does not work\" .)\n\n-- Hannes\n"},{"id":"93427","messageId":"c6c947f60810190711je910155y31652395228fd163@mail.gmail.com","threadId":"15935","inReplyTo":"200810181951.53962.johannes.sixt@telecom.at","subject":"Re: [msysGit] Re: Weird filename encoding issue","fromName":"Alexander Gladysh","fromEmail":"agladysh@gmail.com","sentAt":"2008-10-19T14:11:20Z","receivedAt":"2008-10-19T14:11:20Z","isPatch":false,"sender":{"key":"agladysh@gmail.com","avatar":"https://avatars.githubusercontent.com/u/38239?v=4"},"body":"On Sat, Oct 18, 2008 at 9:51 PM, Johannes Sixt <johannes.sixt@telecom.at> wrote:\n> On Samstag, 18. Oktober 2008, Alexander Gladysh wrote:\n>> 1. Git hooks do not work under msysgit.\n>\n> This is new to me. Why are you saying that? (IOW, please be more precise with\n> your description. We can't help you if all you tell us is \"Does not work\" .)\n\nErr. Sorry. I thought it is a known issue:\n\nhttp://groups.google.com/group/msysgit/browse_thread/thread/956e555b8d515c66\n\nI've got an impression that msysgit 1.5.6.1.1071.g76fb we use does not\ncontain this fix.\n\nOf course, I may apply it by hand on each user's machine... Still have\nto do it though.\n\nAlexander.\n"},{"id":"94030","messageId":"Pine.LNX.4.64.0810270904490.18631@ds9.cixit.se","threadId":"15935","inReplyTo":"200810180458.53853.robin.rosenberg@gmail.com","subject":"Re: [msysGit] Re: Weird filename encoding issue","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2008-10-27T08:08:39Z","receivedAt":"2008-10-27T08:08:39Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Robin Rosenberg:\n\n> Git (or msys) under windows is somewhat stupid here as it involves\n> the eight-bit locale despite running in a unicode OS. To get UTF-8 on\n> XP you can set the codepage to UTF-8 (called 65001 in windows).\n\nThat presents itself with a lot of other issues, however, as the\nchar-based file APIs were only designed to handle up to double-byte\ncharacters. To implement proper Unicode file name support on Windows,\none need to use the \"wide\" APIs (using wchar_t).\n\nUsing those APIs, it does not matter what locale the system is set to.\nEverything is then UTF-16, which is trivially converted to and from\nUTF-8 (and there are even native APIs to do it if one can't be bothered\nto write code for it).\n\nI have been meaning to check out a copy of the Msysgit source tree and\nsee how difficult it would be to replace the file API layer with proper\nUnicode support, but time keeps fleeing from me, so I never seem to get\naround to :-/\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"}]}