# Weird filename encoding issue 10 messages from 2008-10-17 to 2008-10-27. Participants: Alexander Gladysh, Johannes Sixt, Jakub Narebski, Robin Rosenberg, Peter Krefting. Thread: https://gitlist.dev/t/15935 ## Alexander Gladysh, 2008-10-17 07:59 Subject: Weird filename encoding issue Message-ID: URL: https://gitlist.dev/e/c6c947f60810170059s3859d906r9935b5e80d8a41c6%40mail.gmail.com ``` Hi, all! OS X 10.5.5 (English locale), Git 1.5.5.1 Windows XP SP2 (Russian), msysGit 1.5.6.1.1071.g76fb ``` ## Alexander Gladysh, 2008-10-17 08:08 Subject: Re: Weird filename encoding issue Message-ID: URL: https://gitlist.dev/e/c6c947f60810170108q3038b66bj2ad89bdcb7105903%40mail.gmail.com In-Reply-To: ``` On Fri, Oct 17, 2008 at 11:59 AM, Alexander Gladysh wrote: > Hi, all! > > OS X 10.5.5 (English locale), Git 1.5.5.1 > Windows XP SP2 (Russian), msysGit 1.5.6.1.1071.g76fb My apologies, Gmail eaten part of my message. Sorry for such old Git versions, I can not upgrade right now. Perhaps it was fixed in 1.6.x? Steps to reproduce: 1. Create git repo on Windows machine, add there file, named "Копия.txt" (Russian for Copy.txt) in CP1251 (NOT UTF-8). Just in case, hopefully correctly named file is attached. File must be added and committed from git gui, not from command line. $ mkdir test && cd test $ git init $ git gui 2. Share the repo somewhere, so it would be accessible from OS X (repo should still be on Windows machine). 3. Clone repo to OS X machine and see the weird behaviour. Git status can't decide how file should be named, "%CA%EE%EF%E8%FF.txt" or "\312\356\357\350\377.txt". $ git clone /Volumes/Exchange/test/ Initialized empty Git repository in /Users/agladysh/test/.git/ $ cd test $ git status # On branch master # Untracked files: # (use "git add ..." to include in what will be committed) # # %CA%EE%EF%E8%FF.txt nothing added to commit but untracked files present (use "git add" to track) $ ls %CA%EE%EF%E8%FF.txt $ git clean -fd Removing %CA%EE%EF%E8%FF.txt $ git status # On branch master # Changed but not updated: # (use "git add/rm ..." to update what will be committed) # # deleted: "\312\356\357\350\377.txt" # no changes added to commit (use "git add" and/or "git commit -a") Alexander. ``` ## Johannes Sixt, 2008-10-17 08:55 Subject: Re: Weird filename encoding issue Message-ID: <48F852FC.2070506@viscovery.net> URL: https://gitlist.dev/e/48F852FC.2070506%40viscovery.net In-Reply-To: ``` Alexander Gladysh schrieb: > 3. Clone repo to OS X machine and see the weird behaviour. Git status > can't decide how file should be named, "%CA%EE%EF%E8%FF.txt" or > "\312\356\357\350\377.txt". It's not 'git status' that cannot decide how to name the file. It's OS X. > $ git status > # On branch master > # Untracked files: > # (use "git add ..." to include in what will be committed) > # > # %CA%EE%EF%E8%FF.txt > nothing added to commit but untracked files present (use "git add" to track) > $ ls > %CA%EE%EF%E8%FF.txt git asked OS X: Does the file "\312\356\357\350\377.txt" exist? and OS X said: "Yes". Because otherwise, you would have seen the file listed as "deleted:" in the 'git status' call above. But then git also requested a file listing from OS X in order to list the untracked files. And now OS X returned the name "%CA%EE%EF%E8%FF.txt". How could you expect git to tell that this is the same file when OS X cannot decide how to name it? The solution: Do not use file names with some local (let alone Windows specific) encoding if you have to use the files on OS X, too. -- Hannes ``` ## Jakub Narebski, 2008-10-17 08:57 Subject: Re: Weird filename encoding issue Message-ID: URL: https://gitlist.dev/e/m3hc7bk39g.fsf%40localhost.localdomain In-Reply-To: ``` "Alexander Gladysh" writes: > Steps to reproduce: > > 1. Create git repo on Windows machine, add there file, named > "Копия.txt" (Russian for Copy.txt) in CP1251 (NOT UTF-8). Just in > case, hopefully correctly named file is attached. > > File must be added and committed from git gui, not from command line. > > $ mkdir test && cd test > $ git init > > $ git gui > > 2. Share the repo somewhere, so it would be accessible from OS X (repo > should still be on Windows machine). > > 3. Clone repo to OS X machine and see the weird behaviour. Git status > can't decide how file should be named, "%CA%EE%EF%E8%FF.txt" or > "\312\356\357\350\377.txt". I _suspect_ that it is caused by known issue with MacOS X filesystem in default settings. Git assumes that what he puts in filesystem is the same that it reads from filesystem; there is ongoing (although bit stalled) effort to make Git work around this. Unfortunately HFS+ does not, I think, fullfil this requirement. -- Jakub Narebski Poland ShadeHawk on #git ``` ## Robin Rosenberg, 2008-10-18 02:58 Subject: Re: Weird filename encoding issue Message-ID: <200810180458.53853.robin.rosenberg@gmail.com> URL: https://gitlist.dev/e/200810180458.53853.robin.rosenberg%40gmail.com In-Reply-To: <48F852FC.2070506@viscovery.net> ``` fredagen den 17 oktober 2008 10.55.24 skrev Johannes Sixt: > > Alexander Gladysh schrieb: > > 3. Clone repo to OS X machine and see the weird behaviour. Git status > > can't decide how file should be named, "%CA%EE%EF%E8%FF.txt" or > > "\312\356\357\350\377.txt". > > It's not 'git status' that cannot decide how to name the file. It's OS X. > > > $ git status > > # On branch master > > # Untracked files: > > # (use "git add ..." to include in what will be committed) > > # > > # %CA%EE%EF%E8%FF.txt > > nothing added to commit but untracked files present (use "git add" to track) > > $ ls > > %CA%EE%EF%E8%FF.txt > > git asked OS X: > > Does the file "\312\356\357\350\377.txt" exist? > > and OS X said: "Yes". Because otherwise, you would have seen the file > listed as "deleted:" in the 'git status' call above. > > But then git also requested a file listing from OS X in order to list the > untracked files. And now OS X returned the name "%CA%EE%EF%E8%FF.txt". How > could you expect git to tell that this is the same file when OS X cannot > decide how to name it? > > The solution: Do not use file names with some local (let alone Windows > specific) encoding if you have to use the files on OS X, too. That seems to be OS-X's way of saying the is a string of garbage, lets handle it as such. Git (or msys) under windows is somewhat stupid here as it involves the eight-bit locale despite running in a unicode OS. To get UTF-8 on XP you can set the codepage to UTF-8 (called 65001 in windows). That might still cause problems with OS-X for some chararacters as it wants to "decompose" the UTF-8 encoding for accented characters, which perhaps is not a problem for you. I have no idea how well codepage 65001 plays with msys, but it would be nice it would work. I'm not sure how to switch codepage in msys, but if you set it from cmd and then start msys bash from there, I'm pretty sure it will pick it up. -- robin ``` ## Alexander Gladysh, 2008-10-18 06:22 Subject: Re: Weird filename encoding issue Message-ID: URL: https://gitlist.dev/e/c6c947f60810172322o12beeb80xfd39b551b0db7c99%40mail.gmail.com In-Reply-To: <48F852FC.2070506@viscovery.net> ``` Thanks for answers, guys. <...> > git asked OS X: > > Does the file "\312\356\357\350\377.txt" exist? > > and OS X said: "Yes". Because otherwise, you would have seen the file > listed as "deleted:" in the 'git status' call above. > > But then git also requested a file listing from OS X in order to list the > untracked files. And now OS X returned the name "%CA%EE%EF%E8%FF.txt". How > could you expect git to tell that this is the same file when OS X cannot > decide how to name it? > > The solution: Do not use file names with some local (let alone Windows > specific) encoding if you have to use the files on OS X, too. Thank you, but this is not the solution, this is a workaround. 1. Git hooks do not work under msysgit. Thus I'm not able to prevent people on Windows from committing weird filenames to the repo. (Please don't start elitist discussion that people using git must do everything conciously.) Note also, that even if hooks would work, such files still may appear in the repo in theory (somebody may disable hooks after all). 2. Now I have one such file. I'm managing my repo on OS X, and have no access to other machines right now. How can I create commit that renames the file? Git GUI manages to create commit that adds the renamed file, but does not delete old one. Alexander. ``` ## Robin Rosenberg, 2008-10-18 09:35 Subject: Re: [msysGit] Re: Weird filename encoding issue Message-ID: <200810181135.52270.robin.rosenberg@gmail.com> URL: https://gitlist.dev/e/200810181135.52270.robin.rosenberg%40gmail.com In-Reply-To: ``` lördagen den 18 oktober 2008 08.22.24 skrev Alexander Gladysh: > 2. Now I have one such file. I'm managing my repo on OS X, and have no > access to other machines right now. How can I create commit that > renames the file? Git GUI manages to create commit that adds the > renamed file, but does not delete old one. Rename the file and add it. Then git add -u to remove the old name and commit. -- tobin ``` ## Johannes Sixt, 2008-10-18 17:51 Subject: Re: Weird filename encoding issue Message-ID: <200810181951.53962.johannes.sixt@telecom.at> URL: https://gitlist.dev/e/200810181951.53962.johannes.sixt%40telecom.at In-Reply-To: ``` On Samstag, 18. Oktober 2008, Alexander Gladysh wrote: > 1. Git hooks do not work under msysgit. This is new to me. Why are you saying that? (IOW, please be more precise with your description. We can't help you if all you tell us is "Does not work" .) -- Hannes ``` ## Alexander Gladysh, 2008-10-19 14:11 Subject: Re: [msysGit] Re: Weird filename encoding issue Message-ID: URL: https://gitlist.dev/e/c6c947f60810190711je910155y31652395228fd163%40mail.gmail.com In-Reply-To: <200810181951.53962.johannes.sixt@telecom.at> ``` On Sat, Oct 18, 2008 at 9:51 PM, Johannes Sixt wrote: > On Samstag, 18. Oktober 2008, Alexander Gladysh wrote: >> 1. Git hooks do not work under msysgit. > > This is new to me. Why are you saying that? (IOW, please be more precise with > your description. We can't help you if all you tell us is "Does not work" .) Err. Sorry. I thought it is a known issue: http://groups.google.com/group/msysgit/browse_thread/thread/956e555b8d515c66 I've got an impression that msysgit 1.5.6.1.1071.g76fb we use does not contain this fix. Of course, I may apply it by hand on each user's machine... Still have to do it though. Alexander. ``` ## Peter Krefting, 2008-10-27 08:08 Subject: Re: [msysGit] Re: Weird filename encoding issue Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0810270904490.18631%40ds9.cixit.se In-Reply-To: <200810180458.53853.robin.rosenberg@gmail.com> ``` Robin Rosenberg: > Git (or msys) under windows is somewhat stupid here as it involves > the eight-bit locale despite running in a unicode OS. To get UTF-8 on > XP you can set the codepage to UTF-8 (called 65001 in windows). That presents itself with a lot of other issues, however, as the char-based file APIs were only designed to handle up to double-byte characters. To implement proper Unicode file name support on Windows, one need to use the "wide" APIs (using wchar_t). Using those APIs, it does not matter what locale the system is set to. Everything is then UTF-16, which is trivially converted to and from UTF-8 (and there are even native APIs to do it if one can't be bothered to write code for it). I have been meaning to check out a copy of the Msysgit source tree and see how difficult it would be to replace the file API layer with proper Unicode support, but time keeps fleeing from me, so I never seem to get around to :-/ -- \\// Peter - http://www.softwolves.pp.se/ ```