Weird filename encoding issue

Hi, all!

OS X 10.5.5 (English locale), Git 1.5.5.1 Windows XP SP2 (Russian), msysGit 1.5.6.1.1071.g76fb

On Fri, Oct 17, 2008 at 11:59 AM, Alexander Gladysh <agladysh@gmail.com> wrote:

My apologies, Gmail eaten part of my message.

Sorry for such old Git versions, I can not upgrade right now. Perhaps it was fixed in 1.6.x?

Steps to reproduce:

1. Create git repo on Windows machine, add there file, named
"Копия.txt" (Russian for Copy.txt) in CP1251 (NOT UTF-8). Just in
case, hopefully correctly named file is attached.

File must be added and committed from git gui, not from command line.

$ mkdir test && cd test $ git init <put file there> $ git gui

2. Share the repo somewhere, so it would be accessible from OS X (repo
should still be on Windows machine).

3. Clone repo to OS X machine and see the weird behaviour. Git status
can't decide how file should be named, "%CA%EE%EF%E8%FF.txt" or
"\312\356\357\350\377.txt".

$ git clone /Volumes/Exchange/test/ Initialized empty Git repository in /Users/agladysh/test/.git/ $ cd test $ git status # On branch master # Untracked files: # (use "git add <file>..." to include in what will be committed) # # %CA%EE%EF%E8%FF.txt nothing added to commit but untracked files present (use "git add" to track) $ ls %CA%EE%EF%E8%FF.txt $ git clean -fd Removing %CA%EE%EF%E8%FF.txt $ git status # On branch master # Changed but not updated: # (use "git add/rm <file>..." to update what will be committed) # # deleted: "\312\356\357\350\377.txt" # no changes added to commit (use "git add" and/or "git commit -a")

Alexander.

Alexander Gladysh schrieb:

It's not 'git status' that cannot decide how to name the file. It's OS X.

git asked OS X:

   Does the file "\312\356\357\350\377.txt" exist?

and OS X said: "Yes". Because otherwise, you would have seen the file listed as "deleted:" in the 'git status' call above.

But then git also requested a file listing from OS X in order to list the untracked files. And now OS X returned the name "%CA%EE%EF%E8%FF.txt". How could you expect git to tell that this is the same file when OS X cannot decide how to name it?

The solution: Do not use file names with some local (let alone Windows specific) encoding if you have to use the files on OS X, too.

-- Hannes

fredagen den 17 oktober 2008 10.55.24 skrev Johannes Sixt:

That seems to be OS-X's way of saying the is a string of garbage, lets handle it as such.

Git (or msys) under windows is somewhat stupid here as it involves the eight-bit locale despite running in a unicode OS. To get UTF-8 on XP you can set the codepage to UTF-8 (called 65001 in windows). That might still cause problems with OS-X for some chararacters as it wants to "decompose" the UTF-8 encoding for accented characters, which perhaps is not a problem for you.

I have no idea how well codepage 65001 plays with msys, but it would be nice it would work. I'm not sure how to switch codepage in msys, but if you set it from cmd and then start msys bash from there, I'm pretty sure it will pick it up.

-- robin

Robin Rosenberg:

That presents itself with a lot of other issues, however, as the char-based file APIs were only designed to handle up to double-byte characters. To implement proper Unicode file name support on Windows, one need to use the "wide" APIs (using wchar_t).

Using those APIs, it does not matter what locale the system is set to. Everything is then UTF-16, which is trivially converted to and from UTF-8 (and there are even native APIs to do it if one can't be bothered to write code for it).

I have been meaning to check out a copy of the Msysgit source tree and see how difficult it would be to replace the file API layer with proper Unicode support, but time keeps fleeing from me, so I never seem to get around to :-/

Thanks for answers, guys.

<...>

Thank you, but this is not the solution, this is a workaround.

1. Git hooks do not work under msysgit. Thus I'm not able to prevent
people on Windows from committing weird filenames to the repo. (Please
don't start elitist discussion that people using git must do
everything conciously.)

Note also, that even if hooks would work, such files still may appear in the repo in theory (somebody may disable hooks after all).

2. Now I have one such file. I'm managing my repo on OS X, and have no
access to other machines right now. How can I create commit that
renames the file? Git GUI manages to create commit that adds the
renamed file, but does not delete old one.

Alexander.

lördagen den 18 oktober 2008 08.22.24 skrev Alexander Gladysh:

Rename the file and add it. Then git add -u to remove the old name and commit.

-- tobin

On Samstag, 18. Oktober 2008, Alexander Gladysh wrote:

This is new to me. Why are you saying that? (IOW, please be more precise with your description. We can't help you if all you tell us is "Does not work" .)

-- Hannes

On Sat, Oct 18, 2008 at 9:51 PM, Johannes Sixt <johannes.sixt@telecom.at> wrote:

Err. Sorry. I thought it is a known issue:

http://groups.google.com/group/msysgit/browse_thread/thread/956e555b8d515c66

I've got an impression that msysgit 1.5.6.1.1071.g76fb we use does not contain this fix.

Of course, I may apply it by hand on each user's machine... Still have to do it though.

Alexander.

"Alexander Gladysh" <agladysh@gmail.com> writes:

I _suspect_ that it is caused by known issue with MacOS X filesystem in default settings. Git assumes that what he puts in filesystem is the same that it reads from filesystem; there is ongoing (although bit stalled) effort to make Git work around this. Unfortunately HFS+ does not, I think, fullfil this requirement.