Re: OS X and umlauts in file names

From: B Smith-Mannschott <bsmith.occs@gmail.com>
Date: Nov 25, 2009, 09:51 UTC
Message-ID: <28c656e20911250151v7b89e5f8pa15aa2bd7d5f96d@mail.gmail.com>
In-Reply-To: <4B0CEFCA.5020605@syntevo.com>

On Wed, Nov 25, 2009 at 09:50, Thomas Singer <thomas.singer@syntevo.com> wrote:

Show 28 quoted lines

> I've did following:
>
>  toms-mac-mini:git-umlauts tom$ ls
>  Überlänge.txt
>  toms-mac-mini:git-umlauts tom$ git status
>  # On branch master
>  #
>  # Initial commit
>  #
>  # Changes to be committed:
>  #   (use "git rm --cached <file>..." to unstage)
>  #
>  #     new file:   "U\314\210berla\314\210nge.txt"
>  #
>  toms-mac-mini:git-umlauts tom$ git stage "U\314\210berla\314\210nge.txt"
>  fatal: pathspec 'U\314\210berla\314\210nge.txt' did not match any files
>
> Note, that I copy-pasted the file name which 'git status' showed to the
> stage command. IMHO, this should work, especially, because different people
> said Git would treat the file name as byte-array without interpreting it in
> some kind.
>
> From the user with the German OS X (for which the staging is said to work),
> I've got the output of 'env' and hence also tried
>
>  export LANG=de_DE.UTF-8
>
> before doing the above steps, but with the same results. :(

The problem you are having is not because of the *encoding*, it's the Normalization form that's messing things up. The fact is that in Unicode there are two ways to represent many -- but not all -- accented characters.

- "composed": one code point for the accented character)
- "decomposed": two code points: one for the base letter, one or more
combining characters for the accents.

The composed code points are really just backward compatibility to legacy encodings (like LATIN-1). If you want to actually support (rather than just tolerate) unicode you have to know how to deal with the decomposed form, and once you can do that there's little point beyond backward compatibility in continuing to use composed form internally.

The Subversion people have run into this same problem because they made the same error of assuming that any given sequence of glyphs has only one possible representation as unicode code points and thus only one representation as UTF-8 bytes. Dionisos has done written up the issues involved here:

http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames

// Ben