Git, Mac OS X and German special characters

Hi,

There are problems on MacOSX with different UTF8 encodings of filenames. A unicode string has multiple ways to be represented as UTF8 and Git treats them as different filenames. This is the actual bug. It should treat them all as the same filename. In some cases (as on MacOSX), the underlying operating system may use a normalized UTF8 representation in some sort, i.e. change the actual UTF8 filename representation.

Similar problems also exists in SVN, for example. This was reported [here](http://subversion.tigris.org/issues/show_bug.cgi?id=2464). There you can find also lengthy discussions about the topic. And also [here](http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames).

This was already reported for Git earlier and there is also a patch for Git [here](http://lists-archives.org/git/719832-git-mac-os-x-and-german-special-characters.html).

I wonder about the state of this. This hasn't been applied yet. Why?

Regards, Albert

On 10/01/2011 07:44 AM, Albert Zeyer wrote:

Because the patch didn't address repositories carrying files with more than one possible representation of the filename and that could have lead to silent loss of data for unsuspecting users.

The real solution to your problem is, unfortunately, to either use a different and more competent filesystem, or to avoid triggering the bugs in the one you're currently using.

The patch has probably not been applied, because it is not a god one. I have been working on a better version, but that is not 100% ready to be released.

I can post it in a couple of days, (and yes, it does a NFD->NFC conversion in readdir() ) /Torsten On 10/01/2011 02:44 PM, Albert Zeyer wrote:

Please don't cull the list when replying. Reply-to-all is the standard on git@vger.

On 10/01/2011 08:57 AM, Albert Zeyer wrote:

The trouble is that they may represent two different files on a different filesystem. The Linux kernel repo has plenty of files that exist with both uppercase and lowercase characters, like so: SOMEFILE_driver.c somefile_driver.c

This is perfectly valid on all sensible and case-sensitive filesystems, but breaks horribly on HFS. There are other, far more "interesting" cases when you involve special chars such as the german umlaut, or the swedish åäö characters.

On Sat, 01 Oct 2011 09:24:08 +0000, Andreas Ericsson wrote: ...

It also breaks on windows, except in at least one country[1]. And the latter alone is good reason why no VCS should try to forbid to use different characters that some filesystems (and only some) consider the same.

Care to share some?

The question is, should git forbid two filenames that consist of the *same* characters, only differently uni-encoded? I don't think anyone would make two files named 'Büro', with different unicode encodings. But as far as I know that is a shady area.

Andreas

[1] Which has 'i with dot' and 'i without dot' both in uppercase
    and lowercase variant, so I and i are not the 'same'.

On Sat, Oct 1, 2011 at 19:47, Andreas Krey <a.krey@gmx.de> wrote:

So, let's leave git's current behavior as the default and provide a config variable that when set, tells git to handle file names in terms of characters rather than bytes.

Michael Witten <mfwitten@gmail.com> writes:

You meant here _graphemes_, not Unicode codepoint when talking about characters, didn't you?

IIRC the problem with MacOS X is that it accepts different composition when creating a file from what it returns when asking for contents of directory (NFD if I remember correctly, which is less used).

There are some beginnings of sanely handling filesystem encoding in Git (the framework), but it is currently underutilized only to handle case-sensitivity and case-preserving.

2011/10/1 Jakub Narebski <jnareb@gmail.com>:

Yes.

On Sun, Oct 2, 2011 at 12:02 AM, Michael Witten <mfwitten@gmail.com> wrote:

I just read the very lengthy discussion here: http://thread.gmane.org/gmane.comp.version-control.git/70688

Basically all the arguments have already been discussed.

There are varios options. Most of them are not mutual exclusive, so it would also be an option to implement most of them and let the user pick what (s)he prefers.

* TreatFilenamesAsText or however you would call it. I.e. handle
filenames the same when they equal in Unicode.

Linus is very much against this because in rare situations, it could destroy your data, like in this example:

	echo "foo" > Hütte # "Hütte" in NFC
	echo "bar" > Hütte # "Hütte" in NFD

The second write would overwrite silently the file generated by the first write if those filenames would be handled the same. This (and such) behavior is to be avoided, claims Linus, because it would more often lead to not wanted behavior in third party applications.

* On MacOSX, wrap all filesystem functions (like readdir()) to convert
all filenames to NFC.

MacOSX normalizes the UTF8 representation of the filenames to NFD but in most common situations (on most other systems), you end up with the filename being in NFC.

As the filename is anyway normalized on OSX, it doesn't matter wether it is handled as NFC or NFD and NFC will likely generate less trouble. And this patch doesn't even really need an option.

This was one suggestion by Linus itself: http://news.gmane.org/find-root.php?message_id=%3calpine.LFD.1.00.0801211323120.2957%40woody.linux%2dfoundation.org%3e

* Disallow any files with filenames which are not in NFC at all. This
makes some things a bit more safe (like on MacOSX; along with the
previous suggestion) and more clear (you always know that your
filename is in NFC).

* Some more clever readdir() which, when it gets a filename which is
not in the Git index but Unicode-equally to one filename in the Git
index, automatically replaces it by the filename in the index.

This is some sort of half way to a TreatFilenamesAsText option but should produce less trouble.

This probably also doesn't need an extra option as it should very likely generate less trouble (on OSX at least; and for other systems which don't mangle the filename, they don't need to use this code at all).

---

I will probably go and try to implement the clever-readdir(). And/or maybe also the NFC conversation in such a readdir() wrapper.