# Git, Mac OS X and German special characters 9 messages from 2011-10-01 to 2011-10-03. Participants: Albert Zeyer, Andreas Ericsson, Andreas Krey, Michael Witten, Jakub Narebski, Torsten Bögershausen. Thread: https://gitlist.dev/t/28544 ## Albert Zeyer, 2011-10-01 12:44 Subject: Git, Mac OS X and German special characters Message-ID: URL: https://gitlist.dev/e/CAO1Q%2Bjf%3DRO%3DsE90-mQdi%2B%3DfUWH1RLM%2BJTubSgSVGC5uDyhU%2B2A%40mail.gmail.com ``` Hi, There are problems on MacOSX with different UTF8 encodings of filenames. A unicode string has multiple ways to be represented as UTF8 and Git treats them as different filenames. This is the actual bug. It should treat them all as the same filename. In some cases (as on MacOSX), the underlying operating system may use a normalized UTF8 representation in some sort, i.e. change the actual UTF8 filename representation. Similar problems also exists in SVN, for example. This was reported [here](http://subversion.tigris.org/issues/show_bug.cgi?id=2464). There you can find also lengthy discussions about the topic. And also [here](http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames). This was already reported for Git earlier and there is also a patch for Git [here](http://lists-archives.org/git/719832-git-mac-os-x-and-german-special-characters.html). I wonder about the state of this. This hasn't been applied yet. Why? Regards, Albert ``` ## Andreas Ericsson, 2011-10-01 13:39 Subject: Re: Git, Mac OS X and German special characters Message-ID: <4E87182C.8050201@op5.se> URL: https://gitlist.dev/e/4E87182C.8050201%40op5.se In-Reply-To: ``` On 10/01/2011 07:44 AM, Albert Zeyer wrote: > Hi, > > There are problems on MacOSX with different UTF8 encodings of > filenames. A unicode string has multiple ways to be represented as > UTF8 and Git treats them as different filenames. This is the actual > bug. It should treat them all as the same filename. In some cases (as > on MacOSX), the underlying operating system may use a normalized UTF8 > representation in some sort, i.e. change the actual UTF8 filename > representation. > > Similar problems also exists in SVN, for example. This was reported > [here](http://subversion.tigris.org/issues/show_bug.cgi?id=2464). > There you can find also lengthy discussions about the topic. And also > [here](http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames). > > This was already reported for Git earlier and there is also a patch > for Git [here](http://lists-archives.org/git/719832-git-mac-os-x-and-german-special-characters.html). > > I wonder about the state of this. This hasn't been applied yet. Why? > Because the patch didn't address repositories carrying files with more than one possible representation of the filename and that could have lead to silent loss of data for unsuspecting users. The real solution to your problem is, unfortunately, to either use a different and more competent filesystem, or to avoid triggering the bugs in the one you're currently using. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se Tel: +46 8-230225 Fax: +46 8-230231 Considering the successes of the wars on alcohol, poverty, drugs and terror, I think we should give some serious thought to declaring war on peace. ``` ## Andreas Ericsson, 2011-10-01 14:24 Subject: Re: Git, Mac OS X and German special characters Message-ID: <4E872288.10503@op5.se> URL: https://gitlist.dev/e/4E872288.10503%40op5.se In-Reply-To: ``` Please don't cull the list when replying. Reply-to-all is the standard on git@vger. On 10/01/2011 08:57 AM, Albert Zeyer wrote: > On Sat, Oct 1, 2011 at 3:39 PM, Andreas Ericsson wrote: >> On 10/01/2011 07:44 AM, Albert Zeyer wrote: >>> Hi, >>> >>> There are problems on MacOSX with different UTF8 encodings of >>> filenames. A unicode string has multiple ways to be represented as >>> UTF8 and Git treats them as different filenames. This is the actual >>> bug. It should treat them all as the same filename. In some cases (as >>> on MacOSX), the underlying operating system may use a normalized UTF8 >>> representation in some sort, i.e. change the actual UTF8 filename >>> representation. >>> >>> Similar problems also exists in SVN, for example. This was reported >>> [here](http://subversion.tigris.org/issues/show_bug.cgi?id=2464). >>> There you can find also lengthy discussions about the topic. And also >>> [here](http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames). >>> >>> This was already reported for Git earlier and there is also a patch >>> for Git [here](http://lists-archives.org/git/719832-git-mac-os-x-and-german-special-characters.html). >>> >>> I wonder about the state of this. This hasn't been applied yet. Why? >>> >> >> Because the patch didn't address repositories carrying files with >> more than one possible representation of the filename and that >> could have lead to silent loss of data for unsuspecting users. >> >> The real solution to your problem is, unfortunately, to either use >> a different and more competent filesystem, or to avoid triggering >> the bugs in the one you're currently using. > > Well, I think it is a bug in Git itself that it treats different UTF8 > representations of the same filename as different filenames. It > shouldn't have allowed such in the first place. > > But I see your point. I guess I will work myself on a patch here or > extend that one. The trouble is that they may represent two different files on a different filesystem. The Linux kernel repo has plenty of files that exist with both uppercase and lowercase characters, like so: SOMEFILE_driver.c somefile_driver.c This is perfectly valid on all sensible and case-sensitive filesystems, but breaks horribly on HFS. There are other, far more "interesting" cases when you involve special chars such as the german umlaut, or the swedish åäö characters. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se Tel: +46 8-230225 Fax: +46 8-230231 Considering the successes of the wars on alcohol, poverty, drugs and terror, I think we should give some serious thought to declaring war on peace. ``` ## Andreas Krey, 2011-10-01 19:47 Subject: Re: Git, Mac OS X and German special characters Message-ID: <20111001194746.GA16826@inner.h.iocl.org> URL: https://gitlist.dev/e/20111001194746.GA16826%40inner.h.iocl.org In-Reply-To: <4E872288.10503@op5.se> ``` On Sat, 01 Oct 2011 09:24:08 +0000, Andreas Ericsson wrote: ... > The trouble is that they may represent two different files on a > different filesystem. The Linux kernel repo has plenty of files > that exist with both uppercase and lowercase characters, like so: > SOMEFILE_driver.c > somefile_driver.c > > This is perfectly valid on all sensible and case-sensitive > filesystems, but breaks horribly on HFS. It also breaks on windows, except in at least one country[1]. And the latter alone is good reason why no VCS should try to forbid to use different characters that some filesystems (and only some) consider the same. > There are other, far more > "interesting" cases when you involve special chars such as the > german umlaut, or the swedish åäö characters. Care to share some? The question is, should git forbid two filenames that consist of the *same* characters, only differently uni-encoded? I don't think anyone would make two files named 'Büro', with different unicode encodings. But as far as I know that is a shady area. Andreas [1] Which has 'i with dot' and 'i without dot' both in uppercase and lowercase variant, so I and i are not the 'same'. ``` ## Michael Witten, 2011-10-01 22:02 Subject: Re: Git, Mac OS X and German special characters Message-ID: URL: https://gitlist.dev/e/CAMOZ1BuXiQkZG_7mvay-ybm7Q7niwXVhbmbCmkfy%3DwD1AKsasQ%40mail.gmail.com In-Reply-To: <20111001194746.GA16826@inner.h.iocl.org> ``` On Sat, Oct 1, 2011 at 19:47, Andreas Krey wrote: > The question is, should git forbid two filenames that consist > of the *same* characters, only differently uni-encoded? I don't > think anyone would make two files named 'Büro', with different > unicode encodings. But as far as I know that is a shady area. So, let's leave git's current behavior as the default and provide a config variable that when set, tells git to handle file names in terms of characters rather than bytes. ``` ## Jakub Narebski, 2011-10-01 23:14 Subject: Re: Git, Mac OS X and German special characters Message-ID: URL: https://gitlist.dev/e/m3hb3snw4b.fsf%40localhost.localdomain In-Reply-To: ``` Michael Witten writes: > On Sat, Oct 1, 2011 at 19:47, Andreas Krey wrote: > > > The question is, should git forbid two filenames that consist > > of the *same* characters, only differently uni-encoded? I don't > > think anyone would make two files named 'Büro', with different > > unicode encodings. But as far as I know that is a shady area. > > So, let's leave git's current behavior as the default and provide > a config variable that when set, tells git to handle file names > in terms of characters rather than bytes. You meant here _graphemes_, not Unicode codepoint when talking about characters, didn't you? IIRC the problem with MacOS X is that it accepts different composition when creating a file from what it returns when asking for contents of directory (NFD if I remember correctly, which is less used). There are some beginnings of sanely handling filesystem encoding in Git (the framework), but it is currently underutilized only to handle case-sensitivity and case-preserving. -- Jakub Narębski ``` ## Michael Witten, 2011-10-01 23:26 Subject: Re: Git, Mac OS X and German special characters Message-ID: URL: https://gitlist.dev/e/CAMOZ1BvcQtOzPR%3Da8NJ9QHXUMhfmmLWwbKMZqsmQTSE150TaXQ%40mail.gmail.com In-Reply-To: ``` 2011/10/1 Jakub Narebski : > Michael Witten writes: >> On Sat, Oct 1, 2011 at 19:47, Andreas Krey wrote: >> >> > The question is, should git forbid two filenames that consist >> > of the *same* characters, only differently uni-encoded? I don't >> > think anyone would make two files named 'Büro', with different >> > unicode encodings. But as far as I know that is a shady area. >> >> So, let's leave git's current behavior as the default and provide >> a config variable that when set, tells git to handle file names >> in terms of characters rather than bytes. > > You meant here _graphemes_, not Unicode codepoint when talking about > characters, didn't you? Yes. ``` ## Albert Zeyer, 2011-10-01 23:48 Subject: Re: Git, Mac OS X and German special characters Message-ID: URL: https://gitlist.dev/e/CAO1Q%2Bjec0Dm3FXEbcb_yEU5XPW_7V5sf%2BSn8SR0h9fGWJE%3D-%3Dw%40mail.gmail.com In-Reply-To: ``` On Sun, Oct 2, 2011 at 12:02 AM, Michael Witten wrote: > On Sat, Oct 1, 2011 at 19:47, Andreas Krey wrote: > >> The question is, should git forbid two filenames that consist >> of the *same* characters, only differently uni-encoded? I don't >> think anyone would make two files named 'Büro', with different >> unicode encodings. But as far as I know that is a shady area. > > So, let's leave git's current behavior as the default and provide > a config variable that when set, tells git to handle file names > in terms of characters rather than bytes. I just read the very lengthy discussion here: http://thread.gmane.org/gmane.comp.version-control.git/70688 Basically all the arguments have already been discussed. There are varios options. Most of them are not mutual exclusive, so it would also be an option to implement most of them and let the user pick what (s)he prefers. * TreatFilenamesAsText or however you would call it. I.e. handle filenames the same when they equal in Unicode. Linus is very much against this because in rare situations, it could destroy your data, like in this example: echo "foo" > Hütte # "Hütte" in NFC echo "bar" > Hütte # "Hütte" in NFD The second write would overwrite silently the file generated by the first write if those filenames would be handled the same. This (and such) behavior is to be avoided, claims Linus, because it would more often lead to not wanted behavior in third party applications. * On MacOSX, wrap all filesystem functions (like readdir()) to convert all filenames to NFC. MacOSX normalizes the UTF8 representation of the filenames to NFD but in most common situations (on most other systems), you end up with the filename being in NFC. As the filename is anyway normalized on OSX, it doesn't matter wether it is handled as NFC or NFD and NFC will likely generate less trouble. And this patch doesn't even really need an option. This was one suggestion by Linus itself: http://news.gmane.org/find-root.php?message_id=%3calpine.LFD.1.00.0801211323120.2957%40woody.linux%2dfoundation.org%3e * Disallow any files with filenames which are not in NFC at all. This makes some things a bit more safe (like on MacOSX; along with the previous suggestion) and more clear (you always know that your filename is in NFC). * Some more clever readdir() which, when it gets a filename which is not in the Git index but Unicode-equally to one filename in the Git index, automatically replaces it by the filename in the index. This is some sort of half way to a TreatFilenamesAsText option but should produce less trouble. This probably also doesn't need an extra option as it should very likely generate less trouble (on OSX at least; and for other systems which don't mangle the filename, they don't need to use this code at all). --- I will probably go and try to implement the clever-readdir(). And/or maybe also the NFC conversation in such a readdir() wrapper. ``` ## Torsten Bögershausen, 2011-10-03 19:48 Subject: Re: Git, Mac OS X and German special characters Message-ID: <4E8A1197.4050708@web.de> URL: https://gitlist.dev/e/4E8A1197.4050708%40web.de In-Reply-To: ``` The patch has probably not been applied, because it is not a god one. I have been working on a better version, but that is not 100% ready to be released. I can post it in a couple of days, (and yes, it does a NFD->NFC conversion in readdir() ) /Torsten On 10/01/2011 02:44 PM, Albert Zeyer wrote: > Hi, > > There are problems on MacOSX with different UTF8 encodings of > filenames. A unicode string has multiple ways to be represented as > UTF8 and Git treats them as different filenames. This is the actual > bug. It should treat them all as the same filename. In some cases (as > on MacOSX), the underlying operating system may use a normalized UTF8 > representation in some sort, i.e. change the actual UTF8 filename > representation. > > Similar problems also exists in SVN, for example. This was reported > [here](http://subversion.tigris.org/issues/show_bug.cgi?id=2464). > There you can find also lengthy discussions about the topic. And also > [here](http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames). > > This was already reported for Git earlier and there is also a patch > for Git [here](http://lists-archives.org/git/719832-git-mac-os-x-and-german-special-characters.html). > > I wonder about the state of this. This hasn't been applied yet. Why? > > Regards, > Albert > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ```