{"thread":{"id":"28544","subject":"Git, Mac OS X and German special characters","startedAt":"2011-10-01T12:44:14Z","lastAt":"2011-10-03T19:48:39Z","messageCount":9,"participants":["Albert Zeyer","Andreas Ericsson","Andreas Krey","Michael Witten","Jakub Narebski","Torsten Bögershausen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"176633","messageId":"CAO1Q+jf=RO=sE90-mQdi+=fUWH1RLM+JTubSgSVGC5uDyhU+2A@mail.gmail.com","threadId":"28544","inReplyTo":null,"subject":"Git, Mac OS X and German special characters","fromName":"Albert Zeyer","fromEmail":"albzey@googlemail.com","sentAt":"2011-10-01T12:44:14Z","receivedAt":"2011-10-01T12:44:14Z","isPatch":false,"sender":{"key":"albzey@googlemail.com","avatar":null},"body":"Hi,\n\nThere are problems on MacOSX with different UTF8 encodings of\nfilenames. A unicode string has multiple ways to be represented as\nUTF8 and Git treats them as different filenames. This is the actual\nbug. It should treat them all as the same filename. In some cases (as\non MacOSX), the underlying operating system may use a normalized UTF8\nrepresentation in some sort, i.e. change the actual UTF8 filename\nrepresentation.\n\nSimilar problems also exists in SVN, for example. This was reported\n[here](http://subversion.tigris.org/issues/show_bug.cgi?id=2464).\nThere you can find also lengthy discussions about the topic. And also\n[here](http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames).\n\nThis was already reported for Git earlier and there is also a patch\nfor Git [here](http://lists-archives.org/git/719832-git-mac-os-x-and-german-special-characters.html).\n\nI wonder about the state of this. This hasn't been applied yet. Why?\n\nRegards,\nAlbert\n"},{"id":"176635","messageId":"4E87182C.8050201@op5.se","threadId":"28544","inReplyTo":"CAO1Q+jf=RO=sE90-mQdi+=fUWH1RLM+JTubSgSVGC5uDyhU+2A@mail.gmail.com","subject":"Re: Git, Mac OS X and German special characters","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2011-10-01T13:39:56Z","receivedAt":"2011-10-01T13:39:56Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"On 10/01/2011 07:44 AM, Albert Zeyer wrote:\n> Hi,\n> \n> There are problems on MacOSX with different UTF8 encodings of\n> filenames. A unicode string has multiple ways to be represented as\n> UTF8 and Git treats them as different filenames. This is the actual\n> bug. It should treat them all as the same filename. In some cases (as\n> on MacOSX), the underlying operating system may use a normalized UTF8\n> representation in some sort, i.e. change the actual UTF8 filename\n> representation.\n> \n> Similar problems also exists in SVN, for example. This was reported\n> [here](http://subversion.tigris.org/issues/show_bug.cgi?id=2464).\n> There you can find also lengthy discussions about the topic. And also\n> [here](http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames).\n> \n> This was already reported for Git earlier and there is also a patch\n> for Git [here](http://lists-archives.org/git/719832-git-mac-os-x-and-german-special-characters.html).\n> \n> I wonder about the state of this. This hasn't been applied yet. Why?\n> \n\nBecause the patch didn't address repositories carrying files with\nmore than one possible representation of the filename and that\ncould have lead to silent loss of data for unsuspecting users.\n\nThe real solution to your problem is, unfortunately, to either use\na different and more competent filesystem, or to avoid triggering\nthe bugs in the one you're currently using.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"176637","messageId":"4E872288.10503@op5.se","threadId":"28544","inReplyTo":"CAO1Q+jeLEp2ReNc9eOFoJxdGq6oRE3b+O=JvMNU0Kqx_eAX=7w@mail.gmail.com","subject":"Re: Git, Mac OS X and German special characters","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2011-10-01T14:24:08Z","receivedAt":"2011-10-01T14:24:08Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Please don't cull the list when replying. Reply-to-all is the\nstandard on git@vger.\n\nOn 10/01/2011 08:57 AM, Albert Zeyer wrote:\n> On Sat, Oct 1, 2011 at 3:39 PM, Andreas Ericsson wrote:\n>> On 10/01/2011 07:44 AM, Albert Zeyer wrote:\n>>> Hi,\n>>>\n>>> There are problems on MacOSX with different UTF8 encodings of\n>>> filenames. A unicode string has multiple ways to be represented as\n>>> UTF8 and Git treats them as different filenames. This is the actual\n>>> bug. It should treat them all as the same filename. In some cases (as\n>>> on MacOSX), the underlying operating system may use a normalized UTF8\n>>> representation in some sort, i.e. change the actual UTF8 filename\n>>> representation.\n>>>\n>>> Similar problems also exists in SVN, for example. This was reported\n>>> [here](http://subversion.tigris.org/issues/show_bug.cgi?id=2464).\n>>> There you can find also lengthy discussions about the topic. And also\n>>> [here](http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames).\n>>>\n>>> This was already reported for Git earlier and there is also a patch\n>>> for Git [here](http://lists-archives.org/git/719832-git-mac-os-x-and-german-special-characters.html).\n>>>\n>>> I wonder about the state of this. This hasn't been applied yet. Why?\n>>>\n>>\n>> Because the patch didn't address repositories carrying files with\n>> more than one possible representation of the filename and that\n>> could have lead to silent loss of data for unsuspecting users.\n>>\n>> The real solution to your problem is, unfortunately, to either use\n>> a different and more competent filesystem, or to avoid triggering\n>> the bugs in the one you're currently using.\n> \n> Well, I think it is a bug in Git itself that it treats different UTF8\n> representations of the same filename as different filenames. It\n> shouldn't have allowed such in the first place.\n> \n> But I see your point. I guess I will work myself on a patch here or\n> extend that one.\n\n\nThe trouble is that they may represent two different files on a\ndifferent filesystem. The Linux kernel repo has plenty of files\nthat exist with both uppercase and lowercase characters, like so:\nSOMEFILE_driver.c\nsomefile_driver.c\n\nThis is perfectly valid on all sensible and case-sensitive\nfilesystems, but breaks horribly on HFS. There are other, far more\n\"interesting\" cases when you involve special chars such as the\ngerman umlaut, or the swedish åäö characters.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"176655","messageId":"20111001194746.GA16826@inner.h.iocl.org","threadId":"28544","inReplyTo":"4E872288.10503@op5.se","subject":"Re: Git, Mac OS X and German special characters","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2011-10-01T19:47:46Z","receivedAt":"2011-10-01T19:47:46Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Sat, 01 Oct 2011 09:24:08 +0000, Andreas Ericsson wrote:\n...\n> The trouble is that they may represent two different files on a\n> different filesystem. The Linux kernel repo has plenty of files\n> that exist with both uppercase and lowercase characters, like so:\n> SOMEFILE_driver.c\n> somefile_driver.c\n> \n> This is perfectly valid on all sensible and case-sensitive\n> filesystems, but breaks horribly on HFS.\n\nIt also breaks on windows, except in at least one country[1].\nAnd the latter alone is good reason why no VCS should try to\nforbid to use different characters that some filesystems\n(and only some) consider the same.\n\n> There are other, far more\n> \"interesting\" cases when you involve special chars such as the\n> german umlaut, or the swedish åäö characters.\n\nCare to share some?\n\nThe question is, should git forbid two filenames that consist\nof the *same* characters, only differently uni-encoded? I don't\nthink anyone would make two files named 'Büro', with different\nunicode encodings. But as far as I know that is a shady area.\n\nAndreas\n\n[1] Which has 'i with dot' and 'i without dot' both in uppercase\n and lowercase variant, so I and i are not the 'same'.\n"},{"id":"176662","messageId":"CAMOZ1BuXiQkZG_7mvay-ybm7Q7niwXVhbmbCmkfy=wD1AKsasQ@mail.gmail.com","threadId":"28544","inReplyTo":"20111001194746.GA16826@inner.h.iocl.org","subject":"Re: Git, Mac OS X and German special characters","fromName":"Michael Witten","fromEmail":"mfwitten@gmail.com","sentAt":"2011-10-01T22:02:40Z","receivedAt":"2011-10-01T22:02:40Z","isPatch":false,"sender":{"key":"mfwitten@gmail.com","avatar":"https://avatars.githubusercontent.com/u/597101?v=4"},"body":"On Sat, Oct 1, 2011 at 19:47, Andreas Krey wrote:\n\n> The question is, should git forbid two filenames that consist\n> of the *same* characters, only differently uni-encoded? I don't\n> think anyone would make two files named 'Büro', with different\n> unicode encodings. But as far as I know that is a shady area.\n\nSo, let's leave git's current behavior as the default and provide\na config variable that when set, tells git to handle file names\nin terms of characters rather than bytes.\n"},{"id":"176664","messageId":"m3hb3snw4b.fsf@localhost.localdomain","threadId":"28544","inReplyTo":"CAMOZ1BuXiQkZG_7mvay-ybm7Q7niwXVhbmbCmkfy=wD1AKsasQ@mail.gmail.com","subject":"Re: Git, Mac OS X and German special characters","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-10-01T23:14:45Z","receivedAt":"2011-10-01T23:14:45Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Michael Witten writes:\n> On Sat, Oct 1, 2011 at 19:47, Andreas Krey wrote:\n> \n> > The question is, should git forbid two filenames that consist\n> > of the *same* characters, only differently uni-encoded? I don't\n> > think anyone would make two files named 'Büro', with different\n> > unicode encodings. But as far as I know that is a shady area.\n> \n> So, let's leave git's current behavior as the default and provide\n> a config variable that when set, tells git to handle file names\n> in terms of characters rather than bytes.\n\nYou meant here _graphemes_, not Unicode codepoint when talking about\ncharacters, didn't you?\n\nIIRC the problem with MacOS X is that it accepts different composition\nwhen creating a file from what it returns when asking for contents of\ndirectory (NFD if I remember correctly, which is less used).\n\n\nThere are some beginnings of sanely handling filesystem encoding in\nGit (the framework), but it is currently underutilized only to handle\ncase-sensitivity and case-preserving.\n\n-- \nJakub Narębski\n"},{"id":"176666","messageId":"CAMOZ1BvcQtOzPR=a8NJ9QHXUMhfmmLWwbKMZqsmQTSE150TaXQ@mail.gmail.com","threadId":"28544","inReplyTo":"m3hb3snw4b.fsf@localhost.localdomain","subject":"Re: Git, Mac OS X and German special characters","fromName":"Michael Witten","fromEmail":"mfwitten@gmail.com","sentAt":"2011-10-01T23:26:44Z","receivedAt":"2011-10-01T23:26:44Z","isPatch":false,"sender":{"key":"mfwitten@gmail.com","avatar":"https://avatars.githubusercontent.com/u/597101?v=4"},"body":"2011/10/1 Jakub Narebski :\n\n> Michael Witten writes:\n>> On Sat, Oct 1, 2011 at 19:47, Andreas Krey wrote:\n>>\n>> > The question is, should git forbid two filenames that consist\n>> > of the *same* characters, only differently uni-encoded? I don't\n>> > think anyone would make two files named 'Büro', with different\n>> > unicode encodings. But as far as I know that is a shady area.\n>>\n>> So, let's leave git's current behavior as the default and provide\n>> a config variable that when set, tells git to handle file names\n>> in terms of characters rather than bytes.\n>\n> You meant here _graphemes_, not Unicode codepoint when talking about\n> characters, didn't you?\n\nYes.\n"},{"id":"176667","messageId":"CAO1Q+jec0Dm3FXEbcb_yEU5XPW_7V5sf+Sn8SR0h9fGWJE=-=w@mail.gmail.com","threadId":"28544","inReplyTo":"CAMOZ1BuXiQkZG_7mvay-ybm7Q7niwXVhbmbCmkfy=wD1AKsasQ@mail.gmail.com","subject":"Re: Git, Mac OS X and German special characters","fromName":"Albert Zeyer","fromEmail":"albzey@googlemail.com","sentAt":"2011-10-01T23:48:07Z","receivedAt":"2011-10-01T23:48:07Z","isPatch":false,"sender":{"key":"albzey@googlemail.com","avatar":null},"body":"On Sun, Oct 2, 2011 at 12:02 AM, Michael Witten wrote:\n> On Sat, Oct 1, 2011 at 19:47, Andreas Krey wrote:\n>\n>> The question is, should git forbid two filenames that consist\n>> of the *same* characters, only differently uni-encoded? I don't\n>> think anyone would make two files named 'Büro', with different\n>> unicode encodings. But as far as I know that is a shady area.\n>\n> So, let's leave git's current behavior as the default and provide\n> a config variable that when set, tells git to handle file names\n> in terms of characters rather than bytes.\n\nI just read the very lengthy discussion here:\nhttp://thread.gmane.org/gmane.comp.version-control.git/70688\n\nBasically all the arguments have already been discussed.\n\nThere are varios options. Most of them are not mutual exclusive, so it\nwould also be an option to implement most of them and let the user\npick what (s)he prefers.\n\n* TreatFilenamesAsText or however you would call it. I.e. handle\nfilenames the same when they equal in Unicode.\n\nLinus is very much against this because in rare situations, it could\ndestroy your data, like in this example:\n\n\techo \"foo\" > Hütte # \"Hütte\" in NFC\n\techo \"bar\" > Hütte # \"Hütte\" in NFD\n\nThe second write would overwrite silently the file generated by the\nfirst write if those filenames would be handled the same. This (and\nsuch) behavior is to be avoided, claims Linus, because it would more\noften lead to not wanted behavior in third party applications.\n\n* On MacOSX, wrap all filesystem functions (like readdir()) to convert\nall filenames to NFC.\n\nMacOSX normalizes the UTF8 representation of the filenames to NFD but\nin most common situations (on most other systems), you end up with the\nfilename being in NFC.\n\nAs the filename is anyway normalized on OSX, it doesn't matter wether\nit is handled as NFC or NFD and NFC will likely generate less trouble.\nAnd this patch doesn't even really need an option.\n\nThis was one suggestion by Linus itself:\nhttp://news.gmane.org/find-root.php?message_id=%3calpine.LFD.1.00.0801211323120.2957%40woody.linux%2dfoundation.org%3e\n\n* Disallow any files with filenames which are not in NFC at all. This\nmakes some things a bit more safe (like on MacOSX; along with the\nprevious suggestion) and more clear (you always know that your\nfilename is in NFC).\n\n* Some more clever readdir() which, when it gets a filename which is\nnot in the Git index but Unicode-equally to one filename in the Git\nindex, automatically replaces it by the filename in the index.\n\nThis is some sort of half way to a TreatFilenamesAsText option but\nshould produce less trouble.\n\nThis probably also doesn't need an extra option as it should very\nlikely generate less trouble (on OSX at least; and for other systems\nwhich don't mangle the filename, they don't need to use this code at\nall).\n\n---\n\nI will probably go and try to implement the clever-readdir(). And/or\nmaybe also the NFC conversation in such a readdir() wrapper.\n"},{"id":"176786","messageId":"4E8A1197.4050708@web.de","threadId":"28544","inReplyTo":"CAO1Q+jf=RO=sE90-mQdi+=fUWH1RLM+JTubSgSVGC5uDyhU+2A@mail.gmail.com","subject":"Re: Git, Mac OS X and German special characters","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2011-10-03T19:48:39Z","receivedAt":"2011-10-03T19:48:39Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"The patch has probably not been applied, because it is not a god one.\nI have been working on a better version,\nbut that is not 100% ready to be released.\n\nI can post it in a couple of days,\n(and yes, it does a NFD->NFC conversion in readdir() )\n/Torsten\nOn 10/01/2011 02:44 PM, Albert Zeyer wrote:\n> Hi,\n>\n> There are problems on MacOSX with different UTF8 encodings of\n> filenames. A unicode string has multiple ways to be represented as\n> UTF8 and Git treats them as different filenames. This is the actual\n> bug. It should treat them all as the same filename. In some cases (as\n> on MacOSX), the underlying operating system may use a normalized UTF8\n> representation in some sort, i.e. change the actual UTF8 filename\n> representation.\n>\n> Similar problems also exists in SVN, for example. This was reported\n> [here](http://subversion.tigris.org/issues/show_bug.cgi?id=2464).\n> There you can find also lengthy discussions about the topic. And also\n> [here](http://svn.apache.org/repos/asf/subversion/trunk/notes/unicode-composition-for-filenames).\n>\n> This was already reported for Git earlier and there is also a patch\n> for Git [here](http://lists-archives.org/git/719832-git-mac-os-x-and-german-special-characters.html).\n>\n> I wonder about the state of this. This hasn't been applied yet. Why?\n>\n> Regards,\n> Albert\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"}]}