Re: git on MacOSX and files with decomposed utf-8 file names

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Jan 17, 2008, 19:11 UTC
Message-ID: <alpine.LFD.1.00.0801171100330.14959@woody.linux-foundation.org>
In-Reply-To: <F666FFD2-9777-47EA-BEF4-C78906CA8901@simplicidade.org>

On Thu, 17 Jan 2008, Pedro Melo wrote:

Show 5 quoted lines

>
> We have people using windows, people using Macs, and people using several
> flavors of Linux desktops. They all have different settings and if I add a
> file like áéióú that happens to be UTF-8 encoded, it will reach a iso-latin-1
> user as visual garbage.

Yes.

> git will track the file perfectly, we know that, because the sequence of 
> bytes that my system used to create the file will be the same on all 
> "sane" systems, but the file will look "funny" to some users, and we get 
> complaints for some less enlightened ones.

I can't really suggest anything else than trying to make everybody use UTF-8.

[ Not just for filenames, by the way - this is one of the reasons I think
  it is so *important* to not corrupt filenames, exactly because this is 
  in no way filename-specific at all, and filenames are generally "textual 
  data" exactly the same way a text-file is.

  But only totally insane people think that you should force-normalize 
  text-files, even though all the issues are obviously all the same 
  regardless of whether it's a filename or a word in textfile. ]

And yes, I also realize that it's not going to be realistic. We're probably *closer* to that than we used to be, but I don't think you can even make Windows think FAT is UTF-8.

I don't know how NTFS works (I know it is Unicode-aware, and I think it encodes filenames in UCS-2 or possibly UTF-16, but there is an obvious 1:1 translation to UTF-8, and since we use C strings, I'd assume/hope Windows actually uses that unambiguous translation for any filenames).

Under modern Linux and OS X, UTF-8 is basically the only way (older Linux distros may be set up for Latin1, but at least the newer ones seem to all default to a UTF-8 locale).

> The answer is that users should not create filenames with non-ascii characters
> if they want a consistent experience, right?

Oh, absolutely. That takes care of 99.9% of all source projects. Even then you can have problems with case insensitivity (the Linux kernel sources are all US-ASCII filenames, for example, but *literally* has many files that are identical if you ignore case, and that's not unheard of).

So yes, to a first approximation, the answer is to simply avoid using anything but US-ASCII. It's seldom a big limitation when talking about filenames.

			Linus