Re: git on MacOSX and files with decomposed utf-8 file names

On Mon, 21 Jan 2008, Kevin Ballard wrote:

You said it was invisible when you treat things "as text". Here's the quote:

	.. when you treat filenames as text, it DOESN'T MATTER if the 
 	string gets normalized ..

Without ever apparently realizing that "as text" is part of the problem in itself. What is "text" to one person is gibberish to another.

In particular, the biggest reason to not normalize is that you don't know it's text or Unicode in the first place. Which is why git doesn't do it.

And no, even with filenames you don't know that they are "text". People encode stuff in them. And people don't always use UTF-8.

Of course, you could ask everybody to create OS X-only programs that know that under OS X, you only have a subset of filenames. If so, you're complaining about the wrong tool. Especially when the whole point of the tool was to be distributed (not to mention coming from an environment that simply doesn't have the same silly limitations OS X has).

So here's a few clues:

 - "as text" isn't "as unicode": it may well be Latin1 or EUC-JP or
   something. Yes, it's still used. Git doesn't care, and very consciously 
   has avoided forcing character sets, even if the *default* (and notice 
   how it's overridable) commit message encoding may be utf-8.

 - In fact, even in unicode, the difference between "identical" and 
   "equivalent" strings exists, and even in the standard, unicode 
   strings are very much defined to be arbitrary codepoint sequences, not 
   normalized.

So even for the very specific case of unicode text, it's simply not true that "it doesn't matter if the string gets normalized". The unicode spec itself talks about cases where even canonical normalization makes a difference.

Search for this quote:

  "Not all processes are required to respect canonical equivalence. For 
   example:

    * A function that collects a set of the General_Category values 
      present in a string will and should produce a different value for 
      <angstrom sign, semicolon> than for <A, combining ring above, greek 
      question mark>, even though they are canonically equivalent.
    * A function that does a binary comparison of strings will also find 
      these two sequences different."

and notice that first case. Even things that are *very*much* aware of Unicode text do actually have cases where canonical equivalence doesn't mean crud.

		Linus