Re: Rss produced by git is not valid xml?

From: Linus Torvalds <torvalds@osdl.org>
Date: Nov 18, 2005, 20:47 UTC
Message-ID: <Pine.LNX.4.64.0511181237040.13959@g5.osdl.org>
In-Reply-To: <437E3971.3060803@zytor.com>

On Fri, 18 Nov 2005, H. Peter Anvin wrote:

> 
> I think the point is: what do you do with the data?  If it *looks* like valid
> UTF-8, you pretty much have to assume it is; if it's not (it contains invalid
> UTF-8 sequences), what do you do?

Btw, this is not a new issue.

This is true even of data that _claims_ to be UTF-8 but contains sequences that are illegal. A program that just barfs on it is a buggy program.

And yes, I know there are buggy programs out there. I seem to recall some perl(?) problems when it got UTF-8 strings that weren't, and did impossible things.

> There are only a small handful of alternatives, and none are really good:
> 
> 	- Reject it (it's kind of too late, should have been done at
> 	  checkin)

It can't be done at checkin, since it's not _wrong_. It's 8-bit data.

It's like saying that /bin/echo is an illegal program and shouldn't be executed, because it's not encoded in utf-8.

I can well imagine somebody wanting to put a binary signature at the end of a commit. git shouldn't care, and the important thing to realize is that there _is_ no "encoding" for such things. So the commits don't necessarily have to have a font encoding at all, and any visualization tool should just accept that fact.

Show 5 quoted lines

> 	- Show them as SUBSTITUTE characters (U+FFFD).
> 	- Show them as Latin-1 or Windows-1252
> 	- Provide a complex configuration mechanism
> 
> I think Kay is going with the second option.

Which is a fine option. Latin-1 is probably the right choice for the kernel, but not necessarily for other projects.

Another option is to just pass them through unmodified, and encourage the XML parser to handle it. Anything that takes UTF-8 and doesn't have some fallback to handle malformed input is basically buggy. It simply _will_ happen occasionally, quite independently of git. You can either give up, or try to handle it. And giving up is always the wrong choice.

			Linus