From: Linus Torvalds Date: Tue, 11 Oct 2005 20:56:12 GMT Subject: Re: [PATCH] Try URI quoting for embedded TAB and LF in pathnames Message-ID: In-Reply-To: <87slv7zvqj.fsf@penguin.cs.ucla.edu> On Tue, 11 Oct 2005, Paul Eggert wrote: > > Yes, quite true. But we don't have to come up with something that's > perfect in all cases, just something that's good enough to handle > cases that we expect will be common in practice, in a world where > UTF-8 is the preferred encoding for non-ASCII characters. The thing is, I can almost guarantee you that any quoting in the high characters is going to be _worse_ than no quoting at all. Exactly because quoting as UTF-8 is the wrong thing when it isn't actually UTF-8, and quoting as non-UTF-8 is the wrong thing when it _is_. Not quoting at all, on the other hand, is unambigious. If you have a mailer that corrupts your text stream (which-ever type it is), then it's clearly the mailers problem. The _mailer_ at least has a chance in hell to know what character set it is getting mailed as. The other alternative is to quote _everything_ non-ASCII. That's definitely reliable, but it's also unquestionably ugly as hell, especially in the long run. Yes, there are some complex quoting approaches you can do, which quote things "correctly" (ie at a byte stream level) _and_ keep it valid UTF-8 at the same time. For example, you can read it as a UTF-8 stream, but then quote things at a byte level (ie if you quote one "character", you quote _all_ bytes in that character). And you quote if: - the UTF-8 _character_ is in the 0x80-0x9f control range - any _raw_byte_ is in the 0x80-0x9f range (it might not be UTF-8) - any _raw_byte_ is 0xfe-0xff (illegal UTF-8 character) - misformed UTF-8 (non-shortest sequence, or just generally invalid sequences with missing or wrong high bits) but quite frankly, that's a pretty painful thing to write. The upside is that it's easy to decode: you can _unquote_ it just as a byte stream. Linus