Re: [PATCH] Try URI quoting for embedded TAB and LF in pathnames

From: Linus Torvalds <torvalds@osdl.org>
Date: Oct 11, 2005, 20:56 UTC
Message-ID: <Pine.LNX.4.64.0510111346220.14597@g5.osdl.org>
In-Reply-To: <87slv7zvqj.fsf@penguin.cs.ucla.edu>

On Tue, 11 Oct 2005, Paul Eggert wrote:

Show 5 quoted lines

> 
> Yes, quite true.  But we don't have to come up with something that's
> perfect in all cases, just something that's good enough to handle
> cases that we expect will be common in practice, in a world where
> UTF-8 is the preferred encoding for non-ASCII characters.

The thing is, I can almost guarantee you that any quoting in the high characters is going to be _worse_ than no quoting at all.

Exactly because quoting as UTF-8 is the wrong thing when it isn't actually UTF-8, and quoting as non-UTF-8 is the wrong thing when it _is_.

Not quoting at all, on the other hand, is unambigious. If you have a mailer that corrupts your text stream (which-ever type it is), then it's clearly the mailers problem. The _mailer_ at least has a chance in hell to know what character set it is getting mailed as.

The other alternative is to quote _everything_ non-ASCII. That's definitely reliable, but it's also unquestionably ugly as hell, especially in the long run.

Yes, there are some complex quoting approaches you can do, which quote things "correctly" (ie at a byte stream level) _and_ keep it valid UTF-8 at the same time.

For example, you can read it as a UTF-8 stream, but then quote things at a byte level (ie if you quote one "character", you quote _all_ bytes in that character). And you quote if:

 - the UTF-8 _character_ is in the 0x80-0x9f control range
 - any _raw_byte_ is in the 0x80-0x9f range (it might not be UTF-8)
 - any _raw_byte_ is 0xfe-0xff (illegal UTF-8 character)
 - misformed UTF-8 (non-shortest sequence, or just generally invalid 
   sequences with missing or wrong high bits)

but quite frankly, that's a pretty painful thing to write. The upside is that it's easy to decode: you can _unquote_ it just as a byte stream.

			Linus