Re: [PATCH] Try URI quoting for embedded TAB and LF in pathnames

From: Linus Torvalds <torvalds@osdl.org>
Date: Oct 14, 2005, 05:20 UTC
Message-ID: <Pine.LNX.4.64.0510132203220.23590@g5.osdl.org>
In-Reply-To: <87irw1q7eu.fsf@penguin.cs.ucla.edu>

On Thu, 13 Oct 2005, Paul Eggert wrote:

> 
> Perhaps so, but it has a lot of company.  I have even worse problems
> with Mozilla Thunderbird.  And as we observed, Pine also has problems
> sending properly-formatted email containing arbitrary binary data.

No, pine does it right. Exactly because it sends _arbitraty_ binary data.

The fact that I turned the terminal into utf-8 mode in order to generate the bytes (that end up being a garbage string in latin1) is not pine's fault.

The point being that because the transport was 8-bit clean, I could do that. I could mix a latin1-encoding with a UTF-8 encoding, and the other side could see the mixed setting. Now, the other side had no way of knowing that I mixed things (unless it was a smart human and could read and understand what I wrote), so any email client would have trouble showing it.

But it got _transferred_ right, and you could have saved the email, and turned the terminal into latin1 or utf-8 mode, and done a "cat" both ways, and you'd have seen both versions.

> I suspect the vast majority of email clients will screw up in
> relatively common cases involving unusual characters in file names.

Not if they just save it.

Oh, sure, they can't _display_ it, since they don't know what it is, but when they save it, they'd _better_ save it bit-for-bit.

Which is the right thing to do. Then you apply it with "patch", and you get the right answer.

> Using attachments avoids many of the problems, but lots of patches are
> emailed inline and I'd rather not force people to use attachments to
> send diffs.

inline or attachment should not matter to any sane email client. If it does, then the email client isn't sane.

The point is, when you save it, it _has_ to be saved bit-for-bit.

The only difference between a binary attachment and a text thing is that an email client will _try_ to show the text thing to you as text. It has no other meaning.

And trying is better than not trying. Attachments are _inferior_ to inline for that reason.

Show 8 quoted lines

> > I find that email is very robust - it's basically 8-bit clean. No 
> > character encoding, no crap. Just a byte stream. It really _is_ the most 
> > reliable format.
> 
> Hmm.  To test that theory, I just now sent plain-text email to myself,
> containing a carriage-return (CR) byte in the middle of a line.
> 
> The CR byte was transliterated into a LF.  Ooops.

I'm not surprised, since CR/LF is special for a lot of (sad) reasons. Oh, well.

I agree that it makes sense to escape \r, and obviously you _have_ to escape \n. In general, escaping pretty much everything in the 0-31 range is likely the right approach, since those are never printable anyway.

That, btw, is probably true of the patch contents too, not just the filename. The exception being \t (and in patch contents, \n is obviously part of the stream).

> More generally, I suspect inline patches with weird bytes will suffer
> greatly from encoding and recoding by mail agents.

I've had pretty good luck. We do have 8-bit stuff occasionally, but it almost always makes it through.

Spaces and tabs are much worse (yes, they're more common too). That's clearly just crap mailers.

Show 6 quoted lines

> Unfortunately this isn't true for Emacs, and I suspect other mailers
> will have similar problems.  For example, with Emacs I can easily save
> either the exact byte-for-byte message body that my mail transfer
> agent gave me; or I can have Emacs decode the message into its
> constituent characters, reencode the result as UTF-8, and put that
> into a file.

Well, as long as there's a choice.

> In neither case, though, am I saving the original byte
> stream that you presented to your mail user agent.  Even if I save the
> byte-for-byte message body, it is often in quoted-printable format so
> I'll have to decode strings like "=EF" to recover the original bytes.

You have a broken mail client. Now, I'm not a big fan of QP (I think it was making a stupid excuse for bad transport), but QP is a _mail_ level quoting protocol, and the same way a MUA uses QP to encode, the MUA should have de-coded the QP. It shouldn't leave it to somebody else.

I think GNU emacs is a horrible mistake ("do everything - badly"), but you may be able to fix it by letting your mail transport agent do the un-QP for you. A lot of them do, which makes it easier to then use weak MUA's.

Anyway, it sounds like GNU emacs made the wrong choices (hey, I'm not surprised). It should have decoded QP, not the character set. There are lots of tools that do charset conversions, that's not very email-specific.

			Linus