{"thread":{"id":"8162","subject":"testing vger handling of charsets (part 2)","startedAt":"2007-05-15T07:13:40Z","lastAt":"2007-05-16T10:57:01Z","messageCount":7,"participants":["Jeff King","Karl Hasselström","J. Bruce Fields","Junio C Hamano","Jan Hudec"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"42197","messageId":"20070515test.2@coredump.intra.peff.net","threadId":"8162","inReplyTo":null,"subject":"testing vger handling of charsets (part 2)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2007-05-15T07:13:40Z","receivedAt":"2007-05-15T07:13:40Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This is a test message to check how vger reacts to seeing 8bit\ncharacters (like Hasselström) in a message _with_ the right mime\nheader.\n\nIf your mail server doesn't advertise the 8BITMIME extensions, I expect\nthis message to come through OK, since vger will (hopefully) respect the\nexisting content-type header. Karl and Bruce, can you please report?\n"},{"id":"42202","messageId":"20070515083148.GD9096@diana.vm.bytemark.co.uk","threadId":"8162","inReplyTo":"20070515test.2@coredump.intra.peff.net","subject":"Re: testing vger handling of charsets (part 2)","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2007-05-15T08:31:48Z","receivedAt":"2007-05-15T08:31:48Z","isPatch":false,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2007-05-15 03:13:40 -0400, Jeff King wrote:\n\n> This is a test message to check how vger reacts to seeing 8bit\n> characters (like Hasselström) in a message _with_ the right mime\n> header.\n\nLooks OK both in the copy I got directly, and in the copy I got via\nthe list.\n\n-- \nKarl Hasselström, kha@treskal.com\n www.treskal.com/kalle\n"},{"id":"42222","messageId":"20070515145947.GB6794@fieldses.org","threadId":"8162","inReplyTo":"20070515083148.GD9096@diana.vm.bytemark.co.uk","subject":"Re: testing vger handling of charsets (part 2)","fromName":"J. Bruce Fields","fromEmail":"bfields@fieldses.org","sentAt":"2007-05-15T14:59:47Z","receivedAt":"2007-05-15T14:59:47Z","isPatch":false,"sender":{"key":"bfields@citi.umich.edu","avatar":null},"body":"On Tue, May 15, 2007 at 10:31:48AM +0200, Karl Hasselström wrote:\n> On 2007-05-15 03:13:40 -0400, Jeff King wrote:\n> \n> > This is a test message to check how vger reacts to seeing 8bit\n> > characters (like Hasselström) in a message _with_ the right mime\n> > header.\n> \n> Looks OK both in the copy I got directly, and in the copy I got via\n> the list.\n\nYup, same here.\n\n--b.\n"},{"id":"42292","messageId":"7vmz05ffad.fsf@assigned-by-dhcp.cox.net","threadId":"8162","inReplyTo":"20070515test.2@coredump.intra.peff.net","subject":"Re: testing vger handling of charsets (part 2)","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-05-16T06:34:02Z","receivedAt":"2007-05-16T06:34:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> This is a test message to check how vger reacts to seeing 8bit\n> characters (like Hasselström) in a message _with_ the right mime\n> header.\n>\n> If your mail server doesn't advertise the 8BITMIME extensions, I expect\n> this message to come through OK, since vger will (hopefully) respect the\n> existing content-type header. Karl and Bruce, can you please report?\n\nI think you are trying to figure out how vger adds/munges the\nheaders, and the above is not very useful for people but\nyourself unless you explicitly say what headers you gave on your\nend in the body of the message, is it?\n\nJudging from the list responses, I am guessing the situation is\nlike this. Does that match your understanding?\n\nbotched one:\n\n\toutgoing:\n \tbody in utf-8\n \tContent-type: text/plain; charset=utf-8\n no MIME-Version: header\n\n\tvger relayed to recipients:\n \tbody untouched\n Content-type: text/plain; charset=iso-8859-1\n MIME-Version: 1.0\n\ngood one:\n\n\toutgoing:\n \tbody in utf-8\n \tContent-type: text/plain; charset=utf-8\n MIME-Version: 1.0\n\n\tvger relayed to recipients: everything intact.\n\n\nI am not sure what exactly you meant by with/without \"the right\nmime header\", but the above is based on my guess that you meant\nonly MIME-VERSION header.\n"},{"id":"42304","messageId":"20070516092915.GA10660@artax.karlin.mff.cuni.cz","threadId":"8162","inReplyTo":"7vmz05ffad.fsf@assigned-by-dhcp.cox.net","subject":"Re: testing vger handling of charsets (part 2)","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2007-05-16T09:29:15Z","receivedAt":"2007-05-16T09:29:15Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Tue, May 15, 2007 at 11:34:02PM -0700, Junio C Hamano wrote:\n> botched one:\n> \n> \toutgoing:\n> \tbody in utf-8\n> \tContent-type: text/plain; charset=utf-8\n> no MIME-Version: header\n> \n> \tvger relayed to recipients:\n> \tbody untouched\n> Content-type: text/plain; charset=iso-8859-1\n> MIME-Version: 1.0\n\nThe strange thing is, that I got it from vger -- with \nContent-type: text/plain; charset=utf-8\n\nTherefore either:\n - It's not vger, but some other mail software, that munges it.\n - Some software on my side correctly guesses that it should have been\n utf-8, but I don't really believe that.\n\n--------------------------------------------------------------------------------\n \t\t\t\t- Jan Hudec `Bulb' \n"},{"id":"42306","messageId":"20070516105522.GA30256@coredump.intra.peff.net","threadId":"8162","inReplyTo":"7vmz05ffad.fsf@assigned-by-dhcp.cox.net","subject":"Re: testing vger handling of charsets (part 2)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2007-05-16T10:55:24Z","receivedAt":"2007-05-16T10:55:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 15, 2007 at 11:34:02PM -0700, Junio C Hamano wrote:\n\n> I think you are trying to figure out how vger adds/munges the\n> headers, and the above is not very useful for people but\n> yourself unless you explicitly say what headers you gave on your\n> end in the body of the message, is it?\n\nYes, I'm sorry that the message itself looked a bit vague. It was\nactually about the 4th or 5th such message I sent, as the list filter\nkept blocking the previous ones, so with each iteration I made the\nmessage shorter and shorter to try to remove any offending text.\n\nSo Karl and Bruce actually received several explanatory messages that\neveryone else didn't, and I really only expected them to be replying.\n\n> Judging from the list responses, I am guessing the situation is\n> like this. Does that match your understanding?\n\nYes, this is close.\n\n> \toutgoing:\n> \tbody in utf-8\n> \tContent-type: text/plain; charset=utf-8\n> no MIME-Version: header\n> \n> \tvger relayed to recipients:\n> \tbody untouched\n> Content-type: text/plain; charset=iso-8859-1\n> MIME-Version: 1.0\n\nThere is also a \"Content-Transfer-Encoding: 8bit\" that gets switched to\nquoted-printable (and the body is actually encoded as QP). However, the\nchange of charset is the problem.\n\n> I am not sure what exactly you meant by with/without \"the right\n> mime header\", but the above is based on my guess that you meant\n> only MIME-VERSION header.\n\nYes, the two messages differed _only_ in the presence of a MIME-Version\nheader.\n\nSo now that I have the data, let me explain the sequence of events in\nthe bug, which should hopefully explain what everyone has seen.\n\n1. Bruce generates a message containing utf8 characters in the body and\nthe following headers:\n Content-Type: text/plain; charset=utf-8\n Content-Transfer-Encoding: 8bit\nand _no_ MIME-Version header. This is produced by git-format-patch on a\ncommit with non-ascii in the body. The message is sent to vger, and cc'd\nto some recipients, and we move to step 2.\n\n2a. The cc'd recipients receive one copy of the message intact; none of\nthe mailservers along the route do any munging. This message is done.\n\n2b. vger sends the message to each list member in turn. What the user\nsees depends on the mail route. We move to step 3.\n\n3a. If the next hop from vger advertises 8BITMIME in the SMTP session,\nthen vger submits the message intact. This is the case for me, so I see\nall messages intact (and is why I needed responses from others --\nspecifically, I knew Karl and Bruce were seeing the problem). This\nmessage is done.\n\n3b. If the next hop does not advertise 8BITMIME, vger must convert the\nmessage to a 7bit encoding (it chooses quoted-printable). Continue to\nstep 4.\n\n4a. If the message has valid MIME headers, then vger can simply encode,\nre-writing the content-transfer-encoding to quoted-printable and\nencoding the body. vger considers valid mime headers to be a\nMIME-version header and a content-type header. This is the case for the\nsecond message I set, which appears correctly to all recipients.\n\n4b. If the message doesn't have valid MIME headers, then vger adds the\nheaders. Without a MIME-Version header, it ignores the content-type and\nguesses at a suitable one, using text/plain with some totally arbitrary\nlocal charset (in this case \"iso-8859-1\"). This message has now been\nincorrectly munged (claims latin1 charset, but has utf8 characters).\nvger puts an explanation into the X-Warning headers of the munged\nmessage (the only unexplained thing that I had to test is that\nMIME-Version is critical to vger believing the current content-type).\n\nSo recipients see the bug IFF\n the original has utf8 characters\n AND the original lacks a MIME-Version header\n AND their mailserver doesn't claim 8BITMIME\n\nInterestingly, rfc1428 claims that in this case vger should actually set\nthe charset to \"unknown-8bit\":\n\n If no information about the character set in use is available, the\n gateway should upgrade the content by using the character set\n \"unknown-8bit\". The unknown-8bit value of the charset parameter\n indicates only that no reliable information about the character set(s)\n used in the message was available.\n\nThough that really just pushes the problem to the recipients MUA, and I\nhave no idea what the handling of \"unknown-8bit\" is like there.\n\n-Peff\n"},{"id":"42307","messageId":"20070516105700.GB30256@coredump.intra.peff.net","threadId":"8162","inReplyTo":"20070516092915.GA10660@artax.karlin.mff.cuni.cz","subject":"Re: testing vger handling of charsets (part 2)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2007-05-16T10:57:01Z","receivedAt":"2007-05-16T10:57:01Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 16, 2007 at 11:29:15AM +0200, Jan Hudec wrote:\n\n> The strange thing is, that I got it from vger -- with \n> Content-type: text/plain; charset=utf-8\n> \n> Therefore either:\n> - It's not vger, but some other mail software, that munges it.\n> - Some software on my side correctly guesses that it should have been\n> utf-8, but I don't really believe that.\n\nIt's option 3:\n - vger selectively munges based on whether your server claims 8BITMIME\n during the SMTP EHLO. So some list members never saw this issue at\n all.\n\n-Peff\n"}]}