{"thread":{"id":"50743","subject":"\"Problems\" with git format-patch --thread email header ordering","startedAt":"2019-03-14T23:45:13Z","lastAt":"2019-04-05T05:24:57Z","messageCount":5,"participants":["Linus Torvalds","Junio C Hamano","Julian Andres Klode"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"371552","messageId":"CAHk-=whP1stFZNAaJiMi5eZ9rj0MRt20Y_yHVczZPH+O01d+sA@mail.gmail.com","threadId":"50743","inReplyTo":null,"subject":"\"Problems\" with git format-patch --thread email header ordering","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2019-03-14T23:44:51Z","receivedAt":"2019-03-14T23:45:13Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"So Thomas Gleixner just figured out that the google gmail support for\nS/MIME is even more broken than we initially thought, and has been\nrejecting emails that have a non-canonical order of headers in the\nemail. In particular, gmail s/mime parsing hates emails generated with\n\"git format-patch --thread\" (and then encrypted to be s/mime).\n\nWith the git format-patch header ordering, S/MIME senders will get\nincomprehensible (and wrong) bounces that say\n\n Mail delivery failed: returning message to sender (fwd)\n\n 550-5.7.1 Our system has detected that this\n 550-5.7.1 message is not RFC 5322 compliant:\n 550-5.7.1 'From' header is missing.\n\nwhen sending to an gmail S/MIME-enabled recipient.\n\nNow, this is very much a gmail bug, but I've long since given up any\nhope at all that the gmail people would listen to outsiders (and from\nmy interactions with people _inside_ of google, I think they consider\nanybody outside the \"gmail\" team to be outsiders - good luck to any\nGoogle people trying to get gmail issues fixed either).\n\nNote that the gmail bounce about 'From' header is missing is\ncompletely bogus, and claims about RFC 5322 are equally inane. The\nSMTP RFC's very much say that the order of header files is not\nguaranteed, and gmail is wrong.\n\nAlso note that this does not affect any *normal* emails to gmail\nrecipients. It only seems to affect the server-side s/mime decryption\nof the email, so you'll never see it unless the recipient actually has\ns/mime support enabled (and you encode the git format-patch as\ns/mime).\n\nHOWEVER.\n\nWhile it's true that header ordering isn't specified, there's a common\n\"canonical\" order that the headers are listed in. To quote rfc822:\n\n Note: Due to an artifact of the notational conventions, the syn-\n tax indicates that, when present, some fields, must be in\n a particular order. Header fields are NOT required to\n occur in any particular order, except that the message\n body must occur AFTER the headers. It is recommended\n that, if present, headers be sent in the order \"Return-\n Path\", \"Received\", \"Date\", \"From\", \"Subject\", \"Sender\",\n \"To\", \"cc\", etc.\n\nNote how that very basic smtp rfc makes it very clear that headers are\nvery much not required to occur in any particular order, but it does\nhave a recommended ordering.\n\nI suspect some broken code inside gmail uses that \"notational\nconvention with syntax that indicates that some fields must be in a\nparticular order\". The RFC explicitly states that it's wrong, but hey,\nsomebody cut-and-pasted the syntax from the RFC and didn't read the\nnote.\n\nAlso note that rfc 5322 (which is a newer version of 822) doesn't\nreally change that, but does make it clear that trace and resent\nfields must not be re-ordered during retransmission (so \"Received\"\nfields etc should stay ordered). That's not about accepting messages,\nthat's about the transfer of them, though. Gmail is still wrong, and\npointing to the newer rtc doesn't make gmail any more correct.\n\nSo gmail is simply wrong in having some odd ordering requirement.\n\nBut git format-patch does create the email headers out in a different\norder than the one recommended. When you do \"git format-patch\n--thread\" to create the emails, the header ordering looks roughly like\nthis:\n\n Message-Id: \n From: Linus Torvalds \n Date: Thu, 14 Mar 2019 16:29:30 -0700\n Subject: [PATCH 0/6] *** SUBJECT HERE ***\n MIME-Version: 1.0\n Content-Type: text/plain; charset=UTF-8\n Content-Transfer-Encoding: 8bit\n\nand gmail is quite unhappy with the result if it is then sent as\ns/mime. Gmail apparently really wants that\nDate/From/Subject/To/Message-Id ordering.\n\nDon't ask me why. Gmail is simply wrong. But I have a very strong\nsuspicion that it's easier to fix \"git format-patch\" than it is to fix\nwhatever odd gmail issue.\n\nComments? Thomas has munged his s/mime infrastructure to re-order\nthings, but git _could_ do the proper recommended ordering.\n\nAnd yes, if somebody from Google on this list wants to bring this up\nwith the gmail team, I wish you the best of luck. Let me know how it\ngoes.\n\n Linus\n"},{"id":"371562","messageId":"xmqqk1h09224.fsf@gitster-ct.c.googlers.com","threadId":"50743","inReplyTo":"CAHk-=whP1stFZNAaJiMi5eZ9rj0MRt20Y_yHVczZPH+O01d+sA@mail.gmail.com","subject":"Re: \"Problems\" with git format-patch --thread email header ordering","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-03-15T04:47:15Z","receivedAt":"2019-03-15T04:47:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> While it's true that header ordering isn't specified, there's a common\n> \"canonical\" order that the headers are listed in. To quote rfc822:\n> ...\n> body must occur AFTER the headers. It is recommended\n> that, if present, headers be sent in the order \"Return-\n> Path\", \"Received\", \"Date\", \"From\", \"Subject\", \"Sender\",\n> \"To\", \"cc\", etc.\n>\n> But git format-patch does create the email headers out in a different\n> order than the one recommended. When you do \"git format-patch\n> --thread\" to create the emails, the header ordering looks roughly like\n> this:\n>\n> Message-Id: \n> From: Linus Torvalds \n> Date: Thu, 14 Mar 2019 16:29:30 -0700\n> Subject: [PATCH 0/6] *** SUBJECT HERE ***\n> MIME-Version: 1.0\n> Content-Type: text/plain; charset=UTF-8\n> Content-Transfer-Encoding: 8bit\n> ...\n> And yes, if somebody from Google on this list wants to bring this up\n> with the gmail team, I wish you the best of luck. Let me know how it\n> goes.\n\nI obviously won't do the last one myself, but if the issue is only\nto swap from and date, then this may be sufficient, perhaps?\n\nI suspect that there would be fallouts in t/ directory before we can\ncall this a fix.\n\n pretty.c | 11 +++++++++--\n 1 file changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/pretty.c b/pretty.c\nindex f496f0f128..40c7236fbc 100644\n--- a/pretty.c\n+++ b/pretty.c\n@@ -452,6 +452,14 @@ void pp_user_info(struct pretty_print_context *pp,\n \t\tmap_user(pp->mailmap, &mailbuf, &maillen, &namebuf, &namelen);\n \n \tif (cmit_fmt_is_mail(pp->fmt)) {\n+\n+\t\t/* \n+\t\t * Date: before From:\n+\t\t * cf. \n+\t\t */\n+\t\tstrbuf_addf(sb, \"Date: %s\\n\",\n+\t\t\t show_ident_date(&ident, DATE_MODE(RFC2822)));\n+\n \t\tif (pp->from_ident && ident_cmp(pp->from_ident, &ident)) {\n \t\t\tstruct strbuf buf = STRBUF_INIT;\n \n@@ -502,8 +510,7 @@ void pp_user_info(struct pretty_print_context *pp,\n \t\tbreak;\n \tcase CMIT_FMT_EMAIL:\n \tcase CMIT_FMT_MBOXRD:\n-\t\tstrbuf_addf(sb, \"Date: %s\\n\",\n-\t\t\t show_ident_date(&ident, DATE_MODE(RFC2822)));\n+\t\t/* has been done much earlier */\n \t\tbreak;\n \tcase CMIT_FMT_FULLER:\n \t\tstrbuf_addf(sb, \"%sDate: %s\\n\", what,\n"},{"id":"371615","messageId":"CAHk-=whe+R7fkTi0eBnhrBY=_k_du3e5dkNsVe=HgKXLoXekdg@mail.gmail.com","threadId":"50743","inReplyTo":"xmqqk1h09224.fsf@gitster-ct.c.googlers.com","subject":"Re: \"Problems\" with git format-patch --thread email header ordering","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2019-03-15T16:59:25Z","receivedAt":"2019-03-15T16:59:47Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Thu, Mar 14, 2019 at 9:47 PM Junio C Hamano wrote:\n>\n> Linus Torvalds writes:\n>\n> > While it's true that header ordering isn't specified, there's a common\n> > \"canonical\" order that the headers are listed in. To quote rfc822:\n> > ...\n> > body must occur AFTER the headers. It is recommended\n> > that, if present, headers be sent in the order \"Return-\n> > Path\", \"Received\", \"Date\", \"From\", \"Subject\", \"Sender\",\n> > \"To\", \"cc\", etc.\n>\n> I obviously won't do the last one myself, but if the issue is only\n> to swap from and date, then this may be sufficient, perhaps?\n\nI'm not actually sure _what_ the order requirements for gmail are,\nsince gmail itself doesn't seem to honor them. Does the order of the\nMessage-ID header line matter, for example?\n\nI don't think it's the order of the From/Date lines, actually, because\ngoogle itself doesn't do that.\n\nWhat Thomas Found out was that the exact same email with\n\n Message-Id/From/Date/Subject/To\n\n(in that order) does not work, but\n\n Date/From/Subject/To/Message-Id\n\ndoes work. Weird and \"wonderful\". But there might be a lot of other\norderings that work or don't.\n\nHaving looked through some other emails, I know that\n\n From/To/Subject/Date/Message-Id\n Subject/To/References/From/Message-ID/Date\n\nalso works. Which makes me suspect that it's the Message-ID line that matters.\n\nBut it might be something _really_ odd, and maybe not just ordering at\nall (ie it might be something where gmail wants to see the \"From\"\nline, but only in certain circumstances, and only *then* does it\nmatter if the From line comes before some other line or not.\n\nAll we know is\n\n (a) gmail complains about the normal git format-patch ordering with a\n\"no From line\" bounce\n\n (b) the same email with just headers re-ordered goes through\n\nI *suspect* it's that Message-ID line, but .... It's very annoying.\n\n Linus\n"},{"id":"373126","messageId":"20190404180653.curtoo7yot6tpqfk@jak-t480s","threadId":"50743","inReplyTo":"CAHk-=whe+R7fkTi0eBnhrBY=_k_du3e5dkNsVe=HgKXLoXekdg@mail.gmail.com","subject":"Re: \"Problems\" with git format-patch --thread email header ordering","fromName":"Julian Andres Klode","fromEmail":"jak@jak-linux.org","sentAt":"2019-04-04T18:06:53Z","receivedAt":"2019-04-04T18:06:59Z","isPatch":false,"sender":{"key":"jak@jak-linux.org","avatar":null},"body":"On Fri, Mar 15, 2019 at 09:59:25AM -0700, Linus Torvalds wrote:\n> On Thu, Mar 14, 2019 at 9:47 PM Junio C Hamano wrote:\n> >\n> > Linus Torvalds writes:\n> >\n> > > While it's true that header ordering isn't specified, there's a common\n> > > \"canonical\" order that the headers are listed in. To quote rfc822:\n> > > ...\n> > > body must occur AFTER the headers. It is recommended\n> > > that, if present, headers be sent in the order \"Return-\n> > > Path\", \"Received\", \"Date\", \"From\", \"Subject\", \"Sender\",\n> > > \"To\", \"cc\", etc.\n> >\n> > I obviously won't do the last one myself, but if the issue is only\n> > to swap from and date, then this may be sufficient, perhaps?\n> \n> I'm not actually sure _what_ the order requirements for gmail are,\n> since gmail itself doesn't seem to honor them. Does the order of the\n> Message-ID header line matter, for example?\n> \n> I don't think it's the order of the From/Date lines, actually, because\n> google itself doesn't do that.\n> \n> What Thomas Found out was that the exact same email with\n> \n> Message-Id/From/Date/Subject/To\n> \n> (in that order) does not work, but\n> \n> Date/From/Subject/To/Message-Id\n> \n> does work. Weird and \"wonderful\". But there might be a lot of other\n> orderings that work or don't.\n> \n> Having looked through some other emails, I know that\n> \n> From/To/Subject/Date/Message-Id\n> Subject/To/References/From/Message-ID/Date\n> \n> also works. Which makes me suspect that it's the Message-ID line that matters.\n\nI also know that gmail rewrites the Message-ID / creates one if it is\nmissing or \"odd\" (such as ends in a .). It those probably makes sense\nin that twisted world view to require that to be fairly late...\n\n-- \ndebian developer - deb.li/jak | jak-linux.org - free software dev\nubuntu core developer i speak de, en\n"},{"id":"373180","messageId":"xmqq7ec9dnxs.fsf@gitster-ct.c.googlers.com","threadId":"50743","inReplyTo":"20190404180653.curtoo7yot6tpqfk@jak-t480s","subject":"Re: \"Problems\" with git format-patch --thread email header ordering","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-05T05:24:31Z","receivedAt":"2019-04-05T05:24:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Julian Andres Klode writes:\n\n>> What Thomas Found out was that the exact same email with\n>> \n>> Message-Id/From/Date/Subject/To\n>> \n>> (in that order) does not work, but\n>> \n>> Date/From/Subject/To/Message-Id\n>> \n>> does work. Weird and \"wonderful\". But there might be a lot of other\n>> orderings that work or don't.\n>> \n>> Having looked through some other emails, I know that\n>> \n>> From/To/Subject/Date/Message-Id\n>> Subject/To/References/From/Message-ID/Date\n>> \n>> also works. Which makes me suspect that it's the Message-ID line that matters.\n>\n> I also know that gmail rewrites the Message-ID / creates one if it is\n> missing or \"odd\" (such as ends in a .). It those probably makes sense\n> in that twisted world view to require that to be fairly late...\n\nHere is a patch that is not polished enough to be commited yet but\nshould be testable (as I suspect that existing tests need to be\nadjusted).\n\n-- >8 --\n\nSubject: [PATCH] format-patch: move message-id and related headers to the end\n\nIt has been reported that GMail MSA has an undocumented limitation\nthat makes it reject or accept depending on some ordering of headers\nin the incoming message. The current suspicion is that they want to\nsee the Message-Id: header near the end, so let's try doing so and\nsee what happens..\n\nSigned-off-by: Junio C Hamano \n---\n log-tree.c | 30 ++++++++++++++++++++++--------\n 1 file changed, 22 insertions(+), 8 deletions(-)\n\ndiff --git a/log-tree.c b/log-tree.c\nindex 1e56df62a7..111b947264 100644\n--- a/log-tree.c\n+++ b/log-tree.c\n@@ -363,24 +363,27 @@ void log_write_email_headers(struct rev_info *opt, struct commit *commit,\n \tconst char *extra_headers = opt->extra_headers;\n \tconst char *name = oid_to_hex(opt->zero_commit ?\n \t\t\t\t &null_oid : &commit->object.oid);\n+\tstruct strbuf message_ids = STRBUF_INIT;\n \n \t*need_8bit_cte_p = 0; /* unknown */\n \n \tfprintf(opt->diffopt.file, \"From %s Mon Sep 17 00:00:00 2001\\n\", name);\n \tgraph_show_oneline(opt->graph);\n-\tif (opt->message_id) {\n-\t\tfprintf(opt->diffopt.file, \"Message-Id: <%s>\\n\", opt->message_id);\n-\t\tgraph_show_oneline(opt->graph);\n-\t}\n+\n+\tif (opt->message_id)\n+\t\tstrbuf_addf(&message_ids, \"Message-Id: <%s>\\n\", opt->message_id);\n+\n \tif (opt->ref_message_ids && opt->ref_message_ids->nr > 0) {\n \t\tint i, n;\n \t\tn = opt->ref_message_ids->nr;\n-\t\tfprintf(opt->diffopt.file, \"In-Reply-To: <%s>\\n\", opt->ref_message_ids->items[n-1].string);\n+\t\tstrbuf_addf(&message_ids, \"In-Reply-To: <%s>\\n\",\n+\t\t\t opt->ref_message_ids->items[n-1].string);\n \t\tfor (i = 0; i < n; i++)\n-\t\t\tfprintf(opt->diffopt.file, \"%s<%s>\\n\", (i > 0 ? \"\\t\" : \"References: \"),\n-\t\t\t opt->ref_message_ids->items[i].string);\n-\t\tgraph_show_oneline(opt->graph);\n+\t\t\tstrbuf_addf(&message_ids, \"%s<%s>\\n\",\n+\t\t\t\t (i > 0 ? \"\\t\" : \"References: \"),\n+\t\t\t\t opt->ref_message_ids->items[i].string);\n \t}\n+\n \tif (opt->mime_boundary && maybe_multipart) {\n \t\tstatic struct strbuf subject_buffer = STRBUF_INIT;\n \t\tstatic struct strbuf buffer = STRBUF_INIT;\n@@ -425,6 +428,17 @@ void log_write_email_headers(struct rev_info *opt, struct commit *commit,\n \t\topt->diffopt.stat_sep = buffer.buf;\n \t\tstrbuf_release(&filename);\n \t}\n+\n+\tif (message_ids.len) {\n+\t\tstatic struct strbuf buf = STRBUF_INIT;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addbuf(&buf, &message_ids);\n+\t\tif (extra_headers)\n+\t\t\tstrbuf_addstr(&buf, extra_headers);\n+\t\textra_headers = buf.buf;\n+\t}\n+\n \t*extra_headers_p = extra_headers;\n }\n \n-- \n2.21.0-197-g893dd55891\n\n"}]}