{"thread":{"id":"37028","subject":"git format-patch doesn't add Content-type for UTF-8 diffs","startedAt":"2014-06-30T09:03:25Z","lastAt":"2014-07-01T04:38:24Z","messageCount":4,"participants":["Paul Eggert","Jeff King","Torsten Bögershausen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"245176","messageId":"53B127DD.8000807@cs.ucla.edu","threadId":"37028","inReplyTo":null,"subject":"git format-patch doesn't add Content-type for UTF-8 diffs","fromName":"Paul Eggert","fromEmail":"eggert@cs.ucla.edu","sentAt":"2014-06-30T09:03:25Z","receivedAt":"2014-06-30T09:03:25Z","isPatch":false,"sender":{"key":"eggert@cs.ucla.edu","avatar":"https://avatars.githubusercontent.com/u/572024?v=4"},"body":"I've been having trouble sending my Git-generated patches to the tz \nmailing list.  Patches containing UTF-8 text are garbled, e.g., if you \nvisit <http://mm.icann.org/pipermail/tz/2014-June/021086.html> you'll \nsee \"Ãœrümqi\" where the patch actually had \"Ürümqi\".\n\nI've tracked this down to the fact that \"git format-patch\" isn't \noutputting a Content-Type: line in the outgoing email.  I thought it was \nsupposed to do that; the man page implies that it does.\n\nHere's how I can reproduce the bug with the git 1.9.3 that's shipped \nwith Fedora 20.  Notice that the patch is missing the line \n\"Content-Type: text/plain; charset=UTF-8\" that the git-format-patch man \npage implies it should be generating, and this causes the ICANN email \nsoftware to misinterpret the patch's character set encoding.\n\n$ git init\nInitialized empty Git repository in /home/eggert/junk/d/.git/\n$ echo x >x\n$ git add x\n$ git commit -m'x'\n[master (root-commit) 5d0e0ce] x\n  1 file changed, 1 insertion(+)\n  create mode 100644 x\n$ echo '§' >x\n$ git commit -am'added UTF-8'\n[master 57f0669] added UTF-8\n  1 file changed, 1 insertion(+), 1 deletion(-)\n$ git format-patch -1\n0001-added-UTF-8.patch\n$ cat 0001-added-UTF-8.patch\n From 57f066927a1d8e253715b7980460d81cb549b162 Mon Sep 17 00:00:00 2001\nFrom: Paul Eggert <eggert@cs.ucla.edu>\nDate: Mon, 30 Jun 2014 01:49:28 -0700\nSubject: [PATCH] added UTF-8\n\n---\n  x | 2 +-\n  1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/x b/x\nindex 587be6b..3038d22 100644\n--- a/x\n+++ b/x\n@@ -1 +1 @@\n-x\n+§\n--\n1.9.3\n"},{"id":"245208","messageId":"20140630173052.GB16747@sigill.intra.peff.net","threadId":"37028","inReplyTo":"53B127DD.8000807@cs.ucla.edu","subject":"Re: git format-patch doesn't add Content-type for UTF-8 diffs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-06-30T17:30:52Z","receivedAt":"2014-06-30T17:30:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 30, 2014 at 02:03:25AM -0700, Paul Eggert wrote:\n\n> I've been having trouble sending my Git-generated patches to the tz mailing\n> list.  Patches containing UTF-8 text are garbled, e.g., if you visit\n> <http://mm.icann.org/pipermail/tz/2014-June/021086.html> you'll see\n> \"Ãœrümqi\" where the patch actually had \"Ürümqi\".\n> \n> I've tracked this down to the fact that \"git format-patch\" isn't outputting\n> a Content-Type: line in the outgoing email.  I thought it was supposed to do\n> that; the man page implies that it does.\n\nformat-patch will add a content-type header if the commit message\ncontains non-ascii characters, and is marked as an alternate encoding\n(usually this is utf8, but you can use i18n.commitEncoding to store them\nin a different format).\n\nHowever, it doesn't look at the filenames or diff contents at all. If it\nwere to do so, it would have to guess at the correct encoding, since git\ndoesn't know anything about the encoding of filenames or contents.\nWorse, you could actually have several different encodings, across\nmultiple files in the same diff.\n\nTypically, the next stage in the pipeline is to give the output to\nsend-email, or to a MUA. Send-email will detect high-bit characters in\nthis case and ask you which encoding you want. Many MUAs will do some\nkind of auto-detection and fill in the content-type (e.g., I know that\nmutt handles this correctly).\n\nHow do you send the mails after they come out of format-patch?\n\n> Here's how I can reproduce the bug with the git 1.9.3 that's shipped with\n> Fedora 20.  Notice that the patch is missing the line \"Content-Type:\n> text/plain; charset=UTF-8\" that the git-format-patch man page implies it\n> should be generating, and this causes the ICANN email software to\n> misinterpret the patch's character set encoding.\n> [...]\n\nThanks for the reproduction recipe. While I think we have to accept that\nsome hard cases (e.g., multiple encodings in a single diff) can't be\nhandled cleanly, it would be really nice if this all-utf8 case worked\nout of the box. And perhaps the complex cases could use binary diffs\nwhen we see multiple encodings.\n\nOne tricky thing about the implementation is that we stream the output\nfrom format-patch, and write the content-type header (if any) before we\nstart opening the blobs for diff.\n\nI wonder if it would be enough to do:\n\n  1. Always add a content-type header, even if the commit is utf-8 and\n     contains only ascii characters. This _shouldn't_ hurt anything,\n     though I suppose it would if you have latin1 (for example) commit\n     messages and did not correctly set the encoding header in your\n     commits.\n\n  2. When producing diff header lines do not respect core.quotepath if\n     the filename does is not valid for the encoding we claimed earlier.\n\n  3. When producing lines of textual diff, use a binary diff if the\n     contents are not valid for the encoding we claimed earlier.\n\nThat would make the utf-8 case \"just work\", and would prevent us from\never sending malformed contents (i.e., mismatched encodings in the\ncommit message and diff contents). However, it is not perfect:\n\n  1. Right now if you send a diff for a latin1 file but do not use any\n     non-ascii characters in your commit message (and do not set\n     i18n.commitEncoding, so it is \"utf8\"), you get no claimed encoding\n     in the email. If your receiving end is OK with that, everything\n     works, and you get to see text diffs.\n\n     So my scheme would be a slight regression there. But it is somewhat\n     of an accident waiting to happen. If you ever use a utf-8 character\n     in your commit message, that particular email will be marked as\n     utf-8, and your diff will be broken.\n\n  2. We can only check \"is it valid?\" for the encoding. That works well\n     with utf-8, which has rules. But for something like latin1 versus\n     another \"use a code page for the high-bit bytes\" type of encoding,\n     we cannot really tell the difference. However, I do not think we\n     are making anything _worse_ there. You'd already get mojibake in\n     such a case.\n\n-Peff\n"},{"id":"245214","messageId":"53B1B24B.2040609@cs.ucla.edu","threadId":"37028","inReplyTo":"20140630173052.GB16747@sigill.intra.peff.net","subject":"Re: git format-patch doesn't add Content-type for UTF-8 diffs","fromName":"Paul Eggert","fromEmail":"eggert@cs.ucla.edu","sentAt":"2014-06-30T18:54:03Z","receivedAt":"2014-06-30T18:54:03Z","isPatch":false,"sender":{"key":"eggert@cs.ucla.edu","avatar":"https://avatars.githubusercontent.com/u/572024?v=4"},"body":"Jeff King wrote:\n> How do you send the mails after they come out of format-patch?\n\nI run a shell command like this (on Solaris 10):\n\n/usr/lib/sendmail -ONoRecipientAction=add-to tz@iana.org < \n0001-whatever.patch\n\n(The \"NoRecipientAction\" option pacifies the IANA MTA.)\n\nThis is an old machine not under my control, with an old 'git' installed \nthat I don't use and don't particularly want to worry about porting to. \n  I generate the patch file on a different machine with git 1.9.3, and \nscp it into the email-sending machine.\n\nI suppose that I could work around the problem with this shell command:\n\n(grep -q '^Mime-Version: ' 0001-whatever.patch ||\n    printf '%s\\n' \\\n      'MIME-Version: 1.0' \\\n      'Content-Type: text/plain; charset=UTF-8' \\\n      'Content-Transfer-Encoding: 8bit'\n  cat 0001-whatever.patch) |\n/usr/lib/sendmail -ONoRecipientAction=add-to tz@iana.org\n\nbut that's less convenient.\n\n> I wonder if it would be enough to do:\n>\n>   1. Always add a content-type header, even if the commit is utf-8 and\n>      contains only ascii characters.\n\nThat would help for my case, yes.  We use only UTF-8, and to me it \nfeelds weird that patches are mailed properly if the commit log contains \nnon-ASCII characters, but don't work if the commit log is ASCII and the \ndiff contains non-ASCII.\n"},{"id":"245223","messageId":"53B23B40.1070209@web.de","threadId":"37028","inReplyTo":"20140630173052.GB16747@sigill.intra.peff.net","subject":"Re: git format-patch doesn't add Content-type for UTF-8 diffs","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-07-01T04:38:24Z","receivedAt":"2014-07-01T04:38:24Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n>I wonder if it would be enough to do:\n\n>  1. Always add a content-type header, even if the commit is utf-8 and\n>     contains only ascii characters. This _shouldn't_ hurt anything,\n>     though I suppose it would if you have latin1 (for example) commit\n>     messages and did not correctly set the encoding header in your\n>     commits.\n\nDoes it make sense to call this function (from utf8.c)\n\nint is_utf8(const char *text)\n\nand either add the content-type header for utf-8 (or not)\n"}]}