{"thread":{"id":"9076","subject":"Do a better job at guessing unknown character sets","startedAt":"2007-07-17T17:34:44Z","lastAt":"2007-07-17T20:01:22Z","messageCount":3,"participants":["Linus Torvalds","Johannes Schindelin","david@lang.hm"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"47656","messageId":"alpine.LFD.0.999.0707171027100.19166@woody.linux-foundation.org","threadId":"9076","inReplyTo":null,"subject":"Do a better job at guessing unknown character sets","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-07-17T17:34:44Z","receivedAt":"2007-07-17T17:34:44Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nAt least in the kernel development community, we're generally slowly \nconverting to UTF-8 everywhere, and the old default of Latin1 in emails is \nbeing supplanted by UTF-8, and it doesn't necessarily show up as such in \nthe mail headers (because, quite frankly, when people send patches \naround, they want the email client to do as little as humanly possible \nabout the patch)\n\nDespite that, it's often the case that email addresses etc still have \nLatin1, so I've seen emails where this is a mixed bag, with Signed-off \nparts being copied from email (and containing Latin1 characters), and the \nrest of the email being a patch in UTF-8.\n\nSo this suggests a very natural change: if the target character set is \nutf-8 (the default), and if the source already looks like utf-8, just \nassume that it doesn't need any conversion at all.\n\nOnly assume that it needs conversion if it isn't already valid utf-8, in \nwhich case we (for historical reasons) will assume it's Latin1.\n\nBasically no really _valid_ latin1 will ever look like utf-8, so while \nthis changes our historical behaviour, it doesn't do so in practice, and \nmakes the default behaviour saner for the case where the input was already \nin proper format.\n\nWe could do a more fancy guess, of course, but this correctly handled a \nseries of patches I just got from Andrew that had a mixture of Latin1 and \nUTF-8 (in different emails, but without any character set indication).\n\nSigned-off-by: Linus Torvalds <torvalds@linux-foundation.org>\n---\n\nI think this makes sense from a \"the world is moving to utf-8\" standpoint, \neven if obviously some people might consider it a bit ugly to do per-line \n\"guessing\".\n\nComments?\n\n builtin-mailinfo.c |   33 +++++++++++++++++++++++++++++----\n 1 files changed, 29 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin-mailinfo.c b/builtin-mailinfo.c\nindex 489c2c5..a37a4ff 100644\n--- a/builtin-mailinfo.c\n+++ b/builtin-mailinfo.c\n@@ -499,15 +499,40 @@ static int decode_b_segment(char *in, char *ot, char *ep)\n \treturn 0;\n }\n \n+/*\n+ * When there is no known charset, guess.\n+ *\n+ * Right now we assume that if the target is UTF-8 (the default),\n+ * and it already looks like UTF-8 (which includes US-ASCII as its\n+ * subset, of course) then that is what it is and there is nothing\n+ * to do.\n+ *\n+ * Otherwise, we default to assuming it is Latin1 for historical\n+ * reasons.\n+ */\n+static const char *guess_charset(const char *line, const char *target_charset)\n+{\n+\tif (is_encoding_utf8(target_charset)) {\n+\t\tif (is_utf8(line))\n+\t\t\treturn NULL;\n+\t}\n+\treturn \"latin1\";\n+}\n+\n static void convert_to_utf8(char *line, const char *charset)\n {\n-\tstatic const char latin_one[] = \"latin1\";\n-\tconst char *input_charset = *charset ? charset : latin_one;\n-\tchar *out = reencode_string(line, metainfo_charset, input_charset);\n+\tchar *out;\n+\n+\tif (!charset || !*charset) {\n+\t\tcharset = guess_charset(line, metainfo_charset);\n+\t\tif (!charset)\n+\t\t\treturn;\n+\t}\n \n+\tout = reencode_string(line, metainfo_charset, charset);\n \tif (!out)\n \t\tdie(\"cannot convert from %s to %s\\n\",\n-\t\t    input_charset, metainfo_charset);\n+\t\t    charset, metainfo_charset);\n \tstrcpy(line, out);\n \tfree(out);\n }\n"},{"id":"47665","messageId":"Pine.LNX.4.64.0707172053030.14781@racer.site","threadId":"9076","inReplyTo":"alpine.LFD.0.999.0707171027100.19166@woody.linux-foundation.org","subject":"Re: Do a better job at guessing unknown character sets","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-07-17T19:56:21Z","receivedAt":"2007-07-17T19:56:21Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 17 Jul 2007, Linus Torvalds wrote:\n\n> I think this makes sense from a \"the world is moving to utf-8\" \n> standpoint, even if obviously some people might consider it a bit ugly \n> to do per-line \"guessing\".\n> \n> Comments?\n\nIMHO this is a good change.  Encodings are such a hassle, and probably \nonly because the inventors of ASCII just were narrow-minded enough not to \ncare.  With this patch, the hassle factor diminishes AFAICT.\n\nCiao,\nDscho\n\nP.S.: I think that in case of undesired behaviour, even if it is detected \nlate in the game, filter-branch/rewrite-commits will help.\n"},{"id":"47666","messageId":"Pine.LNX.4.64.0707171257420.2467@asgard.lang.hm","threadId":"9076","inReplyTo":"Pine.LNX.4.64.0707172053030.14781@racer.site","subject":"Re: Do a better job at guessing unknown character sets","fromName":"","fromEmail":"david@lang.hm","sentAt":"2007-07-17T20:01:22Z","receivedAt":"2007-07-17T20:01:22Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Tue, 17 Jul 2007, Johannes Schindelin wrote:\n\n> On Tue, 17 Jul 2007, Linus Torvalds wrote:\n>\n>> I think this makes sense from a \"the world is moving to utf-8\"\n>> standpoint, even if obviously some people might consider it a bit ugly\n>> to do per-line \"guessing\".\n>>\n>> Comments?\n>\n> Encodings are such a hassle, and probably\n> only because the inventors of ASCII just were narrow-minded enough not to\n> care.\n\nto be perfectly fair, at the time ASCII was invented it was done to \neliminate the use of the different, incompatible character sets that were \nin use at the time. And it did the job well (I think the only surviver \nfrom those sets is EBCDIC, and only due to the legacy installed base)\n\ncurrent character encodings are doing things that weren't dreamed of by \nanyone at the time.\n\nDavid Lang\n"}]}