{"thread":{"id":"44168","subject":"[PATCH 0/2] Locally alias \"latin-1\" to \"ISO-8859-1\"","startedAt":"2016-09-27T01:22:19Z","lastAt":"2016-09-27T15:33:30Z","messageCount":7,"participants":["Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"302665","messageId":"20160927012211.9378-1-gitster@pobox.com","threadId":"44168","inReplyTo":null,"subject":"[PATCH 0/2] Locally alias \"latin-1\" to \"ISO-8859-1\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-27T01:22:09Z","receivedAt":"2016-09-27T01:22:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Some systems do not seem to ship \"latin-1\" as a valid locale, even\nthough they happilly accept more modern official name \"ISO-8859-1\".\nNaturally, \"iconv -f iso-8859-1\" succeeds while \"iconv -f latin-1\"\nfails on such a system.\n\nWe already have in utf8.c to accomodate overly strict iconv_open()\nthat does not like various spellings of UTF-8 when our users spell\nit differently from the most official \"UTF-8\" form. Piggyback on\nthe mechanism and teach outselves that \"latin-1\" used to be the way\nto say \"ISO-8859-1\".\n\nI feel dirty for doing it this way, but I found it the easiest\nworkaround to apply recent patches we saw on the mailing list.\n\nJunio C Hamano (2):\n utf8: refactor code to decide fallback encoding\n utf8: accept \"latin-1\" as ISO-8859-1\n\n utf8.c | 36 +++++++++++++++++++++++++-----------\n 1 file changed, 25 insertions(+), 11 deletions(-)\n\n-- \n2.10.0-556-g5bbc40b\n\n"},{"id":"302666","messageId":"20160927012211.9378-2-gitster@pobox.com","threadId":"44168","inReplyTo":"20160927012211.9378-1-gitster@pobox.com","subject":"[PATCH 1/2] utf8: refactor code to decide fallback encoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-27T01:22:10Z","receivedAt":"2016-09-27T01:22:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The codepath we use to call iconv_open() has a provision to use a\nfallback encoding when it fails, hoping that \"UTF-8\" being spelled\ndifferently could be the reason why the library function did not\nlike the encoding names we gave it. Essentially, we turn what we\nhave observed to be used as variants of \"UTF-8\" (e.g. \"utf8\") into\nthe most official spelling and use that as a fallback.\n\nWe do the same thing for input and output encoding. Introduce a\nhelper function to do just one side and call that twice.\n\nSigned-off-by: Junio C Hamano \n---\n utf8.c | 29 ++++++++++++++++++-----------\n 1 file changed, 18 insertions(+), 11 deletions(-)\n\ndiff --git a/utf8.c b/utf8.c\nindex 00e10c8..550e785 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -489,6 +489,21 @@ char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv, int *outs\n \treturn out;\n }\n \n+static const char *fallback_encoding(const char *name)\n+{\n+\t/*\n+\t * Some platforms do not have the variously spelled variants of\n+\t * UTF-8, so let's fall back to trying the most official\n+\t * spelling. We do so only as a fallback in case the platform\n+\t * does understand the user's spelling, but not our official\n+\t * one.\n+\t */\n+\tif (is_encoding_utf8(name))\n+\t\treturn \"UTF-8\";\n+\n+\treturn name;\n+}\n+\n char *reencode_string_len(const char *in, int insz,\n \t\t\t const char *out_encoding, const char *in_encoding,\n \t\t\t int *outsz)\n@@ -501,17 +516,9 @@ char *reencode_string_len(const char *in, int insz,\n \n \tconv = iconv_open(out_encoding, in_encoding);\n \tif (conv == (iconv_t) -1) {\n-\t\t/*\n-\t\t * Some platforms do not have the variously spelled variants of\n-\t\t * UTF-8, so let's fall back to trying the most official\n-\t\t * spelling. We do so only as a fallback in case the platform\n-\t\t * does understand the user's spelling, but not our official\n-\t\t * one.\n-\t\t */\n-\t\tif (is_encoding_utf8(in_encoding))\n-\t\t\tin_encoding = \"UTF-8\";\n-\t\tif (is_encoding_utf8(out_encoding))\n-\t\t\tout_encoding = \"UTF-8\";\n+\t\tin_encoding = fallback_encoding(in_encoding);\n+\t\tout_encoding = fallback_encoding(out_encoding);\n+\n \t\tconv = iconv_open(out_encoding, in_encoding);\n \t\tif (conv == (iconv_t) -1)\n \t\t\treturn NULL;\n-- \n2.10.0-556-g5bbc40b\n\n"},{"id":"302667","messageId":"20160927012211.9378-3-gitster@pobox.com","threadId":"44168","inReplyTo":"20160927012211.9378-1-gitster@pobox.com","subject":"[PATCH 2/2] utf8: accept \"latin-1\" as ISO-8859-1","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-27T01:22:11Z","receivedAt":"2016-09-27T01:22:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Even though latin-1 is still seen in e-mail headers, some platforms\nonly install ISO-8859-1. \"iconv -f ISO-8859-1\" succeeds, while\n\"iconv -f latin-1\" fails on such a system.\n\nUsing the same fallback_encoding() mechanism factored out in the\nprevious step, teach ourselves that \"ISO-8859-1\" has a better chance\nof being accepted than \"latin-1\".\n\nSigned-off-by: Junio C Hamano \n---\n utf8.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/utf8.c b/utf8.c\nindex 550e785..0c8e011 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -501,6 +501,13 @@ static const char *fallback_encoding(const char *name)\n \tif (is_encoding_utf8(name))\n \t\treturn \"UTF-8\";\n \n+\t/*\n+\t * Even though latin-1 is still seen in e-mail\n+\t * headers, some platforms only install ISO-8859-1.\n+\t */\n+\tif (!strcasecmp(name, \"latin-1\"))\n+\t\treturn \"ISO-8859-1\";\n+\n \treturn name;\n }\n \n-- \n2.10.0-556-g5bbc40b\n\n"},{"id":"302678","messageId":"20160927055202.4ucddki3xkns45om@sigill.intra.peff.net","threadId":"44168","inReplyTo":"20160927012211.9378-2-gitster@pobox.com","subject":"Re: [PATCH 1/2] utf8: refactor code to decide fallback encoding","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-09-27T05:52:02Z","receivedAt":"2016-09-27T05:52:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 26, 2016 at 06:22:10PM -0700, Junio C Hamano wrote:\n\n> @@ -501,17 +516,9 @@ char *reencode_string_len(const char *in, int insz,\n> \n> \tconv = iconv_open(out_encoding, in_encoding);\n> \tif (conv == (iconv_t) -1) {\n> -\t\t/*\n> -\t\t * Some platforms do not have the variously spelled variants of\n> -\t\t * UTF-8, so let's fall back to trying the most official\n> -\t\t * spelling. We do so only as a fallback in case the platform\n> -\t\t * does understand the user's spelling, but not our official\n> -\t\t * one.\n> -\t\t */\n> -\t\tif (is_encoding_utf8(in_encoding))\n> -\t\t\tin_encoding = \"UTF-8\";\n> -\t\tif (is_encoding_utf8(out_encoding))\n> -\t\t\tout_encoding = \"UTF-8\";\n> +\t\tin_encoding = fallback_encoding(in_encoding);\n> +\t\tout_encoding = fallback_encoding(out_encoding);\n> +\n\nThis comment is interesting. We're concerned about a platform knowing\n\"utf8\" but not \"UTF-8\". When we fallback, we do it for both the input\nand output encodings, because we don't know which may have caused the\nproblem. So is it possible that we improve one case but break the other?\n\nWith just UTF-8, I don't think so. That could only be the case with\nsomething like \"utf8 -> utf-8\" because they both become \"UTF-8\". So\neither it improves the situation or not (because we either understand\nUTF-8 or not).\n\nBut once we introduce other fallbacks, then \"utf8 -> latin1\" may become\n\"UTF-8 -> iso8859-1\". A system that knows only \"utf8\" and \"iso8859-1\"\n_could_ work if we turned the knobs individually, but won't if we turn\nthem both at once. Worse, a system that knows only \"UTF-8\" and \"latin1\"\nworks now, but would break with your patches.\n\nI'm not convinced it's worth worrying about, though. The existence of\nsuch a system is theoretical at this point. I'm not even sure how common\nthe \"know about utf8 but not UTF-8\" thing is, or if we were merely being\noverly cautious.\n\n-Peff\n"},{"id":"302679","messageId":"20160927055744.el2jbxzdqfhjl6qt@sigill.intra.peff.net","threadId":"44168","inReplyTo":"20160927012211.9378-3-gitster@pobox.com","subject":"Re: [PATCH 2/2] utf8: accept \"latin-1\" as ISO-8859-1","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-09-27T05:57:45Z","receivedAt":"2016-09-27T05:57:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 26, 2016 at 06:22:11PM -0700, Junio C Hamano wrote:\n\n> Even though latin-1 is still seen in e-mail headers, some platforms\n> only install ISO-8859-1. \"iconv -f ISO-8859-1\" succeeds, while\n> \"iconv -f latin-1\" fails on such a system.\n> \n> Using the same fallback_encoding() mechanism factored out in the\n> previous step, teach ourselves that \"ISO-8859-1\" has a better chance\n> of being accepted than \"latin-1\".\n\nI was curious if this was the most official or accepted spelling.\nGrepping a few hundred thousand messages from my mail archives, it does\nseem to be the most common.\n\n> diff --git a/utf8.c b/utf8.c\n> index 550e785..0c8e011 100644\n> --- a/utf8.c\n> +++ b/utf8.c\n> @@ -501,6 +501,13 @@ static const char *fallback_encoding(const char *name)\n> \tif (is_encoding_utf8(name))\n> \t\treturn \"UTF-8\";\n> \n> +\t/*\n> +\t * Even though latin-1 is still seen in e-mail\n> +\t * headers, some platforms only install ISO-8859-1.\n> +\t */\n> +\tif (!strcasecmp(name, \"latin-1\"))\n> +\t\treturn \"ISO-8859-1\";\n> +\n\nFor the UTF-8 fallbacks, we actually detect their equivalence via\nsame_encoding() before even hitting iconv. Is it worth doing the same\nhere?\n\nI have to admit that I don't care too deeply about performance for\nsomebody who wants to convert \"latin1\" to \"ISO-8859-1\". If one of your\nencodings is not UTF-8, you are probably Doing It Wrong. :)\n\n-Peff\n"},{"id":"302680","messageId":"xmqq7f9xswm7.fsf@gitster.mtv.corp.google.com","threadId":"44168","inReplyTo":"20160927055744.el2jbxzdqfhjl6qt@sigill.intra.peff.net","subject":"Re: [PATCH 2/2] utf8: accept \"latin-1\" as ISO-8859-1","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-27T06:08:32Z","receivedAt":"2016-09-27T06:08:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I have to admit that I don't care too deeply about performance for\n> somebody who wants to convert \"latin1\" to \"ISO-8859-1\". If one of your\n> encodings is not UTF-8, you are probably Doing It Wrong. :)\n\nExactly. Note that the \"you\" in the above are usually plural,\ncollectively referring to both the sender and the receiver. I\nusually am on the poor receiving end ;-)\n"},{"id":"302709","messageId":"xmqqvaxhqrwd.fsf@gitster.mtv.corp.google.com","threadId":"44168","inReplyTo":"20160927055202.4ucddki3xkns45om@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] utf8: refactor code to decide fallback encoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-09-27T15:33:22Z","receivedAt":"2016-09-27T15:33:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> But once we introduce other fallbacks, then \"utf8 -> latin1\" may become\n> \"UTF-8 -> iso8859-1\". A system that knows only \"utf8\" and \"iso8859-1\"\n> _could_ work if we turned the knobs individually, but won't if we turn\n> them both at once. Worse, a system that knows only \"UTF-8\" and \"latin1\"\n> works now, but would break with your patches.\n>\n> I'm not convinced it's worth worrying about, though. The existence of\n> such a system is theoretical at this point. I'm not even sure how common\n> the \"know about utf8 but not UTF-8\" thing is, or if we were merely being\n> overly cautious.\n\nYeah, I did consider having to try the permutations until it works,\nbut suspecting that somebody takes \"utf8\" without taking \"UTF-8\" is\nto pretty much invalidate the basic premise of the existing code,\ni.e. spelling it as \"UTF-8\" is the most likely to work anywhere as\nlong as UTF-8 is supported, so I stopped worrying about it at that\npoint.\n\nI'd actually welcome a more generic suggestions we can put in our\ndocumentation so that we can _lose_ the fallback entirely (e.g. \"if\nyour contributor spelled 'utf8' and your system, which does take\n'UTF-8', does not like it, then here is what you can do to your\n/etc/locale.alias\").\n\n"}]}