{"thread":{"id":"6352","subject":"[PATCH] Reencode committer info to utf-8 before formatting mail header","startedAt":"2007-01-12T13:06:57Z","lastAt":"2007-01-16T12:43:39Z","messageCount":18,"participants":["David Kågedal","Junio C Hamano","Alex Riesen","Johannes Schindelin","Robin Rosenberg","Horst H. von Brand"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"31552","messageId":"871wm08kcu.fsf@morpheus.local","threadId":"6352","inReplyTo":null,"subject":"[PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"David Kågedal","fromEmail":"davidk@lysator.liu.se","sentAt":"2007-01-12T13:06:57Z","receivedAt":"2007-01-12T13:06:57Z","isPatch":true,"sender":{"key":"davidk@lysator.liu.se","avatar":"https://avatars.githubusercontent.com/u/60530?v=4"},"body":"The add_user_info function formats the commit as a mail message, and\nuses add_rfc2047 to format the From: line.  The add_rfc2047 assumes\nthat the string is encoded as utf-8.\n---\n builtin-mailinfo.c |    2 +-\n commit.c           |   10 +++++++++-\n utf8.c             |    9 +++++++--\n utf8.h             |    2 +-\n 4 files changed, 18 insertions(+), 5 deletions(-)\n\nI was hit by this problem when working with an old repository where I\nhad used latin1, and I tried to use \"git rebase\".\n\nAnother option would have been to use the correct encoding in the\nRFC2047 header, but this was a quicker solution.\n\ndiff --git a/builtin-mailinfo.c b/builtin-mailinfo.c\nindex 583da38..3fd8e00 100644\n--- a/builtin-mailinfo.c\n+++ b/builtin-mailinfo.c\n@@ -513,7 +513,7 @@ static void convert_to_utf8(char *line, char *charset)\n {\n \tstatic char latin_one[] = \"latin1\";\n \tchar *input_charset = *charset ? charset : latin_one;\n-\tchar *out = reencode_string(line, metainfo_charset, input_charset);\n+\tchar *out = reencode_string(line, metainfo_charset, input_charset, NULL);\n \n \tif (!out)\n \t\tdie(\"cannot convert from %s to %s\\n\",\ndiff --git a/commit.c b/commit.c\nindex 496d37a..8477fa7 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -486,6 +486,10 @@ static int add_rfc2047(char *buf, const char *line, int len)\n \tif (!needquote)\n \t\treturn sprintf(buf, \"%.*s\", len, line);\n \n+        if (git_commit_encoding)\n+                line = reencode_string(line, \"utf-8\",\n+                                       git_commit_encoding, &len);\n+\n \tmemcpy(bp, q_utf8, sizeof(q_utf8)-1);\n \tbp += sizeof(q_utf8)-1;\n \tfor (i = 0; i < len; i++) {\n@@ -501,6 +505,10 @@ static int add_rfc2047(char *buf, const char *line, int len)\n \t}\n \tmemcpy(bp, \"?=\", 2);\n \tbp += 2;\n+\n+        if (git_commit_encoding)\n+                free((char *)line);\n+\n \treturn bp - buf;\n }\n \n@@ -687,7 +695,7 @@ static char *logmsg_reencode(const struct commit *commit)\n \t\tout = strdup(commit->buffer);\n \telse\n \t\tout = reencode_string(commit->buffer,\n-\t\t\t\t      output_encoding, encoding);\n+\t\t\t\t      output_encoding, encoding, NULL);\n \tif (out)\n \t\tout = replace_encoding_header(out, output_encoding);\n \ndiff --git a/utf8.c b/utf8.c\nindex 7c80eec..ee9f514 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -291,7 +291,7 @@ int is_encoding_utf8(const char *name)\n  * with iconv.  If the conversion fails, returns NULL.\n  */\n #ifndef NO_ICONV\n-char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding)\n+char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding, int *len)\n {\n \ticonv_t conv;\n \tsize_t insz, outsz, outalloc;\n@@ -302,7 +302,10 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e\n \tconv = iconv_open(out_encoding, in_encoding);\n \tif (conv == (iconv_t) -1)\n \t\treturn NULL;\n-\tinsz = strlen(in);\n+        if (len)\n+                insz = *len;\n+        else\n+                insz = strlen(in);\n \toutsz = insz;\n \toutalloc = outsz + 1; /* for terminating NUL */\n \tout = xmalloc(outalloc);\n@@ -332,6 +335,8 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e\n \t\t}\n \t\telse {\n \t\t\t*outpos = '\\0';\n+                        if (len)\n+                                *len = outpos - out;\n \t\t\tbreak;\n \t\t}\n \t}\ndiff --git a/utf8.h b/utf8.h\nindex a07c5a8..eb64d46 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -8,7 +8,7 @@ int is_encoding_utf8(const char *name);\n void print_wrapped_text(const char *text, int indent, int indent2, int len);\n \n #ifndef NO_ICONV\n-char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding);\n+char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding, int *len);\n #else\n #define reencode_string(a,b,c) NULL\n #endif\n-- \n1.4.4.4.ge10a-dirty\n\n\n-- \nDavid Kågedal\n"},{"id":"31600","messageId":"7vd55jrj38.fsf@assigned-by-dhcp.cox.net","threadId":"6352","inReplyTo":"871wm08kcu.fsf@morpheus.local","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-12T22:11:39Z","receivedAt":"2007-01-12T22:11:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Kågedal <davidk@lysator.liu.se> writes:\n\n> The add_user_info function formats the commit as a mail message, and\n> uses add_rfc2047 to format the From: line.  The add_rfc2047 assumes\n> that the string is encoded as utf-8.\n\npretty_print_commit() labels the commit log message not just the\nauthor name also as UTF-8 when doing plain_non_ascii.\n\nIt might make more sense to just set the log_output_encoding to\nbe always UTF-8 when generating an e-mail output, in\ngit-format-patch.\n\n> diff --git a/utf8.h b/utf8.h\n> index a07c5a8..eb64d46 100644\n> --- a/utf8.h\n> +++ b/utf8.h\n> @@ -8,7 +8,7 @@ int is_encoding_utf8(const char *name);\n>  void print_wrapped_text(const char *text, int indent, int indent2, int len);\n>  \n>  #ifndef NO_ICONV\n> -char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding);\n> +char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding, int *len);\n>  #else\n>  #define reencode_string(a,b,c) NULL\n>  #endif\n\nThis feels fishy...\n"},{"id":"31618","messageId":"7vr6tzogp4.fsf@assigned-by-dhcp.cox.net","threadId":"6352","inReplyTo":"7vd55jrj38.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-13T01:31:35Z","receivedAt":"2007-01-13T01:31:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <junkio@cox.net> writes:\n\n> It might make more sense to just set the log_output_encoding to\n> be always UTF-8 when generating an e-mail output, in\n> git-format-patch.\n\nActually, I do not want to be an UTF-8 imperialist, so how about\ndoing this?\n\n-- >8 --\nUse log output encoding in --pretty=email headers.\n\nPrivate functions add_rfc2047() and pretty_print_commit() assumed\nthey are only emitting UTF-8.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n---\ndiff --git a/commit.c b/commit.c\nindex 496d37a..9b2b842 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -464,20 +464,29 @@ static int get_one_line(const char *msg, unsigned long len)\n \treturn ret;\n }\n \n+/* High bit set, or ISO-2022-INT */\n+static int non_ascii(int ch)\n+{\n+\tch = (ch & 0xff);\n+\treturn ((ch & 0x80) || (ch == 0x1b));\n+}\n+\n static int is_rfc2047_special(char ch)\n {\n-\treturn ((ch & 0x80) || (ch == '=') || (ch == '?') || (ch == '_'));\n+\treturn (non_ascii(ch) || (ch == '=') || (ch == '?') || (ch == '_'));\n }\n \n-static int add_rfc2047(char *buf, const char *line, int len)\n+static int add_rfc2047(char *buf, const char *line, int len,\n+\t\t       const char *encoding)\n {\n \tchar *bp = buf;\n \tint i, needquote;\n-\tstatic const char q_utf8[] = \"=?utf-8?q?\";\n+\tchar q_encoding[128];\n+\tconst char *q_encoding_fmt = \"=?%s?q?\";\n \n \tfor (i = needquote = 0; !needquote && i < len; i++) {\n-\t\tunsigned ch = line[i];\n-\t\tif (ch & 0x80)\n+\t\tint ch = line[i];\n+\t\tif (non_ascii(ch))\n \t\t\tneedquote++;\n \t\tif ((i + 1 < len) &&\n \t\t    (ch == '=' && line[i+1] == '?'))\n@@ -486,8 +495,11 @@ static int add_rfc2047(char *buf, const char *line, int len)\n \tif (!needquote)\n \t\treturn sprintf(buf, \"%.*s\", len, line);\n \n-\tmemcpy(bp, q_utf8, sizeof(q_utf8)-1);\n-\tbp += sizeof(q_utf8)-1;\n+\ti = snprintf(q_encoding, sizeof(q_encoding), q_encoding_fmt, encoding);\n+\tif (sizeof(q_encoding) < i)\n+\t\tdie(\"Insanely long encoding name %s\", encoding);\n+\tmemcpy(bp, q_encoding, i);\n+\tbp += i;\n \tfor (i = 0; i < len; i++) {\n \t\tunsigned ch = line[i] & 0xFF;\n \t\tif (is_rfc2047_special(ch)) {\n@@ -505,7 +517,8 @@ static int add_rfc2047(char *buf, const char *line, int len)\n }\n \n static int add_user_info(const char *what, enum cmit_fmt fmt, char *buf,\n-\t\t\t const char *line, int relative_date)\n+\t\t\t const char *line, int relative_date,\n+\t\t\t const char *encoding)\n {\n \tchar *date;\n \tint namelen;\n@@ -533,7 +546,8 @@ static int add_user_info(const char *what, enum cmit_fmt fmt, char *buf,\n \t\tfiller = \"\";\n \t\tstrcpy(buf, \"From: \");\n \t\tret = strlen(buf);\n-\t\tret += add_rfc2047(buf + ret, line, display_name_length);\n+\t\tret += add_rfc2047(buf + ret, line, display_name_length,\n+\t\t\t\t   encoding);\n \t\tmemcpy(buf + ret, name_tail, namelen - display_name_length);\n \t\tret += namelen - display_name_length;\n \t\tbuf[ret++] = '\\n';\n@@ -668,21 +682,18 @@ static char *replace_encoding_header(char *buf, char *encoding)\n \treturn buf;\n }\n \n-static char *logmsg_reencode(const struct commit *commit)\n+static char *logmsg_reencode(const struct commit *commit,\n+\t\t\t     char *output_encoding)\n {\n \tchar *encoding;\n \tchar *out;\n-\tchar *output_encoding = (git_log_output_encoding\n-\t\t\t\t ? git_log_output_encoding\n-\t\t\t\t : git_commit_encoding);\n+\tchar *utf8 = \"utf-8\";\n \n-\tif (!output_encoding)\n-\t\toutput_encoding = \"utf-8\";\n-\telse if (!*output_encoding)\n+\tif (!*output_encoding)\n \t\treturn NULL;\n \tencoding = get_header(commit, \"encoding\");\n \tif (!encoding)\n-\t\treturn NULL;\n+\t\tencoding = utf8;\n \tif (!strcmp(encoding, output_encoding))\n \t\tout = strdup(commit->buffer);\n \telse\n@@ -691,7 +702,8 @@ static char *logmsg_reencode(const struct commit *commit)\n \tif (out)\n \t\tout = replace_encoding_header(out, output_encoding);\n \n-\tfree(encoding);\n+\tif (encoding != utf8)\n+\t\tfree(encoding);\n \tif (!out)\n \t\treturn NULL;\n \treturn out;\n@@ -711,8 +723,15 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt,\n \tint parents_shown = 0;\n \tconst char *msg = commit->buffer;\n \tint plain_non_ascii = 0;\n-\tchar *reencoded = logmsg_reencode(commit);\n+\tchar *reencoded;\n+\tchar *encoding;\n \n+\tencoding = (git_log_output_encoding\n+\t\t    ? git_log_output_encoding\n+\t\t    : git_commit_encoding);\n+\tif (!encoding)\n+\t\tencoding = \"utf-8\";\n+\treencoded = logmsg_reencode(commit, encoding);\n \tif (reencoded)\n \t\tmsg = reencoded;\n \n@@ -738,7 +757,7 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt,\n \t\t\t\t    i + 1 < len && msg[i+1] == '\\n')\n \t\t\t\t\tin_body = 1;\n \t\t\t}\n-\t\t\telse if (ch & 0x80) {\n+\t\t\telse if (non_ascii(ch)) {\n \t\t\t\tplain_non_ascii = 1;\n \t\t\t\tbreak;\n \t\t\t}\n@@ -797,13 +816,15 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt,\n \t\t\t\toffset += add_user_info(\"Author\", fmt,\n \t\t\t\t\t\t\tbuf + offset,\n \t\t\t\t\t\t\tline + 7,\n-\t\t\t\t\t\t\trelative_date);\n+\t\t\t\t\t\t\trelative_date,\n+\t\t\t\t\t\t\tencoding);\n \t\t\tif (!memcmp(line, \"committer \", 10) &&\n \t\t\t    (fmt == CMIT_FMT_FULL || fmt == CMIT_FMT_FULLER))\n \t\t\t\toffset += add_user_info(\"Commit\", fmt,\n \t\t\t\t\t\t\tbuf + offset,\n \t\t\t\t\t\t\tline + 10,\n-\t\t\t\t\t\t\trelative_date);\n+\t\t\t\t\t\t\trelative_date,\n+\t\t\t\t\t\t\tencoding);\n \t\t\tcontinue;\n \t\t}\n \n@@ -826,7 +847,8 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt,\n \t\t\tint slen = strlen(subject);\n \t\t\tmemcpy(buf + offset, subject, slen);\n \t\t\toffset += slen;\n-\t\t\toffset += add_rfc2047(buf + offset, line, linelen);\n+\t\t\toffset += add_rfc2047(buf + offset, line, linelen,\n+\t\t\t\t\t      encoding);\n \t\t}\n \t\telse {\n \t\t\tmemset(buf + offset, ' ', indent);\n@@ -837,11 +859,17 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt,\n \t\tif (fmt == CMIT_FMT_ONELINE)\n \t\t\tbreak;\n \t\tif (subject && plain_non_ascii) {\n-\t\t\tstatic const char header[] =\n-\t\t\t\t\"Content-Type: text/plain; charset=UTF-8\\n\"\n+\t\t\tint sz;\n+\t\t\tchar header[512];\n+\t\t\tconst char *header_fmt =\n+\t\t\t\t\"Content-Type: text/plain; charset=%s\\n\"\n \t\t\t\t\"Content-Transfer-Encoding: 8bit\\n\";\n-\t\t\tmemcpy(buf + offset, header, sizeof(header)-1);\n-\t\t\toffset += sizeof(header)-1;\n+\t\t\tsz = snprintf(header, sizeof(header), header_fmt,\n+\t\t\t\t      encoding);\n+\t\t\tif (sizeof(header) < sz)\n+\t\t\t\tdie(\"Encoding name %s too long\", encoding);\n+\t\t\tmemcpy(buf + offset, header, sz);\n+\t\t\toffset += sz;\n \t\t}\n \t\tif (after_subject) {\n \t\t\tint slen = strlen(after_subject);\n"},{"id":"31619","messageId":"7vmz4nog5b.fsf@assigned-by-dhcp.cox.net","threadId":"6352","inReplyTo":"7vr6tzogp4.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-13T01:43:28Z","receivedAt":"2007-01-13T01:43:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Side note.  The previous patch does not help if your commit were\nmade in non UTF-8 with not too recent git; the code assumes that\ncommit messages without the new \"encoding\" headers are in UTF-8.\n\nWe might want to help transitioning people by doing something\nlike this on top of the previous patch.  Then when dealing with\nan ancient commit (sorry, I am not saying commits older than 3\nweeks are ancient -- but it will be 6 months from now ;-), you\ncan override that default by setting an environment variable.\n\n---\ndiff --git a/commit.c b/commit.c\nindex 9b2b842..a1b5705 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -692,8 +692,12 @@ static char *logmsg_reencode(const struct commit *commit,\n \tif (!*output_encoding)\n \t\treturn NULL;\n \tencoding = get_header(commit, \"encoding\");\n-\tif (!encoding)\n-\t\tencoding = utf8;\n+\tif (!encoding) {\n+\t\tif (getenv(\"GIT_OLD_COMMIT_ENCODING\"))\n+\t\t\tencoding = strdup(getenv(\"GIT_OLD_COMMIT_ENCODING\"));\n+\t\telse\n+\t\t\tencoding = utf8;\n+\t}\n \tif (!strcmp(encoding, output_encoding))\n \t\tout = strdup(commit->buffer);\n \telse\n"},{"id":"31630","messageId":"20070113110238.GA2795@steel.home","threadId":"6352","inReplyTo":"7vr6tzogp4.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Alex Riesen","fromEmail":"fork0@t-online.de","sentAt":"2007-01-13T11:02:38Z","receivedAt":"2007-01-13T11:02:38Z","isPatch":true,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Junio C Hamano, Sat, Jan 13, 2007 02:31:35 +0100:\n> +/* High bit set, or ISO-2022-INT */\n> +static int non_ascii(int ch)\n> +{\n> +\tch = (ch & 0xff);\n> +\treturn ((ch & 0x80) || (ch == 0x1b));\n> +}\n> +\n\n\"return (ch & 0x0x80) || (ch & 0xff) == 0x1b;\" :)\n"},{"id":"31632","messageId":"Pine.LNX.4.63.0701131217410.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6352","inReplyTo":"7vmz4nog5b.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-13T11:19:47Z","receivedAt":"2007-01-13T11:19:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 12 Jan 2007, Junio C Hamano wrote:\n\n> Side note.  The previous patch does not help if your commit were\n> made in non UTF-8 with not too recent git; the code assumes that\n> commit messages without the new \"encoding\" headers are in UTF-8.\n\nWhy not just use is_utf8() and warn, or error out, if the message is not \nUTF-8? (I tend towards the erroring out, since this _is_ a new feature, \nand gives undesired results with \"old\" commits.)\n\nCiao,\nDscho\n"},{"id":"31634","messageId":"200701131323.53121.robin.rosenberg.lists@dewire.com","threadId":"6352","inReplyTo":"7vmz4nog5b.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2007-01-13T12:23:52Z","receivedAt":"2007-01-13T12:23:52Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"lördag 13 januari 2007 02:43 skrev Junio C Hamano:\n> Side note.  The previous patch does not help if your commit were\n> made in non UTF-8 with not too recent git; the code assumes that\n> commit messages without the new \"encoding\" headers are in UTF-8.\n\nWasn't there a repository option, \"commitencoding\"?  I can't see it being\nused here. I.e., we should err out if the log message is not UTF-8 and the \noption is not set (giving a message telling the user to set it).  If it is \nset we should consider the repository encoding to be the one and if that too \nis wrong (only possible to detect for some encodings), just assume iso-8859-1 \nas anything could in theory be iso-8859-1 encoded.  \n\n-- robin\n"},{"id":"31643","messageId":"7vd55ilsm7.fsf@assigned-by-dhcp.cox.net","threadId":"6352","inReplyTo":"200701131323.53121.robin.rosenberg.lists@dewire.com","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-13T17:54:40Z","receivedAt":"2007-01-13T17:54:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robin Rosenberg <robin.rosenberg.lists@dewire.com> writes:\n\n> lördag 13 januari 2007 02:43 skrev Junio C Hamano:\n>> Side note.  The previous patch does not help if your commit were\n>> made in non UTF-8 with not too recent git; the code assumes that\n>> commit messages without the new \"encoding\" headers are in UTF-8.\n>\n> Wasn't there a repository option, \"commitencoding\"?  I can't see it being\n> used here.\n\ncommitencoding is about what encoding the commit newly created\nin this repository right now should claim to have -- in other\nwords what is fed to commit-tree.\n\nWe are talking about examining existing commit that might have\ncome from another repository or created some time ago when the\nrepository configuration was set differently.\n"},{"id":"31644","messageId":"7v8xg6lsh6.fsf@assigned-by-dhcp.cox.net","threadId":"6352","inReplyTo":"Pine.LNX.4.63.0701131217410.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-13T17:57:41Z","receivedAt":"2007-01-13T17:57:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Why not just use is_utf8() and warn, or error out, if the message is not \n> UTF-8? (I tend towards the erroring out, since this _is_ a new feature, \n> and gives undesired results with \"old\" commits.)\n\nThat sounds sensible.\n"},{"id":"31659","messageId":"7vfyaek1ub.fsf@assigned-by-dhcp.cox.net","threadId":"6352","inReplyTo":"7vr6tzogp4.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-13T22:18:20Z","receivedAt":"2007-01-13T22:18:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On this topic, along with the \"format-patch\" fix (which\nautomatically makes \"rebase without --merge\" do the right thing\nbecause it is \"format-patch piped to am\" in essence), I have\nanother commit to make \"cherry-pick\", \"rebase --merge\" and\n\"commit -c/-C\" do the right thing according to the\ncommitencoding specified in the repository the new commit is\nbeing created.\n\nThe issue is that an existing commit might have come from a\ndifferent repository or from the past when this repository had\ncommitencoding that was different from the current value.\nRunning \"cat-file commit\" to extract the old commit log message\nand feeding it directly to create the new commit would not work,\nbecause the value of commitencoding in this repository may be\ndifferent.\n\nThis should not affect old encoding-unaware setup where people\nuse _only_ a legacy encoding and do not bother to specify any\ncommitencoding.  In such a case, both input and output are the\nsame and while we pretend both are UTF-8, we actually do not\ntrigger conversion.  To support such a configuration is one\nreason I did not actually take Johannes's suggestion to error\nout on an existing commit that does _not_ have encoding header\nbut the contents does not look like a valid UTF-8.\n\nThe series is currently sitting in 'next'.  If people do not see\nproblem with it, I think it should go in v1.5.0.\n"},{"id":"31667","messageId":"200701140042.l0E0gvMT005024@laptop13.inf.utfsm.cl","threadId":"6352","inReplyTo":"20070113110238.GA2795@steel.home","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Horst H. von Brand","fromEmail":"vonbrand@inf.utfsm.cl","sentAt":"2007-01-14T00:42:57Z","receivedAt":"2007-01-14T00:42:57Z","isPatch":true,"sender":{"key":"vonbrand@inf.utfsm.cl","avatar":"https://avatars.githubusercontent.com/u/211384?v=4"},"body":"Alex Riesen <fork0@t-online.de> wrote:\n> Junio C Hamano, Sat, Jan 13, 2007 02:31:35 +0100:\n> > +/* High bit set, or ISO-2022-INT */\n> > +static int non_ascii(int ch)\n> > +{\n> > +\tch = (ch & 0xff);\n> > +\treturn ((ch & 0x80) || (ch == 0x1b));\n> > +}\n> > +\n> \n> \"return (ch & 0x0x80) || (ch & 0xff) == 0x1b;\" :)\n                ^^\n\nIs the same, if ch == 0x9b, it will match the first part anyway.\n\nThe outer parentesis can (should?) go.\n-- \nDr. Horst H. von Brand                   User #22616 counter.li.org\nDepartamento de Informatica                    Fono: +56 32 2654431\nUniversidad Tecnica Federico Santa Maria             +56 32 2654239\nCasilla 110-V, Valparaiso, Chile               Fax:  +56 32 2797513\n"},{"id":"31749","messageId":"20070114192557.GA2378@steel.home","threadId":"6352","inReplyTo":"200701140042.l0E0gvMT005024@laptop13.inf.utfsm.cl","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Alex Riesen","fromEmail":"fork0@t-online.de","sentAt":"2007-01-14T19:25:57Z","receivedAt":"2007-01-14T19:25:57Z","isPatch":true,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Horst H. von Brand, Sun, Jan 14, 2007 01:42:57 +0100:\n> Alex Riesen <fork0@t-online.de> wrote:\n> > Junio C Hamano, Sat, Jan 13, 2007 02:31:35 +0100:\n> > > +/* High bit set, or ISO-2022-INT */\n> > > +static int non_ascii(int ch)\n> > > +{\n> > > +\tch = (ch & 0xff);\n> > > +\treturn ((ch & 0x80) || (ch == 0x1b));\n> > > +}\n> > > +\n> > \n> > \"return (ch & 0x0x80) || (ch & 0xff) == 0x1b;\" :)\n>                 ^^\n\nOops :)\n\n> Is the same, if ch == 0x9b, it will match the first part anyway.\n\nSo it should. 0x9b isn't ASCII.\n\n> The outer parentesis can (should?) go.\n\nIt's \"question of style\", I'm afraid :)\n"},{"id":"31773","messageId":"87k5zo44fq.fsf@morpheus.local","threadId":"6352","inReplyTo":"7vd55jrj38.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"David Kågedal","fromEmail":"davidk@lysator.liu.se","sentAt":"2007-01-15T16:53:29Z","receivedAt":"2007-01-15T16:53:29Z","isPatch":true,"sender":{"key":"davidk@lysator.liu.se","avatar":"https://avatars.githubusercontent.com/u/60530?v=4"},"body":"Junio C Hamano <junkio@cox.net> writes:\n\n>> diff --git a/utf8.h b/utf8.h\n>> index a07c5a8..eb64d46 100644\n>> --- a/utf8.h\n>> +++ b/utf8.h\n>> @@ -8,7 +8,7 @@ int is_encoding_utf8(const char *name);\n>>  void print_wrapped_text(const char *text, int indent, int indent2, int len);\n>>  \n>>  #ifndef NO_ICONV\n>> -char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding);\n>> +char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding, int *len);\n>>  #else\n>>  #define reencode_string(a,b,c) NULL\n>>  #endif\n>\n> This feels fishy...\n\nI admit that I didn't test-compile with NO_ICONV.\n\n-- \nDavid Kågedal\n"},{"id":"31774","messageId":"87fyac44e9.fsf@morpheus.local","threadId":"6352","inReplyTo":"7vmz4nog5b.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"David Kågedal","fromEmail":"davidk@lysator.liu.se","sentAt":"2007-01-15T16:54:22Z","receivedAt":"2007-01-15T16:54:22Z","isPatch":true,"sender":{"key":"davidk@lysator.liu.se","avatar":"https://avatars.githubusercontent.com/u/60530?v=4"},"body":"Junio C Hamano <junkio@cox.net> writes:\n\n> Side note.  The previous patch does not help if your commit were\n> made in non UTF-8 with not too recent git; the code assumes that\n> commit messages without the new \"encoding\" headers are in UTF-8.\n\nThis was exactly the problem I was trying to solve.\n\n> We might want to help transitioning people by doing something\n> like this on top of the previous patch.  Then when dealing with\n> an ancient commit (sorry, I am not saying commits older than 3\n> weeks are ancient -- but it will be 6 months from now ;-), you\n> can override that default by setting an environment variable.\n>\n> ---\n> diff --git a/commit.c b/commit.c\n> index 9b2b842..a1b5705 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -692,8 +692,12 @@ static char *logmsg_reencode(const struct commit *commit,\n>  \tif (!*output_encoding)\n>  \t\treturn NULL;\n>  \tencoding = get_header(commit, \"encoding\");\n> -\tif (!encoding)\n> -\t\tencoding = utf8;\n> +\tif (!encoding) {\n> +\t\tif (getenv(\"GIT_OLD_COMMIT_ENCODING\"))\n> +\t\t\tencoding = strdup(getenv(\"GIT_OLD_COMMIT_ENCODING\"));\n> +\t\telse\n> +\t\t\tencoding = utf8;\n> +\t}\n>  \tif (!strcmp(encoding, output_encoding))\n>  \t\tout = strdup(commit->buffer);\n>  \telse\n>\n>\n>\n\n-- \nDavid Kågedal\n"},{"id":"31776","messageId":"87bql0449j.fsf@morpheus.local","threadId":"6352","inReplyTo":"7vr6tzogp4.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"David Kågedal","fromEmail":"davidk@lysator.liu.se","sentAt":"2007-01-15T16:57:12Z","receivedAt":"2007-01-15T16:57:12Z","isPatch":true,"sender":{"key":"davidk@lysator.liu.se","avatar":"https://avatars.githubusercontent.com/u/60530?v=4"},"body":"Junio C Hamano <junkio@cox.net> writes:\n\n> -static int add_rfc2047(char *buf, const char *line, int len)\n> +static int add_rfc2047(char *buf, const char *line, int len,\n> +\t\t       const char *encoding)\n>  {\n>  \tchar *bp = buf;\n>  \tint i, needquote;\n> -\tstatic const char q_utf8[] = \"=?utf-8?q?\";\n> +\tchar q_encoding[128];\n> +\tconst char *q_encoding_fmt = \"=?%s?q?\";\n\nThis goes against the old principle of being forgiving in what you\naccept, and strict in what you send.  The names of the encoding in the\nheaders should probably be normalized before putting them in an\ne-mail.  I.e. we might accept \"utf-8\", \"utf8\", \"UTF-8\", and \"UTF8\"\n(this depends on iconv, I suppose), but the RFC2047 encoding should be\nthe one blessed by RFC4027.  But I admit that I haven't read the RFC,\nand I'm writing this offline so I can't check right now.\n\n-- \nDavid Kågedal\n"},{"id":"31775","messageId":"877ivo446n.fsf@morpheus.local","threadId":"6352","inReplyTo":"Pine.LNX.4.63.0701131217410.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"David Kågedal","fromEmail":"davidk@lysator.liu.se","sentAt":"2007-01-15T16:58:56Z","receivedAt":"2007-01-15T16:58:56Z","isPatch":true,"sender":{"key":"davidk@lysator.liu.se","avatar":"https://avatars.githubusercontent.com/u/60530?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Hi,\n>\n> On Fri, 12 Jan 2007, Junio C Hamano wrote:\n>\n>> Side note.  The previous patch does not help if your commit were\n>> made in non UTF-8 with not too recent git; the code assumes that\n>> commit messages without the new \"encoding\" headers are in UTF-8.\n>\n> Why not just use is_utf8() and warn, or error out, if the message is not \n> UTF-8? (I tend towards the erroring out, since this _is_ a new feature, \n> and gives undesired results with \"old\" commits.)\n\nWhat do you mean? I have an old repository with latin1 commits without\nany encoding markers.  I want to be able to use format-patch from that\nand at least get a From: line with something readable.  You can't just\nbarf and say \"This isn't UTF-8, go away\".\n\n-- \nDavid Kågedal\n"},{"id":"31815","messageId":"Pine.LNX.4.63.0701161239590.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6352","inReplyTo":"877ivo446n.fsf@morpheus.local","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-16T11:41:37Z","receivedAt":"2007-01-16T11:41:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 15 Jan 2007, David Kågedal wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> > On Fri, 12 Jan 2007, Junio C Hamano wrote:\n> >\n> >> Side note.  The previous patch does not help if your commit were\n> >> made in non UTF-8 with not too recent git; the code assumes that\n> >> commit messages without the new \"encoding\" headers are in UTF-8.\n> >\n> > Why not just use is_utf8() and warn, or error out, if the message is not \n> > UTF-8? (I tend towards the erroring out, since this _is_ a new feature, \n> > and gives undesired results with \"old\" commits.)\n> \n> What do you mean? I have an old repository with latin1 commits without\n> any encoding markers.  I want to be able to use format-patch from that\n> and at least get a From: line with something readable.  You can't just\n> barf and say \"This isn't UTF-8, go away\".\n\nSo what do you want to do instead? Just pretend that the unrecoded -- \nLatin-1 encoded -- text is UTF-8? That's plain wrong.\n\nCiao,\nDscho\n"},{"id":"31817","messageId":"873b6b87lw.fsf@morpheus.local","threadId":"6352","inReplyTo":"Pine.LNX.4.63.0701161239590.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Reencode committer info to utf-8 before formatting mail header","fromName":"David Kågedal","fromEmail":"davidk@lysator.liu.se","sentAt":"2007-01-16T12:43:39Z","receivedAt":"2007-01-16T12:43:39Z","isPatch":true,"sender":{"key":"davidk@lysator.liu.se","avatar":"https://avatars.githubusercontent.com/u/60530?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Hi,\n>\n> On Mon, 15 Jan 2007, David Kågedal wrote:\n>\n>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n>> \n>> > On Fri, 12 Jan 2007, Junio C Hamano wrote:\n>> >\n>> >> Side note.  The previous patch does not help if your commit were\n>> >> made in non UTF-8 with not too recent git; the code assumes that\n>> >> commit messages without the new \"encoding\" headers are in UTF-8.\n>> >\n>> > Why not just use is_utf8() and warn, or error out, if the message is not \n>> > UTF-8? (I tend towards the erroring out, since this _is_ a new feature, \n>> > and gives undesired results with \"old\" commits.)\n>> \n>> What do you mean? I have an old repository with latin1 commits without\n>> any encoding markers.  I want to be able to use format-patch from that\n>> and at least get a From: line with something readable.  You can't just\n>> barf and say \"This isn't UTF-8, go away\".\n>\n> So what do you want to do instead? Just pretend that the unrecoded -- \n> Latin-1 encoded -- text is UTF-8? That's plain wrong.\n\nThat is what git did before I wrote my patch, so it obviously not what\nI want.  I want to be able to tell git what encoding it is.\n\nMy patch reused the i18n.commitencoding configuration parameter for\nthat, but Junio is probably right in that that is only meant for new\ncommits, and an evironment variable makes more sense.\n\nSo just barfing on a commit that isn't utf-8 isn't a complete\nsolution.  But maybe there was some context to your comment above that\nI missed.\n\n-- \nDavid Kågedal\n"}]}