{"thread":{"id":"30939","subject":"[PATCH] Fix Q-encoded multi-octet-char split in email.","startedAt":"2012-07-03T01:41:37Z","lastAt":"2012-08-16T21:52:45Z","messageCount":8,"participants":["katsu","Jeff King","Erik Faye-Lund","Junio C Hamano","Drew Northup"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"194504","messageId":"1341279697-4596-1-git-send-email-gkatsu.ne@gmail.com","threadId":"30939","inReplyTo":null,"subject":"[PATCH] Fix Q-encoded multi-octet-char split in email.","fromName":"katsu","fromEmail":"gkatsu.ne@gmail.com","sentAt":"2012-07-03T01:41:37Z","receivedAt":"2012-07-03T01:41:37Z","isPatch":true,"sender":{"key":"gkatsu.ne@gmail.com","avatar":null},"body":"Issue: Email subject written in multi-octet language like japanese cannot\nbe displayed in correct at destinations's email client, because the\nQ-encoded subject which is longer than 78 octets is split by a octet not by\na character at line breaks.\ne.g.)\n \"=?utf-8?q? [PATCH] ... =E8=83=86=E8=81=A9?=\"\n |\n V\n \"=?utf-8?q? [PATCH] ... =E8=83=86=E8?=\"\n \"=?utf-8?q?=81=A9=?\"\n\nChanges: Add a judge if a character is an part of utf-8 muti-octet, and\nsplit the characters by a character not by a octet at line breaks in\nfunction add_rfc2407() in pretty.c. Like following.\n\n \"=?utf-8?q? [PATCH] ... =E8=83=86?=\"\n \"=?utf-8?q?=E8=81=A9=?\"\n\nSigned-off-by: Takeharu Katsuyama \n---\n pretty.c | 29 ++++++++++++++++++++++++++++-\n 1 files changed, 28 insertions(+), 1 deletions(-)\n mode change 100644 => 100755 pretty.c\n\ndiff --git a/pretty.c b/pretty.c\nold mode 100644\nnew mode 100755\nindex 8b1ea9f..266a8fe\n--- a/pretty.c\n+++ b/pretty.c\n@@ -272,6 +272,12 @@ static void add_rfc2047(struct strbuf *sb, const char *line, int len,\n \tstatic const int max_length = 78; /* per rfc2822 */\n \tint i;\n \tint line_len;\n+\tint utf_ctr, use_utf;\n+\n+\tif (!strcmp(encoding, \"UTF-8\") || !strcmp(encoding, \"utf-8\"))\n+\t\tuse_utf = 1;\n+\telse\n+\t\tuse_utf = 0;\n \n \t/* How many bytes are already used on the current line? */\n \tfor (i = sb->len - 1; i >= 0; i--)\n@@ -293,10 +299,31 @@ needquote:\n \tstrbuf_grow(sb, len * 3 + strlen(encoding) + 100);\n \tstrbuf_addf(sb, \"=?%s?q?\", encoding);\n \tline_len += strlen(encoding) + 5; /* 5 for =??q? */\n+\tutf_ctr = 0;\n \tfor (i = 0; i < len; i++) {\n \t\tunsigned ch = line[i] & 0xFF;\n \n-\t\tif (line_len >= max_length - 2) {\n+\t\t/*\n+\t\t * Judge if it is an utf-8 char, to avoid inserting newline\n+\t\t * in the middle of utf-8 char code.\n+\t\t */\n+\t\tif (use_utf) {\n+\t\t\tif (ch >= 0xC2 && ch <= 0xDF)\t/* 1'st byte of 2-bytes utf-8 */\n+\t\t\t\tutf_ctr = 1;\n+\t\t\telse if (ch >= 0xE0 && ch <= 0xEF)\t/* 3-bytes utf-8 */\n+\t\t\t\tutf_ctr = 2;\n+\t\t\telse if (ch >= 0xF0 && ch <= 0xF7)\t/* 4-bytes utf-8 */\n+\t\t\t\tutf_ctr = 3;\n+\t\t\telse if (ch >= 0xF8 && ch <= 0xFB)\t/* 5-bytes utf-8 */\n+\t\t\t\tutf_ctr = 4;\n+\t\t\telse if (ch >= 0xFC && ch <= 0xFD)\t/* 6-bytes utf-8 */\n+\t\t\t\tutf_ctr = 5;\n+\t\t\telse if (ch >= 0x80 && ch <= 0xBF) /* 2'nd to 6'th byte of utf-8 */\n+\t\t\t\tutf_ctr--;\n+\t\t\telse\n+\t\t\t\tutf_ctr = 0;\n+\t\t}\n+\t\tif (line_len >= (max_length - 2 - utf_ctr *3)) {\n \t\t\tstrbuf_addf(sb, \"?=\\n =?%s?q?\", encoding);\n \t\t\tline_len = strlen(encoding) + 5 + 1; /* =??q? plus SP */\n \t\t}\n-- \n1.7.9\n"},{"id":"194515","messageId":"20120703063511.GA16679@sigill.intra.peff.net","threadId":"30939","inReplyTo":"1341279697-4596-1-git-send-email-gkatsu.ne@gmail.com","subject":"Re: [PATCH] Fix Q-encoded multi-octet-char split in email.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-07-03T06:35:11Z","receivedAt":"2012-07-03T06:35:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 03, 2012 at 10:41:37AM +0900, katsu wrote:\n\n> Issue: Email subject written in multi-octet language like japanese cannot\n> be displayed in correct at destinations's email client, because the\n> Q-encoded subject which is longer than 78 octets is split by a octet not by\n> a character at line breaks.\n> e.g.)\n> \"=?utf-8?q? [PATCH] ... =E8=83=86=E8=81=A9?=\"\n> |\n> V\n> \"=?utf-8?q? [PATCH] ... =E8=83=86=E8?=\"\n> \"=?utf-8?q?=81=A9=?\"\n> \n> Changes: Add a judge if a character is an part of utf-8 muti-octet, and\n> split the characters by a character not by a octet at line breaks in\n> function add_rfc2407() in pretty.c. Like following.\n> \n> \"=?utf-8?q? [PATCH] ... =E8=83=86?=\"\n> \"=?utf-8?q?=E8=81=A9=?\"\n> \n> Signed-off-by: Takeharu Katsuyama \n\nYeah, we definitely don't handle that properly according to the rfc.\nThis patch is is going in the right direction, but I have a few\ncomments:\n\n> --- a/pretty.c\n> +++ b/pretty.c\n> @@ -272,6 +272,12 @@ static void add_rfc2047(struct strbuf *sb, const char *line, int len,\n> \tstatic const int max_length = 78; /* per rfc2822 */\n> \tint i;\n> \tint line_len;\n> +\tint utf_ctr, use_utf;\n> +\n> +\tif (!strcmp(encoding, \"UTF-8\") || !strcmp(encoding, \"utf-8\"))\n> +\t\tuse_utf = 1;\n> +\telse\n> +\t\tuse_utf = 0;\n\nPlease use is_encoding_utf8, which handles both of these spellings, as\nwell as \"utf8\" and \"UTF8\" (it also handles encoding==NULL; I don't think\nthat can happen in this code path, but it is nice to be defensive).\n\n> @@ -293,10 +299,31 @@ needquote:\n> \tstrbuf_grow(sb, len * 3 + strlen(encoding) + 100);\n> \tstrbuf_addf(sb, \"=?%s?q?\", encoding);\n> \tline_len += strlen(encoding) + 5; /* 5 for =??q? */\n> +\tutf_ctr = 0;\n> \tfor (i = 0; i < len; i++) {\n> \t\tunsigned ch = line[i] & 0xFF;\n> \n> -\t\tif (line_len >= max_length - 2) {\n> +\t\t/*\n> +\t\t * Judge if it is an utf-8 char, to avoid inserting newline\n> +\t\t * in the middle of utf-8 char code.\n> +\t\t */\n> +\t\tif (use_utf) {\n> +\t\t\tif (ch >= 0xC2 && ch <= 0xDF)\t/* 1'st byte of 2-bytes utf-8 */\n> +\t\t\t\tutf_ctr = 1;\n> +\t\t\telse if (ch >= 0xE0 && ch <= 0xEF)\t/* 3-bytes utf-8 */\n> +\t\t\t\tutf_ctr = 2;\n> +\t\t\telse if (ch >= 0xF0 && ch <= 0xF7)\t/* 4-bytes utf-8 */\n> +\t\t\t\tutf_ctr = 3;\n> +\t\t\telse if (ch >= 0xF8 && ch <= 0xFB)\t/* 5-bytes utf-8 */\n> +\t\t\t\tutf_ctr = 4;\n> +\t\t\telse if (ch >= 0xFC && ch <= 0xFD)\t/* 6-bytes utf-8 */\n> +\t\t\t\tutf_ctr = 5;\n> +\t\t\telse if (ch >= 0x80 && ch <= 0xBF) /* 2'nd to 6'th byte of utf-8 */\n> +\t\t\t\tutf_ctr--;\n> +\t\t\telse\n> +\t\t\t\tutf_ctr = 0;\n> +\t\t}\n> +\t\tif (line_len >= (max_length - 2 - utf_ctr *3)) {\n\nCan we re-use utf8_width here instead of rewriting these rules?\n\n-Peff\n"},{"id":"194521","messageId":"CABPQNSYbdjzrHyjsXu23ARFBEiZ_a6S+iEn0Pwy-1HaKTTAheQ@mail.gmail.com","threadId":"30939","inReplyTo":"1341279697-4596-1-git-send-email-gkatsu.ne@gmail.com","subject":"Re: [PATCH] Fix Q-encoded multi-octet-char split in email.","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2012-07-03T09:52:20Z","receivedAt":"2012-07-03T09:52:20Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, Jul 3, 2012 at 3:41 AM, katsu wrote:\n> Issue: Email subject written in multi-octet language like japanese cannot\n> be displayed in correct at destinations's email client, because the\n> Q-encoded subject which is longer than 78 octets is split by a octet not by\n> a character at line breaks.\n> e.g.)\n> \"=?utf-8?q? [PATCH] ... =E8=83=86=E8=81=A9?=\"\n> |\n> V\n> \"=?utf-8?q? [PATCH] ... =E8=83=86=E8?=\"\n> \"=?utf-8?q?=81=A9=?\"\n>\n> Changes: Add a judge if a character is an part of utf-8 muti-octet, and\n> split the characters by a character not by a octet at line breaks in\n> function add_rfc2407() in pretty.c.\n\nYou mean add_rfc2047(), right?\n\nAnyway, I'm not an expert here, but can't a \"soft\" newline (as\nspecified in rfc 2045) be used in message headers? If it could, then\nwe wouldn't need to grok the underlying encoding when wrapping, which\nstrikes me as slightly better...\n"},{"id":"194597","messageId":"20120704064450.GA24807@sigill.intra.peff.net","threadId":"30939","inReplyTo":"CAGxub4-9E0W8ZgsPHeTyUyxmPD80LUd7NjSezg5Zt2-nZPBMJA@mail.gmail.com","subject":"Re: [PATCH] Fix Q-encoded multi-octet-char split in email.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-07-04T06:44:50Z","receivedAt":"2012-07-04T06:44:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 04, 2012 at 03:19:31PM +0900, Katsuyama Takeharu wrote:\n\n> diff --git a/pretty.c b/pretty.c\n> --- a/pretty.c\n> +++ b/pretty.c\n> @@ -272,6 +272,13 @@ static void add_rfc2047(struct strbuf *sb, const char\n> *line, int len,\n> static const int max_length = 78; /* per rfc2822 */\n> int i;\n> int line_len;\n> + int utf8_ctr, use_utf8;\n> + const char *utf8_start;\n> +\n> + if (is_encoding_utf8(encoding) && encoding != NULL)\n> + use_utf8 = 1;\n> + else\n> + use_utf8 = 0;\n\nI think you can drop the \"encoding != NULL\" here. If we don't have an\nexplicit encoding, git always assumes utf8 (also, as it happens we never\nhit this point with a NULL encoding in the current code anyway, though\nthat could in theory change in the future).\n\n> > Can we re-use utf8_width here instead of rewriting these rules?\n> \n> Yes you can. But there are an issue which utf8_width seems not to return\n> correct value. It returns 3 even if a provided code has 3 octet utf-8\n> char(e.g. 0xE38292).\n> I expect it returns 2.\n\nHmm. I think I may have led you astray. It seems that the return value\nof utf8_width is not about the byte-width of the character\nrepresentation, but rather about the intended character-width of the\nglyph. But since we are encoding the bytes, we care about the former.\n\nSo I think you would really want to use pick_one_utf8_char and see how\nmany characters it consumed, like this:\n\n const char *p = &line[i];\n pick_one_utf8_char(&p, NULL);\n if (!p) /* not valid utf8, just assume single byte */\n utf8_ctr = 1;\n else\n utf8_ctr = p - &line[i];\n\n-Peff\n"},{"id":"195243","messageId":"7v394py7r2.fsf@alter.siamese.dyndns.org","threadId":"30939","inReplyTo":"20120704064450.GA24807@sigill.intra.peff.net","subject":"Re: [PATCH] Fix Q-encoded multi-octet-char split in email.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-18T05:10:25Z","receivedAt":"2012-07-18T05:10:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ping on a seemingly stalled thread.\n"},{"id":"195262","messageId":"20120718072753.GC12942@sigill.intra.peff.net","threadId":"30939","inReplyTo":"7v394py7r2.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] Fix Q-encoded multi-octet-char split in email.","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-07-18T07:27:53Z","receivedAt":"2012-07-18T07:27:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 17, 2012 at 10:10:25PM -0700, Junio C Hamano wrote:\n\n> Ping on a seemingly stalled thread.\n\nHrm. I could swear that Takeharu sent a follow-up using\npick_one_utf8_char directly that looked OK, but now I can't seem to find\nit in the list archives. I wonder if it was off-list and I didn't\nrealize it.\n\nIf I did not simply dream it, can you re-post the latest patch you\nsent?\n\n-Peff\n"},{"id":"195737","messageId":"CAM9Z-nk97tkE4==Kx8z-1gwKoeUqj087wE2X1q=fUVuK+ckqpA@mail.gmail.com","threadId":"30939","inReplyTo":"20120718072753.GC12942@sigill.intra.peff.net","subject":"Re: [PATCH] Fix Q-encoded multi-octet-char split in email.","fromName":"Drew Northup","fromEmail":"n1xim.email@gmail.com","sentAt":"2012-07-25T11:10:51Z","receivedAt":"2012-07-25T11:10:51Z","isPatch":true,"sender":{"key":"n1xim.email@gmail.com","avatar":null},"body":"On Wed, Jul 18, 2012 at 3:27 AM, Jeff King wrote:\n> On Tue, Jul 17, 2012 at 10:10:25PM -0700, Junio C Hamano wrote:\n>\n>> Ping on a seemingly stalled thread.\n\n> If I did not simply dream it, can you re-post the latest patch you\n> sent?\n>\n> -Peff\n\n...And I thought dreaming I was trapped in a DOS prompt was bad.\nDreaming in patches? OUCH!\n\n-- \n-Drew Northup\n--------------------------------------------------------------\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"197124","messageId":"7vr4r6zeqa.fsf@alter.siamese.dyndns.org","threadId":"30939","inReplyTo":"20120718072753.GC12942@sigill.intra.peff.net","subject":"Re: [PATCH] Fix Q-encoded multi-octet-char split in email.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-16T21:52:45Z","receivedAt":"2012-08-16T21:52:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Tue, Jul 17, 2012 at 10:10:25PM -0700, Junio C Hamano wrote:\n>\n>> Ping on a seemingly stalled thread.\n>\n> Hrm. I could swear that Takeharu sent a follow-up using\n> pick_one_utf8_char directly that looked OK, but now I can't seem to find\n> it in the list archives. I wonder if it was off-list and I didn't\n> realize it.\n>\n> If I did not simply dream it, can you re-post the latest patch you\n> sent?\n\nAnother ping on a seemingly stalled thread.\n"}]}