# [PATCH] Reencode committer info to utf-8 before formatting mail header 18 messages from 2007-01-12 to 2007-01-16. Participants: David Kågedal, Junio C Hamano, Alex Riesen, Johannes Schindelin, Robin Rosenberg, Horst H. von Brand. Thread: https://gitlist.dev/t/6352 ## David Kågedal, 2007-01-12 13:06 Subject: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <871wm08kcu.fsf@morpheus.local> URL: https://gitlist.dev/e/871wm08kcu.fsf%40morpheus.local ``` The add_user_info function formats the commit as a mail message, and uses add_rfc2047 to format the From: line. The add_rfc2047 assumes that the string is encoded as utf-8. --- builtin-mailinfo.c | 2 +- commit.c | 10 +++++++++- utf8.c | 9 +++++++-- utf8.h | 2 +- 4 files changed, 18 insertions(+), 5 deletions(-) I was hit by this problem when working with an old repository where I had used latin1, and I tried to use "git rebase". Another option would have been to use the correct encoding in the RFC2047 header, but this was a quicker solution. diff --git a/builtin-mailinfo.c b/builtin-mailinfo.c index 583da38..3fd8e00 100644 --- a/builtin-mailinfo.c +++ b/builtin-mailinfo.c @@ -513,7 +513,7 @@ static void convert_to_utf8(char *line, char *charset) { static char latin_one[] = "latin1"; char *input_charset = *charset ? charset : latin_one; - char *out = reencode_string(line, metainfo_charset, input_charset); + char *out = reencode_string(line, metainfo_charset, input_charset, NULL); if (!out) die("cannot convert from %s to %s\n", diff --git a/commit.c b/commit.c index 496d37a..8477fa7 100644 --- a/commit.c +++ b/commit.c @@ -486,6 +486,10 @@ static int add_rfc2047(char *buf, const char *line, int len) if (!needquote) return sprintf(buf, "%.*s", len, line); + if (git_commit_encoding) + line = reencode_string(line, "utf-8", + git_commit_encoding, &len); + memcpy(bp, q_utf8, sizeof(q_utf8)-1); bp += sizeof(q_utf8)-1; for (i = 0; i < len; i++) { @@ -501,6 +505,10 @@ static int add_rfc2047(char *buf, const char *line, int len) } memcpy(bp, "?=", 2); bp += 2; + + if (git_commit_encoding) + free((char *)line); + return bp - buf; } @@ -687,7 +695,7 @@ static char *logmsg_reencode(const struct commit *commit) out = strdup(commit->buffer); else out = reencode_string(commit->buffer, - output_encoding, encoding); + output_encoding, encoding, NULL); if (out) out = replace_encoding_header(out, output_encoding); diff --git a/utf8.c b/utf8.c index 7c80eec..ee9f514 100644 --- a/utf8.c +++ b/utf8.c @@ -291,7 +291,7 @@ int is_encoding_utf8(const char *name) * with iconv. If the conversion fails, returns NULL. */ #ifndef NO_ICONV -char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding) +char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding, int *len) { iconv_t conv; size_t insz, outsz, outalloc; @@ -302,7 +302,10 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e conv = iconv_open(out_encoding, in_encoding); if (conv == (iconv_t) -1) return NULL; - insz = strlen(in); + if (len) + insz = *len; + else + insz = strlen(in); outsz = insz; outalloc = outsz + 1; /* for terminating NUL */ out = xmalloc(outalloc); @@ -332,6 +335,8 @@ char *reencode_string(const char *in, const char *out_encoding, const char *in_e } else { *outpos = '\0'; + if (len) + *len = outpos - out; break; } } diff --git a/utf8.h b/utf8.h index a07c5a8..eb64d46 100644 --- a/utf8.h +++ b/utf8.h @@ -8,7 +8,7 @@ int is_encoding_utf8(const char *name); void print_wrapped_text(const char *text, int indent, int indent2, int len); #ifndef NO_ICONV -char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding); +char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding, int *len); #else #define reencode_string(a,b,c) NULL #endif -- 1.4.4.4.ge10a-dirty -- David Kågedal ``` ## Junio C Hamano, 2007-01-12 22:11 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <7vd55jrj38.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vd55jrj38.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <871wm08kcu.fsf@morpheus.local> ``` David Kågedal writes: > The add_user_info function formats the commit as a mail message, and > uses add_rfc2047 to format the From: line. The add_rfc2047 assumes > that the string is encoded as utf-8. pretty_print_commit() labels the commit log message not just the author name also as UTF-8 when doing plain_non_ascii. It might make more sense to just set the log_output_encoding to be always UTF-8 when generating an e-mail output, in git-format-patch. > diff --git a/utf8.h b/utf8.h > index a07c5a8..eb64d46 100644 > --- a/utf8.h > +++ b/utf8.h > @@ -8,7 +8,7 @@ int is_encoding_utf8(const char *name); > void print_wrapped_text(const char *text, int indent, int indent2, int len); > > #ifndef NO_ICONV > -char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding); > +char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding, int *len); > #else > #define reencode_string(a,b,c) NULL > #endif This feels fishy... ``` ## Junio C Hamano, 2007-01-13 01:31 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <7vr6tzogp4.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vr6tzogp4.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <7vd55jrj38.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano writes: > It might make more sense to just set the log_output_encoding to > be always UTF-8 when generating an e-mail output, in > git-format-patch. Actually, I do not want to be an UTF-8 imperialist, so how about doing this? -- >8 -- Use log output encoding in --pretty=email headers. Private functions add_rfc2047() and pretty_print_commit() assumed they are only emitting UTF-8. Signed-off-by: Junio C Hamano --- diff --git a/commit.c b/commit.c index 496d37a..9b2b842 100644 --- a/commit.c +++ b/commit.c @@ -464,20 +464,29 @@ static int get_one_line(const char *msg, unsigned long len) return ret; } +/* High bit set, or ISO-2022-INT */ +static int non_ascii(int ch) +{ + ch = (ch & 0xff); + return ((ch & 0x80) || (ch == 0x1b)); +} + static int is_rfc2047_special(char ch) { - return ((ch & 0x80) || (ch == '=') || (ch == '?') || (ch == '_')); + return (non_ascii(ch) || (ch == '=') || (ch == '?') || (ch == '_')); } -static int add_rfc2047(char *buf, const char *line, int len) +static int add_rfc2047(char *buf, const char *line, int len, + const char *encoding) { char *bp = buf; int i, needquote; - static const char q_utf8[] = "=?utf-8?q?"; + char q_encoding[128]; + const char *q_encoding_fmt = "=?%s?q?"; for (i = needquote = 0; !needquote && i < len; i++) { - unsigned ch = line[i]; - if (ch & 0x80) + int ch = line[i]; + if (non_ascii(ch)) needquote++; if ((i + 1 < len) && (ch == '=' && line[i+1] == '?')) @@ -486,8 +495,11 @@ static int add_rfc2047(char *buf, const char *line, int len) if (!needquote) return sprintf(buf, "%.*s", len, line); - memcpy(bp, q_utf8, sizeof(q_utf8)-1); - bp += sizeof(q_utf8)-1; + i = snprintf(q_encoding, sizeof(q_encoding), q_encoding_fmt, encoding); + if (sizeof(q_encoding) < i) + die("Insanely long encoding name %s", encoding); + memcpy(bp, q_encoding, i); + bp += i; for (i = 0; i < len; i++) { unsigned ch = line[i] & 0xFF; if (is_rfc2047_special(ch)) { @@ -505,7 +517,8 @@ static int add_rfc2047(char *buf, const char *line, int len) } static int add_user_info(const char *what, enum cmit_fmt fmt, char *buf, - const char *line, int relative_date) + const char *line, int relative_date, + const char *encoding) { char *date; int namelen; @@ -533,7 +546,8 @@ static int add_user_info(const char *what, enum cmit_fmt fmt, char *buf, filler = ""; strcpy(buf, "From: "); ret = strlen(buf); - ret += add_rfc2047(buf + ret, line, display_name_length); + ret += add_rfc2047(buf + ret, line, display_name_length, + encoding); memcpy(buf + ret, name_tail, namelen - display_name_length); ret += namelen - display_name_length; buf[ret++] = '\n'; @@ -668,21 +682,18 @@ static char *replace_encoding_header(char *buf, char *encoding) return buf; } -static char *logmsg_reencode(const struct commit *commit) +static char *logmsg_reencode(const struct commit *commit, + char *output_encoding) { char *encoding; char *out; - char *output_encoding = (git_log_output_encoding - ? git_log_output_encoding - : git_commit_encoding); + char *utf8 = "utf-8"; - if (!output_encoding) - output_encoding = "utf-8"; - else if (!*output_encoding) + if (!*output_encoding) return NULL; encoding = get_header(commit, "encoding"); if (!encoding) - return NULL; + encoding = utf8; if (!strcmp(encoding, output_encoding)) out = strdup(commit->buffer); else @@ -691,7 +702,8 @@ static char *logmsg_reencode(const struct commit *commit) if (out) out = replace_encoding_header(out, output_encoding); - free(encoding); + if (encoding != utf8) + free(encoding); if (!out) return NULL; return out; @@ -711,8 +723,15 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt, int parents_shown = 0; const char *msg = commit->buffer; int plain_non_ascii = 0; - char *reencoded = logmsg_reencode(commit); + char *reencoded; + char *encoding; + encoding = (git_log_output_encoding + ? git_log_output_encoding + : git_commit_encoding); + if (!encoding) + encoding = "utf-8"; + reencoded = logmsg_reencode(commit, encoding); if (reencoded) msg = reencoded; @@ -738,7 +757,7 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt, i + 1 < len && msg[i+1] == '\n') in_body = 1; } - else if (ch & 0x80) { + else if (non_ascii(ch)) { plain_non_ascii = 1; break; } @@ -797,13 +816,15 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt, offset += add_user_info("Author", fmt, buf + offset, line + 7, - relative_date); + relative_date, + encoding); if (!memcmp(line, "committer ", 10) && (fmt == CMIT_FMT_FULL || fmt == CMIT_FMT_FULLER)) offset += add_user_info("Commit", fmt, buf + offset, line + 10, - relative_date); + relative_date, + encoding); continue; } @@ -826,7 +847,8 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt, int slen = strlen(subject); memcpy(buf + offset, subject, slen); offset += slen; - offset += add_rfc2047(buf + offset, line, linelen); + offset += add_rfc2047(buf + offset, line, linelen, + encoding); } else { memset(buf + offset, ' ', indent); @@ -837,11 +859,17 @@ unsigned long pretty_print_commit(enum cmit_fmt fmt, if (fmt == CMIT_FMT_ONELINE) break; if (subject && plain_non_ascii) { - static const char header[] = - "Content-Type: text/plain; charset=UTF-8\n" + int sz; + char header[512]; + const char *header_fmt = + "Content-Type: text/plain; charset=%s\n" "Content-Transfer-Encoding: 8bit\n"; - memcpy(buf + offset, header, sizeof(header)-1); - offset += sizeof(header)-1; + sz = snprintf(header, sizeof(header), header_fmt, + encoding); + if (sizeof(header) < sz) + die("Encoding name %s too long", encoding); + memcpy(buf + offset, header, sz); + offset += sz; } if (after_subject) { int slen = strlen(after_subject); ``` ## Junio C Hamano, 2007-01-13 01:43 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <7vmz4nog5b.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vmz4nog5b.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <7vr6tzogp4.fsf@assigned-by-dhcp.cox.net> ``` Side note. The previous patch does not help if your commit were made in non UTF-8 with not too recent git; the code assumes that commit messages without the new "encoding" headers are in UTF-8. We might want to help transitioning people by doing something like this on top of the previous patch. Then when dealing with an ancient commit (sorry, I am not saying commits older than 3 weeks are ancient -- but it will be 6 months from now ;-), you can override that default by setting an environment variable. --- diff --git a/commit.c b/commit.c index 9b2b842..a1b5705 100644 --- a/commit.c +++ b/commit.c @@ -692,8 +692,12 @@ static char *logmsg_reencode(const struct commit *commit, if (!*output_encoding) return NULL; encoding = get_header(commit, "encoding"); - if (!encoding) - encoding = utf8; + if (!encoding) { + if (getenv("GIT_OLD_COMMIT_ENCODING")) + encoding = strdup(getenv("GIT_OLD_COMMIT_ENCODING")); + else + encoding = utf8; + } if (!strcmp(encoding, output_encoding)) out = strdup(commit->buffer); else ``` ## Alex Riesen, 2007-01-13 11:02 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <20070113110238.GA2795@steel.home> URL: https://gitlist.dev/e/20070113110238.GA2795%40steel.home In-Reply-To: <7vr6tzogp4.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano, Sat, Jan 13, 2007 02:31:35 +0100: > +/* High bit set, or ISO-2022-INT */ > +static int non_ascii(int ch) > +{ > + ch = (ch & 0xff); > + return ((ch & 0x80) || (ch == 0x1b)); > +} > + "return (ch & 0x0x80) || (ch & 0xff) == 0x1b;" :) ``` ## Johannes Schindelin, 2007-01-13 11:19 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0701131217410.22628%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <7vmz4nog5b.fsf@assigned-by-dhcp.cox.net> ``` Hi, On Fri, 12 Jan 2007, Junio C Hamano wrote: > Side note. The previous patch does not help if your commit were > made in non UTF-8 with not too recent git; the code assumes that > commit messages without the new "encoding" headers are in UTF-8. Why not just use is_utf8() and warn, or error out, if the message is not UTF-8? (I tend towards the erroring out, since this _is_ a new feature, and gives undesired results with "old" commits.) Ciao, Dscho ``` ## Robin Rosenberg, 2007-01-13 12:23 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <200701131323.53121.robin.rosenberg.lists@dewire.com> URL: https://gitlist.dev/e/200701131323.53121.robin.rosenberg.lists%40dewire.com In-Reply-To: <7vmz4nog5b.fsf@assigned-by-dhcp.cox.net> ``` lördag 13 januari 2007 02:43 skrev Junio C Hamano: > Side note. The previous patch does not help if your commit were > made in non UTF-8 with not too recent git; the code assumes that > commit messages without the new "encoding" headers are in UTF-8. Wasn't there a repository option, "commitencoding"? I can't see it being used here. I.e., we should err out if the log message is not UTF-8 and the option is not set (giving a message telling the user to set it). If it is set we should consider the repository encoding to be the one and if that too is wrong (only possible to detect for some encodings), just assume iso-8859-1 as anything could in theory be iso-8859-1 encoded. -- robin ``` ## Junio C Hamano, 2007-01-13 17:54 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <7vd55ilsm7.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vd55ilsm7.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <200701131323.53121.robin.rosenberg.lists@dewire.com> ``` Robin Rosenberg writes: > lördag 13 januari 2007 02:43 skrev Junio C Hamano: >> Side note. The previous patch does not help if your commit were >> made in non UTF-8 with not too recent git; the code assumes that >> commit messages without the new "encoding" headers are in UTF-8. > > Wasn't there a repository option, "commitencoding"? I can't see it being > used here. commitencoding is about what encoding the commit newly created in this repository right now should claim to have -- in other words what is fed to commit-tree. We are talking about examining existing commit that might have come from another repository or created some time ago when the repository configuration was set differently. ``` ## Junio C Hamano, 2007-01-13 17:57 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <7v8xg6lsh6.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7v8xg6lsh6.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Johannes Schindelin writes: > Why not just use is_utf8() and warn, or error out, if the message is not > UTF-8? (I tend towards the erroring out, since this _is_ a new feature, > and gives undesired results with "old" commits.) That sounds sensible. ``` ## Junio C Hamano, 2007-01-13 22:18 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <7vfyaek1ub.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vfyaek1ub.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <7vr6tzogp4.fsf@assigned-by-dhcp.cox.net> ``` On this topic, along with the "format-patch" fix (which automatically makes "rebase without --merge" do the right thing because it is "format-patch piped to am" in essence), I have another commit to make "cherry-pick", "rebase --merge" and "commit -c/-C" do the right thing according to the commitencoding specified in the repository the new commit is being created. The issue is that an existing commit might have come from a different repository or from the past when this repository had commitencoding that was different from the current value. Running "cat-file commit" to extract the old commit log message and feeding it directly to create the new commit would not work, because the value of commitencoding in this repository may be different. This should not affect old encoding-unaware setup where people use _only_ a legacy encoding and do not bother to specify any commitencoding. In such a case, both input and output are the same and while we pretend both are UTF-8, we actually do not trigger conversion. To support such a configuration is one reason I did not actually take Johannes's suggestion to error out on an existing commit that does _not_ have encoding header but the contents does not look like a valid UTF-8. The series is currently sitting in 'next'. If people do not see problem with it, I think it should go in v1.5.0. ``` ## Horst H. von Brand, 2007-01-14 00:42 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <200701140042.l0E0gvMT005024@laptop13.inf.utfsm.cl> URL: https://gitlist.dev/e/200701140042.l0E0gvMT005024%40laptop13.inf.utfsm.cl In-Reply-To: <20070113110238.GA2795@steel.home> ``` Alex Riesen wrote: > Junio C Hamano, Sat, Jan 13, 2007 02:31:35 +0100: > > +/* High bit set, or ISO-2022-INT */ > > +static int non_ascii(int ch) > > +{ > > + ch = (ch & 0xff); > > + return ((ch & 0x80) || (ch == 0x1b)); > > +} > > + > > "return (ch & 0x0x80) || (ch & 0xff) == 0x1b;" :) ^^ Is the same, if ch == 0x9b, it will match the first part anyway. The outer parentesis can (should?) go. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 2654431 Universidad Tecnica Federico Santa Maria +56 32 2654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 2797513 ``` ## Alex Riesen, 2007-01-14 19:25 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <20070114192557.GA2378@steel.home> URL: https://gitlist.dev/e/20070114192557.GA2378%40steel.home In-Reply-To: <200701140042.l0E0gvMT005024@laptop13.inf.utfsm.cl> ``` Horst H. von Brand, Sun, Jan 14, 2007 01:42:57 +0100: > Alex Riesen wrote: > > Junio C Hamano, Sat, Jan 13, 2007 02:31:35 +0100: > > > +/* High bit set, or ISO-2022-INT */ > > > +static int non_ascii(int ch) > > > +{ > > > + ch = (ch & 0xff); > > > + return ((ch & 0x80) || (ch == 0x1b)); > > > +} > > > + > > > > "return (ch & 0x0x80) || (ch & 0xff) == 0x1b;" :) > ^^ Oops :) > Is the same, if ch == 0x9b, it will match the first part anyway. So it should. 0x9b isn't ASCII. > The outer parentesis can (should?) go. It's "question of style", I'm afraid :) ``` ## David Kågedal, 2007-01-15 16:53 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <87k5zo44fq.fsf@morpheus.local> URL: https://gitlist.dev/e/87k5zo44fq.fsf%40morpheus.local In-Reply-To: <7vd55jrj38.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano writes: >> diff --git a/utf8.h b/utf8.h >> index a07c5a8..eb64d46 100644 >> --- a/utf8.h >> +++ b/utf8.h >> @@ -8,7 +8,7 @@ int is_encoding_utf8(const char *name); >> void print_wrapped_text(const char *text, int indent, int indent2, int len); >> >> #ifndef NO_ICONV >> -char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding); >> +char *reencode_string(const char *in, const char *out_encoding, const char *in_encoding, int *len); >> #else >> #define reencode_string(a,b,c) NULL >> #endif > > This feels fishy... I admit that I didn't test-compile with NO_ICONV. -- David Kågedal ``` ## David Kågedal, 2007-01-15 16:54 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <87fyac44e9.fsf@morpheus.local> URL: https://gitlist.dev/e/87fyac44e9.fsf%40morpheus.local In-Reply-To: <7vmz4nog5b.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano writes: > Side note. The previous patch does not help if your commit were > made in non UTF-8 with not too recent git; the code assumes that > commit messages without the new "encoding" headers are in UTF-8. This was exactly the problem I was trying to solve. > We might want to help transitioning people by doing something > like this on top of the previous patch. Then when dealing with > an ancient commit (sorry, I am not saying commits older than 3 > weeks are ancient -- but it will be 6 months from now ;-), you > can override that default by setting an environment variable. > > --- > diff --git a/commit.c b/commit.c > index 9b2b842..a1b5705 100644 > --- a/commit.c > +++ b/commit.c > @@ -692,8 +692,12 @@ static char *logmsg_reencode(const struct commit *commit, > if (!*output_encoding) > return NULL; > encoding = get_header(commit, "encoding"); > - if (!encoding) > - encoding = utf8; > + if (!encoding) { > + if (getenv("GIT_OLD_COMMIT_ENCODING")) > + encoding = strdup(getenv("GIT_OLD_COMMIT_ENCODING")); > + else > + encoding = utf8; > + } > if (!strcmp(encoding, output_encoding)) > out = strdup(commit->buffer); > else > > > -- David Kågedal ``` ## David Kågedal, 2007-01-15 16:57 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <87bql0449j.fsf@morpheus.local> URL: https://gitlist.dev/e/87bql0449j.fsf%40morpheus.local In-Reply-To: <7vr6tzogp4.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano writes: > -static int add_rfc2047(char *buf, const char *line, int len) > +static int add_rfc2047(char *buf, const char *line, int len, > + const char *encoding) > { > char *bp = buf; > int i, needquote; > - static const char q_utf8[] = "=?utf-8?q?"; > + char q_encoding[128]; > + const char *q_encoding_fmt = "=?%s?q?"; This goes against the old principle of being forgiving in what you accept, and strict in what you send. The names of the encoding in the headers should probably be normalized before putting them in an e-mail. I.e. we might accept "utf-8", "utf8", "UTF-8", and "UTF8" (this depends on iconv, I suppose), but the RFC2047 encoding should be the one blessed by RFC4027. But I admit that I haven't read the RFC, and I'm writing this offline so I can't check right now. -- David Kågedal ``` ## David Kågedal, 2007-01-15 16:58 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <877ivo446n.fsf@morpheus.local> URL: https://gitlist.dev/e/877ivo446n.fsf%40morpheus.local In-Reply-To: ``` Johannes Schindelin writes: > Hi, > > On Fri, 12 Jan 2007, Junio C Hamano wrote: > >> Side note. The previous patch does not help if your commit were >> made in non UTF-8 with not too recent git; the code assumes that >> commit messages without the new "encoding" headers are in UTF-8. > > Why not just use is_utf8() and warn, or error out, if the message is not > UTF-8? (I tend towards the erroring out, since this _is_ a new feature, > and gives undesired results with "old" commits.) What do you mean? I have an old repository with latin1 commits without any encoding markers. I want to be able to use format-patch from that and at least get a From: line with something readable. You can't just barf and say "This isn't UTF-8, go away". -- David Kågedal ``` ## Johannes Schindelin, 2007-01-16 11:41 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0701161239590.22628%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <877ivo446n.fsf@morpheus.local> ``` Hi, On Mon, 15 Jan 2007, David Kågedal wrote: > Johannes Schindelin writes: > > > On Fri, 12 Jan 2007, Junio C Hamano wrote: > > > >> Side note. The previous patch does not help if your commit were > >> made in non UTF-8 with not too recent git; the code assumes that > >> commit messages without the new "encoding" headers are in UTF-8. > > > > Why not just use is_utf8() and warn, or error out, if the message is not > > UTF-8? (I tend towards the erroring out, since this _is_ a new feature, > > and gives undesired results with "old" commits.) > > What do you mean? I have an old repository with latin1 commits without > any encoding markers. I want to be able to use format-patch from that > and at least get a From: line with something readable. You can't just > barf and say "This isn't UTF-8, go away". So what do you want to do instead? Just pretend that the unrecoded -- Latin-1 encoded -- text is UTF-8? That's plain wrong. Ciao, Dscho ``` ## David Kågedal, 2007-01-16 12:43 Subject: Re: [PATCH] Reencode committer info to utf-8 before formatting mail header Message-ID: <873b6b87lw.fsf@morpheus.local> URL: https://gitlist.dev/e/873b6b87lw.fsf%40morpheus.local In-Reply-To: ``` Johannes Schindelin writes: > Hi, > > On Mon, 15 Jan 2007, David Kågedal wrote: > >> Johannes Schindelin writes: >> >> > On Fri, 12 Jan 2007, Junio C Hamano wrote: >> > >> >> Side note. The previous patch does not help if your commit were >> >> made in non UTF-8 with not too recent git; the code assumes that >> >> commit messages without the new "encoding" headers are in UTF-8. >> > >> > Why not just use is_utf8() and warn, or error out, if the message is not >> > UTF-8? (I tend towards the erroring out, since this _is_ a new feature, >> > and gives undesired results with "old" commits.) >> >> What do you mean? I have an old repository with latin1 commits without >> any encoding markers. I want to be able to use format-patch from that >> and at least get a From: line with something readable. You can't just >> barf and say "This isn't UTF-8, go away". > > So what do you want to do instead? Just pretend that the unrecoded -- > Latin-1 encoded -- text is UTF-8? That's plain wrong. That is what git did before I wrote my patch, so it obviously not what I want. I want to be able to tell git what encoding it is. My patch reused the i18n.commitencoding configuration parameter for that, but Junio is probably right in that that is only meant for new commits, and an evironment variable makes more sense. So just barfing on a commit that isn't utf-8 isn't a complete solution. But maybe there was some context to your comment above that I missed. -- David Kågedal ```