{"thread":{"id":"11445","subject":"[RFC/PATCH] fix \"git diff\" to create wrong UTF-8 text","startedAt":"2008-01-01T23:20:14Z","lastAt":"2008-01-04T20:53:33Z","messageCount":9,"participants":["Tsugikazu Shibata","Junio C Hamano","しらいしななこ"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"64308","messageId":"20080102.082014.02281301.tshibata@ab.jp.nec.com","threadId":"11445","inReplyTo":null,"subject":"[RFC/PATCH] fix \"git diff\" to create wrong UTF-8 text","fromName":"Tsugikazu Shibata","fromEmail":"tshibata@ab.jp.nec.com","sentAt":"2008-01-01T23:20:14Z","receivedAt":"2008-01-01T23:20:14Z","isPatch":true,"sender":{"key":"tshibata@ab.jp.nec.com","avatar":null},"body":"Hello,\n\nI met a problem in patch text from \"git diff\" for UTF-8 text.\nPatch text following to \"@@\" sometimes cut the string with max\n80bytes. In case of UTF-8 text written in Japanese and English, most\nof Japanese character are consist of 3 bytes for a character and also\nASCII character is single byte.\nSo, cut the string with 80bytes may cause cut off 1 or 2 byte for a\ncharacter at the bottom. This will cause the broken code of result of\n\"git diff\".\n\nIt seems no problem to read such patch text for the patch command but\nthe problem is not readable for me. ie. Emacs cannot handle the\nencoding for such file and show me octal numbers.\n\nThe patch below is my quick and dirty solution (but It works fine !)\nI tested this patch with using Linux kernel document\n(Documentation/ja_JP/HOWTO)\nI believe this should be work for another language using UTF-8 and\nsolve this issue.\n\nPlease note that this is focused only for UTF-8 but we may need to\nsupport another encoding.\nSo, How can we turn on this UTF-8 processing?\nAny suggestions are welcome.\n\nThanks,\n\nSigined-off-by: Tsugikazu Shibata \n---\n\ndiff -upr git-1.5.3.7/xdiff/xutils.c git-1.5.3.7-dev/xdiff/xutils.c\n--- git-1.5.3.7/xdiff/xutils.c\t2007-12-02 06:21:12.000000000 +0900\n+++ git-1.5.3.7-dev/xdiff/xutils.c\t2007-12-31 01:30:51.000000000 +0900\n@@ -332,6 +332,32 @@ long xdl_atol(char const *str, char cons\n }\n\n\n+/* return utf character size of bytes */\n+int utf8charsize(const unsigned char c) {\n+\tint l;\n+\tif ( c < 0x7f ) l = 1;\n+\telse if (( c > 0xc0) && ( c < 0xdf)) l=2;\n+\telse if (( c > 0xe0) && ( c < 0xef)) l=3;\n+\telse if (( c > 0xf0) && ( c < 0xf7)) l=4;\n+\telse if (( c > 0xf8) && ( c < 0xfb)) l=5;\n+\telse if (( c > 0xfc) && ( c < 0xfd)) l=6;\n+\telse l=1; /* fale safe */\n+\treturn l;\n+}\n+\n+int utf8width(const char *up, int len) {\n+ int cs;\n+ int l=len;\n+ const char *p = up;\n+ while ((l > 0) && (p[0] != '\\0')) {\n+\t\tcs = utf8charsize(p[0]);\n+\t\tif (l >= cs) {\n+\t\t\tl -= cs; p += cs;\n+\t\t} else l=0; /* do not split multi byte char. */\n+ }\n+ return p-up;\n+}\n+\n int xdl_emit_hunk_hdr(long s1, long c1, long s2, long c2,\n \t\t const char *func, long funclen, xdemitcb_t *ecb) {\n \tint nb = 0;\n@@ -368,6 +394,7 @@ int xdl_emit_hunk_hdr(long s1, long c1,\n \t\tbuf[nb++] = ' ';\n \t\tif (funclen > sizeof(buf) - nb - 1)\n \t\t\tfunclen = sizeof(buf) - nb - 1;\n+\t\tfunclen = utf8width(func, funclen);\n \t\tmemcpy(buf + nb, func, funclen);\n \t\tnb += funclen;\n \t}\n"},{"id":"64315","messageId":"7v1w904x29.fsf@gitster.siamese.dyndns.org","threadId":"11445","inReplyTo":"20080102.082014.02281301.tshibata@ab.jp.nec.com","subject":"Re: [RFC/PATCH] fix \"git diff\" to create wrong UTF-8 text","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-02T05:26:54Z","receivedAt":"2008-01-02T05:26:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tsugikazu Shibata writes:\n\n> I believe this should be work for another language using UTF-8 and\n> solve this issue.\n> ...\n> @@ -368,6 +394,7 @@ int xdl_emit_hunk_hdr(long s1, long c1,\n> \t\tbuf[nb++] = ' ';\n> \t\tif (funclen > sizeof(buf) - nb - 1)\n> \t\t\tfunclen = sizeof(buf) - nb - 1;\n> +\t\tfunclen = utf8width(func, funclen);\n> \t\tmemcpy(buf + nb, func, funclen);\n> \t\tnb += funclen;\n> \t}\n\nI'd rather not do this in xdiff/ level for two reasons.\n\nWe consider the functions there strictly \"borrowed code\", and\nideally I'd rather even not to have that \"chop down to funclen\"\nlogic at that level.\n\nThe code at that level does not know what paths it is dealing\nwith and cannot consult git specific data (i.e. attributes); it\nwould make it harder to enhance it later by introducing per-path\nencoding information.\n\nHow about...\n\n * (optional) lift the funclen limit from xdl_emit_hunk_hdr()\n and xdl_emit_diff() and have them preserve the full line;\n\n * Around ll.554 in diff.c::fn_out_consume(), look at\n line[i..eol] and apply the \"chomp between character\" logic\n there. I think it is very sensible to use the UTF-8 logic by\n default as you did above (but I suspect you may be able to\n reuse helper functions in utf8.c such as git_wcwidth(),\n instead of rolling your own). Chomp the funcname line given\n from xdiff layer in this function.\n\n * (future) make the length of the funcname line configurable\n either from the command line or configuration.\n\n * (future) add per-path blob encoding information (default to\n UTF-8) to struct emit_callback, and initialize it from the\n gitattributes(5) mechanism, just like we have added ws_rule\n recently there. Use that to decide how inter-character chomp\n works to customize the logic in diff.c::fn_out_consume() you\n would introduce above.\n\nI think the two future enhancements I listed above as examples\nwould be cleaner to implement if the line chomping is done in\nfn_out_consume().\n"},{"id":"64326","messageId":"7v4pdwzhdl.fsf@gitster.siamese.dyndns.org","threadId":"11445","inReplyTo":"7v1w904x29.fsf@gitster.siamese.dyndns.org","subject":"[PATCH 1/2] utf8_width(): allow non NUL-terminated input","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-02T09:49:58Z","receivedAt":"2008-01-02T09:49:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The original interface assumed that the input string to be\nalways terminated with NUL, but that wasn't too useful.\n\nSigned-off-by: Junio C Hamano \n---\n\n * This is a preparatory step for the next one that follows what\n I outlined in my response. As we are in pre-1.5.4 freeze, I\n won't be applying this to 'master' but this and the one that\n implements the sane truncation might be 'next' material.\n\n utf8.c | 68 ++++++++++++++++++++++++++++++++++++++-------------------------\n utf8.h | 2 +-\n 2 files changed, 42 insertions(+), 28 deletions(-)\n\ndiff --git a/utf8.c b/utf8.c\nindex 9efcdb9..5d641fa 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -157,57 +157,71 @@ static int git_wcwidth(ucs_char_t ch)\n * pointed to by the variable start. The pointer is updated to point at\n * the next character. If it was not valid UTF-8, the pointer is set to NULL.\n */\n-int utf8_width(const char **start)\n+int utf8_width(const char **start, size_t *remainder_p)\n {\n \tunsigned char *s = (unsigned char *)*start;\n \tucs_char_t ch;\n+\tsize_t remainder, incr;\n \n-\tif (*s < 0x80) {\n+\t/*\n+\t * A caller that assumes NUL terminated text can choose\n+\t * not to bother with the remainder length.\n+\t */\n+\tremainder = (remainder_p ? *remainder_p : 999);\n+\n+\tif (remainder < 1) {\n+\t\tgoto invalid;\n+\t} else if (*s < 0x80) {\n \t\t/* 0xxxxxxx */\n \t\tch = *s;\n-\t\t*start += 1;\n+\t\tincr = 1;\n \t} else if ((s[0] & 0xe0) == 0xc0) {\n \t\t/* 110XXXXx 10xxxxxx */\n-\t\tif ((s[1] & 0xc0) != 0x80 ||\n-\t\t\t\t/* overlong? */\n-\t\t\t\t(s[0] & 0xfe) == 0xc0)\n+\t\tif (remainder < 2 ||\n+\t\t (s[1] & 0xc0) != 0x80 ||\n+\t\t (s[0] & 0xfe) == 0xc0)\n \t\t\tgoto invalid;\n \t\tch = ((s[0] & 0x1f) << 6) | (s[1] & 0x3f);\n-\t\t*start += 2;\n+\t\tincr = 2;\n \t} else if ((s[0] & 0xf0) == 0xe0) {\n \t\t/* 1110XXXX 10Xxxxxx 10xxxxxx */\n-\t\tif ((s[1] & 0xc0) != 0x80 ||\n-\t\t\t\t(s[2] & 0xc0) != 0x80 ||\n-\t\t\t\t/* overlong? */\n-\t\t\t\t(s[0] == 0xe0 && (s[1] & 0xe0) == 0x80) ||\n-\t\t\t\t/* surrogate? */\n-\t\t\t\t(s[0] == 0xed && (s[1] & 0xe0) == 0xa0) ||\n-\t\t\t\t/* U+FFFE or U+FFFF? */\n-\t\t\t\t(s[0] == 0xef && s[1] == 0xbf &&\n-\t\t\t\t (s[2] & 0xfe) == 0xbe))\n+\t\tif (remainder < 3 ||\n+\t\t (s[1] & 0xc0) != 0x80 ||\n+\t\t (s[2] & 0xc0) != 0x80 ||\n+\t\t /* overlong? */\n+\t\t (s[0] == 0xe0 && (s[1] & 0xe0) == 0x80) ||\n+\t\t /* surrogate? */\n+\t\t (s[0] == 0xed && (s[1] & 0xe0) == 0xa0) ||\n+\t\t /* U+FFFE or U+FFFF? */\n+\t\t (s[0] == 0xef && s[1] == 0xbf &&\n+\t\t (s[2] & 0xfe) == 0xbe))\n \t\t\tgoto invalid;\n \t\tch = ((s[0] & 0x0f) << 12) |\n \t\t\t((s[1] & 0x3f) << 6) | (s[2] & 0x3f);\n-\t\t*start += 3;\n+\t\tincr = 3;\n \t} else if ((s[0] & 0xf8) == 0xf0) {\n \t\t/* 11110XXX 10XXxxxx 10xxxxxx 10xxxxxx */\n-\t\tif ((s[1] & 0xc0) != 0x80 ||\n-\t\t\t\t(s[2] & 0xc0) != 0x80 ||\n-\t\t\t\t(s[3] & 0xc0) != 0x80 ||\n-\t\t\t\t/* overlong? */\n-\t\t\t\t(s[0] == 0xf0 && (s[1] & 0xf0) == 0x80) ||\n-\t\t\t\t/* > U+10FFFF? */\n-\t\t\t\t(s[0] == 0xf4 && s[1] > 0x8f) || s[0] > 0xf4)\n+\t\tif (remainder < 4 ||\n+\t\t (s[1] & 0xc0) != 0x80 ||\n+\t\t (s[2] & 0xc0) != 0x80 ||\n+\t\t (s[3] & 0xc0) != 0x80 ||\n+\t\t /* overlong? */\n+\t\t (s[0] == 0xf0 && (s[1] & 0xf0) == 0x80) ||\n+\t\t /* > U+10FFFF? */\n+\t\t (s[0] == 0xf4 && s[1] > 0x8f) || s[0] > 0xf4)\n \t\t\tgoto invalid;\n \t\tch = ((s[0] & 0x07) << 18) | ((s[1] & 0x3f) << 12) |\n \t\t\t((s[2] & 0x3f) << 6) | (s[3] & 0x3f);\n-\t\t*start += 4;\n+\t\tincr = 4;\n \t} else {\n invalid:\n \t\t*start = NULL;\n \t\treturn 0;\n \t}\n \n+\t*start += incr;\n+\tif (remainder_p)\n+\t\t*remainder_p = remainder - incr;\n \treturn git_wcwidth(ch);\n }\n \n@@ -218,7 +232,7 @@ int is_utf8(const char *text)\n \t\t\ttext++;\n \t\t\tcontinue;\n \t\t}\n-\t\tutf8_width(&text);\n+\t\tutf8_width(&text, NULL);\n \t\tif (!text)\n \t\t\treturn 0;\n \t}\n@@ -278,7 +292,7 @@ int print_wrapped_text(const char *text, int indent, int indent2, int width)\n \t\t\tcontinue;\n \t\t}\n \t\tif (assume_utf8)\n-\t\t\tw += utf8_width(&text);\n+\t\t\tw += utf8_width(&text, NULL);\n \t\telse {\n \t\t\tw++;\n \t\t\ttext++;\ndiff --git a/utf8.h b/utf8.h\nindex 15db6f1..4295a19 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -1,7 +1,7 @@\n #ifndef GIT_UTF8_H\n #define GIT_UTF8_H\n \n-int utf8_width(const char **start);\n+int utf8_width(const char **start, size_t *remain);\n int is_utf8(const char *text);\n int is_encoding_utf8(const char *name);\n \n-- \n1.5.4.rc2.13.g690bd\n"},{"id":"64327","messageId":"7vy7b8y2ss.fsf@gitster.siamese.dyndns.org","threadId":"11445","inReplyTo":"7v1w904x29.fsf@gitster.siamese.dyndns.org","subject":"[PATCH 2/2] diff: do not chomp hunk-header in the middle of a character","fromName":"Junio C Hamano","fromEmail":"junio@pobox.com","sentAt":"2008-01-02T09:50:11Z","receivedAt":"2008-01-02T09:50:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"We truncate hunk-header line at 80 bytes, but that 80th byte\ncould be in the middle of a character, which is bad. This uses\nutf8_width() function to make sure we do not cut a character in\nthe middle.\n\nSigned-off-by: Junio C Hamano \n---\n\n * And this implements the sane_truncate_line() that\n unconditionally assumes UTF-8.\n\n The 80-byte cutoff is meant to chomp at 80 display columns,\n and in many encodings including UTF-8 80 cols are usually\n more than 80 bytes. To implement 80 column chomping, we\n would need to tweak the low level xdl_find_func() interface\n further so that it gives us back more than 80-bytes. Ideally\n that layer should not be doing any chomping.\n\n Another thing I noticed but did not bother to fix was that\n the emit_line() interface is doing bad things with color.\n It emits:\n\n\tstart-color text (= line, len) reset-color\n\n and the text often ends with LF. We should emit reset-color\n before the terminating LF instead.\n\n diff.c | 28 ++++++++++++++++++++++++++++\n 1 files changed, 28 insertions(+), 0 deletions(-)\n\ndiff --git a/diff.c b/diff.c\nindex 5bdc111..b9159bc 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -10,6 +10,7 @@\n #include \"color.h\"\n #include \"attr.h\"\n #include \"run-command.h\"\n+#include \"utf8.h\"\n \n #ifdef NO_FAST_WORKING_DIRECTORY\n #define FAST_WORKING_DIRECTORY 0\n@@ -465,10 +466,13 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \t}\n }\n \n+typedef unsigned long (*sane_truncate_fn)(char *line, unsigned long len);\n+\n struct emit_callback {\n \tstruct xdiff_emit_state xm;\n \tint nparents, color_diff;\n \tunsigned ws_rule;\n+\tsane_truncate_fn truncate;\n \tconst char **label_path;\n \tstruct diff_words_data *diff_words;\n \tint *found_changesp;\n@@ -521,6 +525,27 @@ static void emit_add_line(const char *reset, struct emit_callback *ecbdata, cons\n \t}\n }\n \n+static unsigned long sane_truncate_line(struct emit_callback *ecb, char *line, unsigned long len)\n+{\n+\tconst char *cp;\n+\tunsigned long width, allot;\n+\n+\tif (ecb->truncate)\n+\t\treturn ecb->truncate(line, len);\n+\tcp = line;\n+\twidth = 0;\n+\tallot = len;\n+\twhile (0 < len) {\n+\t\tint w = utf8_width(&cp, &len);\n+\t\tif (!cp)\n+\t\t\tbreak; /* truncated in the middle? */\n+\t\twidth += w;\n+\t\tif (allot <= width)\n+\t\t\tbreak;\n+\t}\n+\treturn allot - len;\n+}\n+\n static void fn_out_consume(void *priv, char *line, unsigned long len)\n {\n \tint i;\n@@ -551,8 +576,11 @@ static void fn_out_consume(void *priv, char *line, unsigned long len)\n \t\t;\n \tif (2 <= i && i < len && line[i] == ' ') {\n \t\tecbdata->nparents = i - 1;\n+\t\tlen = sane_truncate_line(ecbdata, line, len);\n \t\temit_line(diff_get_color(ecbdata->color_diff, DIFF_FRAGINFO),\n \t\t\t reset, line, len);\n+\t\tif (line[len-1] != '\\n')\n+\t\t\tputchar('\\n');\n \t\treturn;\n \t}\n \n-- \n1.5.4.rc2.13.g690bd\n"},{"id":"64328","messageId":"7vsl1gy2si.fsf@gitster.siamese.dyndns.org","threadId":"11445","inReplyTo":"7v1w904x29.fsf@gitster.siamese.dyndns.org","subject":"[PATCH 3/2] attribute \"coding\": specify blob encoding","fromName":"Junio C Hamano","fromEmail":"junio@pobox.com","sentAt":"2008-01-02T09:50:21Z","receivedAt":"2008-01-02T09:50:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This teaches \"diff hunk header\" function about custom character\nencoding per path via gitattributes(5) so that it can sensibly\nchomp a line without truncating a character in the middle.\n\nSigned-off-by: Junio C Hamano \n---\n\n * This is not intended for serious inclusion, but was done more\n as a demonstration of the concept, hence [3/2].\n\n Makefile | 5 ++-\n coding.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n coding.h | 10 +++++++++\n diff.c | 48 ++++++++++++++++++++++++++++++++++++++++-----\n diffcore.h | 1 +\n 5 files changed, 118 insertions(+), 8 deletions(-)\n create mode 100644 coding.c\n create mode 100644 coding.h\n\ndiff --git a/Makefile b/Makefile\nindex 21c80e6..de205e6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -293,7 +293,8 @@ LIB_H = \\\n \trun-command.h strbuf.h tag.h tree.h git-compat-util.h revision.h \\\n \ttree-walk.h log-tree.h dir.h path-list.h unpack-trees.h builtin.h \\\n \tutf8.h reflog-walk.h patch-ids.h attr.h decorate.h progress.h \\\n-\tmailmap.h remote.h parse-options.h transport.h diffcore.h hash.h\n+\tmailmap.h remote.h parse-options.h transport.h diffcore.h hash.h \\\n+\tcoding.h\n \n DIFF_OBJS = \\\n \tdiff.o diff-lib.o diffcore-break.o diffcore-order.o \\\n@@ -316,7 +317,7 @@ LIB_OBJS = \\\n \talloc.o merge-file.o path-list.o help.o unpack-trees.o $(DIFF_OBJS) \\\n \tcolor.o wt-status.o archive-zip.o archive-tar.o shallow.o utf8.o \\\n \tconvert.o attr.o decorate.o progress.o mailmap.o symlinks.o remote.o \\\n-\ttransport.o bundle.o walker.o parse-options.o ws.o\n+\ttransport.o bundle.o walker.o parse-options.o ws.o coding.o\n \n BUILTIN_OBJS = \\\n \tbuiltin-add.o \\\ndiff --git a/coding.c b/coding.c\nnew file mode 100644\nindex 0000000..63c5dc7\n--- /dev/null\n+++ b/coding.c\n@@ -0,0 +1,62 @@\n+#include \"coding.h\"\n+\n+unsigned long truncate_euc_jp(char *line, unsigned long len)\n+{\n+\t/*\n+\t * line contains len bytes but the end of the string could\n+\t * be incomplete (chomped in the middle of a character).\n+\t * grab complete lines and count the display width, and\n+\t * return the number of bytes to chomp at (which could\n+\t * be smaller than 'len' given, to discard the corrupt\n+\t * characters at the end\n+\t */\n+\tchar *ep = line + len;\n+\tunsigned long sofar = 0;\n+\n+\twhile (line < ep) {\n+\t\tint ch = (*line++ & 0xFF);\n+\t\tint width;\n+\n+\t\tif (ch < 0x80) {\n+\t\t\t/* JIS X 0201 lower half */\n+\t\t\tsofar++;\n+\t\t\twidth = 1;\n+\t\t} else if (0xA1 <= ch && ch <= 0xFE) {\n+\t\t\t/* JIS X 0208 */\n+\t\t\tif (ep <= line)\n+\t\t\t\tgoto invalid;\n+\t\t\tch = (*line++ & 0xFF);\n+\t\t\tif (ch < 0xA1 || 0xFE < ch)\n+\t\t\t\tgoto invalid;\n+\t\t\twidth = 2;\n+\t\t\tsofar += 2;\n+\t\t} else if (ch == 0x8E) {\n+\t\t\t/* JIS X 0201 upper half */\n+\t\t\tif (ep <= line)\n+\t\t\t\tgoto invalid;\n+\t\t\tch = (*line++ & 0xFF);\n+\t\t\tif (ch <= 0xA1 || 0xDF < ch)\n+\t\t\t\tgoto invalid;\n+\t\t\twidth = 1;\n+\t\t\tsofar += 2;\n+\t\t} else if (ch == 0x8F) {\n+\t\t\t/* JIS X 0212 */\n+\t\t\tif (ep <= line)\n+\t\t\t\tgoto invalid;\n+\t\t\tch = (*line++ & 0xFF);\n+\t\t\tif (ch < 0xA1 || 0xFE < ch)\n+\t\t\t\tgoto invalid;\n+\t\t\tif (ep <= line)\n+\t\t\t\tgoto invalid;\n+\t\t\tch = (*line++ & 0xFF);\n+\t\t\tif (ch < 0xA1 || 0xFE < ch)\n+\t\t\t\tgoto invalid;\n+\t\t\twidth = 2;\n+\t\t\tsofar += 3;\n+\t\t} else {\n+\t\tinvalid:\n+\t\t\treturn sofar;\n+\t\t}\n+\t}\n+\treturn sofar;\n+}\ndiff --git a/coding.h b/coding.h\nnew file mode 100644\nindex 0000000..0e02701\n--- /dev/null\n+++ b/coding.h\n@@ -0,0 +1,10 @@\n+#ifndef CODING_H\n+#define CODING_H\n+\n+#include \"cache.h\"\n+\n+typedef unsigned long (*sane_truncate_fn)(char *line, unsigned long len);\n+\n+unsigned long truncate_euc_jp(char *, unsigned long);\n+\n+#endif\ndiff --git a/diff.c b/diff.c\nindex b9159bc..40c8d2b 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -11,6 +11,7 @@\n #include \"attr.h\"\n #include \"run-command.h\"\n #include \"utf8.h\"\n+#include \"coding.h\"\n \n #ifdef NO_FAST_WORKING_DIRECTORY\n #define FAST_WORKING_DIRECTORY 0\n@@ -466,8 +467,6 @@ static void diff_words_show(struct diff_words_data *diff_words)\n \t}\n }\n \n-typedef unsigned long (*sane_truncate_fn)(char *line, unsigned long len);\n-\n struct emit_callback {\n \tstruct xdiff_emit_state xm;\n \tint nparents, color_diff;\n@@ -1125,30 +1124,34 @@ static void emit_binary_diff(mmfile_t *one, mmfile_t *two)\n static void setup_diff_attr_check(struct git_attr_check *check)\n {\n \tstatic struct git_attr *attr_diff;\n+\tstatic struct git_attr *attr_coding;\n \n \tif (!attr_diff) {\n \t\tattr_diff = git_attr(\"diff\", 4);\n+\t\tattr_coding = git_attr(\"coding\", 6);\n \t}\n \tcheck[0].attr = attr_diff;\n+\tcheck[1].attr = attr_coding;\n }\n \n static void diff_filespec_check_attr(struct diff_filespec *one)\n {\n-\tstruct git_attr_check attr_diff_check;\n+\tstruct git_attr_check attr_diff_check[2];\n \tint check_from_data = 0;\n \n \tif (one->checked_attr)\n \t\treturn;\n \n-\tsetup_diff_attr_check(&attr_diff_check);\n+\tsetup_diff_attr_check(attr_diff_check);\n \tone->is_binary = 0;\n \tone->funcname_pattern_ident = NULL;\n+\tone->coding = NULL;\n \n-\tif (!git_checkattr(one->path, 1, &attr_diff_check)) {\n+\tif (!git_checkattr(one->path, 2, attr_diff_check)) {\n \t\tconst char *value;\n \n \t\t/* binaryness */\n-\t\tvalue = attr_diff_check.value;\n+\t\tvalue = attr_diff_check[0].value;\n \t\tif (ATTR_TRUE(value))\n \t\t\t;\n \t\telse if (ATTR_FALSE(value))\n@@ -1161,6 +1164,15 @@ static void diff_filespec_check_attr(struct diff_filespec *one)\n \t\t\t;\n \t\telse\n \t\t\tone->funcname_pattern_ident = value;\n+\n+\t\t/* coding */\n+\t\tvalue = attr_diff_check[1].value;\n+\t\tif (ATTR_TRUE(value) ||\n+\t\t ATTR_FALSE(value) ||\n+\t\t ATTR_UNSET(value))\n+\t\t\t;\n+\t\telse\n+\t\t\tone->coding = value;\n \t}\n \n \tif (check_from_data) {\n@@ -1233,6 +1245,27 @@ static const char *diff_funcname_pattern(struct diff_filespec *one)\n \treturn NULL;\n }\n \n+static struct {\n+\tconst char *coding;\n+\tsane_truncate_fn fn;\n+} builtin_truncate_fn[] = {\n+\t{ \"euc-jp\", truncate_euc_jp },\n+\t{ \"utf-8\", NULL },\n+};\n+\n+static sane_truncate_fn diff_truncate_line_fn(struct diff_filespec *one)\n+{\n+\tint i;\n+\n+\tdiff_filespec_check_attr(one);\n+\tif (!one->coding)\n+\t\treturn NULL;\n+\tfor (i = 0; i < ARRAY_SIZE(builtin_truncate_fn); i++)\n+\t\tif (!strcmp(one->coding, builtin_truncate_fn[i].coding))\n+\t\t\treturn builtin_truncate_fn[i].fn;\n+\treturn NULL;\n+}\n+\n static void builtin_diff(const char *name_a,\n \t\t\t const char *name_b,\n \t\t\t struct diff_filespec *one,\n@@ -1318,6 +1351,9 @@ static void builtin_diff(const char *name_a,\n \t\tecbdata.color_diff = DIFF_OPT_TST(o, COLOR_DIFF);\n \t\tecbdata.found_changesp = &o->found_changes;\n \t\tecbdata.ws_rule = whitespace_rule(name_b ? name_b : name_a);\n+\t\tecbdata.truncate = diff_truncate_line_fn(one);\n+\t\tif (!ecbdata.truncate)\n+\t\t\tecbdata.truncate = diff_truncate_line_fn(two);\n \t\txpp.flags = XDF_NEED_MINIMAL | o->xdl_opts;\n \t\txecfg.ctxlen = o->context;\n \t\txecfg.flags = XDL_EMIT_FUNCNAMES;\ndiff --git a/diffcore.h b/diffcore.h\nindex cc96c20..711a4df 100644\n--- a/diffcore.h\n+++ b/diffcore.h\n@@ -28,6 +28,7 @@ struct diff_filespec {\n \tvoid *data;\n \tvoid *cnt_data;\n \tconst char *funcname_pattern_ident;\n+\tconst char *coding; /* blob text coding */\n \tunsigned long size;\n \tint count; /* Reference count */\n \tint xfrm_flags;\t\t /* for use by the xfrm */\n-- \n1.5.4.rc2.13.g690bd\n"},{"id":"64395","messageId":"200801032128.m03LSIn2022902@mi0.bluebottle.com","threadId":"11445","inReplyTo":"7vsl1gy2si.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH 3/2] attribute \"coding\": specify blob encoding","fromName":"しらいしななこ","fromEmail":"nanako3@bluebottle.com","sentAt":"2008-01-03T21:23:25Z","receivedAt":"2008-01-03T21:23:25Z","isPatch":true,"sender":{"key":"nanako3@lavabit.com","avatar":"https://gravatar.com/avatar/3777b9e201c5883a62b1a6fdf7c53f2d712d1d80989146063ea861e33aad72a8?d=mp&s=160"},"body":"Quoting Junio C Hamano :\n\n> This teaches \"diff hunk header\" function about custom character\n> encoding per path via gitattributes(5) so that it can sensibly\n> chomp a line without truncating a character in the middle.\n>\n> Signed-off-by: Junio C Hamano \n> ---\n>\n> * This is not intended for serious inclusion, but was done more\n> as a demonstration of the concept, hence [3/2].\n\nWhy not? It looks a useful addition for us Japanese people.\n\n> +static struct {\n> +\tconst char *coding;\n> +\tsane_truncate_fn fn;\n> +} builtin_truncate_fn[] = {\n> +\t{ \"euc-jp\", truncate_euc_jp },\n> +\t{ \"utf-8\", NULL },\n> +};\n\nCan you also do JIS and Shift JIS? I ask because many of my old notes are in Shift JIS and I think it is the same for many other people.\n\n-- \nNanako Shiraishi\nhttp://ivory.ap.teacup.com/nanako3/\n\n----------------------------------------------------------------------\nGet a free email account with anti spam protection.\nhttp://www.bluebottle.com/tag/2\n"},{"id":"64401","messageId":"7vejcyo9ql.fsf@gitster.siamese.dyndns.org","threadId":"11445","inReplyTo":"200801032128.m03LSIn2022902@mi0.bluebottle.com","subject":"Re: [PATCH 3/2] attribute \"coding\": specify blob encoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-03T21:54:58Z","receivedAt":"2008-01-03T21:54:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"しらいしななこ writes:\n\n> Quoting Junio C Hamano :\n> \n>> This teaches \"diff hunk header\" function about custom character\n>> encoding per path via gitattributes(5) so that it can sensibly\n>> chomp a line without truncating a character in the middle.\n>>\n>> Signed-off-by: Junio C Hamano \n>> ---\n>>\n>> * This is not intended for serious inclusion, but was done more\n>> as a demonstration of the concept, hence [3/2].\n>\n> Why not? It looks a useful addition for us Japanese people.\n\n (offtopic) I was once told that \"us Japanese people\" is a\n bad thing to say in public because it sets an unfriendly\n tone by creating a psychological divide between \"us\" and\n \"others\". After all I am one of you ;-)\n\nThe reason I do not like the patch as-is is because I have\ndoubts about the way \"coding\" acts in the patch.\n\nThere already is clean/smudge filter mechanism. Even if your\nwork tree has files in euc-jp or Shift_JIS, you could choose to\ninternally use UTF-8 at git object level. Then the part that\ndeals with diff hunk headers (the topic of the patch we are\ndiscussing) would have to work only on UTF-8 data.\n\n\tSide note: when getting the data from a file in the work\n\ttree, we convert into internal representation before\n\trunning diff (see diff.c::diff_populate_filespec()), but\n\twe do not convert it back to external representation by\n\trunning the smudge filter on the diff output. We might\n\toptionally want to but if somebody is going to do this,\n\tthe patch accepting side also needs to be modified to\n\treverse the conversion.\n\nThe solution with clean/smudge is not applicable to everybody.\nIt needs to be agreed upon project-wide what encoding is used as\nthe canonical encoding for the project, and when the project\nchooses to use UTF-8, the above would become a cleaner and\nworkable approach.\n\nIf the project, on the other hand, chooses to use a non UTF-8\nencoding (e.g. euc-jp) as the canonical representation,\nsomething like my patch may be necessary.\n\nBetween these two ways to skin the cat, I do not want to close\nthe door for either one of them too early, although I am\nsomewhat partial to \"internally everything is UTF-8\" approach.\n\nMaybe we would want to use \"coding\" (short, sweet and nice name\nfor an attribute) to mean a canned smudge/clean filter pair that\nruns to/from UTF-8 iconv, making the \"internally, everything is\nUTF-8\" approach a more officially supported option. If we\nchoose to go that route, the way \"coding\" attribute was used in\nthe patch directly conflicts with that design, as the world view\nmy \"coding\" patch takes is \"whatever coding project chooses is\nused internally, and the attribute is used to teach coding\nspecific actions to the underlying logic\".\n\n>> +static struct {\n>> +\tconst char *coding;\n>> +\tsane_truncate_fn fn;\n>> +} builtin_truncate_fn[] = {\n>> +\t{ \"euc-jp\", truncate_euc_jp },\n>> +\t{ \"utf-8\", NULL },\n>> +};\n>\n>Can you also do JIS and Shift JIS? I ask because many of my\n>old notes are in Shift JIS and I think it is the same for many\n>other people. \n\nI guess somebody else could (hint, hint,...). Shift_JIS should\nbe more or less straightforward to add.\n\nWith the current code structure, however, ISO-2022 (you said\n\"JIS\" -- Japanese often use that word to mean 7-bit ISO-2022 and\nso did you in this context) is a bit cumbersome to handle, as\nyou cannot just truncate but also have to add a few escape bytes\nto go back to ASCII at the end of line.\n"},{"id":"64442","messageId":"20080105.011649.26949633.tshibata@ab.jp.nec.com","threadId":"11445","inReplyTo":"7vejcyo9ql.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH 3/2] attribute \"coding\": specify blob encoding","fromName":"Tsugikazu Shibata","fromEmail":"tshibata@ab.jp.nec.com","sentAt":"2008-01-04T16:16:49Z","receivedAt":"2008-01-04T16:16:49Z","isPatch":true,"sender":{"key":"tshibata@ab.jp.nec.com","avatar":null},"body":"On Thu, 03 Jan 2008 13:54:58 -0800, gitster wrote:\n> しらいしななこ writes:\n> \n> > Quoting Junio C Hamano :\n> > \n> >> This teaches \"diff hunk header\" function about custom character\n> >> encoding per path via gitattributes(5) so that it can sensibly\n> >> chomp a line without truncating a character in the middle.\n> >>\n> >> Signed-off-by: Junio C Hamano \n> >> ---\n> >>\n> >> * This is not intended for serious inclusion, but was done more\n> >> as a demonstration of the concept, hence [3/2].\n> >\n...\n> >> +static struct {\n> >> +\tconst char *coding;\n> >> +\tsane_truncate_fn fn;\n> >> +} builtin_truncate_fn[] = {\n> >> +\t{ \"euc-jp\", truncate_euc_jp },\n> >> +\t{ \"utf-8\", NULL },\n> >> +};\n> >\n> >Can you also do JIS and Shift JIS? I ask because many of my\n> >old notes are in Shift JIS and I think it is the same for many\n> >other people. \n> \n> I guess somebody else could (hint, hint,...). Shift_JIS should\n> be more or less straightforward to add.\n> \n> With the current code structure, however, ISO-2022 (you said\n> \"JIS\" -- Japanese often use that word to mean 7-bit ISO-2022 and\n> so did you in this context) is a bit cumbersome to handle, as\n> you cannot just truncate but also have to add a few escape bytes\n> to go back to ASCII at the end of line.\n\nI guess that there are many other encodings and support everything are\nnot reasonable.\nIn my experience, It seems that chopping a multi-byte character\noccurs the field after \"@@\". I believe this field is for the name of\nthe appropriate function. Also, I believe most of the function names\nare in ASCII. \nSo, question is why we should think of this field in case of non\nprogramming language files ? \nIn case of text file using any coding, Should we need to add something\nafter \"@@\"?\nHow about not to add anything after \"@@\" when the file name was .txt\nor no extension (ie. HOWTO)?\n"},{"id":"64464","messageId":"7vwsqpia7m.fsf@gitster.siamese.dyndns.org","threadId":"11445","inReplyTo":"20080105.011649.26949633.tshibata@ab.jp.nec.com","subject":"Re: [PATCH 3/2] attribute \"coding\": specify blob encoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-04T20:53:33Z","receivedAt":"2008-01-04T20:53:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tsugikazu Shibata writes:\n\n> So, question is why we should think of this field in case of non\n> programming language files ? \n> In case of text file using any coding, Should we need to add something\n> after \"@@\"?\n\nSee gitattributes(5), \"Defining a custom hunk-header\". The\nassociated configuration variable for this feature is somewhat\nmisnamed [*1*], but if you do not want to see anything, I think\nyou should be able to define a custom pattern that match\nemptyness.\n\nFor .txt files, I always hoped that someday, somebody clever\nwould come up with a custom pattern to grok AsciiDoc mark-up to\npick a label-looking line so that the hunks in patches to our\ndocumentation would be easier to find. A simpler and more\ngenerally applicable possibility is to find the previous line\nthat starts a paragraph (e.g. you would pick \"For .txt files...\"\nfor this part because it is the non-blank line that immediately\nfollows the first blank line before this part). But I think the\ncode to find an appropriate string for hunk header is not\ncapable of looking at more than one line, so neither of these\nwould be easy to implement without changing the underlying logic\nand semantics of \"diff.*.funcname\" configuration.\n\n[Footnote]\n\n*1* It may be misnamed but it is in line with GNU diff, though.\nThey call -p \"show-c-function\" and -F \"show-function-line\" --\nboth assume function names are what people are interested in\nhaving on hunk headers.\n"}]}