{"thread":{"id":"36628","subject":"[PATCH 2/2] Make it possible to update git_wcwidth()","startedAt":"2014-05-09T21:51:44Z","lastAt":"2014-05-12T17:54:41Z","messageCount":4,"participants":["Torsten Bögershausen","Peter Krefting","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"241231","messageId":"536D4DF0.3060301@web.de","threadId":"36628","inReplyTo":null,"subject":"[PATCH 2/2] Make it possible to update git_wcwidth()","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-05-09T21:51:44Z","receivedAt":"2014-05-09T21:51:44Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"The function git_wcwidth() returns for a given unicode code point the\nwidth on the display:\n-1 for control characters,\n 0 for combining or other non-visible code points\n 1 for e.g. ASCII\n 2 for double-width code points.\n\nThis table had been originally been extracted for one Unicode version,\nprobably 3.2.\n\nMake it possible to update the to a later version of Unicode:\n\nFactor out the table from utf8.c into unicode_width.h and\nadd the script update_unicode.sh to update the table based on the latest\nUnicode specification files.\n\nThanks to\nPeter Krefting <peter@softwolves.pp.se> and\nKevin Bracey <kevin@bracey.fi>\nfor helping with their Unicode knowledge\n\nSigned-off-by: Torsten Bögershausen <tboegi@web.de>\n---\n .gitignore        |   1 +\n Makefile          |   1 +\n unicode_width.h   | 288 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n update_unicode.sh |  37 +++++++\n utf8.c            |  61 +-----------\n 5 files changed, 329 insertions(+), 59 deletions(-)\n create mode 100644 unicode_width.h\n create mode 100755 update_unicode.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex dc600f9..42294e5 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -226,6 +226,7 @@\n /config.mak.autogen\n /config.mak.append\n /configure\n+/unicode\n /tags\n /TAGS\n /cscope*\ndiff --git a/Makefile b/Makefile\nindex a53f3a8..2e2077a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -730,6 +730,7 @@ LIB_H += transport.h\n LIB_H += tree-walk.h\n LIB_H += tree.h\n LIB_H += unpack-trees.h\n+LIB_H += unicode_width.h\n LIB_H += url.h\n LIB_H += urlmatch.h\n LIB_H += userdiff.h\ndiff --git a/unicode_width.h b/unicode_width.h\nnew file mode 100644\nindex 0000000..4db7803\n--- /dev/null\n+++ b/unicode_width.h\n@@ -0,0 +1,288 @@\n+static const struct interval zero_width[] = {\n+{ 0x0300, 0x036F },\n+{ 0x0483, 0x0489 },\n+{ 0x0591, 0x05BD },\n+{ 0x05BF, 0x05BF },\n+{ 0x05C1, 0x05C2 },\n+{ 0x05C4, 0x05C5 },\n+{ 0x05C7, 0x05C7 },\n+{ 0x0600, 0x0604 },\n+{ 0x0610, 0x061A },\n+{ 0x061C, 0x061C },\n+{ 0x064B, 0x065F },\n+{ 0x0670, 0x0670 },\n+{ 0x06D6, 0x06DD },\n+{ 0x06DF, 0x06E4 },\n+{ 0x06E7, 0x06E8 },\n+{ 0x06EA, 0x06ED },\n+{ 0x070F, 0x070F },\n+{ 0x0711, 0x0711 },\n+{ 0x0730, 0x074A },\n+{ 0x07A6, 0x07B0 },\n+{ 0x07EB, 0x07F3 },\n+{ 0x0816, 0x0819 },\n+{ 0x081B, 0x0823 },\n+{ 0x0825, 0x0827 },\n+{ 0x0829, 0x082D },\n+{ 0x0859, 0x085B },\n+{ 0x08E4, 0x08FE },\n+{ 0x0900, 0x0902 },\n+{ 0x093A, 0x093A },\n+{ 0x093C, 0x093C },\n+{ 0x0941, 0x0948 },\n+{ 0x094D, 0x094D },\n+{ 0x0951, 0x0957 },\n+{ 0x0962, 0x0963 },\n+{ 0x0981, 0x0981 },\n+{ 0x09BC, 0x09BC },\n+{ 0x09C1, 0x09C4 },\n+{ 0x09CD, 0x09CD },\n+{ 0x09E2, 0x09E3 },\n+{ 0x0A01, 0x0A02 },\n+{ 0x0A3C, 0x0A3C },\n+{ 0x0A41, 0x0A42 },\n+{ 0x0A47, 0x0A48 },\n+{ 0x0A4B, 0x0A4D },\n+{ 0x0A51, 0x0A51 },\n+{ 0x0A70, 0x0A71 },\n+{ 0x0A75, 0x0A75 },\n+{ 0x0A81, 0x0A82 },\n+{ 0x0ABC, 0x0ABC },\n+{ 0x0AC1, 0x0AC5 },\n+{ 0x0AC7, 0x0AC8 },\n+{ 0x0ACD, 0x0ACD },\n+{ 0x0AE2, 0x0AE3 },\n+{ 0x0B01, 0x0B01 },\n+{ 0x0B3C, 0x0B3C },\n+{ 0x0B3F, 0x0B3F },\n+{ 0x0B41, 0x0B44 },\n+{ 0x0B4D, 0x0B4D },\n+{ 0x0B56, 0x0B56 },\n+{ 0x0B62, 0x0B63 },\n+{ 0x0B82, 0x0B82 },\n+{ 0x0BC0, 0x0BC0 },\n+{ 0x0BCD, 0x0BCD },\n+{ 0x0C3E, 0x0C40 },\n+{ 0x0C46, 0x0C48 },\n+{ 0x0C4A, 0x0C4D },\n+{ 0x0C55, 0x0C56 },\n+{ 0x0C62, 0x0C63 },\n+{ 0x0CBC, 0x0CBC },\n+{ 0x0CBF, 0x0CBF },\n+{ 0x0CC6, 0x0CC6 },\n+{ 0x0CCC, 0x0CCD },\n+{ 0x0CE2, 0x0CE3 },\n+{ 0x0D41, 0x0D44 },\n+{ 0x0D4D, 0x0D4D },\n+{ 0x0D62, 0x0D63 },\n+{ 0x0DCA, 0x0DCA },\n+{ 0x0DD2, 0x0DD4 },\n+{ 0x0DD6, 0x0DD6 },\n+{ 0x0E31, 0x0E31 },\n+{ 0x0E34, 0x0E3A },\n+{ 0x0E47, 0x0E4E },\n+{ 0x0EB1, 0x0EB1 },\n+{ 0x0EB4, 0x0EB9 },\n+{ 0x0EBB, 0x0EBC },\n+{ 0x0EC8, 0x0ECD },\n+{ 0x0F18, 0x0F19 },\n+{ 0x0F35, 0x0F35 },\n+{ 0x0F37, 0x0F37 },\n+{ 0x0F39, 0x0F39 },\n+{ 0x0F71, 0x0F7E },\n+{ 0x0F80, 0x0F84 },\n+{ 0x0F86, 0x0F87 },\n+{ 0x0F8D, 0x0F97 },\n+{ 0x0F99, 0x0FBC },\n+{ 0x0FC6, 0x0FC6 },\n+{ 0x102D, 0x1030 },\n+{ 0x1032, 0x1037 },\n+{ 0x1039, 0x103A },\n+{ 0x103D, 0x103E },\n+{ 0x1058, 0x1059 },\n+{ 0x105E, 0x1060 },\n+{ 0x1071, 0x1074 },\n+{ 0x1082, 0x1082 },\n+{ 0x1085, 0x1086 },\n+{ 0x108D, 0x108D },\n+{ 0x109D, 0x109D },\n+{ 0x1160, 0x11FF },\n+{ 0x135D, 0x135F },\n+{ 0x1712, 0x1714 },\n+{ 0x1732, 0x1734 },\n+{ 0x1752, 0x1753 },\n+{ 0x1772, 0x1773 },\n+{ 0x17B4, 0x17B5 },\n+{ 0x17B7, 0x17BD },\n+{ 0x17C6, 0x17C6 },\n+{ 0x17C9, 0x17D3 },\n+{ 0x17DD, 0x17DD },\n+{ 0x180B, 0x180E },\n+{ 0x18A9, 0x18A9 },\n+{ 0x1920, 0x1922 },\n+{ 0x1927, 0x1928 },\n+{ 0x1932, 0x1932 },\n+{ 0x1939, 0x193B },\n+{ 0x1A17, 0x1A18 },\n+{ 0x1A1B, 0x1A1B },\n+{ 0x1A56, 0x1A56 },\n+{ 0x1A58, 0x1A5E },\n+{ 0x1A60, 0x1A60 },\n+{ 0x1A62, 0x1A62 },\n+{ 0x1A65, 0x1A6C },\n+{ 0x1A73, 0x1A7C },\n+{ 0x1A7F, 0x1A7F },\n+{ 0x1B00, 0x1B03 },\n+{ 0x1B34, 0x1B34 },\n+{ 0x1B36, 0x1B3A },\n+{ 0x1B3C, 0x1B3C },\n+{ 0x1B42, 0x1B42 },\n+{ 0x1B6B, 0x1B73 },\n+{ 0x1B80, 0x1B81 },\n+{ 0x1BA2, 0x1BA5 },\n+{ 0x1BA8, 0x1BA9 },\n+{ 0x1BAB, 0x1BAB },\n+{ 0x1BE6, 0x1BE6 },\n+{ 0x1BE8, 0x1BE9 },\n+{ 0x1BED, 0x1BED },\n+{ 0x1BEF, 0x1BF1 },\n+{ 0x1C2C, 0x1C33 },\n+{ 0x1C36, 0x1C37 },\n+{ 0x1CD0, 0x1CD2 },\n+{ 0x1CD4, 0x1CE0 },\n+{ 0x1CE2, 0x1CE8 },\n+{ 0x1CED, 0x1CED },\n+{ 0x1CF4, 0x1CF4 },\n+{ 0x1DC0, 0x1DE6 },\n+{ 0x1DFC, 0x1DFF },\n+{ 0x200B, 0x200F },\n+{ 0x202A, 0x202E },\n+{ 0x2060, 0x2064 },\n+{ 0x2066, 0x206F },\n+{ 0x20D0, 0x20F0 },\n+{ 0x2CEF, 0x2CF1 },\n+{ 0x2D7F, 0x2D7F },\n+{ 0x2DE0, 0x2DFF },\n+{ 0x302A, 0x302D },\n+{ 0x3099, 0x309A },\n+{ 0xA66F, 0xA672 },\n+{ 0xA674, 0xA67D },\n+{ 0xA69F, 0xA69F },\n+{ 0xA6F0, 0xA6F1 },\n+{ 0xA802, 0xA802 },\n+{ 0xA806, 0xA806 },\n+{ 0xA80B, 0xA80B },\n+{ 0xA825, 0xA826 },\n+{ 0xA8C4, 0xA8C4 },\n+{ 0xA8E0, 0xA8F1 },\n+{ 0xA926, 0xA92D },\n+{ 0xA947, 0xA951 },\n+{ 0xA980, 0xA982 },\n+{ 0xA9B3, 0xA9B3 },\n+{ 0xA9B6, 0xA9B9 },\n+{ 0xA9BC, 0xA9BC },\n+{ 0xAA29, 0xAA2E },\n+{ 0xAA31, 0xAA32 },\n+{ 0xAA35, 0xAA36 },\n+{ 0xAA43, 0xAA43 },\n+{ 0xAA4C, 0xAA4C },\n+{ 0xAAB0, 0xAAB0 },\n+{ 0xAAB2, 0xAAB4 },\n+{ 0xAAB7, 0xAAB8 },\n+{ 0xAABE, 0xAABF },\n+{ 0xAAC1, 0xAAC1 },\n+{ 0xAAEC, 0xAAED },\n+{ 0xAAF6, 0xAAF6 },\n+{ 0xABE5, 0xABE5 },\n+{ 0xABE8, 0xABE8 },\n+{ 0xABED, 0xABED },\n+{ 0xFB1E, 0xFB1E },\n+{ 0xFE00, 0xFE0F },\n+{ 0xFE20, 0xFE26 },\n+{ 0xFEFF, 0xFEFF },\n+{ 0xFFF9, 0xFFFB },\n+{ 0x101FD, 0x101FD },\n+{ 0x10A01, 0x10A03 },\n+{ 0x10A05, 0x10A06 },\n+{ 0x10A0C, 0x10A0F },\n+{ 0x10A38, 0x10A3A },\n+{ 0x10A3F, 0x10A3F },\n+{ 0x11001, 0x11001 },\n+{ 0x11038, 0x11046 },\n+{ 0x11080, 0x11081 },\n+{ 0x110B3, 0x110B6 },\n+{ 0x110B9, 0x110BA },\n+{ 0x110BD, 0x110BD },\n+{ 0x11100, 0x11102 },\n+{ 0x11127, 0x1112B },\n+{ 0x1112D, 0x11134 },\n+{ 0x11180, 0x11181 },\n+{ 0x111B6, 0x111BE },\n+{ 0x116AB, 0x116AB },\n+{ 0x116AD, 0x116AD },\n+{ 0x116B0, 0x116B5 },\n+{ 0x116B7, 0x116B7 },\n+{ 0x16F8F, 0x16F92 },\n+{ 0x1D167, 0x1D169 },\n+{ 0x1D173, 0x1D182 },\n+{ 0x1D185, 0x1D18B },\n+{ 0x1D1AA, 0x1D1AD },\n+{ 0x1D242, 0x1D244 },\n+{ 0xE0001, 0xE0001 },\n+{ 0xE0020, 0xE007F },\n+{ 0xE0100, 0xE01EF }\n+};\n+static const struct interval double_width[] = {\n+{ /* plane */ 0x0, 0x1C },\n+{ /* plane */ 0x1C, 0x21 },\n+{ /* plane */ 0x21, 0x22 },\n+{ /* plane */ 0x22, 0x23 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ /* plane */ 0x0, 0x0 },\n+{ 0x1100, 0x115F },\n+{ 0x2329, 0x232A },\n+{ 0x2E80, 0x2E99 },\n+{ 0x2E9B, 0x2EF3 },\n+{ 0x2F00, 0x2FD5 },\n+{ 0x2FF0, 0x2FFB },\n+{ 0x3000, 0x303E },\n+{ 0x3041, 0x3096 },\n+{ 0x3099, 0x30FF },\n+{ 0x3105, 0x312D },\n+{ 0x3131, 0x318E },\n+{ 0x3190, 0x31BA },\n+{ 0x31C0, 0x31E3 },\n+{ 0x31F0, 0x321E },\n+{ 0x3220, 0x3247 },\n+{ 0x3250, 0x32FE },\n+{ 0x3300, 0x4DBF },\n+{ 0x4E00, 0xA48C },\n+{ 0xA490, 0xA4C6 },\n+{ 0xA960, 0xA97C },\n+{ 0xAC00, 0xD7A3 },\n+{ 0xF900, 0xFAFF },\n+{ 0xFE10, 0xFE19 },\n+{ 0xFE30, 0xFE52 },\n+{ 0xFE54, 0xFE66 },\n+{ 0xFE68, 0xFE6B },\n+{ 0xFF01, 0xFF60 },\n+{ 0xFFE0, 0xFFE6 },\n+{ 0x1B000, 0x1B001 },\n+{ 0x1F200, 0x1F202 },\n+{ 0x1F210, 0x1F23A },\n+{ 0x1F240, 0x1F248 },\n+{ 0x1F250, 0x1F251 },\n+{ 0x20000, 0x2FFFD },\n+{ 0x30000, 0x3FFFD }\n+};\ndiff --git a/update_unicode.sh b/update_unicode.sh\nnew file mode 100755\nindex 0000000..000b937\n--- /dev/null\n+++ b/update_unicode.sh\n@@ -0,0 +1,37 @@\n+#!/bin/sh\n+#See http://www.unicode.org/reports/tr44/\n+#\n+#Me Enclosing_Mark  an enclosing combining mark\n+#Mn Nonspacing_Mark a nonspacing combining mark (zero advance width)\n+#Cf Format          a format control character\n+#\n+UNICODEWIDTH_H=../unicode_width.h\n+if ! test -d unicode; then\n+\tmkdir unicode\n+fi &&\n+( cd unicode &&\n+\tif ! test -f UnicodeData.txt; then\n+\t\twget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt\n+\tfi &&\n+\tif ! test -f EastAsianWidth.txt; then\n+\t\twget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt\n+\tfi &&\n+\tif ! test -d uniset; then\n+\t\tgit clone https://github.com/depp/uniset.git\n+\tfi &&\n+\t(\n+\t\tcd uniset &&\n+\t\tif ! test -x uniset; then\n+\t\t\tautoreconf -i &&\n+\t\t\t./configure --enable-warnings=-Werror CFLAGS='-O0 -ggdb'\n+\t\tfi &&\n+\t\tmake\n+\t) &&\n+\techo \"static const struct interval zero_width[] = {\" >$UNICODEWIDTH_H &&\n+\tUNICODE_DIR=. ./uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD |\n+\tgrep -v plane >>$UNICODEWIDTH_H &&\n+\techo \"};\" >>$UNICODEWIDTH_H &&\n+\techo \"static const struct interval double_width[] = {\" >>$UNICODEWIDTH_H &&\n+\tUNICODE_DIR=. ./uniset/uniset --32 eaw:F,W >>$UNICODEWIDTH_H &&\n+\techo \"};\" >>$UNICODEWIDTH_H\n+)\ndiff --git a/utf8.c b/utf8.c\nindex b5d8136..b30790d 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -80,65 +80,8 @@ static int git_wcwidth(ucs_char_t ch)\n {\n \t/*\n \t * Sorted list of non-overlapping intervals of non-spacing characters,\n-\t * generated by\n-\t *   \"uniset +cat=Me +cat=Mn +cat=Cf -00AD +1160-11FF +200B c\".\n \t */\n-\tstatic const struct interval combining[] = {\n-\t\t{ 0x0300, 0x036F }, { 0x0483, 0x0489 }, { 0x0591, 0x05BD },\n-\t\t{ 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 }, { 0x05C4, 0x05C5 },\n-\t\t{ 0x05C7, 0x05C7 }, { 0x0600, 0x0604 }, { 0x0610, 0x061A },\n-\t\t{ 0x064B, 0x065F }, { 0x0670, 0x0670 }, { 0x06D6, 0x06E4 },\n-\t\t{ 0x06E7, 0x06E8 }, { 0x06EA, 0x06ED }, { 0x070F, 0x070F },\n-\t\t{ 0x0711, 0x0711 }, { 0x0730, 0x074A }, { 0x07A6, 0x07B0 },\n-\t\t{ 0x0901, 0x0902 }, { 0x093C, 0x093C }, { 0x0941, 0x0948 },\n-\t\t{ 0x094D, 0x094D }, { 0x0951, 0x0954 }, { 0x0962, 0x0963 },\n-\t\t{ 0x0981, 0x0981 }, { 0x09BC, 0x09BC }, { 0x09C1, 0x09C4 },\n-\t\t{ 0x09CD, 0x09CD }, { 0x09E2, 0x09E3 }, { 0x0A01, 0x0A02 },\n-\t\t{ 0x0A3C, 0x0A3C }, { 0x0A41, 0x0A42 }, { 0x0A47, 0x0A48 },\n-\t\t{ 0x0A4B, 0x0A4D }, { 0x0A70, 0x0A71 }, { 0x0A81, 0x0A82 },\n-\t\t{ 0x0ABC, 0x0ABC }, { 0x0AC1, 0x0AC5 }, { 0x0AC7, 0x0AC8 },\n-\t\t{ 0x0ACD, 0x0ACD }, { 0x0AE2, 0x0AE3 }, { 0x0B01, 0x0B01 },\n-\t\t{ 0x0B3C, 0x0B3C }, { 0x0B3F, 0x0B3F }, { 0x0B41, 0x0B43 },\n-\t\t{ 0x0B4D, 0x0B4D }, { 0x0B56, 0x0B56 }, { 0x0B82, 0x0B82 },\n-\t\t{ 0x0BC0, 0x0BC0 }, { 0x0BCD, 0x0BCD }, { 0x0C3E, 0x0C40 },\n-\t\t{ 0x0C46, 0x0C48 }, { 0x0C4A, 0x0C4D }, { 0x0C55, 0x0C56 },\n-\t\t{ 0x0CBC, 0x0CBC }, { 0x0CBF, 0x0CBF }, { 0x0CC6, 0x0CC6 },\n-\t\t{ 0x0CCC, 0x0CCD }, { 0x0D41, 0x0D43 }, { 0x0D4D, 0x0D4D },\n-\t\t{ 0x0DCA, 0x0DCA }, { 0x0DD2, 0x0DD4 }, { 0x0DD6, 0x0DD6 },\n-\t\t{ 0x0E31, 0x0E31 }, { 0x0E34, 0x0E3A }, { 0x0E47, 0x0E4E },\n-\t\t{ 0x0EB1, 0x0EB1 }, { 0x0EB4, 0x0EB9 }, { 0x0EBB, 0x0EBC },\n-\t\t{ 0x0EC8, 0x0ECD }, { 0x0F18, 0x0F19 }, { 0x0F35, 0x0F35 },\n-\t\t{ 0x0F37, 0x0F37 }, { 0x0F39, 0x0F39 }, { 0x0F71, 0x0F7E },\n-\t\t{ 0x0F80, 0x0F84 }, { 0x0F86, 0x0F87 }, { 0x0F90, 0x0F97 },\n-\t\t{ 0x0F99, 0x0FBC }, { 0x0FC6, 0x0FC6 }, { 0x102D, 0x1030 },\n-\t\t{ 0x1032, 0x1032 }, { 0x1036, 0x1037 }, { 0x1039, 0x1039 },\n-\t\t{ 0x1058, 0x1059 }, { 0x1160, 0x11FF }, { 0x1712, 0x1714 },\n-\t\t{ 0x1732, 0x1734 }, { 0x1752, 0x1753 }, { 0x1772, 0x1773 },\n-\t\t{ 0x17B4, 0x17B5 }, { 0x17B7, 0x17BD }, { 0x17C6, 0x17C6 },\n-\t\t{ 0x17C9, 0x17D3 }, { 0x17DD, 0x17DD }, { 0x180B, 0x180D },\n-\t\t{ 0x18A9, 0x18A9 }, { 0x1920, 0x1922 }, { 0x1927, 0x1928 },\n-\t\t{ 0x1932, 0x1932 }, { 0x1939, 0x193B }, { 0x200B, 0x200F },\n-\t\t{ 0x202A, 0x202E }, { 0x2060, 0x2063 }, { 0x206A, 0x206F },\n-\t\t{ 0x20D0, 0x20EA }, { 0x302A, 0x302F }, { 0x3099, 0x309A },\n-\t\t{ 0xFB1E, 0xFB1E }, { 0xFE00, 0xFE0F }, { 0xFE20, 0xFE23 },\n-\t\t{ 0xFEFF, 0xFEFF }, { 0xFFF9, 0xFFFB }, { 0x1D167, 0x1D169 },\n-\t\t{ 0x1D173, 0x1D182 }, { 0x1D185, 0x1D18B },\n-\t\t{ 0x1D1AA, 0x1D1AD }, { 0xE0001, 0xE0001 },\n-\t\t{ 0xE0020, 0xE007F }, { 0xE0100, 0xE01EF }\n-\t};\n-\tstatic const struct interval double_width[] = {\n-\t\t{ 0x1100, 0x115F },\n-\t\t{ 0x2329, 0x232A },\n-\t\t{ 0x2E80, 0x303E },\n-\t\t{ 0x3040, 0xA4CF },\n-\t\t{ 0xAC00, 0xD7A3 },\n-\t\t{ 0xF900, 0xFAFF },\n-\t\t{ 0xFE30, 0xFE6F },\n-\t\t{ 0xFF00, 0xFF60 },\n-\t\t{ 0xFFE0, 0xFFE6 },\n-\t\t{ 0x20000, 0x2FFFD },\n-\t\t{ 0x30000, 0x3FFFD }\n-\t};\n+#include \"unicode_width.h\"\n \n \t/* test for 8-bit control characters */\n \tif (ch == 0)\n@@ -147,7 +90,7 @@ static int git_wcwidth(ucs_char_t ch)\n \t\treturn -1;\n \n \t/* binary search in table of non-spacing characters */\n-\tif (bisearch(ch, combining, sizeof(combining)\n+\tif (bisearch(ch, zero_width, sizeof(zero_width)\n \t\t\t\t/ sizeof(struct interval) - 1))\n \t\treturn 0;\n \n-- \n1.9.2.691.g8d8dc6d\n"},{"id":"241290","messageId":"alpine.DEB.2.00.1405121334480.9884@ds9.cixit.se","threadId":"36628","inReplyTo":"536D4DF0.3060301@web.de","subject":"Re: [PATCH 2/2] Make it possible to update git_wcwidth()","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2014-05-12T12:38:37Z","receivedAt":"2014-05-12T12:38:37Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Torsten Bögershausen:\n\n> The function git_wcwidth() returns for a given unicode code point the\n> width on the display:\n> -1 for control characters,\n> 0 for combining or other non-visible code points\n> 1 for e.g. ASCII\n> 2 for double-width code points.\n\nThis all looks sane, but the problem is that this is also \ncontext-dependent since there are a lot of characters with \"ambiguous\" \nwidths, i.e., characters that are \"double\" width for CJK locales (and \nfonts), but \"single\" width for others. This includes Greek and \nCyrillic characters, which are encoded using the double-byte parts of \nthe CJK DBCS encodings.\n\nI'm not quite sure how much impact this would have on day-to-day Git \noperation in a CJK locale, however, as I guess they would mostly \nencounter texts in their own language (which would mostly be \"double\" \nwidth) or English (which would be unambiguously \"single\" width).\n\nAnyone on the list running Git in a CJK locale that would like to \nweigh in here?\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"241305","messageId":"xmqqegzyu96f.fsf@gitster.dls.corp.google.com","threadId":"36628","inReplyTo":"alpine.DEB.2.00.1405121334480.9884@ds9.cixit.se","subject":"Re: [PATCH 2/2] Make it possible to update git_wcwidth()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-12T17:44:05Z","receivedAt":"2014-05-12T17:44:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Peter Krefting <peter@softwolves.pp.se> writes:\n\n> Torsten Bögershausen:\n>\n>> The function git_wcwidth() returns for a given unicode code point the\n>> width on the display:\n>> -1 for control characters,\n>> 0 for combining or other non-visible code points\n>> 1 for e.g. ASCII\n>> 2 for double-width code points.\n>\n> This all looks sane, but the problem is that this is also\n> context-dependent since there are a lot of characters with \"ambiguous\"\n> widths, i.e., characters that are \"double\" width for CJK locales (and\n> fonts), but \"single\" width for others. This includes Greek and\n> Cyrillic characters, which are encoded using the double-byte parts of\n> the CJK DBCS encodings.\n>\n> I'm not quite sure how much impact this would have on day-to-day Git\n> operation in a CJK locale, however, as I guess they would mostly\n> encounter texts in their own language (which would mostly be \"double\"\n> width) or English (which would be unambiguously \"single\" width).\n>\n> Anyone on the list running Git in a CJK locale that would like to\n> weigh in here?\n\nThe issue does appear in the real life.  A solution I've seen used\nin a terminaul emulator program was to give the user a choice to say\n\"I want ambiguous ones to be treated as double (or single)\".  As a\nJ-locale user, I naturally set the configuration to double while\nusing that program (I no longer do).\n"},{"id":"241306","messageId":"xmqq8uq6u967.fsf@gitster.dls.corp.google.com","threadId":"36628","inReplyTo":"536D4DF0.3060301@web.de","subject":"Re: [PATCH 2/2] Make it possible to update git_wcwidth()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-12T17:54:41Z","receivedAt":"2014-05-12T17:54:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> The function git_wcwidth() returns for a given unicode code point the\n> width on the display:\n> -1 for control characters,\n>  0 for combining or other non-visible code points\n>  1 for e.g. ASCII\n>  2 for double-width code points.\n>\n> This table had been originally been extracted for one Unicode version,\n> probably 3.2.\n>\n> Make it possible to update the to a later version of Unicode:\n>\n> Factor out the table from utf8.c into unicode_width.h and\n> add the script update_unicode.sh to update the table based on the latest\n> Unicode specification files.\n>\n> Thanks to\n> Peter Krefting <peter@softwolves.pp.se> and\n> Kevin Bracey <kevin@bracey.fi>\n> for helping with their Unicode knowledge\n>\n> Signed-off-by: Torsten Bögershausen <tboegi@web.de>\n> ---\n\nI would say this is a good idea, but a few nitpicks.\n\n> diff --git a/unicode_width.h b/unicode_width.h\n> new file mode 100644\n> index 0000000..4db7803\n> --- /dev/null\n> +++ b/unicode_width.h\n> @@ -0,0 +1,288 @@\n\nPlease update the script (and the resulting file) to caution against\nmisuse/mismanagement of this file by adding a comment to at least\nstate:\n\n - This is a generated file and you are not supposed to modify it; and\n\n - This is to be included only once from one place in the code and\n   that is why this does not use the usual #ifndef X/#define X/#endif\n   double-inclusion guards.\n\nAn obvious and viable alternative to the second would be to do the\nusual double-inclusion guard.  I do not have much preference either\nway.\n\n> +static const struct interval zero_width[] = {\n> ...\n> +};\n> +static const struct interval double_width[] = {\n> ...\n> +};\n> diff --git a/update_unicode.sh b/update_unicode.sh\n> new file mode 100755\n> index 0000000..000b937\n> --- /dev/null\n> +++ b/update_unicode.sh\n> @@ -0,0 +1,37 @@\n> +#!/bin/sh\n> +#See http://www.unicode.org/reports/tr44/\n> +#\n> +#Me Enclosing_Mark  an enclosing combining mark\n> +#Mn Nonspacing_Mark a nonspacing combining mark (zero advance width)\n> +#Cf Format          a format control character\n\nPlease have a SP after # in these comments to make them readable?\n\n> +#\n> +UNICODEWIDTH_H=../unicode_width.h\n> +if ! test -d unicode; then\n> +\tmkdir unicode\n> +fi &&\n\nStyle:\n\n\tif ! test -d unicode\n        then\n        \t...\n\tfi\n\n> +( cd unicode &&\n> +\tif ! test -f UnicodeData.txt; then\n> +\t\twget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt\n> +\tfi &&\n> +\tif ! test -f EastAsianWidth.txt; then\n> +\t\twget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt\n> +\tfi &&\n> +\tif ! test -d uniset; then\n> +\t\tgit clone https://github.com/depp/uniset.git\n> +\tfi &&\n> +\t(\n> +\t\tcd uniset &&\n> +\t\tif ! test -x uniset; then\n> +\t\t\tautoreconf -i &&\n> +\t\t\t./configure --enable-warnings=-Werror CFLAGS='-O0 -ggdb'\n\nWhat are these \"-O0 -ggdb\" about?\n\n> +\t\tfi &&\n> +\t\tmake\n> +\t) &&\n> +\techo \"static const struct interval zero_width[] = {\" >$UNICODEWIDTH_H &&\n> +\tUNICODE_DIR=. ./uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD |\n> +\tgrep -v plane >>$UNICODEWIDTH_H &&\n> +\techo \"};\" >>$UNICODEWIDTH_H &&\n> +\techo \"static const struct interval double_width[] = {\" >>$UNICODEWIDTH_H &&\n> +\tUNICODE_DIR=. ./uniset/uniset --32 eaw:F,W >>$UNICODEWIDTH_H &&\n> +\techo \"};\" >>$UNICODEWIDTH_H\n> +)\n> @@ -147,7 +90,7 @@ static int git_wcwidth(ucs_char_t ch)\n>  \t\treturn -1;\n>  \n>  \t/* binary search in table of non-spacing characters */\n> -\tif (bisearch(ch, combining, sizeof(combining)\n> +\tif (bisearch(ch, zero_width, sizeof(zero_width)\n>  \t\t\t\t/ sizeof(struct interval) - 1))\n\nTo my eyes, that line looks folded at a funny place.  I think it is\nmore conventional to fold after the operator, i.e.\n\n\tif (bisearch(ch, zero_width, sizeof(zero_width)\t/\n\t\t\t\tsizeof(struct interval) - 1))\n\nbut\n\n\tif (bisearch(ch, zero_width,\n  \t\t     sizeof(zero_width)\t/ sizeof(struct interval) - 1))\n\nmay probably be a lot more logical and readable.  Maybe it is just\nme.\n"}]}