{"thread":{"id":"44599","subject":"[PATCH 1/2] update-unicode.sh: automatically download newer definition files","startedAt":"2016-12-02T21:32:32Z","lastAt":"2016-12-07T01:00:42Z","messageCount":18,"participants":["Beat Bolli","Torsten B��gershausen","Torsten Bögershausen","Junio C Hamano","jbh"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"306816","messageId":"1480713995-16157-1-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":null,"subject":"[PATCH 1/2] update-unicode.sh: automatically download newer definition files","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-02T21:26:34Z","receivedAt":"2016-12-02T21:32:32Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"Checking just for the files' existence is not enough; we should also\ndownload them if a newer version exists on the Unicode servers.\n\nSigned-off-by: Beat Bolli \n---\n update_unicode.sh | 8 ++------\n 1 file changed, 2 insertions(+), 6 deletions(-)\n\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 27af77c..3c84270 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -10,12 +10,8 @@ if ! test -d unicode; then\n \tmkdir unicode\n fi &&\n ( cd unicode &&\n-\tif ! test -f UnicodeData.txt; then\n-\t\twget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt\n-\tfi &&\n-\tif ! test -f EastAsianWidth.txt; then\n-\t\twget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt\n-\tfi &&\n+\twget -N http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt \\\n+\t\thttp://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt &&\n \tif ! test -d uniset; then\n \t\tgit clone https://github.com/depp/uniset.git\n \tfi &&\n-- \n2.7.2\n"},{"id":"306817","messageId":"1480713995-16157-2-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480713995-16157-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH 2/2] unicode: update the tables to Unicode 9.0","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-02T21:26:35Z","receivedAt":"2016-12-02T21:32:53Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"A rerun of the previously fixed update-unicode.sh produces these new\ntables.\n\nSigned-off-by: Beat Bolli \n---\n unicode_width.h | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 111 insertions(+), 11 deletions(-)\n\ndiff --git a/unicode_width.h b/unicode_width.h\nindex 47cdd23..73b5fd6 100644\n--- a/unicode_width.h\n+++ b/unicode_width.h\n@@ -25,7 +25,7 @@ static const struct interval zero_width[] = {\n { 0x0825, 0x0827 },\n { 0x0829, 0x082D },\n { 0x0859, 0x085B },\n-{ 0x08E4, 0x0902 },\n+{ 0x08D4, 0x0902 },\n { 0x093A, 0x093A },\n { 0x093C, 0x093C },\n { 0x0941, 0x0948 },\n@@ -120,6 +120,7 @@ static const struct interval zero_width[] = {\n { 0x17C9, 0x17D3 },\n { 0x17DD, 0x17DD },\n { 0x180B, 0x180E },\n+{ 0x1885, 0x1886 },\n { 0x18A9, 0x18A9 },\n { 0x1920, 0x1922 },\n { 0x1927, 0x1928 },\n@@ -158,7 +159,7 @@ static const struct interval zero_width[] = {\n { 0x1CF4, 0x1CF4 },\n { 0x1CF8, 0x1CF9 },\n { 0x1DC0, 0x1DF5 },\n-{ 0x1DFC, 0x1DFF },\n+{ 0x1DFB, 0x1DFF },\n { 0x200B, 0x200F },\n { 0x202A, 0x202E },\n { 0x2060, 0x2064 },\n@@ -171,13 +172,13 @@ static const struct interval zero_width[] = {\n { 0x3099, 0x309A },\n { 0xA66F, 0xA672 },\n { 0xA674, 0xA67D },\n-{ 0xA69F, 0xA69F },\n+{ 0xA69E, 0xA69F },\n { 0xA6F0, 0xA6F1 },\n { 0xA802, 0xA802 },\n { 0xA806, 0xA806 },\n { 0xA80B, 0xA80B },\n { 0xA825, 0xA826 },\n-{ 0xA8C4, 0xA8C4 },\n+{ 0xA8C4, 0xA8C5 },\n { 0xA8E0, 0xA8F1 },\n { 0xA926, 0xA92D },\n { 0xA947, 0xA951 },\n@@ -204,7 +205,7 @@ static const struct interval zero_width[] = {\n { 0xABED, 0xABED },\n { 0xFB1E, 0xFB1E },\n { 0xFE00, 0xFE0F },\n-{ 0xFE20, 0xFE2D },\n+{ 0xFE20, 0xFE2F },\n { 0xFEFF, 0xFEFF },\n { 0xFFF9, 0xFFFB },\n { 0x101FD, 0x101FD },\n@@ -228,16 +229,21 @@ static const struct interval zero_width[] = {\n { 0x11173, 0x11173 },\n { 0x11180, 0x11181 },\n { 0x111B6, 0x111BE },\n+{ 0x111CA, 0x111CC },\n { 0x1122F, 0x11231 },\n { 0x11234, 0x11234 },\n { 0x11236, 0x11237 },\n+{ 0x1123E, 0x1123E },\n { 0x112DF, 0x112DF },\n { 0x112E3, 0x112EA },\n-{ 0x11301, 0x11301 },\n+{ 0x11300, 0x11301 },\n { 0x1133C, 0x1133C },\n { 0x11340, 0x11340 },\n { 0x11366, 0x1136C },\n { 0x11370, 0x11374 },\n+{ 0x11438, 0x1143F },\n+{ 0x11442, 0x11444 },\n+{ 0x11446, 0x11446 },\n { 0x114B3, 0x114B8 },\n { 0x114BA, 0x114BA },\n { 0x114BF, 0x114C0 },\n@@ -245,6 +251,7 @@ static const struct interval zero_width[] = {\n { 0x115B2, 0x115B5 },\n { 0x115BC, 0x115BD },\n { 0x115BF, 0x115C0 },\n+{ 0x115DC, 0x115DD },\n { 0x11633, 0x1163A },\n { 0x1163D, 0x1163D },\n { 0x1163F, 0x11640 },\n@@ -252,6 +259,16 @@ static const struct interval zero_width[] = {\n { 0x116AD, 0x116AD },\n { 0x116B0, 0x116B5 },\n { 0x116B7, 0x116B7 },\n+{ 0x1171D, 0x1171F },\n+{ 0x11722, 0x11725 },\n+{ 0x11727, 0x1172B },\n+{ 0x11C30, 0x11C36 },\n+{ 0x11C38, 0x11C3D },\n+{ 0x11C3F, 0x11C3F },\n+{ 0x11C92, 0x11CA7 },\n+{ 0x11CAA, 0x11CB0 },\n+{ 0x11CB2, 0x11CB3 },\n+{ 0x11CB5, 0x11CB6 },\n { 0x16AF0, 0x16AF4 },\n { 0x16B30, 0x16B36 },\n { 0x16F8F, 0x16F92 },\n@@ -262,16 +279,28 @@ static const struct interval zero_width[] = {\n { 0x1D185, 0x1D18B },\n { 0x1D1AA, 0x1D1AD },\n { 0x1D242, 0x1D244 },\n+{ 0x1DA00, 0x1DA36 },\n+{ 0x1DA3B, 0x1DA6C },\n+{ 0x1DA75, 0x1DA75 },\n+{ 0x1DA84, 0x1DA84 },\n+{ 0x1DA9B, 0x1DA9F },\n+{ 0x1DAA1, 0x1DAAF },\n+{ 0x1E000, 0x1E006 },\n+{ 0x1E008, 0x1E018 },\n+{ 0x1E01B, 0x1E021 },\n+{ 0x1E023, 0x1E024 },\n+{ 0x1E026, 0x1E02A },\n { 0x1E8D0, 0x1E8D6 },\n+{ 0x1E944, 0x1E94A },\n { 0xE0001, 0xE0001 },\n { 0xE0020, 0xE007F },\n { 0xE0100, 0xE01EF }\n };\n static const struct interval double_width[] = {\n-{ /* plane */ 0x0, 0x1C },\n-{ /* plane */ 0x1C, 0x21 },\n-{ /* plane */ 0x21, 0x22 },\n-{ /* plane */ 0x22, 0x23 },\n+{ /* plane */ 0x0, 0x3D },\n+{ /* plane */ 0x3D, 0x68 },\n+{ /* plane */ 0x68, 0x69 },\n+{ /* plane */ 0x69, 0x6A },\n { /* plane */ 0x0, 0x0 },\n { /* plane */ 0x0, 0x0 },\n { /* plane */ 0x0, 0x0 },\n@@ -286,7 +315,40 @@ static const struct interval double_width[] = {\n { /* plane */ 0x0, 0x0 },\n { /* plane */ 0x0, 0x0 },\n { 0x1100, 0x115F },\n+{ 0x231A, 0x231B },\n { 0x2329, 0x232A },\n+{ 0x23E9, 0x23EC },\n+{ 0x23F0, 0x23F0 },\n+{ 0x23F3, 0x23F3 },\n+{ 0x25FD, 0x25FE },\n+{ 0x2614, 0x2615 },\n+{ 0x2648, 0x2653 },\n+{ 0x267F, 0x267F },\n+{ 0x2693, 0x2693 },\n+{ 0x26A1, 0x26A1 },\n+{ 0x26AA, 0x26AB },\n+{ 0x26BD, 0x26BE },\n+{ 0x26C4, 0x26C5 },\n+{ 0x26CE, 0x26CE },\n+{ 0x26D4, 0x26D4 },\n+{ 0x26EA, 0x26EA },\n+{ 0x26F2, 0x26F3 },\n+{ 0x26F5, 0x26F5 },\n+{ 0x26FA, 0x26FA },\n+{ 0x26FD, 0x26FD },\n+{ 0x2705, 0x2705 },\n+{ 0x270A, 0x270B },\n+{ 0x2728, 0x2728 },\n+{ 0x274C, 0x274C },\n+{ 0x274E, 0x274E },\n+{ 0x2753, 0x2755 },\n+{ 0x2757, 0x2757 },\n+{ 0x2795, 0x2797 },\n+{ 0x27B0, 0x27B0 },\n+{ 0x27BF, 0x27BF },\n+{ 0x2B1B, 0x2B1C },\n+{ 0x2B50, 0x2B50 },\n+{ 0x2B55, 0x2B55 },\n { 0x2E80, 0x2E99 },\n { 0x2E9B, 0x2EF3 },\n { 0x2F00, 0x2FD5 },\n@@ -313,11 +375,49 @@ static const struct interval double_width[] = {\n { 0xFE68, 0xFE6B },\n { 0xFF01, 0xFF60 },\n { 0xFFE0, 0xFFE6 },\n+{ 0x16FE0, 0x16FE0 },\n+{ 0x17000, 0x187EC },\n+{ 0x18800, 0x18AF2 },\n { 0x1B000, 0x1B001 },\n+{ 0x1F004, 0x1F004 },\n+{ 0x1F0CF, 0x1F0CF },\n+{ 0x1F18E, 0x1F18E },\n+{ 0x1F191, 0x1F19A },\n { 0x1F200, 0x1F202 },\n-{ 0x1F210, 0x1F23A },\n+{ 0x1F210, 0x1F23B },\n { 0x1F240, 0x1F248 },\n { 0x1F250, 0x1F251 },\n+{ 0x1F300, 0x1F320 },\n+{ 0x1F32D, 0x1F335 },\n+{ 0x1F337, 0x1F37C },\n+{ 0x1F37E, 0x1F393 },\n+{ 0x1F3A0, 0x1F3CA },\n+{ 0x1F3CF, 0x1F3D3 },\n+{ 0x1F3E0, 0x1F3F0 },\n+{ 0x1F3F4, 0x1F3F4 },\n+{ 0x1F3F8, 0x1F43E },\n+{ 0x1F440, 0x1F440 },\n+{ 0x1F442, 0x1F4FC },\n+{ 0x1F4FF, 0x1F53D },\n+{ 0x1F54B, 0x1F54E },\n+{ 0x1F550, 0x1F567 },\n+{ 0x1F57A, 0x1F57A },\n+{ 0x1F595, 0x1F596 },\n+{ 0x1F5A4, 0x1F5A4 },\n+{ 0x1F5FB, 0x1F64F },\n+{ 0x1F680, 0x1F6C5 },\n+{ 0x1F6CC, 0x1F6CC },\n+{ 0x1F6D0, 0x1F6D2 },\n+{ 0x1F6EB, 0x1F6EC },\n+{ 0x1F6F4, 0x1F6F6 },\n+{ 0x1F910, 0x1F91E },\n+{ 0x1F920, 0x1F927 },\n+{ 0x1F930, 0x1F930 },\n+{ 0x1F933, 0x1F93E },\n+{ 0x1F940, 0x1F94B },\n+{ 0x1F950, 0x1F95E },\n+{ 0x1F980, 0x1F991 },\n+{ 0x1F9C0, 0x1F9C0 },\n { 0x20000, 0x2FFFD },\n { 0x30000, 0x3FFFD }\n };\n-- \n2.7.2\n"},{"id":"306866","messageId":"1480761320-27967-1-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480713995-16157-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH 3/3] unicode_width.h: fix the double_width[] table","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T10:35:20Z","receivedAt":"2016-12-03T10:35:48Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"The function bisearch() in utf8.c does a pure binary search in\ndouble_width. It does not care about the 17 plane offsets which\nunicode/uniset/uniset prepends. Leaving the plane offsets in the table\nmay cause wrong results.\n\nFilter out the plane offsets in the update-unicode.sh and regenerate\nthe table.\n\nSigned-off-by: Beat Bolli \n---\n unicode_width.h | 17 -----------------\n update_unicode.sh | 2 +-\n 2 files changed, 1 insertion(+), 18 deletions(-)\n\ndiff --git a/unicode_width.h b/unicode_width.h\nindex 73b5fd6..02207be 100644\n--- a/unicode_width.h\n+++ b/unicode_width.h\n@@ -297,23 +297,6 @@ static const struct interval zero_width[] = {\n { 0xE0100, 0xE01EF }\n };\n static const struct interval double_width[] = {\n-{ /* plane */ 0x0, 0x3D },\n-{ /* plane */ 0x3D, 0x68 },\n-{ /* plane */ 0x68, 0x69 },\n-{ /* plane */ 0x69, 0x6A },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n { 0x1100, 0x115F },\n { 0x231A, 0x231B },\n { 0x2329, 0x232A },\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 3c84270..4c1ec8d 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -30,7 +30,7 @@ fi &&\n \t\t grep -v plane)\n \t};\n \tstatic const struct interval double_width[] = {\n-\t\t$(uniset/uniset --32 eaw:F,W)\n+\t\t$(uniset/uniset --32 eaw:F,W | grep -v plane)\n \t};\n \tEOF\n )\n-- \n2.7.2\n"},{"id":"306867","messageId":"1480762392-28731-1-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480713995-16157-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH v2 1/3] update-unicode.sh: automatically download newer definition files","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T10:53:10Z","receivedAt":"2016-12-03T10:53:32Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"Checking just for the unicode data files' existence is not sufficient;\nwe should also download them if a newer version exists on the Unicode\nconsortium's servers. Option -N of wget does this nicely for us.\n\nCc: Torsten Bögershausen \nSigned-off-by: Beat Bolli \n---\nDiff to v1:\n - reword the commit message\n - add Thorsten's Cc:\n\n update_unicode.sh | 8 ++------\n 1 file changed, 2 insertions(+), 6 deletions(-)\n\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 27af77c..3c84270 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -10,12 +10,8 @@ if ! test -d unicode; then\n \tmkdir unicode\n fi &&\n ( cd unicode &&\n-\tif ! test -f UnicodeData.txt; then\n-\t\twget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt\n-\tfi &&\n-\tif ! test -f EastAsianWidth.txt; then\n-\t\twget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt\n-\tfi &&\n+\twget -N http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt \\\n+\t\thttp://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt &&\n \tif ! test -d uniset; then\n \t\tgit clone https://github.com/depp/uniset.git\n \tfi &&\n-- \n2.7.2\n"},{"id":"306868","messageId":"1480762392-28731-3-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480762392-28731-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH v2 3/3] unicode_width.h: fix the double_width[] table","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T10:53:12Z","receivedAt":"2016-12-03T10:53:50Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"The function bisearch() in utf8.c does a pure binary search in\ndouble_width. It does not care about the 17 plane offsets which\nunicode/uniset/uniset prepends. Leaving the plane offsets in the table\nmay cause wrong results.\n\nFilter out the plane offsets in update-unicode.sh and regenerate the\ntable.\n\nCc: Torsten Bögershausen \nSigned-off-by: Beat Bolli \n---\nDiff to v1:\n - add Thorsten's Cc:\n\n unicode_width.h | 17 -----------------\n update_unicode.sh | 2 +-\n 2 files changed, 1 insertion(+), 18 deletions(-)\n\ndiff --git a/unicode_width.h b/unicode_width.h\nindex 73b5fd6..02207be 100644\n--- a/unicode_width.h\n+++ b/unicode_width.h\n@@ -297,23 +297,6 @@ static const struct interval zero_width[] = {\n { 0xE0100, 0xE01EF }\n };\n static const struct interval double_width[] = {\n-{ /* plane */ 0x0, 0x3D },\n-{ /* plane */ 0x3D, 0x68 },\n-{ /* plane */ 0x68, 0x69 },\n-{ /* plane */ 0x69, 0x6A },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n { 0x1100, 0x115F },\n { 0x231A, 0x231B },\n { 0x2329, 0x232A },\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 3c84270..4c1ec8d 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -30,7 +30,7 @@ fi &&\n \t\t grep -v plane)\n \t};\n \tstatic const struct interval double_width[] = {\n-\t\t$(uniset/uniset --32 eaw:F,W)\n+\t\t$(uniset/uniset --32 eaw:F,W | grep -v plane)\n \t};\n \tEOF\n )\n-- \n2.7.2\n"},{"id":"306869","messageId":"1480762392-28731-2-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480762392-28731-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH v2 2/3] unicode_width.h: update the tables to Unicode 9.0","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T10:53:11Z","receivedAt":"2016-12-03T10:53:53Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"Rerunning update-unicode.sh fixed in the previous commit produces these new\ntables.\n\nSigned-off-by: Beat Bolli \n---\nDiff to v1:\n - reword the commit message\n\n unicode_width.h | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 111 insertions(+), 11 deletions(-)\n\ndiff --git a/unicode_width.h b/unicode_width.h\nindex 47cdd23..73b5fd6 100644\n--- a/unicode_width.h\n+++ b/unicode_width.h\n@@ -25,7 +25,7 @@ static const struct interval zero_width[] = {\n { 0x0825, 0x0827 },\n { 0x0829, 0x082D },\n { 0x0859, 0x085B },\n-{ 0x08E4, 0x0902 },\n+{ 0x08D4, 0x0902 },\n { 0x093A, 0x093A },\n { 0x093C, 0x093C },\n { 0x0941, 0x0948 },\n@@ -120,6 +120,7 @@ static const struct interval zero_width[] = {\n { 0x17C9, 0x17D3 },\n { 0x17DD, 0x17DD },\n { 0x180B, 0x180E },\n+{ 0x1885, 0x1886 },\n { 0x18A9, 0x18A9 },\n { 0x1920, 0x1922 },\n { 0x1927, 0x1928 },\n@@ -158,7 +159,7 @@ static const struct interval zero_width[] = {\n { 0x1CF4, 0x1CF4 },\n { 0x1CF8, 0x1CF9 },\n { 0x1DC0, 0x1DF5 },\n-{ 0x1DFC, 0x1DFF },\n+{ 0x1DFB, 0x1DFF },\n { 0x200B, 0x200F },\n { 0x202A, 0x202E },\n { 0x2060, 0x2064 },\n@@ -171,13 +172,13 @@ static const struct interval zero_width[] = {\n { 0x3099, 0x309A },\n { 0xA66F, 0xA672 },\n { 0xA674, 0xA67D },\n-{ 0xA69F, 0xA69F },\n+{ 0xA69E, 0xA69F },\n { 0xA6F0, 0xA6F1 },\n { 0xA802, 0xA802 },\n { 0xA806, 0xA806 },\n { 0xA80B, 0xA80B },\n { 0xA825, 0xA826 },\n-{ 0xA8C4, 0xA8C4 },\n+{ 0xA8C4, 0xA8C5 },\n { 0xA8E0, 0xA8F1 },\n { 0xA926, 0xA92D },\n { 0xA947, 0xA951 },\n@@ -204,7 +205,7 @@ static const struct interval zero_width[] = {\n { 0xABED, 0xABED },\n { 0xFB1E, 0xFB1E },\n { 0xFE00, 0xFE0F },\n-{ 0xFE20, 0xFE2D },\n+{ 0xFE20, 0xFE2F },\n { 0xFEFF, 0xFEFF },\n { 0xFFF9, 0xFFFB },\n { 0x101FD, 0x101FD },\n@@ -228,16 +229,21 @@ static const struct interval zero_width[] = {\n { 0x11173, 0x11173 },\n { 0x11180, 0x11181 },\n { 0x111B6, 0x111BE },\n+{ 0x111CA, 0x111CC },\n { 0x1122F, 0x11231 },\n { 0x11234, 0x11234 },\n { 0x11236, 0x11237 },\n+{ 0x1123E, 0x1123E },\n { 0x112DF, 0x112DF },\n { 0x112E3, 0x112EA },\n-{ 0x11301, 0x11301 },\n+{ 0x11300, 0x11301 },\n { 0x1133C, 0x1133C },\n { 0x11340, 0x11340 },\n { 0x11366, 0x1136C },\n { 0x11370, 0x11374 },\n+{ 0x11438, 0x1143F },\n+{ 0x11442, 0x11444 },\n+{ 0x11446, 0x11446 },\n { 0x114B3, 0x114B8 },\n { 0x114BA, 0x114BA },\n { 0x114BF, 0x114C0 },\n@@ -245,6 +251,7 @@ static const struct interval zero_width[] = {\n { 0x115B2, 0x115B5 },\n { 0x115BC, 0x115BD },\n { 0x115BF, 0x115C0 },\n+{ 0x115DC, 0x115DD },\n { 0x11633, 0x1163A },\n { 0x1163D, 0x1163D },\n { 0x1163F, 0x11640 },\n@@ -252,6 +259,16 @@ static const struct interval zero_width[] = {\n { 0x116AD, 0x116AD },\n { 0x116B0, 0x116B5 },\n { 0x116B7, 0x116B7 },\n+{ 0x1171D, 0x1171F },\n+{ 0x11722, 0x11725 },\n+{ 0x11727, 0x1172B },\n+{ 0x11C30, 0x11C36 },\n+{ 0x11C38, 0x11C3D },\n+{ 0x11C3F, 0x11C3F },\n+{ 0x11C92, 0x11CA7 },\n+{ 0x11CAA, 0x11CB0 },\n+{ 0x11CB2, 0x11CB3 },\n+{ 0x11CB5, 0x11CB6 },\n { 0x16AF0, 0x16AF4 },\n { 0x16B30, 0x16B36 },\n { 0x16F8F, 0x16F92 },\n@@ -262,16 +279,28 @@ static const struct interval zero_width[] = {\n { 0x1D185, 0x1D18B },\n { 0x1D1AA, 0x1D1AD },\n { 0x1D242, 0x1D244 },\n+{ 0x1DA00, 0x1DA36 },\n+{ 0x1DA3B, 0x1DA6C },\n+{ 0x1DA75, 0x1DA75 },\n+{ 0x1DA84, 0x1DA84 },\n+{ 0x1DA9B, 0x1DA9F },\n+{ 0x1DAA1, 0x1DAAF },\n+{ 0x1E000, 0x1E006 },\n+{ 0x1E008, 0x1E018 },\n+{ 0x1E01B, 0x1E021 },\n+{ 0x1E023, 0x1E024 },\n+{ 0x1E026, 0x1E02A },\n { 0x1E8D0, 0x1E8D6 },\n+{ 0x1E944, 0x1E94A },\n { 0xE0001, 0xE0001 },\n { 0xE0020, 0xE007F },\n { 0xE0100, 0xE01EF }\n };\n static const struct interval double_width[] = {\n-{ /* plane */ 0x0, 0x1C },\n-{ /* plane */ 0x1C, 0x21 },\n-{ /* plane */ 0x21, 0x22 },\n-{ /* plane */ 0x22, 0x23 },\n+{ /* plane */ 0x0, 0x3D },\n+{ /* plane */ 0x3D, 0x68 },\n+{ /* plane */ 0x68, 0x69 },\n+{ /* plane */ 0x69, 0x6A },\n { /* plane */ 0x0, 0x0 },\n { /* plane */ 0x0, 0x0 },\n { /* plane */ 0x0, 0x0 },\n@@ -286,7 +315,40 @@ static const struct interval double_width[] = {\n { /* plane */ 0x0, 0x0 },\n { /* plane */ 0x0, 0x0 },\n { 0x1100, 0x115F },\n+{ 0x231A, 0x231B },\n { 0x2329, 0x232A },\n+{ 0x23E9, 0x23EC },\n+{ 0x23F0, 0x23F0 },\n+{ 0x23F3, 0x23F3 },\n+{ 0x25FD, 0x25FE },\n+{ 0x2614, 0x2615 },\n+{ 0x2648, 0x2653 },\n+{ 0x267F, 0x267F },\n+{ 0x2693, 0x2693 },\n+{ 0x26A1, 0x26A1 },\n+{ 0x26AA, 0x26AB },\n+{ 0x26BD, 0x26BE },\n+{ 0x26C4, 0x26C5 },\n+{ 0x26CE, 0x26CE },\n+{ 0x26D4, 0x26D4 },\n+{ 0x26EA, 0x26EA },\n+{ 0x26F2, 0x26F3 },\n+{ 0x26F5, 0x26F5 },\n+{ 0x26FA, 0x26FA },\n+{ 0x26FD, 0x26FD },\n+{ 0x2705, 0x2705 },\n+{ 0x270A, 0x270B },\n+{ 0x2728, 0x2728 },\n+{ 0x274C, 0x274C },\n+{ 0x274E, 0x274E },\n+{ 0x2753, 0x2755 },\n+{ 0x2757, 0x2757 },\n+{ 0x2795, 0x2797 },\n+{ 0x27B0, 0x27B0 },\n+{ 0x27BF, 0x27BF },\n+{ 0x2B1B, 0x2B1C },\n+{ 0x2B50, 0x2B50 },\n+{ 0x2B55, 0x2B55 },\n { 0x2E80, 0x2E99 },\n { 0x2E9B, 0x2EF3 },\n { 0x2F00, 0x2FD5 },\n@@ -313,11 +375,49 @@ static const struct interval double_width[] = {\n { 0xFE68, 0xFE6B },\n { 0xFF01, 0xFF60 },\n { 0xFFE0, 0xFFE6 },\n+{ 0x16FE0, 0x16FE0 },\n+{ 0x17000, 0x187EC },\n+{ 0x18800, 0x18AF2 },\n { 0x1B000, 0x1B001 },\n+{ 0x1F004, 0x1F004 },\n+{ 0x1F0CF, 0x1F0CF },\n+{ 0x1F18E, 0x1F18E },\n+{ 0x1F191, 0x1F19A },\n { 0x1F200, 0x1F202 },\n-{ 0x1F210, 0x1F23A },\n+{ 0x1F210, 0x1F23B },\n { 0x1F240, 0x1F248 },\n { 0x1F250, 0x1F251 },\n+{ 0x1F300, 0x1F320 },\n+{ 0x1F32D, 0x1F335 },\n+{ 0x1F337, 0x1F37C },\n+{ 0x1F37E, 0x1F393 },\n+{ 0x1F3A0, 0x1F3CA },\n+{ 0x1F3CF, 0x1F3D3 },\n+{ 0x1F3E0, 0x1F3F0 },\n+{ 0x1F3F4, 0x1F3F4 },\n+{ 0x1F3F8, 0x1F43E },\n+{ 0x1F440, 0x1F440 },\n+{ 0x1F442, 0x1F4FC },\n+{ 0x1F4FF, 0x1F53D },\n+{ 0x1F54B, 0x1F54E },\n+{ 0x1F550, 0x1F567 },\n+{ 0x1F57A, 0x1F57A },\n+{ 0x1F595, 0x1F596 },\n+{ 0x1F5A4, 0x1F5A4 },\n+{ 0x1F5FB, 0x1F64F },\n+{ 0x1F680, 0x1F6C5 },\n+{ 0x1F6CC, 0x1F6CC },\n+{ 0x1F6D0, 0x1F6D2 },\n+{ 0x1F6EB, 0x1F6EC },\n+{ 0x1F6F4, 0x1F6F6 },\n+{ 0x1F910, 0x1F91E },\n+{ 0x1F920, 0x1F927 },\n+{ 0x1F930, 0x1F930 },\n+{ 0x1F933, 0x1F93E },\n+{ 0x1F940, 0x1F94B },\n+{ 0x1F950, 0x1F95E },\n+{ 0x1F980, 0x1F991 },\n+{ 0x1F9C0, 0x1F9C0 },\n { 0x20000, 0x2FFFD },\n { 0x30000, 0x3FFFD }\n };\n-- \n2.7.2\n"},{"id":"306870","messageId":"1480771173-731-2-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480771173-731-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH v3 2/3] update-unicode.sh: strip the plane offsets from the double_width[] table","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T13:19:32Z","receivedAt":"2016-12-03T13:21:23Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"The function bisearch() in utf8.c does a pure binary search in\ndouble_width. It does not care about the 17 plane offsets which\nunicode/uniset/uniset prepends. Leaving the plane offsets in the table\nmay cause wrong results.\n\nFilter out the plane offsets in update-unicode.sh.\n\nCc: Torsten Bögershausen \nSigned-off-by: Beat Bolli \n---\n update_unicode.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 3c84270..4c1ec8d 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -30,7 +30,7 @@ fi &&\n \t\t grep -v plane)\n \t};\n \tstatic const struct interval double_width[] = {\n-\t\t$(uniset/uniset --32 eaw:F,W)\n+\t\t$(uniset/uniset --32 eaw:F,W | grep -v plane)\n \t};\n \tEOF\n )\n-- \n2.7.2\n"},{"id":"306871","messageId":"1480771173-731-1-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480762392-28731-3-git-send-email-dev+git@drbeat.li","subject":"[PATCH v3 1/3] update-unicode.sh: automatically download newer definition files","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T13:19:31Z","receivedAt":"2016-12-03T13:21:29Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"Checking just for the unicode data files' existence is not sufficient;\nwe should also download them if a newer version exists on the Unicode\nconsortium's servers. Option -N of wget does this nicely for us.\n\nCc: Torsten Bögershausen \nSigned-off-by: Beat Bolli \n---\nDiff to v2:\n - reorder the commits: fix all of update-unicode.sh first, then\n regenerate unicode_width.h only once\n\n update_unicode.sh | 8 ++------\n 1 file changed, 2 insertions(+), 6 deletions(-)\n\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 27af77c..3c84270 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -10,12 +10,8 @@ if ! test -d unicode; then\n \tmkdir unicode\n fi &&\n ( cd unicode &&\n-\tif ! test -f UnicodeData.txt; then\n-\t\twget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt\n-\tfi &&\n-\tif ! test -f EastAsianWidth.txt; then\n-\t\twget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt\n-\tfi &&\n+\twget -N http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt \\\n+\t\thttp://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt &&\n \tif ! test -d uniset; then\n \t\tgit clone https://github.com/depp/uniset.git\n \tfi &&\n-- \n2.7.2\n"},{"id":"306872","messageId":"1480771173-731-3-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480771173-731-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH v3 3/3] unicode_width.h: update the tables to Unicode 9.0","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T13:19:33Z","receivedAt":"2016-12-03T13:21:36Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"Rerunning update-unicode.sh that we fixed in the two previous commits\nproduces these new tables.\n\nSigned-off-by: Beat Bolli \n---\n unicode_width.h | 131 +++++++++++++++++++++++++++++++++++++++++++++-----------\n 1 file changed, 107 insertions(+), 24 deletions(-)\n\ndiff --git a/unicode_width.h b/unicode_width.h\nindex 47cdd23..02207be 100644\n--- a/unicode_width.h\n+++ b/unicode_width.h\n@@ -25,7 +25,7 @@ static const struct interval zero_width[] = {\n { 0x0825, 0x0827 },\n { 0x0829, 0x082D },\n { 0x0859, 0x085B },\n-{ 0x08E4, 0x0902 },\n+{ 0x08D4, 0x0902 },\n { 0x093A, 0x093A },\n { 0x093C, 0x093C },\n { 0x0941, 0x0948 },\n@@ -120,6 +120,7 @@ static const struct interval zero_width[] = {\n { 0x17C9, 0x17D3 },\n { 0x17DD, 0x17DD },\n { 0x180B, 0x180E },\n+{ 0x1885, 0x1886 },\n { 0x18A9, 0x18A9 },\n { 0x1920, 0x1922 },\n { 0x1927, 0x1928 },\n@@ -158,7 +159,7 @@ static const struct interval zero_width[] = {\n { 0x1CF4, 0x1CF4 },\n { 0x1CF8, 0x1CF9 },\n { 0x1DC0, 0x1DF5 },\n-{ 0x1DFC, 0x1DFF },\n+{ 0x1DFB, 0x1DFF },\n { 0x200B, 0x200F },\n { 0x202A, 0x202E },\n { 0x2060, 0x2064 },\n@@ -171,13 +172,13 @@ static const struct interval zero_width[] = {\n { 0x3099, 0x309A },\n { 0xA66F, 0xA672 },\n { 0xA674, 0xA67D },\n-{ 0xA69F, 0xA69F },\n+{ 0xA69E, 0xA69F },\n { 0xA6F0, 0xA6F1 },\n { 0xA802, 0xA802 },\n { 0xA806, 0xA806 },\n { 0xA80B, 0xA80B },\n { 0xA825, 0xA826 },\n-{ 0xA8C4, 0xA8C4 },\n+{ 0xA8C4, 0xA8C5 },\n { 0xA8E0, 0xA8F1 },\n { 0xA926, 0xA92D },\n { 0xA947, 0xA951 },\n@@ -204,7 +205,7 @@ static const struct interval zero_width[] = {\n { 0xABED, 0xABED },\n { 0xFB1E, 0xFB1E },\n { 0xFE00, 0xFE0F },\n-{ 0xFE20, 0xFE2D },\n+{ 0xFE20, 0xFE2F },\n { 0xFEFF, 0xFEFF },\n { 0xFFF9, 0xFFFB },\n { 0x101FD, 0x101FD },\n@@ -228,16 +229,21 @@ static const struct interval zero_width[] = {\n { 0x11173, 0x11173 },\n { 0x11180, 0x11181 },\n { 0x111B6, 0x111BE },\n+{ 0x111CA, 0x111CC },\n { 0x1122F, 0x11231 },\n { 0x11234, 0x11234 },\n { 0x11236, 0x11237 },\n+{ 0x1123E, 0x1123E },\n { 0x112DF, 0x112DF },\n { 0x112E3, 0x112EA },\n-{ 0x11301, 0x11301 },\n+{ 0x11300, 0x11301 },\n { 0x1133C, 0x1133C },\n { 0x11340, 0x11340 },\n { 0x11366, 0x1136C },\n { 0x11370, 0x11374 },\n+{ 0x11438, 0x1143F },\n+{ 0x11442, 0x11444 },\n+{ 0x11446, 0x11446 },\n { 0x114B3, 0x114B8 },\n { 0x114BA, 0x114BA },\n { 0x114BF, 0x114C0 },\n@@ -245,6 +251,7 @@ static const struct interval zero_width[] = {\n { 0x115B2, 0x115B5 },\n { 0x115BC, 0x115BD },\n { 0x115BF, 0x115C0 },\n+{ 0x115DC, 0x115DD },\n { 0x11633, 0x1163A },\n { 0x1163D, 0x1163D },\n { 0x1163F, 0x11640 },\n@@ -252,6 +259,16 @@ static const struct interval zero_width[] = {\n { 0x116AD, 0x116AD },\n { 0x116B0, 0x116B5 },\n { 0x116B7, 0x116B7 },\n+{ 0x1171D, 0x1171F },\n+{ 0x11722, 0x11725 },\n+{ 0x11727, 0x1172B },\n+{ 0x11C30, 0x11C36 },\n+{ 0x11C38, 0x11C3D },\n+{ 0x11C3F, 0x11C3F },\n+{ 0x11C92, 0x11CA7 },\n+{ 0x11CAA, 0x11CB0 },\n+{ 0x11CB2, 0x11CB3 },\n+{ 0x11CB5, 0x11CB6 },\n { 0x16AF0, 0x16AF4 },\n { 0x16B30, 0x16B36 },\n { 0x16F8F, 0x16F92 },\n@@ -262,31 +279,59 @@ static const struct interval zero_width[] = {\n { 0x1D185, 0x1D18B },\n { 0x1D1AA, 0x1D1AD },\n { 0x1D242, 0x1D244 },\n+{ 0x1DA00, 0x1DA36 },\n+{ 0x1DA3B, 0x1DA6C },\n+{ 0x1DA75, 0x1DA75 },\n+{ 0x1DA84, 0x1DA84 },\n+{ 0x1DA9B, 0x1DA9F },\n+{ 0x1DAA1, 0x1DAAF },\n+{ 0x1E000, 0x1E006 },\n+{ 0x1E008, 0x1E018 },\n+{ 0x1E01B, 0x1E021 },\n+{ 0x1E023, 0x1E024 },\n+{ 0x1E026, 0x1E02A },\n { 0x1E8D0, 0x1E8D6 },\n+{ 0x1E944, 0x1E94A },\n { 0xE0001, 0xE0001 },\n { 0xE0020, 0xE007F },\n { 0xE0100, 0xE01EF }\n };\n static const struct interval double_width[] = {\n-{ /* plane */ 0x0, 0x1C },\n-{ /* plane */ 0x1C, 0x21 },\n-{ /* plane */ 0x21, 0x22 },\n-{ /* plane */ 0x22, 0x23 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n { 0x1100, 0x115F },\n+{ 0x231A, 0x231B },\n { 0x2329, 0x232A },\n+{ 0x23E9, 0x23EC },\n+{ 0x23F0, 0x23F0 },\n+{ 0x23F3, 0x23F3 },\n+{ 0x25FD, 0x25FE },\n+{ 0x2614, 0x2615 },\n+{ 0x2648, 0x2653 },\n+{ 0x267F, 0x267F },\n+{ 0x2693, 0x2693 },\n+{ 0x26A1, 0x26A1 },\n+{ 0x26AA, 0x26AB },\n+{ 0x26BD, 0x26BE },\n+{ 0x26C4, 0x26C5 },\n+{ 0x26CE, 0x26CE },\n+{ 0x26D4, 0x26D4 },\n+{ 0x26EA, 0x26EA },\n+{ 0x26F2, 0x26F3 },\n+{ 0x26F5, 0x26F5 },\n+{ 0x26FA, 0x26FA },\n+{ 0x26FD, 0x26FD },\n+{ 0x2705, 0x2705 },\n+{ 0x270A, 0x270B },\n+{ 0x2728, 0x2728 },\n+{ 0x274C, 0x274C },\n+{ 0x274E, 0x274E },\n+{ 0x2753, 0x2755 },\n+{ 0x2757, 0x2757 },\n+{ 0x2795, 0x2797 },\n+{ 0x27B0, 0x27B0 },\n+{ 0x27BF, 0x27BF },\n+{ 0x2B1B, 0x2B1C },\n+{ 0x2B50, 0x2B50 },\n+{ 0x2B55, 0x2B55 },\n { 0x2E80, 0x2E99 },\n { 0x2E9B, 0x2EF3 },\n { 0x2F00, 0x2FD5 },\n@@ -313,11 +358,49 @@ static const struct interval double_width[] = {\n { 0xFE68, 0xFE6B },\n { 0xFF01, 0xFF60 },\n { 0xFFE0, 0xFFE6 },\n+{ 0x16FE0, 0x16FE0 },\n+{ 0x17000, 0x187EC },\n+{ 0x18800, 0x18AF2 },\n { 0x1B000, 0x1B001 },\n+{ 0x1F004, 0x1F004 },\n+{ 0x1F0CF, 0x1F0CF },\n+{ 0x1F18E, 0x1F18E },\n+{ 0x1F191, 0x1F19A },\n { 0x1F200, 0x1F202 },\n-{ 0x1F210, 0x1F23A },\n+{ 0x1F210, 0x1F23B },\n { 0x1F240, 0x1F248 },\n { 0x1F250, 0x1F251 },\n+{ 0x1F300, 0x1F320 },\n+{ 0x1F32D, 0x1F335 },\n+{ 0x1F337, 0x1F37C },\n+{ 0x1F37E, 0x1F393 },\n+{ 0x1F3A0, 0x1F3CA },\n+{ 0x1F3CF, 0x1F3D3 },\n+{ 0x1F3E0, 0x1F3F0 },\n+{ 0x1F3F4, 0x1F3F4 },\n+{ 0x1F3F8, 0x1F43E },\n+{ 0x1F440, 0x1F440 },\n+{ 0x1F442, 0x1F4FC },\n+{ 0x1F4FF, 0x1F53D },\n+{ 0x1F54B, 0x1F54E },\n+{ 0x1F550, 0x1F567 },\n+{ 0x1F57A, 0x1F57A },\n+{ 0x1F595, 0x1F596 },\n+{ 0x1F5A4, 0x1F5A4 },\n+{ 0x1F5FB, 0x1F64F },\n+{ 0x1F680, 0x1F6C5 },\n+{ 0x1F6CC, 0x1F6CC },\n+{ 0x1F6D0, 0x1F6D2 },\n+{ 0x1F6EB, 0x1F6EC },\n+{ 0x1F6F4, 0x1F6F6 },\n+{ 0x1F910, 0x1F91E },\n+{ 0x1F920, 0x1F927 },\n+{ 0x1F930, 0x1F930 },\n+{ 0x1F933, 0x1F93E },\n+{ 0x1F940, 0x1F94B },\n+{ 0x1F950, 0x1F95E },\n+{ 0x1F980, 0x1F991 },\n+{ 0x1F9C0, 0x1F9C0 },\n { 0x20000, 0x2FFFD },\n { 0x30000, 0x3FFFD }\n };\n-- \n2.7.2\n"},{"id":"306874","messageId":"20161203164049.GA31244@tb-raspi","threadId":"44599","inReplyTo":"1480771173-731-1-git-send-email-dev+git@drbeat.li","subject":"Re: [PATCH v3 1/3] update-unicode.sh: automatically download newer definition files","fromName":"Torsten B��gershausen","fromEmail":"tboegi@web.de","sentAt":"2016-12-03T16:40:49Z","receivedAt":"2016-12-03T16:41:02Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Sat, Dec 03, 2016 at 02:19:31PM +0100, Beat Bolli wrote:\n> Checking just for the unicode data files' existence is not sufficient;\n> we should also download them if a newer version exists on the Unicode\n> consortium's servers. Option -N of wget does this nicely for us.\n> \n> Cc: Torsten B??gershausen \n\nThe V3 series makes perfect sense, thanks for cleaning up my mess.\n(And can we remove the Cc: line, or replace with it Reviewed-by ?)\n"},{"id":"306875","messageId":"835c0328-e812-1cb7-c49e-714ff0e9ffb3@drbeat.li","threadId":"44599","inReplyTo":"20161203164049.GA31244@tb-raspi","subject":"Re: [PATCH v3 1/3] update-unicode.sh: automatically download newer definition files","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T16:41:42Z","receivedAt":"2016-12-03T16:53:43Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"On 03.12.16 17:40, Torsten =?unknown-8bit?Q?B=C3=B6gershausen?= wrote:\n> On Sat, Dec 03, 2016 at 02:19:31PM +0100, Beat Bolli wrote:\n>> Checking just for the unicode data files' existence is not sufficient;\n>> we should also download them if a newer version exists on the Unicode\n>> consortium's servers. Option -N of wget does this nicely for us.\n>>\n>> Cc: Torsten B??gershausen \n> \n> The V3 series makes perfect sense, thanks for cleaning up my mess.\nYeah, it took me three tries, too :-)\n\n> (And can we remove the Cc: line, or replace with it Reviewed-by ?)\nIf you prefer, sure.\n\nDo you have any other comments?\n\nBeat\n"},{"id":"306878","messageId":"1480798849-13907-2-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480798849-13907-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH v4 2/3] update-unicode.sh: strip the plane offsets from the double_width[] table","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T21:00:48Z","receivedAt":"2016-12-03T21:01:17Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"The function bisearch() in utf8.c does a pure binary search in\ndouble_width. It does not care about the 17 plane offsets which\nunicode/uniset/uniset prepends. Leaving the plane offsets in the table\nmay cause wrong results.\n\nFilter out the plane offsets in update-unicode.sh.\n\nReviewed-by: Torsten Bögershausen \nSigned-off-by: Beat Bolli \n---\n update_unicode.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 3c84270..4c1ec8d 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -30,7 +30,7 @@ fi &&\n \t\t grep -v plane)\n \t};\n \tstatic const struct interval double_width[] = {\n-\t\t$(uniset/uniset --32 eaw:F,W)\n+\t\t$(uniset/uniset --32 eaw:F,W | grep -v plane)\n \t};\n \tEOF\n )\n-- \n2.7.2\n"},{"id":"306879","messageId":"1480798849-13907-3-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"1480798849-13907-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH v4 3/3] unicode_width.h: update the tables to Unicode 9.0","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T21:00:49Z","receivedAt":"2016-12-03T21:01:20Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"Rerunning update-unicode.sh that we fixed in the two previous commits\nproduces these new tables.\n\nSigned-off-by: Beat Bolli \n---\n unicode_width.h | 131 +++++++++++++++++++++++++++++++++++++++++++++-----------\n 1 file changed, 107 insertions(+), 24 deletions(-)\n\ndiff --git a/unicode_width.h b/unicode_width.h\nindex 47cdd23..02207be 100644\n--- a/unicode_width.h\n+++ b/unicode_width.h\n@@ -25,7 +25,7 @@ static const struct interval zero_width[] = {\n { 0x0825, 0x0827 },\n { 0x0829, 0x082D },\n { 0x0859, 0x085B },\n-{ 0x08E4, 0x0902 },\n+{ 0x08D4, 0x0902 },\n { 0x093A, 0x093A },\n { 0x093C, 0x093C },\n { 0x0941, 0x0948 },\n@@ -120,6 +120,7 @@ static const struct interval zero_width[] = {\n { 0x17C9, 0x17D3 },\n { 0x17DD, 0x17DD },\n { 0x180B, 0x180E },\n+{ 0x1885, 0x1886 },\n { 0x18A9, 0x18A9 },\n { 0x1920, 0x1922 },\n { 0x1927, 0x1928 },\n@@ -158,7 +159,7 @@ static const struct interval zero_width[] = {\n { 0x1CF4, 0x1CF4 },\n { 0x1CF8, 0x1CF9 },\n { 0x1DC0, 0x1DF5 },\n-{ 0x1DFC, 0x1DFF },\n+{ 0x1DFB, 0x1DFF },\n { 0x200B, 0x200F },\n { 0x202A, 0x202E },\n { 0x2060, 0x2064 },\n@@ -171,13 +172,13 @@ static const struct interval zero_width[] = {\n { 0x3099, 0x309A },\n { 0xA66F, 0xA672 },\n { 0xA674, 0xA67D },\n-{ 0xA69F, 0xA69F },\n+{ 0xA69E, 0xA69F },\n { 0xA6F0, 0xA6F1 },\n { 0xA802, 0xA802 },\n { 0xA806, 0xA806 },\n { 0xA80B, 0xA80B },\n { 0xA825, 0xA826 },\n-{ 0xA8C4, 0xA8C4 },\n+{ 0xA8C4, 0xA8C5 },\n { 0xA8E0, 0xA8F1 },\n { 0xA926, 0xA92D },\n { 0xA947, 0xA951 },\n@@ -204,7 +205,7 @@ static const struct interval zero_width[] = {\n { 0xABED, 0xABED },\n { 0xFB1E, 0xFB1E },\n { 0xFE00, 0xFE0F },\n-{ 0xFE20, 0xFE2D },\n+{ 0xFE20, 0xFE2F },\n { 0xFEFF, 0xFEFF },\n { 0xFFF9, 0xFFFB },\n { 0x101FD, 0x101FD },\n@@ -228,16 +229,21 @@ static const struct interval zero_width[] = {\n { 0x11173, 0x11173 },\n { 0x11180, 0x11181 },\n { 0x111B6, 0x111BE },\n+{ 0x111CA, 0x111CC },\n { 0x1122F, 0x11231 },\n { 0x11234, 0x11234 },\n { 0x11236, 0x11237 },\n+{ 0x1123E, 0x1123E },\n { 0x112DF, 0x112DF },\n { 0x112E3, 0x112EA },\n-{ 0x11301, 0x11301 },\n+{ 0x11300, 0x11301 },\n { 0x1133C, 0x1133C },\n { 0x11340, 0x11340 },\n { 0x11366, 0x1136C },\n { 0x11370, 0x11374 },\n+{ 0x11438, 0x1143F },\n+{ 0x11442, 0x11444 },\n+{ 0x11446, 0x11446 },\n { 0x114B3, 0x114B8 },\n { 0x114BA, 0x114BA },\n { 0x114BF, 0x114C0 },\n@@ -245,6 +251,7 @@ static const struct interval zero_width[] = {\n { 0x115B2, 0x115B5 },\n { 0x115BC, 0x115BD },\n { 0x115BF, 0x115C0 },\n+{ 0x115DC, 0x115DD },\n { 0x11633, 0x1163A },\n { 0x1163D, 0x1163D },\n { 0x1163F, 0x11640 },\n@@ -252,6 +259,16 @@ static const struct interval zero_width[] = {\n { 0x116AD, 0x116AD },\n { 0x116B0, 0x116B5 },\n { 0x116B7, 0x116B7 },\n+{ 0x1171D, 0x1171F },\n+{ 0x11722, 0x11725 },\n+{ 0x11727, 0x1172B },\n+{ 0x11C30, 0x11C36 },\n+{ 0x11C38, 0x11C3D },\n+{ 0x11C3F, 0x11C3F },\n+{ 0x11C92, 0x11CA7 },\n+{ 0x11CAA, 0x11CB0 },\n+{ 0x11CB2, 0x11CB3 },\n+{ 0x11CB5, 0x11CB6 },\n { 0x16AF0, 0x16AF4 },\n { 0x16B30, 0x16B36 },\n { 0x16F8F, 0x16F92 },\n@@ -262,31 +279,59 @@ static const struct interval zero_width[] = {\n { 0x1D185, 0x1D18B },\n { 0x1D1AA, 0x1D1AD },\n { 0x1D242, 0x1D244 },\n+{ 0x1DA00, 0x1DA36 },\n+{ 0x1DA3B, 0x1DA6C },\n+{ 0x1DA75, 0x1DA75 },\n+{ 0x1DA84, 0x1DA84 },\n+{ 0x1DA9B, 0x1DA9F },\n+{ 0x1DAA1, 0x1DAAF },\n+{ 0x1E000, 0x1E006 },\n+{ 0x1E008, 0x1E018 },\n+{ 0x1E01B, 0x1E021 },\n+{ 0x1E023, 0x1E024 },\n+{ 0x1E026, 0x1E02A },\n { 0x1E8D0, 0x1E8D6 },\n+{ 0x1E944, 0x1E94A },\n { 0xE0001, 0xE0001 },\n { 0xE0020, 0xE007F },\n { 0xE0100, 0xE01EF }\n };\n static const struct interval double_width[] = {\n-{ /* plane */ 0x0, 0x1C },\n-{ /* plane */ 0x1C, 0x21 },\n-{ /* plane */ 0x21, 0x22 },\n-{ /* plane */ 0x22, 0x23 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n-{ /* plane */ 0x0, 0x0 },\n { 0x1100, 0x115F },\n+{ 0x231A, 0x231B },\n { 0x2329, 0x232A },\n+{ 0x23E9, 0x23EC },\n+{ 0x23F0, 0x23F0 },\n+{ 0x23F3, 0x23F3 },\n+{ 0x25FD, 0x25FE },\n+{ 0x2614, 0x2615 },\n+{ 0x2648, 0x2653 },\n+{ 0x267F, 0x267F },\n+{ 0x2693, 0x2693 },\n+{ 0x26A1, 0x26A1 },\n+{ 0x26AA, 0x26AB },\n+{ 0x26BD, 0x26BE },\n+{ 0x26C4, 0x26C5 },\n+{ 0x26CE, 0x26CE },\n+{ 0x26D4, 0x26D4 },\n+{ 0x26EA, 0x26EA },\n+{ 0x26F2, 0x26F3 },\n+{ 0x26F5, 0x26F5 },\n+{ 0x26FA, 0x26FA },\n+{ 0x26FD, 0x26FD },\n+{ 0x2705, 0x2705 },\n+{ 0x270A, 0x270B },\n+{ 0x2728, 0x2728 },\n+{ 0x274C, 0x274C },\n+{ 0x274E, 0x274E },\n+{ 0x2753, 0x2755 },\n+{ 0x2757, 0x2757 },\n+{ 0x2795, 0x2797 },\n+{ 0x27B0, 0x27B0 },\n+{ 0x27BF, 0x27BF },\n+{ 0x2B1B, 0x2B1C },\n+{ 0x2B50, 0x2B50 },\n+{ 0x2B55, 0x2B55 },\n { 0x2E80, 0x2E99 },\n { 0x2E9B, 0x2EF3 },\n { 0x2F00, 0x2FD5 },\n@@ -313,11 +358,49 @@ static const struct interval double_width[] = {\n { 0xFE68, 0xFE6B },\n { 0xFF01, 0xFF60 },\n { 0xFFE0, 0xFFE6 },\n+{ 0x16FE0, 0x16FE0 },\n+{ 0x17000, 0x187EC },\n+{ 0x18800, 0x18AF2 },\n { 0x1B000, 0x1B001 },\n+{ 0x1F004, 0x1F004 },\n+{ 0x1F0CF, 0x1F0CF },\n+{ 0x1F18E, 0x1F18E },\n+{ 0x1F191, 0x1F19A },\n { 0x1F200, 0x1F202 },\n-{ 0x1F210, 0x1F23A },\n+{ 0x1F210, 0x1F23B },\n { 0x1F240, 0x1F248 },\n { 0x1F250, 0x1F251 },\n+{ 0x1F300, 0x1F320 },\n+{ 0x1F32D, 0x1F335 },\n+{ 0x1F337, 0x1F37C },\n+{ 0x1F37E, 0x1F393 },\n+{ 0x1F3A0, 0x1F3CA },\n+{ 0x1F3CF, 0x1F3D3 },\n+{ 0x1F3E0, 0x1F3F0 },\n+{ 0x1F3F4, 0x1F3F4 },\n+{ 0x1F3F8, 0x1F43E },\n+{ 0x1F440, 0x1F440 },\n+{ 0x1F442, 0x1F4FC },\n+{ 0x1F4FF, 0x1F53D },\n+{ 0x1F54B, 0x1F54E },\n+{ 0x1F550, 0x1F567 },\n+{ 0x1F57A, 0x1F57A },\n+{ 0x1F595, 0x1F596 },\n+{ 0x1F5A4, 0x1F5A4 },\n+{ 0x1F5FB, 0x1F64F },\n+{ 0x1F680, 0x1F6C5 },\n+{ 0x1F6CC, 0x1F6CC },\n+{ 0x1F6D0, 0x1F6D2 },\n+{ 0x1F6EB, 0x1F6EC },\n+{ 0x1F6F4, 0x1F6F6 },\n+{ 0x1F910, 0x1F91E },\n+{ 0x1F920, 0x1F927 },\n+{ 0x1F930, 0x1F930 },\n+{ 0x1F933, 0x1F93E },\n+{ 0x1F940, 0x1F94B },\n+{ 0x1F950, 0x1F95E },\n+{ 0x1F980, 0x1F991 },\n+{ 0x1F9C0, 0x1F9C0 },\n { 0x20000, 0x2FFFD },\n { 0x30000, 0x3FFFD }\n };\n-- \n2.7.2\n"},{"id":"306880","messageId":"1480798849-13907-1-git-send-email-dev+git@drbeat.li","threadId":"44599","inReplyTo":"835c0328-e812-1cb7-c49e-714ff0e9ffb3@drbeat.li","subject":"[PATCH v4 1/3] update-unicode.sh: automatically download newer definition files","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-03T21:00:47Z","receivedAt":"2016-12-03T21:01:23Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"Checking just for the unicode data files' existence is not sufficient;\nwe should also download them if a newer version exists on the Unicode\nconsortium's servers. Option -N of wget does this nicely for us.\n\nReviewed-by: Torsten Boegershausen \nSigned-off-by: Beat Bolli \n---\nDiff to v3:\n - change the Cc: into Reviewed-by: on Thorsten's request\n - include the old reroll diffs\n\nDiff to v2:\n - reorder the commits: fix all of update-unicode.sh first, then\n regenerate unicode_width.h only once\n\nDiff to v1:\n - reword the commit message\n - add Thorsten's Cc:\n\n update_unicode.sh | 8 ++------\n 1 file changed, 2 insertions(+), 6 deletions(-)\n\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 27af77c..3c84270 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -10,12 +10,8 @@ if ! test -d unicode; then\n \tmkdir unicode\n fi &&\n ( cd unicode &&\n-\tif ! test -f UnicodeData.txt; then\n-\t\twget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt\n-\tfi &&\n-\tif ! test -f EastAsianWidth.txt; then\n-\t\twget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt\n-\tfi &&\n+\twget -N http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt \\\n+\t\thttp://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt &&\n \tif ! test -d uniset; then\n \t\tgit clone https://github.com/depp/uniset.git\n \tfi &&\n-- \n2.7.2\n"},{"id":"306886","messageId":"20161204075800.GA2415@tb-raspi","threadId":"44599","inReplyTo":"1480798849-13907-1-git-send-email-dev+git@drbeat.li","subject":"Re: [PATCH v4 1/3] update-unicode.sh: automatically download newer definition files","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-12-04T07:58:00Z","receivedAt":"2016-12-04T07:58:12Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Sat, Dec 03, 2016 at 10:00:47PM +0100, Beat Bolli wrote:\n> Checking just for the unicode data files' existence is not sufficient;\n> we should also download them if a newer version exists on the Unicode\n> consortium's servers. Option -N of wget does this nicely for us.\n> \n> Reviewed-by: Torsten Boegershausen \n\nMinor remark (Not sure if this motivates v5, may be Junio can fix it locally?)\ns/oe/ö/\n\nBeside this: Thanks again (and I learned about the -N option of wget)\n"},{"id":"306957","messageId":"xmqqmvga3yod.fsf@gitster.mtv.corp.google.com","threadId":"44599","inReplyTo":"20161204075800.GA2415@tb-raspi","subject":"Re: [PATCH v4 1/3] update-unicode.sh: automatically download newer definition files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-12-05T20:31:46Z","receivedAt":"2016-12-05T20:31:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n> On Sat, Dec 03, 2016 at 10:00:47PM +0100, Beat Bolli wrote:\n>> Checking just for the unicode data files' existence is not sufficient;\n>> we should also download them if a newer version exists on the Unicode\n>> consortium's servers. Option -N of wget does this nicely for us.\n>> \n>> Reviewed-by: Torsten Boegershausen \n>\n> Minor remark (Not sure if this motivates v5, may be Junio can fix it locally?)\n> s/oe/ö/\n>\n> Beside this: Thanks again (and I learned about the -N option of wget)\n\nWill fix up while queuing (only 1/3 needs it, 2/3 has it right).\n\nAlso, I'll do s/update-unicode.sh/update_unicode.sh/ on the title\nand the message to match the reality. At some point we might want\nto fix the reality to match people's expectations, though.\n\nThanks.\n"},{"id":"307091","messageId":"c35518dc-5278-bd89-569c-398e70aa8dc8@drbeat.li","threadId":"44599","inReplyTo":"xmqqmvga3yod.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 1/3] update-unicode.sh: automatically download newer definition files","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-07T00:17:31Z","receivedAt":"2016-12-07T00:17:41Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"On 05.12.16 21:31, Junio C Hamano wrote:\n> Torsten Bögershausen writes:\n> \n>> On Sat, Dec 03, 2016 at 10:00:47PM +0100, Beat Bolli wrote:\n>>> Checking just for the unicode data files' existence is not sufficient;\n>>> we should also download them if a newer version exists on the Unicode\n>>> consortium's servers. Option -N of wget does this nicely for us.\n>>>\n>>> Reviewed-by: Torsten Boegershausen \n>>\n>> Minor remark (Not sure if this motivates v5, may be Junio can fix it locally?)\n>> s/oe/ö/\n>>\n>> Beside this: Thanks again (and I learned about the -N option of wget)\n> \n> Will fix up while queuing (only 1/3 needs it, 2/3 has it right).\n> \n> Also, I'll do s/update-unicode.sh/update_unicode.sh/ on the title\n> and the message to match the reality. At some point we might want\n> to fix the reality to match people's expectations, though.\n\nThanks, Junio. This was a bit sloppy of me.\n\nI really appreciate your regard for the small things!\n\nCheers, Beat\n"},{"id":"307092","messageId":"1481072436.5381.0.camel@gmail.com","threadId":"44599","inReplyTo":"c35518dc-5278-bd89-569c-398e70aa8dc8@drbeat.li","subject":"","fromName":"jbh","fromEmail":"jbhx77@gmail.com","sentAt":"2016-12-07T01:00:36Z","receivedAt":"2016-12-07T01:00:42Z","isPatch":false,"sender":{"key":"jbhx77@gmail.com","avatar":null},"body":"unsubscribe\n"}]}