# [PATCH] Unicode: update of combining code points 6 messages from 2014-04-07 to 2014-04-10. Participants: Torsten Bögershausen, Jonathan Nieder, Junio C Hamano. Thread: https://gitlist.dev/t/36364 ## Torsten Bögershausen, 2014-04-07 19:39 Subject: [PATCH] Unicode: update of combining code points Message-ID: <201404072139.42519.tboegi@web.de> URL: https://gitlist.dev/e/201404072139.42519.tboegi%40web.de ``` Unicode 6.3 defines the following code as combining or accents, git_wcwidth() should return 0. Earlier unicode standards had defined these code point as "reserved": 358 COMBINING DOT ABOVE RIGHT 359 COMBINING ASTERISK BELOW 35A COMBINING DOUBLE RING BELOW 35B COMBINING ZIGZAG ABOVE 35C COMBINING DOUBLE BREVE BELOW 487 COMBINING CYRILLIC POKRYTIE 5A2 HEBREW ACCENT ATNAH HAFUKH, 5BA HEBREW POINT HOLAM HASER FOR VAV 5C5 HEBREW MARK LOWER DOT 5C7 HEBREW POINT QAMATS QATAN 604 ARABIC SIGN SAMVAT 616 ARABIC SMALL HIGH LIGATURE ALEF WITH LAM WITH YEH 617 ARABIC SMALL HIGH ZAIN 618 ARABIC SMALL FATHA 619 ARABIC SMALL DAMMA 61A ARABIC SMALL KASRA 659 ARABIC ZWARAKAY 65A ARABIC VOWEL SIGN SMALL V ABOVE 65B ARABIC VOWEL SIGN INVERTED SMALL V ABOVE 65C ARABIC VOWEL SIGN DOT BELOW 65D ARABIC REVERSED DAMMA 65E ARABIC FATHA WITH TWO DOTS 65F ARABIC WAVY HAMZA BELOW This commit touches only the range 300-6FF, there may be more to be updated. Signed-off-by: Torsten Bögershausen --- utf8.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/utf8.c b/utf8.c index a831d50..77c28d4 100644 --- a/utf8.c +++ b/utf8.c @@ -84,11 +84,10 @@ static int git_wcwidth(ucs_char_t ch) * "uniset +cat=Me +cat=Mn +cat=Cf -00AD +1160-11FF +200B c". */ static const struct interval combining[] = { - { 0x0300, 0x0357 }, { 0x035D, 0x036F }, { 0x0483, 0x0486 }, - { 0x0488, 0x0489 }, { 0x0591, 0x05A1 }, { 0x05A3, 0x05B9 }, - { 0x05BB, 0x05BD }, { 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 }, - { 0x05C4, 0x05C4 }, { 0x0600, 0x0603 }, { 0x0610, 0x0615 }, - { 0x064B, 0x0658 }, { 0x0670, 0x0670 }, { 0x06D6, 0x06E4 }, + { 0x0300, 0x036F }, { 0x0483, 0x0489 }, { 0x0591, 0x05BD }, + { 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 }, { 0x05C4, 0x05C5 }, + { 0x05C7, 0x05C7 }, { 0x0600, 0x0604 }, { 0x0610, 0x061A }, + { 0x064B, 0x065F }, { 0x0670, 0x0670 }, { 0x06D6, 0x06E4 }, { 0x06E7, 0x06E8 }, { 0x06EA, 0x06ED }, { 0x070F, 0x070F }, { 0x0711, 0x0711 }, { 0x0730, 0x074A }, { 0x07A6, 0x07B0 }, { 0x0901, 0x0902 }, { 0x093C, 0x093C }, { 0x0941, 0x0948 }, -- 1.9.0 ``` ## Jonathan Nieder, 2014-04-07 19:54 Subject: Re: [PATCH] Unicode: update of combining code points Message-ID: <20140407195458.GA3912@google.com> URL: https://gitlist.dev/e/20140407195458.GA3912%40google.com In-Reply-To: <201404072139.42519.tboegi@web.de> ``` Hi, Torsten Bögershausen wrote: > Unicode 6.3 defines the following code as combining or accents, > git_wcwidth() should return 0. > > Earlier unicode standards had defined these code point as "reserved": Thanks for the update. Could the commit message also explain how this was noticed and what the user-visible effect is? For example: "Unicode just announced that <...>. That means we should mark the relevant code points as combining characters so git knows they are zero-width and doesn't screw up the alignment when presenting branch names in columns with 'git branch --column'" or something like that. [...] > 358 COMBINING DOT ABOVE RIGHT > 359 COMBINING ASTERISK BELOW I'm not sure this list is needed --- the code + the reference to the Unicode 6.3 standard seems like enough (but if you think otherwise, I don't really mind). > This commit touches only the range 300-6FF, there may be more to be updated. The "there may be more" here sounds ominous. Does that mean Unicode 6.3 also added some zero-width characters in other ranges that should be dealt with in the future? How many such ranges? How do we know when we're done? Just biting off the most important characters first and putting off the rest for later sounds fine to me --- my complaint is that the above comment doesn't make clear what the to-do list is for finishing the update later. Thanks and hope that helps, Jonathan ``` ## Junio C Hamano, 2014-04-08 22:37 Subject: Re: [PATCH] Unicode: update of combining code points Message-ID: URL: https://gitlist.dev/e/xmqq61mj30tg.fsf%40gitster.dls.corp.google.com In-Reply-To: <20140407195458.GA3912@google.com> ``` Jonathan Nieder writes: > Torsten Bögershausen wrote: > >> Unicode 6.3 defines the following code as combining or accents, >> git_wcwidth() should return 0. >> >> Earlier unicode standards had defined these code point as "reserved": > > Thanks for the update. Could the commit message also explain how this > was noticed and what the user-visible effect is? > > For example: > > "Unicode just announced that <...>. That means we should mark the > relevant code points as combining characters so git knows they are > zero-width and doesn't screw up the alignment when presenting branch > names in columns with 'git branch --column'" > > or something like that. Perhaps (the original read clearly enough for me, though). > [...] >> 358 COMBINING DOT ABOVE RIGHT >> 359 COMBINING ASTERISK BELOW > > I'm not sure this list is needed --- the code + the reference to the > Unicode 6.3 standard seems like enough (but if you think otherwise, > I don't really mind). I can go either way. >> This commit touches only the range 300-6FF, there may be more to be updated. > > The "there may be more" here sounds ominous. Indeed it does ;-) > Does that mean Unicode > 6.3 also added some zero-width characters in other ranges that should > be dealt with in the future? How many such ranges? How do we know > when we're done? > > Just biting off the most important characters first and putting off > the rest for later sounds fine to me --- my complaint is that the > above comment doesn't make clear what the to-do list is for finishing > the update later. I'll queue this at the tip of 'pu', not to forget about it while waiting for a clarification. Thanks. ``` ## Torsten Bögershausen, 2014-04-09 16:48 Subject: Re: [PATCH] Unicode: update of combining code points Message-ID: <534579DD.1060607@web.de> URL: https://gitlist.dev/e/534579DD.1060607%40web.de In-Reply-To: ``` On 04/09/2014 12:37 AM, Junio C Hamano wrote: > Jonathan Nieder writes: > >> Torsten Bögershausen wrote: >> >>> Unicode 6.3 defines the following code as combining or accents, >>> git_wcwidth() should return 0. >>> >>> Earlier unicode standards had defined these code point as "reserved": >> Thanks for the update. Could the commit message also explain how this >> was noticed and what the user-visible effect is? >> >> For example: >> >> "Unicode just announced that <...>. That means we should mark the >> relevant code points as combining characters so git knows they are >> zero-width and doesn't screw up the alignment when presenting branch >> names in columns with 'git branch --column'" >> >> or something like that. > Perhaps (the original read clearly enough for me, though). > >> [...] >>> 358 COMBINING DOT ABOVE RIGHT >>> 359 COMBINING ASTERISK BELOW >> I'm not sure this list is needed --- the code + the reference to the >> Unicode 6.3 standard seems like enough (but if you think otherwise, >> I don't really mind). > I can go either way. > >>> This commit touches only the range 300-6FF, there may be more to be updated. >> The "there may be more" here sounds ominous. > Indeed it does ;-) > >> Does that mean Unicode >> 6.3 also added some zero-width characters in other ranges that should >> be dealt with in the future? How many such ranges? How do we know >> when we're done? >> >> Just biting off the most important characters first and putting off >> the rest for later sounds fine to me --- my complaint is that the >> above comment doesn't make clear what the to-do list is for finishing >> the update later. > I'll queue this at the tip of 'pu', not to forget about it while > waiting for a clarification. > > Thanks. Thanks for comments, here comes the long version of the strory: I recently fooled myself by running "git config --global user.name" with a decomposed "ö" on a new Mac OS X machine. While there was little problems on Mac OS, all Windows and Linux machines stumbled over the decomposed ö, to be more exact over 0x308, COMBINING DIARESIS, (the 2 dots), giving all kind of weired output in "git log". Looking into commit.c and utf8.c, how to improve the situation, I made this observations: - Some code from commit.c can possibly be moved into utf8.c, so that we only have 1 utf8 code parser. - A solution would be to run precompose_string() under Mac OS (which is a nop otherwise). This could have saved my day. Probably I will make a patch some day. - Some of the combining code points exist in Unicode 6.3, but not in utf8.c (which seams to be based on Unicode >2.0 <6.3) I found some in the 0x300 area, and looked at the neighbors, and had enough time to read all code pages up to 0x7FF. So if somebody knows how to find out which code points that are combined, accents,,, or in other words should return 0 in git_wcwidth(), please let me know. How about this as a commit message: Unicode: partially update to version 6.3 Unicode 6.3 defines the following code points as combining or accents, git_wcwidth() should return 0. Earlier unicode standards had defined these code point as "reserved": 358--35C 487 5A2, 5BA, 5C5, 5C7 604, 616--61A, 659--65F Note: for this commit only the range 0..7FF has been checked, more updates may be needed. Signed-off-by: Torsten Bögershausen ``` ## Junio C Hamano, 2014-04-09 17:30 Subject: Re: [PATCH] Unicode: update of combining code points Message-ID: URL: https://gitlist.dev/e/xmqqr4561kde.fsf%40gitster.dls.corp.google.com In-Reply-To: <534579DD.1060607@web.de> ``` Torsten Bögershausen writes: > How about this as a commit message: > > Unicode: partially update to version 6.3 > > Unicode 6.3 defines the following code points as combining or accents, > git_wcwidth() should return 0. > > Earlier unicode standards had defined these code point as "reserved": > 358--35C > 487 > 5A2, 5BA, 5C5, 5C7 > 604, 616--61A, 659--65F > > Note: for this commit only the range 0..7FF has been checked, > more updates may be needed. > > Signed-off-by: Torsten Bögershausen Thanks. I do not think you meant to say that the listed codepoints above are the only ones that were "reserved". Rather, the codepoints listed are what are affected by this hange, and these were all reserved. Also it may help end-user visible effect like Jonathan asked in his earlier message. How about extending it like this? -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- utf8.c: partially update to version 6.3 Unicode 6.3 defines more code points as combining or accents. For example, the character "ö" could be expressed as an "o" followed by U+0308 COMBINING DIARESIS (aka umlaut, double-dot-above). We should consider that such a sequence of two codepoints occupies one display column for the alignment purposes, and for that, git_wcwidth() should return 0 for them. Affected codepoints are: U+0358..U+035C U+0487 U+05A2, U+05BA, U+05C5, U+05C7 U+0604, U+0616..U+061A, U+0659..U+065F Earlier unicode standards had defined these as "reserved". Only the range 0..U+07FF has been checked to see which codepoints need to be marked as 0-width while preparing for this commit; more updates may be needed. Signed-off-by: Torsten Bögershausen Signed-off-by: Junio C Hamano -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- ``` ## Torsten Bögershausen, 2014-04-10 04:12 Subject: Re: [PATCH] Unicode: update of combining code points Message-ID: <53461A2B.4000009@web.de> URL: https://gitlist.dev/e/53461A2B.4000009%40web.de In-Reply-To: ``` Excellent, thanks. ```