{"thread":{"id":"36364","subject":"[PATCH] Unicode: update of combining code points","startedAt":"2014-04-07T19:39:41Z","lastAt":"2014-04-10T04:12:27Z","messageCount":6,"participants":["Torsten Bögershausen","Jonathan Nieder","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"238507","messageId":"201404072139.42519.tboegi@web.de","threadId":"36364","inReplyTo":null,"subject":"[PATCH] Unicode: update of combining code points","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-04-07T19:39:41Z","receivedAt":"2014-04-07T19:39:41Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"Unicode 6.3 defines the following code as combining or accents,\ngit_wcwidth() should return 0.\n\nEarlier unicode standards had defined these code point as \"reserved\":\n\n358 COMBINING DOT ABOVE RIGHT\n359 COMBINING ASTERISK BELOW\n35A COMBINING DOUBLE RING BELOW\n35B COMBINING ZIGZAG ABOVE\n35C COMBINING DOUBLE BREVE BELOW\n487 COMBINING CYRILLIC POKRYTIE\n5A2 HEBREW ACCENT ATNAH HAFUKH,\n5BA HEBREW POINT HOLAM HASER FOR VAV\n5C5 HEBREW MARK LOWER DOT\n5C7 HEBREW POINT QAMATS QATAN\n604 ARABIC SIGN SAMVAT\n616 ARABIC SMALL HIGH LIGATURE ALEF WITH LAM WITH YEH\n617 ARABIC SMALL HIGH ZAIN\n618 ARABIC SMALL FATHA\n619 ARABIC SMALL DAMMA\n61A ARABIC SMALL KASRA\n659 ARABIC ZWARAKAY\n65A ARABIC VOWEL SIGN SMALL V ABOVE\n65B ARABIC VOWEL SIGN INVERTED SMALL V ABOVE\n65C ARABIC VOWEL SIGN DOT BELOW\n65D ARABIC REVERSED DAMMA\n65E ARABIC FATHA WITH TWO DOTS\n65F ARABIC WAVY HAMZA BELOW\n\nThis commit touches only the range 300-6FF, there may be more to be updated.\n\nSigned-off-by: Torsten Bögershausen \n---\n utf8.c | 9 ++++-----\n 1 file changed, 4 insertions(+), 5 deletions(-)\n\ndiff --git a/utf8.c b/utf8.c\nindex a831d50..77c28d4 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -84,11 +84,10 @@ static int git_wcwidth(ucs_char_t ch)\n \t * \"uniset +cat=Me +cat=Mn +cat=Cf -00AD +1160-11FF +200B c\".\n \t */\n \tstatic const struct interval combining[] = {\n-\t\t{ 0x0300, 0x0357 }, { 0x035D, 0x036F }, { 0x0483, 0x0486 },\n-\t\t{ 0x0488, 0x0489 }, { 0x0591, 0x05A1 }, { 0x05A3, 0x05B9 },\n-\t\t{ 0x05BB, 0x05BD }, { 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 },\n-\t\t{ 0x05C4, 0x05C4 }, { 0x0600, 0x0603 }, { 0x0610, 0x0615 },\n-\t\t{ 0x064B, 0x0658 }, { 0x0670, 0x0670 }, { 0x06D6, 0x06E4 },\n+\t\t{ 0x0300, 0x036F }, { 0x0483, 0x0489 }, { 0x0591, 0x05BD },\n+\t\t{ 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 }, { 0x05C4, 0x05C5 },\n+\t\t{ 0x05C7, 0x05C7 }, { 0x0600, 0x0604 }, { 0x0610, 0x061A },\n+\t\t{ 0x064B, 0x065F }, { 0x0670, 0x0670 }, { 0x06D6, 0x06E4 },\n \t\t{ 0x06E7, 0x06E8 }, { 0x06EA, 0x06ED }, { 0x070F, 0x070F },\n \t\t{ 0x0711, 0x0711 }, { 0x0730, 0x074A }, { 0x07A6, 0x07B0 },\n \t\t{ 0x0901, 0x0902 }, { 0x093C, 0x093C }, { 0x0941, 0x0948 },\n-- \n1.9.0\n"},{"id":"238509","messageId":"20140407195458.GA3912@google.com","threadId":"36364","inReplyTo":"201404072139.42519.tboegi@web.de","subject":"Re: [PATCH] Unicode: update of combining code points","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2014-04-07T19:54:58Z","receivedAt":"2014-04-07T19:54:58Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nTorsten Bögershausen wrote:\n\n> Unicode 6.3 defines the following code as combining or accents,\n> git_wcwidth() should return 0.\n>\n> Earlier unicode standards had defined these code point as \"reserved\":\n\nThanks for the update. Could the commit message also explain how this\nwas noticed and what the user-visible effect is?\n\nFor example:\n\n \"Unicode just announced that <...>. That means we should mark the\n relevant code points as combining characters so git knows they are\n zero-width and doesn't screw up the alignment when presenting branch\n names in columns with 'git branch --column'\"\n\nor something like that.\n\n[...]\n> 358 COMBINING DOT ABOVE RIGHT\n> 359 COMBINING ASTERISK BELOW\n\nI'm not sure this list is needed --- the code + the reference to the\nUnicode 6.3 standard seems like enough (but if you think otherwise,\nI don't really mind).\n\n> This commit touches only the range 300-6FF, there may be more to be updated.\n\nThe \"there may be more\" here sounds ominous. Does that mean Unicode\n6.3 also added some zero-width characters in other ranges that should\nbe dealt with in the future? How many such ranges? How do we know\nwhen we're done?\n\nJust biting off the most important characters first and putting off\nthe rest for later sounds fine to me --- my complaint is that the\nabove comment doesn't make clear what the to-do list is for finishing\nthe update later.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"238571","messageId":"xmqq61mj30tg.fsf@gitster.dls.corp.google.com","threadId":"36364","inReplyTo":"20140407195458.GA3912@google.com","subject":"Re: [PATCH] Unicode: update of combining code points","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-08T22:37:15Z","receivedAt":"2014-04-08T22:37:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> Torsten Bögershausen wrote:\n>\n>> Unicode 6.3 defines the following code as combining or accents,\n>> git_wcwidth() should return 0.\n>>\n>> Earlier unicode standards had defined these code point as \"reserved\":\n>\n> Thanks for the update. Could the commit message also explain how this\n> was noticed and what the user-visible effect is?\n>\n> For example:\n>\n> \"Unicode just announced that <...>. That means we should mark the\n> relevant code points as combining characters so git knows they are\n> zero-width and doesn't screw up the alignment when presenting branch\n> names in columns with 'git branch --column'\"\n>\n> or something like that.\n\nPerhaps (the original read clearly enough for me, though).\n\n> [...]\n>> 358 COMBINING DOT ABOVE RIGHT\n>> 359 COMBINING ASTERISK BELOW\n>\n> I'm not sure this list is needed --- the code + the reference to the\n> Unicode 6.3 standard seems like enough (but if you think otherwise,\n> I don't really mind).\n\nI can go either way.\n\n>> This commit touches only the range 300-6FF, there may be more to be updated.\n>\n> The \"there may be more\" here sounds ominous.\n\nIndeed it does ;-)\n\n> Does that mean Unicode\n> 6.3 also added some zero-width characters in other ranges that should\n> be dealt with in the future? How many such ranges? How do we know\n> when we're done?\n>\n> Just biting off the most important characters first and putting off\n> the rest for later sounds fine to me --- my complaint is that the\n> above comment doesn't make clear what the to-do list is for finishing\n> the update later.\n\nI'll queue this at the tip of 'pu', not to forget about it while\nwaiting for a clarification.\n\nThanks.\n"},{"id":"238585","messageId":"534579DD.1060607@web.de","threadId":"36364","inReplyTo":"xmqq61mj30tg.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] Unicode: update of combining code points","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-04-09T16:48:29Z","receivedAt":"2014-04-09T16:48:29Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 04/09/2014 12:37 AM, Junio C Hamano wrote:\n> Jonathan Nieder writes:\n>\n>> Torsten Bögershausen wrote:\n>>\n>>> Unicode 6.3 defines the following code as combining or accents,\n>>> git_wcwidth() should return 0.\n>>>\n>>> Earlier unicode standards had defined these code point as \"reserved\":\n>> Thanks for the update. Could the commit message also explain how this\n>> was noticed and what the user-visible effect is?\n>>\n>> For example:\n>>\n>> \"Unicode just announced that <...>. That means we should mark the\n>> relevant code points as combining characters so git knows they are\n>> zero-width and doesn't screw up the alignment when presenting branch\n>> names in columns with 'git branch --column'\"\n>>\n>> or something like that.\n> Perhaps (the original read clearly enough for me, though).\n>\n>> [...]\n>>> 358 COMBINING DOT ABOVE RIGHT\n>>> 359 COMBINING ASTERISK BELOW\n>> I'm not sure this list is needed --- the code + the reference to the\n>> Unicode 6.3 standard seems like enough (but if you think otherwise,\n>> I don't really mind).\n> I can go either way.\n>\n>>> This commit touches only the range 300-6FF, there may be more to be updated.\n>> The \"there may be more\" here sounds ominous.\n> Indeed it does ;-)\n>\n>> Does that mean Unicode\n>> 6.3 also added some zero-width characters in other ranges that should\n>> be dealt with in the future? How many such ranges? How do we know\n>> when we're done?\n>>\n>> Just biting off the most important characters first and putting off\n>> the rest for later sounds fine to me --- my complaint is that the\n>> above comment doesn't make clear what the to-do list is for finishing\n>> the update later.\n> I'll queue this at the tip of 'pu', not to forget about it while\n> waiting for a clarification.\n>\n> Thanks.\nThanks for comments, here comes the long version of the strory:\nI recently fooled myself by running\n\"git config --global user.name\" with a decomposed \"ö\" on a new Mac OS X machine.\n\nWhile there was little problems on Mac OS, all Windows and Linux machines stumbled\nover the decomposed ö, to be more exact over 0x308, COMBINING DIARESIS, (the 2 dots),\ngiving all kind of weired output in \"git log\".\n\nLooking into commit.c and utf8.c, how to improve the situation, I made this observations:\n- Some code from commit.c can possibly be moved into utf8.c, so that we only\n have 1 utf8 code parser.\n- A solution would be to run precompose_string() under Mac OS (which is a nop otherwise).\n This could have saved my day. Probably I will make a patch some day.\n- Some of the combining code points exist in Unicode 6.3, but not in utf8.c\n (which seams to be based on Unicode >2.0 <6.3)\n I found some in the 0x300 area, and looked at the neighbors, and had enough time to\n read all code pages up to 0x7FF. \n\n So if somebody knows how to find out which code points that are combined, accents,,, or in other words should return 0 in git_wcwidth(), please let me know.\n\nHow about this as a commit message:\n\nUnicode: partially update to version 6.3\n\nUnicode 6.3 defines the following code points as combining or accents,\ngit_wcwidth() should return 0.\n\nEarlier unicode standards had defined these code point as \"reserved\":\n358--35C\n487\n5A2, 5BA, 5C5, 5C7\n604, 616--61A, 659--65F\n\nNote: for this commit only the range 0..7FF has been checked,\nmore updates may be needed.\n\nSigned-off-by: Torsten Bögershausen \n"},{"id":"238587","messageId":"xmqqr4561kde.fsf@gitster.dls.corp.google.com","threadId":"36364","inReplyTo":"534579DD.1060607@web.de","subject":"Re: [PATCH] Unicode: update of combining code points","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-09T17:30:05Z","receivedAt":"2014-04-09T17:30:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n> How about this as a commit message:\n>\n> Unicode: partially update to version 6.3\n>\n> Unicode 6.3 defines the following code points as combining or accents,\n> git_wcwidth() should return 0.\n>\n> Earlier unicode standards had defined these code point as \"reserved\":\n> 358--35C\n> 487\n> 5A2, 5BA, 5C5, 5C7\n> 604, 616--61A, 659--65F\n>\n> Note: for this commit only the range 0..7FF has been checked,\n> more updates may be needed.\n>\n> Signed-off-by: Torsten Bögershausen \n\nThanks.\n\nI do not think you meant to say that the listed codepoints above are\nthe only ones that were \"reserved\". Rather, the codepoints listed\nare what are affected by this hange, and these were all reserved.\n\nAlso it may help end-user visible effect like Jonathan asked in his\nearlier message. How about extending it like this?\n\n-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --\nutf8.c: partially update to version 6.3\n\nUnicode 6.3 defines more code points as combining or accents. For\nexample, the character \"ö\" could be expressed as an \"o\" followed by\nU+0308 COMBINING DIARESIS (aka umlaut, double-dot-above). We should\nconsider that such a sequence of two codepoints occupies one display\ncolumn for the alignment purposes, and for that, git_wcwidth()\nshould return 0 for them. Affected codepoints are:\n\n U+0358..U+035C\n U+0487\n U+05A2, U+05BA, U+05C5, U+05C7\n U+0604, U+0616..U+061A, U+0659..U+065F\n\nEarlier unicode standards had defined these as \"reserved\".\n\nOnly the range 0..U+07FF has been checked to see which codepoints\nneed to be marked as 0-width while preparing for this commit; more\nupdates may be needed.\n\nSigned-off-by: Torsten Bögershausen \nSigned-off-by: Junio C Hamano \n-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --\n"},{"id":"238612","messageId":"53461A2B.4000009@web.de","threadId":"36364","inReplyTo":"xmqqr4561kde.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] Unicode: update of combining code points","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-04-10T04:12:27Z","receivedAt":"2014-04-10T04:12:27Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"Excellent, thanks.\n"}]}