patchUnicode: update of combining code points

Unicode 6.3 defines the following code as combining or accents, git_wcwidth() should return 0.

Earlier unicode standards had defined these code point as "reserved":

358 COMBINING DOT ABOVE RIGHT 359 COMBINING ASTERISK BELOW 35A COMBINING DOUBLE RING BELOW 35B COMBINING ZIGZAG ABOVE 35C COMBINING DOUBLE BREVE BELOW 487 COMBINING CYRILLIC POKRYTIE 5A2 HEBREW ACCENT ATNAH HAFUKH, 5BA HEBREW POINT HOLAM HASER FOR VAV 5C5 HEBREW MARK LOWER DOT 5C7 HEBREW POINT QAMATS QATAN 604 ARABIC SIGN SAMVAT 616 ARABIC SMALL HIGH LIGATURE ALEF WITH LAM WITH YEH 617 ARABIC SMALL HIGH ZAIN 618 ARABIC SMALL FATHA 619 ARABIC SMALL DAMMA 61A ARABIC SMALL KASRA 659 ARABIC ZWARAKAY 65A ARABIC VOWEL SIGN SMALL V ABOVE 65B ARABIC VOWEL SIGN INVERTED SMALL V ABOVE 65C ARABIC VOWEL SIGN DOT BELOW 65D ARABIC REVERSED DAMMA 65E ARABIC FATHA WITH TWO DOTS 65F ARABIC WAVY HAMZA BELOW

This commit touches only the range 300-6FF, there may be more to be updated.

Signed-off-by: Torsten Bögershausen <tboegi@web.de>
---
 utf8.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

Hi,

Torsten Bögershausen wrote:

Thanks for the update. Could the commit message also explain how this was noticed and what the user-visible effect is?

For example:

 "Unicode just announced that <...>.  That means we should mark the
  relevant code points as combining characters so git knows they are
  zero-width and doesn't screw up the alignment when presenting branch
  names in columns with 'git branch --column'"

or something like that.

[...]

I'm not sure this list is needed --- the code + the reference to the Unicode 6.3 standard seems like enough (but if you think otherwise, I don't really mind).

The "there may be more" here sounds ominous. Does that mean Unicode 6.3 also added some zero-width characters in other ranges that should be dealt with in the future? How many such ranges? How do we know when we're done?

Just biting off the most important characters first and putting off the rest for later sounds fine to me --- my complaint is that the above comment doesn't make clear what the to-do list is for finishing the update later.

Thanks and hope that helps, Jonathan

Jonathan Nieder <jrnieder@gmail.com> writes:

Perhaps (the original read clearly enough for me, though).

I can go either way.

Indeed it does ;-)

I'll queue this at the tip of 'pu', not to forget about it while waiting for a clarification.

Thanks.

On 04/09/2014 12:37 AM, Junio C Hamano wrote:

Thanks for comments, here comes the long version of the strory: I recently fooled myself by running "git config --global user.name" with a decomposed "ö" on a new Mac OS X machine.

While there was little problems on Mac OS, all Windows and Linux machines stumbled over the decomposed ö, to be more exact over 0x308, COMBINING DIARESIS, (the 2 dots), giving all kind of weired output in "git log".

Looking into commit.c and utf8.c, how to improve the situation, I made this observations:
- Some code from commit.c can possibly be moved into utf8.c, so that we only
  have 1 utf8 code parser.
- A solution would be to run precompose_string() under Mac OS (which is a nop otherwise).
  This could have saved my day. Probably I will make a patch some day.
- Some of the combining code points exist in Unicode 6.3, but not in utf8.c
  (which seams to be based on Unicode >2.0 <6.3)
  I found some in the 0x300 area, and looked at the neighbors, and had enough time to
  read all code pages up to 0x7FF. 

 So if somebody knows how to find out which code points that are combined, accents,,, or in other words should return 0 in git_wcwidth(), please let me know.

How about this as a commit message:

Unicode: partially update to version 6.3

Unicode 6.3 defines the following code points as combining or accents, git_wcwidth() should return 0.

Earlier unicode standards had defined these code point as "reserved": 358--35C 487 5A2, 5BA, 5C5, 5C7 604, 616--61A, 659--65F

Note: for this commit only the range 0..7FF has been checked,
more updates may be needed.

Signed-off-by: Torsten Bögershausen <tboegi@web.de>

Torsten Bögershausen <tboegi@web.de> writes:

Thanks.

I do not think you meant to say that the listed codepoints above are the only ones that were "reserved". Rather, the codepoints listed are what are affected by this hange, and these were all reserved.

Also it may help end-user visible effect like Jonathan asked in his earlier message. How about extending it like this?

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- utf8.c: partially update to version 6.3

Unicode 6.3 defines more code points as combining or accents. For example, the character "ö" could be expressed as an "o" followed by U+0308 COMBINING DIARESIS (aka umlaut, double-dot-above). We should consider that such a sequence of two codepoints occupies one display column for the alignment purposes, and for that, git_wcwidth() should return 0 for them. Affected codepoints are:

    U+0358..U+035C
    U+0487
    U+05A2, U+05BA, U+05C5, U+05C7
    U+0604, U+0616..U+061A, U+0659..U+065F

Earlier unicode standards had defined these as "reserved".

Only the range 0..U+07FF has been checked to see which codepoints need to be marked as 0-width while preparing for this commit; more updates may be needed.

Signed-off-by: Torsten Bögershausen <tboegi@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

Excellent, thanks.