# Re: [PATCH] git-blame: Make the output human readable 5 messages from 2006-03-06 to 2006-03-08. Participants: linux@horizon.com, Sergey Vlasov. Thread: https://gitlist.dev/t/3587 ## linux@horizon.com, 2006-03-06 19:33 Subject: Re: [PATCH] git-blame: Make the output human readable Message-ID: <20060306193326.19262.qmail@science.horizon.com> URL: https://gitlist.dev/e/20060306193326.19262.qmail%40science.horizon.com ``` Well, getting 15 characters in UTF-8 is easy (just stop before the 16th byte for which ((b & 0xc0) != 0x80)), but what about combining characters? You've got accents and stuff to worry about. And the annoying fact that Unicode defined accents as suffixes, so you have to go past the 15th column to include all of the And then there's that fact that many characters are traditionally represented as double-wide forms, even on character terminals. See http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c for details an an example implementation of wcwidth(). Using that, it would be something like (compiles but untested): /* * Return the number of bytes from the nul-terminated utf8 string * that can be printed in at most max columns using a monospaced * font. *actual returns the number of columns actually occupied, * which may be less than max. * * Output is truncated before any control characters or illegal * UTF-8 sequences. */ unsigned fit_columns(char const *utf8, unsigned max, unsigned *actual) { char const * const origin = utf8; unsigned width = 0; unsigned pos = 0; unsigned c; for (;;) { unsigned w; unsigned c = *utf8++; /* Part 1: Parse the next Unicode code point */ if (c < 0x20) { break; /* Control character - stop */ } else if (c < 0x7F) { w = 1; /* Standard ASCII */ } else if (c < 0xC2 || c > 0xF4) { break; /* DEL or illegal Unicode */ } else { /* Multi-byte UTF-8 sequence */ unsigned n; unsigned char byte = *utf8++; if (c < 0xE0) { /* 2-byte sequence: U+0080..U+07FF */ n = 1; c &= 0x1F; } else if (c < 0xF0) { /* 3-byte sequence: U+0800..U+FFFF */ if (c == 0xE0 && byte < 0xA0) break; /* < /U+0800 */ n = 2; c &= 0x0F; } else { /* 4-byte sequence: U+10000..U+10FFFF */ if (byte < 0x90 ? c == 0xF0 : c == 0xF4) break; /* < 10000 or > 10FFFF */ n = 3; c &= 0x07; } for (; n--; byte = *utf8++) { if (byte & 0xc0 != 0x80) goto done; /* Double break */ c = (c << 6) | (byte & 0x3f); } /* Now find the width of it */ w = wcwidth(c); if (w == -1) break; } /* Part 2: Figure out if it will fit */ if (width + w > max) break; /* Would exceed space - stop */ /* Part 3: It fits; update our statistics */ width += w; pos = (unsigned)(utf8 - origin); } done: if (actual) *actual = width; return pos; } ``` ## Sergey Vlasov, 2006-03-08 14:32 Subject: Re: [PATCH] git-blame: Make the output human readable Message-ID: <20060308173249.1faed1d7.vsu@altlinux.ru> URL: https://gitlist.dev/e/20060308173249.1faed1d7.vsu%40altlinux.ru In-Reply-To: <20060306193326.19262.qmail@science.horizon.com> ``` On 6 Mar 2006 14:33:26 -0500 linux@horizon.com wrote: > Well, getting 15 characters in UTF-8 is easy (just stop before the 16th > byte for which ((b & 0xc0) != 0x80)), but what about combining characters? > > You've got accents and stuff to worry about. And the annoying fact that > Unicode defined accents as suffixes, so you have to go past the 15th > column to include all of the > > And then there's that fact that many characters are traditionally > represented as double-wide forms, even on character terminals. > > See http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c for details > an an example implementation of wcwidth(). > [skip] > /* Now find the width of it */ > w = wcwidth(c); And this won't work, unless you also add that wcwidth() implementation to git. The problem is that the wchar_t encoding is not specified anywhere - glibc uses Unicode for it, but other systems can use whatever they want (even locale-dependent). ``` ## linux@horizon.com, 2006-03-08 18:04 Subject: Re: [PATCH] git-blame: Make the output human readable Message-ID: <20060308180422.27978.qmail@science.horizon.com> URL: https://gitlist.dev/e/20060308180422.27978.qmail%40science.horizon.com In-Reply-To: <20060308173249.1faed1d7.vsu@altlinux.ru> ``` > And this won't work, unless you also add that wcwidth() implementation > to git. That was the general idea. It is freely usable. > The problem is that the wchar_t encoding is not specified anywhere - > glibc uses Unicode for it, but other systems can use whatever they want > (even locale-dependent). Why is that a problem? None of the code mentioned even uses wchar_t. The code I wrote converts from UTF-8 to straight Unicode, and that's what Markus Kuhn's wcwidth() expects as an argument. At no time do we ask the compiler for its opinion on the subject. ``` ## Sergey Vlasov, 2006-03-08 18:30 Subject: Re: [PATCH] git-blame: Make the output human readable Message-ID: <20060308183059.GD9555@procyon.home> URL: https://gitlist.dev/e/20060308183059.GD9555%40procyon.home In-Reply-To: <20060308180422.27978.qmail@science.horizon.com> ``` On Wed, Mar 08, 2006 at 01:04:22PM -0500, linux@horizon.com wrote: > > And this won't work, unless you also add that wcwidth() implementation > > to git. > > That was the general idea. It is freely usable. > > > The problem is that the wchar_t encoding is not specified anywhere - > > glibc uses Unicode for it, but other systems can use whatever they want > > (even locale-dependent). > > Why is that a problem? None of the code mentioned even uses wchar_t. > The code I wrote converts from UTF-8 to straight Unicode, and that's > what Markus Kuhn's wcwidth() expects as an argument. wcwidth() is a standard library function which takes a wchar_t: http://www.opengroup.org/onlinepubs/009695399/functions/wcwidth.html It is easy to write a program which assumes that wchar_t is Unicode without noticing it, because it will work fine with glibc... So that mk_wcwidth() must be used unconditionally, and not as a fallback for systems which do not provide wcwidth() in libc. ``` ## linux@horizon.com, 2006-03-08 19:06 Subject: Re: [PATCH] git-blame: Make the output human readable Message-ID: <20060308190600.7290.qmail@science.horizon.com> URL: https://gitlist.dev/e/20060308190600.7290.qmail%40science.horizon.com In-Reply-To: <20060308183059.GD9555@procyon.home> ``` > So that mk_wcwidth() must be used unconditionally, and not as a > fallback for systems which do not provide wcwidth() in libc. Ah, the light dawns! I now understand your confusion. That was exactly the idea; apologies for being unclear. Kind of like the existing issspace(), isdigit(), isalpha(), etc. implementations in git-compat-util.h to avoid the mysteries and vagaries of locales. A UTF-8-only solution is desired. ```