{"thread":{"id":"3587","subject":"Re: [PATCH] git-blame: Make the output human readable","startedAt":"2006-03-06T19:33:26Z","lastAt":"2006-03-08T19:06:00Z","messageCount":5,"participants":["linux@horizon.com","Sergey Vlasov"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"17268","messageId":"20060306193326.19262.qmail@science.horizon.com","threadId":"3587","inReplyTo":null,"subject":"Re: [PATCH] git-blame: Make the output human readable","fromName":"","fromEmail":"linux@horizon.com","sentAt":"2006-03-06T19:33:26Z","receivedAt":"2006-03-06T19:33:26Z","isPatch":true,"sender":{"key":"linux@horizon.com","avatar":null},"body":"Well, getting 15 characters in UTF-8 is easy (just stop before the 16th\nbyte for which ((b & 0xc0) != 0x80)), but what about combining characters?\n\nYou've got accents and stuff to worry about.  And the annoying fact that\nUnicode defined accents as suffixes, so you have to go past the 15th\ncolumn to include all of the \n\nAnd then there's that fact that many characters are traditionally\nrepresented as double-wide forms, even on character terminals.\n\nSee http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c for details\nan an example implementation of wcwidth().\n\nUsing that, it would be something like (compiles but untested):\n\n/*\n * Return the number of bytes from the nul-terminated utf8 string\n * that can be printed in at most max columns using a monospaced\n * font.  *actual returns the number of columns actually occupied,\n * which may be less than max.\n *\n * Output is truncated before any control characters or illegal\n * UTF-8 sequences.\n */\nunsigned\nfit_columns(char const *utf8, unsigned max, unsigned *actual)\n{\n\tchar const * const origin = utf8;\n\tunsigned width = 0;\n\tunsigned pos = 0;\n\tunsigned c;\n\n\tfor (;;) {\n\t\tunsigned w;\n\t\tunsigned c = *utf8++;\n\n\t\t/* Part 1: Parse the next Unicode code point */\n\t\tif (c < 0x20) {\n\t\t\tbreak;\t/* Control character - stop */\n\t\t} else if (c < 0x7F) {\n\t\t\tw = 1;\t/* Standard ASCII */\n\t\t} else if (c < 0xC2 || c > 0xF4) {\n\t\t\tbreak;\t/* DEL or illegal Unicode */\n\t\t} else {\n\t\t\t/* Multi-byte UTF-8 sequence */\n\t\t\tunsigned n;\n\t\t\tunsigned char byte = *utf8++;\n\n\t\t\tif (c < 0xE0) {\n\t\t\t\t/* 2-byte sequence: U+0080..U+07FF */\n\t\t\t\tn = 1;\n\t\t\t\tc &= 0x1F;\n\t\t\t} else if (c < 0xF0) {\n\t\t\t\t/* 3-byte sequence: U+0800..U+FFFF */\n\t\t\t\tif (c == 0xE0 && byte < 0xA0)\n\t\t\t\t\tbreak;\t/* < /U+0800 */\n\t\t\t\tn = 2;\n\t\t\t\tc &= 0x0F;\n\t\t\t} else {\n\t\t\t\t/* 4-byte sequence: U+10000..U+10FFFF */\n\t\t\t\tif (byte < 0x90 ? c == 0xF0 : c == 0xF4)\n\t\t\t\t\tbreak; /* < 10000 or > 10FFFF */\n\t\t\t\tn = 3;\n\t\t\t\tc &= 0x07;\n\t\t\t}\n\n\t\t\tfor (; n--; byte = *utf8++) {\n\t\t\t\tif (byte & 0xc0 != 0x80)\n\t\t\t\t\tgoto done;\t/* Double break */\n\t\t\t\tc = (c << 6) | (byte & 0x3f);\n\t\t\t}\n\t\t\t/* Now find the width of it */\n\t\t\tw = wcwidth(c);\n\t\t\tif (w == -1)\n\t\t\t\tbreak;\n\t\t}\n\n\t\t/* Part 2: Figure out if it will fit */\n\t\tif (width + w > max)\n\t\t\tbreak;\t/* Would exceed space - stop */\n\t\t/* Part 3: It fits; update our statistics */\n\t\twidth += w;\n\t\tpos = (unsigned)(utf8 - origin);\n\t}\n\ndone:\n\tif (actual)\n\t\t*actual = width;\n\treturn pos;\n}\n"},{"id":"17332","messageId":"20060308173249.1faed1d7.vsu@altlinux.ru","threadId":"3587","inReplyTo":"20060306193326.19262.qmail@science.horizon.com","subject":"Re: [PATCH] git-blame: Make the output human readable","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2006-03-08T14:32:49Z","receivedAt":"2006-03-08T14:32:49Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On 6 Mar 2006 14:33:26 -0500 linux@horizon.com wrote:\n\n> Well, getting 15 characters in UTF-8 is easy (just stop before the 16th\n> byte for which ((b & 0xc0) != 0x80)), but what about combining characters?\n> \n> You've got accents and stuff to worry about.  And the annoying fact that\n> Unicode defined accents as suffixes, so you have to go past the 15th\n> column to include all of the \n> \n> And then there's that fact that many characters are traditionally\n> represented as double-wide forms, even on character terminals.\n> \n> See http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c for details\n> an an example implementation of wcwidth().\n> \n[skip]\n> \t\t\t/* Now find the width of it */\n> \t\t\tw = wcwidth(c);\n\nAnd this won't work, unless you also add that wcwidth() implementation\nto git.\n\nThe problem is that the wchar_t encoding is not specified anywhere -\nglibc uses Unicode for it, but other systems can use whatever they want\n(even locale-dependent).\n"},{"id":"17337","messageId":"20060308180422.27978.qmail@science.horizon.com","threadId":"3587","inReplyTo":"20060308173249.1faed1d7.vsu@altlinux.ru","subject":"Re: [PATCH] git-blame: Make the output human readable","fromName":"","fromEmail":"linux@horizon.com","sentAt":"2006-03-08T18:04:22Z","receivedAt":"2006-03-08T18:04:22Z","isPatch":true,"sender":{"key":"linux@horizon.com","avatar":null},"body":"> And this won't work, unless you also add that wcwidth() implementation\n> to git.\n\nThat was the general idea.  It is freely usable.\n\n> The problem is that the wchar_t encoding is not specified anywhere -\n> glibc uses Unicode for it, but other systems can use whatever they want\n> (even locale-dependent).\n\nWhy is that a problem?  None of the code mentioned even uses wchar_t.\nThe code I wrote converts from UTF-8 to straight Unicode, and that's\nwhat Markus Kuhn's wcwidth() expects as an argument.\n\nAt no time do we ask the compiler for its opinion on the subject.\n"},{"id":"17339","messageId":"20060308183059.GD9555@procyon.home","threadId":"3587","inReplyTo":"20060308180422.27978.qmail@science.horizon.com","subject":"Re: [PATCH] git-blame: Make the output human readable","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2006-03-08T18:30:59Z","receivedAt":"2006-03-08T18:30:59Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Wed, Mar 08, 2006 at 01:04:22PM -0500, linux@horizon.com wrote:\n> > And this won't work, unless you also add that wcwidth() implementation\n> > to git.\n> \n> That was the general idea.  It is freely usable.\n> \n> > The problem is that the wchar_t encoding is not specified anywhere -\n> > glibc uses Unicode for it, but other systems can use whatever they want\n> > (even locale-dependent).\n> \n> Why is that a problem?  None of the code mentioned even uses wchar_t.\n> The code I wrote converts from UTF-8 to straight Unicode, and that's\n> what Markus Kuhn's wcwidth() expects as an argument.\n\nwcwidth() is a standard library function which takes a wchar_t:\n\nhttp://www.opengroup.org/onlinepubs/009695399/functions/wcwidth.html\n\nIt is easy to write a program which assumes that wchar_t is Unicode\nwithout noticing it, because it will work fine with glibc...\n\nSo that mk_wcwidth() must be used unconditionally, and not as a\nfallback for systems which do not provide wcwidth() in libc.\n"},{"id":"17341","messageId":"20060308190600.7290.qmail@science.horizon.com","threadId":"3587","inReplyTo":"20060308183059.GD9555@procyon.home","subject":"Re: [PATCH] git-blame: Make the output human readable","fromName":"","fromEmail":"linux@horizon.com","sentAt":"2006-03-08T19:06:00Z","receivedAt":"2006-03-08T19:06:00Z","isPatch":true,"sender":{"key":"linux@horizon.com","avatar":null},"body":"> So that mk_wcwidth() must be used unconditionally, and not as a\n> fallback for systems which do not provide wcwidth() in libc.\n\nAh, the light dawns!  I now understand your confusion.\nThat was exactly the idea; apologies for being unclear.\n\nKind of like the existing issspace(), isdigit(), isalpha(), etc.\nimplementations in git-compat-util.h to avoid the mysteries and vagaries\nof locales.  A UTF-8-only solution is desired.\n"}]}