{"thread":{"id":"48676","subject":"Truncating file names with Unicode characters","startedAt":"2018-06-08T18:26:09Z","lastAt":"2018-06-09T06:24:06Z","messageCount":2,"participants":["Vitaly Potyarkin","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"349725","messageId":"CA+v=Qh7eiLoN4YcnSdtjL4-wp8rTjCpA85BkQD2fRoG7u=9_Fw@mail.gmail.com","threadId":"48676","inReplyTo":null,"subject":"Truncating file names with Unicode characters","fromName":"Vitaly Potyarkin","fromEmail":"sio.wtf@gmail.com","sentAt":"2018-06-08T18:25:29Z","receivedAt":"2018-06-08T18:26:09Z","isPatch":false,"sender":{"key":"sio.wtf@gmail.com","avatar":null},"body":"# Truncating file names with Unicode characters\n\nWhen shortening file names that contain Unicode characters, git performs\ntruncation without awareness of two-byte characters. That often leads to\nsplitting a character in half and displaying a garbage byte that's left.\n\nUnawareness of Unicode also means that filename length is calculated incorrectly\nand some output gets misaligned.\n\nI have tested this with git 2.14.1 on Windows and with git 2.11.0 on Linux. My\nconfiguration includes setting `core.quotepath = off` to display Unicode paths.\n\n# Example: `git log --stat`\n\n## Bad output: half-characters and wrong text alignment\n\nThe last file name gets truncated in the middle of the character (`ˆ` is\nwhat's left of it). Text alignment is off because string lengths are calculated\nin bytes instead of characters.\n\n    Extension/README.md                                |  28 +++++++++\n    .../Catalog.Номенклатура.xml           |  32 ++++++++++\n    .../Configuration.xml                              |   5 +-\n    ...етПереработчика.ObjectModule.txt |  39 ++++++++++++\n    ...cument.ОтчетПереработчика.xml |  68 +++++++++++++++++++++\n    .../Enum.СтавкиНДС.xml                    |  24 ++++++++\n    ...ˆирениеERPПотяркин_2018-06-05.cfe | Bin 0 -> 22018 bytes\n    7 files changed, 195 insertions(+), 1 deletion(-)\n\n## Good output with ASCII file names\n\nTruncation and alignment are done right because each character is represented\nby a single byte.\n\n    .../index.html                                             | 14\n++++++++++++++\n    docs/posts/2017/loops-in-power-query-m-language/index.html | 14\n++++++++++++++\n    .../index.html                                             |  7 +++++++\n    .../temporary-virtual-environment-for-python/index.html    | 14\n++++++++++++++\n    .../index.html                                             | 14\n++++++++++++++\n    docs/posts/2018/getting-started-with-libpq/index.html      | 14\n++++++++++++++\n    .../index.html                                             | 14\n++++++++++++++\n    .../2018/unit-testing-in-power-query-m-language/index.html |  7 +++++++\n    8 files changed, 98 insertions(+)\n"},{"id":"349774","messageId":"20180609062351.GC30224@sigill.intra.peff.net","threadId":"48676","inReplyTo":"CA+v=Qh7eiLoN4YcnSdtjL4-wp8rTjCpA85BkQD2fRoG7u=9_Fw@mail.gmail.com","subject":"Re: Truncating file names with Unicode characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-06-09T06:23:51Z","receivedAt":"2018-06-09T06:24:06Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 08, 2018 at 09:25:29PM +0300, Vitaly Potyarkin wrote:\n\n> # Truncating file names with Unicode characters\n> \n> When shortening file names that contain Unicode characters, git performs\n> truncation without awareness of two-byte characters. That often leads to\n> splitting a character in half and displaying a garbage byte that's left.\n> \n> Unawareness of Unicode also means that filename length is calculated incorrectly\n> and some output gets misaligned.\n\nYeah, I agree it would be nice for this to be more aware of multi-byte\nglyphs.\n\nUnfortunately, we don't actually know what encoding the paths are in\n(and they might not even all be in the same encoding). But hopefully we\ncould at least make the common case work a bit better. If we know that\ncore.quotepath is off, then the user is expecting paths in their\nterminal encoding. So possibly we could just rely on\ni18n.logOutputEncoding at that point (which of course defaults to utf8).\n\nIf anyone is interested in working on it, the relevant code is in\ndiff.c:show_stats(). Looks like we do a straight strlen() to get the\nwidth, which should become utf8_strwidth(). That would solve the\nproblem where we use the wrong width.\n\nTo deal with the broken code points, I think you'd need to copy the name\ninto a strbuf and use strbuf_utf8_replace() to truncate (it doesn't seem\nto be well documented, but you can grep for a few other callers).\n\n-Peff\n"}]}