{"thread":{"id":"55777","subject":"[BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","startedAt":"2021-05-26T22:47:23Z","lastAt":"2021-05-31T03:36:04Z","messageCount":14,"participants":["Yuri","Junio C Hamano","Torsten Bögershausen","Jeff King","Bagas Sanjaya","Thomas Guyot"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"425590","messageId":"f7e2e271-dcec-2886-f33e-62778a429850@rawbw.com","threadId":"55777","inReplyTo":null,"subject":"[BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Yuri","fromEmail":"yuri@rawbw.com","sentAt":"2021-05-26T22:47:15Z","receivedAt":"2021-05-26T22:47:23Z","isPatch":false,"sender":{"key":"yuri@rawbw.com","avatar":null},"body":"I have the file that contains the \"∞\" character in its name.\n\n\nWhen this file was modified, 'git status .' showed it as:\n\n >    modified:   \"file-name-\\342\\210\\236.ext\"\n\n\nIt replaced the UTF8 character with its byte representation, and put the \nfile name in quotes.\n\n\ngit should show such files without escaping when the terminal is able to \nshow UTF8 characters because escaping decreases readability.\n\n$ env | grep TERM\nCOLORTERM=truecolor\nTERM=xterm-256color\n\n$ env | grep LANG\nLANG=C.UTF-8\n\n$ env | grep CTYPE\nLC_CTYPE=en_US.UTF-8\n\n\nThanks,\n\nYuri\n\n"},{"id":"425593","messageId":"xmqq35u9ax5j.fsf@gitster.g","threadId":"55777","inReplyTo":"f7e2e271-dcec-2886-f33e-62778a429850@rawbw.com","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-26T23:32:56Z","receivedAt":"2021-05-26T23:33:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Yuri <yuri@rawbw.com> writes:\n\n> I have the file that contains the \"∞\" character in its name.\n\n\"git config core.quotepath no\"?\n"},{"id":"425597","messageId":"6318ccec-ec96-91a8-fd65-85daf4a9a22b@rawbw.com","threadId":"55777","inReplyTo":"xmqq35u9ax5j.fsf@gitster.g","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Yuri","fromEmail":"yuri@rawbw.com","sentAt":"2021-05-26T23:41:38Z","receivedAt":"2021-05-26T23:41:42Z","isPatch":false,"sender":{"key":"yuri@rawbw.com","avatar":null},"body":"On 5/26/21 4:32 PM, Junio C Hamano wrote:\n> \"git config core.quotepath no\"?\n\n\nI didn't have the 'core.quotepath' value set. 'git config core.quotepath \nno' changed the behavior to no quoting.\n\nSo it looks like the default value of 'core.quotepath' is incorrect: it \nshould be based on terminal capabilities.\n\n\n\nYuri\n\n\n"},{"id":"425654","messageId":"20210527045628.uvesihyhtqrfyfae@tb-raspi4","threadId":"55777","inReplyTo":"6318ccec-ec96-91a8-fd65-85daf4a9a22b@rawbw.com","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2021-05-27T04:56:28Z","receivedAt":"2021-05-27T04:56:48Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Wed, May 26, 2021 at 04:41:38PM -0700, Yuri wrote:\n> On 5/26/21 4:32 PM, Junio C Hamano wrote:\n> > \"git config core.quotepath no\"?\n>\n>\n> I didn't have the 'core.quotepath' value set. 'git config core.quotepath no'\n> changed the behavior to no quoting.\n>\n> So it looks like the default value of 'core.quotepath' is incorrect: it\n> should be based on terminal capabilities.\n>\n\nThis are 2 different things.\nIf you are in a project where only ASCII names are allowed (for whatever reason),\nyou may want `git config core.quotepath no`, regardless what the terminal can do.\n\n(Beside that, are ther terminals that don't handle UTF-8 these days?)\n\nAny, if you prefer UTF-8 as a default,\n\ngit config --global core.quotepath yes\n\nis your friend (like mine)\n\n"},{"id":"425676","messageId":"YK+mWZP+sl3zXECx@coredump.intra.peff.net","threadId":"55777","inReplyTo":"20210527045628.uvesihyhtqrfyfae@tb-raspi4","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-05-27T14:02:01Z","receivedAt":"2021-05-27T14:02:05Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 27, 2021 at 06:56:28AM +0200, Torsten Bögershausen wrote:\n\n> On Wed, May 26, 2021 at 04:41:38PM -0700, Yuri wrote:\n> > On 5/26/21 4:32 PM, Junio C Hamano wrote:\n> > > \"git config core.quotepath no\"?\n> >\n> >\n> > I didn't have the 'core.quotepath' value set. 'git config core.quotepath no'\n> > changed the behavior to no quoting.\n> >\n> > So it looks like the default value of 'core.quotepath' is incorrect: it\n> > should be based on terminal capabilities.\n> >\n> \n> This are 2 different things.\n> If you are in a project where only ASCII names are allowed (for whatever reason),\n> you may want `git config core.quotepath no`, regardless what the terminal can do.\n> \n> (Beside that, are ther terminals that don't handle UTF-8 these days?)\n\nI don't think core.quotepath is just about UTF-8. It is agnostic to the\nencoding of the paths, so it is really a question of whether to just\npass through bytes with the high bit set.\n\nSo I think the more accurate question is: do the paths in your\nrepositories generally contain bytes that your terminal can interpret\nsensibly?  I'd guess the answer is usually yes, even if you are using\nlatin1 or similar (or else \"ls\" would show you mojibake, too).\n\nBut there's a follow-on, too: do all the other things which consume\nquoted path output likewise handle it? Setting core.quotepath will\nimpact all parts of Git, including plumbing. So a script that parses\ndiff-tree output, for example, will see a difference.\n\nI'd guess that most text-processing tools these days are reasonably\nhappy with high-bit chars. But if we were to flip the default, we might\nsee regressions with:\n\n  - very old / obscure systems (I'd guess even old versions of GNU tools\n    are good, but who knows what Solaris sed will do)\n\n  - some scripting languages (like perl and ruby) have internal strings\n    that are encoding-aware, and so they are picky about reading\n    high-bit input from a descriptor, especially if it isn't utf8.\n    The fix is usually easy-ish, but may be a surprise for some folks\n    (OTOH, I can imagine it fixes bugs in sloppily-written scripts which\n    did not anticipate the incoming filenames being quoted ;) ).\n\nAs Git is used more and more internationally, I suspect the value of\ndefaulting core.quotepath=no increases. And as time goes on and people\ntend to standardize on utf8-aware tools and environments, the risk of\ndoing so decreases. So while core.quotepath=yes was a conservative\nchoice in 2007, it might be time to look at switching.\n\n> Any, if you prefer UTF-8 as a default,\n> \n> git config --global core.quotepath yes\n> \n> is your friend (like mine)\n\nJust a nit/clarification for other readers, but I think you have yes/no\nflipped here and earlier in your message.\n\n-Peff\n"},{"id":"425708","messageId":"4dd22f16-72f0-a28a-8be0-aec622acf0d3@rawbw.com","threadId":"55777","inReplyTo":"YK+mWZP+sl3zXECx@coredump.intra.peff.net","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Yuri","fromEmail":"yuri@rawbw.com","sentAt":"2021-05-27T20:50:58Z","receivedAt":"2021-05-27T20:51:05Z","isPatch":false,"sender":{"key":"yuri@rawbw.com","avatar":null},"body":"It's not clear from the conversation if git reads terminal capabilities \nat all.\n\n\nBut the default behavior, without any options set, should be to read \nterminal capabilities, and write non-ASCII characters verbatim when \nterminal supports this and escape them when terminal doesn't support them.\n\nCurrent default behavior appears to be to always escape non-ASCII \ncharacters.\n\n\nThen options can change this basic behavior according to user's choice.\n\n\n\nYuri\n\n\n"},{"id":"425735","messageId":"50e2780a-21f3-499f-7960-76bf24f550f0@gmail.com","threadId":"55777","inReplyTo":"4dd22f16-72f0-a28a-8be0-aec622acf0d3@rawbw.com","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-05-28T04:39:03Z","receivedAt":"2021-05-28T04:39:09Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 28/05/21 03.50, Yuri wrote:\n> It's not clear from the conversation if git reads terminal capabilities \n> at all.\n> \n> \n> But the default behavior, without any options set, should be to read \n> terminal capabilities, and write non-ASCII characters verbatim when \n> terminal supports this and escape them when terminal doesn't support them.\n> \n> Current default behavior appears to be to always escape non-ASCII \n> characters.\n\nSo the current default is only supports ASCII, and escape other \ncharacters, right?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"425736","messageId":"6fef4b1e-1ec7-b697-c311-59caf6408b29@rawbw.com","threadId":"55777","inReplyTo":"50e2780a-21f3-499f-7960-76bf24f550f0@gmail.com","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Yuri","fromEmail":"yuri@rawbw.com","sentAt":"2021-05-28T04:45:53Z","receivedAt":"2021-05-28T04:46:09Z","isPatch":false,"sender":{"key":"yuri@rawbw.com","avatar":null},"body":"On 5/27/21 9:39 PM, Bagas Sanjaya wrote:\n> So the current default is only supports ASCII, and escape other \n> characters, right?\n\n\nIt appears this way.\n\n\nYuri\n\n"},{"id":"425877","messageId":"20210529092752.kifzqt3haddzgsob@tb-raspi4","threadId":"55777","inReplyTo":"6fef4b1e-1ec7-b697-c311-59caf6408b29@rawbw.com","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2021-05-29T09:27:52Z","receivedAt":"2021-05-29T09:28:15Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Thu, May 27, 2021 at 09:45:53PM -0700, Yuri wrote:\n> On 5/27/21 9:39 PM, Bagas Sanjaya wrote:\n> > So the current default is only supports ASCII, and escape other\n> > characters, right?\n>\n>\n> It appears this way.\n>\n\nYes, that is how it is.\n\nAfter reading the wiki here:\n\nhttps://wiki.gentoo.org/wiki/UTF-8\n\n(There are many other web pages as well)\n\nI am not sure that there is a reliable way for Git to detect, if the\nterminal is capable of handling UTF-8.\nThis should work reliable under Linux, Windows, Mac and all the supported\nUnix-ish platforms.\n\nBeside that, the outputs of git commands can be feed into other programs\nvia a pipe usning  \"|\"  on the command line or redirectet to a file.\n\nAnd what is a terminal ?\nWe need to consider that we run programs like `less` or `more´ which\nneed to be UTF-8 compatble.\n\nMost of them are probably UTF-8 compliant (and LANG is set to xx.UTF-8)\nthese days.\nAnd most repositories have been feed with filenames encoded in  UTF-8 as well.\n\nHaving said that, the default could be switched some day in the future.\nBefore that is \"save\", there may be a transition phase,\nwhere users are warned that the default may change.\nScripts calling git need to use `git -c core.quotepath=yes`, or no,\nwhatever input they expect.\n\nSorry for the longish answer.\nChanging one thing for some users may effect hundrets, thousands or millions\nof other users later, cause surprises, need debugging and fixing effort.\n\nDoes someone wants to come up with a patch that anounces a possible change ?\n\n\n\n"},{"id":"425941","messageId":"YLQHPu0bIy4qHEWP@coredump.intra.peff.net","threadId":"55777","inReplyTo":"20210529092752.kifzqt3haddzgsob@tb-raspi4","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-05-30T21:44:30Z","receivedAt":"2021-05-30T21:44:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, May 29, 2021 at 11:27:52AM +0200, Torsten Bögershausen wrote:\n\n> I am not sure that there is a reliable way for Git to detect, if the\n> terminal is capable of handling UTF-8.\n> This should work reliable under Linux, Windows, Mac and all the supported\n> Unix-ish platforms.\n\nYeah, I'm not sure how such a check would be done. On most Linux systems\nI've seen, $LANG will mention \"en_US.UTF-8\" or similar. But I've no idea\nhow portable that convention is, not to mention that people may have\nmore complex setups anyway (e.g., not setting $LANG but setting some of\nLC_*).\n\nBut more importantly, this is not even a UTF-8 problem. It is \"can your\nterminal do something sensible with high-bit characters in filenames of\nyour repositories\". We don't know the encoding of those filenames (and\nyou may even have a mix).\n\n(And likewise \"terminal\" here is really \"whatever consumes Git's output,\nbe it the terminal or some program you've piped to).\n\n> Having said that, the default could be switched some day in the future.\n> Before that is \"save\", there may be a transition phase,\n> where users are warned that the default may change.\n> Scripts calling git need to use `git -c core.quotepath=yes`, or no,\n> whatever input they expect.\n\nYes. If we're going to do anything, I think it would be to say \"most\nterminals and programs deal with high-bit characters OK these days, so\nswitching the default is more likely to fix things than break them\".\n\nI suspect most scripts would be OK either way. They need to handle\nmaybe-quoted filenames already, so it is really just a question of\nwhether the consuming program is OK with the high bits. If so, we could\nprobably get away with just a mention in the release notes, rather than\nan annoying transition phase (which is likely to simply confuse most\nusers, who are unaware of the issue entirely).\n\nBut I'd feel more confident if whoever proposes such a change does some\nresearch on how piping such names into common tools and scripting\nlanguages works (both for utf8 and non-utf8 names).\n\n-Peff\n"},{"id":"425943","messageId":"481fbc88-3e7d-6b00-0997-1a80b87a637d@rawbw.com","threadId":"55777","inReplyTo":"YLQHPu0bIy4qHEWP@coredump.intra.peff.net","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Yuri","fromEmail":"yuri@rawbw.com","sentAt":"2021-05-30T21:55:59Z","receivedAt":"2021-05-30T21:56:13Z","isPatch":false,"sender":{"key":"yuri@rawbw.com","avatar":null},"body":"On 5/30/21 2:44 PM, Jeff King wrote:\n> Yeah, I'm not sure how such a check would be done. On most Linux systems\n> I've seen, $LANG will mention \"en_US.UTF-8\" or similar. But I've no idea\n> how portable that convention is, not to mention that people may have\n> more complex setups anyway (e.g., not setting $LANG but setting some of\n> LC_*).\n\n\nWhen 'locale charmap' prints 'UTF-8' the terminal can be assumed to be \nable to accept UTF-8 characters.\n\n'locale charmap', I think, determines this only based on environment \nvariables.\n\n\n\nYuri\n\n"},{"id":"425946","messageId":"xmqqmtsb2757.fsf@gitster.g","threadId":"55777","inReplyTo":"YLQHPu0bIy4qHEWP@coredump.intra.peff.net","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-30T22:23:16Z","receivedAt":"2021-05-30T22:23:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Yes. If we're going to do anything, I think it would be to say \"most\n> terminals and programs deal with high-bit characters OK these days, so\n> switching the default is more likely to fix things than break them\".\n\nAmen to that.  The conservative setting was from v1.5.3 days in\n2007, and it would be highly disappointing if the situation hasn't\nchanged in the 14 years.\n"},{"id":"425950","messageId":"c1afaf46-0425-110b-6303-f68d3f994b7f@gmail.com","threadId":"55777","inReplyTo":"481fbc88-3e7d-6b00-0997-1a80b87a637d@rawbw.com","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Thomas Guyot","fromEmail":"tguyot@gmail.com","sentAt":"2021-05-31T01:14:35Z","receivedAt":"2021-05-31T01:14:43Z","isPatch":false,"sender":{"key":"tguyot@gmail.com","avatar":"https://avatars.githubusercontent.com/u/403890?v=4"},"body":"On 2021-05-30 17:55, Yuri wrote:\n> On 5/30/21 2:44 PM, Jeff King wrote:\n>> Yeah, I'm not sure how such a check would be done. On most Linux systems\n>> I've seen, $LANG will mention \"en_US.UTF-8\" or similar. But I've no idea\n>> how portable that convention is, not to mention that people may have\n>> more complex setups anyway (e.g., not setting $LANG but setting some of\n>> LC_*).\n> \n> \n> When 'locale charmap' prints 'UTF-8' the terminal can be assumed to be\n> able to accept UTF-8 characters.\n> \n> 'locale charmap', I think, determines this only based on environment\n> variables.\n> \n\nHi Yuri,\n\nEven if the terminal supports UTF8, will it print it properly? The font\nused could have no or minimal utf8 support. Even when it's supported,\nsome characters might look alike and this could have undesired\nconsequences (ex accidentally switching from a normal space to a\nnon-break space while renaming a file that has spaces...).\n\nI believe repos with utf8 files are rare enough and it could be left to\nthe user to select whenever to use utf8 or not... An option like \"auto\"\nor \"detect\" could make it automatic but I'm not convinced it should be\nthe default.\n\n\nOh, and looking at \"locale charmap\", it doesn't check the terminal\ncapabilities at all - it just prints the charmap based on LC_ALL or\nLC_CTYPE value, or default if they're unset. It doesn't mater what\nterminal you're on...\n\nRegards,\n\n--\nThomas\n"},{"id":"425954","messageId":"ec3adb99-ba20-8db6-4bff-eae7524e6361@gmail.com","threadId":"55777","inReplyTo":"c1afaf46-0425-110b-6303-f68d3f994b7f@gmail.com","subject":"Re: [BUG REPORT] File names that contain UTF8 characters are unnecessarily escaped in 'git status .' messages","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-05-31T03:35:49Z","receivedAt":"2021-05-31T03:36:04Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 31/05/21 08.14, Thomas Guyot wrote:\n> Even if the terminal supports UTF8, will it print it properly? The font\n> used could have no or minimal utf8 support. Even when it's supported,\n> some characters might look alike and this could have undesired\n> consequences (ex accidentally switching from a normal space to a\n> non-break space while renaming a file that has spaces...).\n\nOn Linux distributions, Noto and DejaVu fonts are often installed as \ndefault fonts, because Noto has almost complete Unicode coverage and \nDejaVu Mono become goto monospace font.\n\nAnd yeah, we steer clear of using non-monospace fonts (either serif or \nsans serif), because many terminal-only programs depend on text \nalignment which often can be achieved only with monospace fonts, and \nreading texts on terminal screen is vertical-oriented as opposed to \nhorizontal-oriented texts like books.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"}]}