{"thread":{"id":"18731","subject":"non-ascii filenames issue","startedAt":"2009-04-05T09:36:40Z","lastAt":"2009-04-07T08:26:14Z","messageCount":11,"participants":["Gregory Petrosyan","Teemu Likonen","John Tapsell","Jay Soffian","Junio C Hamano","Peter Krefting","Johannes Schindelin","Dmitry Potapov","demerphq"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"110391","messageId":"20090405093640.GA9803@home","threadId":"18731","inReplyTo":null,"subject":"non-ascii filenames issue","fromName":"Gregory Petrosyan","fromEmail":"gregory.petrosyan@gmail.com","sentAt":"2009-04-05T09:36:40Z","receivedAt":"2009-04-05T09:36:40Z","isPatch":false,"sender":{"key":"gregory.petrosyan@gmail.com","avatar":null},"body":"gregory@home:~$ git --version\ngit version 1.6.2.2.404.ge96f3\ngregory@home:~$ mkdir git-test\ngregory@home:~$ cd git-test\ngregory@home:~/git-test$ touch файл\ngregory@home:~/git-test$ ls -a\n. .. файл\ngregory@home:~/git-test$ git init\nInitialized empty Git repository in /home/gregory/git-test/.git/\ngregory@home:~/git-test$ git add .\ngregory@home:~/git-test$ git status\n# On branch master\n#\n# Initial commit\n#\n# Changes to be committed:\n# (use \"git rm --cached ...\" to unstage)\n#\n# new file: \"\\321\\204\\320\\260\\320\\271\\320\\273\"\n#\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \"файл\" should be here instead\n\nThis is on Ubuntu Jaunty beta, with latest git built from source.\nPlease CC me, I am not subscribed.\n\n\tGregory\n"},{"id":"110399","messageId":"87ab6v2zor.fsf@iki.fi","threadId":"18731","inReplyTo":"20090405093640.GA9803@home","subject":"Re: non-ascii filenames issue","fromName":"Teemu Likonen","fromEmail":"tlikonen@iki.fi","sentAt":"2009-04-05T09:54:28Z","receivedAt":"2009-04-05T09:54:28Z","isPatch":false,"sender":{"key":"tlikonen@iki.fi","avatar":null},"body":"On 2009-04-05 13:36 (+0400), Gregory Petrosyan wrote:\n\n> # Changes to be committed:\n> # (use \"git rm --cached ...\" to unstage)\n> #\n> # new file: \"\\321\\204\\320\\260\\320\\271\\320\\273\"\n> #\n> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> \"файл\" should be here instead\n\nIt can be fixed with command:\n\n git config --global core.quotepath false\n"},{"id":"110400","messageId":"20090405100127.GA12126@home","threadId":"18731","inReplyTo":"87ab6v2zor.fsf@iki.fi","subject":"Re: non-ascii filenames issue","fromName":"Gregory Petrosyan","fromEmail":"gregory.petrosyan@gmail.com","sentAt":"2009-04-05T10:01:27Z","receivedAt":"2009-04-05T10:01:27Z","isPatch":false,"sender":{"key":"gregory.petrosyan@gmail.com","avatar":null},"body":"On Sun, Apr 05, 2009 at 12:54:28PM +0300, Teemu Likonen wrote:\n> On 2009-04-05 13:36 (+0400), Gregory Petrosyan wrote:\n> \n> > # Changes to be committed:\n> > # (use \"git rm --cached ...\" to unstage)\n> > #\n> > # new file: \"\\321\\204\\320\\260\\320\\271\\320\\273\"\n> > #\n> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> > \"файл\" should be here instead\n> \n> It can be fixed with command:\n> \n> git config --global core.quotepath false\n\nThanks! That works. Does it make sence to set it to \"false\" by default?\n\n\tGregory\n"},{"id":"110405","messageId":"43d8ce650904050351p72590d52l8861b3901f95201a@mail.gmail.com","threadId":"18731","inReplyTo":"20090405100127.GA12126@home","subject":"Re: non-ascii filenames issue","fromName":"John Tapsell","fromEmail":"johnflux@gmail.com","sentAt":"2009-04-05T10:51:10Z","receivedAt":"2009-04-05T10:51:10Z","isPatch":false,"sender":{"key":"johnflux@gmail.com","avatar":"https://gravatar.com/avatar/25f70d4c0f96396b84a2e34bcd9bdc233462c7b4be29b5fdca8266fc53f30b0c?d=mp&s=160"},"body":"2009/4/5 Gregory Petrosyan :\n> On Sun, Apr 05, 2009 at 12:54:28PM +0300, Teemu Likonen wrote:\n>> On 2009-04-05 13:36 (+0400), Gregory Petrosyan wrote:\n>>\n>> > # Changes to be committed:\n>> > #   (use \"git rm --cached ...\" to unstage)\n>> > #\n>> > #  new file:   \"\\321\\204\\320\\260\\320\\271\\320\\273\"\n>> > #\n>> >                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n>> >                  \"файл\" should be here instead\n>>\n>> It can be fixed with command:\n>>\n>>     git config --global core.quotepath false\n>\n> Thanks! That works. Does it make sence to set it to \"false\" by default?\n\nUnfortunately not, because for some absolutely crazy reason, there is\nno way at all to tell what encoding the string is in. It never\noccured to anyone that it might actually be useful to be able to read\nthe filename in an unambiguous way. The result is this sort of mess.\nJust wait until you try to checkout that file on a new filesystem with\na different encoding. Or try to checkout that file in Windows. It's\nlike git decided to step backwards 30 years.\n\nJohn\n"},{"id":"110422","messageId":"76718490904050923j105e383dsf650afa0a0687858@mail.gmail.com","threadId":"18731","inReplyTo":"43d8ce650904050351p72590d52l8861b3901f95201a@mail.gmail.com","subject":"Re: non-ascii filenames issue","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2009-04-05T16:23:35Z","receivedAt":"2009-04-05T16:23:35Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"On Sun, Apr 5, 2009 at 6:51 AM, John Tapsell wrote:\n> Unfortunately not, because for some absolutely crazy reason\n\nBzzt. http://article.gmane.org/gmane.comp.version-control.git/50830\n\nAnd, as always, patches welcomed.\n\nj.\n"},{"id":"110434","messageId":"7vfxgmrjb7.fsf@gitster.siamese.dyndns.org","threadId":"18731","inReplyTo":"76718490904050923j105e383dsf650afa0a0687858@mail.gmail.com","subject":"Re: non-ascii filenames issue","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-04-05T19:29:00Z","receivedAt":"2009-04-05T19:29:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jay Soffian writes:\n\n> On Sun, Apr 5, 2009 at 6:51 AM, John Tapsell wrote:\n>> Unfortunately not, because for some absolutely crazy reason\n>\n> Bzzt. http://article.gmane.org/gmane.comp.version-control.git/50830\n\nI do not think the message gives enough information on the issue, as \"a\npathname is a slash separated sequence of path components terminated with\na NUL, and a path component is an uninterpreted sequence of bytes\nexcluding NUL and slash\" is simply a UNIX tradition the original git\ndesign took as _given_, so the \"some absolutely crazy reason\" comment does\nnot even deserve refuting.\n\nThere is _no_ reason, crazy or otherwise. If you start from \"a pathname\nis an uninterpreted sequence of bytes\" tradition, it is a design parameter\nand \"how things are\", and you simply do not argue with them. And the\nmessage you quoted doesn't, either.\n\n\tSide note: I am not saying that we should not ever change that\n\tparticular design parameter. I am just explaining why 50830 is\n\tnot a good counterargument to quote against the \"some absolutely\n\tcrazy reason\" accusation.\n\n> And, as always, patches welcomed.\n\nBefore patches, you need a sound design and justification.\n\nAt least you need to consider the following (the early ones are easier):\n\n - Do we unify them to some canonical encoding internally and do the\n matching in the canonical space? What's the internal representation\n (presumably UTF-8)?\n\n - How should a user tell the pathname conversion rules between the\n internal repreasentation and the filesystem representation to git? A\n config variable per a repository?\n\n - How should this interact with patch+apply dataflow (including \"rebase\"\n without -i/-m)? Should pathnames in diffs be in canonical form?\n\n - How should this interact with case challenged and/or unicode corrupting\n filesystems such as NTFS and HFSplus whose creat(), readdir(), and\n stat() contradict with each other?\n\n - What should happen when the pathname in the canonical representation\n recorded in the history cannot be externalized on a particular\n filesystem? Does it gracefully degenerate and give some escape hatch,\n and if so how?\n"},{"id":"110446","messageId":"76718490904051322u48f22143s6744d62a37fa55ec@mail.gmail.com","threadId":"18731","inReplyTo":"7vfxgmrjb7.fsf@gitster.siamese.dyndns.org","subject":"Re: non-ascii filenames issue","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2009-04-05T20:22:23Z","receivedAt":"2009-04-05T20:22:23Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"> I do not think the message gives enough information on the issue\n\nOf course you are correct. I was perturbed by John's message, but your\nthoughtful reply is much more beneficial than my silly link. Thank you\nfor providing the level-headed response as always.\n\nj.\n"},{"id":"110523","messageId":"alpine.DEB.2.00.0904060823400.21376@ds9.cixit.se","threadId":"18731","inReplyTo":"43d8ce650904050351p72590d52l8861b3901f95201a@mail.gmail.com","subject":"Re: non-ascii filenames issue","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-04-06T07:28:21Z","receivedAt":"2009-04-06T07:28:21Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"John Tapsell:\n\n> Unfortunately not, because for some absolutely crazy reason, there is no \n> way at all to tell what encoding the string is in. It never occured to \n> anyone that it might actually be useful to be able to read the filename in \n> an unambiguous way.\n\nIt comes from the Unix tradition, unfortunately, that file names are just a \nstream of bytes, instead of a stream of characters mapped to a byte \nsequence. The \"stream of bytes\" think worked back when everyone used ASCII, \nbut as soon as other character encodings were used (i.e back in the 1970s or \nso), that assumption broke.\n\n> The result is this sort of mess. Just wait until you try to checkout that \n> file on a new filesystem with a different encoding. Or try to checkout \n> that file in Windows. It's like git decided to step backwards 30 years.\n\nSince most people on Linux nowadays probably are running in a UTF-8-based \nlocale, I tried introducing some (very incomplete) patches for the Windows \nport to make this assumption, to allow Windows users to make use of \nnon-ASCII file names (Windows uses Unicode strings for file names). Mac OS \nuses (semi-decomposed) UTF-8 strings, so it should also be able to make use \nof this.\n\nUnfortunately, there seems to be quite some resistance towards deciding on \na platform- and language-independent way of storing file names in Git, but \nrather just going the \"Unix\" way and making it someone elses problem. I find \nthis a bit sad.\n\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"110545","messageId":"alpine.DEB.1.00.0904061109330.10279@pacific.mpi-cbg.de","threadId":"18731","inReplyTo":"alpine.DEB.2.00.0904060823400.21376@ds9.cixit.se","subject":"Re: non-ascii filenames issue","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-04-06T09:12:35Z","receivedAt":"2009-04-06T09:12:35Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 6 Apr 2009, Peter Krefting wrote:\n\n> It comes from the Unix tradition, unfortunately, that file names are \n> just a stream of bytes, instead of a stream of characters mapped to a \n> byte sequence.\n\nHow is that different from .txt not having a defined locale?\n\nReally, please, do not add to the non-information.\n\n> Since most people on Linux nowadays probably are running in a \n> UTF-8-based locale, I tried introducing some (very incomplete) patches \n> for the Windows port to make this assumption, to allow Windows users to \n> make use of non-ASCII file names (Windows uses Unicode strings for file \n> names). Mac OS uses (semi-decomposed) UTF-8 strings, so it should also \n> be able to make use of this.\n\nMost Russian programmers I know do not run in a UTF-8 locale.\n\n> Unfortunately, there seems to be quite some resistance towards deciding \n> on a platform- and language-independent way of storing file names in \n> Git, but rather just going the \"Unix\" way and making it someone elses \n> problem. I find this a bit sad.\n\nI find it a bit unfair that you say that, after many people participated \nin that very informative thread, and after I tried to work with you \npersonally on getting the stuff into 4msysgit.git.\n\nActually, not just only a bit.\n\nCiao,\nDscho\n"},{"id":"110638","messageId":"37fcd2780904061533o73b393b4j1358ea6259fbd33a@mail.gmail.com","threadId":"18731","inReplyTo":"alpine.DEB.1.00.0904061109330.10279@pacific.mpi-cbg.de","subject":"Re: non-ascii filenames issue","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-04-06T22:33:36Z","receivedAt":"2009-04-06T22:33:36Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Mon, Apr 06, 2009 at 11:12:35AM +0200, Johannes Schindelin wrote:\n>\n> Most Russian programmers I know do not run in a UTF-8 locale.\n\nActually, on Linux, people gradually switching to UTF-8 from koi8-r,\nbut on Windows MSCRT does not support UTF-8, so you do have much choice\nhere but to use Windows-1251. BTW, the upcoming Cygwin 1.7 is going to\nhave UTF-8 as the default locale. So, IMHO, UTF-8 is the only reasonable\nchoice for internal file name representation...\n\nDmitry\n"},{"id":"110686","messageId":"9b18b3110904070126i354fc100l69b6ce3c9cd19d49@mail.gmail.com","threadId":"18731","inReplyTo":"alpine.DEB.2.00.0904060823400.21376@ds9.cixit.se","subject":"Re: non-ascii filenames issue","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2009-04-07T08:26:14Z","receivedAt":"2009-04-07T08:26:14Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"2009/4/6 Peter Krefting :\n> John Tapsell:\n>\n>> Unfortunately not, because for some absolutely crazy reason, there is no\n>> way at all to tell what encoding the string is in. It never occured to\n>> anyone that it might actually be useful to be able to read the filename in\n>> an unambiguous way.\n>\n> It comes from the Unix tradition, unfortunately, that file names are just a\n> stream of bytes, instead of a stream of characters mapped to a byte\n> sequence. The \"stream of bytes\" think worked back when everyone used ASCII,\n> but as soon as other character encodings were used (i.e back in the 1970s or\n> so), that assumption broke.\n\nThose interested in this subject may find the following document on\nthe creation of utf8 interesting.\n\nhttp://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt\n\ncheers,\nYves\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"}]}