{"thread":{"id":"15677","subject":"git log and utf-u in filenames","startedAt":"2008-09-25T21:50:02Z","lastAt":"2008-09-27T08:37:12Z","messageCount":8,"participants":["Joey Hess","Jakub Narebski","Alex Riesen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"91639","messageId":"20080925215002.GA30810@kodama.kitenet.net","threadId":"15677","inReplyTo":null,"subject":"git log and utf-u in filenames","fromName":"Joey Hess","fromEmail":"joey@kitenet.net","sentAt":"2008-09-25T21:50:02Z","receivedAt":"2008-09-25T21:50:02Z","isPatch":false,"sender":{"key":"joey@kitenet.net","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Git, particularly git-log seems to not display utf-8 characters in filenames,\ninstead showing an escaped representation. On the other hand, commit messages,\nas git-log(1) notes, are assumed to be utf-8, and the same utf-8 character\nused in a commit message is not escaped, and displays ok.\n\nCan anyone point me at the documentation for this utf-8 filename escaping,\nassuming it's not a bug? And did earlier versions of git (circa 2006) perhaps\nnot do that escaping? I have code in ikiwiki that apparently used to work, but\nis certianly not working with current git, due to this escaping.\n\nHere's an example of the inconsistent handling of the same utf-8 character\n(\"ö\") in commit messages and filenames.\n\njoey@kodama:~/tmp>mkdir utf8; cd utf8; git-init\nInitialized empty Git repository in /home/joey/tmp/utf8/.git/\njoey@kodama:~/tmp/utf8>echo hi > ö\njoey@kodama:~/tmp/utf8>git add ö; git commit -m 'adding file: ö'\nCreated initial commit ee7d809: adding file: ö\n 1 files changed, 1 insertions(+), 0 deletions(-)\n create mode 100644 \"\\303\\266\"\njoey@kodama:~/tmp/utf-8>git log --stat\ncommit ee7d809d1811b1e1ad485ce3e7274316257029ae\nAuthor: Joey Hess \nDate: Thu Sep 25 17:34:10 2008 -0400\n \n adding file: ö\n \n \"\\303\\266\" | 1 +\n 1 files changed, 1 insertions(+), 0 deletions(-)\n\n-- \nsee shy jo\n"},{"id":"91642","messageId":"20080925224354.GA6072@kodama.kitenet.net","threadId":"15677","inReplyTo":"20080925215002.GA30810@kodama.kitenet.net","subject":"Re: git log and utf-u in filenames","fromName":"Joey Hess","fromEmail":"joey@kitenet.net","sentAt":"2008-09-25T22:43:54Z","receivedAt":"2008-09-25T22:43:54Z","isPatch":false,"sender":{"key":"joey@kitenet.net","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Joey Hess wrote:\n> And did earlier versions of git (circa 2006) perhaps\n> not do that escaping? I have code in ikiwiki that apparently used to work, but\n> is certianly not working with current git, due to this escaping.\n\nNo, I guess it's always done that, perhaps something broke on my side\nin the meantime.\n\nBut it doesn't seem right somehow that gitweb, ikiwiki, and seemingly\nany other program that needs to look at git log / commits and figure out\nwhat filename is being changed needs to include their own nasty code[1] to\nconvert the escaped characters back to normal characters.\n\nAnd it seems that anyone who uses a lot of utf-8 in filenames would shortly\nget tired of git commit, git log, etc displaying obfuscated versions of their\nfilenames.\n\nI'm sure it makes sense to use this format internally in git to represent\nfilenames, to avoid needing to worry about encoding issues. But it's a shame\nthat that internal detail is exposed so that everything around git has to\nworry about it.\n\nWould making git-log and git-commit display de-escaped filenames be likely\nto break something?\n\n-- \nsee shy jo\n\n[1] Such as this from gitweb:\n\n# git may return quoted and escaped filenames\nsub unquote {\n my $str = shift;\n\n sub unq {\n my $seq = shift;\n my %es = ( # character escape codes, aka escape sequences\n 't' => \"\\t\", # tab (HT, TAB)\n 'n' => \"\\n\", # newline (NL)\n 'r' => \"\\r\", # return (CR)\n 'f' => \"\\f\", # form feed (FF)\n 'b' => \"\\b\", # backspace (BS)\n 'a' => \"\\a\", # alarm (bell) (BEL)\n 'e' => \"\\e\", # escape (ESC)\n 'v' => \"\\013\", # vertical tab (VT)\n );\n\n if ($seq =~ m/^[0-7]{1,3}$/) {\n # octal char sequence\n return chr(oct($seq));\n } elsif (exists $es{$seq}) {\n # C escape sequence, aka character escape code\n return $es{$seq};\n }\n # quoted ordinary character\n return $seq;\n }\n\n if ($str =~ m/^\"(.*)\"$/) {\n # needs unquoting\n $str = $1;\n $str =~ s/\\\\([^0-7]|[0-7]{1,3})/unq($1)/eg;\n }\n return $str;\n}\n"},{"id":"91644","messageId":"m3k5cz3jp4.fsf@localhost.localdomain","threadId":"15677","inReplyTo":"20080925215002.GA30810@kodama.kitenet.net","subject":"Re: git log and utf-u in filenames","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-09-25T23:11:54Z","receivedAt":"2008-09-25T23:11:54Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Joey Hess writes:\n\n> Git, particularly git-log seems to not display utf-8 characters in filenames,\n> instead showing an escaped representation. On the other hand, commit messages,\n> as git-log(1) notes, are assumed to be utf-8, and the same utf-8 character\n> used in a commit message is not escaped, and displays ok.\n> \n> Can anyone point me at the documentation for this utf-8 filename\n> escaping, assuming it's not a bug? And did earlier versions of git\n> (circa 2006) perhaps not do that escaping? I have code in ikiwiki\n> that apparently used to work, but is certianly not working with\n> current git, due to this escaping.\n\nErr... it always worked like this, mainly I think to have 7bit safe\npatches for sending via email. Now in the time of 8bit transfer\nand using single utf-8 encoding instead of multitude of different\nfilesystem encodings, you can set core.quotepath to false, although\nthis would eliminate only octal escaping for >127 ASCII characters;\nTAB, CR etc. would still be quoted (and they have to be).\n\ngitconfig(7):\n\n core.quotepath::\n The commands that output paths (e.g. 'ls-files',\n 'diff'), when not given the `-z` option, will quote\n \"unusual\" characters in the pathname by enclosing the\n pathname in a double-quote pair and with backslashes the\n same way strings in C source code are quoted. If this\n variable is set to false, the bytes higher than 0x80 are\n not quoted but output as verbatim. Note that double\n quote, backslash and control characters are always\n quoted without `-z` regardless of the setting of this\n variable.\n\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"91645","messageId":"m3fxnn3ji6.fsf@localhost.localdomain","threadId":"15677","inReplyTo":"20080925224354.GA6072@kodama.kitenet.net","subject":"Re: git log and utf-u in filenames","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-09-25T23:15:58Z","receivedAt":"2008-09-25T23:15:58Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Joey Hess writes:\n\n> Joey Hess wrote:\n\n> > And did earlier versions of git (circa 2006) perhaps not do that\n> > escaping? I have code in ikiwiki that apparently used to work, but\n> > is certianly not working with current git, due to this escaping.\n> \n> No, I guess it's always done that, perhaps something broke on my side\n> in the meantime.\n> \n> But it doesn't seem right somehow that gitweb, ikiwiki, and seemingly\n> any other program that needs to look at git log / commits and figure out\n> what filename is being changed needs to include their own nasty code[1] to\n> convert the escaped characters back to normal characters.\n\nWell, in gitweb we could use '-z' option for git-diff-tree and git-ls-tree,\nbut it has its disadvantages, like having to do actual parsing record after\nrecord instead of simplys splitting outout on end of line (\"\\n\") characters.\n\n> Would making git-log and git-commit display de-escaped filenames be likely\n> to break something?\n\ncore.quotepath limits filename escaping, but you still _have_ to quote\n\"\\n\", \"\\t\", and of course '\"' and '\\', if you want for filename to have\nin single line in text format.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"91662","messageId":"20080926063311.GA4939@blimp.localhost","threadId":"15677","inReplyTo":"m3fxnn3ji6.fsf@localhost.localdomain","subject":"Re: git log and utf-u in filenames","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2008-09-26T06:33:11Z","receivedAt":"2008-09-26T06:33:11Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Jakub Narebski, Fri, Sep 26, 2008 01:15:58 +0200:\n> \n> Well, in gitweb we could use '-z' option for git-diff-tree and git-ls-tree,\n> but it has its disadvantages, like having to do actual parsing record after\n> record instead of simplys splitting outout on end of line (\"\\n\") characters.\n> \n\nHow about simply splitting output on end of line (\"\\0\" NUL) characters?\nThe \"\\n\" NL you refer to is just as EOR as NUL.\n"},{"id":"91665","messageId":"200809260931.21883.jnareb@gmail.com","threadId":"15677","inReplyTo":"20080926063311.GA4939@blimp.localhost","subject":"Re: git log and utf-u in filenames","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-09-26T07:31:20Z","receivedAt":"2008-09-26T07:31:20Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Fri, 26 Sep 2008, Alex Riesen wrote:\n> Jakub Narebski, Fri, Sep 26, 2008 01:15:58 +0200:\n> > \n> > Well, in gitweb we could use '-z' option for git-diff-tree and git-ls-tree,\n> > but it has its disadvantages, like having to do actual parsing record after\n> > record instead of simplys splitting outout on end of line (\"\\n\") characters.\n> > \n> \n> How about simply splitting output on end of line (\"\\0\" NUL) characters?\n> The \"\\n\" NL you refer to is just as EOR as NUL.\n\nDoesn't work for \"git diff-tree -z [...]\" output. When there is rename\nor copy detected, NUL is used as separator between fields (beetween\nsource and destination unquoted filename), not only between records:\n\n git diff-tree\n .... TAB LF\n\n git diff-tree -z\n .... NUL NUL\n\n-- \nJakub Narebski\nPoland\n"},{"id":"91680","messageId":"81b0412b0809260649n7d7c0ccbwfde5504157687cda@mail.gmail.com","threadId":"15677","inReplyTo":"200809260931.21883.jnareb@gmail.com","subject":"Re: git log and utf-u in filenames","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2008-09-26T13:49:06Z","receivedAt":"2008-09-26T13:49:06Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"2008/9/26 Jakub Narebski :\n> On Fri, 26 Sep 2008, Alex Riesen wrote:\n>> Jakub Narebski, Fri, Sep 26, 2008 01:15:58 +0200:\n>> >\n>> > Well, in gitweb we could use '-z' option for git-diff-tree and git-ls-tree,\n>> > but it has its disadvantages, like having to do actual parsing record after\n>> > record instead of simplys splitting outout on end of line (\"\\n\") characters.\n>> >\n>>\n>> How about simply splitting output on end of line (\"\\0\" NUL) characters?\n>> The \"\\n\" NL you refer to is just as EOR as NUL.\n>\n> Doesn't work for \"git diff-tree -z [...]\" output. When there is rename\n> or copy detected, NUL is used as separator between fields (beetween\n> source and destination unquoted filename), not only between records:\n>\n> git diff-tree\n> .... TAB LF\n>\n> git diff-tree -z\n> .... NUL NUL\n>\n\nYou still have the marker (Rnnn) from pre- record and\ncan treat the next record correspondingly. Still a split, just a bit more\ncareful handling of the resulting list/array.\n"},{"id":"91733","messageId":"200809271037.13571.jnareb@gmail.com","threadId":"15677","inReplyTo":"81b0412b0809260649n7d7c0ccbwfde5504157687cda@mail.gmail.com","subject":"Re: git log and utf-u in filenames","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-09-27T08:37:12Z","receivedAt":"2008-09-27T08:37:12Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Fri, 26 Sep 2008, Alex Riesen wrote:\n> 2008/9/26 Jakub Narebski :\n\n>>> How about simply splitting output on end of line (\"\\0\" NUL) characters?\n>>> The \"\\n\" NL you refer to is just as EOR as NUL.\n>>\n>> Doesn't work for \"git diff-tree -z [...]\" output. When there is rename\n>> or copy detected, NUL is used as separator between fields (beetween\n>> source and destination unquoted filename), not only between records:\n>>\n>> git diff-tree\n>> .... TAB LF\n>>\n>> git diff-tree -z\n>> .... NUL NUL\n>>\n> \n> You still have the marker (Rnnn) from pre- record and\n> can treat the next record correspondingly. Still a split, just a bit more\n> careful handling of the resulting list/array.\n\nCurrently gitweb does something like this:\n\n open $fd, \"-|\", git_cmd(), \"diff-tree\", '-r', ...\n @difftree = <$fd>;\n close $fd;\n foreach my $line (@difftree) {\n ...\n }\n\nIf gitweb would use git-diff-tree with '-z' option, above code\nwould get more complicated, offsetting simplification of not using\nunquote() (which is already written).\n\n-- \nJakub Narebski\nPoland\n"}]}