{"thread":{"id":"16700","subject":"gitweb and unicode special characters","startedAt":"2008-12-12T18:33:03Z","lastAt":"2008-12-13T22:08:29Z","messageCount":7,"participants":["Praveen A","Jakub Narebski","Edward Z. Yang"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"97719","messageId":"3f2beab60812121033r5d41894t77acc271b7c6955c@mail.gmail.com","threadId":"16700","inReplyTo":null,"subject":"gitweb and unicode special characters","fromName":"Praveen A","fromEmail":"pravi.a@gmail.com","sentAt":"2008-12-12T18:33:03Z","receivedAt":"2008-12-12T18:33:03Z","isPatch":false,"sender":{"key":"pravi.a@gmail.com","avatar":null},"body":"Hi,\n\nGit currently does not handle unicode special characters ZWJ and ZWNJ,\nboth are heavily used in Malayalam and common in other languages\nneeding complex text layout like Sinhala and Arabic.\n\nAn example of this is shown in the commit message here\nhttp://git.savannah.gnu.org/gitweb/?p=smc.git;a=commit;h=c3f368c60aabdc380c77608c614d91b0a628590a\n\n\\20014 and \\20015 should have been ZWNJ and ZWJ respectively. You just\nneed to handle them as any other unicode character - especially it is\na commit message and expectation is normal pain text display.\n\nI hope some one will fix this.\n\n- Praveen\n-- \nപ്രവീണ്‍ അരിമ്പ്രത്തൊടിയില്‍\n I know my rights; I want my phone call!\n What use is a phone call, if you are unable to speak?\n(as seen on /.)\nJoin The DRM Elimination Crew Now!\nhttp://fci.wikia.com/wiki/Anti-DRM-Campaign\n"},{"id":"97727","messageId":"m37i65gp6b.fsf@localhost.localdomain","threadId":"16700","inReplyTo":"3f2beab60812121033r5d41894t77acc271b7c6955c@mail.gmail.com","subject":"Re: gitweb and unicode special characters","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-12-12T19:37:57Z","receivedAt":"2008-12-12T19:37:57Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Praveen A\" writes:\n\n> Git currently does not handle unicode special characters ZWJ and ZWNJ,\n> both are heavily used in Malayalam and common in other languages\n> needing complex text layout like Sinhala and Arabic.\n> \n> An example of this is shown in the commit message here\n> http://git.savannah.gnu.org/gitweb/?p=smc.git;a=commit;h=c3f368c60aabdc380c77608c614d91b0a628590a\n> \n> \\20014 and \\20015 should have been ZWNJ and ZWJ respectively. You just\n> need to handle them as any other unicode character - especially it is\n> a commit message and expectation is normal pain text display.\n> \n> I hope some one will fix this.\n\nWell, I am bit stumped. git_commit calls format_log_line_html, which\nin turn calls esc_html. esc_html looks like this:\n\n sub esc_html ($;%) {\n \tmy $str = shift;\n \tmy %opts = @_;\n \n **\t$str = to_utf8($str);\n \t$str = $cgi->escapeHTML($str);\n \tif ($opts{'-nbsp'}) {\n \t\t$str =~ s/ / /g;\n \t}\n **\t$str =~ s|([[:cntrl:]])|(($1 ne \"\\t\") ? quot_cec($1) : $1)|eg;\n \treturn $str;\n }\n\nThe two important lines are marked with '**'. Not to_utf8 subroutine\nis very simple wrapper:\n\n # decode sequences of octets in utf8 into Perl's internal form,\n # which is utf-8 with utf8 flag set if needed. gitweb writes out\n # in utf-8 thanks to \"binmode STDOUT, ':utf8'\" at beginning\n sub to_utf8 {\n \tmy $str = shift;\n \tif (utf8::valid($str)) {\n \t\tutf8::decode($str);\n \t\treturn $str;\n \t} else {\n \t\treturn decode($fallback_encoding, $str, Encode::FB_DEFAULT);\n \t}\n }\n\nSo it looks like Perl treats \\20014 and \\20015 (ZWNJ and ZWJ) as\nbelonging to '[:cntrl:]' class. I don't know if it is correct from the\npoint of view of Unicode character classes, therefore if it is a bug\nin Perl, or just in gitweb.\n\nWe might need protecting similar to ($1 ne \"\\t\"), like (ord($1) < 127)\nor something... or perhaps we shouldn't use POSIX character class\n[:cntrl:] but something different when dealing with Unicode,\ne.g. \\p{Cc} or \\p{Control}, or perhaps \\p{C} (other). I don't know\nPerl (nor Unicode) enough to decide...\n\n\nP.S. Even that might not help much, as Savannah uses git and gitwev\nversion 1.5.6.5, which is probably version released with some major\ndistribution. As of now we are at 1.6.0.5...\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"97755","messageId":"m3y6ylf3mq.fsf@localhost.localdomain","threadId":"16700","inReplyTo":"m37i65gp6b.fsf@localhost.localdomain","subject":"Re: gitweb and unicode special characters","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-12-12T22:09:05Z","receivedAt":"2008-12-12T22:09:05Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jakub Narebski writes:\n> \"Praveen A\" writes:\n> \n> > Git currently does not handle unicode special characters ZWJ and ZWNJ,\n> > both are heavily used in Malayalam and common in other languages\n> > needing complex text layout like Sinhala and Arabic.\n> > \n> > An example of this is shown in the commit message here\n> > http://git.savannah.gnu.org/gitweb/?p=smc.git;a=commit;h=c3f368c60aabdc380c77608c614d91b0a628590a\n> > \n> > \\20014 and \\20015 should have been ZWNJ and ZWJ respectively. You just\n> > need to handle them as any other unicode character - especially it is\n> > a commit message and expectation is normal pain text display.\n> > \n> > I hope some one will fix this.\n> \n> Well, I am bit stumped. git_commit calls format_log_line_html, which\n> in turn calls esc_html. esc_html looks like this:\n> \n> sub esc_html ($;%) {\n> \tmy $str = shift;\n> \tmy %opts = @_;\n> \n> **\t$str = to_utf8($str);\n> \t$str = $cgi->escapeHTML($str);\n> \tif ($opts{'-nbsp'}) {\n> \t\t$str =~ s/ / /g;\n> \t}\n> **\t$str =~ s|([[:cntrl:]])|(($1 ne \"\\t\") ? quot_cec($1) : $1)|eg;\n> \treturn $str;\n> }\n> \n> The two important lines are marked with '**'.\n[...]\n\n> So it looks like Perl treats \\20014 and \\20015 (ZWNJ and ZWJ) as\n> belonging to '[:cntrl:]' class. I don't know if it is correct from the\n> point of view of Unicode character classes, therefore if it is a bug\n> in Perl, or just in gitweb.\n\nI checked this, via this simple Perl script:\n\n #!/usr/bin/perl\n\n use charnames \":full\";\n\n my $c = ord(\"\\N{ZWNJ}\");\n printf \"oct=%o dec=%d hex=%x\\n\", $c, $c, $c;\n\n \"\\N{ZWNJ}\" =~ /[[:cntrl:]]/ and print \"is [:cntrl:]\";\n\nAnd the answer was:\n\n oct=20014 dex=8204 hex=200c\n is [:cntrl:]\n\n'ZERO WIDTH NON-JOINER' _is_ control character... We probably should\nuse [^[:print:][:space:]] instead of [[:cntrl:]] here.\n\n[...]\n> P.S. Even that might not help much, as Savannah uses git and gitwev\n> version 1.5.6.5, which is probably version released with some major\n> distribution. As of now we are at 1.6.0.5...\n\nWhich can be seen from the fact that gitweb uses octal escapes,\ninstead of hex escapes...\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"97765","messageId":"3f2beab60812121655m6cd868bfhaaf386e6f5457533@mail.gmail.com","threadId":"16700","inReplyTo":"m3y6ylf3mq.fsf@localhost.localdomain","subject":"Re: gitweb and unicode special characters","fromName":"Praveen A","fromEmail":"pravi.a@gmail.com","sentAt":"2008-12-13T00:55:03Z","receivedAt":"2008-12-13T00:55:03Z","isPatch":false,"sender":{"key":"pravi.a@gmail.com","avatar":null},"body":"2008/12/12 Jakub Narebski :\n> Jakub Narebski writes:\n>> \"Praveen A\" writes:\n>>\n>> > Git currently does not handle unicode special characters ZWJ and ZWNJ,\n>> > both are heavily used in Malayalam and common in other languages\n>> > needing complex text layout like Sinhala and Arabic.\n>> >\n>> > An example of this is shown in the commit message here\n>> > http://git.savannah.gnu.org/gitweb/?p=smc.git;a=commit;h=c3f368c60aabdc380c77608c614d91b0a628590a\n>> >\n>> > \\20014 and \\20015 should have been ZWNJ and ZWJ respectively. You just\n>> > need to handle them as any other unicode character - especially it is\n>> > a commit message and expectation is normal pain text display.\n>> >\n>> > I hope some one will fix this.\n>>\n>> Well, I am bit stumped. git_commit calls format_log_line_html, which\n>> in turn calls esc_html. esc_html looks like this:\n>>\n>> sub esc_html ($;%) {\n>> my $str = shift;\n>> my %opts = @_;\n>>\n>> ** $str = to_utf8($str);\n>> $str = $cgi->escapeHTML($str);\n>> if ($opts{'-nbsp'}) {\n>> $str =~ s/ / /g;\n>> }\n>> ** $str =~ s|([[:cntrl:]])|(($1 ne \"\\t\") ? quot_cec($1) : $1)|eg;\n>> return $str;\n>> }\n>>\n>> The two important lines are marked with '**'.\n> [...]\n>\n>> So it looks like Perl treats \\20014 and \\20015 (ZWNJ and ZWJ) as\n>> belonging to '[:cntrl:]' class. I don't know if it is correct from the\n>> point of view of Unicode character classes, therefore if it is a bug\n>> in Perl, or just in gitweb.\n>\n> I checked this, via this simple Perl script:\n>\n> #!/usr/bin/perl\n>\n> use charnames \":full\";\n>\n> my $c = ord(\"\\N{ZWNJ}\");\n> printf \"oct=%o dec=%d hex=%x\\n\", $c, $c, $c;\n>\n> \"\\N{ZWNJ}\" =~ /[[:cntrl:]]/ and print \"is [:cntrl:]\";\n>\n> And the answer was:\n>\n> oct=20014 dex=8204 hex=200c\n> is [:cntrl:]\n>\n> 'ZERO WIDTH NON-JOINER' _is_ control character... We probably should\n> use [^[:print:][:space:]] instead of [[:cntrl:]] here.\n\nThat looks good. But I'm wondering why do we need to filter at all?\nIs it a security concern? It is just description.\n\n>\n> [...]\n>> P.S. Even that might not help much, as Savannah uses git and gitwev\n>> version 1.5.6.5, which is probably version released with some major\n>> distribution. As of now we are at 1.6.0.5...\n>\n> Which can be seen from the fact that gitweb uses octal escapes,\n> instead of hex escapes...\n\nBut we can expect it to work someday when savannah updates their git\nversion, or we can bug them to upgrade if the fix is in official git\nrelease.\n\n- Praveen\nj4v4m4n\n>\n> --\n> Jakub Narebski\n> Poland\n> ShadeHawk on #git\n>\n\n\n\n-- \nപ്രവീണ്‍ അരിമ്പ്രത്തൊടിയില്‍\n I know my rights; I want my phone call!\n What use is a phone call, if you are unable to speak?\n(as seen on /.)\nJoin The DRM Elimination Crew Now!\nhttp://fci.wikia.com/wiki/Anti-DRM-Campaign\n"},{"id":"97773","messageId":"200812130231.06929.jnareb@gmail.com","threadId":"16700","inReplyTo":"3f2beab60812121655m6cd868bfhaaf386e6f5457533@mail.gmail.com","subject":"Re: gitweb and unicode special characters","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-12-13T01:31:05Z","receivedAt":"2008-12-13T01:31:05Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sat, 13 Dec 2008 01:55, Praveen A wrote:\n> 2008/12/12 Jakub Narebski :\n>> Jakub Narebski writes:\n>>> \"Praveen A\" writes:\n>>>\n>>>> Git currently does not handle unicode special characters ZWJ and ZWNJ,\n>>>> both are heavily used in Malayalam and common in other languages\n>>>> needing complex text layout like Sinhala and Arabic.\n>>>>\n>>>> An example of this is shown in the commit message here\n>>>> http://git.savannah.gnu.org/gitweb/?p=smc.git;a=commit;h=c3f368c60aabdc380c77608c614d91b0a628590a\n>>>>\n>>>> \\20014 and \\20015 should have been ZWNJ and ZWJ respectively. You just\n>>>> need to handle them as any other unicode character - especially it is\n>>>> a commit message and expectation is normal pain text display.\n>>>\n>>> [...] git_commit calls format_log_line_html, which\n>>> in turn calls esc_html. esc_html looks like this:\n>>>\n>>> sub esc_html ($;%) {\n\n[...]\n>>> ** $str =~ s|([[:cntrl:]])|(($1 ne \"\\t\") ? quot_cec($1) : $1)|eg;\n>>> return $str;\n>>> }\n>>>\n>>> The two important lines are marked with '**'.\n>> [...]\n>>\n>>> So it looks like Perl treats \\20014 and \\20015 (ZWNJ and ZWJ) as\n>>> belonging to '[:cntrl:]' class. I don't know if it is correct from the\n>>> point of view of Unicode character classes, therefore if it is a bug\n>>> in Perl, or just in gitweb.\n>>\n>> I checked this, via this simple Perl script:\n[...]\n>> \"\\N{ZWNJ}\" =~ /[[:cntrl:]]/ and print \"is [:cntrl:]\";\n>>\n>> And the answer was:\n>>\n>> oct=20014 dex=8204 hex=200c\n>> is [:cntrl:]\n>>\n>> 'ZERO WIDTH NON-JOINER' _is_ control character... We probably should\n>> use [^[:print:][:space:]] instead of [[:cntrl:]] here.\n> \n> That looks good. But I'm wondering why do we need to filter at all?\n> Is it a security concern? It is just description.\n\nFirst, from the new description [^[:print:][:space:]], or even\n[^[:print:]] (whichever we choose) you can see that those characters\nwe are showing using C (\\r, \\v, \\b,...) + octal (in older gitweb) or\nhex (in never gitweb) escapes would be invisible otherwise, or do\nthe strange things like \\b aka backspace character.\n\nSidenote: There is probably one exception we want to add, namely not\nescape '\\r' at the end of line, to be able to deal better with DOS\nline endings (\\r\\n).\n\n\nSecond, and that is I think reason we started to escape control\ncharacters like \\014 or ^L i.e. FORM FEED (FF) character (e.g. in\nCOPYING file), or \\033 or ^[ i.e. ESCAPE (\\e) character (e.g. commit\n20a3847d) is that they are not allowed in XML, which means that they\nare not allowed in XHTML, which means that if they are on the page,\nand MIME-type is 'application/xml+html' forcing strict XML/XHTML mode\nvalidating browsers would not display the page because it is not valid\nXHTML. Mozilla 1.17.2 did this, and it would not show page; I don't\nknow how it works with more modern browsers.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"97777","messageId":"ghv8rf$47v$1@ger.gmane.org","threadId":"16700","inReplyTo":"200812130231.06929.jnareb@gmail.com","subject":"Re: gitweb and unicode special characters","fromName":"Edward Z. Yang","fromEmail":"edwardzyang@thewritingpot.com","sentAt":"2008-12-13T03:06:23Z","receivedAt":"2008-12-13T03:06:23Z","isPatch":false,"sender":{"key":"edwardzyang@thewritingpot.com","avatar":"https://gravatar.com/avatar/a805a0a3c1d7d36e7fe22270596e4d812723652933c59cac267e67c79126fdd0?d=mp&s=160"},"body":"Jakub Narebski wrote:\n> Sidenote: There is probably one exception we want to add, namely not\n> escape '\\r' at the end of line, to be able to deal better with DOS\n> line endings (\\r\\n).\n\nI'm sorry, but I have to disagree. I find being able to see \\r\nline-endings in the pretty-printed format is exceedingly useful for\nfiguring out if a file has been checked in with the wrong line-endings.\nThe number of files that must have \\r line endings are vanishingly small\n(Bat files are perhaps the one example I can think of right now).\n\nCheers,\nEdward\n"},{"id":"97822","messageId":"m3d4fvg23p.fsf@localhost.localdomain","threadId":"16700","inReplyTo":"ghv8rf$47v$1@ger.gmane.org","subject":"Re: gitweb and unicode special characters","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-12-13T22:08:29Z","receivedAt":"2008-12-13T22:08:29Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Edward Z. Yang\" writes:\n> Jakub Narebski wrote:\n\n> > Sidenote: There is probably one exception we want to add, namely not\n> > escape '\\r' at the end of line, to be able to deal better with DOS\n> > line endings (\\r\\n).\n> \n> I'm sorry, but I have to disagree. I find being able to see \\r\n> line-endings in the pretty-printed format is exceedingly useful for\n> figuring out if a file has been checked in with the wrong line-endings.\n> The number of files that must have \\r line endings are vanishingly small\n> (BAT files are perhaps the one example I can think of right now).\n\nWell, it is a bit annoying if you have checked file with wrong line\nendings, and just noticed this... I was thinking about adding '(DOS)'\nor something indicator at the bottom of 'blob' and 'blame' views, but\nI guess I can live with '\\r'...\n\nIn short: I agree, that was not a good idea.\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"}]}