{"thread":{"id":"29192","subject":"[PATCH 1/3] gitweb: Call to_utf8() on input string in chop_and_escape_str()","startedAt":"2011-12-17T09:22:20Z","lastAt":"2011-12-19T16:21:17Z","messageCount":8,"participants":["Jakub Narebski","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"181375","messageId":"1324113743-21498-1-git-send-email-jnareb@gmail.com","threadId":"29192","inReplyTo":null,"subject":"[PATCH 0/3 (resend)] gitweb: Various to_utf8 / esc_html fixes","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-12-17T09:22:20Z","receivedAt":"2011-12-17T09:22:20Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Sorry for resend of this series, but I forgot to generate patches in\nUTF-8 instead of i18n.logoutputencoding=iso-8859-2\n\n\nThis is post-release resend of Jürgen patches (which were sent\nduring feature-freeze).\n\nI have slightly extended commit messages, and added my ACK.\n\nJürgen Kreileder (3):\n  gitweb: Call to_utf8() on input string in chop_and_escape_str()\n  gitweb: esc_html() site name for title in OPML\n  gitweb: Output valid utf8 in git_blame_common('data')\n\n gitweb/gitweb.perl |    8 ++++++--\n 1 files changed, 6 insertions(+), 2 deletions(-)\n\n-- \n1.7.6\n"},{"id":"181374","messageId":"1324113743-21498-2-git-send-email-jnareb@gmail.com","threadId":"29192","inReplyTo":"1324113743-21498-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 1/3] gitweb: Call to_utf8() on input string in chop_and_escape_str()","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-12-17T09:22:21Z","receivedAt":"2011-12-17T09:22:21Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"From: Jürgen Kreileder <jk@blackdown.de>\n\na) To fix the comparison with the chopped string,\n   otherwise we compare bytes with characters, as\n   chop_str() must run to_utf8() for correct operation\nb) To give the title attribute correct encoding;\n   we need to mark strings as UTF-8 before outpur\n\nSigned-off-by: Jürgen Kreileder <jk@blackdown.de>\nAcked-by: Jakub Narębski <jnareb@gmail.com>\n---\n gitweb/gitweb.perl |    1 +\n 1 files changed, 1 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex f80f259..35126cd 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1696,6 +1696,7 @@ sub chop_and_escape_str {\n \tmy ($str) = @_;\n \n \tmy $chopped = chop_str(@_);\n+\t$str = to_utf8($str);\n \tif ($chopped eq $str) {\n \t\treturn esc_html($chopped);\n \t} else {\n-- \n1.7.6\n"},{"id":"181377","messageId":"1324113743-21498-3-git-send-email-jnareb@gmail.com","threadId":"29192","inReplyTo":"1324113743-21498-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 2/3] gitweb: esc_html() site name for title in OPML","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-12-17T09:22:22Z","receivedAt":"2011-12-17T09:22:22Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"From: Jürgen Kreileder <jk@blackdown.de>\n\nThis escapes the site name in OPML (XML uses the same escaping rules\nas HTML).  Also fixes encoding issues because esc_html() uses\nto_utf8().\n\nSigned-off-by: Jürgen Kreileder <jk@blackdown.de>\nAcked-by: Jakub Narębski <jnareb@gmail.com>\n---\n gitweb/gitweb.perl |    3 ++-\n 1 files changed, 2 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 35126cd..dcf4658 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -7863,11 +7863,12 @@ sub git_opml {\n \t\t-charset => 'utf-8',\n \t\t-content_disposition => 'inline; filename=\"opml.xml\"');\n \n+\tmy $title = esc_html($site_name);\n \tprint <<XML;\n <?xml version=\"1.0\" encoding=\"utf-8\"?>\n <opml version=\"1.0\">\n <head>\n-  <title>$site_name OPML Export</title>\n+  <title>$title OPML Export</title>\n </head>\n <body>\n <outline text=\"git RSS feeds\">\n-- \n1.7.6\n"},{"id":"181376","messageId":"1324113743-21498-4-git-send-email-jnareb@gmail.com","threadId":"29192","inReplyTo":"1324113743-21498-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 3/3] gitweb: Output valid utf8 in git_blame_common('data')","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-12-17T09:22:23Z","receivedAt":"2011-12-17T09:22:23Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"From: Jürgen Kreileder <jk@blackdown.de>\n\nOtherwise when javascript-actions are enabled gitweb shown broken\nauthor names in the tooltips on blame pages ('blame_incremental'\nview).\n\nSigned-off-by: Jürgen Kreileder <jk@blackdown.de>\nAcked-by: Jakub Narębski <jnareb@gmail.com>\n---\n gitweb/gitweb.perl |    4 +++-\n 1 files changed, 3 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex dcf4658..d24763b 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -6244,7 +6244,9 @@ sub git_blame_common {\n \t\t\t-type=>\"text/plain\", -charset => \"utf-8\",\n \t\t\t-status=> \"200 OK\");\n \t\tlocal $| = 1; # output autoflush\n-\t\tprint while <$fd>;\n+\t\twhile (my $line = <$fd>) {\n+\t\t\tprint to_utf8($line);\n+\t\t}\n \t\tclose $fd\n \t\t\tor print \"ERROR $!\\n\";\n \n-- \n1.7.6\n"},{"id":"181417","messageId":"7vty4zdmml.fsf@alter.siamese.dyndns.org","threadId":"29192","inReplyTo":"1324113743-21498-1-git-send-email-jnareb@gmail.com","subject":"Re: [PATCH 0/3 (resend)] gitweb: Various to_utf8 / esc_html fixes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-17T19:27:46Z","receivedAt":"2011-12-17T19:27:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> Sorry for resend of this series, but I forgot to generate patches in\n> UTF-8 instead of i18n.logoutputencoding=iso-8859-2\n>\n>\n> This is post-release resend of Jürgen patches (which were sent\n> during feature-freeze).\n>\n> I have slightly extended commit messages, and added my ACK.\n>\n> Jürgen Kreileder (3):\n>   gitweb: Call to_utf8() on input string in chop_and_escape_str()\n>   gitweb: esc_html() site name for title in OPML\n>   gitweb: Output valid utf8 in git_blame_common('data')\n>\n>  gitweb/gitweb.perl |    8 ++++++--\n>  1 files changed, 6 insertions(+), 2 deletions(-)\n\nThanks.\n"},{"id":"181447","messageId":"201112190154.19107.jnareb@gmail.com","threadId":"29192","inReplyTo":"1324113743-21498-1-git-send-email-jnareb@gmail.com","subject":"[PATCH 4/3] gitweb: Fix fallback mode of to_utf8 subroutine","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-12-19T00:54:16Z","receivedAt":"2011-12-19T00:54:16Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"e5d3de5 (gitweb: use Perl built-in utf8 function for UTF-8 decoding.,\n2007-12-04) was meant to make gitweb faster by using Perl's internals\n(see subsection \"Messing with Perl's Internals\" in Encode(3pm) manpage)\n\nSimple benchmark confirms that (old = 00f429a, new = this version);\nnote that it is synthetic benchmark of standalone subroutines, not\nof gitweb itself\n\n        old  new\n  old    -- -65%\n  new  189%   --\n\nUnfortunately it made fallback mode of to_utf8 do not work...  except\nfor default value 'latin1' of $fallback_encoding ('latin1' is Perl\nnative encoding), which is why it was not noticed for such long time.\n\nutf8::valid(STRING) is an internal function that tests whether STRING\nis in a _consistent state_ regarding UTF-8.  It returns true is\nwell-formed UTF-8 and has the UTF-8 flag on _*or*_ if string is held\nas bytes (both these states are 'consistent').  For gitweb the second\noption was true, as output from git commands is opened without ':utf8'\nlayer.\n\nWhat made it work at all for STRING in 'latin1' encoding is the fact\nthat utf8:decode(STRING) turns on UTF-8 flag only if source string is\nvalid UTF-8 and contains multi-byte UTF-8 characters... and that if\nstring doesn't have UTF-8 flag set it is treated as in native Perl\nencoding, i.e.  'latin1' / 'iso-8859-1' (unless native encoding it is\nEBCDIC ;-)).  It was ':utf8' layer that actually converted 'latin1'\n(no UTF-8 flag == native == 'latin1) to 'utf8'.\n\n\nLet's make use of the fact that utf8:decode(STRING) returns false if\nSTRING is invalid as UTF-8 to check whether to enable fallback mode.\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nExcuse me for overly long commit message...\n\nResent as part of to_utf8 fixes for better visibility\n\n gitweb/gitweb.perl |    3 +--\n 1 files changed, 1 insertions(+), 2 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex d24763b..75b0970 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1443,8 +1443,7 @@ sub validate_refname {\n sub to_utf8 {\n \tmy $str = shift;\n \treturn undef unless defined $str;\n-\tif (utf8::valid($str)) {\n-\t\tutf8::decode($str);\n+\tif (utf8::valid($str) && utf8::decode($str)) {\n \t\treturn $str;\n \t} else {\n \t\treturn decode($fallback_encoding, $str, Encode::FB_DEFAULT);\n-- \n1.7.6\n"},{"id":"181464","messageId":"201112191311.58787.jnareb@gmail.com","threadId":"29192","inReplyTo":"201112190154.19107.jnareb@gmail.com","subject":"Re: [PATCH 4/3] gitweb: Fix fallback mode of to_utf8 subroutine","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-12-19T12:11:57Z","receivedAt":"2011-12-19T12:11:57Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jakub Narebski wrote:\n\n> e5d3de5 (gitweb: use Perl built-in utf8 function for UTF-8 decoding.,\n> 2007-12-04) was meant to make gitweb faster by using Perl's internals\n> (see subsection \"Messing with Perl's Internals\" in Encode(3pm) manpage)\n> \n> Simple benchmark confirms that (old = 00f429a, new = this version);\n> note that it is synthetic benchmark of standalone subroutines, not\n> of gitweb itself\n> \n>         old  new\n>   old    -- -65%\n>   new  189%   --\n\nNb. that was about operations / second (higher is better):\n\n           Rate  old  new\n    old  2067/s   -- -65%\n    new  5863/s 184%   --\n\nOr in slightly different benchmark (more smaller lines):\n\n          Rate  old  new\n    old  277/s   -- -73%\n    new 1021/s 268%   --\n\n old$ time ./t9500-gitweb-standalone-no-errors.sh >/dev/null\n\n   real       1m16.788s\n   user       1m0.908s\n   sys        0m14.033s\n   user+sys   1m14.941s\n\n new$ time ./t9500-gitweb-standalone-no-errors.sh >/dev/null\n\n    real      1m12.216s\n    user      0m57.300s\n    sys       0m13.329s\n    user+sys  1m10.639s\n\nThough such benchmarks should have been a part of e5d3de5.\n\n\nP.S. I started to get strange errors\n\n XML Parsing Error: xml processing instruction not at start of external entity\n Location: http://localhost/cgi-bin/gitweb/gitweb.cgi\n Line Number 37, Column 1:\n <?xml version=\"1.0\" encoding=\"utf-8\"?>\n ^\n\nwhile \"show source\" shows that '<?xml version=\"1.0\" encoding=\"utf-8\"?>'\nis the first line.  WTF?!?\n\nP.P.S. Now I am getting errors when running gitweb, but only in some\ncases (via mod_cgi not as standalone script, only when using lynx),\nnamely it looks like it falls back to 'latin1' when doing content\nwhich is valid UTF-8.\n\nWill investigate.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"181471","messageId":"201112191721.17478.jnareb@gmail.com","threadId":"29192","inReplyTo":"201112191311.58787.jnareb@gmail.com","subject":"[PATCH 4/3 v2 (bugfix)] gitweb: Fix fallback mode of to_utf8 subroutine","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-12-19T16:21:17Z","receivedAt":"2011-12-19T16:21:17Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"e5d3de5 (gitweb: use Perl built-in utf8 function for UTF-8 decoding.,\n2007-12-04) was meant to make gitweb faster by using Perl's internals\n(see subsection \"Messing with Perl's Internals\" in Encode(3pm) manpage)\n\nSimple benchmark confirms that (old = 00f429a, new = this version):\nnote that it is synthetic benchmark of standalone subroutines, not\nof gitweb itself (where probably no visible difference in performace\nwill show)\n\n        Rate  old  new\n  old 1582/s   -- -64%\n  new 4453/s 181%   --\n\nUnfortunately it made fallback mode of to_utf8 do not work...  except\nfor default value 'latin1' of $fallback_encoding (because 'latin1' is\nPerl native encoding), which is probably why it was not noticed for so\nlong.\n\n\nutf8::valid(STRING) is an internal function that tests whether STRING\nis in a _consistent state_ regarding UTF-8.  It returns true is\nwell-formed UTF-8 and has the UTF-8 flag on _*or*_ if string is held\nas bytes (both these states are 'consistent').\n\nFor gitweb in most cases the second option was true, as output from\ngit commands is opened without ':utf8' layer.  So utf8::valid is not\nuseful for to_utf8.\n\nWhat made it look as if to_utf8() fallback mode worked correctly\n(though only for $fallback_encoding at its default value 'latin1')\nwas the fact that utf8::decode(STRING) turns on UTF-8 flag only if\nsource string^W octets form a valid UTF-8 and it contains multi-byte\nUTF-8 characters... this means that if string was not valid UTF-8\nit didn't get UTF-8 flag.\n\nWhen string doesn't have UTF-8 flag set, it is treated as if it was in\nnative Perl encoding, which is 'latin1' (unless native encoding is\nEBCDIC ;-)).  So it was ':utf8' layer that actually converted 'latin1'\n(no UTF-8 flag == native == 'latin1) to 'utf8', and not to_utf8()\nsubroutine.  Fallback mode was never triggered.\n\n\nLet's make use of the fact that utf8::decode(STRING) returns false if\nSTRING is invalid as UTF-8 to check whether to enable fallback mode.\n\nNote however that if STRING has UTF-8 flag set already, then\nutf8::decode also returns false, which could cause problems if given\nstring was already converted with to_utf8().  Such double conversion\ncan happen in gitweb.  Therefore we have to check if STRING has UTF-8\nflag set with utf8::is_utf8(); if this subroutine returns true then we\nhave already decoded (converted) string, and don't have to do it\nsecond time.\n\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\nJakub Narebski wrote:\n> P.S. I started to get strange errors\n> \n>  XML Parsing Error: xml processing instruction not at start of external entity\n>  Location: http://localhost/cgi-bin/gitweb/gitweb.cgi\n>  Line Number 37, Column 1:\n>  <?xml version=\"1.0\" encoding=\"utf-8\"?>\n>  ^\n> \n> while \"show source\" shows that '<?xml version=\"1.0\" encoding=\"utf-8\"?>'\n> is the first line.  WTF?!?\n> \n> P.P.S. Now I am getting errors when running gitweb, but only in some\n> cases (via mod_cgi not as standalone script, only when using lynx),\n> namely it looks like it falls back to 'latin1' when doing content\n> which is valid UTF-8.\n> \n> Will investigate.\n\nNow it is fixed; the error was caused by to_utf8 not dealing with double\nencoding for strings outside 7bit ASCII.\n\n gitweb/gitweb.perl |    4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex d24763b..fc41b07 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1443,8 +1443,8 @@ sub validate_refname {\n sub to_utf8 {\n \tmy $str = shift;\n \treturn undef unless defined $str;\n-\tif (utf8::valid($str)) {\n-\t\tutf8::decode($str);\n+\n+\tif (utf8::is_utf8($str) || utf8::decode($str)) {\n \t\treturn $str;\n \t} else {\n \t\treturn decode($fallback_encoding, $str, Encode::FB_DEFAULT);\n-- \n1.7.6\n"}]}