{"thread":{"id":"29509","subject":"[RFC PATCH] gitweb: use CGI with -utf8","startedAt":"2012-02-01T22:50:53Z","lastAt":"2012-02-03T21:09:12Z","messageCount":14,"participants":["Michał Kiedrowicz","Jakub Narebski","Michal Kiedrowicz","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"183511","messageId":"1328136653-20559-1-git-send-email-michal.kiedrowicz@gmail.com","threadId":"29509","inReplyTo":null,"subject":"[RFC PATCH] gitweb: use CGI with -utf8","fromName":"Michał Kiedrowicz","fromEmail":"michal.kiedrowicz@gmail.com","sentAt":"2012-02-01T22:50:53Z","receivedAt":"2012-02-01T22:50:53Z","isPatch":true,"sender":{"key":"michal.kiedrowicz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/14072847?v=4"},"body":"I noticed that gitweb tries a lot to properly process UTF-8 data, for\nexample it prints my name correctly in log and commit information, but\nit echos junk in the search field. It looks like:\n\n\tMichał Kiedrowicz\n\nI don't know CGI well and I never touched gitewb code, but I found this\non http://www.lemoda.net/cgi/perl-unicode/index.html:\n\n\tuse CGI '-utf8';\n\tmy $value = params ('input');\n\nI tried it and that fixed my problem. I'm not sure about the\nconsequences, maybe someone more experienced in CGI might help?\n---\n gitweb/gitweb.perl | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex abb5a79..74d45b1 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -10,7 +10,7 @@\n use 5.008;\n use strict;\n use warnings;\n-use CGI qw(:standard :escapeHTML -nosticky);\n+use CGI qw(:standard :escapeHTML -nosticky -utf8);\n use CGI::Util qw(unescape);\n use CGI::Carp qw(fatalsToBrowser set_message);\n use Encode;\n-- \n1.7.3.4\n"},{"id":"183645","messageId":"m37h05c8c1.fsf@localhost.localdomain","threadId":"29509","inReplyTo":"1328136653-20559-1-git-send-email-michal.kiedrowicz@gmail.com","subject":"Re: [RFC PATCH] gitweb: use CGI with -utf8","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2012-02-02T20:01:41Z","receivedAt":"2012-02-02T20:01:41Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Michał Kiedrowicz writes:\n\n> I noticed that gitweb tries a lot to properly process UTF-8 data, for\n> example it prints my name correctly in log and commit information, but\n> it echos junk in the search field. It looks like:\n> \n> \tMichał Kiedrowicz\n> \n> I don't know CGI well and I never touched gitewb code, but I found this\n> on http://www.lemoda.net/cgi/perl-unicode/index.html:\n> \n> \tuse CGI '-utf8';\n> \tmy $value = params ('input');\n> \n> I tried it and that fixed my problem. I'm not sure about the\n> consequences, maybe someone more experienced in CGI might help?\n\nI have reworded this to form a proper commit message (see\nDocumentation/SubmittingPatches) and I'll resend this as a reply to\nthis email.\n\n> ---\n> gitweb/gitweb.perl | 2 +-\n> 1 files changed, 1 insertions(+), 1 deletions(-)\n> \n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index abb5a79..74d45b1 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -10,7 +10,7 @@\n> use 5.008;\n> use strict;\n> use warnings;\n> -use CGI qw(:standard :escapeHTML -nosticky);\n> +use CGI qw(:standard :escapeHTML -nosticky -utf8);\n> use CGI::Util qw(unescape);\n> use CGI::Carp qw(fatalsToBrowser set_message);\n> use Encode;\n> -- \n\nDoes this actually work for you? Because it doesn't work for me\n(perhaps I have too old CGI module: what CGI.pm and what Perl version\ndo you use?).\n\nSee other solution to this in other reply to this email.\n\n-- \nJakub Narebski\n"},{"id":"183649","messageId":"201202022108.51353.jnareb@gmail.com","threadId":"29509","inReplyTo":"m37h05c8c1.fsf@localhost.localdomain","subject":"[PATCH/RFC (version A)] gitweb: use CGI with -utf8 to process Unicode query parameters correctly","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2012-02-02T20:08:50Z","receivedAt":"2012-02-02T20:08:50Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Gitweb tries hard to properly process UTF-8 data, by marking output\nfrom git commands and contents of files as UTF-8 with to_utf8()\nsubroutine. This ensures that gitweb would print correctly UTF-8\ne.g. in 'log' and 'commit' views.\n\nUnfortunately it misses another source of potentially Unicode input,\nnamely query parameters. The result is that one cannot search for a\nstring containing characters outside US-ASCII. For example searching\nfor \"Michał Kiedrowicz\" (containing letter 'ł' - LATIN SMALL LETTER L\nWITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82\nbytes in UTF-8 and percent-encoded as %C5%81) result in the following\nincorrect data in search field\n\n\tMichał Kiedrowicz\n\nThis is caused by CGI by default treating '0xc5 0x82' bytes as two\ncharacters in Perl legacy encoding latin-1 (iso-8859-1), because 's'\nquery parameter is not processed explicitly as UTF-8 encoded string.\n\nAccording to \"Using Unicode in a Perl CGI script\" article on\nhttp://www.lemoda.net/cgi/perl-unicode/index.html the simplest\nsolution is to just import '-utf8' pragma for CGI module:\n\n\tuse CGI '-utf8';\n\tmy $value = params('input');\n\nAccording to CGI module documentation, the '-utf8' pragma may cause\nproblems with POST requests containing binary files... but gitweb\ncurrently do not use POST requests at all, so this should be not a\nproblem now.\n\nAlternate solution would be to explicity decode query parameters when\nstoring them in %input_params (and perhaps also path_info).\n\n[jn: reworded / rewritten commit message]\n\nSigned-off-by: Michał Kiedrowicz \nSigned-off-by: Jakub Narębski \n---\n gitweb/gitweb.perl | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 9cf7e71..a7441ef 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -10,7 +10,7 @@\n use 5.008;\n use strict;\n use warnings;\n-use CGI qw(:standard :escapeHTML -nosticky);\n+use CGI qw(:standard :escapeHTML -nosticky -utf8);\n use CGI::Util qw(unescape);\n use CGI::Carp qw(fatalsToBrowser set_message);\n use Encode;\n-- \n1.7.6\n"},{"id":"183651","messageId":"201202022110.07127.jnareb@gmail.com","threadId":"29509","inReplyTo":"m37h05c8c1.fsf@localhost.localdomain","subject":"[PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2012-02-02T20:10:06Z","receivedAt":"2012-02-02T20:10:06Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Gitweb tries hard to properly process UTF-8 data, by marking output\nfrom git commands and contents of files as UTF-8 with to_utf8()\nsubroutine. This ensures that gitweb would print correctly UTF-8\ne.g. in 'log' and 'commit' views.\n\nUnfortunately it misses another source of potentially Unicode input,\nnamely query parameters. The result is that one cannot search for a\nstring containing characters outside US-ASCII. For example searching\nfor \"Michał Kiedrowicz\" (containing letter 'ł' - LATIN SMALL LETTER L\nWITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82\nbytes in UTF-8 and percent-encoded as %C5%81) result in the following\nincorrect data in search field\n\n\tMichał Kiedrowicz\n\nThis is caused by CGI by default treating '0xc5 0x82' bytes as two\ncharacters in Perl legacy encoding latin-1 (iso-8859-1), because 's'\nquery parameter is not processed explicitly as UTF-8 encoded string.\n\nThe solution used here follows \"Using Unicode in a Perl CGI script\"\narticle on http://www.lemoda.net/cgi/perl-unicode/index.html:\n\n\tuse CGI;\n\tuse Encode 'decode_utf8;\n\tmy $value = params('input');\n\t$value = decode_utf8($value);\n\nThis is done when filling %input_params hash; this required to move\nfrom explicit $cgi->param(