Re: [PATCH/RFC (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

> On Thu, 2 Feb 2012, Jakub Narebski wrote:
> > On Thu, 2 Feb 2012, Michał Kiedrowicz wrote:
> > > Jakub Narebski <jnareb@gmail.com> wrote:
> > > 
> > > > Gitweb tries hard to properly process UTF-8 data, by marking
> > > > output from git commands and contents of files as UTF-8 with
> > > > to_utf8() subroutine.  This ensures that gitweb would print
> > > > correctly UTF-8 e.g. in 'log' and 'commit' views.
> > > > 
> > > > Unfortunately it misses another source of potentially Unicode
> > > > input, namely query parameters.  The result is that one cannot
> > > > search for a string containing characters outside US-ASCII.
> > > > For example searching for "Michał Kiedrowicz" (containing
> > > > letter 'ł' - LATIN SMALL LETTER L WITH STROKE, with Unicode
> > > > codepoint U+0142, represented with 0xc5 0x82 bytes in UTF-8 and
> > > > percent-encoded as %C5%81) result in the following incorrect
> > > > data in search field
> > > > 
> > > > 	Michał Kiedrowicz
> > > > 
> > > > This is caused by CGI by default treating '0xc5 0x82' bytes as
> > > > two characters in Perl legacy encoding latin-1 (iso-8859-1),
> > > > because 's' query parameter is not processed explicitly as
> > > > UTF-8 encoded string.
> > > > 
> > > > The solution used here follows "Using Unicode in a Perl CGI
> > > > script" article on
> > > > http://www.lemoda.net/cgi/perl-unicode/index.html:
> > > > 
> > > > 	use CGI;
> > > > 	use Encode 'decode_utf8;
> > > > 	my $value = params('input');
> > > > 	$value = decode_utf8($value);
> > > > 
> > > > This is done when filling %input_params hash; this required to
> > > > move from explicit $cgi->param(<label>) to
> > > > $input_params{<name>} in a few places.
> > > 
> > > I'm sorry but this doesn't work for me. I would be happy to help
> > > if you have some questions about it.
> > 
> > Strange.  http://www.lemoda.net/cgi/perl-unicode/index.html says
> > that those two approaches should be equivalent.  The -utf8 pragma
> > version doesn't work for me at all, while this one works in that if
> > finds what it is supposed to, but shows garbage in search form.
> 
> Is it what you mean by "this doesn't work for me", i.e. working
> search, garbage in search field?

> To skip this we need to pass -force=>1  or
> -override=>1 (i.e. further changes to gitweb).
> 
> -utf8 pragma works with more modern CGI.pm, but does not with 3.10.
>