From: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
Date: Fri, 03 Feb 2012 17:45:57 GMT
Subject: Re: [PATCH/RFCv2 (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info
Message-ID: <20120203184557.59042dec@gmail.com>
In-Reply-To: <201202031344.55750.jnareb@gmail.com>

Jakub Narebski <jnareb@gmail.com> wrote:

> Gitweb tries hard to properly process UTF-8 data, by marking output
> from git commands and contents of files as UTF-8 with to_utf8()
> subroutine.  This ensures that gitweb would print correctly UTF-8
> e.g. in 'log' and 'commit' views.
> 
> Unfortunately it misses another source of potentially Unicode input,
> namely query parameters.  The result is that one cannot search for a
> string containing characters outside US-ASCII.  For example searching
> for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L
> WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82
> bytes in UTF-8 and percent-encoded as %C5%81) result in the following
> incorrect data in search field
> 
> 	Michał Kiedrowicz
> 
> This is caused by CGI by default treating '0xc5 0x82' bytes as two
> characters in Perl legacy encoding latin-1 (iso-8859-1), because 's'
> query parameter is not processed explicitly as UTF-8 encoded string.
> 
> The solution used here follows "Using Unicode in a Perl CGI script"
> article on http://www.lemoda.net/cgi/perl-unicode/index.html:
> 
> 	use CGI;
> 	use Encode 'decode_utf8;
> 	my $value = params('input');
> 	$value = decode_utf8($value);
> 
> Decoding UTF-8 is done when filling %input_params hash and $path_info
> variable; the former required to move from explicit $cgi->param(<label>)
> to $input_params{<name>} in a few places, which is a good idea anyway.
> 
> Another required change was to add -override=>1 parameter to
> $cgi->textfield() invocation (in search form).  Otherwise CGI would
> use values from query string if it is present, filling value from
> $cgi->param... without decode_utf8().  As we are using value of
> appropriate parameter anyway, -override=>1 doesn't change the
> situation but makes gitweb fill search field correctly.
> 
> Alternate solution would be to simply use the '-utf8' pragma (via
> "use CGI '-utf8';"), but according to CGI.pm documentation it may
> cause problems with POST requests containing binary files... and
> it requires CGI 3.31 (I think), released with perl v5.8.9.
> 
> Noticed-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
> Signed-off-by: Jakub Narębski <jnareb@gmail.com>
> ---
> On Fri, 3 Feb 2012, Michal Kiedrowicz wrote:
> > Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > > Is it what you mean by "this doesn't work for me", i.e. working
> > > search, garbage in search field?
> > 
> > I mean "garbage in search field". Search works even without the patch
> > (at least on Debian with git-1.7.7.3, perl-5.10.1 and CGI-3.43; I
> > don't have my notebook nearby at the moment to check).
> [...]
> 
> > > Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext)
> > > like in gitweb, CGI.pm would read $cgi->param("s") by itself -
> > > without decoding. 
> > 
> > Makes sense. When I tried calling to_utf8() in the line that defines
> > textfield (this was my first approach to this problem), it haven't
> > changed anything.
> 
> Yes, and it doesn't makes sense in gitweb case - we use value of 
> $cgi->param("s") as default value of text field anyway, but in
> Unicode-aware way.
>  
> > > To skip this we need to pass -force=>1  or
> > > -override=>1 (i.e. further changes to gitweb).
> 
> This patch does this.  
> 
> Does it make work for you?
> 

Yes, it works for me. Search form properly displays "ł". Thanks!