git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH/RFCv2 (version B)] gitweb: Allow UTF-8 encoded CGI query parameters and path_info

From
Jakub Narebski <jnareb@gmail.com>
Date
Feb 3, 2012, 12:44 UTC
Message-ID
<201202031344.55750.jnareb@gmail.com>
In-Reply-To
<20120203083935.5d9d4b18@mkiedrowicz.ivo.pl>

Gitweb tries hard to properly process UTF-8 data, by marking output from git commands and contents of files as UTF-8 with to_utf8() subroutine. This ensures that gitweb would print correctly UTF-8 e.g. in 'log' and 'commit' views.

Unfortunately it misses another source of potentially Unicode input, namely query parameters. The result is that one cannot search for a string containing characters outside US-ASCII. For example searching for "Michał Kiedrowicz" (containing letter 'ł' - LATIN SMALL LETTER L WITH STROKE, with Unicode codepoint U+0142, represented with 0xc5 0x82 bytes in UTF-8 and percent-encoded as %C5%81) result in the following incorrect data in search field

	Michał Kiedrowicz

This is caused by CGI by default treating '0xc5 0x82' bytes as two characters in Perl legacy encoding latin-1 (iso-8859-1), because 's' query parameter is not processed explicitly as UTF-8 encoded string.

The solution used here follows "Using Unicode in a Perl CGI script" article on http://www.lemoda.net/cgi/perl-unicode/index.html:

	use CGI;
	use Encode 'decode_utf8;
	my $value = params('input');
	$value = decode_utf8($value);

Decoding UTF-8 is done when filling %input_params hash and $path_info variable; the former required to move from explicit $cgi->param(<label>) to $input_params{<name>} in a few places, which is a good idea anyway.

Another required change was to add -override=>1 parameter to $cgi->textfield() invocation (in search form). Otherwise CGI would use values from query string if it is present, filling value from $cgi->param... without decode_utf8(). As we are using value of appropriate parameter anyway, -override=>1 doesn't change the situation but makes gitweb fill search field correctly.

Alternate solution would be to simply use the '-utf8' pragma (via "use CGI '-utf8';"), but according to CGI.pm documentation it may cause problems with POST requests containing binary files... and it requires CGI 3.31 (I think), released with perl v5.8.9.

Noticed-by: Michał Kiedrowicz <michal.kiedrowicz@gmail.com>
Signed-off-by: Jakub Narębski <jnareb@gmail.com>
---
On Fri, 3 Feb 2012, Michal Kiedrowicz wrote:
> Jakub Narebski <jnareb@gmail.com> wrote:
Show 6 quoted lines
> > Is it what you mean by "this doesn't work for me", i.e. working
> > search, garbage in search field?
> 
> I mean "garbage in search field". Search works even without the patch
> (at least on Debian with git-1.7.7.3, perl-5.10.1 and CGI-3.43; I
> don't have my notebook nearby at the moment to check).
[...]
Show 7 quoted lines
> > Damn.  If we use $cgi->textfield(-name => "s", -value => $searchtext)
> > like in gitweb, CGI.pm would read $cgi->param("s") by itself -
> > without decoding. 
> 
> Makes sense. When I tried calling to_utf8() in the line that defines
> textfield (this was my first approach to this problem), it haven't
> changed anything.
Yes, and it doesn't makes sense in gitweb case - we use value of 
$cgi->param("s") as default value of text field anyway, but in
Unicode-aware way.
 
> > To skip this we need to pass -force=>1  or
> > -override=>1 (i.e. further changes to gitweb).
This patch does this.  
Does it make work for you?
> > -utf8 pragma works with more modern CGI.pm, but does not with 3.10.

-utf8 pragma was added in CVS revision 1.238 of CGI.pm, which I think is present in CGI 3.31, released with perl v5.8.9. Theoretically gitweb maintains backward compatibility with perl v5.8.3 or something ("use 5.008;" but IIRC 5.8.3 is needed for correct Unicde handling anyway).

 gitweb/gitweb.perl |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 9cf7e71..bd5fff9 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -52,7 +52,7 @@ sub evaluate_uri {
 	# as base URL.
 	# Therefore, if we needed to strip PATH_INFO, then we know that we have
 	# to build the base URL ourselves:
-	our $path_info = $ENV{"PATH_INFO"};
+	our $path_info = decode_utf8($ENV{"PATH_INFO"});
 	if ($path_info) {
 		if ($my_url =~ s,\Q$path_info\E$,, &&
 		    $my_uri =~ s,\Q$path_info\E$,, &&
@@ -816,9 +816,9 @@ sub evaluate_query_params {
 
 	while (my ($name, $symbol) = each %cgi_param_mapping) {
 		if ($symbol eq 'opt') {
-			$input_params{$name} = [ $cgi->param($symbol) ];
+			$input_params{$name} = [ map { decode_utf8($_) } $cgi->param($symbol) ];
 		} else {
-			$input_params{$name} = $cgi->param($symbol);
+			$input_params{$name} = decode_utf8($cgi->param($symbol));
 		}
 	}
 }
@@ -2767,7 +2767,7 @@ sub git_populate_project_tagcloud {
 	}
 
 	my $cloud;
-	my $matched = $cgi->param('by_tag');
+	my $matched = $input_params{'ctag'};
 	if (eval { require HTML::TagCloud; 1; }) {
 		$cloud = HTML::TagCloud->new;
 		foreach my $ctag (sort keys %ctags_lc) {
@@ -3873,7 +3873,7 @@ sub print_search_form {
 	                       -values => ['commit', 'grep', 'author', 'committer', 'pickaxe']) .
 	      $cgi->sup($cgi->a({-href => href(action=>"search_help")}, "?")) .
 	      " search:\n",
-	      $cgi->textfield(-name => "s", -value => $searchtext) . "\n" .
+	      $cgi->textfield(-name => "s", -value => $searchtext, -override => 1) . "\n" .
 	      "<span title=\"Extended regular expression\">" .
 	      $cgi->checkbox(-name => 'sr', -value => 1, -label => 're',
 	                     -checked => $search_use_regexp) .
@@ -5282,7 +5282,7 @@ sub git_project_list_body {
 
 	my $check_forks = gitweb_check_feature('forks');
 	my $show_ctags  = gitweb_check_feature('ctags');
-	my $tagfilter = $show_ctags ? $cgi->param('by_tag') : undef;
+	my $tagfilter = $show_ctags ? $input_params{'ctag'} : undef;
 	$check_forks = undef
 		if ($tagfilter || $searchtext);
 
@@ -5994,7 +5994,7 @@ sub git_project_list {
 	}
 	print $cgi->startform(-method => "get") .
 	      "<p class=\"projsearch\">Search:\n" .
-	      $cgi->textfield(-name => "s", -value => $searchtext) . "\n" .
+	      $cgi->textfield(-name => "s", -value => $searchtext, -override => 1) . "\n" .
 	      "</p>" .
 	      $cgi->end_form() . "\n";
 	git_project_list_body(\@list, $order);
@@ -6197,7 +6197,7 @@ sub git_tag {
 
 sub git_blame_common {
 	my $format = shift || 'porcelain';
-	if ($format eq 'porcelain' && $cgi->param('js')) {
+	if ($format eq 'porcelain' && $input_params{'javascript'}) {
 		$format = 'incremental';
 		$action = 'blame_incremental'; # for page title etc
 	}

-- 
1.7.6
Previous: Michal KiedrowiczNext: Michał Kiedrowicz
Message 11 of 14 in “gitweb: use CGI with -utf8”
  1. gitweb: use CGI with -utf8Michał Kiedrowicz, Feb 1, 2012
  2. Jakub NarebskiFeb 2, 2012
  3. gitweb: use CGI with -utf8 to process Unicode query parameters correctlyJakub Narebski, Feb 2, 2012
  4. Jakub NarebskiFeb 2, 2012
  5. Michał KiedrowiczFeb 2, 2012
  6. gitweb: Allow UTF-8 encoded CGI query parameters and path_infoJakub Narebski, Feb 2, 2012
  7. Michał KiedrowiczFeb 2, 2012
  8. Jakub NarebskiFeb 2, 2012
  9. Jakub NarebskiFeb 2, 2012
  10. Michal KiedrowiczFeb 3, 2012
  11. gitweb: Allow UTF-8 encoded CGI query parameters and path_infoJakub Narebski, Feb 3, 2012
  12. Michał KiedrowiczFeb 3, 2012
  13. Junio C HamanoFeb 3, 2012
  14. Michał KiedrowiczFeb 2, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.