Re: gitweb and unicode special characters

From: Jakub Narebski <jnareb@gmail.com>
Date: Dec 12, 2008, 19:37 UTC
Message-ID: <m37i65gp6b.fsf@localhost.localdomain>
In-Reply-To: <3f2beab60812121033r5d41894t77acc271b7c6955c@mail.gmail.com>

"Praveen A" <pravi.a@gmail.com> writes:

Show 12 quoted lines

> Git currently does not handle unicode special characters ZWJ and ZWNJ,
> both are heavily used in Malayalam and common in other languages
> needing complex text layout like Sinhala and Arabic.
> 
> An example of this is shown in the commit message here
> http://git.savannah.gnu.org/gitweb/?p=smc.git;a=commit;h=c3f368c60aabdc380c77608c614d91b0a628590a
> 
> \20014 and \20015 should have been ZWNJ and ZWJ respectively. You just
> need to handle them as any other unicode character - especially it is
> a commit message and expectation is normal pain text display.
> 
> I hope some one will fix this.

Well, I am bit stumped. git_commit calls format_log_line_html, which in turn calls esc_html. esc_html looks like this:

  sub esc_html ($;%) {
  	my $str = shift;
  	my %opts = @_;
  
  **	$str = to_utf8($str);
  	$str = $cgi->escapeHTML($str);
  	if ($opts{'-nbsp'}) {
  		$str =~ s/ /&nbsp;/g;
  	}
  **	$str =~ s|([[:cntrl:]])|(($1 ne "\t") ? quot_cec($1) : $1)|eg;
  	return $str;
  }

The two important lines are marked with '**'. Not to_utf8 subroutine is very simple wrapper:

  # decode sequences of octets in utf8 into Perl's internal form,
  # which is utf-8 with utf8 flag set if needed.  gitweb writes out
  # in utf-8 thanks to "binmode STDOUT, ':utf8'" at beginning
  sub to_utf8 {
  	my $str = shift;
  	if (utf8::valid($str)) {
  		utf8::decode($str);
  		return $str;
  	} else {
  		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
  	}
  }

So it looks like Perl treats \20014 and \20015 (ZWNJ and ZWJ) as belonging to '[:cntrl:]' class. I don't know if it is correct from the point of view of Unicode character classes, therefore if it is a bug in Perl, or just in gitweb.

We might need protecting similar to ($1 ne "\t"), like (ord($1) < 127) or something... or perhaps we shouldn't use POSIX character class [:cntrl:] but something different when dealing with Unicode, e.g. \p{Cc} or \p{Control}, or perhaps \p{C} (other). I don't know Perl (nor Unicode) enough to decide...

P.S. Even that might not help much, as Savannah uses git and gitwev version 1.5.6.5, which is probably version released with some major distribution. As of now we are at 1.6.0.5...

-- 
Jakub Narebski
Poland
ShadeHawk on #git