[PATCH 4/3 v2 (bugfix)] gitweb: Fix fallback mode of to_utf8 subroutine

e5d3de5 (gitweb: use Perl built-in utf8 function for UTF-8 decoding., 2007-12-04) was meant to make gitweb faster by using Perl's internals (see subsection "Messing with Perl's Internals" in Encode(3pm) manpage)

Simple benchmark confirms that (old = 00f429a, new = this version): note that it is synthetic benchmark of standalone subroutines, not of gitweb itself (where probably no visible difference in performace will show)

        Rate  old  new
  old 1582/s   -- -64%
  new 4453/s 181%   --

Unfortunately it made fallback mode of to_utf8 do not work... except for default value 'latin1' of $fallback_encoding (because 'latin1' is Perl native encoding), which is probably why it was not noticed for so long.

utf8::valid(STRING) is an internal function that tests whether STRING is in a _consistent state_ regarding UTF-8. It returns true is well-formed UTF-8 and has the UTF-8 flag on _*or*_ if string is held as bytes (both these states are 'consistent').

For gitweb in most cases the second option was true, as output from git commands is opened without ':utf8' layer. So utf8::valid is not useful for to_utf8.

What made it look as if to_utf8() fallback mode worked correctly (though only for $fallback_encoding at its default value 'latin1') was the fact that utf8::decode(STRING) turns on UTF-8 flag only if source string^W octets form a valid UTF-8 and it contains multi-byte UTF-8 characters... this means that if string was not valid UTF-8 it didn't get UTF-8 flag.

When string doesn't have UTF-8 flag set, it is treated as if it was in native Perl encoding, which is 'latin1' (unless native encoding is EBCDIC ;-)). So it was ':utf8' layer that actually converted 'latin1' (no UTF-8 flag == native == 'latin1) to 'utf8', and not to_utf8() subroutine. Fallback mode was never triggered.

Let's make use of the fact that utf8::decode(STRING) returns false if STRING is invalid as UTF-8 to check whether to enable fallback mode.

Note however that if STRING has UTF-8 flag set already, then utf8::decode also returns false, which could cause problems if given string was already converted with to_utf8(). Such double conversion can happen in gitweb. Therefore we have to check if STRING has UTF-8 flag set with utf8::is_utf8(); if this subroutine returns true then we have already decoded (converted) string, and don't have to do it second time.

Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
Jakub Narebski wrote:

Now it is fixed; the error was caused by to_utf8 not dealing with double encoding for strings outside 7bit ASCII.

 gitweb/gitweb.perl |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)