[RFD] Handling of non-UTF8 data in gitweb

Hello!

Currently gitweb converts data it receives from git commands to Perl internal utf8 representation via to_utf8() subroutine

 # decode sequences of octets in utf8 into Perl's internal form,
 # which is utf-8 with utf8 flag set if needed.  gitweb writes out
 # in utf-8 thanks to "binmode STDOUT, ':utf8'" at beginning
 sub to_utf8 {
 	my $str = shift;
 	return undef unless defined $str;
 	if (utf8::valid($str)) {
 		utf8::decode($str);
 		return $str;
 	} else {
 		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 	}
 }

Each part of data must be handled separately. It is quite error prone process, as can be seen from quite a number of patches that fix handling of UTF-8 data (latest from Jürgen).

Much, much simpler would be to force opening of all files (including output pipes from git commands) in ':utf8' mode:

  use open qw(:std :utf8);

[Note: perhaps instead of ':utf8' it should be ':encoding(UTF-8)' 
 there...]

But doing this would change gitweb behavior. Currently when encountering something (usually line of output) that is not valid UTF-8, we decode it (to UTF-8) using $fallback_encoding, by default 'latin1'. Note however that this value is per gitweb installation, not per repository.

Using "use open qw(:std :utf8);" would be like changing the value of $fallback_encoding to 'utf8' -- errors would be ignored, and characters which are invalid in UTF-8 encoding would get replaced[1] with substitution character '�' U+FFFD.

Though at least for HTML output we could use Encode::FB_HTMLCREF handling (which would produce &#NNN;) or Encode::FB_XMLCREF (which would produce &#xHHHH;), though this must be done after HTML escaping... and is probaby not worth it (FYI this can be done by setting $PerlIO::encoding::fallback to either of those values[2])

[1] http://perldoc.perl.org/Encode.html#Handling-Malformed-Data
    http://p3rl.org/Encode
[2] http://perldoc.perl.org/PerlIO/encoding.html
    http://p3rl.org/PerlIO::encoding

I don't know if people are relying on the old behavior. I guess it could be emulated by defining our own 'utf-8-with-fallback' encoding, or by defining our own PerlIO layer with PerlIO::via. But it no longer be simple solution (though still automatic).

Alternate approach would be to audit gitweb code, and call to_utf8 before storing extracted output of git command in variable (excluding save types like SHA-1, filemode, timestamp and timezone). The fact that to_utf8 is idempotent and can be called multiple times would help here, I think.

The correct solution would be of course to respect `gui.encoding` per-repository config variable, and `encoding` gitattribute... though the latter is hampered by the fact that there is currently no way to read attribute with "git check-attr" from a given tree: think of a diff of change of encoding of a file!