Re: [RFD] Handling of non-UTF8 data in gitweb

From: Junio C Hamano <gitster@pobox.com>
Date: Dec 7, 2011, 00:37 UTC
Message-ID: <7vehwhcj3q.fsf@alter.siamese.dyndns.org>
In-Reply-To: <201112041709.32212.jnareb@gmail.com>

Jakub Narebski <jnareb@gmail.com> writes:

Show 5 quoted lines

> But doing this would change gitweb behavior.  Currently when 
> encountering something (usually line of output) that is not valid 
> UTF-8, we decode it (to UTF-8) using $fallback_encoding, by default
> 'latin1'.  Note however that this value is per gitweb installation,
> not per repository.

I think we added and you acked 00f429a (gitweb: Handle non UTF-8 text better, 2007-06-03) for a good reason, and I think the above argues that whatever issue the commit tried to address is a non-issue. Is it really true?

> ... I guess
> it could be emulated by defining our own 'utf-8-with-fallback'
> encoding, or by defining our own PerlIO layer with PerlIO::via.
> But it no longer be simple solution (though still automatic).

Between the current "everybody who read from the input must remember to call to_utf8" and "everybody gets to read utf8 automatically for internal processing", even though the latter may involve one-time pain to set up the framework to do so, the pros-and-cons feels obvious to me.