Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string

From: Junio C Hamano <gitster@pobox.com>
Date: May 2, 2018, 08:01 UTC
Message-ID: <xmqqtvrqv632.fsf@gitster-ct.c.googlers.com>
In-Reply-To: <20180501064015.59977-1-shin@kojima.org>

Shin Kojima <shin@kojima.org> writes:

Show 10 quoted lines

> Offset positions should not be counted by byte length, but by actual
> character length.
> ...
>  # escape tabs (convert tabs to spaces)
>  sub untabify {
> -	my $line = shift;
> +	my $line = to_utf8(shift);
>  
>  	while ((my $pos = index($line, "\t")) != -1) {
>  		if (my $count = (8 - ($pos % 8))) {

Some codepaths in the resulting codeflow look even hackier than they already are. For example, format_rem_add_lines_pair() calls untabify() and then feeds its result to esc_html(). The first thing done in esc_html() is to call to_utf8(). I know that to_utf8() cheats and leaves the incoming string as-is if it is already UTF-8, so this may be a safe conversion, but ideally we should be able to say "function X takes non-UTF8 and works on it", "function Y takes UTF8 and works on it", and "function Z takes non-UTF8 and gives UTF8 data back" for each functions clearly, not "function W can take either UTF8 or any other garbage and tries to return UTF8".

Also, does it even "fix" the problem to use to_utf8() here in the first place? Untabify is about aligning the character after a HT to multiple of 8 display position, so we'd want measure display width, which is not the same as either byte count or char count.