Re: Fix UTF Encoding issue

From: Martin Koegler <mkoegler@auto.tuwien.ac.at>
Date: Dec 3, 2007, 16:38 UTC
Message-ID: <20071203163856.GA24269@auto.tuwien.ac.at>
In-Reply-To: <m3prxougmx.fsf@roke.D-201>

On Mon, Dec 03, 2007 at 04:06:48AM -0800, Jakub Narebski wrote:

Show 12 quoted lines

> Ismail Dönmez <ismail@pardus.org.tr> writes:
> > Monday 03 December 2007 Tarihinde 12:14:43 yazm??t?:
> >> Benjamin Close <Benjamin.Close@clearchain.com> writes:
> >>> -	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
> >>> -	if (defined $res) {
> >>> -		return $res;
> >>> -	} else {
> >>> -		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
> >>> -	}
> >>> +	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
> >>> +	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
> >>>  }

This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1 encoding of the same character sequence yields to different results.

Show 18 quoted lines

> >>
> >> I thought the standard catch ... throw idiom in Perl was to do the above
> >> like this:
> >>
> >> 	my $res;
> >>         eval { $res = decode_utf8($str, Encode::FB_CROAK); };
> >>         if ($@) {
> >>         	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
> >> 	}
> >> 	return $res;
> > 
> > I think this is correct, but the current code in gitweb doesn't look correct 
> > since it checks for $res and not $@.
> 
> First version of the patch was created by Martin Koegler. I have
> participated in creating the version which is now in gitweb, but I
> have to say that I wrote it based on decode_utf8
> documentation... which doesn't necessarily agree with facts :-(

eval { $res = decode_utf8(...); }
if ($@) 
     return decode(...);
return $res

or

eval { $res = decode_utf8(...); }
if (defined $res)
      return $res;
else
    return decode(...);

show the same (wrong) behaviour on Debian sarge. They do not always decode non UTF-8 characters correctly, eg. #öäü does not work #äöüä does work

On Debian etch, both versions are working.

> I'm all for the "throw idion" version. Ack.

mfg Martin Kögler