Re: Git-mediawiki : Encoding problems in perl

From: Jeff King <peff@peff.net>
Date: Jun 8, 2011, 15:01 UTC
Message-ID: <20110608150106.GB7805@sigill.intra.peff.net>
In-Reply-To: <BANLkTimy85b3nu05FBjXzdnTJP0RBWdxiQ@mail.gmail.com>

On Wed, Jun 08, 2011 at 03:45:43PM +0200, Jérémie NIKAES wrote:

Show 14 quoted lines

> my $mw = MediaWiki::API->new();
> $mw->edit( {
>         action => 'edit',
>         title => 'Main_page',
>         text => 'été',
> } ) ;
> [...]
> While googling (a lot), I found that utf8 was pretty tricky in perl...
> The only thing that seems to solve things is a simple addition of 'use
> encoding utf8' at the top of our script.
> However
> A) Adding this line requires that I remove 'use strict;'
> B) I found some information about this pragma encoding and it seems to
> be unadvised to use it

From the "utf8" man page:

  Do not use this pragma for anything else than telling Perl that your
  script is written in UTF-8.

which is what you are doing here, since you are telling perl that the string constant is in utf8. So from my understanding, "use utf8" is the right solution.

That being said, this is probably just a small test case, and you are more likely to be reading the data from a file.

For file contents, you can use:

  binmode($handle, ":utf8");

to read everything in as utf8.

For file names themselves, I think it depends where you get them. Presumably from readdir() or from a glob. I think you can use utf8::upgrade($string) on the result to make sure they are interpreted as utf8 (if you already know that is how the bytes in the filename should be interpreted).

But I admit I am not an expert on such matters, and every time I do utf8 things in perl, I end up with a lot of trial and error.

-Peff