Re: Git-mediawiki : Encoding problems in perl

From: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Date: Jun 8, 2011, 15:37 UTC
Message-ID: <vpqhb8049m3.fsf@bauges.imag.fr>
In-Reply-To: <20110608150106.GB7805@sigill.intra.peff.net>

Jeff King <peff@peff.net> writes:

Show 9 quoted lines

> On Wed, Jun 08, 2011 at 03:45:43PM +0200, Jérémie NIKAES wrote:
>
>> my $mw = MediaWiki::API->new();
>> $mw->edit( {
>>         action => 'edit',
>>         title => 'Main_page',
>>         text => 'été',
>> } ) ;
>> [...]

[...]

Show 6 quoted lines

>>From the "utf8" man page:
>
>   Do not use this pragma for anything else than telling Perl that your
>   script is written in UTF-8.
>
> which is what you are doing here,

Actually, this is what the example does, but this is not where the original problem comes from. The code of git-remote-mediawiki contains only us-ascii characters.

The actual code is:

    my $file_content = `git cat-file -p $sha1`;
    chomp($file_content);
    // ...
    $mw->edit( {
    	action => 'edit',
    	summary => $_[1],
    	title => $title,
    	text => $file_content,
    });

If the file is UTF-8 encoded, the page sent to the wiki is double-utf8-encoded.

> That being said, this is probably just a small test case, and you are
> more likely to be reading the data from a file.

Oops, read this too late ;-).

Show 5 quoted lines

> For file contents, you can use:
>
>   binmode($handle, ":utf8");
>
> to read everything in as utf8.

That's not exactly it, since we read the output of "git cat-file", not an actual file.

But something along the lines of:

open(my $git, "-|:encoding(UTF-8)", "git cat-file -p $sha1"); my $file_content = <$git>; close($git);

may do it.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/