{"thread":{"id":"27579","subject":"Git-mediawiki : Encoding problems in perl","startedAt":"2011-06-08T13:45:43Z","lastAt":"2011-06-08T22:36:58Z","messageCount":20,"participants":["Jérémie NIKAES","Steffen Daode Nurpmeso","Jeff King","Matthieu Moy","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"169609","messageId":"BANLkTimy85b3nu05FBjXzdnTJP0RBWdxiQ@mail.gmail.com","threadId":"27579","inReplyTo":null,"subject":"Git-mediawiki : Encoding problems in perl","fromName":"Jérémie NIKAES","fromEmail":"jeremie.nikaes@gmail.com","sentAt":"2011-06-08T13:45:43Z","receivedAt":"2011-06-08T13:45:43Z","isPatch":false,"sender":{"key":"jeremie.nikaes@gmail.com","avatar":null},"body":"Hi,\nWhile working on the git-mediawiki project[1], we ran into some\nproblems regarding utf8 encoding of files. Most of them have been\nsolved, however, one is still pretty annoying.\nLet me illustrate it :\n\nI want to edit a page on mediawiki using the API, with a very simple example :\n\nmy $mw = MediaWiki::API->new();\n$mw->edit( {\n        action => 'edit',\n        title => 'Main_page',\n        text => 'été',\n} ) ;\n\nBut, when I look at the page on mediawiki, I see weird characters : été.\n\nI tried text => encode_utf8('été') with no success.\n\nThis makes pushing changes from git to mediawiki buggy since pulling a\nfile with accentuated characters and pushing it right after changes\nthings on the wiki.\n\nWhile googling (a lot), I found that utf8 was pretty tricky in perl...\nThe only thing that seems to solve things is a simple addition of 'use\nencoding utf8' at the top of our script.\nHowever\nA) Adding this line requires that I remove 'use strict;'\nB) I found some information about this pragma encoding and it seems to\nbe unadvised to use it\n\nDo you have any information regarding this issue ?\n\nThanks,\n-- \nJérémie Nikaes\n[1] https://github.com/Bibzball/Git-Mediawiki\n"},{"id":"169613","messageId":"20110608143726.GA72373@sherwood.local","threadId":"27579","inReplyTo":"BANLkTimy85b3nu05FBjXzdnTJP0RBWdxiQ@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Steffen Daode Nurpmeso","fromEmail":"sdaoden@googlemail.com","sentAt":"2011-06-08T14:37:26Z","receivedAt":"2011-06-08T14:37:26Z","isPatch":false,"sender":{"key":"sdaoden@googlemail.com","avatar":null},"body":"@ Jérémie NIKAES <jeremie.nikaes@gmail.com> wrote (2011-06-08 15:45+0200):\n> But, when I look at the page on mediawiki, I see weird characters : été.\n> \n> I tried text => encode_utf8('été') with no success.\n> \n> Do you have any information regarding this issue ?\n\nI'm not a Perl guru, but i ran into the very same problem when\nwriting a disc-ripper/CDDB lookup/DB writer, and the following\nsnippet helped me out:\n\n    $CDDB{TITLES} = $dinf->{ttitles};\n    foreach (@{$dinf->{ttitles}}) {\n        s/^\\s*(.*?)\\s*$/$1/;\n        my $save = $_;\n        eval { Encode::from_to($_, 'iso-8859-1', 'utf-8'); };\n        $_ = $save if $@;\n        Encode::_utf8_off($_);\n    }\n\nI forget the exact circumstances, but as far as i remember you\nneed to trigger the is-UTF-8 bit on the string object in an\ndiscouraged (acc. to manual) way to make it work the way it\nshould.\n--\nCiao, Steffen\nsdaoden(*)(gmail.com)\n() ascii ribbon campaign - against html e-mail\n/\\ www.asciiribbon.org - against proprietary attachments\n"},{"id":"169615","messageId":"20110608150106.GB7805@sigill.intra.peff.net","threadId":"27579","inReplyTo":"BANLkTimy85b3nu05FBjXzdnTJP0RBWdxiQ@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-08T15:01:06Z","receivedAt":"2011-06-08T15:01:06Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 08, 2011 at 03:45:43PM +0200, Jérémie NIKAES wrote:\n\n> my $mw = MediaWiki::API->new();\n> $mw->edit( {\n>         action => 'edit',\n>         title => 'Main_page',\n>         text => 'été',\n> } ) ;\n> [...]\n> While googling (a lot), I found that utf8 was pretty tricky in perl...\n> The only thing that seems to solve things is a simple addition of 'use\n> encoding utf8' at the top of our script.\n> However\n> A) Adding this line requires that I remove 'use strict;'\n> B) I found some information about this pragma encoding and it seems to\n> be unadvised to use it\n\nFrom the \"utf8\" man page:\n\n  Do not use this pragma for anything else than telling Perl that your\n  script is written in UTF-8.\n\nwhich is what you are doing here, since you are telling perl that the\nstring constant is in utf8. So from my understanding, \"use utf8\" is the\nright solution.\n\nThat being said, this is probably just a small test case, and you are\nmore likely to be reading the data from a file.\n\nFor file contents, you can use:\n\n  binmode($handle, \":utf8\");\n\nto read everything in as utf8.\n\nFor file names themselves, I think it depends where you get them.\nPresumably from readdir() or from a glob. I think you can use\nutf8::upgrade($string) on the result to make sure they are interpreted\nas utf8 (if you already know that is how the bytes in the filename\nshould be interpreted).\n\nBut I admit I am not an expert on such matters, and every time I do utf8\nthings in perl, I end up with a lot of trial and error.\n\n-Peff\n"},{"id":"169624","messageId":"vpqhb8049m3.fsf@bauges.imag.fr","threadId":"27579","inReplyTo":"20110608150106.GB7805@sigill.intra.peff.net","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-06-08T15:37:56Z","receivedAt":"2011-06-08T15:37:56Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Wed, Jun 08, 2011 at 03:45:43PM +0200, Jérémie NIKAES wrote:\n>\n>> my $mw = MediaWiki::API->new();\n>> $mw->edit( {\n>>         action => 'edit',\n>>         title => 'Main_page',\n>>         text => 'été',\n>> } ) ;\n>> [...]\n[...]\n>>From the \"utf8\" man page:\n>\n>   Do not use this pragma for anything else than telling Perl that your\n>   script is written in UTF-8.\n>\n> which is what you are doing here,\n\nActually, this is what the example does, but this is not where the\noriginal problem comes from. The code of git-remote-mediawiki contains\nonly us-ascii characters.\n\nThe actual code is:\n\n    my $file_content = `git cat-file -p $sha1`;\n    chomp($file_content);\n    // ...\n    $mw->edit( {\n    \taction => 'edit',\n    \tsummary => $_[1],\n    \ttitle => $title,\n    \ttext => $file_content,\n    });\n\nIf the file is UTF-8 encoded, the page sent to the wiki is\ndouble-utf8-encoded.\n\n> That being said, this is probably just a small test case, and you are\n> more likely to be reading the data from a file.\n\nOops, read this too late ;-).\n\n> For file contents, you can use:\n>\n>   binmode($handle, \":utf8\");\n>\n> to read everything in as utf8.\n\nThat's not exactly it, since we read the output of \"git cat-file\", not\nan actual file.\n\nBut something along the lines of:\n\nopen(my $git, \"-|:encoding(UTF-8)\", \"git cat-file -p $sha1\");\nmy $file_content = <$git>;\nclose($git);\n\nmay do it.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"169625","messageId":"20110608154513.GB8759@sigill.intra.peff.net","threadId":"27579","inReplyTo":"vpqhb8049m3.fsf@bauges.imag.fr","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-08T15:45:13Z","receivedAt":"2011-06-08T15:45:13Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 08, 2011 at 05:37:56PM +0200, Matthieu Moy wrote:\n\n> The actual code is:\n> \n>     my $file_content = `git cat-file -p $sha1`;\n>     chomp($file_content);\n>     // ...\n>     $mw->edit( {\n>     \taction => 'edit',\n>     \tsummary => $_[1],\n>     \ttitle => $title,\n>     \ttext => $file_content,\n>     });\n> \n> If the file is UTF-8 encoded, the page sent to the wiki is\n> double-utf8-encoded.\n\nI think there might be a way to say \"all descriptors are utf8\"; I don't\nknow if that would apply to pipes from backtick commands or not. I'm\nalso not sure if it would interfere with something like Mediawiki::API\ntalking over the socket.\n\n> That's not exactly it, since we read the output of \"git cat-file\", not\n> an actual file.\n> \n> But something along the lines of:\n> \n> open(my $git, \"-|:encoding(UTF-8)\", \"git cat-file -p $sha1\");\n> my $file_content = <$git>;\n> close($git);\n> \n> may do it.\n\nYeah, I think that's the cleanest solution (assuming it works. :) ).\n\n-Peff\n"},{"id":"169626","messageId":"BANLkTi=DLZQM_jzove_g0wLy_zgBF6Z0Xw@mail.gmail.com","threadId":"27579","inReplyTo":"vpqhb8049m3.fsf@bauges.imag.fr","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jérémie NIKAES","fromEmail":"jeremie.nikaes@gmail.com","sentAt":"2011-06-08T15:46:08Z","receivedAt":"2011-06-08T15:46:08Z","isPatch":false,"sender":{"key":"jeremie.nikaes@gmail.com","avatar":null},"body":"2011/6/8 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:\n\n> open(my $git, \"-|:encoding(UTF-8)\", \"git cat-file -p $sha1\");\n> my $file_content = <$git>;\n> close($git);\n\nYes, that did it ! Thank you boss :-)\nAnd thanks to both of you Peff & Steffen for your suggestions.\n\nAlthough, I'm still encountering issues regarding encoding file names\nthough, I am going to look deeper now that i have this solution.\n\n-- \nJérémie Nikaes\nÉlève ingénieur en deuxième année à l'Ensimag\nIngénierie des Systèmes d'Informations\nTel : +33 (0)6 12 99 78 75\nMail : jeremie.nikaes@gmail.com\n"},{"id":"169627","messageId":"vpqtyc0wc1j.fsf@bauges.imag.fr","threadId":"27579","inReplyTo":"BANLkTi=DLZQM_jzove_g0wLy_zgBF6Z0Xw@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-06-08T15:58:00Z","receivedAt":"2011-06-08T15:58:00Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jérémie NIKAES <jeremie.nikaes@gmail.com> writes:\n\n> 2011/6/8 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:\n>\n>> open(my $git, \"-|:encoding(UTF-8)\", \"git cat-file -p $sha1\");\n\nthere should probably have been a $/ = 1; or some other perl magic to\nmake sure we don't read only the first line there:\n\n>> my $file_content = <$git>;\n>> close($git);\n>\n> Yes, that did it ! Thank you boss :-)\n\nThen, make it a helper function to call like\n\nmy $file_content = run_git(\"cat-file -p $sha1\");\n\nand use it where needed.\n\n> Although, I'm still encountering issues regarding encoding file names\n> though, I am going to look deeper now that i have this solution.\n\nMy advice, at least in the short-term (already discussed offline): use\nurlencode ( http://php.net/manual/en/function.urlencode.php ) on pull,\nand don't bother with encoding on push. Non-ascii characters in\nfilenames are a nightmare ...\n\nIf you go for utf8 filenames, you should test that your script works in\nvarious environments, like\n\nLANG=fr_FR.ISO-8859-1 xterm\n(launch a terminal with latin-1 encoding inside)\n\nand Mac OS X (which does some weird utf-8-normalization on filenames),\nand probably windows (no idea how filename encoding works there).\n\nIf it doesn't work in one of them, you'll have to provide a fall-back to\nplain ascii for these users, which will most likely be the short-term\nsolution I'm proposing.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"169629","messageId":"BANLkTinKV3k7uu=+QzvUjrLdRD0TUj67pw@mail.gmail.com","threadId":"27579","inReplyTo":"vpqtyc0wc1j.fsf@bauges.imag.fr","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jérémie NIKAES","fromEmail":"jeremie.nikaes@gmail.com","sentAt":"2011-06-08T16:15:15Z","receivedAt":"2011-06-08T16:15:15Z","isPatch":false,"sender":{"key":"jeremie.nikaes@gmail.com","avatar":null},"body":"2011/6/8 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:\n\n> there should probably have been a $/ = 1; or some other perl magic to\n> make sure we don't read only the first line there:\n>\n\nYes, it indeed currently reads only the first line. I'm going to see\nwhat kind of magic I need to use.\n\n> Then, make it a helper function to call like\n>\n> my $file_content = run_git(\"cat-file -p $sha1\");\n>\n> and use it where needed.\n\nGood idea, doing it right now\n\n\n> My advice, at least in the short-term (already discussed offline): use\n> urlencode ( http://php.net/manual/en/function.urlencode.php ) on pull,\n> and don't bother with encoding on push. Non-ascii characters in\n> filenames are a nightmare ...\n>\n\nYes I tried uri_escape, but that only works in the direction mediawiki -> git.\nA page named \"Eté\" on mediawiki comes as a Et%C3%A9.mw file on the repo.\nHowever, when I try to send that file \"Et%C3%A9\" with the mediawiki\nAPI, I get this error\n\n\"Can't use an undefined value as a HASH reference at\n/usr/local/share/perl/5.10.1/MediaWiki/API.pm line 554.\"\n\nSo I tried to backslash the '%' but it does not do it...\nAny idea ? Thanks\n\n--\nJérémie Nikaes\n"},{"id":"169630","messageId":"20110608161816.GA13175@sigill.intra.peff.net","threadId":"27579","inReplyTo":"BANLkTinKV3k7uu=+QzvUjrLdRD0TUj67pw@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-08T16:18:17Z","receivedAt":"2011-06-08T16:18:17Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 08, 2011 at 06:15:15PM +0200, Jérémie NIKAES wrote:\n\n> 2011/6/8 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:\n> \n> > there should probably have been a $/ = 1; or some other perl magic to\n> > make sure we don't read only the first line there:\n> >\n> \n> Yes, it indeed currently reads only the first line. I'm going to see\n> what kind of magic I need to use.\n\nYou need to set $/ to undef. Use \"local\" to prevent it from polluting\nother parts of the code, like:\n\n  my $var = do { local $/; <$handle> };\n\n-Peff\n"},{"id":"169631","messageId":"BANLkTinU0Rj_xqwV4CivdrL8r__rU+wZew@mail.gmail.com","threadId":"27579","inReplyTo":"20110608161816.GA13175@sigill.intra.peff.net","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jérémie NIKAES","fromEmail":"jeremie.nikaes@gmail.com","sentAt":"2011-06-08T16:26:25Z","receivedAt":"2011-06-08T16:26:25Z","isPatch":false,"sender":{"key":"jeremie.nikaes@gmail.com","avatar":null},"body":"2011/6/8 Jeff King <peff@peff.net>:\n\n> You need to set $/ to undef. Use \"local\" to prevent it from polluting\n> other parts of the code, like:\n>\n>  my $var = do { local $/; <$handle> };\n>\n> -Peff\n>\n\nWorked as a charm. Thanks a lot.\n-- \nJérémie Nikaes\n"},{"id":"169632","messageId":"vpqmxhss2yw.fsf@bauges.imag.fr","threadId":"27579","inReplyTo":"BANLkTinKV3k7uu=+QzvUjrLdRD0TUj67pw@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-06-08T16:27:35Z","receivedAt":"2011-06-08T16:27:35Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jérémie NIKAES <jeremie.nikaes@gmail.com> writes:\n\n> Yes I tried uri_escape, but that only works in the direction mediawiki -> git.\n> A page named \"Eté\" on mediawiki comes as a Et%C3%A9.mw file on the repo.\n> However, when I try to send that file \"Et%C3%A9\" with the mediawiki\n> API, I get this error\n>\n> \"Can't use an undefined value as a HASH reference at\n> /usr/local/share/perl/5.10.1/MediaWiki/API.pm line 554.\"\n>\n> So I tried to backslash the '%' but it does not do it...\n\nWhat if you uri_unescape before sending to MediaWiki?\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"169634","messageId":"BANLkTim2W8FxMTLyYE5+iRQWseZ6H_oZjA@mail.gmail.com","threadId":"27579","inReplyTo":"vpqmxhss2yw.fsf@bauges.imag.fr","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jérémie NIKAES","fromEmail":"jeremie.nikaes@gmail.com","sentAt":"2011-06-08T16:30:38Z","receivedAt":"2011-06-08T16:30:38Z","isPatch":false,"sender":{"key":"jeremie.nikaes@gmail.com","avatar":null},"body":"2011/6/8 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:\n> Jérémie NIKAES <jeremie.nikaes@gmail.com> writes:\n>\n>> Yes I tried uri_escape, but that only works in the direction mediawiki -> git.\n>> A page named \"Eté\" on mediawiki comes as a Et%C3%A9.mw file on the repo.\n>> However, when I try to send that file \"Et%C3%A9\" with the mediawiki\n>> API, I get this error\n>>\n>> \"Can't use an undefined value as a HASH reference at\n>> /usr/local/share/perl/5.10.1/MediaWiki/API.pm line 554.\"\n>>\n>> So I tried to backslash the '%' but it does not do it...\n>\n> What if you uri_unescape before sending to MediaWiki?\n\nSame problem, same error. Unfortunately.\n-- \nJérémie Nikaes\n"},{"id":"169638","messageId":"m3y61cdzmi.fsf@localhost.localdomain","threadId":"27579","inReplyTo":"BANLkTimy85b3nu05FBjXzdnTJP0RBWdxiQ@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-06-08T17:04:17Z","receivedAt":"2011-06-08T17:04:17Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jérémie NIKAES <jeremie.nikaes@gmail.com> writes:\n\n> While working on the git-mediawiki project[1], we ran into some\n> problems regarding utf8 encoding of files. Most of them have been\n> solved, however, one is still pretty annoying.\n> Let me illustrate it :\n> \n> I want to edit a page on mediawiki using the API, with a very simple example :\n> \n> my $mw = MediaWiki::API->new();\n> $mw->edit( {\n>         action => 'edit',\n>         title => 'Main_page',\n>         text => 'été',\n> } ) ;\n> \n> But, when I look at the page on mediawiki, I see weird characters : été.\n\nTake a look at\n\n  http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default\n\nespecially accepted answer.\n\n\nIn short (I don't agree with everything there, and not everything is\nneeded for all but extremal Unicode usage): if your script is written\nusing UTF-8 like in above examples, use\n\n  use utf8;\n\nIf this is simplification, and this text comes from other file or is\nresult of output of some command, use\n\n  use utf8::all;\n\nor take a look what it does and put relevant parts in your script.\n \n> I tried text => encode_utf8('été') with no success.\n> \n> This makes pushing changes from git to mediawiki buggy since pulling a\n> file with accentuated characters and pushing it right after changes\n> things on the wiki.\n> \n> While googling (a lot), I found that utf8 was pretty tricky in perl...\n> The only thing that seems to solve things is a simple addition of 'use\n> encoding utf8' at the top of our script.\n> However\n> A) Adding this line requires that I remove 'use strict;'\n\n  use encoding ':utf8';\n\nor\n\n  use encoding 'utf8';\n\n> B) I found some information about this pragma encoding and it seems to\n> be unadvised to use it\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"169640","messageId":"m3tyc0dzi1.fsf@localhost.localdomain","threadId":"27579","inReplyTo":"BANLkTinKV3k7uu=+QzvUjrLdRD0TUj67pw@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-06-08T17:07:17Z","receivedAt":"2011-06-08T17:07:17Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jérémie NIKAES <jeremie.nikaes@gmail.com> writes:\n\n> 2011/6/8 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:\n\n[...]\n\n> > My advice, at least in the short-term (already discussed offline): use\n> > urlencode ( http://php.net/manual/en/function.urlencode.php ) on pull,\n> > and don't bother with encoding on push. Non-ascii characters in\n> > filenames are a nightmare ...\n> >\n> \n> Yes I tried uri_escape, but that only works in the direction mediawiki -> git.\n> A page named \"Eté\" on mediawiki comes as a Et%C3%A9.mw file on the repo.\n> However, when I try to send that file \"Et%C3%A9\" with the mediawiki\n> API, I get this error\n> \n> \"Can't use an undefined value as a HASH reference at\n> /usr/local/share/perl/5.10.1/MediaWiki/API.pm line 554.\"\n\nCan you show us this line end its neighourhood?\n \nIt might be bug in MediaWiki::API...\n\n> So I tried to backslash the '%' but it does not do it...\n\nDecode if from URI encoding to UTF-8 and mark as UTF-8 before sending\nto mediawiki API.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"169642","messageId":"vpqsjrkqmdj.fsf@bauges.imag.fr","threadId":"27579","inReplyTo":"BANLkTinKV3k7uu=+QzvUjrLdRD0TUj67pw@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-06-08T17:11:20Z","receivedAt":"2011-06-08T17:11:20Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jérémie NIKAES <jeremie.nikaes@gmail.com> writes:\n\n> Yes I tried uri_escape, but that only works in the direction mediawiki -> git.\n> A page named \"Eté\" on mediawiki comes as a Et%C3%A9.mw file on the repo.\n> However, when I try to send that file \"Et%C3%A9\" with the mediawiki\n> API, I get this error\n>\n> \"Can't use an undefined value as a HASH reference at\n> /usr/local/share/perl/5.10.1/MediaWiki/API.pm line 554.\"\n>\n> So I tried to backslash the '%' but it does not do it...\n> Any idea ? Thanks\n\nOK, I know that's cheating, but reading the doc helped ;-)\n\nhttp://search.cpan.org/~exobuzz/MediaWiki-API-0.35/lib/MediaWiki/API.pm#MediaWiki::API-%3Eedit%28_$query_hashref,_$options_hashref_%29\n\n  The options hashref currently has one optional parameter (skip_encoding\n  => 1). This is described above in the MediaWiki::API->api call\n  documentation.\n  \nwhich leads us to:\n\n  MediaWiki's API uses UTF-8 and any 8 bit character string parameters are\n  encoded automatically by the API call. If your parameters are already in\n  UTF-8 this will be detected and the encoding will be skipped. If your\n  parameters for some reason contain UTF-8 data but no UTF-8 flag is set\n  (i.e. you did not use the \"use utf8;\" pragma) you should prevent\n  re-encoding by passing an option skip_encoding => 1 in the\n  $options_hash.\n\nIn other words, Perl and MediaWiki::API use some black magic to detect\nUTF-8, and you want to disable it like this:\n\n$mw->edit( {\n    action => 'edit',\n    title => $title,\n    text => $text,\n}, {\n    skip_encoding => 1\n} ) || die $mw->{error}->{code} . ': ' . $mw->{error}->{details};\n\nTried it, worked :-).\n\nThis may well be an alternative solution to the earlier UTF-8 problem.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"169652","messageId":"BANLkTikZGX4A66r6ubUF6P60JNu_BMgMyw@mail.gmail.com","threadId":"27579","inReplyTo":"m3y61cdzmi.fsf@localhost.localdomain","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jérémie NIKAES","fromEmail":"jeremie.nikaes@gmail.com","sentAt":"2011-06-08T17:59:32Z","receivedAt":"2011-06-08T17:59:32Z","isPatch":false,"sender":{"key":"jeremie.nikaes@gmail.com","avatar":null},"body":"2011/6/8 Jakub Narebski <jnareb@gmail.com>:\n\n>  http://stackoverflow.com/questions/6162484/why-does-modern-perl-avoid-utf-8-by-default\n>\n> especially accepted answer.\n\nThanks for the link, do have some interesting things in here.\n\n>  use encoding ':utf8';\n>\n> or\n>\n>  use encoding 'utf8';\n\nThat did it while keeping the use strict; intact.\nThanks a lot\n\n-- \nJérémie Nikaes\n"},{"id":"169653","messageId":"BANLkTinE9+1cVk09KCP-iW=+GFbRdDAZFg@mail.gmail.com","threadId":"27579","inReplyTo":"vpqsjrkqmdj.fsf@bauges.imag.fr","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jérémie NIKAES","fromEmail":"jeremie.nikaes@gmail.com","sentAt":"2011-06-08T18:03:26Z","receivedAt":"2011-06-08T18:03:26Z","isPatch":false,"sender":{"key":"jeremie.nikaes@gmail.com","avatar":null},"body":"2011/6/8 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:\n\n> $mw->edit( {\n>    action => 'edit',\n>    title => $title,\n>    text => $text,\n> }, {\n>    skip_encoding => 1\n> } ) || die $mw->{error}->{code} . ': ' . $mw->{error}->{details};\n>\n> Tried it, worked :-).\n>\nYep this works if you manually set your $title variable earlier in the\ncode. However, I still have the problem which I think is on the git\nside\n\n- I pull the \"Eté.mw\" file from mediawiki\n- I edit it\n- When I commit it I get this message from git :\n[master sha1] commit message\n1 files changed [...]\ncreate mode 100644 \"Bl\\303\\251.mw\"\n\nAs a result, when I parse commit information, the title of the file is\nindeed Bl\\303\\251... so a new page is created on the mediawiki.\n\n-- \nJérémie Nikaes\n"},{"id":"169655","messageId":"vpqboy8i3rz.fsf@bauges.imag.fr","threadId":"27579","inReplyTo":"BANLkTinE9+1cVk09KCP-iW=+GFbRdDAZFg@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-06-08T18:20:16Z","receivedAt":"2011-06-08T18:20:16Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jérémie NIKAES <jeremie.nikaes@gmail.com> writes:\n\n> Yep this works if you manually set your $title variable earlier in the\n> code.\n\nOr if you uri_escape the file on pull.\n\n> However, I still have the problem which I think is on the git side\n>\n> - I pull the \"Eté.mw\" file from mediawiki\n> - I edit it\n> - When I commit it I get this message from git :\n> [master sha1] commit message\n> 1 files changed [...]\n> create mode 100644 \"Bl\\303\\251.mw\"\n\nI guess you mean \"Et\\303\\251.mw\" ?\n\n> As a result, when I parse commit information, the title of the file is\n> indeed Bl\\303\\251...\n\nThe -z option of many Git commands is your friend.\n\nEspecially the one of \"git diff --raw\".\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"169701","messageId":"20110608215130.GA13021@sigill.intra.peff.net","threadId":"27579","inReplyTo":"BANLkTinE9+1cVk09KCP-iW=+GFbRdDAZFg@mail.gmail.com","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-06-08T21:51:32Z","receivedAt":"2011-06-08T21:51:32Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 08, 2011 at 08:03:26PM +0200, Jérémie NIKAES wrote:\n\n> - I pull the \"Eté.mw\" file from mediawiki\n> - I edit it\n> - When I commit it I get this message from git :\n> [master sha1] commit message\n> 1 files changed [...]\n> create mode 100644 \"Bl\\303\\251.mw\"\n> \n> As a result, when I parse commit information, the title of the file is\n> indeed Bl\\303\\251... so a new page is created on the mediawiki.\n\nIck. I hope you aren't parsing the output of \"git commit\"; it's not\nguaranteed to be stable.\n\nBut if you are parsing \"diff\", then yes, filenames with high-bit\ncharacters (or special characters like tab or double-quote) may be\nquoted C-style, and you should be unquoting them. Or, as Matthieu\nsuggested, use \"-z\" to get a NUL-terminated, non-quoted version.\n\n-Peff\n"},{"id":"169709","messageId":"BANLkTikTDPmvj4oepiJdFj9EpSTNpS_Prg@mail.gmail.com","threadId":"27579","inReplyTo":"20110608215130.GA13021@sigill.intra.peff.net","subject":"Re: Git-mediawiki : Encoding problems in perl","fromName":"Jérémie NIKAES","fromEmail":"jeremie.nikaes@gmail.com","sentAt":"2011-06-08T22:36:58Z","receivedAt":"2011-06-08T22:36:58Z","isPatch":false,"sender":{"key":"jeremie.nikaes@gmail.com","avatar":null},"body":"2011/6/8 Jeff King <peff@peff.net>:\n>\n> But if you are parsing \"diff\", then yes, filenames with high-bit\n> characters (or special characters like tab or double-quote) may be\n> quoted C-style, and you should be unquoting them. Or, as Matthieu\n> suggested, use \"-z\" to get a NUL-terminated, non-quoted version.\n\nYes, we are parsing \"diff\". The -z helped a lot with non-iso characters.\nEverything seems to be working fine now without using uri_escape. The\nproblem is, as Matthieu said, different file systems may handle these\ncharacters in file names differently, so in the long run it could be\nbetter to use uri_escape / uri_unescape.\n\nThe problem I run into now is that we are using\nuse encoding 'utf-8'\nas Jakub suggested.\nUsing this mode, when you uri_escape and uri_unescape a string, you\ndon't get the original string. I must be missing something but my head\nis kind of fuzzy with all the different existing methods to encode\nthings in utf8 and it is getting pretty late.\n\nThanks a lot to everyone who helped today, a RFC patch should follow tomorrow.\n\n-- \nJérémie Nikaes\n"}]}