Re: Git-mediawiki : Encoding problems in perl

From: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Date: Jun 8, 2011, 15:58 UTC
Message-ID: <vpqtyc0wc1j.fsf@bauges.imag.fr>
In-Reply-To: <BANLkTi=DLZQM_jzove_g0wLy_zgBF6Z0Xw@mail.gmail.com>

Jérémie NIKAES <jeremie.nikaes@gmail.com> writes:

> 2011/6/8 Matthieu Moy <Matthieu.Moy@grenoble-inp.fr>:
>
>> open(my $git, "-|:encoding(UTF-8)", "git cat-file -p $sha1");

there should probably have been a $/ = 1; or some other perl magic to make sure we don't read only the first line there:

>> my $file_content = <$git>;
>> close($git);
>
> Yes, that did it ! Thank you boss :-)

Then, make it a helper function to call like

my $file_content = run_git("cat-file -p $sha1");

and use it where needed.

> Although, I'm still encountering issues regarding encoding file names
> though, I am going to look deeper now that i have this solution.

My advice, at least in the short-term (already discussed offline): use urlencode ( http://php.net/manual/en/function.urlencode.php ) on pull, and don't bother with encoding on push. Non-ascii characters in filenames are a nightmare ...

If you go for utf8 filenames, you should test that your script works in various environments, like

LANG=fr_FR.ISO-8859-1 xterm (launch a terminal with latin-1 encoding inside)

and Mac OS X (which does some weird utf-8-normalization on filenames), and probably windows (no idea how filename encoding works there).

If it doesn't work in one of them, you'll have to provide a fall-back to plain ascii for these users, which will most likely be the short-term solution I'm proposing.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/