{"thread":{"id":"14253","subject":"[PATCH/rfc] gitweb: open files (e.g. indextext.html) in utf8 mode","startedAt":"2008-07-02T12:13:17Z","lastAt":"2008-07-03T09:39:38Z","messageCount":3,"participants":["Gerrit Pape","Jakub Narebski","Lea Wiemann"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"81967","messageId":"20080702121317.10819.qmail@bca5b84cb0e0a0.315fe32.mid.smarden.org","threadId":"14253","inReplyTo":null,"subject":"[PATCH/rfc] gitweb: open files (e.g. indextext.html) in utf8 mode","fromName":"Gerrit Pape","fromEmail":"pape@smarden.org","sentAt":"2008-07-02T12:13:17Z","receivedAt":"2008-07-02T12:13:17Z","isPatch":true,"sender":{"key":"pape@smarden.org","avatar":"https://avatars.githubusercontent.com/u/143170252?v=4"},"body":"From: =?utf-8?q?Recai=20Okta=C5=9F?= \n\ngitweb used to use utf8 only in stdout. As a result, included files\nlike indextext.html appeared garbled if they contain utf8 characters.\nNow utf8 is also used when reading files.\n\nThe patch was submitted through\n http://bugs.debian.org/487465\n\nSigned-off-by: Gerrit Pape \n---\n gitweb/gitweb.perl | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 90cd99b..96cb4e0 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -16,7 +16,7 @@ use Encode;\n use Fcntl ':mode';\n use File::Find qw();\n use File::Basename qw(basename);\n-binmode STDOUT, ':utf8';\n+use open qw(:std :utf8);\n \n BEGIN {\n \tCGI->compile() if $ENV{'MOD_PERL'};\n-- \n1.5.6\n"},{"id":"81973","messageId":"m3prpwflus.fsf@localhost.localdomain","threadId":"14253","inReplyTo":"20080702121317.10819.qmail@bca5b84cb0e0a0.315fe32.mid.smarden.org","subject":"Re: [PATCH/rfc] gitweb: open files (e.g. indextext.html) in utf8 mode","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-07-02T13:37:38Z","receivedAt":"2008-07-02T13:37:38Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Gerrit Pape writes:\n\n> From: =?utf-8?q?Recai=20Okta=C5=9F?= \n\nYou don't need to use quoted-printable in 'From:' header embedded in\nthe mail body. It should probably read\n\n From: \"Recai Oktaş\" \n \n(provided that you can use utf-8 in email).\n\n> gitweb used to use utf8 only in stdout. As a result, included files\n> like indextext.html appeared garbled if they contain utf8 characters.\n> Now utf8 is also used when reading files.\n\nIt would better read as:\n\n Gitweb used to use utf8 mode only on STDOUT (actually \":utf8\" output\n layer), relying on using to_utf8(...) to convert input data from uft8\n to Perl internal form. As a result, included files such as $home_text\n (indextext.html in default build configuration), or repository's\n README.html appeared garbled if they did contain UTF-8 characters.\n\n Now uft8 mode is used for all open invovations, also when reading files.\n\n> The patch was submitted through\n> http://bugs.debian.org/487465\n> \n\nProbably should have here\n\n Reported-by: Recai Oktaş \n> Signed-off-by: Gerrit Pape \n> ---\n> gitweb/gitweb.perl | 2 +-\n> 1 files changed, 1 insertions(+), 1 deletions(-)\n> \n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 90cd99b..96cb4e0 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -16,7 +16,7 @@ use Encode;\n> use Fcntl ':mode';\n> use File::Find qw();\n> use File::Basename qw(basename);\n> -binmode STDOUT, ':utf8';\n> +use open qw(:std :utf8);\n> \n> BEGIN {\n> \tCGI->compile() if $ENV{'MOD_PERL'};\n\nIt would be wonderfull if such simple solution worked. We would be\nthen able to remove to_utf8() subroutine and do not worry that we\nforgot to convert some string to Perl internal encoding, which could\nresult to curring wide (non US-ASCII) UTF-8 character to be cut in\nhalf. (On the other hand we wouldn't have $fallback_encoding).\n\nUnfortunately there are two problem (or rather a problem and a half)\nwith this approach.\n\n\nFirst is that with this patch gitweb doesn't pass gitweb test\nt/t9500-gitweb-standalone-no-errors.sh (this is with perl v5.8.6)\n\n* ok 63: encode(commit): utf8\n* ok 64: encode(commit): iso-8859-1\n* ok 65: encode(log): utf-8 and iso-8859-1\n[...]\n* FAIL 71: URL: no project URLs, no base URL\n gitweb_run \"p=.git;a=summary\"\n[Wed Jul 2 13:10:15 2008] gitweb.perl: utf8 \"\\xC4\" does not map to Unicode \\\nat /path/to/git/t/trash directory/../../gitweb/gitweb.perl line 2298, \\\n<$fd> line 1.\n[Wed Jul 2 13:10:15 2008] gitweb.perl: Malformed UTF-8 character \\\n(unexpected end of string) at [...]/gitweb/gitweb.perl line 2303, \\\n<$fd> line 1.\n\nwhich is\n\n\topen my $fd, '-|', git_cmd(), 'for-each-ref',\n\t\t($limit ? '--count='.($limit+1) : ()), '--sort=-committerdate',\n\t\t'--format=%(objectname) %(refname) %(subject)%00%(committer)',\n\t\t'refs/heads'\n\t\tor return;\n2298:\twhile (my $line = <$fd>) {\n\t\tmy %ref_item;\n\n\t\tchomp $line;\n\t\tmy ($refinfo, $committerinfo) = split(/\\0/, $line);\n2303:\t\tmy ($hash, $name, $title) = split(' ', $refinfo, 3);\n\n\nSecond, what is minimal Perl version and Perl configuration (installed\nmodules) that support \"use open qw(:std :utf8);\"? We do have some\nminimal requirements for gitweb, and it would be nice if we didn't add\nto them. But we already require PerlIO, so it probably doesn't matter.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"82112","messageId":"486C9E5A.9@gmail.com","threadId":"14253","inReplyTo":"m3prpwflus.fsf@localhost.localdomain","subject":"Re: [PATCH/rfc] gitweb: open files (e.g. indextext.html) in utf8 mode","fromName":"Lea Wiemann","fromEmail":"lewiemann@gmail.com","sentAt":"2008-07-03T09:39:38Z","receivedAt":"2008-07-03T09:39:38Z","isPatch":true,"sender":{"key":"lewiemann@gmail.com","avatar":null},"body":"Jakub Narebski wrote:\n> Second, what is minimal Perl version and Perl configuration (installed\n> modules) that support \"use open qw(:std :utf8);\"?\n\nopen is in core (-> corelist), and \"qw(:std :utf)\" works here with Perl\n5.8.8. Perl 5.6 doesn't have it, but gitweb doesn't support Perl 5.6\nanyway (e.g. \"binmode STDOUT, ':utf8';\" doesn't work with Perl 5.6). So\nit should be fine compatibility-wise.\n\n-- Lea\n"}]}