{"thread":{"id":"53126","subject":"[PATCH] gitweb: fix UTF-8 encoding when using CGI::Fast","startedAt":"2020-03-29T00:30:34Z","lastAt":"2020-03-29T19:14:06Z","messageCount":3,"participants":["Julien Moutinho","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"394258","messageId":"20200329002028.26080-1-julm+git@sourcephile.fr","threadId":"53126","inReplyTo":null,"subject":"[PATCH] gitweb: fix UTF-8 encoding when using CGI::Fast","fromName":"Julien Moutinho","fromEmail":"julm+git@sourcephile.fr","sentAt":"2020-03-29T00:20:28Z","receivedAt":"2020-03-29T00:30:34Z","isPatch":true,"sender":{"key":"julm+git@sourcephile.fr","avatar":null},"body":"FCGI streams are implemented using the older stream API: TIEHANDLE,\ntherefore applying PerlIO layers using binmode() has no effect to them.\nThe solution in this patch is to redefine the FCGI::Stream::PRINT function\nto use UTF-8 as output encoding, except within git_blob_plain() and git_snapshot()\nwhich must still output in raw binary mode.\n\nThis problem and solution were previously reported back in 2012:\n- http://git.661346.n2.nabble.com/Gitweb-running-as-FCGI-does-not-print-its-output-in-UTF-8-td7573415.html\n- http://stackoverflow.com/questions/5005104\n\nSigned-off-by: Julien Moutinho <julm+git@sourcephile.fr>\n---\n gitweb/gitweb.perl | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 65a3a9e..1a02a12 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1291,9 +1291,23 @@ sub run_request {\n our ($pre_dispatch_hook, $post_dispatch_hook, $pre_listen_hook);\n our $CGI = 'CGI';\n our $cgi;\n+our $FCGI_Stream_PRINT_raw = \\&FCGI::Stream::PRINT;\n sub configure_as_fcgi {\n \trequire CGI::Fast;\n \tour $CGI = 'CGI::Fast';\n+\t# FCGI is not Unicode aware hence the UTF-8 encoding must be done manually.\n+\t# However no encoding must be done within git_blob_plain() and git_snapshot()\n+\t# which must still output in raw binary mode.\n+\tno warnings 'redefine';\n+\tmy $enc = Encode::find_encoding('UTF-8');\n+\t*FCGI::Stream::PRINT = sub {\n+\t\tmy @OUTPUT = @_;\n+\t\tfor (my $i = 1; $i < @_; $i++) {\n+\t\t\t$OUTPUT[$i] = $enc->encode($_[$i], Encode::FB_CROAK|Encode::LEAVE_SRC);\n+\t\t}\n+\t\t@_ = @OUTPUT;\n+\t\tgoto $FCGI_Stream_PRINT_raw;\n+\t};\n \n \tmy $request_number = 0;\n \t# let each child service 100 requests\n@@ -7079,6 +7093,7 @@ sub git_blob_plain {\n \t\t\t($sandbox ? 'attachment' : 'inline')\n \t\t\t. '; filename=\"' . $save_as . '\"');\n \tlocal $/ = undef;\n+\tlocal *FCGI::Stream::PRINT = $FCGI_Stream_PRINT_raw;\n \tbinmode STDOUT, ':raw';\n \tprint <$fd>;\n \tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n@@ -7417,6 +7432,7 @@ sub git_snapshot {\n \n \topen my $fd, \"-|\", $cmd\n \t\tor die_error(500, \"Execute git-archive failed\");\n+\tlocal *FCGI::Stream::PRINT = $FCGI_Stream_PRINT_raw;\n \tbinmode STDOUT, ':raw';\n \tprint <$fd>;\n \tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n-- \n2.25.1\n\n"},{"id":"394289","messageId":"xmqqr1xbb0xr.fsf@gitster.c.googlers.com","threadId":"53126","inReplyTo":"20200329002028.26080-1-julm+git@sourcephile.fr","subject":"Re: [PATCH] gitweb: fix UTF-8 encoding when using CGI::Fast","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-29T16:06:24Z","receivedAt":"2020-03-29T16:06:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Julien Moutinho <julm+git@sourcephile.fr> writes:\n\n>  \trequire CGI::Fast;\n>  \tour $CGI = 'CGI::Fast';\n> +\t# FCGI is not Unicode aware hence the UTF-8 encoding must be done manually.\n> +\t# However no encoding must be done within git_blob_plain() and git_snapshot()\n> +\t# which must still output in raw binary mode.\n\nI guess this comment would be sufficient to help future developers\nwhen they find that newer version of CGI::Fast has become Unicode\naware later can make this part conditional to the version of the\nmodule, perhaps?\n\nWould \"use CGI::Fast (-utf8)\" instead of the whole thing help, by\nthe way?\n\n> +\tno warnings 'redefine';\n> +\tmy $enc = Encode::find_encoding('UTF-8');\n> +\t*FCGI::Stream::PRINT = sub {\n> +\t\tmy @OUTPUT = @_;\n> +\t\tfor (my $i = 1; $i < @_; $i++) {\n> +\t\t\t$OUTPUT[$i] = $enc->encode($_[$i], Encode::FB_CROAK|Encode::LEAVE_SRC);\n> +\t\t}\n> +\t\t@_ = @OUTPUT;\n> +\t\tgoto $FCGI_Stream_PRINT_raw;\n> +\t};\n\n\n\n>  \tmy $request_number = 0;\n>  \t# let each child service 100 requests\n> @@ -7079,6 +7093,7 @@ sub git_blob_plain {\n>  \t\t\t($sandbox ? 'attachment' : 'inline')\n>  \t\t\t. '; filename=\"' . $save_as . '\"');\n>  \tlocal $/ = undef;\n> +\tlocal *FCGI::Stream::PRINT = $FCGI_Stream_PRINT_raw;\n>  \tbinmode STDOUT, ':raw';\n>  \tprint <$fd>;\n>  \tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n> @@ -7417,6 +7432,7 @@ sub git_snapshot {\n>  \n>  \topen my $fd, \"-|\", $cmd\n>  \t\tor die_error(500, \"Execute git-archive failed\");\n> +\tlocal *FCGI::Stream::PRINT = $FCGI_Stream_PRINT_raw;\n>  \tbinmode STDOUT, ':raw';\n>  \tprint <$fd>;\n>  \tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n"},{"id":"394296","messageId":"20200329191400.eh377knm4fxm66pd@sourcephile.fr","threadId":"53126","inReplyTo":"xmqqr1xbb0xr.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] gitweb: fix UTF-8 encoding when using CGI::Fast","fromName":"Julien Moutinho","fromEmail":"julm+git@sourcephile.fr","sentAt":"2020-03-29T19:14:00Z","receivedAt":"2020-03-29T19:14:06Z","isPatch":true,"sender":{"key":"julm+git@sourcephile.fr","avatar":null},"body":"Le dim. 29 mars 2020 09h06 -0700, Junio C Hamano a écrit :\n> Julien Moutinho <julm+git@sourcephile.fr> writes:\n> >  \trequire CGI::Fast;\n> >  \tour $CGI = 'CGI::Fast';\n> > +\t# FCGI is not Unicode aware hence the UTF-8 encoding must be done manually.\n> > +\t# However no encoding must be done within git_blob_plain() and git_snapshot()\n> > +\t# which must still output in raw binary mode.\n> \n> I guess this comment would be sufficient to help future developers\n> when they find that newer version of CGI::Fast has become Unicode\n> aware later can make this part conditional to the version of the\n> module, perhaps?\nSure, though as long as CGI::Fast will be relying on FCGI,\nI would not bet on any improvement on this bug\nwhich has been waiting to be fixed on FCGI's bugtracker since 2013:\nhttps://rt.cpan.org/Public/Bug/Display.html?id=89383\n\n> Would \"use CGI::Fast (-utf8)\" instead of the whole thing help, by\n> the way?\nUnfortunately not, the -utf8 option (aka. $CGI::$PARAM_UTF8)\ncontrols the decoding of the input parameters,\nnot the encoding of the output.\n- https://metacpan.org/pod/CGI#-utf8\n- https://stackoverflow.com/questions/5005104/how-to-force-fastcgi-to-encode-form-data-as-utf-8-as-cgi-pm-has-option/7097698#7097698\n\n> > our $FCGI_Stream_PRINT_raw = \\&FCGI::Stream::PRINT;\n> [...]\n> > +\tlocal *FCGI::Stream::PRINT = $FCGI_Stream_PRINT_raw;\nI had forgotten to test the patch without FastCGI,\nbut AFAICS it is innocuous in non-FastCGI mode:\nPerl does not chokes on \\&FCGI::Stream::PRINT despite it being not reachable,\nand the local binding emits no redefine warning since it's not a redefine\nin this case, but a define.\n\nRegards,\n"}]}