{"thread":{"id":"49303","subject":"gitweb: HTML output is not always encoded in UTF-8 when using --fastcgi.","startedAt":"2018-09-08T17:30:15Z","lastAt":"2018-09-08T18:15:09Z","messageCount":2,"participants":["Julien Moutinho"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"357704","messageId":"20180908171532.uyz7i5oy6w6otp2r@localhost","threadId":"49303","inReplyTo":null,"subject":"gitweb: HTML output is not always encoded in UTF-8 when using --fastcgi.","fromName":"Julien Moutinho","fromEmail":"julm+git@autogeree.net","sentAt":"2018-09-08T17:15:32Z","receivedAt":"2018-09-08T17:30:15Z","isPatch":false,"sender":{"key":"julm+git@autogeree.net","avatar":null},"body":"Description\n===========\nAn old (2011) description of the problem is here:\nhttps://stackoverflow.com/questions/7285215/nginx-fastcgi-utf-8-encoding-output-iso-8859-1-instead-of-utf8#answer-18149487\n\nBasically, gitweb's HTML output is not always encoded in UTF-8\nwhen using --fastcgi.\n\n\nReproduction\n============\ngitweb v2.18.0\nperl   v5.28.0\n\n| echo Système >test.git/description\n\nAccording to the 2011 problem report,\nthe problem only appears when using gitweb.cgi --fastcgi\nnot when gitweb.cgi is spawned by fcgiwrap.\n\nAnd apparently, the text must not contain one character\nwhich cannot be correctly converted to ISO-8859-1,\nor an UTF-8 encoding is done (not sure by what);\nwhich made this bug harder to spot.\n\n\nExplanation\n===========\nAccording to Christian Hansen (chansen), the problem is that:\n> FCGI streams are implemented using the older stream API,\n> TIEHANDLE. Applying PerlIO layers using binmode() has no effect.\nhttps://stackoverflow.com/questions/5005104/how-to-force-fastcgi-to-encode-form-data-as-utf-8-as-cgi-pm-has-option#answer-7097698\nhttps://perldoc.perl.org/perltie.html\n\nIndeed:\n> FCGI.pm isn't Unicode aware,\n> only characters within the range 0x00-0xFF are supported.\nhttps://metacpan.org/pod/FCGI#LIMITATIONS\n\nBut, as stated in gitweb's to_utf8():\n> gitweb writes out in utf-8\n> thanks to \"binmode STDOUT, ':utf8'\" at beginning\"\n\n\nCorrection\n==========\nChristian Hansen suggested that:\n\"The proper solution would be to encode your data before outputting it,\nbut if thats not an option I can offer this hotpatch:\"\n\n| my $enc = Encode::find_encoding('UTF-8');\n| my $org = \\&FCGI::Stream::PRINT;\n| no warnings 'redefine';\n| local *FCGI::Stream::PRINT = sub {\n|     my @OUTPUT = @_;\n|     for (my $i = 1; $i < @_; $i++) {\n|         $OUTPUT[$i] = $enc->encode($_[$i], Encode::FB_CROAK|Encode::LEAVE_SRC);\n|     }\n|     @_ = @OUTPUT;\n|     goto $org;\n| };\n\nAs a quick workaround this hotpatch can even be put in $GITWEB_CONFIG\nby removing the `local` before `*FCGI::Stream::PRINT`.\n"},{"id":"357705","messageId":"20180908181439.jdbvdrzndvdy7gkc@localhost","threadId":"49303","inReplyTo":"20180908171532.uyz7i5oy6w6otp2r@localhost","subject":"Re: gitweb: HTML output is not always encoded in UTF-8 when using --fastcgi.","fromName":"Julien Moutinho","fromEmail":"julm+git@autogeree.net","sentAt":"2018-09-08T18:14:39Z","receivedAt":"2018-09-08T18:15:09Z","isPatch":false,"sender":{"key":"julm+git@autogeree.net","avatar":null},"body":"Le sam. 08 sept. 2018 à 19:15:32 +0200, Julien Moutinho a écrit :\n> As a quick workaround this hotpatch can even be put in $GITWEB_CONFIG\n> by removing the `local` before `*FCGI::Stream::PRINT`.\n\nTurns out to require more care than that,\ndue to $per_request_config reloading $GITWEB_CONFIG at each request,\noverwriting FCGI::Stream::PRINT multiple times, messing the encoding.\nThis seems to work(around):\n| if ($first_request) {\n|         my $enc = Encode::find_encoding('UTF-8');\n|         my $org = \\&FCGI::Stream::PRINT;\n|         no warnings 'redefine';\n|         *FCGI::Stream::PRINT = sub {\n|             my @OUTPUT = @_;\n|             for (my $i = 1; $i < @_; $i++) {\n|                 $OUTPUT[$i] = $enc->encode($_[$i], Encode::FB_CROAK|Encode::LEAVE_SRC);\n|             }\n|             @_ = @OUTPUT;\n|             goto $org;\n|         };\n| }\n"}]}