{"thread":{"id":"42086","subject":"[PATCH] gitweb: apply fallback encoding before highlight","startedAt":"2016-04-20T11:32:28Z","lastAt":"2016-05-05T10:22:08Z","messageCount":8,"participants":["Shin Kojima","Junio C Hamano","Jakub Narębski"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"283926","messageId":"1461151948-38583-1-git-send-email-shin@kojima.org","threadId":"42086","inReplyTo":null,"subject":"[PATCH] gitweb: apply fallback encoding before highlight","fromName":"Shin Kojima","fromEmail":"shin@kojima.org","sentAt":"2016-04-20T11:32:28Z","receivedAt":"2016-04-20T11:32:28Z","isPatch":true,"sender":{"key":"shin@kojima.org","avatar":"https://avatars.githubusercontent.com/u/343556?v=4"},"body":"Some multi-byte character encodings (such as Shift_JIS and GBK) have\ncharacters whose final bytes is an ASCII '\\' (0x5c), and they\nwill be displayed as funny-characters even if $fallback_encoding is\ncorrect.  This is because `highlight` command always expects UTF-8\nencoded strings from STDIN.\n\n    $ echo 'my $v = \"申\";' | highlight --syntax perl | w3m -T text/html -dump\n    my $v = \"申\";\n\n    $ echo 'my $v = \"申\";' | iconv -f UTF-8 -t Shift_JIS | highlight \\\n        --syntax perl | iconv -f Shift_JIS -t UTF-8 | w3m -T text/html -dump\n\n    iconv: (stdin):9:135: cannot convert\n    my $v = \"\n\nThis patch prepare git blob objects to be encoded into UTF-8 before\nhighlighting in the manner of `to_utf8` subroutine.\n---\n gitweb/gitweb.perl | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 05d7910..2fddf75 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -3935,6 +3935,9 @@ sub run_highlighter {\n \n \tclose $fd;\n \topen $fd, quote_command(git_cmd(), \"cat-file\", \"blob\", $hash).\" | \".\n+\t          quote_command($^X, '-CO', '-MEncode=decode,FB_DEFAULT', '-pse',\n+\t            '$_ = decode($fe, $_, FB_DEFAULT) if !utf8::decode($_);',\n+\t            '--', \"-fe=$fallback_encoding\").\" | \".\n \t          quote_command($highlight_bin).\n \t          \" --replace-tabs=8 --fragment --syntax $syntax |\"\n \t\tor die_error(500, \"Couldn't open file or run syntax highlighter\");\n-- \n2.8.1\n"},{"id":"285172","messageId":"xmqqbn4ouz7u.fsf@gitster.mtv.corp.google.com","threadId":"42086","inReplyTo":"1461151948-38583-1-git-send-email-shin@kojima.org","subject":"Re: [PATCH] gitweb: apply fallback encoding before highlight","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-05-02T17:49:41Z","receivedAt":"2016-05-02T17:49:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shin Kojima <shin@kojima.org> writes:\n\n> Some multi-byte character encodings (such as Shift_JIS and GBK) have\n> characters whose final bytes is an ASCII '\\' (0x5c), and they\n> will be displayed as funny-characters even if $fallback_encoding is\n> correct.  This is because `highlight` command always expects UTF-8\n> encoded strings from STDIN.\n>\n>     $ echo 'my $v = \"申\";' | highlight --syntax perl | w3m -T text/html -dump\n>     my $v = \"申\";\n>\n>     $ echo 'my $v = \"申\";' | iconv -f UTF-8 -t Shift_JIS | highlight \\\n>         --syntax perl | iconv -f Shift_JIS -t UTF-8 | w3m -T text/html -dump\n>\n>     iconv: (stdin):9:135: cannot convert\n>     my $v = \"\n>\n> This patch prepare git blob objects to be encoded into UTF-8 before\n> highlighting in the manner of `to_utf8` subroutine.\n> ---\n\nThe single liner Perl invoked from the script felt a bit too dense\nto my taste but other than that I have no complaints to what the\npatched code does.\n\nJakub, does it look good to you, too?\n\nPlease sign-off your patch (see Documentation/SubmittingPatches).\n\nThanks.\n\n\n>  gitweb/gitweb.perl | 3 +++\n>  1 file changed, 3 insertions(+)\n>\n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 05d7910..2fddf75 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -3935,6 +3935,9 @@ sub run_highlighter {\n>  \n>  \tclose $fd;\n>  \topen $fd, quote_command(git_cmd(), \"cat-file\", \"blob\", $hash).\" | \".\n> +\t          quote_command($^X, '-CO', '-MEncode=decode,FB_DEFAULT', '-pse',\n> +\t            '$_ = decode($fe, $_, FB_DEFAULT) if !utf8::decode($_);',\n> +\t            '--', \"-fe=$fallback_encoding\").\" | \".\n>  \t          quote_command($highlight_bin).\n>  \t          \" --replace-tabs=8 --fragment --syntax $syntax |\"\n>  \t\tor die_error(500, \"Couldn't open file or run syntax highlighter\");\n"},{"id":"285178","messageId":"CANQwDwfZPoNd7OTpSkEmf4amV0hh8kiuk3omwLCS0O=MZ1DB=A@mail.gmail.com","threadId":"42086","inReplyTo":"xmqqbn4ouz7u.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] gitweb: apply fallback encoding before highlight","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-05-02T18:12:52Z","receivedAt":"2016-05-02T18:12:52Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, May 2, 2016 at 7:49 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Shin Kojima <shin@kojima.org> writes:\n>\n>> This patch prepare git blob objects to be encoded into UTF-8 before\n>> highlighting in the manner of `to_utf8` subroutine.\n>> ---\n>\n> The single liner Perl invoked from the script felt a bit too dense\n> to my taste but other than that I have no complaints to what the\n> patched code does.\n>\n> Jakub, does it look good to you, too?\n\nYes, it looks all right to me. $^X is current Perl. -CO means that\nthe output is utf8 (for `highlight` command), -p means read all lines\nand print them (it could be replaced by \"print\" command in one-liner),\n-s is here to pass $fallback_encoding as $fe (it could be replaced,\nbut it would require some fiddling with quoting $s), -e '...' means\nexecute one line.\n\n> Please sign-off your patch (see Documentation/SubmittingPatches).\n>\n> Thanks.\n>\n>\n>>  gitweb/gitweb.perl | 3 +++\n>>  1 file changed, 3 insertions(+)\n>>\n>> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n>> index 05d7910..2fddf75 100755\n>> --- a/gitweb/gitweb.perl\n>> +++ b/gitweb/gitweb.perl\n>> @@ -3935,6 +3935,9 @@ sub run_highlighter {\n>>\n>>       close $fd;\n>>       open $fd, quote_command(git_cmd(), \"cat-file\", \"blob\", $hash).\" | \".\n>> +               quote_command($^X, '-CO', '-MEncode=decode,FB_DEFAULT', '-pse',\n>> +                 '$_ = decode($fe, $_, FB_DEFAULT) if !utf8::decode($_);',\n>> +                 '--', \"-fe=$fallback_encoding\").\" | \".\n>>                 quote_command($highlight_bin).\n>>                 \" --replace-tabs=8 --fragment --syntax $syntax |\"\n>>               or die_error(500, \"Couldn't open file or run syntax highlighter\");\n\n\n\n-- \nJakub Narebski\n"},{"id":"285319","messageId":"1462280451-43388-1-git-send-email-shin@kojima.org","threadId":"42086","inReplyTo":"xmqqbn4ouz7u.fsf@gitster.mtv.corp.google.com","subject":"[PATCH v2] gitweb: apply fallback encoding before highlight","fromName":"Shin Kojima","fromEmail":"shin@kojima.org","sentAt":"2016-05-03T13:00:51Z","receivedAt":"2016-05-03T13:00:51Z","isPatch":true,"sender":{"key":"shin@kojima.org","avatar":"https://avatars.githubusercontent.com/u/343556?v=4"},"body":"Some multi-byte character encodings (such as Shift_JIS and GBK) have\ncharacters whose final bytes is an ASCII '\\' (0x5c), and they\nwill be displayed as funny-characters even if $fallback_encoding is\ncorrect.  This is because `highlight` command always expects UTF-8\nencoded strings from STDIN.\n\n    $ echo 'my $v = \"申\";' | highlight --syntax perl | w3m -T text/html -dump\n    my $v = \"申\";\n\n    $ echo 'my $v = \"申\";' | iconv -f UTF-8 -t Shift_JIS | highlight \\\n        --syntax perl | iconv -f Shift_JIS -t UTF-8 | w3m -T text/html -dump\n\n    iconv: (stdin):9:135: cannot convert\n    my $v = \"\n\nThis patch prepare git blob objects to be encoded into UTF-8 before\nhighlighting in the manner of `to_utf8` subroutine.\n\nSigned-off-by: Shin Kojima <shin@kojima.org>\n---\n\nChanges for v2:\n    - Add Signed-off-by\n\nThanks,\nShin Kojima\n\n gitweb/gitweb.perl | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 05d7910..2fddf75 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -3935,6 +3935,9 @@ sub run_highlighter {\n \n \tclose $fd;\n \topen $fd, quote_command(git_cmd(), \"cat-file\", \"blob\", $hash).\" | \".\n+\t          quote_command($^X, '-CO', '-MEncode=decode,FB_DEFAULT', '-pse',\n+\t            '$_ = decode($fe, $_, FB_DEFAULT) if !utf8::decode($_);',\n+\t            '--', \"-fe=$fallback_encoding\").\" | \".\n \t          quote_command($highlight_bin).\n \t          \" --replace-tabs=8 --fragment --syntax $syntax |\"\n \t\tor die_error(500, \"Couldn't open file or run syntax highlighter\");\n-- \n2.8.2\n"},{"id":"285354","messageId":"xmqq7ffbng8p.fsf@gitster.mtv.corp.google.com","threadId":"42086","inReplyTo":"1462280451-43388-1-git-send-email-shin@kojima.org","subject":"Re: [PATCH v2] gitweb: apply fallback encoding before highlight","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-05-03T18:33:42Z","receivedAt":"2016-05-03T18:33:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shin Kojima <shin@kojima.org> writes:\n\n> Some multi-byte character encodings (such as Shift_JIS and GBK) have\n> characters whose final bytes is an ASCII '\\' (0x5c), and they\n> will be displayed as funny-characters even if $fallback_encoding is\n> correct.\n\nJust out of curiosity, do people still use Shift_JIS aka MS-Kanji?\nIt feels so last-decade, if not last-century ;-)\n"},{"id":"285440","messageId":"20160504083449.GA46041@skmba.local","threadId":"42086","inReplyTo":"xmqq7ffbng8p.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2] gitweb: apply fallback encoding before highlight","fromName":"Shin Kojima","fromEmail":"shin@kojima.org","sentAt":"2016-05-04T08:34:55Z","receivedAt":"2016-05-04T08:34:55Z","isPatch":true,"sender":{"key":"shin@kojima.org","avatar":"https://avatars.githubusercontent.com/u/343556?v=4"},"body":"On Tue, May 03, 2016 at 11:33:42AM -0700, Junio C Hamano wrote:\n> Shin Kojima <shin@kojima.org> writes:\n> \n> > Some multi-byte character encodings (such as Shift_JIS and GBK) have\n> > characters whose final bytes is an ASCII '\\' (0x5c), and they\n> > will be displayed as funny-characters even if $fallback_encoding is\n> > correct.\n> \n> Just out of curiosity, do people still use Shift_JIS aka MS-Kanji?\n> It feels so last-decade, if not last-century ;-)\n\nYes, they do. There are still tons of code from 90's lying around.\n\nFor migrating our codebase from cp932 (Windows31J/MS-Kanji), I keep\nfailing to persuade my boss saying it has no incentives to do so.\n\nI can say this patch, to consider $fallback_encoding while\nhighlighting, is fairly rational.  But I also feel this is too much\njust for specific outdated character encodings, it is completely\nuseless for the most part of gitweb users in the world.\n\nI would rather prefer to generate feedback from you all to convince\nour management if this patch is not acceptable.\n"},{"id":"285498","messageId":"xmqqeg9hhb1e.fsf@gitster.mtv.corp.google.com","threadId":"42086","inReplyTo":"20160504083449.GA46041@skmba.local","subject":"Re: [PATCH v2] gitweb: apply fallback encoding before highlight","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-05-04T19:34:53Z","receivedAt":"2016-05-04T19:34:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shin Kojima <shin@kojima.org> writes:\n\n> I can say this patch, to consider $fallback_encoding while\n> highlighting, is fairly rational.  But I also feel this is too much\n> just for specific outdated character encodings, it is completely\n> useless for the most part of gitweb users in the world.\n\nOh, don't get me wrong. I do think what the patch does is very\nsensible and have no intention of rejecting it.\n\nUnless somebody finds a new bug in it, but in that case, we won't be\nrejecting it but would be improving on it.\n\nAs I said, the question was \"Just out of curiosity\", since it's been\nso long since I was in any part of software work done in Japan.\n"},{"id":"285566","messageId":"20160505102203.GA53576@skmba.local","threadId":"42086","inReplyTo":"xmqqeg9hhb1e.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2] gitweb: apply fallback encoding before highlight","fromName":"Shin Kojima","fromEmail":"shin@kojima.org","sentAt":"2016-05-05T10:22:08Z","receivedAt":"2016-05-05T10:22:08Z","isPatch":true,"sender":{"key":"shin@kojima.org","avatar":"https://avatars.githubusercontent.com/u/343556?v=4"},"body":"> Oh, don't get me wrong. I do think what the patch does is very\n> sensible and have no intention of rejecting it.\n\nI'm sorry for making you worry, my poor English had caused some\nmisunderstanding.  I raised this Shift_JIS related problem (a.k.a\n\"ダメ文字\" in Japanese) might attract your interest knowingly.\n\nI would like to hear frank opinions from engineers who have high\nabilities like you. ;)\n\n> As I said, the question was \"Just out of curiosity\", since it's been\n> so long since I was in any part of software work done in Japan.\n\nHaving said that, many people in Japan are still suffering from these\ncharacter encoding barriers.  This was a huge shock for me, since\nI was studying information science in China as a Japanese foreign\nstudent.\n"}]}