{"thread":{"id":"48391","subject":"[PATCH] gitweb: Measure offsets against UTF-8 flagged string","startedAt":"2018-05-01T06:40:27Z","lastAt":"2018-05-04T02:38:22Z","messageCount":6,"participants":["Shin Kojima","Junio C Hamano","Jakub Narebski"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"346222","messageId":"20180501064015.59977-1-shin@kojima.org","threadId":"48391","inReplyTo":null,"subject":"[PATCH] gitweb: Measure offsets against UTF-8 flagged string","fromName":"Shin Kojima","fromEmail":"shin@kojima.org","sentAt":"2018-05-01T06:40:15Z","receivedAt":"2018-05-01T06:40:27Z","isPatch":true,"sender":{"key":"shin@kojima.org","avatar":"https://avatars.githubusercontent.com/u/343556?v=4"},"body":"Offset positions should not be counted by byte length, but by actual\ncharacter length.\n\n>    5183 \t# We need to untabify lines before split()'ing them;\n>    5184 \t# otherwise offsets would be invalid.\n\nHorizontal tab is not the only case we need to consider.  Please excuse\nme for using your name here, but the following URL can not find \"match\"\noccurances while using `git-instaweb` on the git repository.\n\n    http://127.0.0.1:1234/?p=.git&a=search&h=HEAD&st=grep&s=Nar%C4%99bski\n\nSigned-off-by: Shin Kojima <shin@kojima.org>\n---\n gitweb/gitweb.perl | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 2594a4bad..a5a9093a1 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1697,7 +1697,7 @@ sub unquote {\n \n # escape tabs (convert tabs to spaces)\n sub untabify {\n-\tmy $line = shift;\n+\tmy $line = to_utf8(shift);\n \n \twhile ((my $pos = index($line, \"\\t\")) != -1) {\n \t\tif (my $count = (8 - ($pos % 8))) {\n-- \n2.17.0\n\n"},{"id":"346436","messageId":"xmqqtvrqv632.fsf@gitster-ct.c.googlers.com","threadId":"48391","inReplyTo":"20180501064015.59977-1-shin@kojima.org","subject":"Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-02T08:01:53Z","receivedAt":"2018-05-02T08:02:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shin Kojima <shin@kojima.org> writes:\n\n> Offset positions should not be counted by byte length, but by actual\n> character length.\n> ...\n>  # escape tabs (convert tabs to spaces)\n>  sub untabify {\n> -\tmy $line = shift;\n> +\tmy $line = to_utf8(shift);\n>  \n>  \twhile ((my $pos = index($line, \"\\t\")) != -1) {\n>  \t\tif (my $count = (8 - ($pos % 8))) {\n\nSome codepaths in the resulting codeflow look even hackier than they\nalready are.  For example, format_rem_add_lines_pair() calls\nuntabify() and then feeds its result to esc_html().  The first thing\ndone in esc_html() is to call to_utf8().  I know that to_utf8()\ncheats and leaves the incoming string as-is if it is already UTF-8,\nso this may be a safe conversion, but ideally we should be able to\nsay \"function X takes non-UTF8 and works on it\", \"function Y takes\nUTF8 and works on it\", and \"function Z takes non-UTF8 and gives UTF8\ndata back\" for each functions clearly, not \"function W can take\neither UTF8 or any other garbage and tries to return UTF8\".\n\nAlso, does it even \"fix\" the problem to use to_utf8() here in the\nfirst place?  Untabify is about aligning the character after a HT to\nmultiple of 8 display position, so we'd want measure display width,\nwhich is not the same as either byte count or char count.\n"},{"id":"346447","messageId":"20180502114743.cmetkapax2y5mipj@skmbp","threadId":"48391","inReplyTo":"xmqqtvrqv632.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string","fromName":"Shin Kojima","fromEmail":"shin@kojima.org","sentAt":"2018-05-02T11:47:45Z","receivedAt":"2018-05-02T11:47:54Z","isPatch":true,"sender":{"key":"shin@kojima.org","avatar":"https://avatars.githubusercontent.com/u/343556?v=4"},"body":"> ideally we should be able to say \"function X takes non-UTF8 and\n> works on it\", \"function Y takes UTF8 and works on it\", and \"function\n> Z takes non-UTF8 and gives UTF8 data back\" for each functions\n> clearly, not \"function W can take either UTF8 or any other garbage\n> and tries to return UTF8\".\n\nYes, I totally agree with that.\n\n\n> Some codepaths in the resulting codeflow look even harder than they\n> already are.  For example, format_rem_add_lines_pair() calls\n> untabify() and then feeds its result to esc_html().\n\nHonestly, I discovered this problem exactly from\nformat_rem_add_lines_pair().  In my environment($fallback_encoding\n= 'cp932'), some commitdiff shows garbled text only inside color-words\nportions.\n\nI added a reproduce process at the end of this message.\n\nAfter my investigation, I thought format_rem_add_lines_pair() tries to\nuse split()/index()/substr()/etc against raw blob contents and that\nproduces funny characters.  These builtin functions should be used to\ndecoded string.\n\nuntabify() looks proper place for me to decode blob contents\nbeforehand, as it definitely is not to be used for binary contests\nlike images and compressed snapshot files.\n\nI'm sure using to_utf8() in untabify() fixes this problem.  In fact,\nthere is also another similar problem in blame function that assumes\nblob contents as if utf8 encoded:\n\n    binmode $fd, ':utf8';\n\nI personally consider text blob contents should be decoded as soon as\npossible and esc_html() should not contain to_utf8(), but the\ncodeflow is slightly vast and I couldn't eliminate the possibility of\ncalls from somewhere else that does not care character encodings.\n\nSo yes, I would appreciate hearing your thoughts.\n\n\n> Also, does it even \"fix\" the problem to use to_utf8() here in the\n> first place?  Untabify is about aligning the character after a HT to\n> multiple of 8 display position, so we'd want measure display width,\n> which is not the same as either byte count or char count.\n\nFollowing is a reproduce process:\n\n    $ git --version\n        git version 2.17.0\n\n    $ mkdir test\n    $ cd test\n    $ git init\n    $ echo 'モバイル' | iconv -f UTF-8 -t Shift_JIS > dummy\n    $ git add .\n    $ git commit -m 'init'\n    $ echo 'インスタント' | iconv -f UTF-8 -t Shift_JIS > dummy\n    $ git commit -am 'change'\n    $ git instaweb\n    $ echo 'our $fallback_encoding = \"cp932\";' >> .git/gitweb/gitweb_config.perl\n    $ w3m -dump 'http://127.0.0.1:1234/?p=.git;a=commitdiff'\n\nWhat I got:\n\n    gitprojects / .git / commitdiff\n    [commit   ] ? search: [                    ] [ ]re\n    summary | shortlog | log | commit | commitdiff | tree\n    raw | patch | inline | side by side (parent: 79e26fe)\n    change master\n\n    author    Shin Kojima <shin@kojima.org>\n              Wed, 2 May 2018 10:55:01 +0000 (19:55 +0900)\n    committer Shin Kojima <shin@kojima.org>\n              Wed, 2 May 2018 10:55:01 +0000 (19:55 +0900)\n\n    dummy  patch | blob | history\n\n\n    diff --git a/dummy b/dummy\n    index ac37f38..31fb96a 100644 (file)\n    --- a/dummy\n    +++ b/dummy\n    @@ -1 +1 @@\n    -ｃoイル\n    +Cンスタント\n    Unnamed repository; edit this file 'description' to name the repository.\n    RSS Atom\n\nWhat I expected:\n\n    gitprojects / .git / commitdiff\n    [commit   ] ? search: [                    ] [ ]re\n    summary | shortlog | log | commit | commitdiff | tree\n    raw | patch | inline | side by side (parent: 79e26fe)\n    change master\n\n    author    Shin Kojima <shin@kojima.org>\n              Wed, 2 May 2018 10:55:01 +0000 (19:55 +0900)\n    committer Shin Kojima <shin@kojima.org>\n              Wed, 2 May 2018 10:55:01 +0000 (19:55 +0900)\n\n    dummy  patch | blob | history\n\n\n    diff --git a/dummy b/dummy\n    index ac37f38..31fb96a 100644 (file)\n    --- a/dummy\n    +++ b/dummy\n    @@ -1 +1 @@\n    -モバイル\n    +インスタント\n    Unnamed repository; edit this file 'description' to name the repository.\n    RSS Atom\n\n\n-- \nShin Kojima\n"},{"id":"346512","messageId":"86k1skzzc4.fsf@gmail.com","threadId":"48391","inReplyTo":"xmqqtvrqv632.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-03T12:40:59Z","receivedAt":"2018-05-03T12:41:10Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> Shin Kojima <shin@kojima.org> writes:\n>\n>> Offset positions should not be counted by byte length, but by actual\n>> character length.\n>> ...\n>>  # escape tabs (convert tabs to spaces)\n>>  sub untabify {\n>> -\tmy $line = shift;\n>> +\tmy $line = to_utf8(shift);\n>>  \n>>  \twhile ((my $pos = index($line, \"\\t\")) != -1) {\n>>  \t\tif (my $count = (8 - ($pos % 8))) {\n>\n> Some codepaths in the resulting codeflow look even hackier than they\n> already are.  For example, format_rem_add_lines_pair() calls\n> untabify() and then feeds its result to esc_html().  The first thing\n> done in esc_html() is to call to_utf8().  I know that to_utf8()\n> cheats and leaves the incoming string as-is if it is already UTF-8,\n> so this may be a safe conversion, but ideally we should be able to\n> say \"function X takes non-UTF8 and works on it\", \"function Y takes\n> UTF8 and works on it\", and \"function Z takes non-UTF8 and gives UTF8\n> data back\" for each functions clearly, not \"function W can take\n> either UTF8 or any other garbage and tries to return UTF8\".\n\nThe problem with handling encoding in sane way, that is encode it on\ninput (to UTF-8), and decode on output (to plain text or HTML) is the\n$fallback_encoding.\n\nGitweb assumes that everything uses UTF-8 encoding.  If the source is\nnot in UTF-8, but for example uses latin-1 encoding, then there we could\nstumble upon byte sequences which are not valid UTF-8.  If that happens,\nthen gitweb tries to convert it to UTF-8 using $fallback_encoding.  If\n$fallback_encoding is single-byte encoding, like latin-1, where any byte\nsequence is valid, then that's all.  If there is an error during\nconversion to UTF-8, then Unicode REPLACEMENT CHARACTER, code point\nU+FFFD, is used.\n\nBut there are places where gitweb outputs plain text; the intention is\nto use source data as is - to have it as one would have in the console.\nSome input paths are common for plain text and HTML output; because of\nthat problem the data is not converted to UTF-8 on input.\n\n\nThe to_utf8() function tries to be clever, and do not convert alredy\nconverted data.\n\n> Also, does it even \"fix\" the problem to use to_utf8() here in the\n> first place?  Untabify is about aligning the character after a HT to\n> multiple of 8 display position, so we'd want measure display width,\n> which is not the same as either byte count or char count.\n\nI think the problem is not with aligning, otherwise we would simply get\nbad aling, and not visible corruption.  The ACTUAL PROBLEM is most\nprobably because of concatenating strings marked as UTF-8 and strings\nnot marked as UTF-8.  Strange things happen then in Perl, unfortunetaly.\n\n\nOne solution would be to force conversion to UTF-8 on input via \"open\"\npragma (e.g. \"use open ':encoding(UTF-8)';\").  But there is no\nUTF-8-with_fallback encoding available - we would have to write one, and\ninstall it as module (or fake it via Perl trickery).  This mechanism is\nalmost the same to what we currently use in gitwbe.\n\nAnother would be to use the trick that Perl 6 uses when encountering\nbyte sequence that is invalid UTF-8 - encode it using private plane in\nUnicode using UTF-8, thus achieving lossless conversion / decoding.  But\nthis also as far as I know is not available from CPAN, so we would have\nto implement it ourself.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"346517","messageId":"20180503151627.45pt2veqcjzbk44q@skmbp","threadId":"48391","inReplyTo":"86k1skzzc4.fsf@gmail.com","subject":"Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string","fromName":"Shin Kojima","fromEmail":"shin@kojima.org","sentAt":"2018-05-03T15:16:29Z","receivedAt":"2018-05-03T15:16:38Z","isPatch":true,"sender":{"key":"shin@kojima.org","avatar":"https://avatars.githubusercontent.com/u/343556?v=4"},"body":"> One solution would be to force conversion to UTF-8 on input via \"open\"\n> pragma (e.g. \"use open ':encoding(UTF-8)';\").  But there is no\n> UTF-8-with_fallback encoding available - we would have to write one, and\n> install it as module (or fake it via Perl trickery).  This mechanism is\n> almost the same to what we currently use in gitwbe.\n\nYes, I tried using `Encode::Guess` with \"open\" pragma, but no luck.\nhttps://perldoc.perl.org/Encode/Guess.html\n\nI'm also afraid of \"open\" pragma does not work properly while using\ngit_blame_common().  Let's say someone using non-ASCII characters in\nhis/her name, committing non-UTF8 encoded characters.  git-blame will\ncombine them in the same line.  Following is an example:\n\n$ git blame dummy | xxd\n00000000: 3461 6464 3565 6331 2028 e585 90e5 b3b6  4add5ec1 (......\n00000010: 20e6 96b0 2032 3031 382d 3035 2d30 3320   ... 2018-05-03\n00000020: 3232 3a34 383a 3432 202b 3039 3030 2031  22:48:42 +0900 1\n00000030: 2920 8367 8389 8343 0a                   ) .g...C.\n\n    * e585 90e5 b3b6 20e6 96b0 : my name, encoded with UTF-8\n    * 8367 8389 8343           : \"トライ\" encoded with Shift_JIS\n\nIt means I need to split each lines of git-blame output at the very\nbeginning, then convert the first-half as UTF-8 and the second-half as\nShift_JIS.\n\nSincerely,\n\n-- \nShin Kojima\n"},{"id":"346616","messageId":"xmqqin84taau.fsf@gitster-ct.c.googlers.com","threadId":"48391","inReplyTo":"86k1skzzc4.fsf@gmail.com","subject":"Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-04T02:38:17Z","receivedAt":"2018-05-04T02:38:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> I think the problem is not with aligning, otherwise we would simply get\n> bad aling, and not visible corruption.  The ACTUAL PROBLEM is most\n> probably because of concatenating strings marked as UTF-8 and strings\n> not marked as UTF-8.  Strange things happen then in Perl, unfortunetaly.\n\nSounds quite right---the patch needs to be retitled, if the bug is\nnot about \"measuring offset\", which I think is what fooled me when I\nsent my response.\n\n"}]}