{"thread":{"id":"8342","subject":"[PATCH] gitweb: handle non UTF-8 text","startedAt":"2007-05-28T20:47:34Z","lastAt":"2007-06-03T18:41:04Z","messageCount":9,"participants":["Martin Koegler","Petr Baudis","Jakub Narebski","Robin Rosenberg","Junio C Hamano","Alexandre Julliard"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"43517","messageId":"1180385254576-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"8342","inReplyTo":null,"subject":"[PATCH] gitweb: handle non UTF-8 text","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2007-05-28T20:47:34Z","receivedAt":"2007-05-28T20:47:34Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"gitweb assumes, that everything is in UTF-8. If a text contains invalid\nUTF-8 character sequences, the text must be in a different encoding.\n\nThis patch interprets such a text as latin1.\n\nSigned-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\nFor correct UTF-8, the patch does not change anything.\n\nIf commit/blob/... is not in UTF-8, it displays the text\nwith a very high probability correct. \n\nAs git itself is not aware of any encoding, I know no better\npossibility to handle non UTF-8 text in gitweb.\n\n gitweb/gitweb.perl |   27 +++++++++++++++++----------\n 1 files changed, 17 insertions(+), 10 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex c3921cb..dfd564d 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -621,12 +621,19 @@ sub esc_url {\n \treturn $str;\n }\n \n+sub my_decode_utf8 {\n+\tmy $str = shift;\n+\tmy $res;\n+\teval { $res = decode_utf8 ($str, 1); };\n+\treturn $res || decode('latin1', $str);\n+}\n+\n # replace invalid utf8 character with SUBSTITUTION sequence\n sub esc_html ($;%) {\n \tmy $str = shift;\n \tmy %opts = @_;\n \n-\t$str = decode_utf8($str);\n+\t$str = my_decode_utf8($str);\n \t$str = $cgi->escapeHTML($str);\n \tif ($opts{'-nbsp'}) {\n \t\t$str =~ s/ /&nbsp;/g;\n@@ -640,7 +647,7 @@ sub esc_path {\n \tmy $str = shift;\n \tmy %opts = @_;\n \n-\t$str = decode_utf8($str);\n+\t$str = my_decode_utf8($str);\n \t$str = $cgi->escapeHTML($str);\n \tif ($opts{'-nbsp'}) {\n \t\t$str =~ s/ /&nbsp;/g;\n@@ -925,7 +932,7 @@ sub format_subject_html {\n \n \tif (length($short) < length($long)) {\n \t\treturn $cgi->a({-href => $href, -class => \"list subject\",\n-\t\t                -title => decode_utf8($long)},\n+\t\t                -title => my_decode_utf8($long)},\n \t\t       esc_html($short) . $extra);\n \t} else {\n \t\treturn $cgi->a({-href => $href, -class => \"list subject\"},\n@@ -1239,7 +1246,7 @@ sub git_get_projects_list {\n \t\t\tif (check_export_ok(\"$projectroot/$path\")) {\n \t\t\t\tmy $pr = {\n \t\t\t\t\tpath => $path,\n-\t\t\t\t\towner => decode_utf8($owner),\n+\t\t\t\t\towner => my_decode_utf8($owner),\n \t\t\t\t};\n \t\t\t\tpush @list, $pr;\n \t\t\t\t(my $forks_path = $path) =~ s/\\.git$//;\n@@ -1269,7 +1276,7 @@ sub git_get_project_owner {\n \t\t\t$pr = unescape($pr);\n \t\t\t$ow = unescape($ow);\n \t\t\tif ($pr eq $project) {\n-\t\t\t\t$owner = decode_utf8($ow);\n+\t\t\t\t$owner = my_decode_utf8($ow);\n \t\t\t\tlast;\n \t\t\t}\n \t\t}\n@@ -1759,7 +1766,7 @@ sub get_file_owner {\n \t}\n \tmy $owner = $gcos;\n \t$owner =~ s/[,;].*$//;\n-\treturn decode_utf8($owner);\n+\treturn my_decode_utf8($owner);\n }\n \n ## ......................................................................\n@@ -1842,7 +1849,7 @@ sub git_header_html {\n \n \tmy $title = \"$site_name\";\n \tif (defined $project) {\n-\t\t$title .= \" - \" . decode_utf8($project);\n+\t\t$title .= \" - \" . my_decode_utf8($project);\n \t\tif (defined $action) {\n \t\t\t$title .= \"/$action\";\n \t\t\tif (defined $file_name) {\n@@ -2116,7 +2123,7 @@ sub git_print_page_path {\n \n \tprint \"<div class=\\\"page_path\\\">\";\n \tprint $cgi->a({-href => href(action=>\"tree\", hash_base=>$hb),\n-\t              -title => 'tree root'}, decode_utf8(\"[$project]\"));\n+\t              -title => 'tree root'}, my_decode_utf8(\"[$project]\"));\n \tprint \" / \";\n \tif (defined $name) {\n \t\tmy @dirname = split '/', $name;\n@@ -2936,7 +2943,7 @@ sub git_project_list_body {\n \t\t($pr->{'age'}, $pr->{'age_string'}) = @aa;\n \t\tif (!defined $pr->{'descr'}) {\n \t\t\tmy $descr = git_get_project_description($pr->{'path'}) || \"\";\n-\t\t\t$pr->{'descr_long'} = decode_utf8($descr);\n+\t\t\t$pr->{'descr_long'} = my_decode_utf8($descr);\n \t\t\t$pr->{'descr'} = chop_str($descr, 25, 5);\n \t\t}\n \t\tif (!defined $pr->{'owner'}) {\n@@ -3981,7 +3988,7 @@ sub git_snapshot {\n \tmy $git = git_cmd_str();\n \tmy $name = $project;\n \t$name =~ s/\\047/\\047\\\\\\047\\047/g;\n-\tmy $filename = decode_utf8(basename($project));\n+\tmy $filename = my_decode_utf8(basename($project));\n \tmy $cmd;\n \tif ($suffix eq 'zip') {\n \t\t$filename .= \"-$hash.$suffix\";\n-- \n1.5.2.846.g9a144\n"},{"id":"43527","messageId":"20070528232139.GU4489@pasky.or.cz","threadId":"8342","inReplyTo":"1180385254576-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"Re: [PATCH] gitweb: handle non UTF-8 text","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2007-05-28T23:21:39Z","receivedAt":"2007-05-28T23:21:39Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Mon, May 28, 2007 at 10:47:34PM CEST, Martin Koegler wrote:\n> gitweb assumes, that everything is in UTF-8. If a text contains invalid\n> UTF-8 character sequences, the text must be in a different encoding.\n> \n> This patch interprets such a text as latin1.\n> \n> Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n> ---\n> For correct UTF-8, the patch does not change anything.\n> \n> If commit/blob/... is not in UTF-8, it displays the text\n> with a very high probability correct. \n> \n> As git itself is not aware of any encoding, I know no better\n> possibility to handle non UTF-8 text in gitweb.\n\nI don't think this is a reasonable approach; I actually dispute the high\nprobability - in western Europe it's obvious to assume latin1, but does\nmajority of users using non-ascii characters come from there? Or rather\nfrom central Europe (like me, Petr Baudiš? ;-))? Somewhere else?\n\nIf we do something like this, we should do it properly and look at\nconfigured i18n.commitEncoding for the project. (But as config lookup\nmay be expensive, probably do it only when we need it.)\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nEver try. Ever fail. No matter. // Try again. Fail again. Fail better.\n\t\t-- Samuel Beckett\n"},{"id":"43563","messageId":"200705291121.12119.jnareb@gmail.com","threadId":"8342","inReplyTo":"20070528232139.GU4489@pasky.or.cz","subject":"Re: [PATCH] gitweb: handle non UTF-8 text","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-05-29T09:21:11Z","receivedAt":"2007-05-29T09:21:11Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"[Cc: authors of git-cvsserver]\n\nOn Tue, 29 May 2007, Petr Baudis wrote:\n> On Mon, May 28, 2007 at 10:47:34PM CEST, Martin Koegler wrote:\n\n>> gitweb assumes, that everything is in UTF-8. If a text contains invalid\n>> UTF-8 character sequences, the text must be in a different encoding.\n\nBut it doesn't tell us _what_ is the encoding. For commit messages,\nwith reasonable new git, we have 'encoding' header if git known that\ncommit message was not in utf-8.\n\nBy the way, I winder why we don't have such header for tag objects\n(i18n.tagEncoding ;-)...\n \n>> This patch interprets such a text as latin1.\n\nMeaning that it tries to recode text from latin1 (iso-8859-1) to utf-8\n(not changing gitweb output encoding, which is utf-8).\n\nIt would be much better, and much easier at least for commit message\nto add --encoding=utf-8 to git-rev-list / git-log invocation.\n\n>> Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n>> ---\n>> For correct UTF-8, the patch does not change anything.\n>> \n>> If commit/blob/... is not in UTF-8, it displays the text\n>> with a very high probability correct. \n\nIt is commit (with its 'encoding' header, and `--encoding' option\nwe can use instead of doing it in gitweb, provided that git was\ncompiled with iconv support), tag (similar to commit, but IIRC\nwithout 'encoding' header, and `--encoding' option), blob (with\nno place to store encoding) and pathname in tree (which can be\ndifferent from blob encoding).\n\nAnd I doubt very much about this \"very high probability to be\ncorrect\".\n\n>> As git itself is not aware of any encoding, I know no better\n>> possibility to handle non UTF-8 text in gitweb.\n> \n> I don't think this is a reasonable approach; I actually dispute the high\n> probability - in western Europe it's obvious to assume latin1, but does\n> majority of users using non-ascii characters come from there? Or rather\n> from central Europe (like me, Petr Baudiš? ;-))? Somewhere else?\n\nI also don't think that hardcoding latin1 (iso-8859-1) as default\nalternate encoding is a good idea. I don't think using iso-8859-1\n(outside us-ascii) is _nowadays_ that common. On the other hand I think\nthat not all users of koi8r, eucjp or iso-2022-jp converted (and can\nconvert) to utf-8; latin1 users can.\n\nAnd using latin1 (other encoding) _only_ when there is an invalid utf-8\nsequence is not a good idea either; I think that that there are some\nlatin1 sequences outside us-ascii which are valid utf-8 sequences. That\nkind of magic is wrong, wrong, wrong...\n\n> If we do something like this, we should do it properly and look at\n> configured i18n.commitEncoding for the project. (But as config lookup\n> may be expensive, probably do it only when we need it.)\n\nI think it would be best to make it into %feature, overridable\nor not (which would look at i18n.commitEncoding instead of at\ngitweb.commitEncoding, but still a feature).\n\nAbout config lookup: we can either \"borrow\" config reading code in Perl\nfrom git-cvsserver, perhaps via putting it into Git.pm. Or we can\nimplement at last core git support for dumping whole config in\nunambiguous machine parseable output: \"git config --dump\", e.g.\n  key <LF> value <NUL>\nor\n  key <NUL>\n(the second for \"boolean\" variables without set value).\n\nHaving alternate (read-only) config parser has its advantages and\ndisadvantages. Advantage is that we avoid fork+exec (performance),\nand having two implementations is always good for having format\nstandarized. Disadvantage is that is yet another code to maintain,\nand that config parsing (even read-only config parsing) is a bit tricky\nwith current git config file format.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"43586","messageId":"20070529215536.GA13250@auto.tuwien.ac.at","threadId":"8342","inReplyTo":"200705291121.12119.jnareb@gmail.com","subject":"Re: [PATCH] gitweb: handle non UTF-8 text","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2007-05-29T21:55:36Z","receivedAt":"2007-05-29T21:55:36Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"On Tue, May 29, 2007 at 11:21:11AM +0200, Jakub Narebski wrote:\n> On Tue, 29 May 2007, Petr Baudis wrote:\n> > On Mon, May 28, 2007 at 10:47:34PM CEST, Martin Koegler wrote:\n> \n> >> gitweb assumes, that everything is in UTF-8. If a text contains invalid\n> >> UTF-8 character sequences, the text must be in a different encoding.\n> \n> But it doesn't tell us _what_ is the encoding. For commit messages,\n> with reasonable new git, we have 'encoding' header if git known that\n> commit message was not in utf-8.\n> \n> By the way, I winder why we don't have such header for tag objects\n> (i18n.tagEncoding ;-)...\n\nWhy do I need to set i18n.commitEncoding on a normal Linux systems?  We\nhave a locale, which contains this information. With this, its more\nlikely, that the commits can be read correctly later, if somebody\nforget to set \"i18n.commitEncoding\" in a repository.\n\n> >> This patch interprets such a text as latin1.\n> \n> Meaning that it tries to recode text from latin1 (iso-8859-1) to utf-8\n> (not changing gitweb output encoding, which is utf-8).\n> \n> It would be much better, and much easier at least for commit message\n> to add --encoding=utf-8 to git-rev-list / git-log invocation.\n\nIt does not help for old commits, where the encoding was not specified\ncorrectly. If my research is correct, the encoding handling was\nintroduced at the end of 2006 and released this february.\n\n> >> Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n> >> ---\n> >> For correct UTF-8, the patch does not change anything.\n> >> \n> >> If commit/blob/... is not in UTF-8, it displays the text\n> >> with a very high probability correct. \n> \n> It is commit (with its 'encoding' header, and `--encoding' option\n> we can use instead of doing it in gitweb, provided that git was\n> compiled with iconv support), tag (similar to commit, but IIRC\n> without 'encoding' header, and `--encoding' option), blob (with\n> no place to store encoding) and pathname in tree (which can be\n> different from blob encoding).\n> \n> And I doubt very much about this \"very high probability to be\n> correct\".\n\nFor normal text, this should be true:\n\nWe can divide ISO-8859-1 into some groups:\na) 0x00-0x7f: shared with UTF-8\nb) 0x80-0xBF: continuation characters in UTF-8 (0x80-0x9F are control characters/unused)\nc) 0xC0-0xDF: start of a two byte UTF-8 character\nd) 0xE0-0xEF: start of a tree byte UTF-8 character\ne) 0xF0-0xFF: start of other longer UTF-8 sequences\n\nTo misinterpret a ISO-8859-1 text as UTF-8, each character of class\nc/d/e must be followed by the correct number of character of class b.\n\nCharacter of class b are \"special character\", characters of class\nc/d/e are mostly special letters. As \"special character\" are normally not part\nof a word (at least in German),  any occurence of c/d/e at the begin\nor the in the middle of the word will therefore result in a invalid UTF-8\nsequence. Only a occurence of c/d/e at the end of an word, which is\nfollowed by the correct number occurences of class b result in a correct UTF-8\nsequence.\n\nIn german, the commonly used character of c/d/e are: ÄÖÜäöüß\nThe uppercase ÄÖÜ appear ony at the beginning of a word => invalid combination.\n\nOther combinations:\n* äöü followed by two \"special characters\"  (I don't know, where such a combination could occur).\n* ß followed by one \"special character\" (I regard this as the most likly misinterpretation).\n\nI can not speak for other languages. If you doubt, please look at an\ncharacter table (eg. http://en.wikipedia.org/wiki/ISO-8859-1#ISO-8859-1)\nand think about the possibiltiy of UTF-8 compatible combinations in your languague.\n\nAs gitweb is processing a line of text at once, one UTF-8 compatible\ncombinations has no effect, if any other non UTF-8 combatible\ncharacter sequence occurs.\n\n> >> As git itself is not aware of any encoding, I know no better\n> >> possibility to handle non UTF-8 text in gitweb.\n> > \n> > I don't think this is a reasonable approach; I actually dispute the high\n> > probability - in western Europe it's obvious to assume latin1, but does\n> > majority of users using non-ascii characters come from there? Or rather\n> > from central Europe (like me, Petr Baudiš? ;-))? Somewhere else?\n> \n> I also don't think that hardcoding latin1 (iso-8859-1) as default\n> alternate encoding is a good idea. I don't think using iso-8859-1\n> (outside us-ascii) is _nowadays_ that common. On the other hand I think\n> that not all users of koi8r, eucjp or iso-2022-jp converted (and can\n> convert) to utf-8; latin1 users can.\n\nUTF-8 is not the universal, dropin solution for ISO-8859-1. It has some drawbacks:\n- Some operations are slower, eg.\n$$ hexdump -C s\n00000000  78 0a 78 0a 78 0a 78 0a  78 0a 78 0a 78 0a 78 0a  |x.x.x.x.x.x.x.x.|\n*\n01000000\n$ grep --version\ngrep (GNU grep) 2.5.1\n$LANG=en_US.ISO-8859-15 time grep \"[a]\" s\nCommand exited with non-zero status 1\n0.38user 0.05system 0:00.46elapsed 93%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+219minor)pagefaults 0swaps\n$ LANG=en_US.UTF-8 time grep \"[a]\" s\nCommand exited with non-zero status 1\n10.86user 0.31system 0:14.29elapsed 78%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+17151minor)pagefaults 0swaps\n- Anything using string length/character position is more complicated.\n\nFor some problems, UTF-16 might be a simpler solution.\n\nBut I agree, that there should be the possibilty to choose a the\nfallback encoding.\n\n> And using latin1 (other encoding) _only_ when there is an invalid utf-8\n> sequence is not a good idea either; I think that that there are some\n> latin1 sequences outside us-ascii which are valid utf-8 sequences. That\n> kind of magic is wrong, wrong, wrong...\n\nPlease tell me a better alternative. The non UTF-8 will be in the history\n(in blobs/trees/commits/..) forever, where it can not be changed.\n\nI need a solution for this. I can use this patch on my system, but I\nwould like to see support other encodings in upstream gitweb.\n\n> > If we do something like this, we should do it properly and look at\n> > configured i18n.commitEncoding for the project. (But as config lookup\n> > may be expensive, probably do it only when we need it.)\n> \n> I think it would be best to make it into %feature, overridable\n> or not (which would look at i18n.commitEncoding instead of at\n> gitweb.commitEncoding, but still a feature).\n\nI would use i18n.commitEncoding only as last fallback. In a project\nmore different encodings could be used and the guessing logic may need\nadditional parameter, so I would create a own set of config parameters\nfor this.\n\n> About config lookup: we can either \"borrow\" config reading code in Perl\n> from git-cvsserver, perhaps via putting it into Git.pm. Or we can\n> implement at last core git support for dumping whole config in\n> unambiguous machine parseable output: \"git config --dump\", e.g.\n>   key <LF> value <NUL>\n> or\n>   key <NUL>\n> (the second for \"boolean\" variables without set value).\n\nIf we use a new file (in the gitweb config format), the whole thing\nwill be faster and less complicated.\n\nmfg Martin Kögler\n"},{"id":"43661","messageId":"200705302218.19408.robin.rosenberg.lists@dewire.com","threadId":"8342","inReplyTo":"20070529215536.GA13250@auto.tuwien.ac.at","subject":"Re: [PATCH] gitweb: handle non UTF-8 text","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2007-05-30T20:18:19Z","receivedAt":"2007-05-30T20:18:19Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"tisdag 29 maj 2007 skrev Martin Koegler:\n> On Tue, May 29, 2007 at 11:21:11AM +0200, Jakub Narebski wrote:\n> > On Tue, 29 May 2007, Petr Baudis wrote:\n> > > On Mon, May 28, 2007 at 10:47:34PM CEST, Martin Koegler wrote:\n> > \n> > >> gitweb assumes, that everything is in UTF-8. If a text contains invalid\n> > >> UTF-8 character sequences, the text must be in a different encoding.\n> > \n> > But it doesn't tell us _what_ is the encoding. For commit messages,\n> > with reasonable new git, we have 'encoding' header if git known that\n> > commit message was not in utf-8.\n> > \n> > By the way, I winder why we don't have such header for tag objects\n> > (i18n.tagEncoding ;-)...\n> \n> Why do I need to set i18n.commitEncoding on a normal Linux systems?  We\nI've asked the same question.. :(\n> have a locale, which contains this information. With this, its more\n> likely, that the commits can be read correctly later, if somebody\n> forget to set \"i18n.commitEncoding\" in a repository.\nNo 'if'. Users are virtually guaranteed to forget this setting.\n\n> \n> UTF-8 is not the universal, dropin solution for ISO-8859-1. It has some drawbacks:\n> - Some operations are slower, eg.\n> - Anything using string length/character position is more complicated.\nWe'll have to live with that. A nice property of valid UTF-8 is that many operations can\nbe performed without decoding (like looking for a substring).\n\n> \n> For some problems, UTF-16 might be a simpler solution.\nUTF-16 is also variable width (one or two code units). Most apps get away by pretending it is \nfixed width, simply because that works for most people, but then I'm not sure people in asia \naren't really happy with that assumption either. \n\n> I would use i18n.commitEncoding only as last fallback. In a project\n> more different encodings could be used and the guessing logic may need\n> additional parameter, so I would create a own set of config parameters\n> for this.\n\nThere aren't many simple ways of guessing. The UTF-8 vs other test is simple \nand very reliable for western encodings (and merely good for others, if I'm not misinformed).\nThe i18n.commitEncoding is just a hint. Another hint is the host's encoding.\n\n1. if lookslike(UTF-8) => assume UTF-8 else...\n2. commit's encoding is valid for the text => use it else...\n3. i18n.commitEncoding ...\n4. gitweb.commitencoding  ....\n5. server's location charset ...\n6. assume iso-8859-1\n\nYet another would be to have an extra option to switch encoding on-demand in the gui.\n\nBTW, there's another thread on notes. Maybe they be used to \"fix\" badly encoded messages\nif and when they get a final implementation.\n\n-- robn\n"},{"id":"43770","messageId":"200706012305.41183.jnareb@gmail.com","threadId":"8342","inReplyTo":"20070529215536.GA13250@auto.tuwien.ac.at","subject":"Re: [PATCH] gitweb: handle non UTF-8 text","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-06-01T21:05:40Z","receivedAt":"2007-06-01T21:05:40Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 29 May 2007, Martin Koegler wrote:\n> On Tue, May 29, 2007 at 11:21:11AM +0200, Jakub Narebski wrote:\n>> On Tue, 29 May 2007, Petr Baudis wrote:\n>>> On Mon, May 28, 2007 at 10:47:34PM CEST, Martin Koegler wrote:\n>> \n>>>> gitweb assumes, that everything is in UTF-8. If a text contains invalid\n>>>> UTF-8 character sequences, the text must be in a different encoding.\n>> \n>> But it doesn't tell us _what_ is the encoding. For commit messages,\n>> with reasonable new git, we have 'encoding' header if git known that\n>> commit message was not in utf-8.\n>> \n>> By the way, I winder why we don't have such header for tag objects\n>> (i18n.tagEncoding ;-)...\n> \n> Why do I need to set i18n.commitEncoding on a normal Linux systems?  We\n> have a locale, which contains this information. With this, its more\n> likely, that the commits can be read correctly later, if somebody\n> forget to set \"i18n.commitEncoding\" in a repository.\n\nBecause repository is (or at least can be) _shared_. People working on\nthe same repository can have set different locale. Web server running\ngitweb can have different locale.\n\n>>>> This patch interprets such a text as latin1.\n>> \n>> Meaning that it tries to recode text from latin1 (iso-8859-1) to utf-8\n>> (not changing gitweb output encoding, which is utf-8).\n\nAnd this (i.e. what does \"interprets\" mean) is what should be in the\ncommit message too.\n\n>> It would be much better, and much easier at least for commit message\n>> to add --encoding=utf-8 to git-rev-list / git-log invocation.\n> \n> It does not help for old commits, where the encoding was not specified\n> correctly. If my research is correct, the encoding handling was\n> introduced at the end of 2006 and released this february.\n\nTrue. But it _can_ help.\n\n>>>> If commit/blob/... is not in UTF-8, it displays the text\n>>>> with a very high probability correct. \n>>\n>> And I doubt very much about this \"very high probability to be\n>> correct\".\n> \n> For normal text, this should be true:\n> \n> We can divide ISO-8859-1 into some groups:\n> a) 0x00-0x7f: shared with UTF-8\n> b) 0x80-0xBF: continuation characters in UTF-8 (0x80-0x9F are control characters/unused)\n> c) 0xC0-0xDF: start of a two byte UTF-8 character\n> d) 0xE0-0xEF: start of a tree byte UTF-8 character\n> e) 0xF0-0xFF: start of other longer UTF-8 sequences\n> \n> To misinterpret a ISO-8859-1 text as UTF-8, each character of class\n> c/d/e must be followed by the correct number of character of class b.\n[cut]\n> As gitweb is processing a line of text at once, one UTF-8 compatible\n> combinations has no effect, if any other non UTF-8 combatible\n> character sequence occurs.\n\nThanks for the explanation. In short: if characters not shared with UTF-8\n(outside US-ASCII), \"special characters\" occur usually solo, there is\nlow probability that line in non-UTF-8 encoding will be valid UTF-8.\nWhich perhaps is valid for German and latin1 aka. iso-8859-1; not\nnecessarily so for example for Polish and iso-8859-2, see\n  zażółć gęsią jaźń\nwhich is perfectly good fragment containing all Polish special\ncharacters, and as you can see those characters occur one after another.\nWell, it still could be invalid UTF-8 sequence; what about koi8r and\neucjp (or other non-UTF-8 encoding for Asian languages)?\n\n> But I agree, that there should be the possibilty to choose a the\n> fallback encoding.\n\nI think for the beginning it would be enough to have\n\n  # assume this charset if line contains non-UTF-8 characters\n  our $fallback_encoding = \"latin1\";\n\nor something like that (perhaps different wording in the comment,\nperhaps different name of the variable) in the gitweb.perl for your\nidea to be accepted.\n\nThat, and using to_utf8 (as before e3ad95a8) and not my_decode_utf8\nas subroutine name. If only it would be possible to avoid I think\nquote costly \"eval {....}\" invocation...\n\n\n[cut]\n\nThere are six sources of possibly non-UTF-8 input: commits, tags,\ntrees (file names), blobs, gitweb files and results of system calls.\n\nOnly first one, commits, comes with encoding specified... if commit\nwas made with new enough git, and if committer correctly specified\nencoding. Commits are read using git-rev-list, which accept --encoding\nparameter, so we can convert it easily to utf-8... if git was compiled\nwith iconv support. It is possible that due to repository, gitweb user\nor global configuration (i18n.logOutputEncoding, i18n.commitEncoding)\nthis is done automatically. On the other hand I think it is easiest\nto have accidental wrongly encoded sequence in commit message.\n\nSecond one, tags, really _should_ have encoding header like commits.\nOn the other hand usually the message is version + PGP signature, so\nthere is no place for any encoding. Tags are read using git-cat-file,\nwhich does not do any encoding/decoding.\n\nThird, filenames in tree objects, \"suffers\" from git design decision:\nfor performance and simplicity git stories filenames in tree 'as is',\nand relies on the fact that filenames are the same in tree objects,\nin the index (dircache), in the filesystem during saving, and as read\nfrom filesystem. Moreover I think that names encoding on filesystem\nmight depend on filesystem in question and be different from locale\nspecified encoding (locale is user local, filesystem is global).\nOn the other hand side one ususually does not use special characters\nin filenames because of the problems they cause.\n\nFourth, blobs (file contents). They can use different encoding than\ncommit messages; moreover different files can use different encoding.\nEncoding has to be specified externally; there is no place for encoding\nheader in the blob object structure.\n\nFifth, gitweb files include files read and transformed such as \nGIT_DIR/description file, or projects index file $projects_list,\nand files containing fragments of HTML like README.html or header/footer\nfiles.\n\nSixth, we sometimes have to decode to utf8 results of system calls\nlike getpwuid to get owner of a file (of a project), or decode to utf8\npath (fragment) to the repository.\n\n\nThere are two places to specify gitweb output charset. First is charset\nused in HTML output, which is also default charset (binmode) of STDOUT\nstream. Gitweb uses utf-8 here, and utf-8 is recommended for XML and for\nXHTML by W3C, although we could theoretically add an option to use\ndifferent charset by default, and decode (or not) to this charset, instead\nof recoding everything (see above) to utf-8.\n\nSecond place is default charset for text/plain blob_plain output:\n  # default blob_plain mimetype and default charset for text/plain blob\n  our $default_blob_plain_mimetype = 'text/plain';\n  our $default_text_plain_charset  = undef;\nand for other *_plain output written as text/plain; charset=utf-8, and\nwhich is actually dumpled :raw to STDOUT.\n\n\nSo what should be the solution? Add global, per gitweb installation\nconfigureation variables $input_encoding and $fallback_input_encoding?\nWhat do you think? Do you have other ideas?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"43831","messageId":"7vvee6nguh.fsf@assigned-by-dhcp.cox.net","threadId":"8342","inReplyTo":"200706012305.41183.jnareb@gmail.com","subject":"Re: [PATCH] gitweb: handle non UTF-8 text","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-06-02T22:15:18Z","receivedAt":"2007-06-02T22:15:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> On Tue, 29 May 2007, Martin Koegler wrote:\n> ...\n>> But I agree, that there should be the possibilty to choose a the\n>> fallback encoding.\n>\n> I think for the beginning it would be enough to have\n>\n>   # assume this charset if line contains non-UTF-8 characters\n>   our $fallback_encoding = \"latin1\";\n>\n> or something like that (perhaps different wording in the comment,\n> perhaps different name of the variable) in the gitweb.perl for your\n> idea to be accepted.\n>\n> That, and using to_utf8 (as before e3ad95a8) and not my_decode_utf8\n> as subroutine name. If only it would be possible to avoid I think\n> quote costly \"eval {....}\" invocation...\n\nExcept that I had an impression that block form of \"eval\" (as\nopposed to \"parse and evaluate string\" kind) was not costly at\nall.\n\nPlease make it so.\n\nI'll read the other parts of your message again -- I might have\nfurther comments.\n"},{"id":"43892","messageId":"200706031742.45216.jnareb@gmail.com","threadId":"8342","inReplyTo":"7vvee6nguh.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] gitweb: handle non UTF-8 text","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-06-03T15:42:44Z","receivedAt":"2007-06-03T15:42:44Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Alexandre, I hope that the patch attached would solve your problem.\n\nJunio C Hamano wrote:\n> Jakub Narebski <jnareb@gmail.com> writes:\n> \n>> On Tue, 29 May 2007, Martin Koegler wrote:\n>> ...\n>>> But I agree, that there should be the possibilty to choose the\n>>> fallback encoding.\n>>\n>> I think for the beginning it would be enough to have\n>>\n>>   # assume this charset if line contains non-UTF-8 characters\n>>   our $fallback_encoding = \"latin1\";\n\nAdded this, with more elaborate comment, just before %feature hash.\n\n>> or something like that (perhaps different wording in the comment,\n>> perhaps different name of the variable) in the gitweb.perl for your\n>> idea to be accepted.\n>>\n>> That, and using to_utf8 (as before e3ad95a8) and not my_decode_utf8\n>> as subroutine name. If only it would be possible to avoid I think\n>> quote costly \"eval {....}\" invocation...\n> \n> Except that I had an impression that block form of \"eval\" (as\n> opposed to \"parse and evaluate string\" kind) was not costly at\n> all.\n\nI have checked the time it took to run the gitweb test (t9500),\nand the time to run (user+sys) increased about 1% after this patch,\nwhich is even within range of error I think.\n \n> Please make it so.\n\nI have changed the name from my_decode_utf8 to the name used for thin\nwrapper before commit e3ad95a8 \"gitweb: use decode_utf8 directly\", namely\nto_utf8, and put it in the place where old to_utf8 subroutine was.\n\nInstead of bit hackish \"return $res || decode('latin1', $str);\" use\n\"if (defined $res) { ... } else { ... }\"; it avoids calling decode()\nunnecessary for '' and '0' strings, which are also false, but do not mean\nthat decode_utf8 failed.\n\nIt uses explicit constant names, Encode::FB_CROAK instead of 1, and\nEncode::FB_DEFAULT instead of default undef/0.\n\nIt adds very, very basic test: it does check _only_ if there are any\nerrors or warning which would go to web server log; it does not check\nif the output is correct. It uses helper files from other i18n tests.\n\nAdded comments.\n\n\nStill the main change is by Martin Koegler and he should be author of\nthis commit, I think. I have added S-o-b: by me.\n\n-- >8 --\nFrom: Martin Koegler <mkoegler@auto.tuwien.ac.at>\nSubject: [PATCH] gitweb: Handle non UTF-8 text better\n\ngitweb assumes that everything is in UTF-8. If a text contains invalid\nUTF-8 character sequences, the text must be in a different encoding.\n\nThis commit introduces $fallback_encoding which would be used as input\nencoding if gitweb encounters text with is not valid UTF-8.\n\nAdd basic test for this in t/t9500-gitweb-standalone-no-errors.sh\n\nSigned-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\nSigned-off-by: Jakub Narebski <jnareb@gmail.com>\n---\n gitweb/gitweb.perl                     |   41 ++++++++++++++++++++++++-------\n t/t9500-gitweb-standalone-no-errors.sh |   28 +++++++++++++++++++++\n 2 files changed, 59 insertions(+), 10 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex c3921cb..e92596c 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -94,6 +94,13 @@ our $default_text_plain_charset  = undef;\n # (relative to the current git repository)\n our $mimetypes_file = undef;\n \n+# assume this charset if line contains non-UTF-8 characters;\n+# it should be valid encoding (see Encoding::Supported(3pm) for list),\n+# for which encoding all byte sequences are valid, for example\n+# 'iso-8859-1' aka 'latin1' (it is decoded without checking, so it\n+# could be even 'utf-8' for the old behavior)\n+our $fallback_encoding = 'latin1';\n+\n # You define site-wide feature defaults here; override them with\n # $GITWEB_CONFIG as necessary.\n our %feature = (\n@@ -602,6 +609,20 @@ sub validate_refname {\n \treturn $input;\n }\n \n+# decode sequences of octets in utf8 into Perl's internal form,\n+# which is utf-8 with utf8 flag set if needed.  gitweb writes out\n+# in utf-8 thanks to \"binmode STDOUT, ':utf8'\" at beginning\n+sub to_utf8 {\n+\tmy $str = shift;\n+\tmy $res;\n+\teval { $res = decode_utf8($str, Encode::FB_CROAK); };\n+\tif (defined $res) {\n+\t\treturn $res;\n+\t} else {\n+\t\treturn decode($fallback_encoding, $str, Encode::FB_DEFAULT);\n+\t}\n+}\n+\n # quote unsafe chars, but keep the slash, even when it's not\n # correct, but quoted slashes look too horrible in bookmarks\n sub esc_param {\n@@ -626,7 +647,7 @@ sub esc_html ($;%) {\n \tmy $str = shift;\n \tmy %opts = @_;\n \n-\t$str = decode_utf8($str);\n+\t$str = to_utf8($str);\n \t$str = $cgi->escapeHTML($str);\n \tif ($opts{'-nbsp'}) {\n \t\t$str =~ s/ /&nbsp;/g;\n@@ -640,7 +661,7 @@ sub esc_path {\n \tmy $str = shift;\n \tmy %opts = @_;\n \n-\t$str = decode_utf8($str);\n+\t$str = to_utf8($str);\n \t$str = $cgi->escapeHTML($str);\n \tif ($opts{'-nbsp'}) {\n \t\t$str =~ s/ /&nbsp;/g;\n@@ -925,7 +946,7 @@ sub format_subject_html {\n \n \tif (length($short) < length($long)) {\n \t\treturn $cgi->a({-href => $href, -class => \"list subject\",\n-\t\t                -title => decode_utf8($long)},\n+\t\t                -title => to_utf8($long)},\n \t\t       esc_html($short) . $extra);\n \t} else {\n \t\treturn $cgi->a({-href => $href, -class => \"list subject\"},\n@@ -1239,7 +1260,7 @@ sub git_get_projects_list {\n \t\t\tif (check_export_ok(\"$projectroot/$path\")) {\n \t\t\t\tmy $pr = {\n \t\t\t\t\tpath => $path,\n-\t\t\t\t\towner => decode_utf8($owner),\n+\t\t\t\t\towner => to_utf8($owner),\n \t\t\t\t};\n \t\t\t\tpush @list, $pr;\n \t\t\t\t(my $forks_path = $path) =~ s/\\.git$//;\n@@ -1269,7 +1290,7 @@ sub git_get_project_owner {\n \t\t\t$pr = unescape($pr);\n \t\t\t$ow = unescape($ow);\n \t\t\tif ($pr eq $project) {\n-\t\t\t\t$owner = decode_utf8($ow);\n+\t\t\t\t$owner = to_utf8($ow);\n \t\t\t\tlast;\n \t\t\t}\n \t\t}\n@@ -1759,7 +1780,7 @@ sub get_file_owner {\n \t}\n \tmy $owner = $gcos;\n \t$owner =~ s/[,;].*$//;\n-\treturn decode_utf8($owner);\n+\treturn to_utf8($owner);\n }\n \n ## ......................................................................\n@@ -1842,7 +1863,7 @@ sub git_header_html {\n \n \tmy $title = \"$site_name\";\n \tif (defined $project) {\n-\t\t$title .= \" - \" . decode_utf8($project);\n+\t\t$title .= \" - \" . to_utf8($project);\n \t\tif (defined $action) {\n \t\t\t$title .= \"/$action\";\n \t\t\tif (defined $file_name) {\n@@ -2116,7 +2137,7 @@ sub git_print_page_path {\n \n \tprint \"<div class=\\\"page_path\\\">\";\n \tprint $cgi->a({-href => href(action=>\"tree\", hash_base=>$hb),\n-\t              -title => 'tree root'}, decode_utf8(\"[$project]\"));\n+\t              -title => 'tree root'}, to_utf8(\"[$project]\"));\n \tprint \" / \";\n \tif (defined $name) {\n \t\tmy @dirname = split '/', $name;\n@@ -2936,7 +2957,7 @@ sub git_project_list_body {\n \t\t($pr->{'age'}, $pr->{'age_string'}) = @aa;\n \t\tif (!defined $pr->{'descr'}) {\n \t\t\tmy $descr = git_get_project_description($pr->{'path'}) || \"\";\n-\t\t\t$pr->{'descr_long'} = decode_utf8($descr);\n+\t\t\t$pr->{'descr_long'} = to_utf8($descr);\n \t\t\t$pr->{'descr'} = chop_str($descr, 25, 5);\n \t\t}\n \t\tif (!defined $pr->{'owner'}) {\n@@ -3981,7 +4002,7 @@ sub git_snapshot {\n \tmy $git = git_cmd_str();\n \tmy $name = $project;\n \t$name =~ s/\\047/\\047\\\\\\047\\047/g;\n-\tmy $filename = decode_utf8(basename($project));\n+\tmy $filename = to_utf8(basename($project));\n \tmy $cmd;\n \tif ($suffix eq 'zip') {\n \t\t$filename .= \"-$hash.$suffix\";\ndiff --git a/t/t9500-gitweb-standalone-no-errors.sh b/t/t9500-gitweb-standalone-no-errors.sh\nindex b92ab63..44ae503 100755\n--- a/t/t9500-gitweb-standalone-no-errors.sh\n+++ b/t/t9500-gitweb-standalone-no-errors.sh\n@@ -487,4 +487,32 @@ test_expect_success \\\n \t'gitweb_run \"p=.git;a=atom\"'\n test_debug 'cat gitweb.log'\n \n+# ----------------------------------------------------------------------\n+# encoding/decoding\n+\n+test_expect_success \\\n+\t'encode(commit): utf8' \\\n+\t'. ../t3901-utf8.txt &&\n+\t echo \"UTF-8\" >> file &&\n+\t git add file &&\n+\t git commit -F ../t3900/1-UTF-8.txt &&\n+\t gitweb_run \"p=.git;a=commit\"'\n+test_debug 'cat gitweb.log'\n+\n+test_expect_success \\\n+\t'encode(commit): iso-8859-1' \\\n+\t'. ../t3901-8859-1.txt &&\n+\t echo \"ISO-8859-1\" >> file &&\n+\t git add file &&\n+\t git config i18n.commitencoding ISO-8859-1 &&\n+\t git commit -F ../t3900/ISO-8859-1.txt &&\n+\t git config --unset i18n.commitencoding &&\n+\t gitweb_run \"p=.git;a=commit\"'\n+test_debug 'cat gitweb.log'\n+\n+test_expect_success \\\n+\t'encode(log): utf-8 and iso-8859-1' \\\n+\t'gitweb_run \"p=.git;a=log\"'\n+test_debug 'cat gitweb.log'\n+\n test_done\n-- \n1.5.2\n"},{"id":"43896","messageId":"87lkf0swxr.fsf@wine.dyndns.org","threadId":"8342","inReplyTo":"200706031742.45216.jnareb@gmail.com","subject":"Re: [PATCH] gitweb: handle non UTF-8 text","fromName":"Alexandre Julliard","fromEmail":"julliard@winehq.org","sentAt":"2007-06-03T18:41:04Z","receivedAt":"2007-06-03T18:41:04Z","isPatch":true,"sender":{"key":"julliard@winehq.org","avatar":null},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> Alexandre, I hope that the patch attached would solve your problem.\n\nYes, it works fine for me, thanks!\n\n-- \nAlexandre Julliard\njulliard@winehq.org\n"}]}