{"thread":{"id":"13592","subject":"[PATCH] gitweb: Fix chop_str not to cut in middle of utf8 multibyte chars.","startedAt":"2008-05-20T20:55:41Z","lastAt":"2008-05-24T13:34:23Z","messageCount":6,"participants":["Anders Waldenborg","Jakub Narebski","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"77345","messageId":"g0vdse$rj6$1@ger.gmane.org","threadId":"13592","inReplyTo":null,"subject":"[PATCH] gitweb: Fix chop_str not to cut in middle of utf8 multibyte chars.","fromName":"Anders Waldenborg","fromEmail":"anders@0x63.nu","sentAt":"2008-05-20T20:55:41Z","receivedAt":"2008-05-20T20:55:41Z","isPatch":true,"sender":{"key":"anders@0x63.nu","avatar":"https://avatars.githubusercontent.com/u/1566016?v=4"},"body":"\nWithout this fix at least author name in short log may cut in middle of a\nmultibyte char. When the result comes to esc_html to_utf8 is called again,\nwhich doesn't find valid utf8 and decodes using $fallback_encoding making\nit even worse.\n\nSigned-off-by: Anders Waldenborg \n---\n gitweb/gitweb.perl | 4 ++++\n 1 files changed, 4 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 2facf2d..8308e22 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -866,6 +866,10 @@ sub chop_str {\n my $add_len = shift || 10;\n my $where = shift || 'right'; # 'left' | 'center' | 'right'\n\n+ # Make sure perl knows it is utf8 encoded so we don't\n+ # cut in the middle of a utf8 multibyte char.\n+ $str = to_utf8($str);\n+\n # allow only $len chars, but don't cut a word if it would fit in $add_len\n # if it doesn't fit, cut it if it's still longer than the dots we would add\n # remove chopped character entities entirely\n"},{"id":"77353","messageId":"m3lk244o16.fsf@localhost.localdomain","threadId":"13592","inReplyTo":"g0vdse$rj6$1@ger.gmane.org","subject":"Re: [PATCH] gitweb: Fix chop_str not to cut in middle of utf8 multibyte chars.","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-20T22:19:01Z","receivedAt":"2008-05-20T22:19:01Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Anders Waldenborg writes:\n\n> Without this fix at least author name in short log may cut in middle of a\n> multibyte char. When the result comes to esc_html to_utf8 is called again,\n> which doesn't find valid utf8 and decodes using $fallback_encoding making\n> it even worse.\n\nThanks a lot. This is certainly a good thing, although I think that\nthe proper solution (but which would need much more work) would be to\nensure that all information is stored in Perl internal form, and not\nonly ensured on output.\n\nI would change title of this commit to be more descriptive what this\ncommit does; or rather make current subject the first sentence of\ncommit summary, have something like the following for commit message:\n\ngitweb: Convert string to Perl internal form before chopping in chop_str\n\nFix chop_str not to cut in middle of utf8 multibyte chars. Without\nthis fix at least author name in short log may cut in middle of a\nmultibyte char. When the result comes to esc_html to_utf8 is called\nagain, which doesn't find valid utf8 and decodes using\n$fallback_encoding making it even worse.\n \n> Signed-off-by: Anders Waldenborg \n> ---\n> gitweb/gitweb.perl | 4 ++++\n> 1 files changed, 4 insertions(+), 0 deletions(-)\n> \n> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n> index 2facf2d..8308e22 100755\n> --- a/gitweb/gitweb.perl\n> +++ b/gitweb/gitweb.perl\n> @@ -866,6 +866,10 @@ sub chop_str {\n> my $add_len = shift || 10;\n> my $where = shift || 'right'; # 'left' | 'center' | 'right'\n> \n> + # Make sure perl knows it is utf8 encoded so we don't\n> + # cut in the middle of a utf8 multibyte char.\n> + $str = to_utf8($str);\n> +\n\nI like the comment here. It explains the whys of code.\n\n> # allow only $len chars, but don't cut a word if it would fit in $add_len\n> # if it doesn't fit, cut it if it's still longer than the dots we would add\n> # remove chopped character entities entirely\n> \n\nThis patch is whitespace damaged, by the way.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"77375","messageId":"7vve185d6s.fsf@gitster.siamese.dyndns.org","threadId":"13592","inReplyTo":"m3lk244o16.fsf@localhost.localdomain","subject":"Re: [PATCH] gitweb: Fix chop_str not to cut in middle of utf8 multibyte chars.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-05-21T07:27:55Z","receivedAt":"2008-05-21T07:27:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski writes:\n\n> Anders Waldenborg writes:\n> ...\n> Fix chop_str not to cut in middle of utf8 multibyte chars. Without\n> this fix at least author name in short log may cut in middle of a\n> multibyte char. When the result comes to esc_html to_utf8 is called\n> again, which doesn't find valid utf8 and decodes using\n> $fallback_encoding making it even worse.\n> \n>> Signed-off-by: Anders Waldenborg \n>> ---\n>> gitweb/gitweb.perl | 4 ++++\n>> 1 files changed, 4 insertions(+), 0 deletions(-)\n>> \n>> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\n>> index 2facf2d..8308e22 100755\n>> --- a/gitweb/gitweb.perl\n>> +++ b/gitweb/gitweb.perl\n>> @@ -866,6 +866,10 @@ sub chop_str {\n>> my $add_len = shift || 10;\n>> my $where = shift || 'right'; # 'left' | 'center' | 'right'\n>> \n>> + # Make sure perl knows it is utf8 encoded so we don't\n>> + # cut in the middle of a utf8 multibyte char.\n>> + $str = to_utf8($str);\n>> +\n>\n> I like the comment here. It explains the whys of code.\n>\n>> # allow only $len chars, but don't cut a word if it would fit in $add_len\n>> # if it doesn't fit, cut it if it's still longer than the dots we would add\n>> # remove chopped character entities entirely\n>> \n>\n> This patch is whitespace damaged, by the way.\n\nI haven't followed the codepath but what do the callers do to the string\nreturned from chop_str? Don't they assume the string hasn't been decoded\n(because the old implementation of chop_str did not do this to_utf8), and\nemit the result directly to the output because it also assumes the\nundecoded format is what the outside world wants? In other words, don't\nthey now need to do different things because returned string has gone\nthrough the to_utf8() processing already?\n\nMaybe I am worrying too much, after getting burned by decode_utf/encode_utf\ndata chains in another popular scripting language, and it is possible that\nwith Perl you may not have to be so careful...\n\n\n \n"},{"id":"77376","messageId":"4833D314.4010904@0x63.nu","threadId":"13592","inReplyTo":"7vve185d6s.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] gitweb: Fix chop_str not to cut in middle of utf8 multibyte chars.","fromName":"Anders Waldenborg","fromEmail":"anders@0x63.nu","sentAt":"2008-05-21T07:45:24Z","receivedAt":"2008-05-21T07:45:24Z","isPatch":true,"sender":{"key":"anders@0x63.nu","avatar":"https://avatars.githubusercontent.com/u/1566016?v=4"},"body":"> I haven't followed the codepath but what do the callers do to the string\n> returned from chop_str? Don't they assume the string hasn't been decoded\n> (because the old implementation of chop_str did not do this to_utf8), and\n> emit the result directly to the output because it also assumes the\n> undecoded format is what the outside world wants? In other words, don't\n> they now need to do different things because returned string has gone\n> through the to_utf8() processing already?\n\nThe to_utf8() (defined in gitweb.perl, not part of perl it self) is kind \nof sneaky, it checks if the string already is valid utf8. (guess it \nshould be called ensure_utf8())\n\nchop_str needs to work on decoded string, otherwise character count goes \nall wrong. But maybe it is better to add the to_utf8() to the callsites?\n\n anders\n"},{"id":"77383","messageId":"20080521114443.GA1576@0x63.nu","threadId":"13592","inReplyTo":"m3lk244o16.fsf@localhost.localdomain","subject":"[PATCH] gitweb: Convert string to internal form before chopping in chop_str","fromName":"Anders Waldenborg","fromEmail":"anders@0x63.nu","sentAt":"2008-05-21T11:44:43Z","receivedAt":"2008-05-21T11:44:43Z","isPatch":true,"sender":{"key":"anders@0x63.nu","avatar":"https://avatars.githubusercontent.com/u/1566016?v=4"},"body":"Fix chop_str not to cut in middle of utf8 multibyte chars. Without\nthis fix at least author name in short log may cut in middle of a\nmultibyte char. When the result comes to esc_html to_utf8 is called\nagain, which doesn't find valid utf8 and decodes using\n$fallback_encoding making it even worse.\n\nThis also have the nice side effect that it actually tries to show the\nfirst 10 _characters_, not the number of characters that happened to fit\ninto 10 bytes.\n\nSigned-off-by: Anders Waldenborg \n---\n\nDescription updated as per your suggesion, and hopefully without the\nembarrassing whitespace corruption.\n\ngitweb/gitweb.perl | 4 ++++\n 1 files changed, 4 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 2facf2d..8308e22 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -866,6 +866,10 @@ sub chop_str {\n \tmy $add_len = shift || 10;\n \tmy $where = shift || 'right'; # 'left' | 'center' | 'right'\n \n+\t# Make sure perl knows it is utf8 encoded so we don't\n+\t# cut in the middle of a utf8 multibyte char.\n+\t$str = to_utf8($str);\n+\n \t# allow only $len chars, but don't cut a word if it would fit in $add_len\n \t# if it doesn't fit, cut it if it's still longer than the dots we would add\n \t# remove chopped character entities entirely\n-- \n1.5.5.1\n"},{"id":"77656","messageId":"200805241534.25517.jnareb@gmail.com","threadId":"13592","inReplyTo":"4833D314.4010904@0x63.nu","subject":"Re: [PATCH] gitweb: Fix chop_str not to cut in middle of utf8 multibyte chars.","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-05-24T13:34:23Z","receivedAt":"2008-05-24T13:34:23Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 21 May 2008, Anders Waldenborg wrote:\n> Junio C Hamano wrote:\n\n>> I haven't followed the codepath but what do the callers do to the string\n>> returned from chop_str? Don't they assume the string hasn't been decoded\n>> (because the old implementation of chop_str did not do this to_utf8), and\n>> emit the result directly to the output because it also assumes the\n>> undecoded format is what the outside world wants? In other words, don't\n>> they now need to do different things because returned string has gone\n>> through the to_utf8() processing already?\n> \n> The to_utf8() (defined in gitweb.perl, not part of perl it self) is kind \n> of sneaky, it checks if the string already is valid utf8. (guess it \n> should be called ensure_utf8())\n\nPerhaps it should...\n\n> chop_str needs to work on decoded string, otherwise character count goes \n> all wrong. But maybe it is better to add the to_utf8() to the callsites?\n\nOr do \"binmode $fd, :utf8\".\n\nBut yes, I guess converting to Perl internal form on input would be\ngood idea. Gitweb currently does it partially...\n\n-- \nJakub Narebski\nPoland\n"}]}