Re: gitweb and unicode special characters

From: Jakub Narebski <jnareb@gmail.com>
Date: Dec 13, 2008, 01:31 UTC
Message-ID: <200812130231.06929.jnareb@gmail.com>
In-Reply-To: <3f2beab60812121655m6cd868bfhaaf386e6f5457533@mail.gmail.com>

On Sat, 13 Dec 2008 01:55, Praveen A wrote:

Show 19 quoted lines

> 2008/12/12 Jakub Narebski <jnareb@gmail.com>:
>> Jakub Narebski <jnareb@gmail.com> writes:
>>> "Praveen A" <pravi.a@gmail.com> writes:
>>>
>>>> Git currently does not handle unicode special characters ZWJ and ZWNJ,
>>>> both are heavily used in Malayalam and common in other languages
>>>> needing complex text layout like Sinhala and Arabic.
>>>>
>>>> An example of this is shown in the commit message here
>>>> http://git.savannah.gnu.org/gitweb/?p=smc.git;a=commit;h=c3f368c60aabdc380c77608c614d91b0a628590a
>>>>
>>>> \20014 and \20015 should have been ZWNJ and ZWJ respectively. You just
>>>> need to handle them as any other unicode character - especially it is
>>>> a commit message and expectation is normal pain text display.
>>>
>>> [...] git_commit calls format_log_line_html, which
>>> in turn calls esc_html.  esc_html looks like this:
>>>
>>>   sub esc_html ($;%) {

[...]

Show 13 quoted lines

>>>   **  $str =~ s|([[:cntrl:]])|(($1 ne "\t") ? quot_cec($1) : $1)|eg;
>>>       return $str;
>>>   }
>>>
>>> The two important lines are marked with '**'.
>> [...]
>>
>>> So it looks like Perl treats \20014 and \20015 (ZWNJ and ZWJ) as
>>> belonging to '[:cntrl:]' class. I don't know if it is correct from the
>>> point of view of Unicode character classes, therefore if it is a bug
>>> in Perl, or just in gitweb.
>>
>> I checked this, via this simple Perl script:

[...]

Show 12 quoted lines

>>  "\N{ZWNJ}" =~ /[[:cntrl:]]/ and print "is [:cntrl:]";
>>
>> And the answer was:
>>
>>  oct=20014 dex=8204 hex=200c
>>  is [:cntrl:]
>>
>> 'ZERO WIDTH NON-JOINER' _is_ control character... We probably should
>> use [^[:print:][:space:]] instead of [[:cntrl:]] here.
> 
> That looks good. But I'm wondering why do we need to filter at all?
> Is it a security concern? It is just description.

First, from the new description [^[:print:][:space:]], or even [^[:print:]] (whichever we choose) you can see that those characters we are showing using C (\r, \v, \b,...) + octal (in older gitweb) or hex (in never gitweb) escapes would be invisible otherwise, or do the strange things like \b aka backspace character.

Sidenote: There is probably one exception we want to add, namely not
escape '\r' at the end of line, to be able to deal better with DOS
line endings (\r\n).

Second, and that is I think reason we started to escape control characters like \014 or ^L i.e. FORM FEED (FF) character (e.g. in COPYING file), or \033 or ^[ i.e. ESCAPE (\e) character (e.g. commit 20a3847d) is that they are not allowed in XML, which means that they are not allowed in XHTML, which means that if they are on the page, and MIME-type is 'application/xml+html' forcing strict XML/XHTML mode validating browsers would not display the page because it is not valid XHTML. Mozilla 1.17.2 did this, and it would not show page; I don't know how it works with more modern browsers.

-- 
Jakub Narebski
Poland