git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 0/9] icase match on non-ascii

From
Duy Nguyen <pclouds@gmail.com>
Date
Jul 8, 2015, 12:13 UTC
Message-ID
<CACsJy8DPVrVBrnmq2SBDkR+TNgaDBgnZGseHwR82ap2+6yR+rw@mail.gmail.com>
In-Reply-To
<559D0A4F.9040307@web.de>
On Wed, Jul 8, 2015 at 6:32 PM, Torsten Bögershausen <tboegi@web.de> wrote:
Show 12 quoted lines
> On 2015-07-08 12.38, Nguyễn Thái Ngọc Duy wrote:
>> Side note, I almost added the third has_non_ascii() function. Maybe we
>> should consider merging the two existing has_non_ascii() functions
>> back, or rename one to something else.
>>
> Side question:
>
> has_non_ascii can mean different things:
>  UTF-8, ISO-8859-1, ISO-8859-x...
>
> In short: everything.
> Should we be more critical here ?

For the purpose of this series, no. What we need to ask is "can we optimize this?" and that leads to "ok we can safely optimize for ascii-only patterns, don't optimize otherwise".

> May be this can be used from commit.c:
> static int verify_utf8(struct strbuf *buf)
Except the pcre/utf8 patch, the rest should work for all single-byte encodings..
> Other question:
> Should the "non-ascii" characters in the test scripts be octal-escaped ?

I would prefer something readable from a text editor. Even though I don't speak Icelandic (the strings were copied from gettext test script) I can see uppercase/lowercase of letters. But I don't know how many people have fonts covering more than just ascii..

Show 12 quoted lines
> Third question:
> What happens on systems, which don't have gettext, (for whatever reasons)
> --- a/gettext.c
> +++ b/gettext.c
> @@ -166,12 +166,17 @@ void git_setup_gettext(void)
> textdomain("git");
> }
>
> +int is_utf8_locale(void)
> +{
> + return !strcmp(charset, "UTF-8");
> +}

Hm.. pcre on utf-8 is screwed. I really don't want to go through nl_langinfo, $LC_ALL, $LC_CTYPE and $LANG to detect utf8 like what's done in compat/regex/regcomp.c.. But I guess there's no other way, people can disable gettext and expect git to work properly with utf-8 if their pcre library supports it.

> 4th question:
> What happens on systems which don't have locale support at all ?

I suppose by locale here you do not mean "gettext" any more, then icase does not work. We simply delegate the work to system regex/pcre. If they don't support locale, nothing we can do. Git itself does not know how to fold case outside ascii (except maybe utf8, I don't know how smart our utf-8 impl is). gettext support does not matter here, except pcre/utf8 case above.

> As one may suspect, I'm not a friend of being dependent on gettext and/or
> locale, at least not for this kind of business.
>
> Would it make more sense to have a command line option ?

I assume you only care about utf-8 here (and utf8.c knows how to fold case), you'll need to improve compat regex to take advantage of it first, maybe make sure it can live side by side with system regex, also make sure pcre is compiled with utf-8 support if you use pcre. And if setting LANG to let git know you want to use utf-8 is too subtle, then yes a command line option makes sense :)

-- 
Duy
Previous: Torsten BögershausenNext: Junio C Hamano
Message 27 of 53 in “Git grep does not support multi-byte characters (like UTF-8)”
  1. Plamen TotevJul 6, 2015
  2. Duy NguyenJul 6, 2015
  3. Plamen TotevJul 7, 2015
  4. Duy NguyenJul 7, 2015
  5. Junio C HamanoJul 7, 2015
  6. Plamen TotevJul 7, 2015
  7. Duy NguyenJul 8, 2015
  8. Junio C HamanoJul 8, 2015
  9. grep: use regcomp() for icase search with non-ascii patternsNguyễn Thái Ngọc Duy, Jul 6, 2015
  10. René ScharfeJul 6, 2015
  11. Duy NguyenJul 6, 2015
  12. Plamen TotevJul 7, 2015
  13. 0/9 icase match on non-asciiNguyễn Thái Ngọc Duy, Jul 8, 2015
  14. 1/9 grep: allow -F -i combinationNguyễn Thái Ngọc Duy, Jul 8, 2015
  15. 2/9 grep: break down an "if" stmt in preparation for next changesNguyễn Thái Ngọc Duy, Jul 8, 2015
  16. 3/9 grep/icase: avoid kwsset on literal non-ascii stringsNguyễn Thái Ngọc Duy, Jul 8, 2015
  17. 4/9 grep/icase: avoid kwsset when -F is specifiedNguyễn Thái Ngọc Duy, Jul 8, 2015
  18. 5/9 grep/pcre: prepare locale-dependent tables for icase matchingNguyễn Thái Ngọc Duy, Jul 8, 2015
  19. Duy NguyenJul 8, 2015
  20. 6/9 gettext: add is_utf8_locale()Nguyễn Thái Ngọc Duy, Jul 8, 2015
  21. 7/9 grep/pcre: support utf-8Nguyễn Thái Ngọc Duy, Jul 8, 2015
  22. Plamen TotevJul 11, 2015
  23. 8/9 diffcore-pickaxe: "share" regex error handling codeNguyễn Thái Ngọc Duy, Jul 8, 2015
  24. 9/9 diffcore-pickaxe: support case insensitive match on non-asciiNguyễn Thái Ngọc Duy, Jul 8, 2015
  25. Eric SunshineJul 9, 2015
  26. Torsten BögershausenJul 8, 2015
  27. Duy NguyenJul 8, 2015
  28. Junio C HamanoJul 8, 2015
  29. Duy NguyenJul 8, 2015
  30. 0/9 icase match on non-asciiNguyễn Thái Ngọc Duy, Jul 14, 2015
  31. 1/9 grep: allow -F -i combinationNguyễn Thái Ngọc Duy, Jul 14, 2015
  32. 2/9 grep: break down an "if" stmt in preparation for next changesNguyễn Thái Ngọc Duy, Jul 14, 2015
  33. 3/9 grep/icase: avoid kwsset on literal non-ascii stringsNguyễn Thái Ngọc Duy, Jul 14, 2015
  34. 4/9 grep/icase: avoid kwsset when -F is specifiedNguyễn Thái Ngọc Duy, Jul 14, 2015
  35. 5/9 grep/pcre: prepare locale-dependent tables for icase matchingNguyễn Thái Ngọc Duy, Jul 14, 2015
  36. 6/9 gettext: add is_utf8_locale()Nguyễn Thái Ngọc Duy, Jul 14, 2015
  37. 7/9 grep/pcre: support utf-8Nguyễn Thái Ngọc Duy, Jul 14, 2015
  38. 8/9 diffcore-pickaxe: "share" regex error handling codeNguyễn Thái Ngọc Duy, Jul 14, 2015
  39. 9/9 diffcore-pickaxe: support case insensitive match on non-asciiNguyễn Thái Ngọc Duy, Jul 14, 2015
  40. Torsten BögershausenJul 14, 2015
  41. Duy NguyenJul 15, 2015
  42. Torsten BögershausenJul 15, 2015
  43. 00/10 icase match on non-asciiNguyễn Thái Ngọc Duy, Aug 21, 2015
  44. 01/10 grep: allow -F -i combinationNguyễn Thái Ngọc Duy, Aug 21, 2015
  45. 02/10 grep: break down an "if" stmt in preparation for next changesNguyễn Thái Ngọc Duy, Aug 21, 2015
  46. 03/10 test-regex: expose full regcomp() to the command lineNguyễn Thái Ngọc Duy, Aug 21, 2015
  47. 04/10 grep/icase: avoid kwsset on literal non-ascii stringsNguyễn Thái Ngọc Duy, Aug 21, 2015
  48. 05/10 grep/icase: avoid kwsset when -F is specifiedNguyễn Thái Ngọc Duy, Aug 21, 2015
  49. 06/10 grep/pcre: prepare locale-dependent tables for icase matchingNguyễn Thái Ngọc Duy, Aug 21, 2015
  50. 07/10 gettext: add is_utf8_locale()Nguyễn Thái Ngọc Duy, Aug 21, 2015
  51. 08/10 grep/pcre: support utf-8Nguyễn Thái Ngọc Duy, Aug 21, 2015
  52. 09/10 diffcore-pickaxe: "share" regex error handling codeNguyễn Thái Ngọc Duy, Aug 21, 2015
  53. 10/10 diffcore-pickaxe: support case insensitive match on non-asciiNguyễn Thái Ngọc Duy, Aug 21, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.