git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Git grep does not support multi-byte characters (like UTF-8)

From
Duy Nguyen <pclouds@gmail.com>
Date
Jul 7, 2015, 12:22 UTC
Message-ID
<CACsJy8A035JsR-z34g56QWLVkN0Tg0tOqmrT+Hp6ecsVBhm25g@mail.gmail.com>
In-Reply-To
<775251698.1328032.1436259534851.JavaMail.apache@nm31.abv.bg>
On Tue, Jul 7, 2015 at 3:58 PM, Plamen Totev <plamen.totev@abv.bg> wrote:
> Nguyen, thanks for the help and the patch. Also the escaping suggested by Scharfe seems as good choice. But i dig some more into the problem and I found some other thing. That's why I replied on the main thread not on the patch. I hope you'll excuse me if this is a bad practice.
So far this is very good reporting. I can't complain :)
> git grep -i -P also does not works because the PCRE_UTF8 is not set and pcre library does not treat the string as UTF-8.

We do prefer utf-8, but i don't know if we can assume utf-8 everywhere yet. I guess it's ok in this case.

> pickaxe search also uses kwsearch so the case insensitive search with it does not work (e.g. git log -i -S).  Maybe this is a less of a problem here as one is expected to search for exact string (hence knows the case)

Will fix (i'm close to being done with git-grep, not counting the pcre bug you just found)

> There is a interesting corner case. is_fixed treats all patterns containing nulls as fixed. So what about if the string contains non-ASCII symbols as well as nulls and the search is case insensitive :) I have to admin that my knowledge in UTF-8 is not enough to answer the question if this could occur during normal usage. For example the second byte in multi-byte symbol is NULL. I would guess that's not true as it would break a lot of programs that depend on NULL delimited string but it's good if somebody could confirm.

For utf-8, if NUL occurs in a byte stream, it must be ASCII NUL, not part of any multibyte character. Utf-8 is really well tuned for C strings.

> GNU grep indeed uses escaped regular expressions when the string is using multi-byte encoding and the search is case insensitive. If the encoding is UTF-8 then this strategy could be used in git too. Especially that git already have support and helper functions to work with UTF-8. As for the other multi-byte encodings - I think the things would become more complicated. As far I know in UTF-8 the '{' character for example is two bytes not one. Maybe really a support could be added only for the UTF-8 and if the string is not UTF-8 to issue a warning.

In the worst case we could reuse the trick we do elsewhere in git: convert to utf-8 with iconv, do whatever we need to (escaping...) then convert back before feeding it to regcomp and friends.

Show 6 quoted lines
> So maybe the following makes sense when a grep search is performed:
> * check if the multi-byte encoding is used. If it's and the search is case insensitive and the encoding is not UTF-8 give a warning;
> * if pcre is used and the string is UTF-8 encoded set the PCRE_UTF8 flag;
> * if the search is case insensitive, the string is fixed and the encoding  used is UTF-8 use regcomp instead of kwsearch and escape any regex special characters in the pattern;
>
> And the question with the behavior of pickaxe search remains open. Using kwset does not work with case insensitive non-ASCII searches. Instead of fixing grep.c maybe it's better if new function is introduced that performs keyword searches so it could be used by both grep, diffcore-pickaxe and any other code in the future that may require such functionality. Or maybe diffcore-pickaxe should use grep instead of directly kwset/regcomp

That would function be called "grep". More refactor would be needed. "git grep regcomp" reveals some many places. Many some of them would benefit from kws if we provide this new function you mentioned.

Show 18 quoted lines
> Regards,
> Plamen Totev
>
>
>
>>-------- Оригинално писмо --------
>>От: Duy Nguyen pclouds@gmail.com
>>Относно: Re: Git grep does not support multi-byte characters (like UTF-8)
>>До: Plamen Totev <plamen.totev@abv.bg>
>>Изпратено на: 06.07.2015 15:23
>
>> I think we over-optimized a bit. If you your system provides regex
>> with locale support (e.g. Linux) and you don't explicitly use fallback
>> regex implementation, it should work. I suppose your test patterns
>> look "fixed" (i.e. no regex special characters)? Can you try just add
>> "." and see if case insensitive matching works?
>
> This is indeed the problem. When I added the "." the matching works just fine.
-- 
Duy
Previous: Plamen TotevNext: Junio C Hamano
Message 4 of 53 in “Git grep does not support multi-byte characters (like UTF-8)”
  1. Plamen TotevJul 6, 2015
  2. Duy NguyenJul 6, 2015
  3. Plamen TotevJul 7, 2015
  4. Duy NguyenJul 7, 2015
  5. Junio C HamanoJul 7, 2015
  6. Plamen TotevJul 7, 2015
  7. Duy NguyenJul 8, 2015
  8. Junio C HamanoJul 8, 2015
  9. grep: use regcomp() for icase search with non-ascii patternsNguyễn Thái Ngọc Duy, Jul 6, 2015
  10. René ScharfeJul 6, 2015
  11. Duy NguyenJul 6, 2015
  12. Plamen TotevJul 7, 2015
  13. 0/9 icase match on non-asciiNguyễn Thái Ngọc Duy, Jul 8, 2015
  14. 1/9 grep: allow -F -i combinationNguyễn Thái Ngọc Duy, Jul 8, 2015
  15. 2/9 grep: break down an "if" stmt in preparation for next changesNguyễn Thái Ngọc Duy, Jul 8, 2015
  16. 3/9 grep/icase: avoid kwsset on literal non-ascii stringsNguyễn Thái Ngọc Duy, Jul 8, 2015
  17. 4/9 grep/icase: avoid kwsset when -F is specifiedNguyễn Thái Ngọc Duy, Jul 8, 2015
  18. 5/9 grep/pcre: prepare locale-dependent tables for icase matchingNguyễn Thái Ngọc Duy, Jul 8, 2015
  19. Duy NguyenJul 8, 2015
  20. 6/9 gettext: add is_utf8_locale()Nguyễn Thái Ngọc Duy, Jul 8, 2015
  21. 7/9 grep/pcre: support utf-8Nguyễn Thái Ngọc Duy, Jul 8, 2015
  22. Plamen TotevJul 11, 2015
  23. 8/9 diffcore-pickaxe: "share" regex error handling codeNguyễn Thái Ngọc Duy, Jul 8, 2015
  24. 9/9 diffcore-pickaxe: support case insensitive match on non-asciiNguyễn Thái Ngọc Duy, Jul 8, 2015
  25. Eric SunshineJul 9, 2015
  26. Torsten BögershausenJul 8, 2015
  27. Duy NguyenJul 8, 2015
  28. Junio C HamanoJul 8, 2015
  29. Duy NguyenJul 8, 2015
  30. 0/9 icase match on non-asciiNguyễn Thái Ngọc Duy, Jul 14, 2015
  31. 1/9 grep: allow -F -i combinationNguyễn Thái Ngọc Duy, Jul 14, 2015
  32. 2/9 grep: break down an "if" stmt in preparation for next changesNguyễn Thái Ngọc Duy, Jul 14, 2015
  33. 3/9 grep/icase: avoid kwsset on literal non-ascii stringsNguyễn Thái Ngọc Duy, Jul 14, 2015
  34. 4/9 grep/icase: avoid kwsset when -F is specifiedNguyễn Thái Ngọc Duy, Jul 14, 2015
  35. 5/9 grep/pcre: prepare locale-dependent tables for icase matchingNguyễn Thái Ngọc Duy, Jul 14, 2015
  36. 6/9 gettext: add is_utf8_locale()Nguyễn Thái Ngọc Duy, Jul 14, 2015
  37. 7/9 grep/pcre: support utf-8Nguyễn Thái Ngọc Duy, Jul 14, 2015
  38. 8/9 diffcore-pickaxe: "share" regex error handling codeNguyễn Thái Ngọc Duy, Jul 14, 2015
  39. 9/9 diffcore-pickaxe: support case insensitive match on non-asciiNguyễn Thái Ngọc Duy, Jul 14, 2015
  40. Torsten BögershausenJul 14, 2015
  41. Duy NguyenJul 15, 2015
  42. Torsten BögershausenJul 15, 2015
  43. 00/10 icase match on non-asciiNguyễn Thái Ngọc Duy, Aug 21, 2015
  44. 01/10 grep: allow -F -i combinationNguyễn Thái Ngọc Duy, Aug 21, 2015
  45. 02/10 grep: break down an "if" stmt in preparation for next changesNguyễn Thái Ngọc Duy, Aug 21, 2015
  46. 03/10 test-regex: expose full regcomp() to the command lineNguyễn Thái Ngọc Duy, Aug 21, 2015
  47. 04/10 grep/icase: avoid kwsset on literal non-ascii stringsNguyễn Thái Ngọc Duy, Aug 21, 2015
  48. 05/10 grep/icase: avoid kwsset when -F is specifiedNguyễn Thái Ngọc Duy, Aug 21, 2015
  49. 06/10 grep/pcre: prepare locale-dependent tables for icase matchingNguyễn Thái Ngọc Duy, Aug 21, 2015
  50. 07/10 gettext: add is_utf8_locale()Nguyễn Thái Ngọc Duy, Aug 21, 2015
  51. 08/10 grep/pcre: support utf-8Nguyễn Thái Ngọc Duy, Aug 21, 2015
  52. 09/10 diffcore-pickaxe: "share" regex error handling codeNguyễn Thái Ngọc Duy, Aug 21, 2015
  53. 10/10 diffcore-pickaxe: support case insensitive match on non-asciiNguyễn Thái Ngọc Duy, Aug 21, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.