git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 06/11] grep/pcre: prepare locale-dependent tables for icase matching

From
Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date
Jun 23, 2016, 16:29 UTC
Message-ID
<20160623162907.23295-7-pclouds@gmail.com>
In-Reply-To
<20160623162907.23295-1-pclouds@gmail.com>

The default tables are usually built with C locale and only suitable for LANG=C or similar. This should make case insensitive search work correctly for all single-byte charsets.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 grep.c                             |  8 ++++++--
 grep.h                             |  1 +
 t/t7813-grep-icase-iso.sh (new +x) | 19 +++++++++++++++++++
 3 files changed, 26 insertions(+), 2 deletions(-)
 create mode 100755 t/t7813-grep-icase-iso.sh
diff --git a/grep.c b/grep.c
index 6de58f3..22f4d99 100644
--- a/grep.c
+++ b/grep.c
@@ -324,11 +324,14 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)
 	int erroffset;
 	int options = PCRE_MULTILINE;
 
-	if (opt->ignore_case)
+	if (opt->ignore_case) {
+		if (has_non_ascii(p->pattern))
+			p->pcre_tables = pcre_maketables();
 		options |= PCRE_CASELESS;
+	}
 
 	p->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,
-			NULL);
+				      p->pcre_tables);
 	if (!p->pcre_regexp)
 		compile_regexp_failed(p, error);
 
@@ -362,6 +365,7 @@ static void free_pcre_regexp(struct grep_pat *p)
 {
 	pcre_free(p->pcre_regexp);
 	pcre_free(p->pcre_extra_info);
+	pcre_free((void *)p->pcre_tables);
 }
 #else /* !USE_LIBPCRE */
 static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)
diff --git a/grep.h b/grep.h
index 95f197a..cee4357 100644
--- a/grep.h
+++ b/grep.h
@@ -48,6 +48,7 @@ struct grep_pat {
 	regex_t regexp;
 	pcre *pcre_regexp;
 	pcre_extra *pcre_extra_info;
+	const unsigned char *pcre_tables;
 	kwset_t kws;
 	unsigned fixed:1;
 	unsigned ignore_case:1;
diff --git a/t/t7813-grep-icase-iso.sh b/t/t7813-grep-icase-iso.sh
new file mode 100755
index 0000000..efef7fb
--- /dev/null
+++ b/t/t7813-grep-icase-iso.sh
@@ -0,0 +1,19 @@
+#!/bin/sh
+
+test_description='grep icase on non-English locales'
+
+. ./lib-gettext.sh
+
+test_expect_success GETTEXT_ISO_LOCALE 'setup' '
+	printf "TILRAUN: Hall� Heimur!" >file &&
+	git add file &&
+	LC_ALL="$is_IS_iso_locale" &&
+	export LC_ALL
+'
+
+test_expect_success GETTEXT_ISO_LOCALE,LIBPCRE 'grep pcre string' '
+	git grep --perl-regexp -i "TILRAUN: H.ll� Heimur!" &&
+	git grep --perl-regexp -i "TILRAUN: H.LL� HEIMUR!"
+'
+
+test_done
-- 
2.8.2.526.g02eed6d
Previous: Junio C HamanoNext: Junio C Hamano
Message 15 of 39 in “nd/icase updates”
  1. 00/11 nd/icase updatesNguyễn Thái Ngọc Duy, Jun 23, 2016
  2. 02/11 test-regex: isolate the bug test codeNguyễn Thái Ngọc Duy, Jun 23, 2016
  3. 05/11 grep/icase: avoid kwsset when -F is specifiedNguyễn Thái Ngọc Duy, Jun 23, 2016
  4. Junio C HamanoJun 23, 2016
  5. 09/11 diffcore-pickaxe: "share" regex error handling codeNguyễn Thái Ngọc Duy, Jun 23, 2016
  6. Jeff KingJun 23, 2016
  7. 11/11 grep.c: reuse "icase" variableNguyễn Thái Ngọc Duy, Jun 23, 2016
  8. 03/11 test-regex: expose full regcomp() to the command lineNguyễn Thái Ngọc Duy, Jun 23, 2016
  9. 08/11 grep/pcre: support utf-8Nguyễn Thái Ngọc Duy, Jun 23, 2016
  10. 10/11 diffcore-pickaxe: support case insensitive match on non-asciiNguyễn Thái Ngọc Duy, Jun 23, 2016
  11. 07/11 gettext: add is_utf8_locale()Nguyễn Thái Ngọc Duy, Jun 23, 2016
  12. 01/11 grep: break down an "if" stmt in preparation for next changesNguyễn Thái Ngọc Duy, Jun 23, 2016
  13. 04/11 grep/icase: avoid kwsset on literal non-ascii stringsNguyễn Thái Ngọc Duy, Jun 23, 2016
  14. Junio C HamanoJun 23, 2016
  15. 06/11 grep/pcre: prepare locale-dependent tables for icase matchingNguyễn Thái Ngọc Duy, Jun 23, 2016
  16. Junio C HamanoJun 23, 2016
  17. 00/12 nd/icase updatesNguyễn Thái Ngọc Duy, Jun 25, 2016
  18. 03/12 test-regex: expose full regcomp() to the command lineNguyễn Thái Ngọc Duy, Jun 25, 2016
  19. 04/12 grep/icase: avoid kwsset on literal non-ascii stringsNguyễn Thái Ngọc Duy, Jun 25, 2016
  20. 07/12 grep/pcre: prepare locale-dependent tables for icase matchingNguyễn Thái Ngọc Duy, Jun 25, 2016
  21. 06/12 grep: rewrite an if/else condition to avoid duplicate expressionNguyễn Thái Ngọc Duy, Jun 25, 2016
  22. 10/12 diffcore-pickaxe: Add regcomp_or_die()Nguyễn Thái Ngọc Duy, Jun 25, 2016
  23. 09/12 grep/pcre: support utf-8Nguyễn Thái Ngọc Duy, Jun 25, 2016
  24. 05/12 grep/icase: avoid kwsset when -F is specifiedNguyễn Thái Ngọc Duy, Jun 25, 2016
  25. 12/12 grep.c: reuse "icase" variableNguyễn Thái Ngọc Duy, Jun 25, 2016
  26. 08/12 gettext: add is_utf8_locale()Nguyễn Thái Ngọc Duy, Jun 25, 2016
  27. 01/12 grep: break down an "if" stmt in preparation for next changesNguyễn Thái Ngọc Duy, Jun 25, 2016
  28. 11/12 diffcore-pickaxe: support case insensitive match on non-asciiNguyễn Thái Ngọc Duy, Jun 25, 2016
  29. 02/12 test-regex: isolate the bug test codeNguyễn Thái Ngọc Duy, Jun 25, 2016
  30. Junio C HamanoJun 27, 2016
  31. Junio C HamanoJun 27, 2016
  32. Duy NguyenJun 30, 2016
  33. Junio C HamanoJul 1, 2016
  34. Duy NguyenJul 1, 2016
  35. Junio C HamanoJul 1, 2016
  36. Junio C HamanoJul 1, 2016
  37. Junio C HamanoJul 1, 2016
  38. Junio C HamanoJul 1, 2016
  39. Junio C HamanoJul 1, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.