{"thread":{"id":"42699","subject":"[PATCH 00/11] nd/icase updates","startedAt":"2016-06-23T16:29:34Z","lastAt":"2016-07-01T20:07:47Z","messageCount":39,"participants":["Nguyễn Thái Ngọc Duy","Jeff King","Junio C Hamano","Duy Nguyen"],"isPatch":true,"patchVersion":1,"patchTotal":11},"messages":[{"id":"289955","messageId":"20160623162907.23295-1-pclouds@gmail.com","threadId":"42699","inReplyTo":null,"subject":"[PATCH 00/11] nd/icase updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:28:56Z","receivedAt":"2016-06-23T16:29:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This update drops 1/12, which is an unnecessary change, and changes a\ncouple of echo/printf to test_write_lines. One of those echo uses\nbacklashes and causes problems with Debian dash.\n\nInterdiff therefore is not really interesting\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 46c5ba1..8c516a9 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -809,7 +809,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (!opt.pattern_list)\n \t\tdie(_(\"no pattern given.\"));\n-\tif (opt.ignore_case)\n+\tif (!opt.fixed && opt.ignore_case)\n \t\topt.regflags |= REG_ICASE;\n \n \tcompile_grep_patterns(&opt);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 4176625..169fd8d 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -5,7 +5,7 @@ test_description='grep icase on non-English locales'\n . ./lib-gettext.sh\n \n test_expect_success GETTEXT_LOCALE 'setup' '\n-\tprintf \"TILRAUN: Halló Heimur!\" >file &&\n+\ttest_write_lines \"TILRAUN: Halló Heimur!\" >file &&\n \tgit add file &&\n \tLC_ALL=\"$is_IS_locale\" &&\n \texport LC_ALL\n@@ -27,7 +27,7 @@ test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 icase' '\n '\n \n test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 string with \"+\"' '\n-\tprintf \"TILRAUN: Hallóó Heimur!\" >file2 &&\n+\ttest_write_lines \"TILRAUN: Hallóó Heimur!\" >file2 &&\n \tgit add file2 &&\n \tgit grep -l --perl-regexp \"TILRAUN: H.lló+ Heimur!\" >actual &&\n \techo file >expected &&\n@@ -38,26 +38,26 @@ test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 string with \"+\"' '\n test_expect_success REGEX_LOCALE 'grep literal string, with -F' '\n \tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n-\techo \"fixed TILRAUN: Halló Heimur!\" >expect1 &&\n+\ttest_write_lines \"fixed TILRAUN: Halló Heimur!\" >expect1 &&\n \ttest_cmp expect1 debug1 &&\n \n \tgit grep --debug -i -F \"TILRAUN: HALLÓ HEIMUR!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug2 &&\n-\techo \"fixed TILRAUN: HALLÓ HEIMUR!\" >expect2 &&\n+\ttest_write_lines \"fixed TILRAUN: HALLÓ HEIMUR!\" >expect2 &&\n \ttest_cmp expect2 debug2\n '\n \n test_expect_success REGEX_LOCALE 'grep string with regex, with -F' '\n-\tprintf \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n+\ttest_write_lines \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n \n \tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n-\techo \"fixed \\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n+\ttest_write_lines \"fixed \\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n \ttest_cmp expect1 debug1 &&\n \n \tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\HALLÓ \\$HE[]IMUR!\\$\" 2>&1 >/dev/null |\n \t\t grep fixed >debug2 &&\n-\techo \"fixed \\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n+\ttest_write_lines \"fixed \\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n \ttest_cmp expect2 debug2\n '\n \n\nNguyễn Thái Ngọc Duy (11):\n grep: break down an \"if\" stmt in preparation for next changes\n test-regex: isolate the bug test code\n test-regex: expose full regcomp() to the command line\n grep/icase: avoid kwsset on literal non-ascii strings\n grep/icase: avoid kwsset when -F is specified\n grep/pcre: prepare locale-dependent tables for icase matching\n gettext: add is_utf8_locale()\n grep/pcre: support utf-8\n diffcore-pickaxe: \"share\" regex error handling code\n diffcore-pickaxe: support case insensitive match on non-ascii\n grep.c: reuse \"icase\" variable\n\n diffcore-pickaxe.c | 27 ++++++++----\n gettext.c | 24 ++++++++++-\n gettext.h | 1 +\n grep.c | 47 +++++++++++++++++----\n grep.h | 1 +\n quote.c | 37 +++++++++++++++++\n quote.h | 1 +\n t/t0070-fundamental.sh | 2 +-\n t/t7812-grep-icase-non-ascii.sh (new +x) | 71 ++++++++++++++++++++++++++++++++\n t/t7813-grep-icase-iso.sh (new +x) | 19 +++++++++\n test-regex.c | 59 +++++++++++++++++++++++++-\n 11 files changed, 269 insertions(+), 20 deletions(-)\n create mode 100755 t/t7812-grep-icase-non-ascii.sh\n create mode 100755 t/t7813-grep-icase-iso.sh\n\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289956","messageId":"20160623162907.23295-3-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 02/11] test-regex: isolate the bug test code","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:28:58Z","receivedAt":"2016-06-23T16:29:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is in preparation to turn test-regex into some generic regex\ntesting command.\n\nHelped-by: Eric Sunshine \nHelped-by: Ramsay Jones \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n t/t0070-fundamental.sh | 2 +-\n test-regex.c | 12 ++++++++++--\n 2 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t0070-fundamental.sh b/t/t0070-fundamental.sh\nindex 5ed69a6..991ed2a 100755\n--- a/t/t0070-fundamental.sh\n+++ b/t/t0070-fundamental.sh\n@@ -31,7 +31,7 @@ test_expect_success 'git_mkstemps_mode does not fail if fd 0 is not open' '\n \n test_expect_success 'check for a bug in the regex routines' '\n \t# if this test fails, re-build git with NO_REGEX=1\n-\ttest-regex\n+\ttest-regex --bug\n '\n \n test_done\ndiff --git a/test-regex.c b/test-regex.c\nindex 0dc598e..67a1a65 100644\n--- a/test-regex.c\n+++ b/test-regex.c\n@@ -1,6 +1,6 @@\n #include \"git-compat-util.h\"\n \n-int main(int argc, char **argv)\n+static int test_regex_bug(void)\n {\n \tchar *pat = \"[^={} \\t]+\";\n \tchar *str = \"={}\\nfred\";\n@@ -16,5 +16,13 @@ int main(int argc, char **argv)\n \tif (m[0].rm_so == 3) /* matches '\\n' when it should not */\n \t\tdie(\"regex bug confirmed: re-build git with NO_REGEX=1\");\n \n-\texit(0);\n+\treturn 0;\n+}\n+\n+int main(int argc, char **argv)\n+{\n+\tif (argc == 2 && !strcmp(argv[1], \"--bug\"))\n+\t\treturn test_regex_bug();\n+\telse\n+\t\tusage(\"test-regex --bug\");\n }\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289957","messageId":"20160623162907.23295-6-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 05/11] grep/icase: avoid kwsset when -F is specified","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:29:01Z","receivedAt":"2016-06-23T16:29:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the previous commit, we can't use kws on icase search\noutside ascii range. But we can't simply pass the pattern to\nregcomp/pcre like the previous commit because it may contain regex\nspecial characters, so we need to quote the regex first.\n\nTo avoid misquote traps that could lead to undefined behavior, we\nalways stick to basic regex engine in this case. We don't need fancy\nfeatures for grepping a literal string anyway.\n\nbasic_regex_quote_buf() assumes that if the pattern is in a multibyte\nencoding, ascii chars must be unambiguously encoded as single\nbytes. This is true at least for UTF-8. For others, let's wait until\npeople yell up. Chances are nobody uses multibyte, non utf-8 charsets\nanymore.\n\nNoticed-by: Plamen Totev \nHelped-by: René Scharfe \nHelped-by: Eric Sunshine \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 25 ++++++++++++++++++++++++-\n quote.c | 37 +++++++++++++++++++++++++++++++++++++\n quote.h | 1 +\n t/t7812-grep-icase-non-ascii.sh | 26 ++++++++++++++++++++++++++\n 4 files changed, 88 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex f192727..6de58f3 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -5,6 +5,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"commit.h\"\n+#include \"quote.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -397,6 +398,24 @@ static int is_fixed(const char *s, size_t len)\n \treturn 1;\n }\n \n+static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint err;\n+\n+\tbasic_regex_quote_buf(&sb, p->pattern);\n+\terr = regcomp(&p->regexp, sb.buf, opt->regflags & ~REG_EXTENDED);\n+\tif (opt->debug)\n+\t\tfprintf(stderr, \"fixed %s\\n\", sb.buf);\n+\tstrbuf_release(&sb);\n+\tif (err) {\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, &p->regexp, errbuf, sizeof(errbuf));\n+\t\tregfree(&p->regexp);\n+\t\tcompile_regexp_failed(p, errbuf);\n+\t}\n+}\n+\n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tint icase, ascii_only;\n@@ -408,7 +427,11 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tascii_only = !has_non_ascii(p->pattern);\n \n \tif (opt->fixed) {\n-\t\tp->fixed = 1;\n+\t\tp->fixed = !icase || ascii_only;\n+\t\tif (!p->fixed) {\n+\t\t\tcompile_fixed_regexp(p, opt);\n+\t\t\treturn;\n+\t\t}\n \t} else if ((!icase || ascii_only) &&\n \t\t is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\ndiff --git a/quote.c b/quote.c\nindex fe884d2..c67adb7 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -440,3 +440,40 @@ void tcl_quote_buf(struct strbuf *sb, const char *src)\n \t}\n \tstrbuf_addch(sb, '\"');\n }\n+\n+void basic_regex_quote_buf(struct strbuf *sb, const char *src)\n+{\n+\tchar c;\n+\n+\tif (*src == '^') {\n+\t\t/* only beginning '^' is special and needs quoting */\n+\t\tstrbuf_addch(sb, '\\\\');\n+\t\tstrbuf_addch(sb, *src++);\n+\t}\n+\tif (*src == '*')\n+\t\t/* beginning '*' is not special, no quoting */\n+\t\tstrbuf_addch(sb, *src++);\n+\n+\twhile ((c = *src++)) {\n+\t\tswitch (c) {\n+\t\tcase '[':\n+\t\tcase '.':\n+\t\tcase '\\\\':\n+\t\tcase '*':\n+\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tcase '$':\n+\t\t\t/* only the end '$' is special and needs quoting */\n+\t\t\tif (*src == '\\0')\n+\t\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\ndiff --git a/quote.h b/quote.h\nindex 99e04d3..362d315 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -67,5 +67,6 @@ extern char *quote_path_relative(const char *in, const char *prefix,\n extern void perl_quote_buf(struct strbuf *sb, const char *src);\n extern void python_quote_buf(struct strbuf *sb, const char *src);\n extern void tcl_quote_buf(struct strbuf *sb, const char *src);\n+extern void basic_regex_quote_buf(struct strbuf *sb, const char *src);\n \n #endif\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex b78a774..1929809 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -20,4 +20,30 @@ test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success REGEX_LOCALE 'grep literal string, with -F' '\n+\tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\ttest_write_lines \"fixed TILRAUN: Halló Heimur!\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"TILRAUN: HALLÓ HEIMUR!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\ttest_write_lines \"fixed TILRAUN: HALLÓ HEIMUR!\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n+test_expect_success REGEX_LOCALE 'grep string with regex, with -F' '\n+\ttest_write_lines \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\ttest_write_lines \"fixed \\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\HALLÓ \\$HE[]IMUR!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\ttest_write_lines \"fixed \\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n test_done\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289958","messageId":"20160623162907.23295-10-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 09/11] diffcore-pickaxe: \"share\" regex error handling code","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:29:05Z","receivedAt":"2016-06-23T16:29:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"There's another regcomp code block coming in this function. By moving\nthe error handling code out of this block, we don't have to add the\nsame error handling code in the new block.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n diffcore-pickaxe.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 7715c13..69c6567 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -204,20 +204,13 @@ void diffcore_pickaxe(struct diff_options *o)\n \tint opts = o->pickaxe_opts;\n \tregex_t regex, *regexp = NULL;\n \tkwset_t kws = NULL;\n+\tint err = 0;\n \n \tif (opts & (DIFF_PICKAXE_REGEX | DIFF_PICKAXE_KIND_G)) {\n-\t\tint err;\n \t\tint cflags = REG_EXTENDED | REG_NEWLINE;\n \t\tif (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE))\n \t\t\tcflags |= REG_ICASE;\n \t\terr = regcomp(®ex, needle, cflags);\n-\t\tif (err) {\n-\t\t\t/* The POSIX.2 people are surely sick */\n-\t\t\tchar errbuf[1024];\n-\t\t\tregerror(err, ®ex, errbuf, 1024);\n-\t\t\tregfree(®ex);\n-\t\t\tdie(\"invalid regex: %s\", errbuf);\n-\t\t}\n \t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n@@ -225,6 +218,13 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\tkwsincr(kws, needle, strlen(needle));\n \t\tkwsprep(kws);\n \t}\n+\tif (err) {\n+\t\t/* The POSIX.2 people are surely sick */\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, ®ex, errbuf, 1024);\n+\t\tregfree(®ex);\n+\t\tdie(\"invalid regex: %s\", errbuf);\n+\t}\n \n \t/* Might want to warn when both S and G are on; I don't care... */\n \tpickaxe(&diff_queued_diff, o, regexp, kws,\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289959","messageId":"20160623162907.23295-12-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 11/11] grep.c: reuse \"icase\" variable","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:29:07Z","receivedAt":"2016-06-23T16:29:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 5 +----\n 1 file changed, 1 insertion(+), 4 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 6e99b01..cb058a5 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -445,10 +445,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tp->fixed = 0;\n \n \tif (p->fixed) {\n-\t\tif (opt->regflags & REG_ICASE || p->ignore_case)\n-\t\t\tp->kws = kwsalloc(tolower_trans_tbl);\n-\t\telse\n-\t\t\tp->kws = kwsalloc(NULL);\n+\t\tp->kws = kwsalloc(icase ? tolower_trans_tbl : NULL);\n \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n \t\tkwsprep(p->kws);\n \t\treturn;\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289960","messageId":"20160623162907.23295-4-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 03/11] test-regex: expose full regcomp() to the command line","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:28:59Z","receivedAt":"2016-06-23T16:29:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n test-regex.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 49 insertions(+), 2 deletions(-)\n\ndiff --git a/test-regex.c b/test-regex.c\nindex 67a1a65..eff26f5 100644\n--- a/test-regex.c\n+++ b/test-regex.c\n@@ -1,4 +1,21 @@\n #include \"git-compat-util.h\"\n+#include \"gettext.h\"\n+\n+struct reg_flag {\n+\tconst char *name;\n+\tint flag;\n+};\n+\n+static struct reg_flag reg_flags[] = {\n+\t{ \"EXTENDED\",\t REG_EXTENDED\t},\n+\t{ \"NEWLINE\",\t REG_NEWLINE\t},\n+\t{ \"ICASE\",\t REG_ICASE\t},\n+\t{ \"NOTBOL\",\t REG_NOTBOL\t},\n+#ifdef REG_STARTEND\n+\t{ \"STARTEND\",\t REG_STARTEND\t},\n+#endif\n+\t{ NULL, 0 }\n+};\n \n static int test_regex_bug(void)\n {\n@@ -21,8 +38,38 @@ static int test_regex_bug(void)\n \n int main(int argc, char **argv)\n {\n+\tconst char *pat;\n+\tconst char *str;\n+\tint flags = 0;\n+\tregex_t r;\n+\tregmatch_t m[1];\n+\n \tif (argc == 2 && !strcmp(argv[1], \"--bug\"))\n \t\treturn test_regex_bug();\n-\telse\n-\t\tusage(\"test-regex --bug\");\n+\telse if (argc < 3)\n+\t\tusage(\"test-regex --bug\\n\"\n+\t\t \"test-regex []\");\n+\n+\targv++;\n+\tpat = *argv++;\n+\tstr = *argv++;\n+\twhile (*argv) {\n+\t\tstruct reg_flag *rf;\n+\t\tfor (rf = reg_flags; rf->name; rf++)\n+\t\t\tif (!strcmp(*argv, rf->name)) {\n+\t\t\t\tflags |= rf->flag;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\tif (!rf->name)\n+\t\t\tdie(\"do not recognize %s\", *argv);\n+\t\targv++;\n+\t}\n+\tgit_setup_gettext();\n+\n+\tif (regcomp(&r, pat, flags))\n+\t\tdie(\"failed regcomp() for pattern '%s'\", pat);\n+\tif (regexec(&r, str, 1, m, 0))\n+\t\treturn 1;\n+\n+\treturn 0;\n }\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289961","messageId":"20160623162907.23295-9-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 08/11] grep/pcre: support utf-8","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:29:04Z","receivedAt":"2016-06-23T16:29:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In the previous change in this function, we add locale support for\nsingle-byte encodings only. It looks like pcre only supports utf-* as\nmultibyte encodings, the others are left in the cold (which is\nfine).\n\nWe need to enable PCRE_UTF8 so pcre can find character boundary\ncorrectly. It's needed for case folding (when --ignore-case is used)\nor '*', '+' or similar syntax is used.\n\nThe \"has_non_ascii()\" check is to be on the conservative side. If\nthere's non-ascii in the pattern, the searched content could still be\nin utf-8, but we can treat it just like a byte stream and everything\nshould work. If we force utf-8 based on locale only and pcre validates\nutf-8 and the file content is in non-utf8 encoding, things break.\n\nNoticed-by: Plamen Totev \nHelped-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 2 ++\n t/t7812-grep-icase-non-ascii.sh | 15 +++++++++++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/grep.c b/grep.c\nindex 22f4d99..6e99b01 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -329,6 +329,8 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \t\t\tp->pcre_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n \t}\n+\tif (is_utf8_locale() && has_non_ascii(p->pattern))\n+\t\toptions |= PCRE_UTF8;\n \n \tp->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n \t\t\t\t p->pcre_tables);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 1929809..08ae4c9 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -20,6 +20,21 @@ test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 icase' '\n+\tgit grep --perl-regexp \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LLÓ HEIMUR!\"\n+'\n+\n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 string with \"+\"' '\n+\ttest_write_lines \"TILRAUN: Hallóó Heimur!\" >file2 &&\n+\tgit add file2 &&\n+\tgit grep -l --perl-regexp \"TILRAUN: H.lló+ Heimur!\" >actual &&\n+\techo file >expected &&\n+\techo file2 >>expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success REGEX_LOCALE 'grep literal string, with -F' '\n \tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289962","messageId":"20160623162907.23295-11-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 10/11] diffcore-pickaxe: support case insensitive match on non-ascii","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:29:06Z","receivedAt":"2016-06-23T16:29:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the \"grep -F -i\" case, we can't use kws on icase search\noutside ascii range, so we quote the string and pass it to regcomp as\na basic regexp and let regex engine deal with case sensitivity.\n\nThe new test is put in t7812 instead of t4209-log-pickaxe because\nlib-gettext.sh might cause problems elsewhere, probably.\n\nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n diffcore-pickaxe.c | 11 +++++++++++\n t/t7812-grep-icase-non-ascii.sh | 7 +++++++\n 2 files changed, 18 insertions(+)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 69c6567..0a5f877 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -7,6 +7,8 @@\n #include \"diffcore.h\"\n #include \"xdiff-interface.h\"\n #include \"kwset.h\"\n+#include \"commit.h\"\n+#include \"quote.h\"\n \n typedef int (*pickaxe_fn)(mmfile_t *one, mmfile_t *two,\n \t\t\t struct diff_options *o,\n@@ -212,6 +214,15 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\t\tcflags |= REG_ICASE;\n \t\terr = regcomp(®ex, needle, cflags);\n \t\tregexp = ®ex;\n+\t} else if (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE) &&\n+\t\t has_non_ascii(needle)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tint cflags = REG_NEWLINE | REG_ICASE;\n+\n+\t\tbasic_regex_quote_buf(&sb, needle);\n+\t\terr = regcomp(®ex, sb.buf, cflags);\n+\t\tstrbuf_release(&sb);\n+\t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n \t\t\t ? tolower_trans_tbl : NULL);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 08ae4c9..169fd8d 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -61,4 +61,11 @@ test_expect_success REGEX_LOCALE 'grep string with regex, with -F' '\n \ttest_cmp expect2 debug2\n '\n \n+test_expect_success REGEX_LOCALE 'pickaxe -i on non-ascii' '\n+\tgit commit -m first &&\n+\tgit log --format=%f -i -S\"TILRAUN: HALLÓ HEIMUR!\" >actual &&\n+\techo first >expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289963","messageId":"20160623162907.23295-8-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 07/11] gettext: add is_utf8_locale()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:29:03Z","receivedAt":"2016-06-23T16:29:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This function returns true if git is running under an UTF-8\nlocale. pcre in the next patch will need this.\n\nis_encoding_utf8() is used instead of strcmp() to catch both \"utf-8\"\nand \"utf8\" suffixes.\n\nWhen built with no gettext support, we peek in several env variables\nto detect UTF-8. pcre library might support utf-8 even if libc is\nbuilt without locale support.. The peeking code is a copy from\ncompat/regex/regcomp.c\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n gettext.c | 24 ++++++++++++++++++++++--\n gettext.h | 1 +\n 2 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/gettext.c b/gettext.c\nindex a268a2c..db727ea 100644\n--- a/gettext.c\n+++ b/gettext.c\n@@ -18,6 +18,8 @@\n #\tendif\n #endif\n \n+static const char *charset;\n+\n /*\n * Guess the user's preferred languages from the value in LANGUAGE environment\n * variable and LC_MESSAGES locale category if NO_GETTEXT is not defined.\n@@ -65,7 +67,6 @@ static int test_vsnprintf(const char *fmt, ...)\n \treturn ret;\n }\n \n-static const char *charset;\n static void init_gettext_charset(const char *domain)\n {\n \t/*\n@@ -172,8 +173,27 @@ int gettext_width(const char *s)\n {\n \tstatic int is_utf8 = -1;\n \tif (is_utf8 == -1)\n-\t\tis_utf8 = !strcmp(charset, \"UTF-8\");\n+\t\tis_utf8 = is_utf8_locale();\n \n \treturn is_utf8 ? utf8_strwidth(s) : strlen(s);\n }\n #endif\n+\n+int is_utf8_locale(void)\n+{\n+#ifdef NO_GETTEXT\n+\tif (!charset) {\n+\t\tconst char *env = getenv(\"LC_ALL\");\n+\t\tif (!env || !*env)\n+\t\t\tenv = getenv(\"LC_CTYPE\");\n+\t\tif (!env || !*env)\n+\t\t\tenv = getenv(\"LANG\");\n+\t\tif (!env)\n+\t\t\tenv = \"\";\n+\t\tif (strchr(env, '.'))\n+\t\t\tenv = strchr(env, '.') + 1;\n+\t\tcharset = xstrdup(env);\n+\t}\n+#endif\n+\treturn is_encoding_utf8(charset);\n+}\ndiff --git a/gettext.h b/gettext.h\nindex 33696a4..7eee64a 100644\n--- a/gettext.h\n+++ b/gettext.h\n@@ -90,5 +90,6 @@ const char *Q_(const char *msgid, const char *plu, unsigned long n)\n #endif\n \n const char *get_preferred_languages(void);\n+extern int is_utf8_locale(void);\n \n #endif\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289964","messageId":"20160623162907.23295-2-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 01/11] grep: break down an \"if\" stmt in preparation for next changes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:28:57Z","receivedAt":"2016-06-23T16:30:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 7b2b96a..609f218 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -403,7 +403,9 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \n-\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n+\tif (opt->fixed) {\n+\t\tp->fixed = 1;\n+\t} else if (is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse\n \t\tp->fixed = 0;\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289965","messageId":"20160623162907.23295-5-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 04/11] grep/icase: avoid kwsset on literal non-ascii strings","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:29:00Z","receivedAt":"2016-06-23T16:30:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When we detect the pattern is just a literal string, we avoid heavy\nregex engine and use fast substring search implemented in kwsset.c.\nBut kws uses git-ctype which is locale-independent so it does not know\nhow to fold case properly outside ascii range. Let regcomp or pcre\ntake care of this case instead. Slower, but accurate.\n\nNoticed-by: Plamen Totev \nHelped-by: René Scharfe \nHelped-by: Eric Sunshine \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 7 ++++++-\n t/t7812-grep-icase-non-ascii.sh (new +x) | 23 +++++++++++++++++++++++\n 2 files changed, 29 insertions(+), 1 deletion(-)\n create mode 100755 t/t7812-grep-icase-non-ascii.sh\n\ndiff --git a/grep.c b/grep.c\nindex 609f218..f192727 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -4,6 +4,7 @@\n #include \"xdiff-interface.h\"\n #include \"diff.h\"\n #include \"diffcore.h\"\n+#include \"commit.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -398,14 +399,18 @@ static int is_fixed(const char *s, size_t len)\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n+\tint icase, ascii_only;\n \tint err;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n+\ticase\t = opt->regflags & REG_ICASE || p->ignore_case;\n+\tascii_only = !has_non_ascii(p->pattern);\n \n \tif (opt->fixed) {\n \t\tp->fixed = 1;\n-\t} else if (is_fixed(p->pattern, p->patternlen))\n+\t} else if ((!icase || ascii_only) &&\n+\t\t is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse\n \t\tp->fixed = 0;\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nnew file mode 100755\nindex 0000000..b78a774\n--- /dev/null\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_LOCALE 'setup' '\n+\ttest_write_lines \"TILRAUN: Halló Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_have_prereq GETTEXT_LOCALE &&\n+test-regex \"HALLÓ\" \"Halló\" ICASE &&\n+test_set_prereq REGEX_LOCALE\n+\n+test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n+\tgit grep -i \"TILRAUN: Halló Heimur!\" &&\n+\tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289966","messageId":"20160623162907.23295-7-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH 06/11] grep/pcre: prepare locale-dependent tables for icase matching","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-23T16:29:02Z","receivedAt":"2016-06-23T16:30:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The default tables are usually built with C locale and only suitable\nfor LANG=C or similar. This should make case insensitive search work\ncorrectly for all single-byte charsets.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 8 ++++++--\n grep.h | 1 +\n t/t7813-grep-icase-iso.sh (new +x) | 19 +++++++++++++++++++\n 3 files changed, 26 insertions(+), 2 deletions(-)\n create mode 100755 t/t7813-grep-icase-iso.sh\n\ndiff --git a/grep.c b/grep.c\nindex 6de58f3..22f4d99 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -324,11 +324,14 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \tint erroffset;\n \tint options = PCRE_MULTILINE;\n \n-\tif (opt->ignore_case)\n+\tif (opt->ignore_case) {\n+\t\tif (has_non_ascii(p->pattern))\n+\t\t\tp->pcre_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n+\t}\n \n \tp->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n-\t\t\tNULL);\n+\t\t\t\t p->pcre_tables);\n \tif (!p->pcre_regexp)\n \t\tcompile_regexp_failed(p, error);\n \n@@ -362,6 +365,7 @@ static void free_pcre_regexp(struct grep_pat *p)\n {\n \tpcre_free(p->pcre_regexp);\n \tpcre_free(p->pcre_extra_info);\n+\tpcre_free((void *)p->pcre_tables);\n }\n #else /* !USE_LIBPCRE */\n static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\ndiff --git a/grep.h b/grep.h\nindex 95f197a..cee4357 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -48,6 +48,7 @@ struct grep_pat {\n \tregex_t regexp;\n \tpcre *pcre_regexp;\n \tpcre_extra *pcre_extra_info;\n+\tconst unsigned char *pcre_tables;\n \tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\ndiff --git a/t/t7813-grep-icase-iso.sh b/t/t7813-grep-icase-iso.sh\nnew file mode 100755\nindex 0000000..efef7fb\n--- /dev/null\n+++ b/t/t7813-grep-icase-iso.sh\n@@ -0,0 +1,19 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_ISO_LOCALE 'setup' '\n+\tprintf \"TILRAUN: Hall� Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_iso_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_expect_success GETTEXT_ISO_LOCALE,LIBPCRE 'grep pcre string' '\n+\tgit grep --perl-regexp -i \"TILRAUN: H.ll� Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LL� HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"289991","messageId":"20160623191614.GA1841@sigill.intra.peff.net","threadId":"42699","inReplyTo":"20160623162907.23295-10-pclouds@gmail.com","subject":"Re: [PATCH 09/11] diffcore-pickaxe: \"share\" regex error handling code","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-06-23T19:16:14Z","receivedAt":"2016-06-23T19:16:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 23, 2016 at 06:29:05PM +0200, Nguyễn Thái Ngọc Duy wrote:\n\n> diff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\n> index 7715c13..69c6567 100644\n> --- a/diffcore-pickaxe.c\n> +++ b/diffcore-pickaxe.c\n> @@ -204,20 +204,13 @@ void diffcore_pickaxe(struct diff_options *o)\n> \tint opts = o->pickaxe_opts;\n> \tregex_t regex, *regexp = NULL;\n> \tkwset_t kws = NULL;\n> +\tint err = 0;\n> \n> \tif (opts & (DIFF_PICKAXE_REGEX | DIFF_PICKAXE_KIND_G)) {\n> -\t\tint err;\n> \t\tint cflags = REG_EXTENDED | REG_NEWLINE;\n> \t\tif (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE))\n> \t\t\tcflags |= REG_ICASE;\n> \t\terr = regcomp(®ex, needle, cflags);\n> -\t\tif (err) {\n> -\t\t\t/* The POSIX.2 people are surely sick */\n> -\t\t\tchar errbuf[1024];\n> -\t\t\tregerror(err, ®ex, errbuf, 1024);\n> -\t\t\tregfree(®ex);\n> -\t\t\tdie(\"invalid regex: %s\", errbuf);\n> -\t\t}\n> \t\tregexp = ®ex;\n> \t} else {\n> \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n> @@ -225,6 +218,13 @@ void diffcore_pickaxe(struct diff_options *o)\n> \t\tkwsincr(kws, needle, strlen(needle));\n> \t\tkwsprep(kws);\n> \t}\n> +\tif (err) {\n> +\t\t/* The POSIX.2 people are surely sick */\n> +\t\tchar errbuf[1024];\n> +\t\tregerror(err, ®ex, errbuf, 1024);\n> +\t\tregfree(®ex);\n> +\t\tdie(\"invalid regex: %s\", errbuf);\n> +\t}\n\nHrm. I wondered what happens if we see an error in the kwset code block,\nwhich did not put anything useful in \"regex\" at all.\n\nIt's OK right now, because \"err\" is newly promoted to the top of the\nfunction, and so we know that kwset cannot call it. But it seems like\nan accident waiting to happen. Calling it \"regex_err\" or something might\nhelp.\n\nBut I also wonder if a function wouldn't be better. You could even roll\nit up with regcomp, like:\n\n static void regcomp_or_die(regex_t *regex, const char *pattern, int flags)\n {\n\tint err = regcomp(regex, pattern, flags);\n\tif (err) {\n\t\tchar buf[1024];\n\t\tregerror(err, ®ex, buf, sizeof(buf));\n\t\tregfree(®ex);\n\t\tdie(\"invalid regex: %s\", buf);\n\t}\n }\n\nI think you could also skip the regfree(), since we are about to die. I\nalso think the error message would probably be better if it mentioned\nthe text of \"pattern\" itself (since it might be coming from config, or\nyou might have provided several patterns, or you might have thought\nsomething was supposed to be a non-regex).\n\n-Peff\n"},{"id":"289995","messageId":"xmqq37o3is11.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"20160623162907.23295-5-pclouds@gmail.com","subject":"Re: [PATCH 04/11] grep/icase: avoid kwsset on literal non-ascii strings","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-23T20:12:10Z","receivedAt":"2016-06-23T20:12:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n> {\n> +\tint icase, ascii_only;\n> \tint err;\n> \n> \tp->word_regexp = opt->word_regexp;\n> \tp->ignore_case = opt->ignore_case;\n> +\ticase\t = opt->regflags & REG_ICASE || p->ignore_case;\n> +\tascii_only = !has_non_ascii(p->pattern);\n> \n> \tif (opt->fixed) {\n> \t\tp->fixed = 1;\n> -\t} else if (is_fixed(p->pattern, p->patternlen))\n> +\t} else if ((!icase || ascii_only) &&\n> +\t\t is_fixed(p->pattern, p->patternlen))\n> \t\tp->fixed = 1;\n\n... we are not told to do \"fixed\" explicitly with \"-F\", and that is\nsafe for a literal pattern if the pattern is only ascii (with or\nwithout -i) but not safe with \"-i\" when dealing with non-ascii\npattern.\n\nMakes perfect sense.\n\n> \telse\n> \t\tp->fixed = 0;\n> diff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\n> new file mode 100755\n> index 0000000..b78a774\n> --- /dev/null\n> +++ b/t/t7812-grep-icase-non-ascii.sh\n> @@ -0,0 +1,23 @@\n> +#!/bin/sh\n> +\n> +test_description='grep icase on non-English locales'\n> +\n> +. ./lib-gettext.sh\n> +\n> +test_expect_success GETTEXT_LOCALE 'setup' '\n> +\ttest_write_lines \"TILRAUN: Halló Heimur!\" >file &&\n> +\tgit add file &&\n> +\tLC_ALL=\"$is_IS_locale\" &&\n> +\texport LC_ALL\n> +'\n> +\n> +test_have_prereq GETTEXT_LOCALE &&\n> +test-regex \"HALLÓ\" \"Halló\" ICASE &&\n> +test_set_prereq REGEX_LOCALE\n> +\n> +test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n> +\tgit grep -i \"TILRAUN: Halló Heimur!\" &&\n> +\tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n> +'\n> +\n> +test_done\n"},{"id":"289997","messageId":"xmqqtwgjhcub.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"20160623162907.23295-6-pclouds@gmail.com","subject":"Re: [PATCH 05/11] grep/icase: avoid kwsset when -F is specified","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-23T20:25:32Z","receivedAt":"2016-06-23T20:25:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> @@ -408,7 +427,11 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n> \tascii_only = !has_non_ascii(p->pattern);\n> \n> \tif (opt->fixed) {\n> -\t\tp->fixed = 1;\n> +\t\tp->fixed = !icase || ascii_only;\n> +\t\tif (!p->fixed) {\n> +\t\t\tcompile_fixed_regexp(p, opt);\n> +\t\t\treturn;\n> +\t\t}\n> \t} else if ((!icase || ascii_only) &&\n> \t\t is_fixed(p->pattern, p->patternlen))\n> \t\tp->fixed = 1;\n\nMakes me feel somewhat dirty to see that we do the same \"!icase ||\nascii_only\" on both sides of the if/else cascade.\n\nI wonder if it would be more readable to structure it like this\ninstead?\n\n\tif (opt->fixed || is_fixed(...))\n\t\tp->fixed = (!icase || ascii_only);\n\telse\n \tp->fixed = 0;\n\n\tif (p->fixed)\n \t/* do the kws thing and return */\n\telse if (opt->fixed)\n \t/* do the \"quote into regexp\" thing and return */\n\n\t/* pcre and other from the original follows */\n\n"},{"id":"289998","messageId":"xmqqpor7hchx.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"Re: [PATCH 00/11] nd/icase updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-23T20:32:58Z","receivedAt":"2016-06-23T20:33:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> This update drops 1/12, which is an unnecessary change, and changes a\n> couple of echo/printf to test_write_lines. One of those echo uses\n> backlashes and causes problems with Debian dash.\n>\n> Interdiff therefore is not really interesting\n\nNeither Debian dash (that understands XSI echo) nor other shells\n(that do not do XSI echo) was at fault, though. It was \"one of\nthoese echo was written in an unportable way with backslashes\" ;-)\n\nIn any case, I re-read the whole thing, and except for a small nit I\nsent separately, all looked sensible.\n\nThanks, will replace what has been queued on 'pu'.\n"},{"id":"290095","messageId":"20160625052238.13615-1-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160623162907.23295-1-pclouds@gmail.com","subject":"[PATCH v2 00/12] nd/icase updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:26Z","receivedAt":"2016-06-25T05:24:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"v2 fixes Junio's and Jeff's comments (both good). The sharing \"!icase\n|| ascii_only\" is made a separate commit (6/12) because I think it\ntakes some seconds to realize that the conversion is correct and it's\ntechnically not needed in 5/12 (and it's sort of the opposite of 1/12)\n\nInterdiff\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 0a5f877..55067ca 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -200,19 +200,30 @@ static void pickaxe(struct diff_queue_struct *q, struct diff_options *o,\n \t*q = outq;\n }\n \n+static void regcomp_or_die(regex_t *regex, const char *needle, int cflags)\n+{\n+\tint err = regcomp(regex, needle, cflags);\n+\tif (err) {\n+\t\t/* The POSIX.2 people are surely sick */\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, regex, errbuf, 1024);\n+\t\tregfree(regex);\n+\t\tdie(\"invalid regex: %s\", errbuf);\n+\t}\n+}\n+\n void diffcore_pickaxe(struct diff_options *o)\n {\n \tconst char *needle = o->pickaxe;\n \tint opts = o->pickaxe_opts;\n \tregex_t regex, *regexp = NULL;\n \tkwset_t kws = NULL;\n-\tint err = 0;\n \n \tif (opts & (DIFF_PICKAXE_REGEX | DIFF_PICKAXE_KIND_G)) {\n \t\tint cflags = REG_EXTENDED | REG_NEWLINE;\n \t\tif (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE))\n \t\t\tcflags |= REG_ICASE;\n-\t\terr = regcomp(®ex, needle, cflags);\n+\t\tregcomp_or_die(®ex, needle, cflags);\n \t\tregexp = ®ex;\n \t} else if (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE) &&\n \t\t has_non_ascii(needle)) {\n@@ -220,7 +231,7 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\tint cflags = REG_NEWLINE | REG_ICASE;\n \n \t\tbasic_regex_quote_buf(&sb, needle);\n-\t\terr = regcomp(®ex, sb.buf, cflags);\n+\t\tregcomp_or_die(®ex, sb.buf, cflags);\n \t\tstrbuf_release(&sb);\n \t\tregexp = ®ex;\n \t} else {\n@@ -229,13 +240,6 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\tkwsincr(kws, needle, strlen(needle));\n \t\tkwsprep(kws);\n \t}\n-\tif (err) {\n-\t\t/* The POSIX.2 people are surely sick */\n-\t\tchar errbuf[1024];\n-\t\tregerror(err, ®ex, errbuf, 1024);\n-\t\tregfree(®ex);\n-\t\tdie(\"invalid regex: %s\", errbuf);\n-\t}\n \n \t/* Might want to warn when both S and G are on; I don't care... */\n \tpickaxe(&diff_queued_diff, o, regexp, kws,\ndiff --git a/grep.c b/grep.c\nindex cb058a5..92587a8 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -432,15 +432,8 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \ticase\t = opt->regflags & REG_ICASE || p->ignore_case;\n \tascii_only = !has_non_ascii(p->pattern);\n \n-\tif (opt->fixed) {\n+\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = !icase || ascii_only;\n-\t\tif (!p->fixed) {\n-\t\t\tcompile_fixed_regexp(p, opt);\n-\t\t\treturn;\n-\t\t}\n-\t} else if ((!icase || ascii_only) &&\n-\t\t is_fixed(p->pattern, p->patternlen))\n-\t\tp->fixed = 1;\n \telse\n \t\tp->fixed = 0;\n \n@@ -449,6 +442,9 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n \t\tkwsprep(p->kws);\n \t\treturn;\n+\t} else if (opt->fixed) {\n+\t\tcompile_fixed_regexp(p, opt);\n+\t\treturn;\n \t}\n \n \tif (opt->pcre) {\n\nNguyễn Thái Ngọc Duy (12):\n grep: break down an \"if\" stmt in preparation for next changes\n test-regex: isolate the bug test code\n test-regex: expose full regcomp() to the command line\n grep/icase: avoid kwsset on literal non-ascii strings\n grep/icase: avoid kwsset when -F is specified\n grep: rewrite an if/else condition to avoid duplicate expression\n grep/pcre: prepare locale-dependent tables for icase matching\n gettext: add is_utf8_locale()\n grep/pcre: support utf-8\n diffcore-pickaxe: Add regcomp_or_die()\n diffcore-pickaxe: support case insensitive match on non-ascii\n grep.c: reuse \"icase\" variable\n\n diffcore-pickaxe.c | 33 +++++++++++----\n gettext.c | 24 ++++++++++-\n gettext.h | 1 +\n grep.c | 43 +++++++++++++++----\n grep.h | 1 +\n quote.c | 37 +++++++++++++++++\n quote.h | 1 +\n t/t0070-fundamental.sh | 2 +-\n t/t7812-grep-icase-non-ascii.sh (new +x) | 71 ++++++++++++++++++++++++++++++++\n t/t7813-grep-icase-iso.sh (new +x) | 19 +++++++++\n test-regex.c | 59 +++++++++++++++++++++++++-\n 11 files changed, 270 insertions(+), 21 deletions(-)\n create mode 100755 t/t7812-grep-icase-non-ascii.sh\n create mode 100755 t/t7813-grep-icase-iso.sh\n\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290096","messageId":"20160625052238.13615-4-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 03/12] test-regex: expose full regcomp() to the command line","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:29Z","receivedAt":"2016-06-25T05:24:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n test-regex.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 49 insertions(+), 2 deletions(-)\n\ndiff --git a/test-regex.c b/test-regex.c\nindex 67a1a65..eff26f5 100644\n--- a/test-regex.c\n+++ b/test-regex.c\n@@ -1,4 +1,21 @@\n #include \"git-compat-util.h\"\n+#include \"gettext.h\"\n+\n+struct reg_flag {\n+\tconst char *name;\n+\tint flag;\n+};\n+\n+static struct reg_flag reg_flags[] = {\n+\t{ \"EXTENDED\",\t REG_EXTENDED\t},\n+\t{ \"NEWLINE\",\t REG_NEWLINE\t},\n+\t{ \"ICASE\",\t REG_ICASE\t},\n+\t{ \"NOTBOL\",\t REG_NOTBOL\t},\n+#ifdef REG_STARTEND\n+\t{ \"STARTEND\",\t REG_STARTEND\t},\n+#endif\n+\t{ NULL, 0 }\n+};\n \n static int test_regex_bug(void)\n {\n@@ -21,8 +38,38 @@ static int test_regex_bug(void)\n \n int main(int argc, char **argv)\n {\n+\tconst char *pat;\n+\tconst char *str;\n+\tint flags = 0;\n+\tregex_t r;\n+\tregmatch_t m[1];\n+\n \tif (argc == 2 && !strcmp(argv[1], \"--bug\"))\n \t\treturn test_regex_bug();\n-\telse\n-\t\tusage(\"test-regex --bug\");\n+\telse if (argc < 3)\n+\t\tusage(\"test-regex --bug\\n\"\n+\t\t \"test-regex []\");\n+\n+\targv++;\n+\tpat = *argv++;\n+\tstr = *argv++;\n+\twhile (*argv) {\n+\t\tstruct reg_flag *rf;\n+\t\tfor (rf = reg_flags; rf->name; rf++)\n+\t\t\tif (!strcmp(*argv, rf->name)) {\n+\t\t\t\tflags |= rf->flag;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\tif (!rf->name)\n+\t\t\tdie(\"do not recognize %s\", *argv);\n+\t\targv++;\n+\t}\n+\tgit_setup_gettext();\n+\n+\tif (regcomp(&r, pat, flags))\n+\t\tdie(\"failed regcomp() for pattern '%s'\", pat);\n+\tif (regexec(&r, str, 1, m, 0))\n+\t\treturn 1;\n+\n+\treturn 0;\n }\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290097","messageId":"20160625052238.13615-5-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 04/12] grep/icase: avoid kwsset on literal non-ascii strings","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:30Z","receivedAt":"2016-06-25T05:24:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When we detect the pattern is just a literal string, we avoid heavy\nregex engine and use fast substring search implemented in kwsset.c.\nBut kws uses git-ctype which is locale-independent so it does not know\nhow to fold case properly outside ascii range. Let regcomp or pcre\ntake care of this case instead. Slower, but accurate.\n\nNoticed-by: Plamen Totev \nHelped-by: René Scharfe \nHelped-by: Eric Sunshine \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 7 ++++++-\n t/t7812-grep-icase-non-ascii.sh (new +x) | 23 +++++++++++++++++++++++\n 2 files changed, 29 insertions(+), 1 deletion(-)\n create mode 100755 t/t7812-grep-icase-non-ascii.sh\n\ndiff --git a/grep.c b/grep.c\nindex f430d7e..451275d 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -4,6 +4,7 @@\n #include \"xdiff-interface.h\"\n #include \"diff.h\"\n #include \"diffcore.h\"\n+#include \"commit.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -398,14 +399,18 @@ static int is_fixed(const char *s, size_t len)\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n+\tint icase, ascii_only;\n \tint err;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n+\ticase\t = opt->regflags & REG_ICASE || p->ignore_case;\n+\tascii_only = !has_non_ascii(p->pattern);\n \n \tif (opt->fixed)\n \t\tp->fixed = 1;\n-\telse if (is_fixed(p->pattern, p->patternlen))\n+\telse if ((!icase || ascii_only) &&\n+\t\t is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse\n \t\tp->fixed = 0;\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nnew file mode 100755\nindex 0000000..b78a774\n--- /dev/null\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_LOCALE 'setup' '\n+\ttest_write_lines \"TILRAUN: Halló Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_have_prereq GETTEXT_LOCALE &&\n+test-regex \"HALLÓ\" \"Halló\" ICASE &&\n+test_set_prereq REGEX_LOCALE\n+\n+test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n+\tgit grep -i \"TILRAUN: Halló Heimur!\" &&\n+\tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290098","messageId":"20160625052238.13615-8-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 07/12] grep/pcre: prepare locale-dependent tables for icase matching","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:33Z","receivedAt":"2016-06-25T05:24:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The default tables are usually built with C locale and only suitable\nfor LANG=C or similar. This should make case insensitive search work\ncorrectly for all single-byte charsets.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 8 ++++++--\n grep.h | 1 +\n t/t7813-grep-icase-iso.sh (new +x) | 19 +++++++++++++++++++\n 3 files changed, 26 insertions(+), 2 deletions(-)\n create mode 100755 t/t7813-grep-icase-iso.sh\n\ndiff --git a/grep.c b/grep.c\nindex 4b6b7bc..8cf4247 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -324,11 +324,14 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \tint erroffset;\n \tint options = PCRE_MULTILINE;\n \n-\tif (opt->ignore_case)\n+\tif (opt->ignore_case) {\n+\t\tif (has_non_ascii(p->pattern))\n+\t\t\tp->pcre_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n+\t}\n \n \tp->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n-\t\t\tNULL);\n+\t\t\t\t p->pcre_tables);\n \tif (!p->pcre_regexp)\n \t\tcompile_regexp_failed(p, error);\n \n@@ -362,6 +365,7 @@ static void free_pcre_regexp(struct grep_pat *p)\n {\n \tpcre_free(p->pcre_regexp);\n \tpcre_free(p->pcre_extra_info);\n+\tpcre_free((void *)p->pcre_tables);\n }\n #else /* !USE_LIBPCRE */\n static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\ndiff --git a/grep.h b/grep.h\nindex 95f197a..cee4357 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -48,6 +48,7 @@ struct grep_pat {\n \tregex_t regexp;\n \tpcre *pcre_regexp;\n \tpcre_extra *pcre_extra_info;\n+\tconst unsigned char *pcre_tables;\n \tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\ndiff --git a/t/t7813-grep-icase-iso.sh b/t/t7813-grep-icase-iso.sh\nnew file mode 100755\nindex 0000000..efef7fb\n--- /dev/null\n+++ b/t/t7813-grep-icase-iso.sh\n@@ -0,0 +1,19 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_ISO_LOCALE 'setup' '\n+\tprintf \"TILRAUN: Hall� Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_iso_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_expect_success GETTEXT_ISO_LOCALE,LIBPCRE 'grep pcre string' '\n+\tgit grep --perl-regexp -i \"TILRAUN: H.ll� Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LL� HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290099","messageId":"20160625052238.13615-7-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 06/12] grep: rewrite an if/else condition to avoid duplicate expression","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:32Z","receivedAt":"2016-06-25T05:24:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\"!icase || ascii_only\" is repeated twice in this if/else chain as this\nseries evolves. Rewrite it (and basically revert the first if\ncondition back to before the \"grep: break down an \"if\" stmt...\" commit).\n\nHelped-by: Junio C Hamano \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 5 +----\n 1 file changed, 1 insertion(+), 4 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 5a8c52a..4b6b7bc 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -426,11 +426,8 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \ticase\t = opt->regflags & REG_ICASE || p->ignore_case;\n \tascii_only = !has_non_ascii(p->pattern);\n \n-\tif (opt->fixed)\n+\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = !icase || ascii_only;\n-\telse if ((!icase || ascii_only) &&\n-\t\t is_fixed(p->pattern, p->patternlen))\n-\t\tp->fixed = 1;\n \telse\n \t\tp->fixed = 0;\n \n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290100","messageId":"20160625052238.13615-11-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 10/12] diffcore-pickaxe: Add regcomp_or_die()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:36Z","receivedAt":"2016-06-25T05:24:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"There's another regcomp code block coming in this function that needs\nthe same error handling. This function can help avoid duplicating\nerror handling code.\n\nHelped-by: Jeff King \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n diffcore-pickaxe.c | 22 +++++++++++++---------\n 1 file changed, 13 insertions(+), 9 deletions(-)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 7715c13..2093b6a 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -198,6 +198,18 @@ static void pickaxe(struct diff_queue_struct *q, struct diff_options *o,\n \t*q = outq;\n }\n \n+static void regcomp_or_die(regex_t *regex, const char *needle, int cflags)\n+{\n+\tint err = regcomp(regex, needle, cflags);\n+\tif (err) {\n+\t\t/* The POSIX.2 people are surely sick */\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, regex, errbuf, 1024);\n+\t\tregfree(regex);\n+\t\tdie(\"invalid regex: %s\", errbuf);\n+\t}\n+}\n+\n void diffcore_pickaxe(struct diff_options *o)\n {\n \tconst char *needle = o->pickaxe;\n@@ -206,18 +218,10 @@ void diffcore_pickaxe(struct diff_options *o)\n \tkwset_t kws = NULL;\n \n \tif (opts & (DIFF_PICKAXE_REGEX | DIFF_PICKAXE_KIND_G)) {\n-\t\tint err;\n \t\tint cflags = REG_EXTENDED | REG_NEWLINE;\n \t\tif (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE))\n \t\t\tcflags |= REG_ICASE;\n-\t\terr = regcomp(®ex, needle, cflags);\n-\t\tif (err) {\n-\t\t\t/* The POSIX.2 people are surely sick */\n-\t\t\tchar errbuf[1024];\n-\t\t\tregerror(err, ®ex, errbuf, 1024);\n-\t\t\tregfree(®ex);\n-\t\t\tdie(\"invalid regex: %s\", errbuf);\n-\t\t}\n+\t\tregcomp_or_die(®ex, needle, cflags);\n \t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290101","messageId":"20160625052238.13615-10-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 09/12] grep/pcre: support utf-8","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:35Z","receivedAt":"2016-06-25T05:24:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In the previous change in this function, we add locale support for\nsingle-byte encodings only. It looks like pcre only supports utf-* as\nmultibyte encodings, the others are left in the cold (which is\nfine).\n\nWe need to enable PCRE_UTF8 so pcre can find character boundary\ncorrectly. It's needed for case folding (when --ignore-case is used)\nor '*', '+' or similar syntax is used.\n\nThe \"has_non_ascii()\" check is to be on the conservative side. If\nthere's non-ascii in the pattern, the searched content could still be\nin utf-8, but we can treat it just like a byte stream and everything\nshould work. If we force utf-8 based on locale only and pcre validates\nutf-8 and the file content is in non-utf8 encoding, things break.\n\nNoticed-by: Plamen Totev \nHelped-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 2 ++\n t/t7812-grep-icase-non-ascii.sh | 15 +++++++++++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/grep.c b/grep.c\nindex 8cf4247..3d63612 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -329,6 +329,8 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \t\t\tp->pcre_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n \t}\n+\tif (is_utf8_locale() && has_non_ascii(p->pattern))\n+\t\toptions |= PCRE_UTF8;\n \n \tp->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n \t\t\t\t p->pcre_tables);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 1929809..08ae4c9 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -20,6 +20,21 @@ test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 icase' '\n+\tgit grep --perl-regexp \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LLÓ HEIMUR!\"\n+'\n+\n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 string with \"+\"' '\n+\ttest_write_lines \"TILRAUN: Hallóó Heimur!\" >file2 &&\n+\tgit add file2 &&\n+\tgit grep -l --perl-regexp \"TILRAUN: H.lló+ Heimur!\" >actual &&\n+\techo file >expected &&\n+\techo file2 >>expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success REGEX_LOCALE 'grep literal string, with -F' '\n \tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290102","messageId":"20160625052238.13615-6-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 05/12] grep/icase: avoid kwsset when -F is specified","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:31Z","receivedAt":"2016-06-25T05:24:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the previous commit, we can't use kws on icase search\noutside ascii range. But we can't simply pass the pattern to\nregcomp/pcre like the previous commit because it may contain regex\nspecial characters, so we need to quote the regex first.\n\nTo avoid misquote traps that could lead to undefined behavior, we\nalways stick to basic regex engine in this case. We don't need fancy\nfeatures for grepping a literal string anyway.\n\nbasic_regex_quote_buf() assumes that if the pattern is in a multibyte\nencoding, ascii chars must be unambiguously encoded as single\nbytes. This is true at least for UTF-8. For others, let's wait until\npeople yell up. Chances are nobody uses multibyte, non utf-8 charsets\nanymore.\n\nNoticed-by: Plamen Totev \nHelped-by: René Scharfe \nHelped-by: Eric Sunshine \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 24 +++++++++++++++++++++++-\n quote.c | 37 +++++++++++++++++++++++++++++++++++++\n quote.h | 1 +\n t/t7812-grep-icase-non-ascii.sh | 26 ++++++++++++++++++++++++++\n 4 files changed, 87 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 451275d..5a8c52a 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -5,6 +5,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"commit.h\"\n+#include \"quote.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -397,6 +398,24 @@ static int is_fixed(const char *s, size_t len)\n \treturn 1;\n }\n \n+static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint err;\n+\n+\tbasic_regex_quote_buf(&sb, p->pattern);\n+\terr = regcomp(&p->regexp, sb.buf, opt->regflags & ~REG_EXTENDED);\n+\tif (opt->debug)\n+\t\tfprintf(stderr, \"fixed %s\\n\", sb.buf);\n+\tstrbuf_release(&sb);\n+\tif (err) {\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, &p->regexp, errbuf, sizeof(errbuf));\n+\t\tregfree(&p->regexp);\n+\t\tcompile_regexp_failed(p, errbuf);\n+\t}\n+}\n+\n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tint icase, ascii_only;\n@@ -408,7 +427,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tascii_only = !has_non_ascii(p->pattern);\n \n \tif (opt->fixed)\n-\t\tp->fixed = 1;\n+\t\tp->fixed = !icase || ascii_only;\n \telse if ((!icase || ascii_only) &&\n \t\t is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n@@ -423,6 +442,9 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n \t\tkwsprep(p->kws);\n \t\treturn;\n+\t} else if (opt->fixed) {\n+\t\tcompile_fixed_regexp(p, opt);\n+\t\treturn;\n \t}\n \n \tif (opt->pcre) {\ndiff --git a/quote.c b/quote.c\nindex fe884d2..c67adb7 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -440,3 +440,40 @@ void tcl_quote_buf(struct strbuf *sb, const char *src)\n \t}\n \tstrbuf_addch(sb, '\"');\n }\n+\n+void basic_regex_quote_buf(struct strbuf *sb, const char *src)\n+{\n+\tchar c;\n+\n+\tif (*src == '^') {\n+\t\t/* only beginning '^' is special and needs quoting */\n+\t\tstrbuf_addch(sb, '\\\\');\n+\t\tstrbuf_addch(sb, *src++);\n+\t}\n+\tif (*src == '*')\n+\t\t/* beginning '*' is not special, no quoting */\n+\t\tstrbuf_addch(sb, *src++);\n+\n+\twhile ((c = *src++)) {\n+\t\tswitch (c) {\n+\t\tcase '[':\n+\t\tcase '.':\n+\t\tcase '\\\\':\n+\t\tcase '*':\n+\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tcase '$':\n+\t\t\t/* only the end '$' is special and needs quoting */\n+\t\t\tif (*src == '\\0')\n+\t\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\ndiff --git a/quote.h b/quote.h\nindex 99e04d3..362d315 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -67,5 +67,6 @@ extern char *quote_path_relative(const char *in, const char *prefix,\n extern void perl_quote_buf(struct strbuf *sb, const char *src);\n extern void python_quote_buf(struct strbuf *sb, const char *src);\n extern void tcl_quote_buf(struct strbuf *sb, const char *src);\n+extern void basic_regex_quote_buf(struct strbuf *sb, const char *src);\n \n #endif\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex b78a774..1929809 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -20,4 +20,30 @@ test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success REGEX_LOCALE 'grep literal string, with -F' '\n+\tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\ttest_write_lines \"fixed TILRAUN: Halló Heimur!\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"TILRAUN: HALLÓ HEIMUR!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\ttest_write_lines \"fixed TILRAUN: HALLÓ HEIMUR!\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n+test_expect_success REGEX_LOCALE 'grep string with regex, with -F' '\n+\ttest_write_lines \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\ttest_write_lines \"fixed \\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\HALLÓ \\$HE[]IMUR!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\ttest_write_lines \"fixed \\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n test_done\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290103","messageId":"20160625052238.13615-13-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 12/12] grep.c: reuse \"icase\" variable","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:38Z","receivedAt":"2016-06-25T05:24:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 5 +----\n 1 file changed, 1 insertion(+), 4 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 3d63612..92587a8 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -438,10 +438,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tp->fixed = 0;\n \n \tif (p->fixed) {\n-\t\tif (opt->regflags & REG_ICASE || p->ignore_case)\n-\t\t\tp->kws = kwsalloc(tolower_trans_tbl);\n-\t\telse\n-\t\t\tp->kws = kwsalloc(NULL);\n+\t\tp->kws = kwsalloc(icase ? tolower_trans_tbl : NULL);\n \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n \t\tkwsprep(p->kws);\n \t\treturn;\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290104","messageId":"20160625052238.13615-9-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 08/12] gettext: add is_utf8_locale()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:34Z","receivedAt":"2016-06-25T05:24:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This function returns true if git is running under an UTF-8\nlocale. pcre in the next patch will need this.\n\nis_encoding_utf8() is used instead of strcmp() to catch both \"utf-8\"\nand \"utf8\" suffixes.\n\nWhen built with no gettext support, we peek in several env variables\nto detect UTF-8. pcre library might support utf-8 even if libc is\nbuilt without locale support.. The peeking code is a copy from\ncompat/regex/regcomp.c\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n gettext.c | 24 ++++++++++++++++++++++--\n gettext.h | 1 +\n 2 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/gettext.c b/gettext.c\nindex a268a2c..db727ea 100644\n--- a/gettext.c\n+++ b/gettext.c\n@@ -18,6 +18,8 @@\n #\tendif\n #endif\n \n+static const char *charset;\n+\n /*\n * Guess the user's preferred languages from the value in LANGUAGE environment\n * variable and LC_MESSAGES locale category if NO_GETTEXT is not defined.\n@@ -65,7 +67,6 @@ static int test_vsnprintf(const char *fmt, ...)\n \treturn ret;\n }\n \n-static const char *charset;\n static void init_gettext_charset(const char *domain)\n {\n \t/*\n@@ -172,8 +173,27 @@ int gettext_width(const char *s)\n {\n \tstatic int is_utf8 = -1;\n \tif (is_utf8 == -1)\n-\t\tis_utf8 = !strcmp(charset, \"UTF-8\");\n+\t\tis_utf8 = is_utf8_locale();\n \n \treturn is_utf8 ? utf8_strwidth(s) : strlen(s);\n }\n #endif\n+\n+int is_utf8_locale(void)\n+{\n+#ifdef NO_GETTEXT\n+\tif (!charset) {\n+\t\tconst char *env = getenv(\"LC_ALL\");\n+\t\tif (!env || !*env)\n+\t\t\tenv = getenv(\"LC_CTYPE\");\n+\t\tif (!env || !*env)\n+\t\t\tenv = getenv(\"LANG\");\n+\t\tif (!env)\n+\t\t\tenv = \"\";\n+\t\tif (strchr(env, '.'))\n+\t\t\tenv = strchr(env, '.') + 1;\n+\t\tcharset = xstrdup(env);\n+\t}\n+#endif\n+\treturn is_encoding_utf8(charset);\n+}\ndiff --git a/gettext.h b/gettext.h\nindex 33696a4..7eee64a 100644\n--- a/gettext.h\n+++ b/gettext.h\n@@ -90,5 +90,6 @@ const char *Q_(const char *msgid, const char *plu, unsigned long n)\n #endif\n \n const char *get_preferred_languages(void);\n+extern int is_utf8_locale(void);\n \n #endif\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290105","messageId":"20160625052238.13615-2-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 01/12] grep: break down an \"if\" stmt in preparation for next changes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:27Z","receivedAt":"2016-06-25T05:24:42Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n grep.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 7b2b96a..f430d7e 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -403,7 +403,9 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \n-\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n+\tif (opt->fixed)\n+\t\tp->fixed = 1;\n+\telse if (is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse\n \t\tp->fixed = 0;\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290106","messageId":"20160625052238.13615-12-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 11/12] diffcore-pickaxe: support case insensitive match on non-ascii","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:37Z","receivedAt":"2016-06-25T05:24:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the \"grep -F -i\" case, we can't use kws on icase search\noutside ascii range, so we quote the string and pass it to regcomp as\na basic regexp and let regex engine deal with case sensitivity.\n\nThe new test is put in t7812 instead of t4209-log-pickaxe because\nlib-gettext.sh might cause problems elsewhere, probably.\n\nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n diffcore-pickaxe.c | 11 +++++++++++\n t/t7812-grep-icase-non-ascii.sh | 7 +++++++\n 2 files changed, 18 insertions(+)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 2093b6a..55067ca 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -7,6 +7,8 @@\n #include \"diffcore.h\"\n #include \"xdiff-interface.h\"\n #include \"kwset.h\"\n+#include \"commit.h\"\n+#include \"quote.h\"\n \n typedef int (*pickaxe_fn)(mmfile_t *one, mmfile_t *two,\n \t\t\t struct diff_options *o,\n@@ -223,6 +225,15 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\t\tcflags |= REG_ICASE;\n \t\tregcomp_or_die(®ex, needle, cflags);\n \t\tregexp = ®ex;\n+\t} else if (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE) &&\n+\t\t has_non_ascii(needle)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tint cflags = REG_NEWLINE | REG_ICASE;\n+\n+\t\tbasic_regex_quote_buf(&sb, needle);\n+\t\tregcomp_or_die(®ex, sb.buf, cflags);\n+\t\tstrbuf_release(&sb);\n+\t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n \t\t\t ? tolower_trans_tbl : NULL);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 08ae4c9..169fd8d 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -61,4 +61,11 @@ test_expect_success REGEX_LOCALE 'grep string with regex, with -F' '\n \ttest_cmp expect2 debug2\n '\n \n+test_expect_success REGEX_LOCALE 'pickaxe -i on non-ascii' '\n+\tgit commit -m first &&\n+\tgit log --format=%f -i -S\"TILRAUN: HALLÓ HEIMUR!\" >actual &&\n+\techo first >expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290107","messageId":"20160625052238.13615-3-pclouds@gmail.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"[PATCH v2 02/12] test-regex: isolate the bug test code","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-25T05:22:28Z","receivedAt":"2016-06-25T05:24:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is in preparation to turn test-regex into some generic regex\ntesting command.\n\nHelped-by: Eric Sunshine \nHelped-by: Ramsay Jones \nSigned-off-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Junio C Hamano \n---\n t/t0070-fundamental.sh | 2 +-\n test-regex.c | 12 ++++++++++--\n 2 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t0070-fundamental.sh b/t/t0070-fundamental.sh\nindex 5ed69a6..991ed2a 100755\n--- a/t/t0070-fundamental.sh\n+++ b/t/t0070-fundamental.sh\n@@ -31,7 +31,7 @@ test_expect_success 'git_mkstemps_mode does not fail if fd 0 is not open' '\n \n test_expect_success 'check for a bug in the regex routines' '\n \t# if this test fails, re-build git with NO_REGEX=1\n-\ttest-regex\n+\ttest-regex --bug\n '\n \n test_done\ndiff --git a/test-regex.c b/test-regex.c\nindex 0dc598e..67a1a65 100644\n--- a/test-regex.c\n+++ b/test-regex.c\n@@ -1,6 +1,6 @@\n #include \"git-compat-util.h\"\n \n-int main(int argc, char **argv)\n+static int test_regex_bug(void)\n {\n \tchar *pat = \"[^={} \\t]+\";\n \tchar *str = \"={}\\nfred\";\n@@ -16,5 +16,13 @@ int main(int argc, char **argv)\n \tif (m[0].rm_so == 3) /* matches '\\n' when it should not */\n \t\tdie(\"regex bug confirmed: re-build git with NO_REGEX=1\");\n \n-\texit(0);\n+\treturn 0;\n+}\n+\n+int main(int argc, char **argv)\n+{\n+\tif (argc == 2 && !strcmp(argv[1], \"--bug\"))\n+\t\treturn test_regex_bug();\n+\telse\n+\t\tusage(\"test-regex --bug\");\n }\n-- \n2.8.2.526.g02eed6d\n\n"},{"id":"290236","messageId":"xmqqeg7id6ns.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"20160625052238.13615-1-pclouds@gmail.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-27T14:53:59Z","receivedAt":"2016-06-27T14:54:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> v2 fixes Junio's and Jeff's comments (both good). The sharing \"!icase\n> || ascii_only\" is made a separate commit (6/12) because I think it\n> takes some seconds to realize that the conversion is correct and it's\n> technically not needed in 5/12 (and it's sort of the opposite of 1/12)\n>\n> Interdiff\n\nOK. regcomp_or_die() does make the code simpler.\n\n> diff --git a/grep.c b/grep.c\n> index cb058a5..92587a8 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -432,15 +432,8 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n> \ticase\t = opt->regflags & REG_ICASE || p->ignore_case;\n> \tascii_only = !has_non_ascii(p->pattern);\n> \n> +\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n> \t\tp->fixed = !icase || ascii_only;\n> \telse\n> \t\tp->fixed = 0;\n> \n> @@ -449,6 +442,9 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n> \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n> \t\tkwsprep(p->kws);\n> \t\treturn;\n> +\t} else if (opt->fixed) {\n> +\t\tcompile_fixed_regexp(p, opt);\n> +\t\treturn;\n> \t}\n\nThis if/elseif/else cascade made a lot simpler and while the\ndiscussion is fresh in my brain it makes sense, but it may deserve a\nbit of commenting.\n\nAnd while attempting to do so, I found one possible issue there.\n\nCan't p->ignore_case be true even when opt->regflags does not have\nREG_ICASE? The user never asked us to do a regexp match in such a\ncase, and the logical place to compensate for that case would be\ninside compile_fixed_regexp(), where we use regexp engine behind\nuser's back for our convenience, I would think.\n\nIn the current code, compile_fixed_regexp() is only called when we\nwant ICASE, but hardcoding that assumption to it unnecessarily robs\nflexibility (and the function name does not tell us it is only for\nicase in the first place), so I taught it to do the REG_ICASE thing\nonly when opt->ignore_case is set.\n\nHow does this look?\n\n\n grep.c | 23 ++++++++++++++++++++++-\n 1 file changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 92587a8..3a3a9f4 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -407,17 +407,21 @@ static int is_fixed(const char *s, size_t len)\n static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint err;\n+\tint regflags;\n \n \tbasic_regex_quote_buf(&sb, p->pattern);\n-\terr = regcomp(&p->regexp, sb.buf, opt->regflags & ~REG_EXTENDED);\n+\tregflags = opt->regflags & ~REG_EXTENDED;\n+\tif (opt->ignore_case)\n+\t\tregflags |= REG_ICASE;\n+\terr = regcomp(&p->regexp, sb.buf, regflags);\n \tif (opt->debug)\n \t\tfprintf(stderr, \"fixed %s\\n\", sb.buf);\n \tstrbuf_release(&sb);\n \tif (err) {\n \t\tchar errbuf[1024];\n \t\tregerror(err, &p->regexp, errbuf, sizeof(errbuf));\n \t\tregfree(&p->regexp);\n \t\tcompile_regexp_failed(p, errbuf);\n \t}\n }\n@@ -425,38 +429,55 @@ static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tint icase, ascii_only;\n \tint err;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \ticase\t = opt->regflags & REG_ICASE || p->ignore_case;\n \tascii_only = !has_non_ascii(p->pattern);\n \n+\t/*\n+\t * Even when -F (fixed) asks us to do a non-regexp search, we\n+\t * may not be able to correctly case-fold when -i\n+\t * (ignore-case) is asked (in which case, we'll synthesize a\n+\t * regexp to match the pattern that matches regexp special\n+\t * characters literally, while ignoring case differences). On\n+\t * the other hand, even without -F, if the pattern does not\n+\t * have any regexp special characters and there is no need for\n+\t * case-folding search, we can internally turn it into a\n+\t * simple string match using kws. p->fixed tells us if we\n+\t * want to use kws.\n+\t */\n \tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = !icase || ascii_only;\n \telse\n \t\tp->fixed = 0;\n \n \tif (p->fixed) {\n \t\tp->kws = kwsalloc(icase ? tolower_trans_tbl : NULL);\n \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n \t\tkwsprep(p->kws);\n \t\treturn;\n \t} else if (opt->fixed) {\n+\t\t/*\n+\t\t * We only come here when the pattern has the regexp\n+\t\t * special characters in it, which need to be matched\n+\t\t * literally, while ignoring case.\n+\t\t */\n \t\tcompile_fixed_regexp(p, opt);\n \t\treturn;\n \t}\n \n \tif (opt->pcre) {\n \t\tcompile_pcre_regexp(p, opt);\n \t\treturn;\n \t}\n \n \terr = regcomp(&p->regexp, p->pattern, opt->regflags);\n \tif (err) {\n \t\tchar errbuf[1024];\n \t\tregerror(err, &p->regexp, errbuf, 1024);\n \t\tregfree(&p->regexp);\n \t\tcompile_regexp_failed(p, errbuf);\n \t}\n }\n"},{"id":"290237","messageId":"xmqqa8i6d6dl.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"xmqqeg7id6ns.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-06-27T15:00:06Z","receivedAt":"2016-06-27T15:00:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> \tif (p->fixed) {\n> \t\tp->kws = kwsalloc(icase ? tolower_trans_tbl : NULL);\n> \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n> \t\tkwsprep(p->kws);\n> \t\treturn;\n> \t} else if (opt->fixed) {\n> +\t\t/*\n> +\t\t * We only come here when the pattern has the regexp\n> +\t\t * special characters in it, which need to be matched\n> +\t\t * literally, while ignoring case.\n> +\t\t */\n\nThis comment is wrong.\n\n\t\t/*\n\t\t * We come here only when the pattern has non-ascii\n\t\t * characters we cannot case-fold, and we were asked\n\t\t * to ignore-case.\n\t\t */\n\n> \t\tcompile_fixed_regexp(p, opt);\n> \t\treturn;\n> \t}\n"},{"id":"290598","messageId":"CACsJy8BU0fDVR54hMpA6qVknj+QxWR9Z-i1gRgpaJ6hp+SB2xQ@mail.gmail.com","threadId":"42699","inReplyTo":"xmqqeg7id6ns.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-06-30T15:45:52Z","receivedAt":"2016-06-30T15:59:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jun 27, 2016 at 4:53 PM, Junio C Hamano wrote:\n>> diff --git a/grep.c b/grep.c\n>> index cb058a5..92587a8 100644\n>> --- a/grep.c\n>> +++ b/grep.c\n>> @@ -432,15 +432,8 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n>> icase = opt->regflags & REG_ICASE || p->ignore_case;\n>> ascii_only = !has_non_ascii(p->pattern);\n>>\n>> + if (opt->fixed || is_fixed(p->pattern, p->patternlen))\n>> p->fixed = !icase || ascii_only;\n>> else\n>> p->fixed = 0;\n>>\n>> @@ -449,6 +442,9 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n>> kwsincr(p->kws, p->pattern, p->patternlen);\n>> kwsprep(p->kws);\n>> return;\n>> + } else if (opt->fixed) {\n>> + compile_fixed_regexp(p, opt);\n>> + return;\n>> }\n>\n> This if/elseif/else cascade made a lot simpler and while the\n> discussion is fresh in my brain it makes sense, but it may deserve a\n> bit of commenting.\n>\n> And while attempting to do so, I found one possible issue there.\n>\n> Can't p->ignore_case be true even when opt->regflags does not have\n> REG_ICASE? The user never asked us to do a regexp match in such a\n> case, and the logical place to compensate for that case would be\n> inside compile_fixed_regexp(), where we use regexp engine behind\n> user's back for our convenience, I would think.\n>\n> In the current code, compile_fixed_regexp() is only called when we\n> want ICASE, but hardcoding that assumption to it unnecessarily robs\n> flexibility (and the function name does not tell us it is only for\n> icase in the first place), so I taught it to do the REG_ICASE thing\n> only when opt->ignore_case is set.\n>\n> How does this look?\n>\n>\n> grep.c | 23 ++++++++++++++++++++++-\n> 1 file changed, 22 insertions(+), 1 deletion(-)\n>\n> diff --git a/grep.c b/grep.c\n> index 92587a8..3a3a9f4 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -407,17 +407,21 @@ static int is_fixed(const char *s, size_t len)\n> static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n> {\n> struct strbuf sb = STRBUF_INIT;\n> int err;\n> + int regflags;\n>\n> basic_regex_quote_buf(&sb, p->pattern);\n> - err = regcomp(&p->regexp, sb.buf, opt->regflags & ~REG_EXTENDED);\n> + regflags = opt->regflags & ~REG_EXTENDED;\n> + if (opt->ignore_case)\n> + regflags |= REG_ICASE;\n> + err = regcomp(&p->regexp, sb.buf, regflags);\n\nMakes sense. But then if opt->ignore_case is false and regflags\nhappens to have REG_ICASE set, should we clear it as well?\n\nThe rest looks good (after your comment fixup). I see you already have\nall the changes in your SQUASH??? commit. Do you want me to resend or\nyou will just squash this in locally?\n-- \nDuy\n"},{"id":"290694","messageId":"xmqq4m89z0gl.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"CACsJy8BU0fDVR54hMpA6qVknj+QxWR9Z-i1gRgpaJ6hp+SB2xQ@mail.gmail.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-01T18:18:18Z","receivedAt":"2016-07-01T18:19:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> Makes sense. But then if opt->ignore_case is false and regflags\n> happens to have REG_ICASE set, should we clear it as well?\n\nI think .ignore_case is set iff '-i' is given, and .regflags has\nREG_ICASE only if '-i' is given and the user said she does not want\nliteral string match (i.e. no '-F').\n\nSo... can .regflags have REG_ICASE when .ignore_case is false?\n"},{"id":"290697","messageId":"CACsJy8BBQqxzT8jdaysyeDv3ja=DTD6R-whA3Lhr3pEs-2x8dA@mail.gmail.com","threadId":"42699","inReplyTo":"xmqq4m89z0gl.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-01T18:46:30Z","receivedAt":"2016-07-01T18:48:03Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jul 1, 2016 at 8:18 PM, Junio C Hamano wrote:\n> Duy Nguyen writes:\n>\n>> Makes sense. But then if opt->ignore_case is false and regflags\n>> happens to have REG_ICASE set, should we clear it as well?\n>\n> I think .ignore_case is set iff '-i' is given, and .regflags has\n> REG_ICASE only if '-i' is given and the user said she does not want\n> literal string match (i.e. no '-F').\n>\n> So... can .regflags have REG_ICASE when .ignore_case is false?\n\nYeah reg_icase is more like a subset of ignore_case. Ignore what I wrote.\n-- \nDuy\n"},{"id":"290698","messageId":"xmqqziq1xk8j.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"CACsJy8BBQqxzT8jdaysyeDv3ja=DTD6R-whA3Lhr3pEs-2x8dA@mail.gmail.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-01T18:54:04Z","receivedAt":"2016-07-01T18:54:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On Fri, Jul 1, 2016 at 8:18 PM, Junio C Hamano wrote:\n>> Duy Nguyen writes:\n>>\n>>> Makes sense. But then if opt->ignore_case is false and regflags\n>>> happens to have REG_ICASE set, should we clear it as well?\n>>\n>> I think .ignore_case is set iff '-i' is given, and .regflags has\n>> REG_ICASE only if '-i' is given and the user said she does not want\n>> literal string match (i.e. no '-F').\n>>\n>> So... can .regflags have REG_ICASE when .ignore_case is false?\n>\n> Yeah reg_icase is more like a subset of ignore_case. Ignore what I wrote.\n\nThe rule of the game can be updated to make whoever looks at\n.ignore_case currently instead look at .regflags & REG_ICASE; that\nway, we could remove .ignore_case and stop worrying about one is\nsubset of the other.\n\nIt is just it felt funny to be mucking with .regflags when the user\nexplicitly said -F, and that is why they are separate.\n"},{"id":"290699","messageId":"xmqqvb0pxjfi.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"CACsJy8BU0fDVR54hMpA6qVknj+QxWR9Z-i1gRgpaJ6hp+SB2xQ@mail.gmail.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-01T19:11:29Z","receivedAt":"2016-07-01T19:14:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> The rest looks good (after your comment fixup). I see you already have\n> all the changes in your SQUASH??? commit. Do you want me to resend or\n> you will just squash this in locally?\n\nSquashing in would need to redo this into a few relevant commits,\nso it won't be \"just squash this in locally\" I am afraid, but let me\ntry.\n\n"},{"id":"290703","messageId":"xmqqk2h5xi3q.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"xmqqvb0pxjfi.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-01T19:40:09Z","receivedAt":"2016-07-01T19:40:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Duy Nguyen writes:\n>\n>> The rest looks good (after your comment fixup). I see you already have\n>> all the changes in your SQUASH??? commit. Do you want me to resend or\n>> you will just squash this in locally?\n>\n> Squashing in would need to redo this into a few relevant commits,\n> so it won't be \"just squash this in locally\" I am afraid, but let me\n> try.\n\nOk, there was a miniscule conflicts but otherwise the squashed\nmaterial was all coming from a single step in the original, so\nI did so myself. Let's start merging the result to 'next' ;-)\n\nThanks.\n"},{"id":"290705","messageId":"xmqqbn2hxgve.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"xmqqk2h5xi3q.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-01T20:06:45Z","receivedAt":"2016-07-01T20:07:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Junio C Hamano writes:\n>\n>> Duy Nguyen writes:\n>>\n>>> The rest looks good (after your comment fixup). I see you already have\n>>> all the changes in your SQUASH??? commit. Do you want me to resend or\n>>> you will just squash this in locally?\n>>\n>> Squashing in would need to redo this into a few relevant commits,\n>> so it won't be \"just squash this in locally\" I am afraid, but let me\n>> try.\n>\n> Ok, there was a miniscule conflicts but otherwise the squashed\n> material was all coming from a single step in the original, so\n> I did so myself. Let's start merging the result to 'next' ;-)\n>\n> Thanks.\n\nIOW, this on top of the fixup we discussed.\n\n sideband.c | 9 +++------\n 1 file changed, 3 insertions(+), 6 deletions(-)\n\ndiff --git a/sideband.c b/sideband.c\nindex afa0136..2782df8 100644\n--- a/sideband.c\n+++ b/sideband.c\n@@ -62,10 +62,7 @@ int recv_sideband(const char *me, int in_stream, int out)\n \t\t\t *\n \t\t\t * The output is accumulated in a buffer and\n \t\t\t * each line is printed to stderr using\n-\t\t\t * fwrite(3). This is a \"best effort\"\n-\t\t\t * approach to support inter-process atomicity\n-\t\t\t * (single fwrite(3) call is likely to end up\n-\t\t\t * in single atomic write() system calls).\n+\t\t\t * write(2) to ensure inter-process atomicity.\n \t\t\t */\n \t\t\twhile ((brk = strpbrk(b, \"\\n\\r\"))) {\n \t\t\t\tint linelen = brk - b;\n@@ -78,7 +75,7 @@ int recv_sideband(const char *me, int in_stream, int out)\n \t\t\t\t} else {\n \t\t\t\t\tstrbuf_addf(&outbuf, \"%c\", *brk);\n \t\t\t\t}\n-\t\t\t\tfwrite(outbuf.buf, 1, outbuf.len, stderr);\n+\t\t\t\txwrite(2, outbuf.buf, outbuf.len);\n \t\t\t\tstrbuf_reset(&outbuf);\n \n \t\t\t\tb = brk + 1;\n@@ -101,7 +98,7 @@ int recv_sideband(const char *me, int in_stream, int out)\n \n \tif (outbuf.len) {\n \t\tstrbuf_addf(&outbuf, \"\\n\");\n-\t\tfwrite(outbuf.buf, 1, outbuf.len, stderr);\n+\t\txwrite(2, outbuf.buf, outbuf.len);\n \t}\n \tstrbuf_release(&outbuf);\n \treturn retval;\n"},{"id":"290706","messageId":"xmqq7fd5xgtx.fsf@gitster.mtv.corp.google.com","threadId":"42699","inReplyTo":"xmqqbn2hxgve.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 00/12] nd/icase updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-01T20:07:38Z","receivedAt":"2016-07-01T20:07:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> Ok, there was a miniscule conflicts but otherwise the squashed\n>> material was all coming from a single step in the original, so\n>> I did so myself. Let's start merging the result to 'next' ;-)\n>>\n>> Thanks.\n>\n> IOW, this on top of the fixup we discussed.\n>\n> sideband.c | 9 +++------\n\nNO, sorry, a wrong thread.\n"}]}