{"thread":{"id":"39783","subject":"Git grep does not support multi-byte characters (like UTF-8)","startedAt":"2015-07-06T11:28:16Z","lastAt":"2015-08-21T12:47:20Z","messageCount":53,"participants":["Plamen Totev","Duy Nguyen","Nguyễn Thái Ngọc Duy","René Scharfe","Junio C Hamano","Torsten Bögershausen","Eric Sunshine"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"265548","messageId":"2008630603.1189842.1436182096558.JavaMail.apache@nm33.abv.bg","threadId":"39783","inReplyTo":null,"subject":"Git grep does not support multi-byte characters (like UTF-8)","fromName":"Plamen Totev","fromEmail":"plamen.totev@abv.bg","sentAt":"2015-07-06T11:28:16Z","receivedAt":"2015-07-06T11:28:16Z","isPatch":false,"sender":{"key":"plamen.totev@abv.bg","avatar":null},"body":"Hello, \n\nIt looks like the git grep command does not support multi-byte character sets like UTF-8. As a result some of the grep functionality is not working. For example if you search for non Latin words the ignore case flag does not have effect(the search is case sensitive). I suspect there are some regular expressions that will not work as expected too. \n\nWhen I'm using the git from the shell I could use the GNU grep utility instead. But some tools like gitweb use git grep so they are not working properly with UTF-8 files with non Latin characters. \n\nMuch of the grep code seems to be copied from the GNU grep utility but the multi-byte support is left out. I just wondered what could be the reason. \n\nRegards, \nPlamen Totev\n"},{"id":"265550","messageId":"CACsJy8BH_QJss57uMJNE=ojNT5vBWKN=eEdrBBS38g6As-UH6A@mail.gmail.com","threadId":"39783","inReplyTo":"2008630603.1189842.1436182096558.JavaMail.apache@nm33.abv.bg","subject":"Re: Git grep does not support multi-byte characters (like UTF-8)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-06T12:23:52Z","receivedAt":"2015-07-06T12:23:52Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jul 6, 2015 at 6:28 PM, Plamen Totev wrote:\n> Hello,\n>\n> It looks like the git grep command does not support multi-byte character sets like UTF-8. As a result some of the grep functionality is not working. For example if you search for non Latin words the ignore case flag does not have effect(the search is case sensitive). I suspect there are some regular expressions that will not work as expected too.\n\nI think we over-optimized a bit. If you your system provides regex\nwith locale support (e.g. Linux) and you don't explicitly use fallback\nregex implementation, it should work. I suppose your test patterns\nlook \"fixed\" (i.e. no regex special characters)? Can you try just add\n\".\" and see if case insensitive matching works?\n-- \nDuy\n"},{"id":"265551","messageId":"1436186551-32544-1-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"2008630603.1189842.1436182096558.JavaMail.apache@nm33.abv.bg","subject":"[PATCH] grep: use regcomp() for icase search with non-ascii patterns","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-06T12:42:31Z","receivedAt":"2015-07-06T12:42:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Noticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 14 +++++++++++---\n 1 file changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex b58c7c6..48db15a 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -378,7 +378,7 @@ static void free_pcre_regexp(struct grep_pat *p)\n }\n #endif /* !USE_LIBPCRE */\n \n-static int is_fixed(const char *s, size_t len)\n+static int is_fixed(const char *s, size_t len, int ignore_icase)\n {\n \tsize_t i;\n \n@@ -391,6 +391,13 @@ static int is_fixed(const char *s, size_t len)\n \tfor (i = 0; i < len; i++) {\n \t\tif (is_regex_special(s[i]))\n \t\t\treturn 0;\n+\t\t/*\n+\t\t * The builtin substring search can only deal with case\n+\t\t * insensitivity in ascii range. If there is something outside\n+\t\t * of that range, fall back to regcomp.\n+\t\t */\n+\t\tif (ignore_icase && (unsigned char)s[i] >= 128)\n+\t\t\treturn 0;\n \t}\n \n \treturn 1;\n@@ -398,18 +405,19 @@ static int is_fixed(const char *s, size_t len)\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n+\tint ignore_icase = opt->regflags & REG_ICASE || p->ignore_case;\n \tint err;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \n-\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n+\tif (opt->fixed || is_fixed(p->pattern, p->patternlen, ignore_icase))\n \t\tp->fixed = 1;\n \telse\n \t\tp->fixed = 0;\n \n \tif (p->fixed) {\n-\t\tif (opt->regflags & REG_ICASE || p->ignore_case)\n+\t\tif (ignore_case)\n \t\t\tp->kws = kwsalloc(tolower_trans_tbl);\n \t\telse\n \t\t\tp->kws = kwsalloc(NULL);\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265623","messageId":"559AE0B9.2040704@web.de","threadId":"39783","inReplyTo":"1436186551-32544-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH] grep: use regcomp() for icase search with non-ascii patterns","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2015-07-06T20:10:33Z","receivedAt":"2015-07-06T20:10:33Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 06.07.2015 um 14:42 schrieb Nguyễn Thái Ngọc Duy:\n> Noticed-by: Plamen Totev \n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> grep.c | 14 +++++++++++---\n> 1 file changed, 11 insertions(+), 3 deletions(-)\n>\n> diff --git a/grep.c b/grep.c\n> index b58c7c6..48db15a 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -378,7 +378,7 @@ static void free_pcre_regexp(struct grep_pat *p)\n> }\n> #endif /* !USE_LIBPCRE */\n>\n> -static int is_fixed(const char *s, size_t len)\n> +static int is_fixed(const char *s, size_t len, int ignore_icase)\n> {\n> \tsize_t i;\n>\n> @@ -391,6 +391,13 @@ static int is_fixed(const char *s, size_t len)\n> \tfor (i = 0; i < len; i++) {\n> \t\tif (is_regex_special(s[i]))\n> \t\t\treturn 0;\n> +\t\t/*\n> +\t\t * The builtin substring search can only deal with case\n> +\t\t * insensitivity in ascii range. If there is something outside\n> +\t\t * of that range, fall back to regcomp.\n> +\t\t */\n> +\t\tif (ignore_icase && (unsigned char)s[i] >= 128)\n> +\t\t\treturn 0;\n\nHow about \"isascii(s[i])\"?\n\n> \t}\n>\n> \treturn 1;\n> @@ -398,18 +405,19 @@ static int is_fixed(const char *s, size_t len)\n>\n> static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n> {\n> +\tint ignore_icase = opt->regflags & REG_ICASE || p->ignore_case;\n> \tint err;\n>\n> \tp->word_regexp = opt->word_regexp;\n> \tp->ignore_case = opt->ignore_case;\n\nUsing p->ignore_case before this line, as in initialization of the new \nvariable ignore_icase above, changes the meaning.\n\n>\n> -\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n> +\tif (opt->fixed || is_fixed(p->pattern, p->patternlen, ignore_icase))\n> \t\tp->fixed = 1;\n> \telse\n> \t\tp->fixed = 0;\n>\n> \tif (p->fixed) {\n> -\t\tif (opt->regflags & REG_ICASE || p->ignore_case)\n> +\t\tif (ignore_case)\n\nignore_icase instead? ignore_case is for the config variable \ncore.ignorecase. Tricky.\n\n> \t\t\tp->kws = kwsalloc(tolower_trans_tbl);\n> \t\telse\n> \t\t\tp->kws = kwsalloc(NULL);\n>\n\nSo the optimization before this patch was that if a string was searched \nfor without -F then it would be treated as a fixed string anyway unless \nit contained regex special characters. Searching for fixed strings \nusing the kwset functions is faster than using regcomp and regexec, \nwhich makes the exercise worthwhile.\n\nYour patch disables the optimization if non-ASCII characters are \nsearched for because kwset handles case transformations only for ASCII \nchars.\n\nAnother consequence of this limitation is that -Fi (explicit \ncase-insensitive fixed-string search) doesn't work properly with \nnon-ASCII chars neither. How can we handle this one? Fall back to \nregcomp by escaping all special characters? Or at least warn?\n\nTests would be nice. :)\n\nRené\n"},{"id":"265635","messageId":"CACsJy8BDdxoN3m=FW9sCztz4Egjw7xnxXGZUkWQ3hAK9yXisnA@mail.gmail.com","threadId":"39783","inReplyTo":"559AE0B9.2040704@web.de","subject":"Re: [PATCH] grep: use regcomp() for icase search with non-ascii patterns","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-06T23:02:11Z","receivedAt":"2015-07-06T23:02:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 7, 2015 at 3:10 AM, René Scharfe wrote:\n> Am 06.07.2015 um 14:42 schrieb Nguyễn Thái Ngọc Duy:\n>>\n>> Noticed-by: Plamen Totev \n>> Signed-off-by: Nguyễn Thái Ngọc Duy \n>> ---\n>> grep.c | 14 +++++++++++---\n>> 1 file changed, 11 insertions(+), 3 deletions(-)\n>>\n>> diff --git a/grep.c b/grep.c\n>> index b58c7c6..48db15a 100644\n>> --- a/grep.c\n>> +++ b/grep.c\n>> @@ -378,7 +378,7 @@ static void free_pcre_regexp(struct grep_pat *p)\n>> }\n>> #endif /* !USE_LIBPCRE */\n>>\n>> -static int is_fixed(const char *s, size_t len)\n>> +static int is_fixed(const char *s, size_t len, int ignore_icase)\n>> {\n>> size_t i;\n>>\n>> @@ -391,6 +391,13 @@ static int is_fixed(const char *s, size_t len)\n>> for (i = 0; i < len; i++) {\n>> if (is_regex_special(s[i]))\n>> return 0;\n>> + /*\n>> + * The builtin substring search can only deal with case\n>> + * insensitivity in ascii range. If there is something\n>> outside\n>> + * of that range, fall back to regcomp.\n>> + */\n>> + if (ignore_icase && (unsigned char)s[i] >= 128)\n>> + return 0;\n>\n>\n> How about \"isascii(s[i])\"?\n\nYes, better.\n\n>\n>> }\n>>\n>> return 1;\n>> @@ -398,18 +405,19 @@ static int is_fixed(const char *s, size_t len)\n>>\n>> static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n>> {\n>> + int ignore_icase = opt->regflags & REG_ICASE || p->ignore_case;\n>> int err;\n>>\n>> p->word_regexp = opt->word_regexp;\n>> p->ignore_case = opt->ignore_case;\n>\n>\n> Using p->ignore_case before this line, as in initialization of the new\n> variable ignore_icase above, changes the meaning.\n\nOops.\n\n>> - if (opt->fixed || is_fixed(p->pattern, p->patternlen))\n>> + if (opt->fixed || is_fixed(p->pattern, p->patternlen,\n>> ignore_icase))\n>> p->fixed = 1;\n>> else\n>> p->fixed = 0;\n>>\n>> if (p->fixed) {\n>> - if (opt->regflags & REG_ICASE || p->ignore_case)\n>> + if (ignore_case)\n>\n>\n> ignore_icase instead? ignore_case is for the config variable\n> core.ignorecase. Tricky.\n\nMaybe we can test isascii separately and save the result in\nhas_non_ascii, then we can avoid ignore_(i)case\n\n>\n>> p->kws = kwsalloc(tolower_trans_tbl);\n>> else\n>> p->kws = kwsalloc(NULL);\n>>\n>\n> So the optimization before this patch was that if a string was searched for\n> without -F then it would be treated as a fixed string anyway unless it\n> contained regex special characters. Searching for fixed strings using the\n> kwset functions is faster than using regcomp and regexec, which makes the\n> exercise worthwhile.\n>\n> Your patch disables the optimization if non-ASCII characters are searched\n> for because kwset handles case transformations only for ASCII chars.\n>\n> Another consequence of this limitation is that -Fi (explicit\n> case-insensitive fixed-string search) doesn't work properly with non-ASCII\n> chars neither. How can we handle this one? Fall back to regcomp by\n> escaping all special characters? Or at least warn?\n\nHehe.. I noticed it too shortly after sending the patch. I was torn\nbetween simply documenting the limitation and waiting for the next\nperson to come and fix it, or quoting the regex then passing to\nregcomp. GNU grep does the quoting in this case, but that code is\nGPLv3 so we can't simply copy over. It could be a problem if we need\nto quote a regex in a multibyte charset where ascii is not a subset.\nBut i guess we can just go with utf-8..\n\n> Tests would be nice. :)\n\nYeah.. but we now rely on system regcomp which may behave differently\nacross platforms. Then we need some locale to be always there. Some\nplatforms (like Gentoo) even allow building glibc without i18n.. So\nI'm not sure how we know when to test or skip.\n-- \nDuy\n"},{"id":"265659","messageId":"775251698.1328032.1436259534851.JavaMail.apache@nm31.abv.bg","threadId":"39783","inReplyTo":"CACsJy8BH_QJss57uMJNE=ojNT5vBWKN=eEdrBBS38g6As-UH6A@mail.gmail.com","subject":"Re: Git grep does not support multi-byte characters (like UTF-8)","fromName":"Plamen Totev","fromEmail":"plamen.totev@abv.bg","sentAt":"2015-07-07T08:58:54Z","receivedAt":"2015-07-07T08:58:54Z","isPatch":false,"sender":{"key":"plamen.totev@abv.bg","avatar":null},"body":"Nguyen, thanks for the help and the patch. Also the escaping suggested by Scharfe seems as good choice. But i dig some more into the problem and I found some other thing. That's why I replied on the main thread not on the patch. I hope you'll excuse me if this is a bad practice.\n\ngit grep -i -P also does not works because the PCRE_UTF8 is not set and pcre library does not treat the string as UTF-8.\n\npickaxe search also uses kwsearch so the case insensitive search with it does not work (e.g. git log -i -S).  Maybe this is a less of a problem here as one is expected to search for exact string (hence knows the case)\n\nThere is a interesting corner case. is_fixed treats all patterns containing nulls as fixed. So what about if the string contains non-ASCII symbols as well as nulls and the search is case insensitive :) I have to admin that my knowledge in UTF-8 is not enough to answer the question if this could occur during normal usage. For example the second byte in multi-byte symbol is NULL. I would guess that's not true as it would break a lot of programs that depend on NULL delimited string but it's good if somebody could confirm.\n\nGNU grep indeed uses escaped regular expressions when the string is using multi-byte encoding and the search is case insensitive. If the encoding is UTF-8 then this strategy could be used in git too. Especially that git already have support and helper functions to work with UTF-8. As for the other multi-byte encodings - I think the things would become more complicated. As far I know in UTF-8 the '{' character for example is two bytes not one. Maybe really a support could be added only for the UTF-8 and if the string is not UTF-8 to issue a warning.\n\nSo maybe the following makes sense when a grep search is performed:\n* check if the multi-byte encoding is used. If it's and the search is case insensitive and the encoding is not UTF-8 give a warning;\n* if pcre is used and the string is UTF-8 encoded set the PCRE_UTF8 flag;\n* if the search is case insensitive, the string is fixed and the encoding  used is UTF-8 use regcomp instead of kwsearch and escape any regex special characters in the pattern;\n\nAnd the question with the behavior of pickaxe search remains open. Using kwset does not work with case insensitive non-ASCII searches. Instead of fixing grep.c maybe it's better if new function is introduced that performs keyword searches so it could be used by both grep, diffcore-pickaxe and any other code in the future that may require such functionality. Or maybe diffcore-pickaxe should use grep instead of directly kwset/regcomp\n\nRegards,\nPlamen Totev\n\n\n\n>-------- Оригинално писмо -------- \n>От: Duy Nguyen pclouds@gmail.com \n>Относно: Re: Git grep does not support multi-byte characters (like UTF-8) \n>До: Plamen Totev \n>Изпратено на: 06.07.2015 15:23 \n\n> I think we over-optimized a bit. If you your system provides regex \n> with locale support (e.g. Linux) and you don't explicitly use fallback \n> regex implementation, it should work. I suppose your test patterns \n> look \"fixed\" (i.e. no regex special characters)? Can you try just add \n> \".\" and see if case insensitive matching works? \n\nThis is indeed the problem. When I added the \".\" the matching works just fine.\n"},{"id":"265665","messageId":"CACsJy8A035JsR-z34g56QWLVkN0Tg0tOqmrT+Hp6ecsVBhm25g@mail.gmail.com","threadId":"39783","inReplyTo":"775251698.1328032.1436259534851.JavaMail.apache@nm31.abv.bg","subject":"Re: Git grep does not support multi-byte characters (like UTF-8)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-07T12:22:28Z","receivedAt":"2015-07-07T12:22:28Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 7, 2015 at 3:58 PM, Plamen Totev wrote:\n> Nguyen, thanks for the help and the patch. Also the escaping suggested by Scharfe seems as good choice. But i dig some more into the problem and I found some other thing. That's why I replied on the main thread not on the patch. I hope you'll excuse me if this is a bad practice.\n\nSo far this is very good reporting. I can't complain :)\n\n> git grep -i -P also does not works because the PCRE_UTF8 is not set and pcre library does not treat the string as UTF-8.\n\nWe do prefer utf-8, but i don't know if we can assume utf-8 everywhere\nyet. I guess it's ok in this case.\n\n> pickaxe search also uses kwsearch so the case insensitive search with it does not work (e.g. git log -i -S). Maybe this is a less of a problem here as one is expected to search for exact string (hence knows the case)\n\nWill fix (i'm close to being done with git-grep, not counting the pcre\nbug you just found)\n\n> There is a interesting corner case. is_fixed treats all patterns containing nulls as fixed. So what about if the string contains non-ASCII symbols as well as nulls and the search is case insensitive :) I have to admin that my knowledge in UTF-8 is not enough to answer the question if this could occur during normal usage. For example the second byte in multi-byte symbol is NULL. I would guess that's not true as it would break a lot of programs that depend on NULL delimited string but it's good if somebody could confirm.\n\nFor utf-8, if NUL occurs in a byte stream, it must be ASCII NUL, not\npart of any multibyte character. Utf-8 is really well tuned for C\nstrings.\n\n> GNU grep indeed uses escaped regular expressions when the string is using multi-byte encoding and the search is case insensitive. If the encoding is UTF-8 then this strategy could be used in git too. Especially that git already have support and helper functions to work with UTF-8. As for the other multi-byte encodings - I think the things would become more complicated. As far I know in UTF-8 the '{' character for example is two bytes not one. Maybe really a support could be added only for the UTF-8 and if the string is not UTF-8 to issue a warning.\n\nIn the worst case we could reuse the trick we do elsewhere in git:\nconvert to utf-8 with iconv, do whatever we need to (escaping...) then\nconvert back before feeding it to regcomp and friends.\n\n> So maybe the following makes sense when a grep search is performed:\n> * check if the multi-byte encoding is used. If it's and the search is case insensitive and the encoding is not UTF-8 give a warning;\n> * if pcre is used and the string is UTF-8 encoded set the PCRE_UTF8 flag;\n> * if the search is case insensitive, the string is fixed and the encoding used is UTF-8 use regcomp instead of kwsearch and escape any regex special characters in the pattern;\n>\n> And the question with the behavior of pickaxe search remains open. Using kwset does not work with case insensitive non-ASCII searches. Instead of fixing grep.c maybe it's better if new function is introduced that performs keyword searches so it could be used by both grep, diffcore-pickaxe and any other code in the future that may require such functionality. Or maybe diffcore-pickaxe should use grep instead of directly kwset/regcomp\n\nThat would function be called \"grep\". More refactor would be needed.\n\"git grep regcomp\" reveals some many places. Many some of them would\nbenefit from kws if we provide this new function you mentioned.\n\n> Regards,\n> Plamen Totev\n>\n>\n>\n>>-------- Оригинално писмо --------\n>>От: Duy Nguyen pclouds@gmail.com\n>>Относно: Re: Git grep does not support multi-byte characters (like UTF-8)\n>>До: Plamen Totev \n>>Изпратено на: 06.07.2015 15:23\n>\n>> I think we over-optimized a bit. If you your system provides regex\n>> with locale support (e.g. Linux) and you don't explicitly use fallback\n>> regex implementation, it should work. I suppose your test patterns\n>> look \"fixed\" (i.e. no regex special characters)? Can you try just add\n>> \".\" and see if case insensitive matching works?\n>\n> This is indeed the problem. When I added the \".\" the matching works just fine.\n\n\n\n-- \nDuy\n"},{"id":"265731","messageId":"790424569.1382943.1436279158871.JavaMail.apache@nm31.abv.bg","threadId":"39783","inReplyTo":"CACsJy8BDdxoN3m=FW9sCztz4Egjw7xnxXGZUkWQ3hAK9yXisnA@mail.gmail.com","subject":"Re: [PATCH] grep: use regcomp() for icase search with non-ascii patterns","fromName":"Plamen Totev","fromEmail":"plamen.totev@abv.bg","sentAt":"2015-07-07T14:25:58Z","receivedAt":"2015-07-07T14:25:58Z","isPatch":true,"sender":{"key":"plamen.totev@abv.bg","avatar":null},"body":"\nOn 07.07. 2015 at 02:02, Duy Nguyen  wrote: \n> On Tue, Jul 7, 2015 at 3:10 AM, René Scharfe wrote: \n> > Am 06.07.2015 um 14:42 schrieb Nguyễn Thái Ngọc Duy: \n\n> > So the optimization before this patch was that if a string was searched for \n> > without -F then it would be treated as a fixed string anyway unless it \n> > contained regex special characters. Searching for fixed strings using the \n> > kwset functions is faster than using regcomp and regexec, which makes the \n> > exercise worthwhile. \n> > \n> > Your patch disables the optimization if non-ASCII characters are searched \n> > for because kwset handles case transformations only for ASCII chars. \n> > \n> > Another consequence of this limitation is that -Fi (explicit \n> > case-insensitive fixed-string search) doesn't work properly with non-ASCII \n> > chars neither. How can we handle this one? Fall back to regcomp by \n> > escaping all special characters? Or at least warn? \n> \n> Hehe.. I noticed it too shortly after sending the patch. I was torn \n> between simply documenting the limitation and waiting for the next \n> person to come and fix it, or quoting the regex then passing to \n> regcomp. GNU grep does the quoting in this case, but that code is \n> GPLv3 so we can't simply copy over. It could be a problem if we need \n> to quote a regex in a multibyte charset where ascii is not a subset. \n> But i guess we can just go with utf-8.. \n\nI played a little bit with the code and I came up with this function to escape\nregular expressions in  utf-8. Hope it helps.\n\nstatic void escape_regexp(const char *pattern, size_t len,\n                char **new_pattern, size_t *new_len)\n{\n        const char *p = pattern;\n        char *np = *new_pattern = xmalloc(2 * len);\n        int chrlen;\n        *new_len = len;\n\n        while (len) {\n                chrlen = mbs_chrlen(&p, &len, \"utf-8\");\n                if (chrlen == 1 && is_regex_special(*pattern))\n                        *np++ = '\\\\';\n\n                memcpy(np, pattern, chrlen);\n                np += chrlen;\n                pattern = p;\n        }\n\n        *new_len = np - *new_pattern;\n}\n"},{"id":"265743","messageId":"xmqqr3ok3qad.fsf@gitster.dls.corp.google.com","threadId":"39783","inReplyTo":"775251698.1328032.1436259534851.JavaMail.apache@nm31.abv.bg","subject":"Re: Git grep does not support multi-byte characters (like UTF-8)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-07-07T16:07:06Z","receivedAt":"2015-07-07T16:07:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Plamen Totev writes:\n\n> pickaxe search also uses kwsearch so the case insensitive search with\n> it does not work (e.g. git log -i -S). Maybe this is a less of a\n> problem here as one is expected to search for exact string (hence\n> knows the case)\n\nYou reasoned correctly, I think. Pickaxe, as one of the building\nblocks to implement Linus's ultimate change tracking tool [*1*],\nshould never pay attention to \"-i\". It is a step to finding the\ncommit that touches the exact code block given (i.e. \"how do you\ndrill down?\" part of $gmane/217 message).\n\nThanks.\n\n[Footnote]\n*1* http://article.gmane.org/gmane.comp.version-control.git/217\n"},{"id":"265757","messageId":"663318203.435786.1436292501411.JavaMail.apache@nm32.abv.bg","threadId":"39783","inReplyTo":"xmqqr3ok3qad.fsf@gitster.dls.corp.google.com","subject":"Re: Git grep does not support multi-byte characters (like UTF-8)","fromName":"Plamen Totev","fromEmail":"plamen.totev@abv.bg","sentAt":"2015-07-07T18:08:21Z","receivedAt":"2015-07-07T18:08:21Z","isPatch":false,"sender":{"key":"plamen.totev@abv.bg","avatar":null},"body":"Junio C Hamano writes:\n\n> Plamen Totev writes: \n> \n> > pickaxe search also uses kwsearch so the case insensitive search with \n> > it does not work (e.g. git log -i -S). Maybe this is a less of a \n> > problem here as one is expected to search for exact string (hence \n> > knows the case) \n> \n> You reasoned correctly, I think. Pickaxe, as one of the building \n> blocks to implement Linus's ultimate change tracking tool [*1*], \n> should never pay attention to \"-i\". It is a step to finding the \n> commit that touches the exact code block given (i.e. \"how do you \n> drill down?\" part of $gmane/217 message). \n> \n> Thanks. \n> \n> [Footnote] \n> *1* http://article.gmane.org/gmane.comp.version-control.git/217\n\nNow that I read the link again and gave the matter a thought I'm not so sure.\nIn some contexts the case of the words does not matter. In SQL for example.\nLet's consider a SQL script file that contains the following line:\n\nselect name, address from customers;\n\nAt some point we decide to change the coding style to:\n\nSELECT name, address FROM customers;\n\nWhat should pickaxe search return - the first commit where the line is introduced\nor the commit with the refactoring? From this point of view I think the -i switch makes sense.\nThe SQL is not the only case insensitive language - BASIC and Pascal come into my mind \n(those two I was using while I was in the high school :)).\n\nAlso I think it makes sense (maybe even more?) for natural languages.\nFor example after editing a text a sentence could be split into two.\nThen the first word of the second sentence may change its case.\nOf course the natural languages always  complicate the things a bit.\nAn ultimate tracking tools should be able to handle typo fixes, punctuation changes, etc.\n\nBut I'm getting a bit off-topic. What I wanted to say is that in some contexts it makes sense\n(at least to me) to have case insensitive pickaxe search.\nOr I'm missing something and there is a better tools to use is such cases?\n\nRegards,\nPlamen Totev\n"},{"id":"265799","messageId":"CACsJy8De6Wt4J5ZFx5rEg2eRBt=7PsLim=31FvtLEKsX-6SMyQ@mail.gmail.com","threadId":"39783","inReplyTo":"663318203.435786.1436292501411.JavaMail.apache@nm32.abv.bg","subject":"Re: Git grep does not support multi-byte characters (like UTF-8)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T02:19:00Z","receivedAt":"2015-07-08T02:19:00Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 8, 2015 at 1:08 AM, Plamen Totev wrote:\n> Junio C Hamano writes:\n>\n>> Plamen Totev writes:\n>>\n>> > pickaxe search also uses kwsearch so the case insensitive search with\n>> > it does not work (e.g. git log -i -S). Maybe this is a less of a\n>> > problem here as one is expected to search for exact string (hence\n>> > knows the case)\n>>\n>> You reasoned correctly, I think. Pickaxe, as one of the building\n>> blocks to implement Linus's ultimate change tracking tool [*1*],\n>> should never pay attention to \"-i\". It is a step to finding the\n>> commit that touches the exact code block given (i.e. \"how do you\n>> drill down?\" part of $gmane/217 message).\n>>\n>> Thanks.\n>>\n>> [Footnote]\n>> *1* http://article.gmane.org/gmane.comp.version-control.git/217\n>\n> Now that I read the link again and gave the matter a thought I'm not so sure.\n> In some contexts the case of the words does not matter. In SQL for example.\n> Let's consider a SQL script file that contains the following line:\n>\n> select name, address from customers;\n>\n> At some point we decide to change the coding style to:\n>\n> SELECT name, address FROM customers;\n\nOn top of this, pickaxe already supports icase even kws is used. But\nit only works for ascii, so either we fix it and support non-ascii, or\nwe remove icase support entirely from diffcore_pickaxe(). I vote the\nformer.\n-- \nDuy\n"},{"id":"265804","messageId":"xmqq1tgj1ca9.fsf@gitster.dls.corp.google.com","threadId":"39783","inReplyTo":"CACsJy8De6Wt4J5ZFx5rEg2eRBt=7PsLim=31FvtLEKsX-6SMyQ@mail.gmail.com","subject":"Re: Git grep does not support multi-byte characters (like UTF-8)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-07-08T04:52:30Z","receivedAt":"2015-07-08T04:52:30Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On top of this, pickaxe already supports icase even kws is used. But\n> it only works for ascii, so either we fix it and support non-ascii, or\n> we remove icase support entirely from diffcore_pickaxe(). I vote the\n> former.\n\nI think that is a different issue. The pickaxe has a single very\nnarrowly-defined intended use case [*1*] and I do not care too much\nhow any use that is outside the intended use case behaves. As long\nas its intended use case does not suffer (1) correctness-wise, (2)\nperformance-wise and (3) code-cleanliness-wise, due to changes to\nsupport such enhancements, I am perfectly fine.\n\nAscii-only icase match is one example of a feature that is outside\nthe intended use case, and implementation of it using kws is nearly\nfree if I am not mistaken, not making the primary use case suffer in\nany way.\n\nI however am highly skeptical that the same thing can be done with\nnon-ascii icase. As long as it can be added without makinng the\nprimary use case suffer in any way, I do not mind it very much.\n\nThanks.\n\n\n[Footnote]\n\n*1* The requirement is very simple. You get a string that is unique\nin a blob that exists at the revision your traversal begins, and you\nwant to find the point where the blob at the corresponding path does\nnot have that exact string with minimal effort. You do not need to\nensure that the input string is unique (it is a user error and the\nbehaviour is undefined) and for simplicity you are also allowed to\nfire when the blob has more than one copies of the string (even\nthough the expected use is to find the place where the blob has\nzero).\n\nAny other cases, e.g. the string was not unique in the blob, the\nuser specified \"ignore-case\" and other irrelevant options, are\nallowed to be incorrect or slow or both, as $gmane/217 does not need\nsuch uses to implement it ;-)\n"},{"id":"265827","messageId":"1436351919-2520-1-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436186551-32544-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 0/9] icase match on non-ascii","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:30Z","receivedAt":"2015-07-08T10:38:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This series fix case insensitive matching on non-ascii charsets.\n\"grep -i\", \"grep -F -i\", \"grep --pcre-regexp -i\" and \"log -i -S\" are\nfixed.\n\nSide note, I almost added the third has_non_ascii() function. Maybe we\nshould consider merging the two existing has_non_ascii() functions\nback, or rename one to something else.\n\nPatch 5 is \"funny\". The patch itself is in iso-8859-1, but my name in\nthe commit message is in utf-8. Let's see how it goes on the apply\nside. We may need to fix something in this area..\n\nNguyễn Thái Ngọc Duy (9):\n grep: allow -F -i combination\n grep: break down an \"if\" stmt in preparation for next changes\n grep/icase: avoid kwsset on literal non-ascii strings\n grep/icase: avoid kwsset when -F is specified\n grep/pcre: prepare locale-dependent tables for icase matching\n gettext: add is_utf8_locale()\n grep/pcre: support utf-8\n diffcore-pickaxe: \"share\" regex error handling code\n diffcore-pickaxe: support case insensitive match on non-ascii\n\n builtin/grep.c | 2 +-\n diffcore-pickaxe.c | 27 ++++++++++-----\n gettext.c | 7 +++-\n gettext.h | 5 +++\n grep.c | 43 ++++++++++++++++++++---\n grep.h | 1 +\n quote.c | 37 ++++++++++++++++++++\n quote.h | 1 +\n t/t7812-grep-icase-non-ascii.sh (new +x) | 58 ++++++++++++++++++++++++++++++++\n t/t7813-grep-icase-non-ascii.sh (new +x) | 19 +++++++++++\n 10 files changed, 186 insertions(+), 14 deletions(-)\n create mode 100755 t/t7812-grep-icase-non-ascii.sh\n create mode 100755 t/t7813-grep-icase-non-ascii.sh\n\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265825","messageId":"1436351919-2520-2-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 1/9] grep: allow -F -i combination","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:31Z","receivedAt":"2015-07-08T10:38:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"-F means \"no regex\", not \"case sensitive\" so it should not override -i\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/grep.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex d04f440..2d392e9 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -806,7 +806,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (!opt.pattern_list)\n \t\tdie(_(\"no pattern given.\"));\n-\tif (!opt.fixed && opt.ignore_case)\n+\tif (opt.ignore_case)\n \t\topt.regflags |= REG_ICASE;\n \n \tcompile_grep_patterns(&opt);\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265826","messageId":"1436351919-2520-3-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 2/9] grep: break down an \"if\" stmt in preparation for next changes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:32Z","receivedAt":"2015-07-08T10:38:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex b58c7c6..bd32f66 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -403,9 +403,11 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \n-\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n+\tif (is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n-\telse\n+\telse if (opt->fixed) {\n+\t\tp->fixed = 1;\n+\t} else\n \t\tp->fixed = 0;\n \n \tif (p->fixed) {\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265829","messageId":"1436351919-2520-4-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 3/9] grep/icase: avoid kwsset on literal non-ascii strings","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:33Z","receivedAt":"2015-07-08T10:38:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When we detect the pattern is just a literal string, we avoid heavy\nregex engine and use fast substring search implemented in kwsset.c.\nBut kws uses git-ctype which is locale-independent so it does not know\nhow to fold case properly outside ascii range. Let regcomp or pcre\ntake care of this case instead. Slower, but accurate.\n\nHelped-by: René Scharfe \nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 7 ++++++-\n t/t7812-grep-icase-non-ascii.sh (new +x) | 19 +++++++++++++++++++\n 2 files changed, 25 insertions(+), 1 deletion(-)\n create mode 100755 t/t7812-grep-icase-non-ascii.sh\n\ndiff --git a/grep.c b/grep.c\nindex bd32f66..d795b0e 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -4,6 +4,7 @@\n #include \"xdiff-interface.h\"\n #include \"diff.h\"\n #include \"diffcore.h\"\n+#include \"commit.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -398,12 +399,16 @@ static int is_fixed(const char *s, size_t len)\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n+\tint icase_non_ascii;\n \tint err;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n+\ticase_non_ascii =\n+\t\t(opt->regflags & REG_ICASE || p->ignore_case) &&\n+\t\thas_non_ascii(p->pattern);\n \n-\tif (is_fixed(p->pattern, p->patternlen))\n+\tif (!icase_non_ascii && is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse if (opt->fixed) {\n \t\tp->fixed = 1;\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nnew file mode 100755\nindex 0000000..63a2630\n--- /dev/null\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -0,0 +1,19 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_LOCALE 'setup' '\n+\tprintf \"TILRAUN: Halló Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_expect_success GETTEXT_LOCALE 'grep literal string, no -F' '\n+\tgit grep -i \"TILRAUN: Halló Heimur!\" &&\n+\tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265828","messageId":"1436351919-2520-5-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 4/9] grep/icase: avoid kwsset when -F is specified","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:34Z","receivedAt":"2015-07-08T10:38:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the previous commit, we can't use kws on icase search\noutside ascii range. But we can't simply pass the pattern to\nregcomp/pcre like the previous commit because it may contain regex\nspecial characters, so we need to quote the regex first.\n\nTo avoid misquote traps that could lead to undefined behavior, we\nalways stick to basic regex engine in this case. We don't need fancy\nfeatures for grepping a literal string anyway.\n\nbasic_regex_quote_buf() assumes that if the pattern is in a multibyte\nencoding, ascii chars must be unambiguously encoded as single\nbytes. This is true at least for UTF-8. For others, let's wait until\npeople yell up. Chances are nobody uses multibyte, non utf-8 charsets\nany more..\n\nHelped-by: René Scharfe \nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 25 ++++++++++++++++++++++++-\n quote.c | 37 +++++++++++++++++++++++++++++++++++++\n quote.h | 1 +\n t/t7812-grep-icase-non-ascii.sh | 26 ++++++++++++++++++++++++++\n 4 files changed, 88 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex d795b0e..8fce54f 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -5,6 +5,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"commit.h\"\n+#include \"quote.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -397,6 +398,24 @@ static int is_fixed(const char *s, size_t len)\n \treturn 1;\n }\n \n+static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint err;\n+\n+\tbasic_regex_quote_buf(&sb, p->pattern);\n+\terr = regcomp(&p->regexp, sb.buf, opt->regflags & ~REG_EXTENDED);\n+\tif (opt->debug)\n+\t\tfprintf(stderr, \"fixed%s\\n\", sb.buf);\n+\tstrbuf_release(&sb);\n+\tif (err) {\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, &p->regexp, errbuf, 1024);\n+\t\tregfree(&p->regexp);\n+\t\tcompile_regexp_failed(p, errbuf);\n+\t}\n+}\n+\n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tint icase_non_ascii;\n@@ -411,7 +430,11 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tif (!icase_non_ascii && is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse if (opt->fixed) {\n-\t\tp->fixed = 1;\n+\t\tp->fixed = !icase_non_ascii;\n+\t\tif (!p->fixed) {\n+\t\t\tcompile_fixed_regexp(p, opt);\n+\t\t\treturn;\n+\t\t}\n \t} else\n \t\tp->fixed = 0;\n \ndiff --git a/quote.c b/quote.c\nindex 7920e18..43a8057 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -439,3 +439,40 @@ void tcl_quote_buf(struct strbuf *sb, const char *src)\n \t}\n \tstrbuf_addch(sb, '\"');\n }\n+\n+void basic_regex_quote_buf(struct strbuf *sb, const char *src)\n+{\n+\tchar c;\n+\n+\tif (*src == '^') {\n+\t\t/* only beginning '^' is special and needs quoting */\n+\t\tstrbuf_addch(sb, '\\\\');\n+\t\tstrbuf_addch(sb, *src++);\n+\t}\n+\tif (*src == '*')\n+\t\t/* beginning '*' is not special, no quoting */\n+\t\tstrbuf_addch(sb, *src++);\n+\n+\twhile ((c = *src++)) {\n+\t\tswitch (c) {\n+\t\tcase '[':\n+\t\tcase '.':\n+\t\tcase '\\\\':\n+\t\tcase '*':\n+\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tcase '$':\n+\t\t\t/* only the end '$' is special and needs quoting */\n+\t\t\tif (*src == '\\0')\n+\t\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\ndiff --git a/quote.h b/quote.h\nindex 99e04d3..362d315 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -67,5 +67,6 @@ extern char *quote_path_relative(const char *in, const char *prefix,\n extern void perl_quote_buf(struct strbuf *sb, const char *src);\n extern void python_quote_buf(struct strbuf *sb, const char *src);\n extern void tcl_quote_buf(struct strbuf *sb, const char *src);\n+extern void basic_regex_quote_buf(struct strbuf *sb, const char *src);\n \n #endif\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 63a2630..c945589 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -16,4 +16,30 @@ test_expect_success GETTEXT_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success GETTEXT_LOCALE 'grep literal string, with -F' '\n+\tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\techo \"fixedTILRAUN: Halló Heimur!\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"TILRAUN: HALLÓ HEIMUR!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\techo \"fixedTILRAUN: HALLÓ HEIMUR!\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n+test_expect_success GETTEXT_LOCALE 'grep string with regex, with -F' '\n+\tprintf \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\techo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\HALLÓ \\$HE[]IMUR!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\techo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265831","messageId":"1436351919-2520-6-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 5/9] grep/pcre: prepare locale-dependent tables for icase matching","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:35Z","receivedAt":"2015-07-08T10:38:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The default tables are usually built with C locale and only suitable\nfor LANG=C or similar. This should make case insensitive search work\ncorrectly for all single-byte charsets.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 7 +++++--\n grep.h | 1 +\n t/t7813-grep-icase-non-ascii.sh (new +x) | 19 +++++++++++++++++++\n 3 files changed, 25 insertions(+), 2 deletions(-)\n create mode 100755 t/t7813-grep-icase-non-ascii.sh\n\ndiff --git a/grep.c b/grep.c\nindex 8fce54f..c79aa70 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -324,11 +324,13 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \tint erroffset;\n \tint options = PCRE_MULTILINE;\n \n-\tif (opt->ignore_case)\n+\tif (opt->ignore_case) {\n+\t\tp->pcre_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n+\t}\n \n \tp->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n-\t\t\tNULL);\n+\t\t\t\t p->pcre_tables);\n \tif (!p->pcre_regexp)\n \t\tcompile_regexp_failed(p, error);\n \n@@ -362,6 +364,7 @@ static void free_pcre_regexp(struct grep_pat *p)\n {\n \tpcre_free(p->pcre_regexp);\n \tpcre_free(p->pcre_extra_info);\n+\tpcre_free((void *)p->pcre_tables);\n }\n #else /* !USE_LIBPCRE */\n static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\ndiff --git a/grep.h b/grep.h\nindex 95f197a..cee4357 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -48,6 +48,7 @@ struct grep_pat {\n \tregex_t regexp;\n \tpcre *pcre_regexp;\n \tpcre_extra *pcre_extra_info;\n+\tconst unsigned char *pcre_tables;\n \tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\ndiff --git a/t/t7813-grep-icase-non-ascii.sh b/t/t7813-grep-icase-non-ascii.sh\nnew file mode 100755\nindex 0000000..efef7fb\n--- /dev/null\n+++ b/t/t7813-grep-icase-non-ascii.sh\n@@ -0,0 +1,19 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_ISO_LOCALE 'setup' '\n+\tprintf \"TILRAUN: Hall� Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_iso_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_expect_success GETTEXT_ISO_LOCALE,LIBPCRE 'grep pcre string' '\n+\tgit grep --perl-regexp -i \"TILRAUN: H.ll� Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LL� HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265830","messageId":"1436351919-2520-7-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 6/9] gettext: add is_utf8_locale()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:36Z","receivedAt":"2015-07-08T10:38:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This function returns true if git is running under an UTF-8\nlocale. pcre in the next patch will need this.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n gettext.c | 7 ++++++-\n gettext.h | 5 +++++\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/gettext.c b/gettext.c\nindex 7378ba2..601bc80 100644\n--- a/gettext.c\n+++ b/gettext.c\n@@ -166,12 +166,17 @@ void git_setup_gettext(void)\n \ttextdomain(\"git\");\n }\n \n+int is_utf8_locale(void)\n+{\n+\treturn !strcmp(charset, \"UTF-8\");\n+}\n+\n /* return the number of columns of string 's' in current locale */\n int gettext_width(const char *s)\n {\n \tstatic int is_utf8 = -1;\n \tif (is_utf8 == -1)\n-\t\tis_utf8 = !strcmp(charset, \"UTF-8\");\n+\t\tis_utf8 = is_utf8_locale();\n \n \treturn is_utf8 ? utf8_strwidth(s) : strlen(s);\n }\ndiff --git a/gettext.h b/gettext.h\nindex 33696a4..5e733d4 100644\n--- a/gettext.h\n+++ b/gettext.h\n@@ -31,6 +31,7 @@\n #ifndef NO_GETTEXT\n extern void git_setup_gettext(void);\n extern int gettext_width(const char *s);\n+extern int is_utf8_locale(void);\n #else\n static inline void git_setup_gettext(void)\n {\n@@ -39,6 +40,10 @@ static inline int gettext_width(const char *s)\n {\n \treturn strlen(s);\n }\n+static inline int is_utf8_locale(void)\n+{\n+\treturn 0;\n+}\n #endif\n \n #ifdef GETTEXT_POISON\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265832","messageId":"1436351919-2520-8-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 7/9] grep/pcre: support utf-8","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:37Z","receivedAt":"2015-07-08T10:38:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In the previous change in this function, we add locale support for\nsingle-byte encodings only. It looks like pcre only supports utf-* as\nmultibyte encodings, the others are left in the cold (which is\nfine). We need to enable PCRE_UTF8 so pcre can parse the string\ncorrectly before folding case.\n\nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 2 ++\n t/t7812-grep-icase-non-ascii.sh | 6 ++++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/grep.c b/grep.c\nindex c79aa70..7c9e437 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -326,6 +326,8 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \n \tif (opt->ignore_case) {\n \t\tp->pcre_tables = pcre_maketables();\n+\t\tif (is_utf8_locale())\n+\t\t\toptions |= PCRE_UTF8;\n \t\toptions |= PCRE_CASELESS;\n \t}\n \ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex c945589..1306cc0 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -16,6 +16,12 @@ test_expect_success GETTEXT_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre string' '\n+\tgit grep --perl-regexp \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LLÓ HEIMUR!\"\n+'\n+\n test_expect_success GETTEXT_LOCALE 'grep literal string, with -F' '\n \tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265833","messageId":"1436351919-2520-9-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 8/9] diffcore-pickaxe: \"share\" regex error handling code","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:38Z","receivedAt":"2015-07-08T10:38:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"There's another regcomp code block coming in this function. By moving\nthe error handling code out of this block, we don't have to add the\nsame error handling code in the new block.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n diffcore-pickaxe.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 185f86b..7a718fc 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -204,20 +204,13 @@ void diffcore_pickaxe(struct diff_options *o)\n \tint opts = o->pickaxe_opts;\n \tregex_t regex, *regexp = NULL;\n \tkwset_t kws = NULL;\n+\tint err = 0;\n \n \tif (opts & (DIFF_PICKAXE_REGEX | DIFF_PICKAXE_KIND_G)) {\n-\t\tint err;\n \t\tint cflags = REG_EXTENDED | REG_NEWLINE;\n \t\tif (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE))\n \t\t\tcflags |= REG_ICASE;\n \t\terr = regcomp(®ex, needle, cflags);\n-\t\tif (err) {\n-\t\t\t/* The POSIX.2 people are surely sick */\n-\t\t\tchar errbuf[1024];\n-\t\t\tregerror(err, ®ex, errbuf, 1024);\n-\t\t\tregfree(®ex);\n-\t\t\tdie(\"invalid regex: %s\", errbuf);\n-\t\t}\n \t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n@@ -225,6 +218,13 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\tkwsincr(kws, needle, strlen(needle));\n \t\tkwsprep(kws);\n \t}\n+\tif (err) {\n+\t\t/* The POSIX.2 people are surely sick */\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, ®ex, errbuf, 1024);\n+\t\tregfree(®ex);\n+\t\tdie(\"invalid regex: %s\", errbuf);\n+\t}\n \n \t/* Might want to warn when both S and G are on; I don't care... */\n \tpickaxe(&diff_queued_diff, o, regexp, kws,\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265834","messageId":"1436351919-2520-10-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 9/9] diffcore-pickaxe: support case insensitive match on non-ascii","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T10:38:39Z","receivedAt":"2015-07-08T10:38:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the \"grep -F -i\" case, we can't use kws on icase search\noutside ascii range, quote we quote the string and pass it to regcomp\nas a basic regexp and let regex engine deal with case sensitivity.\n\nThe new test is put in t7812 instead of t4209-log-pickaxe because\nlib-gettext.sh might cause problems elsewhere, probably..\n\nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n diffcore-pickaxe.c | 11 +++++++++++\n t/t7812-grep-icase-non-ascii.sh | 7 +++++++\n 2 files changed, 18 insertions(+)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 7a718fc..6946c15 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -7,6 +7,8 @@\n #include \"diffcore.h\"\n #include \"xdiff-interface.h\"\n #include \"kwset.h\"\n+#include \"commit.h\"\n+#include \"quote.h\"\n \n typedef int (*pickaxe_fn)(mmfile_t *one, mmfile_t *two,\n \t\t\t struct diff_options *o,\n@@ -212,6 +214,15 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\t\tcflags |= REG_ICASE;\n \t\terr = regcomp(®ex, needle, cflags);\n \t\tregexp = ®ex;\n+\t} else if (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE) &&\n+\t\t has_non_ascii(needle)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tint cflags = REG_NEWLINE | REG_ICASE;\n+\n+\t\tbasic_regex_quote_buf(&sb, needle);\n+\t\terr = regcomp(®ex, sb.buf, cflags);\n+\t\tstrbuf_release(&sb);\n+\t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n \t\t\t ? tolower_trans_tbl : NULL);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 1306cc0..20a39d0 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -48,4 +48,11 @@ test_expect_success GETTEXT_LOCALE 'grep string with regex, with -F' '\n \ttest_cmp expect2 debug2\n '\n \n+test_expect_success GETTEXT_LOCALE 'pickaxe -i on non-ascii' '\n+\tgit commit -m first &&\n+\tgit log --format=%f -i -S\"TILRAUN: HALLÓ HEIMUR!\" >actual &&\n+\techo first >expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"265836","messageId":"CACsJy8AiuiY=scqaH57wYVk4sWvvb0ddJzJGs+iaRj9ouHfCqw@mail.gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-6-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v2 5/9] grep/pcre: prepare locale-dependent tables for icase matching","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T11:00:34Z","receivedAt":"2015-07-08T11:00:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 8, 2015 at 5:38 PM, Nguyễn Thái Ngọc Duy wrote:\n> diff --git a/grep.c b/grep.c\n> index 8fce54f..c79aa70 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -324,11 +324,13 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n> int erroffset;\n> int options = PCRE_MULTILINE;\n>\n> - if (opt->ignore_case)\n> + if (opt->ignore_case) {\n> + p->pcre_tables = pcre_maketables();\n\nThis affects ascii-only case too because I didn't use has_non_ascii()\nhere. I guess it's ok because the doc says there are default tables\nanyway. But I'm not really familiar with pcre... If they have\noptimizations for when the last argument of pcre_compile() is NULL, we\nhave a regression.\n\n> options |= PCRE_CASELESS;\n> + }\n>\n> p->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n> - NULL);\n> + p->pcre_tables);\n> if (!p->pcre_regexp)\n> compile_regexp_failed(p, error);\n>\n-- \nDuy\n"},{"id":"265838","messageId":"559D0A4F.9040307@web.de","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v2 0/9] icase match on non-ascii","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-07-08T11:32:31Z","receivedAt":"2015-07-08T11:32:31Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2015-07-08 12.38, Nguyễn Thái Ngọc Duy wrote:\n> Side note, I almost added the third has_non_ascii() function. Maybe we\n> should consider merging the two existing has_non_ascii() functions\n> back, or rename one to something else.\n>\nSide question:\n\nhas_non_ascii can mean different things:\n UTF-8, ISO-8859-1, ISO-8859-x...\n\nIn short: everything.\nShould we be more critical here ?\n\nMay be this can be used from commit.c:\nstatic int verify_utf8(struct strbuf *buf)\n\nOther question:\nShould the \"non-ascii\" characters in the test scripts be octal-escaped ?\n\nThird question:\nWhat happens on systems, which don't have gettext, (for whatever reasons)\n--- a/gettext.c\n+++ b/gettext.c\n@@ -166,12 +166,17 @@ void git_setup_gettext(void)\ntextdomain(\"git\");\n}\n\n+int is_utf8_locale(void)\n+{\n+ return !strcmp(charset, \"UTF-8\");\n+}\n\n\n4th question:\nWhat happens on systems which don't have locale support at all ?\n\n\nAs one may suspect, I'm not a friend of being dependent on gettext and/or\nlocale, at least not for this kind of business.\n\nWould it make more sense to have a command line option ?\n"},{"id":"265843","messageId":"CACsJy8DPVrVBrnmq2SBDkR+TNgaDBgnZGseHwR82ap2+6yR+rw@mail.gmail.com","threadId":"39783","inReplyTo":"559D0A4F.9040307@web.de","subject":"Re: [PATCH v2 0/9] icase match on non-ascii","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T12:13:27Z","receivedAt":"2015-07-08T12:13:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 8, 2015 at 6:32 PM, Torsten Bögershausen wrote:\n> On 2015-07-08 12.38, Nguyễn Thái Ngọc Duy wrote:\n>> Side note, I almost added the third has_non_ascii() function. Maybe we\n>> should consider merging the two existing has_non_ascii() functions\n>> back, or rename one to something else.\n>>\n> Side question:\n>\n> has_non_ascii can mean different things:\n> UTF-8, ISO-8859-1, ISO-8859-x...\n>\n> In short: everything.\n> Should we be more critical here ?\n\nFor the purpose of this series, no. What we need to ask is \"can we\noptimize this?\" and that leads to \"ok we can safely optimize for\nascii-only patterns, don't optimize otherwise\".\n\n> May be this can be used from commit.c:\n> static int verify_utf8(struct strbuf *buf)\n\nExcept the pcre/utf8 patch, the rest should work for all single-byte encodings..\n\n> Other question:\n> Should the \"non-ascii\" characters in the test scripts be octal-escaped ?\n\nI would prefer something readable from a text editor. Even though I\ndon't speak Icelandic (the strings were copied from gettext test\nscript) I can see uppercase/lowercase of letters. But I don't know how\nmany people have fonts covering more than just ascii..\n\n> Third question:\n> What happens on systems, which don't have gettext, (for whatever reasons)\n> --- a/gettext.c\n> +++ b/gettext.c\n> @@ -166,12 +166,17 @@ void git_setup_gettext(void)\n> textdomain(\"git\");\n> }\n>\n> +int is_utf8_locale(void)\n> +{\n> + return !strcmp(charset, \"UTF-8\");\n> +}\n\nHm.. pcre on utf-8 is screwed. I really don't want to go through\nnl_langinfo, $LC_ALL, $LC_CTYPE and $LANG to detect utf8 like what's\ndone in compat/regex/regcomp.c.. But I guess there's no other way,\npeople can disable gettext and expect git to work properly with utf-8\nif their pcre library supports it.\n\n> 4th question:\n> What happens on systems which don't have locale support at all ?\n\nI suppose by locale here you do not mean \"gettext\" any more, then\nicase does not work. We simply delegate the work to system regex/pcre.\nIf they don't support locale, nothing we can do. Git itself does not\nknow how to fold case outside ascii (except maybe utf8, I don't know\nhow smart our utf-8 impl is). gettext support does not matter here,\nexcept pcre/utf8 case above.\n\n> As one may suspect, I'm not a friend of being dependent on gettext and/or\n> locale, at least not for this kind of business.\n>\n> Would it make more sense to have a command line option ?\n\nI assume you only care about utf-8 here (and utf8.c knows how to fold\ncase), you'll need to improve compat regex to take advantage of it\nfirst, maybe make sure it can live side by side with system regex,\nalso make sure pcre is compiled with utf-8 support if you use pcre.\nAnd if setting LANG to let git know you want to use utf-8 is too\nsubtle, then yes a command line option makes sense :)\n-- \nDuy\n"},{"id":"265856","messageId":"CAPc5daWG7=tBm_G+G6Qkmd7bd=5Dd-8bimj1p+KW_FSTOFfRaw@mail.gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v2 0/9] icase match on non-ascii","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-07-08T15:36:02Z","receivedAt":"2015-07-08T15:36:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"> Patch 5 is \"funny\". The patch itself is in iso-8859-1, but my name in\n> the commit message is in utf-8.\n\nAs an e-mail message is a single file, by definition that is not merely\n\"funny\" but just \"broken\", no matter what encoding your MUA\ndeclares the contents are in, no?\n"},{"id":"265885","messageId":"CACsJy8CMm-iHAeV9OUCY7B_F=gikDg9QqftG29aqM0W_q3yF7Q@mail.gmail.com","threadId":"39783","inReplyTo":"CAPc5daWG7=tBm_G+G6Qkmd7bd=5Dd-8bimj1p+KW_FSTOFfRaw@mail.gmail.com","subject":"Re: [PATCH v2 0/9] icase match on non-ascii","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-08T23:28:40Z","receivedAt":"2015-07-08T23:28:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 8, 2015 at 10:36 PM, Junio C Hamano wrote:\n>> Patch 5 is \"funny\". The patch itself is in iso-8859-1, but my name in\n>> the commit message is in utf-8.\n>\n> As an e-mail message is a single file, by definition that is not merely\n> \"funny\" but just \"broken\", no matter what encoding your MUA\n> declares the contents are in, no?\n\nYes. But if your MUA is not so strict and does not reject invalid byte\nsequences, git-am might be able to process it. What are the options we\nhave? Teach git-format-patch to generate patches with non-ascii chars\nas binary (utf-8 may be treated specially and kept as-is after\nvalidation)?\n-- \nDuy\n"},{"id":"265961","messageId":"CAPig+cTuTAHWwPDrbpOovN6HBvZRKHLGM-FPpEU3ZxXpH-aSCw@mail.gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-10-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v2 9/9] diffcore-pickaxe: support case insensitive match on non-ascii","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-07-09T22:55:08Z","receivedAt":"2015-07-09T22:55:08Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Jul 8, 2015 at 6:38 AM, Nguyễn Thái Ngọc Duy wrote:\n> Similar to the \"grep -F -i\" case, we can't use kws on icase search\n> outside ascii range, quote we quote the string and pass it to regcomp\n\ns/quote we quote/so we quote/\n\n(or something)\n\n> as a basic regexp and let regex engine deal with case sensitivity.\n>\n> The new test is put in t7812 instead of t4209-log-pickaxe because\n> lib-gettext.sh might cause problems elsewhere, probably..\n>\n> Noticed-by: Plamen Totev \n> Signed-off-by: Nguyễn Thái Ngọc Duy \n"},{"id":"266034","messageId":"1115951657.770231.1436602045776.JavaMail.apache@nm32.abv.bg","threadId":"39783","inReplyTo":"1436351919-2520-8-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v2 7/9] grep/pcre: support utf-8","fromName":"Plamen Totev","fromEmail":"plamen.totev@abv.bg","sentAt":"2015-07-11T08:07:25Z","receivedAt":"2015-07-11T08:07:25Z","isPatch":true,"sender":{"key":"plamen.totev@abv.bg","avatar":null},"body":"Nguyễn Thái Ngọc Duy  writes:\n> In the previous change in this function, we add locale support for \n> single-byte encodings only. It looks like pcre only supports utf-* as \n> multibyte encodings, the others are left in the cold (which is \n> fine). We need to enable PCRE_UTF8 so pcre can parse the string \n> correctly before folding case. \n\n> if (opt->ignore_case) { \n> p->pcre_tables = pcre_maketables(); \n> +\tif (is_utf8_locale()) \n> +\toptions |= PCRE_UTF8; \n> options |= PCRE_CASELESS; \n> } \n\nWe need to set the PCRE_UTF8 flag in all cases when the locale is UTF-8\nnot only when the search is case insensitive.\nOtherwise pcre threats the encoding as single byte and if the regex contains\nquantifiers it will not work as expected. The quantifier will try to match the\nsecond byte of the multi-byte symbol instead of the whole symbol.\n\nFor example lets have file that contains the string\n\nTILRAUN: HALLÓÓÓ HEIMUR!\n\nthe following command\n\ngit grep -P \"HALLÓ{3}\"\n\nwill not match the file while \n\ngit grep -P \"HAL{2}ÓÓÓ\"\n\nwill. That's because the L symbol is a single byte.\n\nRegards,\nPlamen Totev\n"},{"id":"266118","messageId":"1436880280-18194-1-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436351919-2520-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 0/9] icase match on non-ascii","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:31Z","receivedAt":"2015-07-14T13:24:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Compared to v2\n\n - fix grep/pcre on utf-8 even in case is sensitive\n - peek at $LANG and friends anyway for utf-8 detection even\n when gettext support is not built in git\n - s/quote we quote/so we quote/ in 9/9\n - rename t7813, s/non-ascii/iso/\n\nNguyễn Thái Ngọc Duy (9):\n grep: allow -F -i combination\n grep: break down an \"if\" stmt in preparation for next changes\n grep/icase: avoid kwsset on literal non-ascii strings\n grep/icase: avoid kwsset when -F is specified\n grep/pcre: prepare locale-dependent tables for icase matching\n gettext: add is_utf8_locale()\n grep/pcre: support utf-8\n diffcore-pickaxe: \"share\" regex error handling code\n diffcore-pickaxe: support case insensitive match on non-ascii\n\n builtin/grep.c | 2 +-\n diffcore-pickaxe.c | 27 +++++++++----\n gettext.c | 24 +++++++++++-\n gettext.h | 1 +\n grep.c | 44 +++++++++++++++++++--\n grep.h | 1 +\n quote.c | 37 ++++++++++++++++++\n quote.h | 1 +\n t/t7812-grep-icase-non-ascii.sh (new +x) | 67 ++++++++++++++++++++++++++++++++\n t/t7813-grep-icase-iso.sh (new +x) | 19 +++++++++\n 10 files changed, 208 insertions(+), 15 deletions(-)\n create mode 100755 t/t7812-grep-icase-non-ascii.sh\n create mode 100755 t/t7813-grep-icase-iso.sh\n\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266119","messageId":"1436880280-18194-2-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 1/9] grep: allow -F -i combination","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:32Z","receivedAt":"2015-07-14T13:24:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"-F means \"no regex\", not \"case sensitive\" so it should not override -i\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/grep.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex d04f440..2d392e9 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -806,7 +806,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (!opt.pattern_list)\n \t\tdie(_(\"no pattern given.\"));\n-\tif (!opt.fixed && opt.ignore_case)\n+\tif (opt.ignore_case)\n \t\topt.regflags |= REG_ICASE;\n \n \tcompile_grep_patterns(&opt);\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266120","messageId":"1436880280-18194-3-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 2/9] grep: break down an \"if\" stmt in preparation for next changes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:33Z","receivedAt":"2015-07-14T13:24:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex b58c7c6..bd32f66 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -403,9 +403,11 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \n-\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n+\tif (is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n-\telse\n+\telse if (opt->fixed) {\n+\t\tp->fixed = 1;\n+\t} else\n \t\tp->fixed = 0;\n \n \tif (p->fixed) {\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266121","messageId":"1436880280-18194-4-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 3/9] grep/icase: avoid kwsset on literal non-ascii strings","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:34Z","receivedAt":"2015-07-14T13:24:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When we detect the pattern is just a literal string, we avoid heavy\nregex engine and use fast substring search implemented in kwsset.c.\nBut kws uses git-ctype which is locale-independent so it does not know\nhow to fold case properly outside ascii range. Let regcomp or pcre\ntake care of this case instead. Slower, but accurate.\n\nHelped-by: René Scharfe \nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 7 ++++++-\n t/t7812-grep-icase-non-ascii.sh (new +x) | 19 +++++++++++++++++++\n 2 files changed, 25 insertions(+), 1 deletion(-)\n create mode 100755 t/t7812-grep-icase-non-ascii.sh\n\ndiff --git a/grep.c b/grep.c\nindex bd32f66..d795b0e 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -4,6 +4,7 @@\n #include \"xdiff-interface.h\"\n #include \"diff.h\"\n #include \"diffcore.h\"\n+#include \"commit.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -398,12 +399,16 @@ static int is_fixed(const char *s, size_t len)\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n+\tint icase_non_ascii;\n \tint err;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n+\ticase_non_ascii =\n+\t\t(opt->regflags & REG_ICASE || p->ignore_case) &&\n+\t\thas_non_ascii(p->pattern);\n \n-\tif (is_fixed(p->pattern, p->patternlen))\n+\tif (!icase_non_ascii && is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse if (opt->fixed) {\n \t\tp->fixed = 1;\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nnew file mode 100755\nindex 0000000..63a2630\n--- /dev/null\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -0,0 +1,19 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_LOCALE 'setup' '\n+\tprintf \"TILRAUN: Halló Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_expect_success GETTEXT_LOCALE 'grep literal string, no -F' '\n+\tgit grep -i \"TILRAUN: Halló Heimur!\" &&\n+\tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266122","messageId":"1436880280-18194-5-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 4/9] grep/icase: avoid kwsset when -F is specified","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:35Z","receivedAt":"2015-07-14T13:24:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the previous commit, we can't use kws on icase search\noutside ascii range. But we can't simply pass the pattern to\nregcomp/pcre like the previous commit because it may contain regex\nspecial characters, so we need to quote the regex first.\n\nTo avoid misquote traps that could lead to undefined behavior, we\nalways stick to basic regex engine in this case. We don't need fancy\nfeatures for grepping a literal string anyway.\n\nbasic_regex_quote_buf() assumes that if the pattern is in a multibyte\nencoding, ascii chars must be unambiguously encoded as single\nbytes. This is true at least for UTF-8. For others, let's wait until\npeople yell up. Chances are nobody uses multibyte, non utf-8 charsets\nany more..\n\nHelped-by: René Scharfe \nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 25 ++++++++++++++++++++++++-\n quote.c | 37 +++++++++++++++++++++++++++++++++++++\n quote.h | 1 +\n t/t7812-grep-icase-non-ascii.sh | 26 ++++++++++++++++++++++++++\n 4 files changed, 88 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex d795b0e..8fce54f 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -5,6 +5,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"commit.h\"\n+#include \"quote.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -397,6 +398,24 @@ static int is_fixed(const char *s, size_t len)\n \treturn 1;\n }\n \n+static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint err;\n+\n+\tbasic_regex_quote_buf(&sb, p->pattern);\n+\terr = regcomp(&p->regexp, sb.buf, opt->regflags & ~REG_EXTENDED);\n+\tif (opt->debug)\n+\t\tfprintf(stderr, \"fixed%s\\n\", sb.buf);\n+\tstrbuf_release(&sb);\n+\tif (err) {\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, &p->regexp, errbuf, 1024);\n+\t\tregfree(&p->regexp);\n+\t\tcompile_regexp_failed(p, errbuf);\n+\t}\n+}\n+\n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tint icase_non_ascii;\n@@ -411,7 +430,11 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tif (!icase_non_ascii && is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse if (opt->fixed) {\n-\t\tp->fixed = 1;\n+\t\tp->fixed = !icase_non_ascii;\n+\t\tif (!p->fixed) {\n+\t\t\tcompile_fixed_regexp(p, opt);\n+\t\t\treturn;\n+\t\t}\n \t} else\n \t\tp->fixed = 0;\n \ndiff --git a/quote.c b/quote.c\nindex 7920e18..43a8057 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -439,3 +439,40 @@ void tcl_quote_buf(struct strbuf *sb, const char *src)\n \t}\n \tstrbuf_addch(sb, '\"');\n }\n+\n+void basic_regex_quote_buf(struct strbuf *sb, const char *src)\n+{\n+\tchar c;\n+\n+\tif (*src == '^') {\n+\t\t/* only beginning '^' is special and needs quoting */\n+\t\tstrbuf_addch(sb, '\\\\');\n+\t\tstrbuf_addch(sb, *src++);\n+\t}\n+\tif (*src == '*')\n+\t\t/* beginning '*' is not special, no quoting */\n+\t\tstrbuf_addch(sb, *src++);\n+\n+\twhile ((c = *src++)) {\n+\t\tswitch (c) {\n+\t\tcase '[':\n+\t\tcase '.':\n+\t\tcase '\\\\':\n+\t\tcase '*':\n+\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tcase '$':\n+\t\t\t/* only the end '$' is special and needs quoting */\n+\t\t\tif (*src == '\\0')\n+\t\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\ndiff --git a/quote.h b/quote.h\nindex 99e04d3..362d315 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -67,5 +67,6 @@ extern char *quote_path_relative(const char *in, const char *prefix,\n extern void perl_quote_buf(struct strbuf *sb, const char *src);\n extern void python_quote_buf(struct strbuf *sb, const char *src);\n extern void tcl_quote_buf(struct strbuf *sb, const char *src);\n+extern void basic_regex_quote_buf(struct strbuf *sb, const char *src);\n \n #endif\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 63a2630..c945589 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -16,4 +16,30 @@ test_expect_success GETTEXT_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success GETTEXT_LOCALE 'grep literal string, with -F' '\n+\tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\techo \"fixedTILRAUN: Halló Heimur!\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"TILRAUN: HALLÓ HEIMUR!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\techo \"fixedTILRAUN: HALLÓ HEIMUR!\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n+test_expect_success GETTEXT_LOCALE 'grep string with regex, with -F' '\n+\tprintf \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\techo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\HALLÓ \\$HE[]IMUR!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\techo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266123","messageId":"1436880280-18194-6-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 5/9] grep/pcre: prepare locale-dependent tables for icase matching","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:36Z","receivedAt":"2015-07-14T13:24:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The default tables are usually built with C locale and only suitable\nfor LANG=C or similar. This should make case insensitive search work\ncorrectly for all single-byte charsets.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 8 ++++++--\n grep.h | 1 +\n t/t7813-grep-icase-iso.sh (new +x) | 19 +++++++++++++++++++\n 3 files changed, 26 insertions(+), 2 deletions(-)\n create mode 100755 t/t7813-grep-icase-iso.sh\n\ndiff --git a/grep.c b/grep.c\nindex 8fce54f..f0fbf99 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -324,11 +324,14 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \tint erroffset;\n \tint options = PCRE_MULTILINE;\n \n-\tif (opt->ignore_case)\n+\tif (opt->ignore_case) {\n+\t\tif (has_non_ascii(p->pattern))\n+\t\t\tp->pcre_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n+\t}\n \n \tp->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n-\t\t\tNULL);\n+\t\t\t\t p->pcre_tables);\n \tif (!p->pcre_regexp)\n \t\tcompile_regexp_failed(p, error);\n \n@@ -362,6 +365,7 @@ static void free_pcre_regexp(struct grep_pat *p)\n {\n \tpcre_free(p->pcre_regexp);\n \tpcre_free(p->pcre_extra_info);\n+\tpcre_free((void *)p->pcre_tables);\n }\n #else /* !USE_LIBPCRE */\n static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\ndiff --git a/grep.h b/grep.h\nindex 95f197a..cee4357 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -48,6 +48,7 @@ struct grep_pat {\n \tregex_t regexp;\n \tpcre *pcre_regexp;\n \tpcre_extra *pcre_extra_info;\n+\tconst unsigned char *pcre_tables;\n \tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\ndiff --git a/t/t7813-grep-icase-iso.sh b/t/t7813-grep-icase-iso.sh\nnew file mode 100755\nindex 0000000..efef7fb\n--- /dev/null\n+++ b/t/t7813-grep-icase-iso.sh\n@@ -0,0 +1,19 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_ISO_LOCALE 'setup' '\n+\tprintf \"TILRAUN: Hall� Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_iso_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_expect_success GETTEXT_ISO_LOCALE,LIBPCRE 'grep pcre string' '\n+\tgit grep --perl-regexp -i \"TILRAUN: H.ll� Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LL� HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266124","messageId":"1436880280-18194-7-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 6/9] gettext: add is_utf8_locale()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:37Z","receivedAt":"2015-07-14T13:24:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This function returns true if git is running under an UTF-8\nlocale. pcre in the next patch will need this.\n\nis_encoding_utf8() is used instead of strcmp() to catch both \"utf-8\"\nand \"utf8\" suffixes.\n\nWhen built with no gettext support, we peek in several env variables\nto detect UTF-8. pcre library might support utf-8 even if libc or git\nis built without locale support.. The peeking code is a copy from\ncompat/regex/regcomp.c.\n\nHelped-by: Torsten Bögershausen \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n gettext.c | 24 ++++++++++++++++++++++--\n gettext.h | 1 +\n 2 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/gettext.c b/gettext.c\nindex 7378ba2..aaf1688 100644\n--- a/gettext.c\n+++ b/gettext.c\n@@ -18,6 +18,8 @@\n #\tendif\n #endif\n \n+static const char *charset;\n+\n /*\n * Guess the user's preferred languages from the value in LANGUAGE environment\n * variable and LC_MESSAGES locale category if NO_GETTEXT is not defined.\n@@ -65,7 +67,6 @@ static int test_vsnprintf(const char *fmt, ...)\n \treturn ret;\n }\n \n-static const char *charset;\n static void init_gettext_charset(const char *domain)\n {\n \t/*\n@@ -171,8 +172,27 @@ int gettext_width(const char *s)\n {\n \tstatic int is_utf8 = -1;\n \tif (is_utf8 == -1)\n-\t\tis_utf8 = !strcmp(charset, \"UTF-8\");\n+\t\tis_utf8 = is_utf8_locale();\n \n \treturn is_utf8 ? utf8_strwidth(s) : strlen(s);\n }\n #endif\n+\n+int is_utf8_locale(void)\n+{\n+#ifdef NO_GETTEXT\n+\tif (!charset) {\n+\t\tconst char *env = getenv(\"LC_ALL\");\n+\t\tif (!env || !*env)\n+\t\t\tenv = getenv(\"LC_CTYPE\");\n+\t\tif (!env || !*env)\n+\t\t\tenv = getenv(\"LANG\");\n+\t\tif (!env)\n+\t\t\tenv = \"\";\n+\t\tif (strchr(env, '.'))\n+\t\t\tenv = strchr(env, '.') + 1;\n+\t\tcharset = xstrdup(env);\n+\t}\n+#endif\n+\treturn is_encoding_utf8(charset);\n+}\ndiff --git a/gettext.h b/gettext.h\nindex 33696a4..7eee64a 100644\n--- a/gettext.h\n+++ b/gettext.h\n@@ -90,5 +90,6 @@ const char *Q_(const char *msgid, const char *plu, unsigned long n)\n #endif\n \n const char *get_preferred_languages(void);\n+extern int is_utf8_locale(void);\n \n #endif\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266125","messageId":"1436880280-18194-8-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 7/9] grep/pcre: support utf-8","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:38Z","receivedAt":"2015-07-14T13:24:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In the previous change in this function, we add locale support for\nsingle-byte encodings only. It looks like pcre only supports utf-* as\nmultibyte encodings, the others are left in the cold (which is\nfine).\n\nWe need to enable PCRE_UTF8 so pcre can find character boundary\ncorrectly. It's needed for case folding (when --ignore-case is used)\nor '*', '+' or similar syntax is used.\n\nThe \"has_non_ascii()\" check is to be on the conservative side. If\nthere's non-ascii in the pattern, the searched content could still be\nin utf-8, but we can treat it just like a byte stream and everything\nshould work. If we force utf-8 based on locale only and pcre validates\nutf-8 and the file content is in non-utf8 encoding, things break.\n\nNoticed-by: Plamen Totev \nHelped-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 2 ++\n t/t7812-grep-icase-non-ascii.sh | 15 +++++++++++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/grep.c b/grep.c\nindex f0fbf99..07621c1 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -329,6 +329,8 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \t\t\tp->pcre_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n \t}\n+\tif (is_utf8_locale() && has_non_ascii(p->pattern))\n+\t\toptions |= PCRE_UTF8;\n \n \tp->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n \t\t\t\t p->pcre_tables);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex c945589..e861a15 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -16,6 +16,21 @@ test_expect_success GETTEXT_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 icase' '\n+\tgit grep --perl-regexp \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LLÓ HEIMUR!\"\n+'\n+\n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 string with \"+\"' '\n+\tprintf \"TILRAUN: Hallóó Heimur!\" >file2 &&\n+\tgit add file2 &&\n+\tgit grep -l --perl-regexp \"TILRAUN: H.lló+ Heimur!\" >actual &&\n+\techo file >expected &&\n+\techo file2 >>expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success GETTEXT_LOCALE 'grep literal string, with -F' '\n \tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266126","messageId":"1436880280-18194-9-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 8/9] diffcore-pickaxe: \"share\" regex error handling code","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:39Z","receivedAt":"2015-07-14T13:24:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"There's another regcomp code block coming in this function. By moving\nthe error handling code out of this block, we don't have to add the\nsame error handling code in the new block.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n diffcore-pickaxe.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 185f86b..7a718fc 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -204,20 +204,13 @@ void diffcore_pickaxe(struct diff_options *o)\n \tint opts = o->pickaxe_opts;\n \tregex_t regex, *regexp = NULL;\n \tkwset_t kws = NULL;\n+\tint err = 0;\n \n \tif (opts & (DIFF_PICKAXE_REGEX | DIFF_PICKAXE_KIND_G)) {\n-\t\tint err;\n \t\tint cflags = REG_EXTENDED | REG_NEWLINE;\n \t\tif (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE))\n \t\t\tcflags |= REG_ICASE;\n \t\terr = regcomp(®ex, needle, cflags);\n-\t\tif (err) {\n-\t\t\t/* The POSIX.2 people are surely sick */\n-\t\t\tchar errbuf[1024];\n-\t\t\tregerror(err, ®ex, errbuf, 1024);\n-\t\t\tregfree(®ex);\n-\t\t\tdie(\"invalid regex: %s\", errbuf);\n-\t\t}\n \t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n@@ -225,6 +218,13 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\tkwsincr(kws, needle, strlen(needle));\n \t\tkwsprep(kws);\n \t}\n+\tif (err) {\n+\t\t/* The POSIX.2 people are surely sick */\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, ®ex, errbuf, 1024);\n+\t\tregfree(®ex);\n+\t\tdie(\"invalid regex: %s\", errbuf);\n+\t}\n \n \t/* Might want to warn when both S and G are on; I don't care... */\n \tpickaxe(&diff_queued_diff, o, regexp, kws,\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266127","messageId":"1436880280-18194-10-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 9/9] diffcore-pickaxe: support case insensitive match on non-ascii","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-14T13:24:40Z","receivedAt":"2015-07-14T13:24:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the \"grep -F -i\" case, we can't use kws on icase search\noutside ascii range, so we quote the string and pass it to regcomp as\na basic regexp and let regex engine deal with case sensitivity.\n\nThe new test is put in t7812 instead of t4209-log-pickaxe because\nlib-gettext.sh might cause problems elsewhere, probably..\n\nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n diffcore-pickaxe.c | 11 +++++++++++\n t/t7812-grep-icase-non-ascii.sh | 7 +++++++\n 2 files changed, 18 insertions(+)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 7a718fc..6946c15 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -7,6 +7,8 @@\n #include \"diffcore.h\"\n #include \"xdiff-interface.h\"\n #include \"kwset.h\"\n+#include \"commit.h\"\n+#include \"quote.h\"\n \n typedef int (*pickaxe_fn)(mmfile_t *one, mmfile_t *two,\n \t\t\t struct diff_options *o,\n@@ -212,6 +214,15 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\t\tcflags |= REG_ICASE;\n \t\terr = regcomp(®ex, needle, cflags);\n \t\tregexp = ®ex;\n+\t} else if (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE) &&\n+\t\t has_non_ascii(needle)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tint cflags = REG_NEWLINE | REG_ICASE;\n+\n+\t\tbasic_regex_quote_buf(&sb, needle);\n+\t\terr = regcomp(®ex, sb.buf, cflags);\n+\t\tstrbuf_release(&sb);\n+\t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n \t\t\t ? tolower_trans_tbl : NULL);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex e861a15..d07fa20 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -57,4 +57,11 @@ test_expect_success GETTEXT_LOCALE 'grep string with regex, with -F' '\n \ttest_cmp expect2 debug2\n '\n \n+test_expect_success GETTEXT_LOCALE 'pickaxe -i on non-ascii' '\n+\tgit commit -m first &&\n+\tgit log --format=%f -i -S\"TILRAUN: HALLÓ HEIMUR!\" >actual &&\n+\techo first >expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"266132","messageId":"55A53C13.5090405@web.de","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 0/9] icase match on non-ascii","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-07-14T16:42:59Z","receivedAt":"2015-07-14T16:42:59Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"(I haven't been able to do more debugging yet,\nbut this doesn't fully work on my Mac OS X box:)\n\nInitialized empty Git repository in\n/Users/tb/NoBackup/projects/git/tb.150714_Duy_grep_utf8/t/trash\ndirectory.t7812-grep-icase-non-ascii/.git/\n# lib-gettext: Found 'is_IS.UTF-8' as an is_IS UTF-8 locale\n# lib-gettext: Found 'is_IS.ISO8859-1' as an is_IS ISO-8859-1 locale\nexpecting success:\n printf \"TILRAUN: Halló Heimur!\" >file &&\n git add file &&\n LC_ALL=\"$is_IS_locale\" &&\n export LC_ALL\n\nok 1 - setup\n\nexpecting success:\n git grep -i \"TILRAUN: Halló Heimur!\" &&\n git grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n\nfile:TILRAUN: Halló Heimur!\nnot ok 2 - grep literal string, no -F\n# \n# git grep -i \"TILRAUN: Halló Heimur!\" &&\n# git grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n# \n\nskipping test: grep pcre utf-8 icase\n git grep --perl-regexp \"TILRAUN: H.lló Heimur!\" &&\n git grep --perl-regexp -i \"TILRAUN: H.lló Heimur!\" &&\n git grep --perl-regexp -i \"TILRAUN: H.LLÓ HEIMUR!\"\n\nok 3 # skip grep pcre utf-8 icase (missing LIBPCRE of GETTEXT_LOCALE,LIBPCRE)\n\nskipping test: grep pcre utf-8 string with \"+\"\n printf \"TILRAUN: Hallóó Heimur!\" >file2 &&\n git add file2 &&\n git grep -l --perl-regexp \"TILRAUN: H.lló+ Heimur!\" >actual &&\n echo file >expected &&\n echo file2 >>expected &&\n test_cmp expected actual\n\nok 4 # skip grep pcre utf-8 string with \"+\" (missing LIBPCRE of\nGETTEXT_LOCALE,LIBPCRE)\n\nexpecting success:\n git grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n grep fixed >debug1 &&\n echo \"fixedTILRAUN: Halló Heimur!\" >expect1 &&\n test_cmp expect1 debug1 &&\n\n git grep --debug -i -F \"TILRAUN: HALLÓ HEIMUR!\" 2>&1 >/dev/null |\n grep fixed >debug2 &&\n echo \"fixedTILRAUN: HALLÓ HEIMUR!\" >expect2 &&\n test_cmp expect2 debug2\n\nok 5 - grep literal string, with -F\n\nexpecting success:\n printf \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n\n git grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1 >/dev/null |\n grep fixed >debug1 &&\n echo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n test_cmp expect1 debug1 &&\n\n git grep --debug -i -F \"^*TILR^AUN:.* \\\\HALLÓ \\$HE[]IMUR!\\$\" 2>&1 >/dev/null |\n grep fixed >debug2 &&\n echo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n test_cmp expect2 debug2\n\n--- expect1 2015-07-14 16:38:22.000000000 +0000\n+++ debug1 2015-07-14 16:38:22.000000000 +0000\n@@ -1 +1 @@\n-fixed\\^*TILR^AUN:\\.\\* \\Halló $He\\[]imur!\\$\n+fixed\\^*TILR^AUN:\\.\\* \\\\Halló $He\\[]imur!\\$\nnot ok 6 - grep string with regex, with -F\n# \n# printf \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n# \n# git grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1\n>/dev/null |\n# grep fixed >debug1 &&\n# echo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n# test_cmp expect1 debug1 &&\n# \n# git grep --debug -i -F \"^*TILR^AUN:.* \\\\HALLÓ \\$HE[]IMUR!\\$\" 2>&1\n>/dev/null |\n# grep fixed >debug2 &&\n# echo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n# test_cmp expect2 debug2\n# \n\nexpecting success:\n git commit -m first &&\n git log --format=%f -i -S\"TILRAUN: HALLÓ HEIMUR!\" >actual &&\n echo first >expected &&\n test_cmp expected actual\n\n[master (root-commit) e6052d5] first\n Author: A U Thor \n 1 file changed, 1 insertion(+)\n create mode 100644 file\n--- expected 2015-07-14 16:38:22.000000000 +0000\n+++ actual 2015-07-14 16:38:22.000000000 +0000\n@@ -1 +0,0 @@\n-first\nnot ok 7 - pickaxe -i on non-ascii\n# \n# git commit -m first &&\n# git log --format=%f -i -S\"TILRAUN: HALLÓ HEIMUR!\" >actual &&\n# echo first >expected &&\n# test_cmp expected actual\n# \n\n# failed 3 among 7 test(s)\n1..7\n"},{"id":"266147","messageId":"CACsJy8BXGt25tGoarry=UQS6E4Q54kicG5oY9uoU6W_hw1wYtg@mail.gmail.com","threadId":"39783","inReplyTo":"55A53C13.5090405@web.de","subject":"Re: [PATCH v3 0/9] icase match on non-ascii","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-07-15T09:39:24Z","receivedAt":"2015-07-15T09:39:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 14, 2015 at 11:42 PM, Torsten Bögershausen wrote:\n> (I haven't been able to do more debugging yet,\n> but this doesn't fully work on my Mac OS X box:)\n>\n> Initialized empty Git repository in\n> /Users/tb/NoBackup/projects/git/tb.150714_Duy_grep_utf8/t/trash\n> directory.t7812-grep-icase-non-ascii/.git/\n> # lib-gettext: Found 'is_IS.UTF-8' as an is_IS UTF-8 locale\n> # lib-gettext: Found 'is_IS.ISO8859-1' as an is_IS ISO-8859-1 locale\n> expecting success:\n> printf \"TILRAUN: Halló Heimur!\" >file &&\n> git add file &&\n> LC_ALL=\"$is_IS_locale\" &&\n> export LC_ALL\n>\n> ok 1 - setup\n>\n> expecting success:\n> git grep -i \"TILRAUN: Halló Heimur!\" &&\n> git grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n>\n> file:TILRAUN: Halló Heimur!\n> not ok 2 - grep literal string, no -F\n> #\n> # git grep -i \"TILRAUN: Halló Heimur!\" &&\n> # git grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n> #\n\nI don't know if there's an easy way to test if regexec() on your\nsystem supports locale (at least for is_IS). I can reproduce the same\nby using compat regex. So it's not a good news because compat regex is\nused in a few platforms, so this test will fail on those.\n\nI don't see any way around it, except dropping all the tests. I don't\nthink there is a way for us to test regex locale support at runtime.\n-- \nDuy\n"},{"id":"266162","messageId":"55A6B9D9.2070102@web.de","threadId":"39783","inReplyTo":"CACsJy8BXGt25tGoarry=UQS6E4Q54kicG5oY9uoU6W_hw1wYtg@mail.gmail.com","subject":"Re: [PATCH v3 0/9] icase match on non-ascii","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-07-15T19:51:53Z","receivedAt":"2015-07-15T19:51:53Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"> I don't see any way around it, except dropping all the tests. I don't\n> think there is a way for us to test regex locale support at runtime.\n> \n(I don't think dropping all tests is a good way forward)\nEither there is runtime code similar to test-regex.c,\nor how about something like this:\n\ncommit a1cdac0fc0df1dad20f4dc196688a73c11b00480\nAuthor: Torsten Bögershausen \nDate: Wed Jul 15 21:43:47 2015 +0200\n\n t7812: More LIBPCRE preconditions\n\n Some (e.g. BSD based) regex libraries are not able to handle\n UTF-8 strings case-insensitive (if asked so)\n\n Exclude some test cases by using the LIBPCRE precondition\n\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex d07fa20..30d3d68 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -11,9 +11,12 @@ test_expect_success GETTEXT_LOCALE 'setup' '\n \texport LC_ALL\n '\n\n-test_expect_success GETTEXT_LOCALE 'grep literal string, no -F' '\n-\tgit grep -i \"TILRAUN: Halló Heimur!\" &&\n-\tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n+test_expect_success GETTEXT_LOCALE 'grep literal low string, no -F' '\n+\tgit grep -i \"TILRAUN: Halló Heimur!\"\n+'\n+\n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep literal up string, no -F' '\n+\tgit grep -i \"TILRAUN: HALLÓ. HEIMUR!\"\n '\n\n test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 icase' '\n@@ -31,33 +34,37 @@ test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8\nstring with \"+\"' '\n \ttest_cmp expected actual\n '\n\n-test_expect_success GETTEXT_LOCALE 'grep literal string, with -F' '\n+test_expect_success GETTEXT_LOCALE 'grep literal low string, with -F' '\n \tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n \techo \"fixedTILRAUN: Halló Heimur!\" >expect1 &&\n-\ttest_cmp expect1 debug1 &&\n+\ttest_cmp expect1 debug1\n+'\n\n+test_expect_success GETTEXT_LOCALE 'grep literal up string, with -F' '\n \tgit grep --debug -i -F \"TILRAUN: HALLÓ HEIMUR!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug2 &&\n \techo \"fixedTILRAUN: HALLÓ HEIMUR!\" >expect2 &&\n \ttest_cmp expect2 debug2\n '\n\n-test_expect_success GETTEXT_LOCALE 'grep string with regex, with -F' '\n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep string with regex, with -F' '\n \tprintf \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n\n \tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n \techo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n-\ttest_cmp expect1 debug1 &&\n+\ttest_cmp expect1 debug1\n+'\n\n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep up string with regex, with -F' '\n \tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\HALLÓ \\$HE[]IMUR!\\$\" 2>&1 >/dev/null |\n \t\t grep fixed >debug2 &&\n \techo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n \ttest_cmp expect2 debug2\n '\n\n-test_expect_success GETTEXT_LOCALE 'pickaxe -i on non-ascii' '\n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'pickaxe -i on non-ascii' '\n \tgit commit -m first &&\n \tgit log --format=%f -i -S\"TILRAUN: HALLÓ HEIMUR!\" >actual &&\n \techo first >expected &&\n"},{"id":"268411","messageId":"1440161240-28554-1-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1436880280-18194-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 00/10] icase match on non-ascii","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:10Z","receivedAt":"2015-08-21T12:47:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Sorry it took more than a month for a simple reroll. Free time (with\nenergy left) is rare these days. v4 adds system regex's icase support\ndetection and only runs tests in these cases. This should fix test\nfailures on Windows where compat regex does not support icase.\n\nDiff from v3 below\n\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex d07fa20..a5475bb 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -11,7 +11,11 @@ test_expect_success GETTEXT_LOCALE 'setup' '\n \texport LC_ALL\n '\n \n-test_expect_success GETTEXT_LOCALE 'grep literal string, no -F' '\n+test_have_prereq GETTEXT_LOCALE &&\n+test-regex \"HALLÓ\" \"Halló\" ICASE &&\n+test_set_prereq REGEX_LOCALE\n+\n+test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: Halló Heimur!\" &&\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n@@ -31,7 +35,7 @@ test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 string with \"+\"' '\n \ttest_cmp expected actual\n '\n \n-test_expect_success GETTEXT_LOCALE 'grep literal string, with -F' '\n+test_expect_success REGEX_LOCALE 'grep literal string, with -F' '\n \tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n \techo \"fixedTILRAUN: Halló Heimur!\" >expect1 &&\n@@ -43,7 +47,7 @@ test_expect_success GETTEXT_LOCALE 'grep literal string, with -F' '\n \ttest_cmp expect2 debug2\n '\n \n-test_expect_success GETTEXT_LOCALE 'grep string with regex, with -F' '\n+test_expect_success REGEX_LOCALE 'grep string with regex, with -F' '\n \tprintf \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n \n \tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1 >/dev/null |\n@@ -57,7 +61,7 @@ test_expect_success GETTEXT_LOCALE 'grep string with regex, with -F' '\n \ttest_cmp expect2 debug2\n '\n \n-test_expect_success GETTEXT_LOCALE 'pickaxe -i on non-ascii' '\n+test_expect_success REGEX_LOCALE 'pickaxe -i on non-ascii' '\n \tgit commit -m first &&\n \tgit log --format=%f -i -S\"TILRAUN: HALLÓ HEIMUR!\" >actual &&\n \techo first >expected &&\ndiff --git a/test-regex.c b/test-regex.c\nindex 0dc598e..3b5641c 100644\n--- a/test-regex.c\n+++ b/test-regex.c\n@@ -1,19 +1,63 @@\n #include \"git-compat-util.h\"\n+#include \"gettext.h\"\n+\n+struct reg_flag {\n+\tconst char *name;\n+\tint flag;\n+};\n+\n+static struct reg_flag reg_flags[] = {\n+\t{ \"EXTENDED\",\t REG_EXTENDED\t},\n+\t{ \"NEWLINE\",\t REG_NEWLINE\t},\n+\t{ \"ICASE\",\t REG_ICASE\t},\n+\t{ \"NOTBOL\",\t REG_NOTBOL\t},\n+#ifdef REG_STARTEND\n+\t{ \"STARTEND\",\t REG_STARTEND\t},\n+#endif\n+\t{ NULL, 0 }\n+};\n \n int main(int argc, char **argv)\n {\n-\tchar *pat = \"[^={} \\t]+\";\n-\tchar *str = \"={}\\nfred\";\n+\tconst char *pat;\n+\tconst char *str;\n+\tint flags = 0;\n \tregex_t r;\n \tregmatch_t m[1];\n \n-\tif (regcomp(&r, pat, REG_EXTENDED | REG_NEWLINE))\n+\tif (argc == 1) {\n+\t\t/* special case, bug check */\n+\t\tpat = \"[^={} \\t]+\";\n+\t\tstr = \"={}\\nfred\";\n+\t\tflags = REG_EXTENDED | REG_NEWLINE;\n+\t} else {\n+\t\targv++;\n+\t\tpat = *argv++;\n+\t\tstr = *argv++;\n+\t\twhile (*argv) {\n+\t\t\tstruct reg_flag *rf;\n+\t\t\tfor (rf = reg_flags; rf->name; rf++)\n+\t\t\t\tif (!strcmp(*argv, rf->name)) {\n+\t\t\t\t\tflags |= rf->flag;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\tif (!rf->name)\n+\t\t\t\tdie(\"do not recognize %s\", *argv);\n+\t\t\targv++;\n+\t\t}\n+\t\tgit_setup_gettext();\n+\t}\n+\n+\tif (regcomp(&r, pat, flags))\n \t\tdie(\"failed regcomp() for pattern '%s'\", pat);\n-\tif (regexec(&r, str, 1, m, 0))\n-\t\tdie(\"no match of pattern '%s' to string '%s'\", pat, str);\n+\tif (regexec(&r, str, 1, m, 0)) {\n+\t\tif (argc == 1)\n+\t\t\tdie(\"no match of pattern '%s' to string '%s'\", pat, str);\n+\t\treturn 1;\n+\t}\n \n \t/* http://sourceware.org/bugzilla/show_bug.cgi?id=3957 */\n-\tif (m[0].rm_so == 3) /* matches '\\n' when it should not */\n+\tif (argc == 1 && m[0].rm_so == 3) /* matches '\\n' when it should not */\n \t\tdie(\"regex bug confirmed: re-build git with NO_REGEX=1\");\n \n \texit(0);\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268413","messageId":"1440161240-28554-2-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 01/10] grep: allow -F -i combination","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:11Z","receivedAt":"2015-08-21T12:47:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"-F means \"no regex\", not \"case sensitive\" so it should not override -i\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/grep.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex d04f440..2d392e9 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -806,7 +806,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \n \tif (!opt.pattern_list)\n \t\tdie(_(\"no pattern given.\"));\n-\tif (!opt.fixed && opt.ignore_case)\n+\tif (opt.ignore_case)\n \t\topt.regflags |= REG_ICASE;\n \n \tcompile_grep_patterns(&opt);\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268412","messageId":"1440161240-28554-3-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 02/10] grep: break down an \"if\" stmt in preparation for next changes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:12Z","receivedAt":"2015-08-21T12:47:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex b58c7c6..bd32f66 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -403,9 +403,11 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \n-\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n+\tif (is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n-\telse\n+\telse if (opt->fixed) {\n+\t\tp->fixed = 1;\n+\t} else\n \t\tp->fixed = 0;\n \n \tif (p->fixed) {\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268414","messageId":"1440161240-28554-4-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 03/10] test-regex: expose full regcomp() to the command line","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:13Z","receivedAt":"2015-08-21T12:47:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n test-regex.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++------\n 1 file changed, 50 insertions(+), 6 deletions(-)\n\ndiff --git a/test-regex.c b/test-regex.c\nindex 0dc598e..3b5641c 100644\n--- a/test-regex.c\n+++ b/test-regex.c\n@@ -1,19 +1,63 @@\n #include \"git-compat-util.h\"\n+#include \"gettext.h\"\n+\n+struct reg_flag {\n+\tconst char *name;\n+\tint flag;\n+};\n+\n+static struct reg_flag reg_flags[] = {\n+\t{ \"EXTENDED\",\t REG_EXTENDED\t},\n+\t{ \"NEWLINE\",\t REG_NEWLINE\t},\n+\t{ \"ICASE\",\t REG_ICASE\t},\n+\t{ \"NOTBOL\",\t REG_NOTBOL\t},\n+#ifdef REG_STARTEND\n+\t{ \"STARTEND\",\t REG_STARTEND\t},\n+#endif\n+\t{ NULL, 0 }\n+};\n \n int main(int argc, char **argv)\n {\n-\tchar *pat = \"[^={} \\t]+\";\n-\tchar *str = \"={}\\nfred\";\n+\tconst char *pat;\n+\tconst char *str;\n+\tint flags = 0;\n \tregex_t r;\n \tregmatch_t m[1];\n \n-\tif (regcomp(&r, pat, REG_EXTENDED | REG_NEWLINE))\n+\tif (argc == 1) {\n+\t\t/* special case, bug check */\n+\t\tpat = \"[^={} \\t]+\";\n+\t\tstr = \"={}\\nfred\";\n+\t\tflags = REG_EXTENDED | REG_NEWLINE;\n+\t} else {\n+\t\targv++;\n+\t\tpat = *argv++;\n+\t\tstr = *argv++;\n+\t\twhile (*argv) {\n+\t\t\tstruct reg_flag *rf;\n+\t\t\tfor (rf = reg_flags; rf->name; rf++)\n+\t\t\t\tif (!strcmp(*argv, rf->name)) {\n+\t\t\t\t\tflags |= rf->flag;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\tif (!rf->name)\n+\t\t\t\tdie(\"do not recognize %s\", *argv);\n+\t\t\targv++;\n+\t\t}\n+\t\tgit_setup_gettext();\n+\t}\n+\n+\tif (regcomp(&r, pat, flags))\n \t\tdie(\"failed regcomp() for pattern '%s'\", pat);\n-\tif (regexec(&r, str, 1, m, 0))\n-\t\tdie(\"no match of pattern '%s' to string '%s'\", pat, str);\n+\tif (regexec(&r, str, 1, m, 0)) {\n+\t\tif (argc == 1)\n+\t\t\tdie(\"no match of pattern '%s' to string '%s'\", pat, str);\n+\t\treturn 1;\n+\t}\n \n \t/* http://sourceware.org/bugzilla/show_bug.cgi?id=3957 */\n-\tif (m[0].rm_so == 3) /* matches '\\n' when it should not */\n+\tif (argc == 1 && m[0].rm_so == 3) /* matches '\\n' when it should not */\n \t\tdie(\"regex bug confirmed: re-build git with NO_REGEX=1\");\n \n \texit(0);\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268415","messageId":"1440161240-28554-5-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 04/10] grep/icase: avoid kwsset on literal non-ascii strings","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:14Z","receivedAt":"2015-08-21T12:47:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When we detect the pattern is just a literal string, we avoid heavy\nregex engine and use fast substring search implemented in kwsset.c.\nBut kws uses git-ctype which is locale-independent so it does not know\nhow to fold case properly outside ascii range. Let regcomp or pcre\ntake care of this case instead. Slower, but accurate.\n\nHelped-by: René Scharfe \nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 7 ++++++-\n t/t7812-grep-icase-non-ascii.sh (new +x) | 23 +++++++++++++++++++++++\n 2 files changed, 29 insertions(+), 1 deletion(-)\n create mode 100755 t/t7812-grep-icase-non-ascii.sh\n\ndiff --git a/grep.c b/grep.c\nindex bd32f66..d795b0e 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -4,6 +4,7 @@\n #include \"xdiff-interface.h\"\n #include \"diff.h\"\n #include \"diffcore.h\"\n+#include \"commit.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -398,12 +399,16 @@ static int is_fixed(const char *s, size_t len)\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n+\tint icase_non_ascii;\n \tint err;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n+\ticase_non_ascii =\n+\t\t(opt->regflags & REG_ICASE || p->ignore_case) &&\n+\t\thas_non_ascii(p->pattern);\n \n-\tif (is_fixed(p->pattern, p->patternlen))\n+\tif (!icase_non_ascii && is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse if (opt->fixed) {\n \t\tp->fixed = 1;\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nnew file mode 100755\nindex 0000000..6eff490\n--- /dev/null\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_LOCALE 'setup' '\n+\tprintf \"TILRAUN: Halló Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_have_prereq GETTEXT_LOCALE &&\n+test-regex \"HALLÓ\" \"Halló\" ICASE &&\n+test_set_prereq REGEX_LOCALE\n+\n+test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n+\tgit grep -i \"TILRAUN: Halló Heimur!\" &&\n+\tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268416","messageId":"1440161240-28554-6-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 05/10] grep/icase: avoid kwsset when -F is specified","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:15Z","receivedAt":"2015-08-21T12:47:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the previous commit, we can't use kws on icase search\noutside ascii range. But we can't simply pass the pattern to\nregcomp/pcre like the previous commit because it may contain regex\nspecial characters, so we need to quote the regex first.\n\nTo avoid misquote traps that could lead to undefined behavior, we\nalways stick to basic regex engine in this case. We don't need fancy\nfeatures for grepping a literal string anyway.\n\nbasic_regex_quote_buf() assumes that if the pattern is in a multibyte\nencoding, ascii chars must be unambiguously encoded as single\nbytes. This is true at least for UTF-8. For others, let's wait until\npeople yell up. Chances are nobody uses multibyte, non utf-8 charsets\nany more..\n\nHelped-by: René Scharfe \nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 25 ++++++++++++++++++++++++-\n quote.c | 37 +++++++++++++++++++++++++++++++++++++\n quote.h | 1 +\n t/t7812-grep-icase-non-ascii.sh | 26 ++++++++++++++++++++++++++\n 4 files changed, 88 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex d795b0e..8fce54f 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -5,6 +5,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"commit.h\"\n+#include \"quote.h\"\n \n static int grep_source_load(struct grep_source *gs);\n static int grep_source_is_binary(struct grep_source *gs);\n@@ -397,6 +398,24 @@ static int is_fixed(const char *s, size_t len)\n \treturn 1;\n }\n \n+static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint err;\n+\n+\tbasic_regex_quote_buf(&sb, p->pattern);\n+\terr = regcomp(&p->regexp, sb.buf, opt->regflags & ~REG_EXTENDED);\n+\tif (opt->debug)\n+\t\tfprintf(stderr, \"fixed%s\\n\", sb.buf);\n+\tstrbuf_release(&sb);\n+\tif (err) {\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, &p->regexp, errbuf, 1024);\n+\t\tregfree(&p->regexp);\n+\t\tcompile_regexp_failed(p, errbuf);\n+\t}\n+}\n+\n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tint icase_non_ascii;\n@@ -411,7 +430,11 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tif (!icase_non_ascii && is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = 1;\n \telse if (opt->fixed) {\n-\t\tp->fixed = 1;\n+\t\tp->fixed = !icase_non_ascii;\n+\t\tif (!p->fixed) {\n+\t\t\tcompile_fixed_regexp(p, opt);\n+\t\t\treturn;\n+\t\t}\n \t} else\n \t\tp->fixed = 0;\n \ndiff --git a/quote.c b/quote.c\nindex 7920e18..43a8057 100644\n--- a/quote.c\n+++ b/quote.c\n@@ -439,3 +439,40 @@ void tcl_quote_buf(struct strbuf *sb, const char *src)\n \t}\n \tstrbuf_addch(sb, '\"');\n }\n+\n+void basic_regex_quote_buf(struct strbuf *sb, const char *src)\n+{\n+\tchar c;\n+\n+\tif (*src == '^') {\n+\t\t/* only beginning '^' is special and needs quoting */\n+\t\tstrbuf_addch(sb, '\\\\');\n+\t\tstrbuf_addch(sb, *src++);\n+\t}\n+\tif (*src == '*')\n+\t\t/* beginning '*' is not special, no quoting */\n+\t\tstrbuf_addch(sb, *src++);\n+\n+\twhile ((c = *src++)) {\n+\t\tswitch (c) {\n+\t\tcase '[':\n+\t\tcase '.':\n+\t\tcase '\\\\':\n+\t\tcase '*':\n+\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tcase '$':\n+\t\t\t/* only the end '$' is special and needs quoting */\n+\t\t\tif (*src == '\\0')\n+\t\t\t\tstrbuf_addch(sb, '\\\\');\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\n+\t\tdefault:\n+\t\t\tstrbuf_addch(sb, c);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\ndiff --git a/quote.h b/quote.h\nindex 99e04d3..362d315 100644\n--- a/quote.h\n+++ b/quote.h\n@@ -67,5 +67,6 @@ extern char *quote_path_relative(const char *in, const char *prefix,\n extern void perl_quote_buf(struct strbuf *sb, const char *src);\n extern void python_quote_buf(struct strbuf *sb, const char *src);\n extern void tcl_quote_buf(struct strbuf *sb, const char *src);\n+extern void basic_regex_quote_buf(struct strbuf *sb, const char *src);\n \n #endif\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 6eff490..aba6b15 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -20,4 +20,30 @@ test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success REGEX_LOCALE 'grep literal string, with -F' '\n+\tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\techo \"fixedTILRAUN: Halló Heimur!\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"TILRAUN: HALLÓ HEIMUR!\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\techo \"fixedTILRAUN: HALLÓ HEIMUR!\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n+test_expect_success REGEX_LOCALE 'grep string with regex, with -F' '\n+\tprintf \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" >file &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\Halló \\$He[]imur!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug1 &&\n+\techo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\Halló \\$He\\\\[]imur!\\\\\\$\" >expect1 &&\n+\ttest_cmp expect1 debug1 &&\n+\n+\tgit grep --debug -i -F \"^*TILR^AUN:.* \\\\HALLÓ \\$HE[]IMUR!\\$\" 2>&1 >/dev/null |\n+\t\t grep fixed >debug2 &&\n+\techo \"fixed\\\\^*TILR^AUN:\\\\.\\\\* \\\\\\\\HALLÓ \\$HE\\\\[]IMUR!\\\\\\$\" >expect2 &&\n+\ttest_cmp expect2 debug2\n+'\n+\n test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268417","messageId":"1440161240-28554-7-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 06/10] grep/pcre: prepare locale-dependent tables for icase matching","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:16Z","receivedAt":"2015-08-21T12:47:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The default tables are usually built with C locale and only suitable\nfor LANG=C or similar. This should make case insensitive search work\ncorrectly for all single-byte charsets.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 8 ++++++--\n grep.h | 1 +\n t/t7813-grep-icase-iso.sh (new +x) | 19 +++++++++++++++++++\n 3 files changed, 26 insertions(+), 2 deletions(-)\n create mode 100755 t/t7813-grep-icase-iso.sh\n\ndiff --git a/grep.c b/grep.c\nindex 8fce54f..f0fbf99 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -324,11 +324,14 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \tint erroffset;\n \tint options = PCRE_MULTILINE;\n \n-\tif (opt->ignore_case)\n+\tif (opt->ignore_case) {\n+\t\tif (has_non_ascii(p->pattern))\n+\t\t\tp->pcre_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n+\t}\n \n \tp->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n-\t\t\tNULL);\n+\t\t\t\t p->pcre_tables);\n \tif (!p->pcre_regexp)\n \t\tcompile_regexp_failed(p, error);\n \n@@ -362,6 +365,7 @@ static void free_pcre_regexp(struct grep_pat *p)\n {\n \tpcre_free(p->pcre_regexp);\n \tpcre_free(p->pcre_extra_info);\n+\tpcre_free((void *)p->pcre_tables);\n }\n #else /* !USE_LIBPCRE */\n static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\ndiff --git a/grep.h b/grep.h\nindex 95f197a..cee4357 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -48,6 +48,7 @@ struct grep_pat {\n \tregex_t regexp;\n \tpcre *pcre_regexp;\n \tpcre_extra *pcre_extra_info;\n+\tconst unsigned char *pcre_tables;\n \tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\ndiff --git a/t/t7813-grep-icase-iso.sh b/t/t7813-grep-icase-iso.sh\nnew file mode 100755\nindex 0000000..efef7fb\n--- /dev/null\n+++ b/t/t7813-grep-icase-iso.sh\n@@ -0,0 +1,19 @@\n+#!/bin/sh\n+\n+test_description='grep icase on non-English locales'\n+\n+. ./lib-gettext.sh\n+\n+test_expect_success GETTEXT_ISO_LOCALE 'setup' '\n+\tprintf \"TILRAUN: Hall� Heimur!\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"$is_IS_iso_locale\" &&\n+\texport LC_ALL\n+'\n+\n+test_expect_success GETTEXT_ISO_LOCALE,LIBPCRE 'grep pcre string' '\n+\tgit grep --perl-regexp -i \"TILRAUN: H.ll� Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LL� HEIMUR!\"\n+'\n+\n+test_done\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268418","messageId":"1440161240-28554-8-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 07/10] gettext: add is_utf8_locale()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:17Z","receivedAt":"2015-08-21T12:47:17Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This function returns true if git is running under an UTF-8\nlocale. pcre in the next patch will need this.\n\nis_encoding_utf8() is used instead of strcmp() to catch both \"utf-8\"\nand \"utf8\" suffixes.\n\nWhen built with no gettext support, we peek in several env variables\nto detect UTF-8. pcre library might support utf-8 even if libc is\nbuilt without locale support.. The peeking code is a copy from\ncompat/regex/regcomp.c\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n gettext.c | 24 ++++++++++++++++++++++--\n gettext.h | 1 +\n 2 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/gettext.c b/gettext.c\nindex a268a2c..db727ea 100644\n--- a/gettext.c\n+++ b/gettext.c\n@@ -18,6 +18,8 @@\n #\tendif\n #endif\n \n+static const char *charset;\n+\n /*\n * Guess the user's preferred languages from the value in LANGUAGE environment\n * variable and LC_MESSAGES locale category if NO_GETTEXT is not defined.\n@@ -65,7 +67,6 @@ static int test_vsnprintf(const char *fmt, ...)\n \treturn ret;\n }\n \n-static const char *charset;\n static void init_gettext_charset(const char *domain)\n {\n \t/*\n@@ -172,8 +173,27 @@ int gettext_width(const char *s)\n {\n \tstatic int is_utf8 = -1;\n \tif (is_utf8 == -1)\n-\t\tis_utf8 = !strcmp(charset, \"UTF-8\");\n+\t\tis_utf8 = is_utf8_locale();\n \n \treturn is_utf8 ? utf8_strwidth(s) : strlen(s);\n }\n #endif\n+\n+int is_utf8_locale(void)\n+{\n+#ifdef NO_GETTEXT\n+\tif (!charset) {\n+\t\tconst char *env = getenv(\"LC_ALL\");\n+\t\tif (!env || !*env)\n+\t\t\tenv = getenv(\"LC_CTYPE\");\n+\t\tif (!env || !*env)\n+\t\t\tenv = getenv(\"LANG\");\n+\t\tif (!env)\n+\t\t\tenv = \"\";\n+\t\tif (strchr(env, '.'))\n+\t\t\tenv = strchr(env, '.') + 1;\n+\t\tcharset = xstrdup(env);\n+\t}\n+#endif\n+\treturn is_encoding_utf8(charset);\n+}\ndiff --git a/gettext.h b/gettext.h\nindex 33696a4..7eee64a 100644\n--- a/gettext.h\n+++ b/gettext.h\n@@ -90,5 +90,6 @@ const char *Q_(const char *msgid, const char *plu, unsigned long n)\n #endif\n \n const char *get_preferred_languages(void);\n+extern int is_utf8_locale(void);\n \n #endif\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268419","messageId":"1440161240-28554-9-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 08/10] grep/pcre: support utf-8","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:18Z","receivedAt":"2015-08-21T12:47:18Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In the previous change in this function, we add locale support for\nsingle-byte encodings only. It looks like pcre only supports utf-* as\nmultibyte encodings, the others are left in the cold (which is\nfine).\n\nWe need to enable PCRE_UTF8 so pcre can find character boundary\ncorrectly. It's needed for case folding (when --ignore-case is used)\nor '*', '+' or similar syntax is used.\n\nThe \"has_non_ascii()\" check is to be on the conservative side. If\nthere's non-ascii in the pattern, the searched content could still be\nin utf-8, but we can treat it just like a byte stream and everything\nshould work. If we force utf-8 based on locale only and pcre validates\nutf-8 and the file content is in non-utf8 encoding, things break.\n\nNoticed-by: Plamen Totev \nHelped-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n grep.c | 2 ++\n t/t7812-grep-icase-non-ascii.sh | 15 +++++++++++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/grep.c b/grep.c\nindex f0fbf99..07621c1 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -329,6 +329,8 @@ static void compile_pcre_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \t\t\tp->pcre_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n \t}\n+\tif (is_utf8_locale() && has_non_ascii(p->pattern))\n+\t\toptions |= PCRE_UTF8;\n \n \tp->pcre_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n \t\t\t\t p->pcre_tables);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex aba6b15..8896410 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -20,6 +20,21 @@ test_expect_success REGEX_LOCALE 'grep literal string, no -F' '\n \tgit grep -i \"TILRAUN: HALLÓ HEIMUR!\"\n '\n \n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 icase' '\n+\tgit grep --perl-regexp \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.lló Heimur!\" &&\n+\tgit grep --perl-regexp -i \"TILRAUN: H.LLÓ HEIMUR!\"\n+'\n+\n+test_expect_success GETTEXT_LOCALE,LIBPCRE 'grep pcre utf-8 string with \"+\"' '\n+\tprintf \"TILRAUN: Hallóó Heimur!\" >file2 &&\n+\tgit add file2 &&\n+\tgit grep -l --perl-regexp \"TILRAUN: H.lló+ Heimur!\" >actual &&\n+\techo file >expected &&\n+\techo file2 >>expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success REGEX_LOCALE 'grep literal string, with -F' '\n \tgit grep --debug -i -F \"TILRAUN: Halló Heimur!\" 2>&1 >/dev/null |\n \t\t grep fixed >debug1 &&\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268420","messageId":"1440161240-28554-10-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 09/10] diffcore-pickaxe: \"share\" regex error handling code","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:19Z","receivedAt":"2015-08-21T12:47:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"There's another regcomp code block coming in this function. By moving\nthe error handling code out of this block, we don't have to add the\nsame error handling code in the new block.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n diffcore-pickaxe.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 185f86b..7a718fc 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -204,20 +204,13 @@ void diffcore_pickaxe(struct diff_options *o)\n \tint opts = o->pickaxe_opts;\n \tregex_t regex, *regexp = NULL;\n \tkwset_t kws = NULL;\n+\tint err = 0;\n \n \tif (opts & (DIFF_PICKAXE_REGEX | DIFF_PICKAXE_KIND_G)) {\n-\t\tint err;\n \t\tint cflags = REG_EXTENDED | REG_NEWLINE;\n \t\tif (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE))\n \t\t\tcflags |= REG_ICASE;\n \t\terr = regcomp(®ex, needle, cflags);\n-\t\tif (err) {\n-\t\t\t/* The POSIX.2 people are surely sick */\n-\t\t\tchar errbuf[1024];\n-\t\t\tregerror(err, ®ex, errbuf, 1024);\n-\t\t\tregfree(®ex);\n-\t\t\tdie(\"invalid regex: %s\", errbuf);\n-\t\t}\n \t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n@@ -225,6 +218,13 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\tkwsincr(kws, needle, strlen(needle));\n \t\tkwsprep(kws);\n \t}\n+\tif (err) {\n+\t\t/* The POSIX.2 people are surely sick */\n+\t\tchar errbuf[1024];\n+\t\tregerror(err, ®ex, errbuf, 1024);\n+\t\tregfree(®ex);\n+\t\tdie(\"invalid regex: %s\", errbuf);\n+\t}\n \n \t/* Might want to warn when both S and G are on; I don't care... */\n \tpickaxe(&diff_queued_diff, o, regexp, kws,\n-- \n2.3.0.rc1.137.g477eb31\n"},{"id":"268421","messageId":"1440161240-28554-11-git-send-email-pclouds@gmail.com","threadId":"39783","inReplyTo":"1440161240-28554-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 10/10] diffcore-pickaxe: support case insensitive match on non-ascii","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2015-08-21T12:47:20Z","receivedAt":"2015-08-21T12:47:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Similar to the \"grep -F -i\" case, we can't use kws on icase search\noutside ascii range, so we quote the string and pass it to regcomp as\na basic regexp and let regex engine deal with case sensitivity.\n\nThe new test is put in t7812 instead of t4209-log-pickaxe because\nlib-gettext.sh might cause problems elsewhere, probably..\n\nNoticed-by: Plamen Totev \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n diffcore-pickaxe.c | 11 +++++++++++\n t/t7812-grep-icase-non-ascii.sh | 7 +++++++\n 2 files changed, 18 insertions(+)\n\ndiff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c\nindex 7a718fc..6946c15 100644\n--- a/diffcore-pickaxe.c\n+++ b/diffcore-pickaxe.c\n@@ -7,6 +7,8 @@\n #include \"diffcore.h\"\n #include \"xdiff-interface.h\"\n #include \"kwset.h\"\n+#include \"commit.h\"\n+#include \"quote.h\"\n \n typedef int (*pickaxe_fn)(mmfile_t *one, mmfile_t *two,\n \t\t\t struct diff_options *o,\n@@ -212,6 +214,15 @@ void diffcore_pickaxe(struct diff_options *o)\n \t\t\tcflags |= REG_ICASE;\n \t\terr = regcomp(®ex, needle, cflags);\n \t\tregexp = ®ex;\n+\t} else if (DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE) &&\n+\t\t has_non_ascii(needle)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tint cflags = REG_NEWLINE | REG_ICASE;\n+\n+\t\tbasic_regex_quote_buf(&sb, needle);\n+\t\terr = regcomp(®ex, sb.buf, cflags);\n+\t\tstrbuf_release(&sb);\n+\t\tregexp = ®ex;\n \t} else {\n \t\tkws = kwsalloc(DIFF_OPT_TST(o, PICKAXE_IGNORE_CASE)\n \t\t\t ? tolower_trans_tbl : NULL);\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex 8896410..a5475bb 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -61,4 +61,11 @@ test_expect_success REGEX_LOCALE 'grep string with regex, with -F' '\n \ttest_cmp expect2 debug2\n '\n \n+test_expect_success REGEX_LOCALE 'pickaxe -i on non-ascii' '\n+\tgit commit -m first &&\n+\tgit log --format=%f -i -S\"TILRAUN: HALLÓ HEIMUR!\" >actual &&\n+\techo first >expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.3.0.rc1.137.g477eb31\n"}]}