{"thread":{"id":"59029","subject":"BUG: git grep behave oddly with alternatives","startedAt":"2023-01-03T09:54:06Z","lastAt":"2023-01-14T16:09:04Z","messageCount":22,"participants":["Marco Nenciarini","René Scharfe","Junio C Hamano","Jeff King","Ævar Arnfjörð Bjarmason","Diomidis Spinellis"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"469762","messageId":"f82ae28a-fb56-8d1f-96c8-550b61439d3a@enterprisedb.com","threadId":"59029","inReplyTo":null,"subject":"BUG: git grep behave oddly with alternatives","fromName":"Marco Nenciarini","fromEmail":"marco.nenciarini@enterprisedb.com","sentAt":"2023-01-03T09:53:42Z","receivedAt":"2023-01-03T09:54:06Z","isPatch":false,"sender":{"key":"marco.nenciarini@enterprisedb.com","avatar":null},"body":"I've installed latest git from brew and suddenly git grep started \nbehaving oddly when using alternatives.\n\n```\n$ echo abd > test.file\n$ git grep --untracked 'a\\(b\\|c\\)d' test.file\n$ git grep --untracked 'a\\(b\\)d' test.file\ntest.file:abd\n```\n\nIt should have matched in both cases.\n\n\nIf I switch to exented pattern it starts working again\n\n```\n$ git grep --untracked -E 'a(b|c)d' test.file\ntest.file:abd\n```\n\n[System Info]\ngit version:\ngit version 2.39.0\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nfeature: fsmonitor--daemon\nuname: Darwin 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 \nPST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64 x86_64\ncompiler info: clang: 14.0.0 (clang-1400.0.29.202)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /usr/local/bin/bash\n\n\n-- \nMarco Nenciarini - EnterpriseDB\nmarco.nenciarini@enterprisedb.com | www.enterprisedb.com\n"},{"id":"469770","messageId":"634f47a8-b370-81cb-00e7-d93ffc7534a8@web.de","threadId":"59029","inReplyTo":"f82ae28a-fb56-8d1f-96c8-550b61439d3a@enterprisedb.com","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-01-03T16:29:51Z","receivedAt":"2023-01-03T16:30:29Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 03.01.23 um 10:53 schrieb Marco Nenciarini:\n> I've installed latest git from brew and suddenly git grep started behaving oddly when using alternatives.\n>\n> ```\n> $ echo abd > test.file\n> $ git grep --untracked 'a\\(b\\|c\\)d' test.file\n> $ git grep --untracked 'a\\(b\\)d' test.file\n> test.file:abd\n> ```\n>\n> It should have matched in both cases.\n>\n>\n> If I switch to exented pattern it starts working again\n>\n> ```\n> $ git grep --untracked -E 'a(b|c)d' test.file\n> test.file:abd\n> ```\n\nThis is expected: Basic regular expressions don't support alternation.\n\nUnder which circumstances did it work for you without -E or -P?\n\n>\n> [System Info]\n> git version:\n> git version 2.39.0\n> cpu: x86_64\n> no commit associated with this build\n> sizeof-long: 8\n> sizeof-size_t: 8\n> shell-path: /bin/sh\n> feature: fsmonitor--daemon\n> uname: Darwin 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 PST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64 x86_64\n> compiler info: clang: 14.0.0 (clang-1400.0.29.202)\n> libc info: no libc information available\n> $SHELL (typically, interactive shell): /usr/local/bin/bash\n>\n>\n\n"},{"id":"469774","messageId":"1f61b660-b2d0-ba93-3182-05a9ab97b00e@enterprisedb.com","threadId":"59029","inReplyTo":"634f47a8-b370-81cb-00e7-d93ffc7534a8@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Marco Nenciarini","fromEmail":"marco.nenciarini@enterprisedb.com","sentAt":"2023-01-03T18:13:03Z","receivedAt":"2023-01-03T18:13:18Z","isPatch":false,"sender":{"key":"marco.nenciarini@enterprisedb.com","avatar":null},"body":"On 03/01/23 17:29, René Scharfe wrote:\n> Am 03.01.23 um 10:53 schrieb Marco Nenciarini:\n>> I've installed latest git from brew and suddenly git grep started behaving oddly when using alternatives.\n>>\n>> ```\n>> $ echo abd > test.file\n>> $ git grep --untracked 'a\\(b\\|c\\)d' test.file\n>> $ git grep --untracked 'a\\(b\\)d' test.file\n>> test.file:abd\n>> ```\n>>\n>> It should have matched in both cases.\n>>\n>>\n>> If I switch to exented pattern it starts working again\n>>\n>> ```\n>> $ git grep --untracked -E 'a(b|c)d' test.file\n>> test.file:abd\n>> ```\n> \n> This is expected: Basic regular expressions don't support alternation.\n> \n> Under which circumstances did it work for you without -E or -P?\n> \n\nIt has always worked until I installed 2.39.0 on my mac. I've also \nchecked with other developers that are using a mac and they reports the \nsame. On Linux it works regardless the git version.\n\nUsing grep directly also works, so it is a different behavior between \ngrep and git grep, that is surprising.\n\n>>\n>> [System Info]\n>> git version:\n>> git version 2.39.0\n>> cpu: x86_64\n>> no commit associated with this build\n>> sizeof-long: 8\n>> sizeof-size_t: 8\n>> shell-path: /bin/sh\n>> feature: fsmonitor--daemon\n>> uname: Darwin 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 PST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64 x86_64\n>> compiler info: clang: 14.0.0 (clang-1400.0.29.202)\n>> libc info: no libc information available\n>> $SHELL (typically, interactive shell): /usr/local/bin/bash\n>>\n>>\n> \n\n-- \nMarco Nenciarini - EnterpriseDB\nmarco.nenciarini@enterprisedb.com | www.enterprisedb.com\n\n"},{"id":"469776","messageId":"343a891e-d737-0ace-26a9-3839d3bd5583@web.de","threadId":"59029","inReplyTo":"1f61b660-b2d0-ba93-3182-05a9ab97b00e@enterprisedb.com","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-01-03T20:52:27Z","receivedAt":"2023-01-03T20:52:42Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 03.01.23 um 19:13 schrieb Marco Nenciarini:\n> On 03/01/23 17:29, René Scharfe wrote:\n>> Am 03.01.23 um 10:53 schrieb Marco Nenciarini:\n>>> I've installed latest git from brew and suddenly git grep started behaving oddly when using alternatives.\n>>>\n>>> ```\n>>> $ echo abd > test.file\n>>> $ git grep --untracked 'a\\(b\\|c\\)d' test.file\n>>> $ git grep --untracked 'a\\(b\\)d' test.file\n>>> test.file:abd\n>>> ```\n>>>\n>>> It should have matched in both cases.\n>>>\n>>>\n>>> If I switch to exented pattern it starts working again\n>>>\n>>> ```\n>>> $ git grep --untracked -E 'a(b|c)d' test.file\n>>> test.file:abd\n>>> ```\n>>\n>> This is expected: Basic regular expressions don't support alternation.\n>>\n>> Under which circumstances did it work for you without -E or -P?\n>>\n>\n> It has always worked until I installed 2.39.0 on my mac. I've also checked with other developers that are using a mac and they reports the same. On Linux it works regardless the git version.\n>\n> Using grep directly also works, so it is a different behavior between grep and git grep, that is surprising.\n\nMeaning you used Apple's version of git before?\n\n   $ echo abd > test.file\n   $ /usr/bin/git grep --untracked 'b\\|c' test.file\n   test.file:abd\n   $ /usr/bin/git version\n   git version 2.37.1 (Apple Git-137.1)\n\n   $ git grep --untracked 'b\\|c' test.file\n   $ git version\n   git version 2.39.0\n\nSo I can reproduce your findings on macOS Ventura.  Same with grep:\n\n   $ grep 'b\\|c' test.file\n   abd\n   $ grep --version\n   grep (BSD grep, GNU compatible) 2.6.0-FreeBSD\n\nre_format(7) says:\n\n   \"Obsolete (“basic”) regular expressions differ in several respects.\n    ‘|’ is an ordinary character and there is no equivalent for its\n    functionality.\".\n\nUnder the headline \"ENHANCED FEATURES\" it continues, however:\n\n   \"When the REG_ENHANCED flag is passed to one of the regcomp()\n    variants, additional features are activated.\"\n\nAnd:\n\n   \"For enhanced basic REs, ‘+’, ‘?’ and ‘|’ remain regular characters,\n    but ‘\\+’, ‘\\?’ and ‘\\|’ have the same special meaning as the\n    unescaped characters do for extended REs, i.e., one or more\n    matches, zero or one matches and alteration, respectively.\"\n\nSo apparently Apple chose a middle ground between basic and extended\nregular expressions for its grep and git grep.\n\nUnder \"IMPLEMENTATION CHOICES\" it says:\n\n   \"The regex implementation in Mac OS X 10.8 and later is based on a\n    heavily modified subset of TRE (http://laurikari.net/tre/).\"\n\nThe manpage of GNU grep says:\n\n   \"grep understands three different versions of regular expression\n    syntax: “basic” (BRE), “extended” (ERE) and “perl” (PCRE).  In\n    GNU grep there is no difference in available functionality\n    between basic and extended syntax.  In other implementations,\n    basic regular expressions are less powerful.\"\n\nAnd under the headline \"Basic vs Extended Regular Expressions\":\n\n   \"In basic regular expressions the meta-characters ?, +, {, |, (,\n    and ) lose their special meaning; instead use the backslashed\n    versions \\?, \\+, \\{, \\|, \\(, and \\).\"\n\nSo GNU grep apparently made the same choice as Apple, probably far\nearlier.\n\nWhen I compile git with NO_REGEX and thus with the fallback in\ncompat/regex/, the result supports alternation as well:\n\n   $ ./git grep --untracked 'b\\|c' test.file\n   test.file:abd\n   $ nm ./git | grep regcomp\n   0000000100255978 T _git_regcomp\n\nBased on that I suggest:\n\n--- >8 ---\nSubject: grep: use REG_ENHANCED on macOS\n\nGNU grep(1) and regcomp(3) use enhanced basic regular expressions by\ndefault, which means that it e.g. supports alternation, e.g. \"a\\|b\"\nmatches both \"a\" and \"b\".  The same is true for our compat/regex/\nimplementation.\n\nOn macOS Ventura (and possibly earlier) grep(1) also uses enhanced BREs,\nbut regcomp(3) requires the flag REG_ENHANCED to turn them on.  Do that\nfor git grep if possible, for consistency with the platform's grep(1)\nand our fallback library.\n\nIt would be simpler to use REG_ENHANCED everywhere it is defined instead\nof adding a Makefile setting, but it's a non-standard extension and\nmight mean something else on other platforms.\n\nReported-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>\nSigned-off-by: René Scharfe <l.s.r@web.de>\n---\n Makefile         | 8 ++++++++\n config.mak.uname | 1 +\n grep.c           | 4 ++++\n 3 files changed, 13 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex db447d0738..15e7edc9d2 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -289,6 +289,10 @@ include shared.mak\n # Define NO_REGEX if your C library lacks regex support with REG_STARTEND\n # feature.\n #\n+# Define GIT_GREP_USES_REG_ENHANCED if your C library provides the flag\n+# REG_ENHANCED to enable enhanced basic regular expressions and you'd\n+# like to use it in git grep.\n+#\n # Define HAVE_DEV_TTY if your system can open /dev/tty to interact with the\n # user.\n #\n@@ -2037,6 +2041,10 @@ endif\n ifdef NO_REGEX\n \tCOMPAT_CFLAGS += -Icompat/regex\n \tCOMPAT_OBJS += compat/regex/regex.o\n+else\n+ifdef GIT_GREP_USES_REG_ENHANCED\n+\tCOMPAT_CFLAGS += -DGIT_GREP_USES_REG_ENHANCED\n+endif\n endif\n ifdef NATIVE_CRLF\n \tBASIC_CFLAGS += -DNATIVE_CRLF\ndiff --git a/config.mak.uname b/config.mak.uname\nindex d63629fe80..14c145ae42 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -147,6 +147,7 @@ ifeq ($(uname_S),Darwin)\n \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n \tHAVE_NS_GET_EXECUTABLE_PATH = YesPlease\n \tCSPRNG_METHOD = arc4random\n+\tGIT_GREP_USES_REG_ENHANCED = YesPlease\n\n \t# Workaround for `gettext` being keg-only and not even being linked via\n \t# `brew link --force gettext`, should be obsolete as of\ndiff --git a/grep.c b/grep.c\nindex 06eed69493..a8430daaba 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -502,6 +502,10 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tregflags |= REG_ICASE;\n \tif (opt->pattern_type_option == GREP_PATTERN_TYPE_ERE)\n \t\tregflags |= REG_EXTENDED;\n+#if defined(GIT_GREP_USES_REG_ENHANCED) && defined(REG_ENHANCED)\n+\telse\n+\t\tregflags |= REG_ENHANCED;\n+#endif\n \terr = regcomp(&p->regexp, p->pattern, regflags);\n \tif (err) {\n \t\tchar errbuf[1024];\n--\n2.39.0\n"},{"id":"469779","messageId":"xmqq8riivncm.fsf@gitster.g","threadId":"59029","inReplyTo":"343a891e-d737-0ace-26a9-3839d3bd5583@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-04T06:13:29Z","receivedAt":"2023-01-04T06:13:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n>    \"For enhanced basic REs, ‘+’, ‘?’ and ‘|’ remain regular characters,\n>     but ‘\\+’, ‘\\?’ and ‘\\|’ have the same special meaning as the\n>     unescaped characters do for extended REs, i.e., one or more\n>     matches, zero or one matches and alteration, respectively.\"\n>\n> So apparently Apple chose a middle ground between basic and extended\n> regular expressions for its grep and git grep.\n\nSounds like GNU extension to me.\n\n> So GNU grep apparently made the same choice as Apple, probably far\n> earlier.\n\nYup.\n\n> Based on that I suggest:\n> ...\n>\n> It would be simpler to use REG_ENHANCED everywhere it is defined instead\n> of adding a Makefile setting, but it's a non-standard extension and\n> might mean something else on other platforms.\n\nOK.  Very conservative and good.\n\n> Reported-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>\n> Signed-off-by: René Scharfe <l.s.r@web.de>\n> ---\n>  Makefile         | 8 ++++++++\n>  config.mak.uname | 1 +\n>  grep.c           | 4 ++++\n>  3 files changed, 13 insertions(+)\n>\n> diff --git a/Makefile b/Makefile\n> index db447d0738..15e7edc9d2 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -289,6 +289,10 @@ include shared.mak\n>  # Define NO_REGEX if your C library lacks regex support with REG_STARTEND\n>  # feature.\n>  #\n> +# Define GIT_GREP_USES_REG_ENHANCED if your C library provides the flag\n> +# REG_ENHANCED to enable enhanced basic regular expressions and you'd\n> +# like to use it in git grep.\n> +#\n>  # Define HAVE_DEV_TTY if your system can open /dev/tty to interact with the\n>  # user.\n>  #\n> @@ -2037,6 +2041,10 @@ endif\n>  ifdef NO_REGEX\n>  \tCOMPAT_CFLAGS += -Icompat/regex\n>  \tCOMPAT_OBJS += compat/regex/regex.o\n> +else\n> +ifdef GIT_GREP_USES_REG_ENHANCED\n> +\tCOMPAT_CFLAGS += -DGIT_GREP_USES_REG_ENHANCED\n> +endif\n>  endif\n>  ifdef NATIVE_CRLF\n>  \tBASIC_CFLAGS += -DNATIVE_CRLF\n> diff --git a/config.mak.uname b/config.mak.uname\n> index d63629fe80..14c145ae42 100644\n> --- a/config.mak.uname\n> +++ b/config.mak.uname\n> @@ -147,6 +147,7 @@ ifeq ($(uname_S),Darwin)\n>  \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n>  \tHAVE_NS_GET_EXECUTABLE_PATH = YesPlease\n>  \tCSPRNG_METHOD = arc4random\n> +\tGIT_GREP_USES_REG_ENHANCED = YesPlease\n>\n>  \t# Workaround for `gettext` being keg-only and not even being linked via\n>  \t# `brew link --force gettext`, should be obsolete as of\n> diff --git a/grep.c b/grep.c\n> index 06eed69493..a8430daaba 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -502,6 +502,10 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n>  \t\tregflags |= REG_ICASE;\n>  \tif (opt->pattern_type_option == GREP_PATTERN_TYPE_ERE)\n>  \t\tregflags |= REG_EXTENDED;\n> +#if defined(GIT_GREP_USES_REG_ENHANCED) && defined(REG_ENHANCED)\n> +\telse\n> +\t\tregflags |= REG_ENHANCED;\n> +#endif\n>  \terr = regcomp(&p->regexp, p->pattern, regflags);\n>  \tif (err) {\n>  \t\tchar errbuf[1024];\n> --\n> 2.39.0\n"},{"id":"469783","messageId":"Y7Uu35HwUx2EVfAg@coredump.intra.peff.net","threadId":"59029","inReplyTo":"343a891e-d737-0ace-26a9-3839d3bd5583@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-04T07:46:39Z","receivedAt":"2023-01-04T07:46:50Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 03, 2023 at 09:52:27PM +0100, René Scharfe wrote:\n\n> diff --git a/Makefile b/Makefile\n> index db447d0738..15e7edc9d2 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -289,6 +289,10 @@ include shared.mak\n>  # Define NO_REGEX if your C library lacks regex support with REG_STARTEND\n>  # feature.\n>  #\n> +# Define GIT_GREP_USES_REG_ENHANCED if your C library provides the flag\n> +# REG_ENHANCED to enable enhanced basic regular expressions and you'd\n> +# like to use it in git grep.\n\nI didn't test, but just from looking at the patch I'd expect this to\naffect other parts of Git besides git-grep. E.g., \"git log --grep\".\nWhich raises two questions:\n\n - would a more generalized name be better? USE_REG_ENHANCED or\n   something? That might be _too_ general, but see below.\n\n - should this cover other cases? Grepping for \"regcomp\", would people\n   want this to behave consistently for \"git config --get-regexp\", or\n   diff funcnames, and so on?\n\nIf so, then I could envision a USE_REG_ENHANCED which just wraps the\nsystem regcomp and adds the REG_ENHANCED flag when REG_EXTENDED is not\nset?\n\n-Peff\n"},{"id":"469790","messageId":"e5165840-331c-e9b6-b45f-62abab860d79@web.de","threadId":"59029","inReplyTo":"Y7Uu35HwUx2EVfAg@coredump.intra.peff.net","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-01-04T16:36:21Z","receivedAt":"2023-01-04T16:37:11Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 04.01.23 um 08:46 schrieb Jeff King:\n> On Tue, Jan 03, 2023 at 09:52:27PM +0100, René Scharfe wrote:\n>\n>> diff --git a/Makefile b/Makefile\n>> index db447d0738..15e7edc9d2 100644\n>> --- a/Makefile\n>> +++ b/Makefile\n>> @@ -289,6 +289,10 @@ include shared.mak\n>>  # Define NO_REGEX if your C library lacks regex support with REG_STARTEND\n>>  # feature.\n>>  #\n>> +# Define GIT_GREP_USES_REG_ENHANCED if your C library provides the flag\n>> +# REG_ENHANCED to enable enhanced basic regular expressions and you'd\n>> +# like to use it in git grep.\n>\n> I didn't test, but just from looking at the patch I'd expect this to\n> affect other parts of Git besides git-grep. E.g., \"git log --grep\".\n> Which raises two questions:\n>\n>  - would a more generalized name be better? USE_REG_ENHANCED or\n>    something? That might be _too_ general, but see below.\n>\n>  - should this cover other cases? Grepping for \"regcomp\", would people\n>    want this to behave consistently for \"git config --get-regexp\", or\n>    diff funcnames, and so on?\n>\n> If so, then I could envision a USE_REG_ENHANCED which just wraps the\n> system regcomp and adds the REG_ENHANCED flag when REG_EXTENDED is not\n> set?\n\nGood point.  I don't know what people want, though.  re_format(7) on\nmacOS/BSD and regex(7) on Linux call basic REs \"obsolete\" and extended\nREs \"modern\", so they seem to push people away from the old kind,\nenhanced or not.\n\nBut making a consistent choice for all regex use makes sense --\nplatforms that use compat/regex/ get the same enhanced flavor\neverywhere.\n\nRené\n"},{"id":"469819","messageId":"Y7flVcALZQgz0VPl@coredump.intra.peff.net","threadId":"59029","inReplyTo":"e5165840-331c-e9b6-b45f-62abab860d79@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-06T09:09:41Z","receivedAt":"2023-01-06T09:09:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 04, 2023 at 05:36:21PM +0100, René Scharfe wrote:\n\n> > I didn't test, but just from looking at the patch I'd expect this to\n> > affect other parts of Git besides git-grep. E.g., \"git log --grep\".\n> > Which raises two questions:\n> >\n> >  - would a more generalized name be better? USE_REG_ENHANCED or\n> >    something? That might be _too_ general, but see below.\n> >\n> >  - should this cover other cases? Grepping for \"regcomp\", would people\n> >    want this to behave consistently for \"git config --get-regexp\", or\n> >    diff funcnames, and so on?\n> >\n> > If so, then I could envision a USE_REG_ENHANCED which just wraps the\n> > system regcomp and adds the REG_ENHANCED flag when REG_EXTENDED is not\n> > set?\n> \n> Good point.  I don't know what people want, though.  re_format(7) on\n> macOS/BSD and regex(7) on Linux call basic REs \"obsolete\" and extended\n> REs \"modern\", so they seem to push people away from the old kind,\n> enhanced or not.\n\nOh, good point. I was just grepping for regcomp(), but of course any\ncase which is already passing REG_EXTENDED would not be affected anyway.\nAnd most places are already using that. E.g., the config code always\ndoes so, and it looks like pickaxe \"-G\" does so.\n\nFor diffs, we have diff.*.xfuncname, which uses EREs. We do still\nsupport regular \"funcname\" for backwards compatibility, but we only\ndocument the extended version. Ironically, that option was introduced\nbecause BREs did not portably support things like alternation, even with\nthe \"enhanced\" syntax. ;) See 45d9414fa5 (diff.*.xfuncname which uses\n\"extended\" regex's for hunk header selection, 2008-09-18).\n\nSo I think we are embracing the \"everyone should use EREs\" mentality\nalready. The only spots I see that use BREs are:\n\n  - grep.c, which handles \"git grep\" and \"git log --grep\"\n\n  - line-range.c, presumably for \"-L\" function matching\n\n  - deprecated non-ERE funcname patterns\n\nYour patch is handling the first, which is by the far most important. I\nwould be OK leaving the others as-is, but I also wouldn't mind a patch\nthat works at the regcomp() level to make things automatically\nconsistent.\n\n-Peff\n"},{"id":"469895","messageId":"26a0d4ca-3d97-ace4-1a1f-92b1ee6715a6@web.de","threadId":"59029","inReplyTo":"Y7flVcALZQgz0VPl@coredump.intra.peff.net","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-01-08T00:42:04Z","receivedAt":"2023-01-08T00:42:25Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 06.01.23 um 10:09 schrieb Jeff King:\n> On Wed, Jan 04, 2023 at 05:36:21PM +0100, René Scharfe wrote:\n>\n>>> I didn't test, but just from looking at the patch I'd expect this to\n>>> affect other parts of Git besides git-grep. E.g., \"git log --grep\".\n>>> Which raises two questions:\n>>>\n>>>  - would a more generalized name be better? USE_REG_ENHANCED or\n>>>    something? That might be _too_ general, but see below.\n>>>\n>>>  - should this cover other cases? Grepping for \"regcomp\", would people\n>>>    want this to behave consistently for \"git config --get-regexp\", or\n>>>    diff funcnames, and so on?\n>>>\n>>> If so, then I could envision a USE_REG_ENHANCED which just wraps the\n>>> system regcomp and adds the REG_ENHANCED flag when REG_EXTENDED is not\n>>> set?\n>>\n>> Good point.  I don't know what people want, though.  re_format(7) on\n>> macOS/BSD and regex(7) on Linux call basic REs \"obsolete\" and extended\n>> REs \"modern\", so they seem to push people away from the old kind,\n>> enhanced or not.\n>\n> Oh, good point. I was just grepping for regcomp(), but of course any\n> case which is already passing REG_EXTENDED would not be affected anyway.\n> And most places are already using that. E.g., the config code always\n> does so, and it looks like pickaxe \"-G\" does so.\n>\n> For diffs, we have diff.*.xfuncname, which uses EREs. We do still\n> support regular \"funcname\" for backwards compatibility, but we only\n> document the extended version. Ironically, that option was introduced\n> because BREs did not portably support things like alternation, even with\n> the \"enhanced\" syntax. ;) See 45d9414fa5 (diff.*.xfuncname which uses\n> \"extended\" regex's for hunk header selection, 2008-09-18).\n>\n> So I think we are embracing the \"everyone should use EREs\" mentality\n> already. The only spots I see that use BREs are:\n>\n>   - grep.c, which handles \"git grep\" and \"git log --grep\"\n>\n>   - line-range.c, presumably for \"-L\" function matching\n>\n>   - deprecated non-ERE funcname patterns\n>\n> Your patch is handling the first, which is by the far most important. I\n> would be OK leaving the others as-is, but I also wouldn't mind a patch\n> that works at the regcomp() level to make things automatically\n> consistent.\n\nThere's also the code handling \"git log -i -S nonäscii\" in\ndiffcore_pickaxe(), but it quotes special characters and thus\neffectively does a fixed-string search, so you're right in omitting it\nabove.\n\nREG_ENHANCED on macOS affects REG_EXTENDED as well.  It unlocks e.g.\nnon-greedy repetitions and inline comments.  Sounds nice, but also\npotentially surprising.  I was unable to find a current version of\nthe re_format(7) manpage online, unfortunately.\n\nApple's latest version of Git sets NO_REGEX and thus uses\ncompat/regex, if I read their source correctly:\n\nhttps://github.com/apple-oss-distributions/Git/blob/Git-128/src/git/Makefile#L1302\n\nThe easiest and most consistent option would be to do the same.  But\nwe can't do that, because it would break multibyte support, which was\nfixed by 1819ad327b (grep: fix multibyte regex handling under macOS,\n2022-08-26), which started to use the system regex functions again.\n\nWhich begs the question: Isn't that a problem for the platforms that\nstill have to use NO_REGEX?  Shouldn't we fix compat/regex?\n\nAnyway, here's an attempt at a more general, but still targeted fix\nfor macOS:\n\n--- >8 ---\nSubject: [PATCH] use enhanced basic regular expressions on macOS\n\nWhen 1819ad327b (grep: fix multibyte regex handling under macOS,\n2022-08-26) started to use the native regex library instead of Git's\nown (compat/regex/), it lost support for alternation in basic\nregular expressions.\n\nBring it back by enabling the flag REG_ENHANCED on macOS when\ncompiling basic regular expressions.\n\nReported-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>\nSuggested-by: Jeff King <peff@peff.net>\nSigned-off-by: René Scharfe <l.s.r@web.de>\n---\n Makefile                  | 9 +++++++++\n compat/regcomp_enhanced.c | 9 +++++++++\n config.mak.uname          | 1 +\n git-compat-util.h         | 5 +++++\n 4 files changed, 24 insertions(+)\n create mode 100644 compat/regcomp_enhanced.c\n\ndiff --git a/Makefile b/Makefile\nindex db447d0738..46e30be673 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -289,6 +289,10 @@ include shared.mak\n # Define NO_REGEX if your C library lacks regex support with REG_STARTEND\n # feature.\n #\n+# Define USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS if your C library provides\n+# the flag REG_ENHANCED and you'd like to use it to enable enhanced basic\n+# regular expressions.\n+#\n # Define HAVE_DEV_TTY if your system can open /dev/tty to interact with the\n # user.\n #\n@@ -2037,6 +2041,11 @@ endif\n ifdef NO_REGEX\n \tCOMPAT_CFLAGS += -Icompat/regex\n \tCOMPAT_OBJS += compat/regex/regex.o\n+else\n+ifdef USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS\n+\tCOMPAT_CFLAGS += -DUSE_ENHANCED_BASIC_REGULAR_EXPRESSIONS\n+\tCOMPAT_OBJS += compat/regcomp_enhanced.o\n+endif\n endif\n ifdef NATIVE_CRLF\n \tBASIC_CFLAGS += -DNATIVE_CRLF\ndiff --git a/compat/regcomp_enhanced.c b/compat/regcomp_enhanced.c\nnew file mode 100644\nindex 0000000000..84193ce53b\n--- /dev/null\n+++ b/compat/regcomp_enhanced.c\n@@ -0,0 +1,9 @@\n+#include \"../git-compat-util.h\"\n+#undef regcomp\n+\n+int git_regcomp(regex_t *preg, const char *pattern, int cflags)\n+{\n+\tif (!(cflags & REG_EXTENDED))\n+\t\tcflags |= REG_ENHANCED;\n+\treturn regcomp(preg, pattern, cflags);\n+}\ndiff --git a/config.mak.uname b/config.mak.uname\nindex d63629fe80..7d25995265 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -147,6 +147,7 @@ ifeq ($(uname_S),Darwin)\n \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n \tHAVE_NS_GET_EXECUTABLE_PATH = YesPlease\n \tCSPRNG_METHOD = arc4random\n+\tUSE_ENHANCED_BASIC_REGULAR_EXPRESSIONS = YesPlease\n\n \t# Workaround for `gettext` being keg-only and not even being linked via\n \t# `brew link --force gettext`, should be obsolete as of\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 76e4b11131..1efa834089 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1338,6 +1338,11 @@ static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,\n \treturn regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);\n }\n\n+#ifdef USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS\n+int git_regcomp(regex_t *preg, const char *pattern, int cflags);\n+#define regcomp git_regcomp\n+#endif\n+\n #ifndef DIR_HAS_BSD_GROUP_SEMANTICS\n # define FORCE_DIR_SET_GID S_ISGID\n #else\n--\n2.39.0\n"},{"id":"469896","messageId":"xmqqpmbp24td.fsf@gitster.g","threadId":"59029","inReplyTo":"26a0d4ca-3d97-ace4-1a1f-92b1ee6715a6@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-08T01:27:58Z","receivedAt":"2023-01-08T01:28:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n> diff --git a/Makefile b/Makefile\n> index db447d0738..46e30be673 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -289,6 +289,10 @@ include shared.mak\n>  # Define NO_REGEX if your C library lacks regex support with REG_STARTEND\n>  # feature.\n>  #\n> +# Define USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS if your C library provides\n> +# the flag REG_ENHANCED and you'd like to use it to enable enhanced basic\n> +# regular expressions.\n> +#\n\nI wondered if we should mention macOS somewhere in the description\nto help those users that may be affeced, but it seems that looking\nfor \"REG_ENHANCED BSD\" with a search engine finds pages that\nindicate this is available on BSD's in general?\n\n> @@ -2037,6 +2041,11 @@ endif\n>  ifdef NO_REGEX\n>  \tCOMPAT_CFLAGS += -Icompat/regex\n>  \tCOMPAT_OBJS += compat/regex/regex.o\n> +else\n> +ifdef USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS\n> +\tCOMPAT_CFLAGS += -DUSE_ENHANCED_BASIC_REGULAR_EXPRESSIONS\n> +\tCOMPAT_OBJS += compat/regcomp_enhanced.o\n> +endif\n\nOK.\n\n> diff --git a/compat/regcomp_enhanced.c b/compat/regcomp_enhanced.c\n> new file mode 100644\n> index 0000000000..84193ce53b\n> --- /dev/null\n> +++ b/compat/regcomp_enhanced.c\n> @@ -0,0 +1,9 @@\n> +#include \"../git-compat-util.h\"\n> +#undef regcomp\n> +int git_regcomp(regex_t *preg, const char *pattern, int cflags)\n> +{\n> +\tif (!(cflags & REG_EXTENDED))\n> +\t\tcflags |= REG_ENHANCED;\n> +\treturn regcomp(preg, pattern, cflags);\n> +}\n\nOK.  I like the \"we only want to affect BRE\" bit, that is carefully\ndone.\n\n> diff --git a/config.mak.uname b/config.mak.uname\n> index d63629fe80..7d25995265 100644\n> --- a/config.mak.uname\n> +++ b/config.mak.uname\n> @@ -147,6 +147,7 @@ ifeq ($(uname_S),Darwin)\n>  \tFREAD_READS_DIRECTORIES = UnfortunatelyYes\n>  \tHAVE_NS_GET_EXECUTABLE_PATH = YesPlease\n>  \tCSPRNG_METHOD = arc4random\n> +\tUSE_ENHANCED_BASIC_REGULAR_EXPRESSIONS = YesPlease\n\nOK.  This would give macOS folks who have already been using the\nenhanced mode (without us asking) with their older libraries the\nbehaviour they are more familiar with.  Good.\n\n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index 76e4b11131..1efa834089 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -1338,6 +1338,11 @@ static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,\n>  \treturn regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);\n>  }\n>\n> +#ifdef USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS\n> +int git_regcomp(regex_t *preg, const char *pattern, int cflags);\n> +#define regcomp git_regcomp\n> +#endif\n\nOK.\n"},{"id":"470098","messageId":"Y78GXZvyrOrXhe7n@coredump.intra.peff.net","threadId":"59029","inReplyTo":"26a0d4ca-3d97-ace4-1a1f-92b1ee6715a6@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-11T18:56:29Z","receivedAt":"2023-01-11T18:56:38Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 08, 2023 at 01:42:04AM +0100, René Scharfe wrote:\n\n> REG_ENHANCED on macOS affects REG_EXTENDED as well.  It unlocks e.g.\n> non-greedy repetitions and inline comments.  Sounds nice, but also\n> potentially surprising.  I was unable to find a current version of\n> the re_format(7) manpage online, unfortunately.\n\nI'm not quite sure what you mean here by \"non-greedy repetitions\".\nSomething like:\n\n  # prefer \"foo bar\" to \"foo bar bar\"; only matters for colorizing or\n  # --only-matching\n  git grep -E 'foo.*?bar'\n\n? If so, then yeah, that changes the meaning of a bare \"?\" and people\nmight be surprised by it.\n\n> Apple's latest version of Git sets NO_REGEX and thus uses\n> compat/regex, if I read their source correctly:\n> \n> https://github.com/apple-oss-distributions/Git/blob/Git-128/src/git/Makefile#L1302\n> \n> The easiest and most consistent option would be to do the same.  But\n> we can't do that, because it would break multibyte support, which was\n> fixed by 1819ad327b (grep: fix multibyte regex handling under macOS,\n> 2022-08-26), which started to use the system regex functions again.\n\nLooks like that NO_REGEX line was dropped by 1819ad327b. So I don't\nthink Apple added it themselves; their version is just based on an older\nversion of Git (looks like 2.24.3).\n\n> Which begs the question: Isn't that a problem for the platforms that\n> still have to use NO_REGEX?  Shouldn't we fix compat/regex?\n\nI'm not sure. I always assumed our fallback was similar-ish to what was\nin glibc and was thus pretty featureful, but that may not be true (it\nactually comes from gawk). It looks like we just didn't pull over the\nmulti-byte parts in a997bf423d (compat/regex: get the gawk regex engine\nto compile within git, 2010-08-17).\n\n> Anyway, here's an attempt at a more general, but still targeted fix\n> for macOS:\n> \n> --- >8 ---\n> Subject: [PATCH] use enhanced basic regular expressions on macOS\n\nThis seems pretty sensible, regardless of whether we improve multi-byte\nsupport for compat/regex.\n\n-Peff\n"},{"id":"470221","messageId":"4165031d-e7f1-0477-2245-6a8e312de522@web.de","threadId":"59029","inReplyTo":"Y78GXZvyrOrXhe7n@coredump.intra.peff.net","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-01-12T17:13:13Z","receivedAt":"2023-01-12T17:53:30Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 11.01.23 um 19:56 schrieb Jeff King:\n> On Sun, Jan 08, 2023 at 01:42:04AM +0100, René Scharfe wrote:\n>\n>> REG_ENHANCED on macOS affects REG_EXTENDED as well.  It unlocks e.g.\n>> non-greedy repetitions and inline comments.  Sounds nice, but also\n>> potentially surprising.  I was unable to find a current version of\n>> the re_format(7) manpage online, unfortunately.\n>\n> I'm not quite sure what you mean here by \"non-greedy repetitions\".\n> Something like:\n>\n>   # prefer \"foo bar\" to \"foo bar bar\"; only matters for colorizing or\n>   # --only-matching\n>   git grep -E 'foo.*?bar'\n>\n> ? If so, then yeah, that changes the meaning of a bare \"?\" and people\n> might be surprised by it.\n\nRight.  To be fair, question mark is a special character and you'd\nprobably need to quote it anyway if you want to match a literal\nquestion mark.  Otherwise I get:\n\n   $ git grep -E 'foo.*?bar'\n   fatal: command line, 'foo.*?bar': repetition-operator operand invalid\n\n>> Apple's latest version of Git sets NO_REGEX and thus uses\n>> compat/regex, if I read their source correctly:\n>>\n>> https://github.com/apple-oss-distributions/Git/blob/Git-128/src/git/Makefile#L1302\n>>\n>> The easiest and most consistent option would be to do the same.  But\n>> we can't do that, because it would break multibyte support, which was\n>> fixed by 1819ad327b (grep: fix multibyte regex handling under macOS,\n>> 2022-08-26), which started to use the system regex functions again.\n>\n> Looks like that NO_REGEX line was dropped by 1819ad327b. So I don't\n> think Apple added it themselves; their version is just based on an older\n> version of Git (looks like 2.24.3).\n\nMakes sense.\n\n>> Which begs the question: Isn't that a problem for the platforms that\n>> still have to use NO_REGEX?  Shouldn't we fix compat/regex?\n>\n> I'm not sure. I always assumed our fallback was similar-ish to what was\n> in glibc and was thus pretty featureful, but that may not be true (it\n> actually comes from gawk). It looks like we just didn't pull over the\n> multi-byte parts in a997bf423d (compat/regex: get the gawk regex engine\n> to compile within git, 2010-08-17).\n\nGAWK removed NO_MBSUPPORT, NO_MBSUPPORT and mbsupport.h in the meantime.\nI guess that means they support multi-byte characters everywhere now.\n\nRené\n"},{"id":"470223","messageId":"230112.86mt6nzlk5.gmgdl@evledraar.gmail.com","threadId":"59029","inReplyTo":"4165031d-e7f1-0477-2245-6a8e312de522@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-12T17:52:22Z","receivedAt":"2023-01-12T18:20:21Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jan 12 2023, René Scharfe wrote:\n\n> Am 11.01.23 um 19:56 schrieb Jeff King:\n>> On Sun, Jan 08, 2023 at 01:42:04AM +0100, René Scharfe wrote:\n> [...]\n>>> Which begs the question: Isn't that a problem for the platforms that\n>>> still have to use NO_REGEX?  Shouldn't we fix compat/regex?\n>>\n>> I'm not sure. I always assumed our fallback was similar-ish to what was\n>> in glibc and was thus pretty featureful, but that may not be true (it\n>> actually comes from gawk). It looks like we just didn't pull over the\n>> multi-byte parts in a997bf423d (compat/regex: get the gawk regex engine\n>> to compile within git, 2010-08-17).\n>\n> GAWK removed NO_MBSUPPORT, NO_MBSUPPORT and mbsupport.h in the meantime.\n> I guess that means they support multi-byte characters everywhere now.\n>\n> René\n\nNote that GAWK doesn't really have its own regex engine, their is mostly\nglibc.git's, which they've ported over to their codebase with some\ndifficulty over the years (it not being in gnulib, like most such shared\nGNU libraries).\n\nOurs is then an old copy of gawk.git's, which I attempted to update a\nfew years back, but that patch series ran into some issues I can't\nremember, so we still have that very old version.\n"},{"id":"470234","messageId":"Y8CBrtmL45tA/N8z@coredump.intra.peff.net","threadId":"59029","inReplyTo":"4165031d-e7f1-0477-2245-6a8e312de522@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-12T21:54:54Z","receivedAt":"2023-01-12T22:07:18Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 12, 2023 at 06:13:13PM +0100, René Scharfe wrote:\n\n> > I'm not quite sure what you mean here by \"non-greedy repetitions\".\n> > Something like:\n> >\n> >   # prefer \"foo bar\" to \"foo bar bar\"; only matters for colorizing or\n> >   # --only-matching\n> >   git grep -E 'foo.*?bar'\n> >\n> > ? If so, then yeah, that changes the meaning of a bare \"?\" and people\n> > might be surprised by it.\n> \n> Right.  To be fair, question mark is a special character and you'd\n> probably need to quote it anyway if you want to match a literal\n> question mark.  Otherwise I get:\n> \n>    $ git grep -E 'foo.*?bar'\n>    fatal: command line, 'foo.*?bar': repetition-operator operand invalid\n\nThis is on macOS, I assume? With glibc it seems to be quietly ignored:\n\n  $ git grep -E -o 'foo.*?ba' .clang-format\n  .clang-format:foo, bar, ba\n\nSo it is not treated literally (as it would be without -E). But nor does\nit make the match non-greedy (otherwise it would have output \"foo, ba\",\nas \"git grep -P\" does).\n\nSo it does seem like all bets are off for what people can and should\nexpect here. Which isn't to say we should make things worse. I mostly\nwondered if REG_ENHANCED might take us closer to what glibc was doing by\ndefault, but it doesn't seem like it.\n\n-Peff\n"},{"id":"470244","messageId":"230113.86ilhazved.gmgdl@evledraar.gmail.com","threadId":"59029","inReplyTo":"Y8CBrtmL45tA/N8z@coredump.intra.peff.net","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-13T08:28:59Z","receivedAt":"2023-01-13T08:34:16Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jan 12 2023, Jeff King wrote:\n\n> On Thu, Jan 12, 2023 at 06:13:13PM +0100, René Scharfe wrote:\n>\n>> > I'm not quite sure what you mean here by \"non-greedy repetitions\".\n>> > Something like:\n>> >\n>> >   # prefer \"foo bar\" to \"foo bar bar\"; only matters for colorizing or\n>> >   # --only-matching\n>> >   git grep -E 'foo.*?bar'\n>> >\n>> > ? If so, then yeah, that changes the meaning of a bare \"?\" and people\n>> > might be surprised by it.\n>> \n>> Right.  To be fair, question mark is a special character and you'd\n>> probably need to quote it anyway if you want to match a literal\n>> question mark.  Otherwise I get:\n>> \n>>    $ git grep -E 'foo.*?bar'\n>>    fatal: command line, 'foo.*?bar': repetition-operator operand invalid\n>\n> This is on macOS, I assume? With glibc it seems to be quietly ignored:\n>\n>   $ git grep -E -o 'foo.*?ba' .clang-format\n>   .clang-format:foo, bar, ba\n>\n> So it is not treated literally (as it would be without -E). But nor does\n> it make the match non-greedy (otherwise it would have output \"foo, ba\",\n> as \"git grep -P\" does).\n>\n> So it does seem like all bets are off for what people can and should\n> expect here. Which isn't to say we should make things worse. I mostly\n> wondered if REG_ENHANCED might take us closer to what glibc was doing by\n> default, but it doesn't seem like it.\n\nThere's a couple of ways out of this that I don't see in this thread:\n\n- Declare it not a problem: We have -G, -E and -P to map to BRE, ERE and\n  PCRE. One view is to say the first two must match POSIX, another is\n  tha whatever the platform thinks they should do is how they should\n  act.\n\n  Of course that makes git regex invocations \"unportable\", but it might\n  be acceptable. People can always use PCRE if they want \"portability\".\n\n- Just mandate PCRE for Mac OS, then map -E to -P. We could do this with\n  the pcre2convert() API and PCRE2_CONVERT_POSIX_EXTENDED flag,\n  i.e. PCRE's own \"translate this to ERE\".\n\n  But Perl/PCRE syntax is already a superset of ERE syntax, minus things\n  like (*VERB), (?: etc., which people are unlikely to write without\n  intending to get the PCRE semantics.\n"},{"id":"470286","messageId":"xmqq5yda5p4l.fsf@gitster.g","threadId":"59029","inReplyTo":"230113.86ilhazved.gmgdl@evledraar.gmail.com","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-13T17:19:06Z","receivedAt":"2023-01-13T17:32:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n> On Thu, Jan 12 2023, Jeff King wrote:\n>\n>> So it does seem like all bets are off for what people can and should\n>> expect here. Which isn't to say we should make things worse. I mostly\n>> wondered if REG_ENHANCED might take us closer to what glibc was doing by\n>> default, but it doesn't seem like it.\n\nI thought that René's \"Use enhanced only when doing BRE\" was fairly\nfocused, but I am very tempted to accept ...\n\n> There's a couple of ways out of this that I don't see in this thread:\n>\n> - Declare it not a problem: We have -G, -E and -P to map to BRE, ERE and\n>   PCRE. One view is to say the first two must match POSIX, another is\n>   tha whatever the platform thinks they should do is how they should\n>   act.\n\n... this view.  The story \"BRE and ERE work via what system\nlibraries provide, and 'git grep' matches what system grep' does\" is\nan easy to understand view.\n\n> - Just mandate PCRE for Mac OS, then map -E to -P. We could do this with\n>   the pcre2convert() API and PCRE2_CONVERT_POSIX_EXTENDED flag,\n>   i.e. PCRE's own \"translate this to ERE\".\n\nPresumably this is to ensure -E works identically everywhere?  If\nso, then we should do that everywhere, not just on macOS.  But again\nit makes \"git grep -E\" slightly incompatible with \"grep\" on systems\n(including macOS), so...\n"},{"id":"470287","messageId":"5f6908c8-3f84-92a3-b9d1-8689c500c1e9@web.de","threadId":"59029","inReplyTo":"230113.86ilhazved.gmgdl@evledraar.gmail.com","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-01-13T17:24:44Z","receivedAt":"2023-01-13T17:37:26Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 13.01.23 um 09:28 schrieb Ævar Arnfjörð Bjarmason:\n>\n> There's a couple of ways out of this that I don't see in this thread:\n>\n> - Declare it not a problem: We have -G, -E and -P to map to BRE, ERE and\n>   PCRE. One view is to say the first two must match POSIX, another is\n>   tha whatever the platform thinks they should do is how they should\n>   act.\n\nI like that and it was my first reaction -- alternation is a\nnon-standard extension to BREs.  But we supported it explicitly since\n3632cfc248 (Use compatibility regex library for OSX/Darwin, 2008-09-07),\nso not having it anymore is a regression that we should fix, I think.\n\n>   Of course that makes git regex invocations \"unportable\", but it might\n>   be acceptable. People can always use PCRE if they want \"portability\".\n>\n> - Just mandate PCRE for Mac OS, then map -E to -P. We could do this with\n>   the pcre2convert() API and PCRE2_CONVERT_POSIX_EXTENDED flag,\n>   i.e. PCRE's own \"translate this to ERE\".\n>\n>   But Perl/PCRE syntax is already a superset of ERE syntax, minus things\n>   like (*VERB), (?: etc., which people are unlikely to write without\n>   intending to get the PCRE semantics.\n\nPCRE2_CONVERT_POSIX_EXTENDED is a flag for pcre2_pattern_convert(),\nwhich allows converting an ERE into a PCRE2 regex\n(https://pcre.org/current/doc/html/pcre2convert.html).  So this feature\nallows using PCRE for every kind of regexes, right?\n\nThere may be feature differences for EREs between libc on macOS, glibc\nand/or GAWK, but I'm not aware of any complaints so far.  So I currently\ndon't see the need for that.\n\nRené\n"},{"id":"470314","messageId":"e9913631-0534-ce47-ed73-2caff0f9d04a@web.de","threadId":"59029","inReplyTo":"5f6908c8-3f84-92a3-b9d1-8689c500c1e9@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-01-13T23:03:58Z","receivedAt":"2023-01-13T23:04:18Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 13.01.23 um 18:24 schrieb René Scharfe:\n> Am 13.01.23 um 09:28 schrieb Ævar Arnfjörð Bjarmason:\n>>\n>> - Just mandate PCRE for Mac OS, then map -E to -P. We could do this with\n>>   the pcre2convert() API and PCRE2_CONVERT_POSIX_EXTENDED flag,\n>>   i.e. PCRE's own \"translate this to ERE\".\n>>\n>>   But Perl/PCRE syntax is already a superset of ERE syntax, minus things\n>>   like (*VERB), (?: etc., which people are unlikely to write without\n>>   intending to get the PCRE semantics.\n>\n> PCRE2_CONVERT_POSIX_EXTENDED is a flag for pcre2_pattern_convert(),\n> which allows converting an ERE into a PCRE2 regex\n> (https://pcre.org/current/doc/html/pcre2convert.html).  So this feature\n> allows using PCRE for every kind of regexes, right?\n>\n> There may be feature differences for EREs between libc on macOS, glibc\n> and/or GAWK, but I'm not aware of any complaints so far.  So I currently\n> don't see the need for that.\n\nThought that maybe this could help us matching NUL characters and fix the\nTODO in t7815, because PCRE2 can handle binary files just fine.  But no:\npcre2_pattern_convert() uses (NUL*) in its result, meaning that NUL is\ntreated as a line separator and . (dot) won't match it.  And it's still\nexperimental according to the documentation link I mentioned above.\n\nRené\n"},{"id":"470322","messageId":"0e25e6b0-2eb8-40ee-7999-f2863a545a15@web.de","threadId":"59029","inReplyTo":"xmqq5yda5p4l.fsf@gitster.g","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-01-14T06:44:05Z","receivedAt":"2023-01-14T06:44:28Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 13.01.23 um 18:19 schrieb Junio C Hamano:\n> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n>\n>> On Thu, Jan 12 2023, Jeff King wrote:\n>>\n>>> So it does seem like all bets are off for what people can and should\n>>> expect here. Which isn't to say we should make things worse. I mostly\n>>> wondered if REG_ENHANCED might take us closer to what glibc was doing by\n>>> default, but it doesn't seem like it.\n>\n> I thought that René's \"Use enhanced only when doing BRE\" was fairly\n> focused, but I am very tempted to accept ...\n>\n>> There's a couple of ways out of this that I don't see in this thread:\n>>\n>> - Declare it not a problem: We have -G, -E and -P to map to BRE, ERE and\n>>   PCRE. One view is to say the first two must match POSIX, another is\n>>   tha whatever the platform thinks they should do is how they should\n>>   act.\n>\n> ... this view.  The story \"BRE and ERE work via what system\n> libraries provide, and 'git grep' matches what system grep' does\" is\n> an easy to understand view.\n\nThat was my stance in my first reply as well.  But 3632cfc248 (Use\ncompatibility regex library for OSX/Darwin, 2008-09-07) explicitly\nadded alternation support for BREs on macOS, and 1819ad327b (grep: fix\nmultibyte regex handling under macOS, 2022-08-26) removed it seemingly\nby accident.  And grep(1) does support them on macOS 13.1:\n\n   $ uname -rs\n   Darwin 22.2.0\n   $ which grep\n   /usr/bin/grep\n   $ grep --version\n   grep (BSD grep, GNU compatible) 2.6.0-FreeBSD\n   $ grep '\\(REG_STARTEND\\|NeededForASAN\\)' Makefile\n   # Define NO_REGEX if your C library lacks regex support with REG_STARTEND\n   NO_REGEX = NeededForASAN\n\nRené\n\n"},{"id":"470331","messageId":"b2e597aa-044d-e136-43b9-afc84a1e3794@web.de","threadId":"59029","inReplyTo":"0e25e6b0-2eb8-40ee-7999-f2863a545a15@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-01-14T08:31:44Z","receivedAt":"2023-01-14T08:32:10Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 14.01.23 um 07:44 schrieb René Scharfe:\n> Am 13.01.23 um 18:19 schrieb Junio C Hamano:\n>> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n>>\n>>> On Thu, Jan 12 2023, Jeff King wrote:\n>>>\n>>>> So it does seem like all bets are off for what people can and should\n>>>> expect here. Which isn't to say we should make things worse. I mostly\n>>>> wondered if REG_ENHANCED might take us closer to what glibc was doing by\n>>>> default, but it doesn't seem like it.\n>>\n>> I thought that René's \"Use enhanced only when doing BRE\" was fairly\n>> focused, but I am very tempted to accept ...\n>>\n>>> There's a couple of ways out of this that I don't see in this thread:\n>>>\n>>> - Declare it not a problem: We have -G, -E and -P to map to BRE, ERE and\n>>>   PCRE. One view is to say the first two must match POSIX, another is\n>>>   tha whatever the platform thinks they should do is how they should\n>>>   act.\n>>\n>> ... this view.  The story \"BRE and ERE work via what system\n>> libraries provide, and 'git grep' matches what system grep' does\" is\n>> an easy to understand view.\n>\n> That was my stance in my first reply as well.  But 3632cfc248 (Use\n> compatibility regex library for OSX/Darwin, 2008-09-07) explicitly\n> added alternation support for BREs on macOS, and 1819ad327b (grep: fix\n> multibyte regex handling under macOS, 2022-08-26) removed it seemingly\n> by accident.  And grep(1) does support them on macOS 13.1:\n>\n>    $ uname -rs\n>    Darwin 22.2.0\n>    $ which grep\n>    /usr/bin/grep\n>    $ grep --version\n>    grep (BSD grep, GNU compatible) 2.6.0-FreeBSD\n>    $ grep '\\(REG_STARTEND\\|NeededForASAN\\)' Makefile\n>    # Define NO_REGEX if your C library lacks regex support with REG_STARTEND\n>    NO_REGEX = NeededForASAN\n\nAnd I neglected to copy the author of 1819ad327b until now. :-|\n\n@Diomidis: Here's a link to the start of this thread:\nhttps://lore.kernel.org/git/f82ae28a-fb56-8d1f-96c8-550b61439d3a@enterprisedb.com/\n\nRené\n"},{"id":"470338","messageId":"ab50c16f-dbb0-0661-4d08-fa150d31f88b@aueb.gr","threadId":"59029","inReplyTo":"b2e597aa-044d-e136-43b9-afc84a1e3794@web.de","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Diomidis Spinellis","fromEmail":"dds@aueb.gr","sentAt":"2023-01-14T12:45:54Z","receivedAt":"2023-01-14T12:54:18Z","isPatch":false,"sender":{"key":"dds@aueb.gr","avatar":"https://avatars.githubusercontent.com/u/1212716?v=4"},"body":"On 14-Jan-23 10:31, René Scharfe wrote:\n> Am 14.01.23 um 07:44 schrieb René Scharfe:\n>> Am 13.01.23 um 18:19 schrieb Junio C Hamano:\n>>> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n>>>\n>>>> On Thu, Jan 12 2023, Jeff King wrote:\n>>>> There's a couple of ways out of this that I don't see in this thread:\n>>>>\n>>>> - Declare it not a problem: We have -G, -E and -P to map to BRE, ERE and\n>>>>    PCRE. One view is to say the first two must match POSIX, another is\n>>>>    tha whatever the platform thinks they should do is how they should\n>>>>    act.\n>>>\n>>> ... this view.  The story \"BRE and ERE work via what system\n>>> libraries provide, and 'git grep' matches what system grep' does\" is\n>>> an easy to understand view.\n>>\n>> That was my stance in my first reply as well.  But 3632cfc248 (Use\n>> compatibility regex library for OSX/Darwin, 2008-09-07) explicitly\n>> added alternation support for BREs on macOS, and 1819ad327b (grep: fix\n>> multibyte regex handling under macOS, 2022-08-26) removed it seemingly\n>> by accident.  And grep(1) does support them on macOS 13.1:\n\nIndeed, I removed the alternation handling functionality by accident.  I \nwas not aware of the BRE-handling difference between the GNU and the \nmacOS native regex library.  I think having git-grep behave the same as \ngrep(1) on each platform is consistent with the principle of least \nastonishment (POLA).  This would mean that on macOS plain git-grep \nshould use enhanced basic REs as proposed in René's patch.\n\nDiomidis\n"},{"id":"470349","messageId":"xmqq5yd9m738.fsf@gitster.g","threadId":"59029","inReplyTo":"ab50c16f-dbb0-0661-4d08-fa150d31f88b@aueb.gr","subject":"Re: BUG: git grep behave oddly with alternatives","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-14T16:08:59Z","receivedAt":"2023-01-14T16:09:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Diomidis Spinellis <dds@aueb.gr> writes:\n\n> Indeed, I removed the alternation handling functionality by accident.\n> I was not aware of the BRE-handling difference between the GNU and the\n> macOS native regex library.  I think having git-grep behave the same\n> as grep(1) on each platform is consistent with the principle of least\n> astonishment (POLA).  This would mean that on macOS plain git-grep\n> should use enhanced basic REs as proposed in René's patch.\n\nThanks.  I am fine with what René's patch does, which we already\nqueued, then ;-)\n\nLet's merge 54463d32 (use enhanced basic regular expressions on\nmacOS, 2023-01-08) down to 'next'.\n"}]}