{"thread":{"id":"59475","subject":"git bug: Perl compatible regular expressions do not work as expected","startedAt":"2023-03-25T12:31:23Z","lastAt":"2023-03-28T17:56:14Z","messageCount":18,"participants":["Mario Grgic","Kristoffer Haugsbakk","demerphq","René Scharfe","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"474169","messageId":"MW4PR20MB5517583CBEEF34B1E87CCF1290859@MW4PR20MB5517.namprd20.prod.outlook.com","threadId":"59475","inReplyTo":null,"subject":"git bug: Perl compatible regular expressions do not work as expected","fromName":"Mario Grgic","fromEmail":"mario_grgic@hotmail.com","sentAt":"2023-03-25T12:31:15Z","receivedAt":"2023-03-25T12:31:23Z","isPatch":false,"sender":{"key":"mario_grgic@hotmail.com","avatar":null},"body":"Thank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\nInitilize a new git repo, create a simple C file that you commit into the new\nrepo. Then search the git history using Perl compatible regular expression. \n\nBasically,\n\nmkdir test\ncd test\ngit init\n\nCreate a file \n\ncat <<END > test.c\nint main(int argc, const char *argv[])\n{\n    return 0;\n}\nEND \n\ngit commit -m 'added test file'\n\nThen run:\n\ngit log --all -p -G '\\bmain\\b'\n\ngit is compiled with pcre2 library support but does not find main word in the\nfile we just added.\n\n\nWhat did you expect to happen? (Expected behavior)\ngit compiled with pcre2 library should support Perl compatible regular\nexpressions\n\nWhat happened instead? (Actual behavior)\nNothing is found, when in fact the search for term is present in the git\nhistory\n\nWhat's different between what you expected and what actually happened?\ngit should print out the commit containing the search term.\n\nAnything else you want to add:\n\ngit was compiled with pcre2 library. Here is output of otool -L\n\n$ otool -L git\ngit:\n\t/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1069.24.0)\n\t/usr/local/lib/libpcre2-8.0.dylib (compatibility version 12.0.0, current version 12.2.0)\n\t/usr/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)\n\t/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)\n\t/usr/local/lib/libintl.8.dylib (compatibility version 11.0.0, current version 11.0.0)\n\t/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)\n\t/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1677.104.0)\n\t/usr/lib/libcharset.1.dylib (compatibility version 2.0.0, current version 2.0.0)\n\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n\n[System Info]\ngit version:\ngit version 2.40.0\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nfeature: fsmonitor--daemon\nuname: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Tue Jun 21 21:18:39 PDT 2022; root:xnu-6153.141.66~1/RELEASE_X86_64 x86_64\ncompiler info: clang: 12.0.0 (clang-1200.0.32.21)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /bin/bash\n\n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show\n\n"},{"id":"474170","messageId":"dd473fc3-cd03-4ef1-9fe1-8909e30a0edb@app.fastmail.com","threadId":"59475","inReplyTo":"MW4PR20MB5517583CBEEF34B1E87CCF1290859@MW4PR20MB5517.namprd20.prod.outlook.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Kristoffer Haugsbakk","fromEmail":"code@khaugsbakk.name","sentAt":"2023-03-25T12:42:12Z","receivedAt":"2023-03-25T12:43:59Z","isPatch":false,"sender":{"key":"code@khaugsbakk.name","avatar":"https://avatars.githubusercontent.com/u/2229597?v=4"},"body":"On Sat, Mar 25, 2023, at 13:31, Mario Grgic wrote:\n> What happened instead? (Actual behavior)\n> Nothing is found, when in fact the search for term is present in the git\n> history\n>\n> What's different between what you expected and what actually happened?\n> git should print out the commit containing the search term.\n\nWeird. It works for me.\n\n    $ mkdir test\n    $ cd test\n    $ git init\n    $ [add file]\n    $ # This was missing from the instructions\n    $ git add --all\n    $ git commit -m 'added test file'\n    $ git log --all -p -G '\\bmain\\b'\n    commit 56fbac5e12649c4de95071cc1872569d7c34055e (HEAD -> main)\n    Author: Kristoffer Haugsbakk <code@khaugsbakk.name>\n    Date:   Sat Mar 25 13:33:51 2023 +0100\n\n        added test file\n\n    diff --git a/test.c b/test.c\n    new file mode 100644\n    index 0000000..0bffa6a\n    --- /dev/null\n    +++ b/test.c\n    @@ -0,0 +1,5 @@\n    +int main(int argc, const char *argv[])\n    +{\n    +    return 0;\n    +}\n    +\n\n.\n\n    [System Info]\n    git version:\n    git version 2.40.0\n    cpu: x86_64\n    no commit associated with this build\n    sizeof-long: 8\n    sizeof-size_t: 8\n    shell-path: /bin/sh\n    uname: Linux 5.4.0-144-generic #161~18.04.1-Ubuntu SMP Fri Feb 10 15:55:22 UTC 2023 x86_64\n    compiler info: gnuc: 7.5\n    libc info: glibc: 2.27\n    $SHELL (typically, interactive shell): /bin/bash\n\n\n    [Enabled Hooks]\n\n-- \nKristoffer Haugsbakk\n"},{"id":"474171","messageId":"MW4PR20MB55178582F83B537AC055AD5990859@MW4PR20MB5517.namprd20.prod.outlook.com","threadId":"59475","inReplyTo":"dd473fc3-cd03-4ef1-9fe1-8909e30a0edb@app.fastmail.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Mario Grgic","fromEmail":"mario_grgic@hotmail.com","sentAt":"2023-03-25T12:59:15Z","receivedAt":"2023-03-25T12:59:40Z","isPatch":false,"sender":{"key":"mario_grgic@hotmail.com","avatar":null},"body":"I did some bisecting and this is broken for me on git-2.39.2 as well, but works in git-2.38.4 and previous versions (tried 2.30.0 as well).\n\n\n> On Mar 25, 2023, at 8:42 AM, Kristoffer Haugsbakk <code@khaugsbakk.name> wrote:\n> \n> On Sat, Mar 25, 2023, at 13:31, Mario Grgic wrote:\n>> What happened instead? (Actual behavior)\n>> Nothing is found, when in fact the search for term is present in the git\n>> history\n>> \n>> What's different between what you expected and what actually happened?\n>> git should print out the commit containing the search term.\n> \n> Weird. It works for me.\n> \n>    $ mkdir test\n>    $ cd test\n>    $ git init\n>    $ [add file]\n>    $ # This was missing from the instructions\n>    $ git add --all\n>    $ git commit -m 'added test file'\n>    $ git log --all -p -G '\\bmain\\b'\n>    commit 56fbac5e12649c4de95071cc1872569d7c34055e (HEAD -> main)\n>    Author: Kristoffer Haugsbakk <code@khaugsbakk.name>\n>    Date:   Sat Mar 25 13:33:51 2023 +0100\n> \n>        added test file\n> \n>    diff --git a/test.c b/test.c\n>    new file mode 100644\n>    index 0000000..0bffa6a\n>    --- /dev/null\n>    +++ b/test.c\n>    @@ -0,0 +1,5 @@\n>    +int main(int argc, const char *argv[])\n>    +{\n>    +    return 0;\n>    +}\n>    +\n> \n> .\n> \n>    [System Info]\n>    git version:\n>    git version 2.40.0\n>    cpu: x86_64\n>    no commit associated with this build\n>    sizeof-long: 8\n>    sizeof-size_t: 8\n>    shell-path: /bin/sh\n>    uname: Linux 5.4.0-144-generic #161~18.04.1-Ubuntu SMP Fri Feb 10 15:55:22 UTC 2023 x86_64\n>    compiler info: gnuc: 7.5\n>    libc info: glibc: 2.27\n>    $SHELL (typically, interactive shell): /bin/bash\n> \n> \n>    [Enabled Hooks]\n> \n> -- \n> Kristoffer Haugsbakk\n\n"},{"id":"474172","messageId":"CANgJU+Vn8ZLGcAYbuDeNkv6T5YdX6t20BqGQDPB0VL_TzoGSWg@mail.gmail.com","threadId":"59475","inReplyTo":"MW4PR20MB5517583CBEEF34B1E87CCF1290859@MW4PR20MB5517.namprd20.prod.outlook.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2023-03-25T13:04:23Z","receivedAt":"2023-03-25T13:04:38Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"On Sat, 25 Mar 2023 at 14:00, Mario Grgic <mario_grgic@hotmail.com> wrote:\n> git log --all -p -G '\\bmain\\b'\n\nI think you want -P not -p.\n\ncheers,\nYves\n"},{"id":"474173","messageId":"MW4PR20MB5517888E63C13099E284B97590859@MW4PR20MB5517.namprd20.prod.outlook.com","threadId":"59475","inReplyTo":"CANgJU+Vn8ZLGcAYbuDeNkv6T5YdX6t20BqGQDPB0VL_TzoGSWg@mail.gmail.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Mario Grgic","fromEmail":"mario_grgic@hotmail.com","sentAt":"2023-03-25T13:09:59Z","receivedAt":"2023-03-25T13:10:09Z","isPatch":false,"sender":{"key":"mario_grgic@hotmail.com","avatar":null},"body":"The lowercase -p is to print the output in patch format. You can rewrite the command line as \n\n git log --all --patch --perl-regexp -G '\\bmain\\b’\n\nI still get no output in any git version after 2.38.4"},{"id":"474174","messageId":"CANgJU+XWUkJhJW3b+gmU3xdkVV+2rA9hKBzQDKFhdJn59T+v9A@mail.gmail.com","threadId":"59475","inReplyTo":"MW4PR20MB5517888E63C13099E284B97590859@MW4PR20MB5517.namprd20.prod.outlook.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2023-03-25T13:24:24Z","receivedAt":"2023-03-25T13:24:41Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"On Sat, 25 Mar 2023 at 14:10, Mario Grgic <mario_grgic@hotmail.com> wrote:\n>\n> The lowercase -p is to print the output in patch format. You can rewrite the command line as\n>\n>  git log --all --patch --perl-regexp -G '\\bmain\\b’\n>\n> I still get no output in any git version after 2.38.4\n\nRIght, sorry about that. My bad.\n\nYves\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"474175","messageId":"MW4PR20MB551709099740DB3777C5CEAD90859@MW4PR20MB5517.namprd20.prod.outlook.com","threadId":"59475","inReplyTo":"MW4PR20MB5517583CBEEF34B1E87CCF1290859@MW4PR20MB5517.namprd20.prod.outlook.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Mario Grgic","fromEmail":"mario_grgic@hotmail.com","sentAt":"2023-03-25T14:16:09Z","receivedAt":"2023-03-25T14:16:18Z","isPatch":false,"sender":{"key":"mario_grgic@hotmail.com","avatar":null},"body":"According to git bisect, this got broken in the following commit:\n\n\ncommit 1819ad327b7a1f19540a819813b70a0e8a7f798f\nAuthor: Diomidis Spinellis <dds@aueb.gr>\nDate:   Fri Aug 26 11:58:15 2022 +0300\n\n    grep: fix multibyte regex handling under macOS\n    \n    The commit 29de20504e (Makefile: fix default regex settings on\n    Darwin, 2013-05-11) fixed t0070-fundamental.sh under Darwin (macOS) by\n    adopting Git's regex library.  However, this library is compiled with\n    NO_MBSUPPORT, which causes git-grep to work incorrectly on multibyte\n    (e.g. UTF-8) files.  Current macOS versions pass t0070-fundamental.sh\n    with the native macOS regex library, which also supports multibyte\n    characters.\n    \n    Adjust the Makefile to use the native regex library, and call\n    setlocale(3) to set CTYPE according to the user's preference.\n    The setlocale call is required on all platforms, but in platforms\n    supporting gettext(3), setlocale was called as a side-effect of\n    initializing gettext.  Therefore, move the CTYPE setlocale call from\n    gettext.c to common-main.c and the corresponding locale.h include\n    into git-compat-util.h.\n    \n    Thanks to the global initialization of CTYPE setlocale, the test-tool\n    regex command now works correctly with supported multibyte regexes, and\n    is used to set the MB_REGEX test prerequisite by assessing a platform's\n    support for them.\n    \n    Signed-off-by: Diomidis Spinellis <dds@aueb.gr>\n    Signed-off-by: Junio C Hamano <gitster@pobox.com>\n\n Makefile          |  2 +-\n common-main.c     |  1 +\n gettext.c         |  2 --\n git-compat-util.h |  1 +\n t/t7810-grep.sh   | 15 +++++++++++++++\n 5 files changed, 18 insertions(+), 3 deletions(-)\n\n"},{"id":"474176","messageId":"MW4PR20MB551779102E85B305E9F675E990859@MW4PR20MB5517.namprd20.prod.outlook.com","threadId":"59475","inReplyTo":"MW4PR20MB5517583CBEEF34B1E87CCF1290859@MW4PR20MB5517.namprd20.prod.outlook.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Mario Grgic","fromEmail":"mario_grgic@hotmail.com","sentAt":"2023-03-25T15:39:20Z","receivedAt":"2023-03-25T15:39:30Z","isPatch":false,"sender":{"key":"mario_grgic@hotmail.com","avatar":null},"body":"Confirming that putting back NO_REGEX = YesPlease in the Makefile fixes the problem. I.e. the following patch fixes it for me:\n\n--- Makefile\t2023-03-25 11:24:01.000000000 -0400\n+++ Makefile.patched\t2023-03-25 11:25:11.000000000 -0400\n@@ -1554,6 +1554,7 @@\n \t\tAPPLE_COMMON_CRYPTO = YesPlease\n \t\tCOMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO\n \tendif\n+\tNO_REGEX = YesPlease\n \tPTHREAD_LIBS =\n endif\n \n\n\nPerhaps the real problem is that the build system should not use macOS native regex library is its configured to use —pcre2 library.\n\n\n\n"},{"id":"474180","messageId":"eba23dc4-c036-fd1b-a1f0-028e8fff602b@web.de","threadId":"59475","inReplyTo":"MW4PR20MB5517888E63C13099E284B97590859@MW4PR20MB5517.namprd20.prod.outlook.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-03-25T18:09:35Z","receivedAt":"2023-03-25T18:09:44Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 25.03.23 um 14:09 schrieb Mario Grgic:\n> The lowercase -p is to print the output in patch format. You can rewrite the command line as\n>\n>  git log --all --patch --perl-regexp -G '\\bmain\\b’\n>\n> I still get no output in any git version after 2.38.4\n\n-G doesn't support Perl regular expressions.  --perl-regexp only affects\n--grep, --grep-reflog, --author, and --committer.  Neither POSIX basic\nnor extended regular expressions support \\b as word boundary.  GNU regex\nand our compat/regex/ do, as extensions.  macOS regex supports it if the\nflag REG_ENHANCED is given to regcomp(3).\n\nSo perhaps this is rather a feature request to support Perl regular\nexpressions for -G (and probably -S as well).  Or to enable REG_ENHANCED\nfor them, at least, like 54463d32ef (use enhanced basic regular\nexpressions on macOS, 2023-01-08) did to get alternations for git grep\non macOS.\n\nRené\n\n"},{"id":"474241","messageId":"xmqqh6u6cg4l.fsf@gitster.g","threadId":"59475","inReplyTo":"eba23dc4-c036-fd1b-a1f0-028e8fff602b@web.de","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-27T16:29:14Z","receivedAt":"2023-03-27T16:29:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n> Am 25.03.23 um 14:09 schrieb Mario Grgic:\n>> The lowercase -p is to print the output in patch format. You can rewrite the command line as\n>>\n>>  git log --all --patch --perl-regexp -G '\\bmain\\b’\n>>\n>> I still get no output in any git version after 2.38.4\n>\n> -G doesn't support Perl regular expressions.  --perl-regexp only affects\n> --grep, --grep-reflog, --author, and --committer.  Neither POSIX basic\n> nor extended regular expressions support \\b as word boundary.  GNU regex\n> and our compat/regex/ do, as extensions.  macOS regex supports it if the\n> flag REG_ENHANCED is given to regcomp(3).\n\nGood summary to unconfuse speculations in the thread.\n\n> So perhaps this is rather a feature request to support Perl regular\n> expressions for -G (and probably -S as well).  \n\nPerhaps.  I used to be a \"it would be wonderful if pcre were usable\neverywhere\" dreamer, but after seeing our share of bugs caused by\nuse of pcre, I am not a huge proponent anymore.  I do not object to\nsuch an enhancement at all, as long as it is done cleanly and in\nsuch a way that it is clear pcre cannot be used by accident when the\nuser does not ask for it.\n\n> Or to enable REG_ENHANCED\n> for them, at least, like 54463d32ef (use enhanced basic regular\n> expressions on macOS, 2023-01-08) did to get alternations for git grep\n> on macOS.\n\nThis one sounds like a reasonable thing, which may not have huge\nunintended fallout, to do.  I am a bit surprised that we have to\ncover each individual callsite of regcomp(3), though.  Doesn't the\n54463d32ef fix use \"#define regcomp git_regcomp\" to cover everybody?\n\n\n\n"},{"id":"474242","messageId":"xmqqcz4ucg1w.fsf@gitster.g","threadId":"59475","inReplyTo":"MW4PR20MB551779102E85B305E9F675E990859@MW4PR20MB5517.namprd20.prod.outlook.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-27T16:30:51Z","receivedAt":"2023-03-27T16:30:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mario Grgic <mario_grgic@hotmail.com> writes:\n\n> Confirming that putting back NO_REGEX = YesPlease in the Makefile fixes the problem. I.e. the following patch fixes it for me:\n>\n> --- Makefile\t2023-03-25 11:24:01.000000000 -0400\n> +++ Makefile.patched\t2023-03-25 11:25:11.000000000 -0400\n> @@ -1554,6 +1554,7 @@\n>  \t\tAPPLE_COMMON_CRYPTO = YesPlease\n>  \t\tCOMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO\n>  \tendif\n> +\tNO_REGEX = YesPlease\n>  \tPTHREAD_LIBS =\n>  endif\n\nIt will unfortunately break multibyte support on macOS by reverting\nwhat 1819ad32 (grep: fix multibyte regex handling under macOS,\n2022-08-26) did.\n"},{"id":"474246","messageId":"MW4PR20MB5517E2591A77978709695AC3908B9@MW4PR20MB5517.namprd20.prod.outlook.com","threadId":"59475","inReplyTo":"xmqqcz4ucg1w.fsf@gitster.g","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Mario Grgic","fromEmail":"mario_grgic@hotmail.com","sentAt":"2023-03-27T17:22:29Z","receivedAt":"2023-03-27T17:22:38Z","isPatch":false,"sender":{"key":"mario_grgic@hotmail.com","avatar":null},"body":"In my case, I compiled git with pcre2 support, using third party PCRE2 library: https://github.com/PCRE2Project/pcre2\n and PCRE and multibyte support in git works with it just fine.\n\nE.g. adding and committing a file:\n\n\n```\n#!/usr/bin/env node\n\nconsole.log('顔🏁');\n```\n\nand searching it with\n\ngit log --all -p  -G '顔🏁’\n\nworks. \n\nAgain, perhaps the solution here is to distinguish two cases:\n\n1. git is compiled with native regex library on macOS \n2. git is compiled with third party regex library with PCRE and multibyte support on macOS\n\nin case 2 everything works. \n\nCase 1 is perhaps more work on macOS and a feature request.\n\n\n\n> On Mar 27, 2023, at 12:30 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> \n> Mario Grgic <mario_grgic@hotmail.com> writes:\n> \n>> Confirming that putting back NO_REGEX = YesPlease in the Makefile fixes the problem. I.e. the following patch fixes it for me:\n>> \n>> --- Makefile\t2023-03-25 11:24:01.000000000 -0400\n>> +++ Makefile.patched\t2023-03-25 11:25:11.000000000 -0400\n>> @@ -1554,6 +1554,7 @@\n>> \t\tAPPLE_COMMON_CRYPTO = YesPlease\n>> \t\tCOMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO\n>> \tendif\n>> +\tNO_REGEX = YesPlease\n>> \tPTHREAD_LIBS =\n>> endif\n> \n> It will unfortunately break multibyte support on macOS by reverting\n> what 1819ad32 (grep: fix multibyte regex handling under macOS,\n> 2022-08-26) did.\n\n"},{"id":"474247","messageId":"03fd7ddb-8241-1a0a-3e82-d8083e4ce0f7@web.de","threadId":"59475","inReplyTo":"xmqqh6u6cg4l.fsf@gitster.g","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-03-27T17:23:35Z","receivedAt":"2023-03-27T17:23:49Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 27.03.23 um 18:29 schrieb Junio C Hamano:\n> René Scharfe <l.s.r@web.de> writes:\n>\n>> Or to enable REG_ENHANCED\n>> for them, at least, like 54463d32ef (use enhanced basic regular\n>> expressions on macOS, 2023-01-08) did to get alternations for git grep\n>> on macOS.\n>\n> This one sounds like a reasonable thing, which may not have huge\n> unintended fallout, to do.  I am a bit surprised that we have to\n> cover each individual callsite of regcomp(3), though.  Doesn't the\n> 54463d32ef fix use \"#define regcomp git_regcomp\" to cover everybody?\n\n54463d32ef only affects basic regular expressions (BRE), but -G and -S\nuse extended ones (ERE).  macOS allows both to be \"enhanced\".\n\nI have a hard time finding a readable reference to the documentation to\nbrings us all onto the same page regarding what REG_ENHANCED actually\ndoes.  [1] is in raw troff format, [2] isn't very pretty and shows ads.\nAnyway, the rather lengthy section \"ENHANCED FEATURES\" explains it.\n\nRené\n\n\n[1] https://opensource.apple.com/source/Libc/Libc-1439.40.11/regex/FreeBSD/re_format.7.auto.html\n[2] https://www.unix.com/man-page/osx/7/re_format/\n"},{"id":"474259","messageId":"xmqqtty5nbm4.fsf@gitster.g","threadId":"59475","inReplyTo":"MW4PR20MB5517E2591A77978709695AC3908B9@MW4PR20MB5517.namprd20.prod.outlook.com","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-27T21:11:15Z","receivedAt":"2023-03-27T21:11:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mario Grgic <mario_grgic@hotmail.com> writes:\n\n[administrivia: do not top post]\n\n>> Mario Grgic <mario_grgic@hotmail.com> writes:\n>>  ...\n>>> +\tNO_REGEX = YesPlease\n>>> \tPTHREAD_LIBS =\n>>> endif\n>> \n>> It will unfortunately break multibyte support on macOS by reverting\n>> what 1819ad32 (grep: fix multibyte regex handling under macOS,\n>> 2022-08-26) did.\n\n> In my case, I compiled git with pcre2 support, using third party\n> PCRE2 library: https://github.com/PCRE2Project/pcre2 and PCRE and\n> multibyte support in git works with it just fine.\n\nSorry, you misunderstood.  1819ad32 is about enabling multi-byte\nsupport for normal regexp types, and does not have anything to do\nwith pcre.  By setting NO_REGEX, the build will not link with\nApple's regex library but the one from compat/ and that version is\nwhat is used for -G and -E (not -P).  -G/-E patterns with multi-byte\nwould not work with compat/ stuff, but they should work when linked\nwith Apple's regex library.\n\n"},{"id":"474260","messageId":"xmqqjzz1nalp.fsf@gitster.g","threadId":"59475","inReplyTo":"03fd7ddb-8241-1a0a-3e82-d8083e4ce0f7@web.de","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-27T21:33:06Z","receivedAt":"2023-03-27T21:33:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n> 54463d32ef only affects basic regular expressions (BRE), but -G and -S\n> use extended ones (ERE).  macOS allows both to be \"enhanced\".\n>\n> I have a hard time finding a readable reference to the documentation to\n> brings us all onto the same page regarding what REG_ENHANCED actually\n> does.  [1] is in raw troff format, [2] isn't very pretty and shows ads.\n\nWow, [2] is also misleading in that it loses the distinction between\n'+' and '\\+'; in [1] at least you can see the differences between \\&+\nand \\e+ ;-)\n\n> Anyway, the rather lengthy section \"ENHANCED FEATURES\" explains it.\n\nI suspect that 54463d32ef was done in a conservative way to avoid\nunintended side effects to make ERE \"enhanced\".  I am not 100%\ncertain, but after reading the documentation you pointed at, I do\nnot see a valid expression without ENHANCED flag starting to mean\ntotally different thing with it (well, an extra '?' turning a\npattern from greedy to minimal may count as such a change in\nsemantics, but I do not see anybody sensible adding an extra '?'\nin a pattern in the first place).\n\nPerhaps it is safe to go ahead and loosen what 54463d32ef did, with\nsomething like this (this is not even compile tested, of course)?\n\n Makefile                  | 9 +++++++++\n compat/regcomp_enhanced.c | 2 ++\n git-compat-util.h         | 2 +-\n 3 files changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git c/Makefile w/Makefile\nindex 50ee51fde3..1fe0eb399e 100644\n--- c/Makefile\n+++ w/Makefile\n@@ -293,6 +293,10 @@ include shared.mak\n # the flag REG_ENHANCED and you'd like to use it to enable enhanced basic\n # regular expressions.\n #\n+# Define USE_ENHANCED_REGULAR_EXPRESSIONS if your C library provides\n+# the flag REG_ENHANCED and you'd like to use it to enable enhanced\n+# regular expressions for both BRE and ERE.\n+#\n # Define HAVE_DEV_TTY if your system can open /dev/tty to interact with the\n # user.\n #\n@@ -2041,11 +2045,16 @@ ifdef NO_REGEX\n \tCOMPAT_CFLAGS += -Icompat/regex\n \tCOMPAT_OBJS += compat/regex/regex.o\n else\n+ifdef USE_ENHANCED_REGULAR_EXPRESSIONS\n+\tCOMPAT_CFLAGS += -DUSE_ENHANCED_REGULAR_EXPRESSIONS\n+\tCOMPAT_OBJS += compat/regcomp_enhanced.o\n+else\n ifdef USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS\n \tCOMPAT_CFLAGS += -DUSE_ENHANCED_BASIC_REGULAR_EXPRESSIONS\n \tCOMPAT_OBJS += compat/regcomp_enhanced.o\n endif\n endif\n+endif\n ifdef NATIVE_CRLF\n \tBASIC_CFLAGS += -DNATIVE_CRLF\n endif\ndiff --git c/compat/regcomp_enhanced.c w/compat/regcomp_enhanced.c\nindex 84193ce53b..aadbbac6d3 100644\n--- c/compat/regcomp_enhanced.c\n+++ w/compat/regcomp_enhanced.c\n@@ -3,7 +3,9 @@\n \n int git_regcomp(regex_t *preg, const char *pattern, int cflags)\n {\n+#ifndef USE_ENHANCED_REGULAR_EXPRESSIONS\n \tif (!(cflags & REG_EXTENDED))\n+#endif\n \t\tcflags |= REG_ENHANCED;\n \treturn regcomp(preg, pattern, cflags);\n }\ndiff --git c/git-compat-util.h w/git-compat-util.h\nindex 1e6592624d..95d9b75a15 100644\n--- c/git-compat-util.h\n+++ w/git-compat-util.h\n@@ -1377,7 +1377,7 @@ static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,\n \treturn regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);\n }\n \n-#ifdef USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS\n+#if defined(USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS) || defined(USE_ENHANCED_REGULAR_EXPRESSIONS)\n int git_regcomp(regex_t *preg, const char *pattern, int cflags);\n #define regcomp git_regcomp\n #endif\n\n"},{"id":"474264","messageId":"MW4PR20MB5517CBF944D108A95665025A90889@MW4PR20MB5517.namprd20.prod.outlook.com","threadId":"59475","inReplyTo":"xmqqtty5nbm4.fsf@gitster.g","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Mario Grgic","fromEmail":"mario_grgic@hotmail.com","sentAt":"2023-03-28T00:03:26Z","receivedAt":"2023-03-28T00:03:37Z","isPatch":false,"sender":{"key":"mario_grgic@hotmail.com","avatar":null},"body":"Interesting. I could be wrong. Let me clarify what I did and what I observe. I edited the Makefile and put back the NO_REGEX = YesPlease line. Then I configured the build as \n\n./configure —with-libpcre2=/usr/local \n\nthen built and installed git. \n\notool -L git shows git binary is dynamically linking the libpcre in /usr/local\n\nHowever, it looks like this third party library is also used for -G  searches (plain POSIX regex or PCRE), since it accepts and correctly finds things like ‘\\btext\\b’. That is \n\ngit log —all -p -G ‘\\main\\b’\n\nworks\n\nFurthermore, -G works with multibyte strings as well: \n\ngit log --all -p  -G '顔🏁’\n\nWhat doesn’t work is multibyte strings with PCRESs, so for example this does not work:\n\ngit log --all -p  -G ‘\\b顔🏁\\b’\n\nThis doesn’t seem to be consistent with the thesis that regex library from compat is used for -G searches and that it doesn’t support multibyte strings?\n\n\n> On Mar 27, 2023, at 5:11 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> \n> Mario Grgic <mario_grgic@hotmail.com> writes:\n> \n> [administrivia: do not top post]\n> \n>>> Mario Grgic <mario_grgic@hotmail.com> writes:\n>>> ...\n>>>> +\tNO_REGEX = YesPlease\n>>>> \tPTHREAD_LIBS =\n>>>> endif\n>>> \n>>> It will unfortunately break multibyte support on macOS by reverting\n>>> what 1819ad32 (grep: fix multibyte regex handling under macOS,\n>>> 2022-08-26) did.\n> \n>> In my case, I compiled git with pcre2 support, using third party\n>> PCRE2 library: https://github.com/PCRE2Project/pcre2 and PCRE and\n>> multibyte support in git works with it just fine.\n> \n> Sorry, you misunderstood.  1819ad32 is about enabling multi-byte\n> support for normal regexp types, and does not have anything to do\n> with pcre.  By setting NO_REGEX, the build will not link with\n> Apple's regex library but the one from compat/ and that version is\n> what is used for -G and -E (not -P).  -G/-E patterns with multi-byte\n> would not work with compat/ stuff, but they should work when linked\n> with Apple's regex library.\n> \n\n"},{"id":"474280","messageId":"xmqqsfdpj8ci.fsf@gitster.g","threadId":"59475","inReplyTo":"xmqqjzz1nalp.fsf@gitster.g","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-28T13:47:41Z","receivedAt":"2023-03-28T13:47:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> I suspect that 54463d32ef was done in a conservative way to avoid\n> unintended side effects to make ERE \"enhanced\".  I am not 100%\n> certain, but after reading the documentation you pointed at, I do\n> not see a valid expression without ENHANCED flag starting to mean\n> totally different thing with it (well, an extra '?' turning a\n> pattern from greedy to minimal may count as such a change in\n> semantics, but I do not see anybody sensible adding an extra '?'\n> in a pattern in the first place).\n\nSorry, but that is nonsense.  We cannot avoid being backward\nincompatible if we suddenly flip the \"enhanced\" bit for BRE.  A\nsane pattern written expecting non-enhanced BRE can change its\nmeaning when the \"enhanced\" mode is enabled.\n\nBut if \"enhanced\" is what users want, and if that is what the other\ntools on the platform use, then perhaps flipping the \"enhanced\" bit\nmay not be a bad idea.\n\n"},{"id":"474347","messageId":"4e03ea47-b0aa-d69e-6c54-fcbadb3b0641@web.de","threadId":"59475","inReplyTo":"xmqqsfdpj8ci.fsf@gitster.g","subject":"Re: git bug: Perl compatible regular expressions do not work as expected","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-03-28T17:56:01Z","receivedAt":"2023-03-28T17:56:14Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 28.03.23 um 15:47 schrieb Junio C Hamano:\n> Junio C Hamano <gitster@pobox.com> writes:\n>\n>> I suspect that 54463d32ef was done in a conservative way to avoid\n>> unintended side effects to make ERE \"enhanced\".  I am not 100%\n>> certain, but after reading the documentation you pointed at, I do\n>> not see a valid expression without ENHANCED flag starting to mean\n>> totally different thing with it (well, an extra '?' turning a\n>> pattern from greedy to minimal may count as such a change in\n>> semantics, but I do not see anybody sensible adding an extra '?'\n>> in a pattern in the first place).\n>\n> Sorry, but that is nonsense.  We cannot avoid being backward\n> incompatible if we suddenly flip the \"enhanced\" bit for BRE.  A\n> sane pattern written expecting non-enhanced BRE can change its\n> meaning when the \"enhanced\" mode is enabled.\n\nRight, and there are even more enhancements for ERE, i.e. more\nincompatibilities.  Not sure how big of a problem that actually would\nbe, though.\n\n> But if \"enhanced\" is what users want, and if that is what the other\n> tools on the platform use, then perhaps flipping the \"enhanced\" bit\n> may not be a bad idea.\n\nApple's grep(1) uses it, with and without -E:\n\nhttps://github.com/apple-oss-distributions/text_cmds/search?q=REG_ENHANCED\n\nE.g. awk(1) and sed(1) don't, AFAICS.\n\nRené\n"}]}