{"thread":{"id":"59630","subject":"Compatibility between GNU and Git grep -P","startedAt":"2023-04-21T21:11:53Z","lastAt":"2023-04-30T06:51:07Z","messageCount":3,"participants":["Paul Eggert","Carlo Arenas"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"475818","messageId":"508ca102-63a9-6334-fee8-7a1ae84c7a23@cs.ucla.edu","threadId":"59630","inReplyTo":"c5j7tduynkzhqpcgqc7iei4mmlnlwvfohmj7ryfhifpay6hhtn@ha3apuuzhxzz","subject":"Compatibility between GNU and Git grep -P","fromName":"Paul Eggert","fromEmail":"eggert@cs.ucla.edu","sentAt":"2023-04-21T21:11:48Z","receivedAt":"2023-04-21T21:11:53Z","isPatch":false,"sender":{"key":"eggert@cs.ucla.edu","avatar":"https://avatars.githubusercontent.com/u/572024?v=4"},"body":"In <https://lists.gnu.org/r/grep-devel/2023-04/msg00017.html> Carlo \nMarcelo Arenas Belón wrote:\n\n> After using this for a while think the following will be better suited\n> for a release because:\n> \n> * the unreleased PCRE2 code is still changing and is unlikely to be released\n>    for a couple of months.\n> * the current way to configure PCRE2 make it difficult to link with the\n>    unreleased code (this might be an independent bug), but it is likely that\n>    the wrong headers might be used by mistake.\n> * the tests and documentation were not completely accurate.\n\nThanks for looking into this. I'm concerned about the resulting patches, \nthough, because I see recent activity in on the Git grep -P side here:\n\nhttps://lore.kernel.org/git/xmqqzgaf2zpt.fsf@gitster.g/\n\nBleeding-edge (i.e., \"master\") GNU grep uses PCRE2_UCP | \nPCRE2_EXTRA_ASCII_BSD with unreleased PCRE2 (which introduces \nPCRE2_EXTRA_ASCII_BSD), and it uses neither flag with the current PCRE2 \nrelease. You're proposing to change GNU grep to never use either flag, \nregardless of PCRE2 release.\n\nIn contrast, bleeding-edge (i.e., \"next\") Git grep -P always uses \nPCRE2_UCP and never uses PCRE2_EXTRA_ASCII_BSD. I.e., it disagrees with \nGNU grep regardless of whether your proposed changes were adopted.\n\nGiven Jim's strong desire that \\d should match only ASCII digits, I \ndoubt whether GNU grep will simply use PCRE2_UCP without \nPCRE2_EXTRA_ASCII_BSD.\n\nIf we want the two grep -P's to stay compatible, I see two ways forward:\n\n1. Leave GNU grep alone and modify Git grep to behave like GNU grep (see \nattached patch to Git).\n\n2. Adopt your proposed change to GNU grep, and revert the recent change \nto Git grep so that it never uses PCRE2_UCP.\n\nEither way, we should see what the Git folks say about this.\n\nFrom 5f5e54157a01c540bde02c305c8ee5e1a39d4f1c Mon Sep 17 00:00:00 2001\nFrom: Paul Eggert <eggert@cs.ucla.edu>\nDate: Fri, 21 Apr 2023 14:06:25 -0700\nSubject: [PATCH] grep: be compatible with GNU grep -P\n\nUse PCRE2_UCP only when PCRE2_EXTRA_ASCII_BSD is defined,\nfor compatibility with GNU grep.\n---\n grep.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 073559f2cd..e9dc8dc0bc 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -320,8 +320,13 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \t\t}\n \t\toptions |= PCRE2_CASELESS;\n \t}\n-\tif (!opt->ignore_locale && is_utf8_locale() && !literal)\n-\t\toptions |= (PCRE2_UTF | PCRE2_UCP | PCRE2_MATCH_INVALID_UTF);\n+\tif (!opt->ignore_locale && is_utf8_locale() && !literal) {\n+\t\toptions |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);\n+#ifdef PCRE2_EXTRA_ASCII_BSD\n+\t\t/* Be compatible with GNU grep -P '\\d'.  */\n+\t\toptions |= (PCRE2_UCP | PCRE2_EXTRA_ASCII_BSD);\n+#endif\n+\t}\n \n #ifndef GIT_PCRE2_VERSION_10_35_OR_HIGHER\n \t/*\n-- \n2.39.2\n\n"},{"id":"475822","messageId":"CAPUEsph9MvGhyh0zTz3o5ELj0XKMdPNvghrg6VDRh25UipgXwQ@mail.gmail.com","threadId":"59630","inReplyTo":"508ca102-63a9-6334-fee8-7a1ae84c7a23@cs.ucla.edu","subject":"Re: Compatibility between GNU and Git grep -P","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2023-04-21T22:05:35Z","receivedAt":"2023-04-21T22:05:52Z","isPatch":false,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Fri, Apr 21, 2023 at 2:11 PM Paul Eggert <eggert@cs.ucla.edu> wrote:\n>\n> In <https://lists.gnu.org/r/grep-devel/2023-04/msg00017.html> Carlo\n> Marcelo Arenas Belón wrote:\n>\n> > After using this for a while think the following will be better suited\n> > for a release because:\n> >\n> > * the unreleased PCRE2 code is still changing and is unlikely to be released\n> >    for a couple of months.\n> > * the current way to configure PCRE2 make it difficult to link with the\n> >    unreleased code (this might be an independent bug), but it is likely that\n> >    the wrong headers might be used by mistake.\n> > * the tests and documentation were not completely accurate.\n\nJust to clarify; those points were made about the GNU grep codebase, hence are\nnot really relevant about git's which had an independent thread[1] and\nthat will be better to use instead to avoid more confusion.\n\n> Thanks for looking into this. I'm concerned about the resulting patches,\n> though, because I see recent activity in on the Git grep -P side here:\n>\n> https://lore.kernel.org/git/xmqqzgaf2zpt.fsf@gitster.g/\n\nThis is really not that recent, and has been released already with git\n2.40, so at least at that point in time git and grep 3.9 were\nconsistent.  That was changed with grep 3.10 though.\n\nFWIW, it doesn't seem git had any issues (other than the crasher with\nPCRE2 10.34) with the transition to matching multibyte digits with\n'\\d' and which is what perl (and therefore PCRE2) does, but as I\nexplained in the other thread I think it might be wise (on the context\nof what is usually matched against with git) to not do that in the\nlong term, and was therefore working on adding the necessary features\nto PCRE2 to be able to do so.  Note that no decision has been made\nthough, which is why I didn't even bother sending an RFC patch.\n\n> Given Jim's strong desire that \\d should match only ASCII digits, I\n> doubt whether GNU grep will simply use PCRE2_UCP without\n> PCRE2_EXTRA_ASCII_BSD.\n\nMy assumption is that you would also need PCRE2_EXTRA_ASCII_DIGIT, and\nindeed bleeding edge pcre2grep[2] had a compatibility option added\nassuming as much.\n\n> Either way, we should see what the Git folks say about this.\n\nThe proposed patch for git would IMHO just cause the same risk I was\ntrying to prevent with my proposed change to GNU grep.\n\nThere are no plans to release PCRE2 10.43 and based on its regular\ncadence wouldn't be for another couple of months, so this code is a\nlittle premature and will need updating eitherway.\n\nCarlo\n\n[1] https://lore.kernel.org/git/2554712d-e386-3bab-bc6c-1f0e85d999db@cs.ucla.edu/\n[2] https://github.com/PCRE2Project/pcre2/commit/3bbdb6dd713b39868934fdc978ba61b81da6d1c5\n"},{"id":"476315","messageId":"b6a77d9a-f862-685f-f697-a58acc3a077b@cs.ucla.edu","threadId":"59630","inReplyTo":"CAPUEsph9MvGhyh0zTz3o5ELj0XKMdPNvghrg6VDRh25UipgXwQ@mail.gmail.com","subject":"Re: Compatibility between GNU and Git grep -P","fromName":"Paul Eggert","fromEmail":"eggert@cs.ucla.edu","sentAt":"2023-04-30T06:51:00Z","receivedAt":"2023-04-30T06:51:07Z","isPatch":false,"sender":{"key":"eggert@cs.ucla.edu","avatar":"https://avatars.githubusercontent.com/u/572024?v=4"},"body":"On 2023-04-21 15:05, Carlo Arenas wrote:\n\n> This is really not that recent, and has been released already with git\n> 2.40, so at least at that point in time git and grep 3.9 were\n> consistent.  That was changed with grep 3.10 though.\n\nOK, so currently GNU grep -P and git grep -P treat \\d differently (the \nformer matches only ASCII digits; the latter matches all decimal \ndigits). And the next GNU grep -P (i.e., the one currently on Savannah \nmaster), when combined with a future PCRE2 release, is planned to change \nbehavior in other areas, but the two programs will continue to treat \\d \ndifferently. Not good, obviously.\n\n\n> FWIW, it doesn't seem git had any issues (other than the crasher with\n> PCRE2 10.34) with the transition to matching multibyte digits with\n> '\\d' and which is what perl (and therefore PCRE2) does\n\nI suspect the issues that Jim is worried about have more to do with \npeople attacking grep-using programs with data that unexpectedly match \n\\d. This is more likely to happen with GNU grep -P (which gets all sorts \nof weird junk thrown at it) than with Git grep -P (which tends to lead a \nmore cloistered life). If my suspicion is right, then even if Git users \ndon't have issues with PCRE2_UCP and \\d, that doesn't mean GNU grep \nusers would be free of such issues.\n\n\n> My assumption is that you would also need PCRE2_EXTRA_ASCII_DIGIT, and\n> indeed bleeding edge pcre2grep[2] had a compatibility option added\n> assuming as much.\n\nAlthough I wasn't aware of that future PCRE2 option, I am not sure GNU \ngrep -P should use it. As things stand, the next GNU grep -P release, \nwhen combined with the next PCRE2 release, will have \\d match ASCII \ndigits only, and will have [[:digit:]] match all decimal digits. This is \ncompatible with how plain GNU grep [[:digit:]] works (plain GNU grep \nlacks \\d of course, so there's no compatibility issue there).\n\nQuite possibly GNU grep -P should retain [[:digit:]] compatibility with \nplain grep by not using PCRE2_EXTRA_ASCII_DIGIT. Though it'd be \nunfortunate that \\d would not mean the same thing as [[:digit:]], that \nmight be better than the alternative of having [[:digit:]] mean \nsomething different with -P than without -P.\n\n\n> The proposed patch for git would IMHO just cause the same risk I was\n> trying to prevent with my proposed change to GNU grep.\n\nIt sounds like this ship has already sailed. At best we can now try to \nrepair it.\n\nFor now I installed the attached documentation patch. I left the grep \ncode alone as we are so close to a release.\n\nFrom 8d3afeebcc2bdf2e8fd4ed1c5256e54be95f36a1 Mon Sep 17 00:00:00 2001\nFrom: Paul Eggert <eggert@cs.ucla.edu>\nDate: Sat, 29 Apr 2023 23:41:14 -0700\nSubject: [PATCH] doc: improve doc for -P '\\d'\nMIME-Version: 1.0\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: 8bit\n\nThis follows up to Carlo Marcelo Arenas Belón’s email\n<https://lists.gnu.org/r/grep-devel/2023-04/msg00017.html>\nthat proposed changing the code too.  These patches change\nonly the documentation since we’re so near a release.\n* NEWS: Be less optimistic about the fix for -P '\\d',\nand warn that behavior is likely to change again.\n* doc/grep.texi (grep Programs): Be less specific about -P \\d\nbehavior, since it’s still in flux.  Warn about mismatching\nUnicode versions, or disagreements about obscure constructs.\n---\n NEWS          | 14 ++++++++------\n doc/grep.texi | 13 +++++--------\n 2 files changed, 13 insertions(+), 14 deletions(-)\n\ndiff --git a/NEWS b/NEWS\nindex c15764c..995d14e 100644\n--- a/NEWS\n+++ b/NEWS\n@@ -4,11 +4,12 @@ GNU grep NEWS                                    -*- outline -*-\n \n ** Bug fixes\n \n-  With -P, patterns like [\\d] now work again.  The fix relies on PCRE2\n-  support for the PCRE2_EXTRA_ASCII_BSD flag planned for PCRE2 10.43.\n-  With PCRE2 version 10.42 or earlier, behavior reverts to that of\n-  grep 3.8, in that patterns like \\w and \\b use ASCII rather than\n-  Unicode interpretations.\n+  With -P, patterns like [\\d] now work again.  Fixing this has caused\n+  grep to revert to the behavior of grep 3.8, in that patterns like \\w\n+  and \\b go back to using ASCII rather than Unicode interpretations.\n+  However, future versions of GNU grep and/or PCRE2 are likely to fix\n+  this and change the behavior of \\w and \\b back to Unicode again,\n+  without breaking [\\d] as 3.10 did.\n   [bug introduced in grep 3.10]\n \n   grep no longer fails on files dated after the year 2038,\n@@ -25,7 +26,8 @@ GNU grep NEWS                                    -*- outline -*-\n \n   previous versions of grep wouldn't respect the user provided settings for\n   PCRE_CFLAGS and PCRE_LIBS when building if a libpcre2-8 pkg-config module\n-  found in the system.\n+  was found.\n+\n \n * Noteworthy changes in release 3.10 (2023-03-22) [stable]\n \ndiff --git a/doc/grep.texi b/doc/grep.texi\nindex ce6d6dc..ff31d5d 100644\n--- a/doc/grep.texi\n+++ b/doc/grep.texi\n@@ -1154,18 +1154,15 @@ For documentation, refer to @url{https://www.pcre.org/}, with these caveats:\n @samp{\\d} matches only the ten ASCII digits\n (and @samp{\\D} matches the complement), regardless of locale.\n Use @samp{\\p@{Nd@}} to also match non-ASCII digits.\n-\n-When @command{grep} is built with PCRE2 10.42 and earlier,\n-@samp{\\d} and @samp{\\D} ignore in-regexp directives like @samp{(?aD)}\n-and work like @samp{[0-9]} and @samp{[^0-9]} respectively.\n-However, later versions of PCRE2 likely will fix this,\n-and the plan is for @command{grep} to respect those directives if possible.\n+(The behavior of @samp{\\d} and @samp{\\D} is unspecified after\n+in-regexp directives like @samp{(?aD)}.)\n \n @item\n Although PCRE tracks the syntax and semantics of Perl's regular\n-expressions, the match is not always exact, partly because Perl\n+expressions, the match is not always exact.  For example, Perl\n evolves and a Perl installation may predate or postdate the PCRE2\n-installation on the same host.\n+installation on the same host, or their Unicode versions may differ,\n+or Perl and PCRE2 may disagree about an obscure construct.\n \n @item\n By default, @command{grep} applies each regexp to a line at a time,\n-- \n2.39.2\n\n"}]}