Re: Compatibility between GNU and Git grep -P

From: Paul Eggert <eggert@cs.ucla.edu>
Date: Apr 30, 2023, 06:51 UTC
Message-ID: <b6a77d9a-f862-685f-f697-a58acc3a077b@cs.ucla.edu>
In-Reply-To: <CAPUEsph9MvGhyh0zTz3o5ELj0XKMdPNvghrg6VDRh25UipgXwQ@mail.gmail.com>

On 2023-04-21 15:05, Carlo Arenas wrote:

> This is really not that recent, and has been released already with git
> 2.40, so at least at that point in time git and grep 3.9 were
> consistent.  That was changed with grep 3.10 though.

OK, so currently GNU grep -P and git grep -P treat \d differently (the former matches only ASCII digits; the latter matches all decimal digits). And the next GNU grep -P (i.e., the one currently on Savannah master), when combined with a future PCRE2 release, is planned to change behavior in other areas, but the two programs will continue to treat \d differently. Not good, obviously.

> FWIW, it doesn't seem git had any issues (other than the crasher with
> PCRE2 10.34) with the transition to matching multibyte digits with
> '\d' and which is what perl (and therefore PCRE2) does

I suspect the issues that Jim is worried about have more to do with people attacking grep-using programs with data that unexpectedly match \d. This is more likely to happen with GNU grep -P (which gets all sorts of weird junk thrown at it) than with Git grep -P (which tends to lead a more cloistered life). If my suspicion is right, then even if Git users don't have issues with PCRE2_UCP and \d, that doesn't mean GNU grep users would be free of such issues.

> My assumption is that you would also need PCRE2_EXTRA_ASCII_DIGIT, and
> indeed bleeding edge pcre2grep[2] had a compatibility option added
> assuming as much.

Although I wasn't aware of that future PCRE2 option, I am not sure GNU grep -P should use it. As things stand, the next GNU grep -P release, when combined with the next PCRE2 release, will have \d match ASCII digits only, and will have [[:digit:]] match all decimal digits. This is compatible with how plain GNU grep [[:digit:]] works (plain GNU grep lacks \d of course, so there's no compatibility issue there).

Quite possibly GNU grep -P should retain [[:digit:]] compatibility with plain grep by not using PCRE2_EXTRA_ASCII_DIGIT. Though it'd be unfortunate that \d would not mean the same thing as [[:digit:]], that might be better than the alternative of having [[:digit:]] mean something different with -P than without -P.

> The proposed patch for git would IMHO just cause the same risk I was
> trying to prevent with my proposed change to GNU grep.

It sounds like this ship has already sailed. At best we can now try to repair it.

For now I installed the attached documentation patch. I left the grep code alone as we are so close to a release.

From 8d3afeebcc2bdf2e8fd4ed1c5256e54be95f36a1 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 29 Apr 2023 23:41:14 -0700
Subject: [PATCH] doc: improve doc for -P '\d'
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This follows up to Carlo Marcelo Arenas Belón’s email
<https://lists.gnu.org/r/grep-devel/2023-04/msg00017.html>
that proposed changing the code too.  These patches change
only the documentation since we’re so near a release.
* NEWS: Be less optimistic about the fix for -P '\d',
and warn that behavior is likely to change again.
* doc/grep.texi (grep Programs): Be less specific about -P \d
behavior, since it’s still in flux.  Warn about mismatching
Unicode versions, or disagreements about obscure constructs.
---
 NEWS          | 14 ++++++++------
 doc/grep.texi | 13 +++++--------
 2 files changed, 13 insertions(+), 14 deletions(-)

diff --git a/NEWS b/NEWS
index c15764c..995d14e 100644
--- a/NEWS
+++ b/NEWS
@@ -4,11 +4,12 @@ GNU grep NEWS                                    -*- outline -*-
 
 ** Bug fixes
 
-  With -P, patterns like [\d] now work again.  The fix relies on PCRE2
-  support for the PCRE2_EXTRA_ASCII_BSD flag planned for PCRE2 10.43.
-  With PCRE2 version 10.42 or earlier, behavior reverts to that of
-  grep 3.8, in that patterns like \w and \b use ASCII rather than
-  Unicode interpretations.
+  With -P, patterns like [\d] now work again.  Fixing this has caused
+  grep to revert to the behavior of grep 3.8, in that patterns like \w
+  and \b go back to using ASCII rather than Unicode interpretations.
+  However, future versions of GNU grep and/or PCRE2 are likely to fix
+  this and change the behavior of \w and \b back to Unicode again,
+  without breaking [\d] as 3.10 did.
   [bug introduced in grep 3.10]
 
   grep no longer fails on files dated after the year 2038,
@@ -25,7 +26,8 @@ GNU grep NEWS                                    -*- outline -*-
 
   previous versions of grep wouldn't respect the user provided settings for
   PCRE_CFLAGS and PCRE_LIBS when building if a libpcre2-8 pkg-config module
-  found in the system.
+  was found.
+
 
 * Noteworthy changes in release 3.10 (2023-03-22) [stable]
 
diff --git a/doc/grep.texi b/doc/grep.texi
index ce6d6dc..ff31d5d 100644
--- a/doc/grep.texi
+++ b/doc/grep.texi
@@ -1154,18 +1154,15 @@ For documentation, refer to @url{https://www.pcre.org/}, with these caveats:
 @samp{\d} matches only the ten ASCII digits
 (and @samp{\D} matches the complement), regardless of locale.
 Use @samp{\p@{Nd@}} to also match non-ASCII digits.
-
-When @command{grep} is built with PCRE2 10.42 and earlier,
-@samp{\d} and @samp{\D} ignore in-regexp directives like @samp{(?aD)}
-and work like @samp{[0-9]} and @samp{[^0-9]} respectively.
-However, later versions of PCRE2 likely will fix this,
-and the plan is for @command{grep} to respect those directives if possible.
+(The behavior of @samp{\d} and @samp{\D} is unspecified after
+in-regexp directives like @samp{(?aD)}.)
 
 @item
 Although PCRE tracks the syntax and semantics of Perl's regular
-expressions, the match is not always exact, partly because Perl
+expressions, the match is not always exact.  For example, Perl
 evolves and a Perl installation may predate or postdate the PCRE2
-installation on the same host.
+installation on the same host, or their Unicode versions may differ,
+or Perl and PCRE2 may disagree about an obscure construct.
 
 @item
 By default, @command{grep} applies each regexp to a line at a time,

-- 
2.39.2