Compatibility between GNU and Git grep -P

In <https://lists.gnu.org/r/grep-devel/2023-04/msg00017.html> Carlo Marcelo Arenas Belón wrote:

Thanks for looking into this. I'm concerned about the resulting patches, though, because I see recent activity in on the Git grep -P side here:

https://lore.kernel.org/git/xmqqzgaf2zpt.fsf@gitster.g/

Bleeding-edge (i.e., "master") GNU grep uses PCRE2_UCP | PCRE2_EXTRA_ASCII_BSD with unreleased PCRE2 (which introduces PCRE2_EXTRA_ASCII_BSD), and it uses neither flag with the current PCRE2 release. You're proposing to change GNU grep to never use either flag, regardless of PCRE2 release.

In contrast, bleeding-edge (i.e., "next") Git grep -P always uses PCRE2_UCP and never uses PCRE2_EXTRA_ASCII_BSD. I.e., it disagrees with GNU grep regardless of whether your proposed changes were adopted.

Given Jim's strong desire that \d should match only ASCII digits, I doubt whether GNU grep will simply use PCRE2_UCP without PCRE2_EXTRA_ASCII_BSD.

If we want the two grep -P's to stay compatible, I see two ways forward:

1. Leave GNU grep alone and modify Git grep to behave like GNU grep (see 
attached patch to Git).

2. Adopt your proposed change to GNU grep, and revert the recent change 
to Git grep so that it never uses PCRE2_UCP.

Either way, we should see what the Git folks say about this.

From 5f5e54157a01c540bde02c305c8ee5e1a39d4f1c Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Fri, 21 Apr 2023 14:06:25 -0700
Subject: [PATCH] grep: be compatible with GNU grep -P

Use PCRE2_UCP only when PCRE2_EXTRA_ASCII_BSD is defined,
for compatibility with GNU grep.
---
 grep.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

On Fri, Apr 21, 2023 at 2:11 PM Paul Eggert <eggert@cs.ucla.edu> wrote:

Just to clarify; those points were made about the GNU grep codebase, hence are not really relevant about git's which had an independent thread[1] and that will be better to use instead to avoid more confusion.

This is really not that recent, and has been released already with git 2.40, so at least at that point in time git and grep 3.9 were consistent. That was changed with grep 3.10 though.

FWIW, it doesn't seem git had any issues (other than the crasher with PCRE2 10.34) with the transition to matching multibyte digits with '\d' and which is what perl (and therefore PCRE2) does, but as I explained in the other thread I think it might be wise (on the context of what is usually matched against with git) to not do that in the long term, and was therefore working on adding the necessary features to PCRE2 to be able to do so. Note that no decision has been made though, which is why I didn't even bother sending an RFC patch.

My assumption is that you would also need PCRE2_EXTRA_ASCII_DIGIT, and indeed bleeding edge pcre2grep[2] had a compatibility option added assuming as much.

The proposed patch for git would IMHO just cause the same risk I was trying to prevent with my proposed change to GNU grep.

There are no plans to release PCRE2 10.43 and based on its regular cadence wouldn't be for another couple of months, so this code is a little premature and will need updating eitherway.

Carlo

[1] https://lore.kernel.org/git/2554712d-e386-3bab-bc6c-1f0e85d999db@cs.ucla.edu/ [2] https://github.com/PCRE2Project/pcre2/commit/3bbdb6dd713b39868934fdc978ba61b81da6d1c5

On 2023-04-21 15:05, Carlo Arenas wrote:

OK, so currently GNU grep -P and git grep -P treat \d differently (the former matches only ASCII digits; the latter matches all decimal digits). And the next GNU grep -P (i.e., the one currently on Savannah master), when combined with a future PCRE2 release, is planned to change behavior in other areas, but the two programs will continue to treat \d differently. Not good, obviously.

I suspect the issues that Jim is worried about have more to do with people attacking grep-using programs with data that unexpectedly match \d. This is more likely to happen with GNU grep -P (which gets all sorts of weird junk thrown at it) than with Git grep -P (which tends to lead a more cloistered life). If my suspicion is right, then even if Git users don't have issues with PCRE2_UCP and \d, that doesn't mean GNU grep users would be free of such issues.

Although I wasn't aware of that future PCRE2 option, I am not sure GNU grep -P should use it. As things stand, the next GNU grep -P release, when combined with the next PCRE2 release, will have \d match ASCII digits only, and will have [[:digit:]] match all decimal digits. This is compatible with how plain GNU grep [[:digit:]] works (plain GNU grep lacks \d of course, so there's no compatibility issue there).

Quite possibly GNU grep -P should retain [[:digit:]] compatibility with plain grep by not using PCRE2_EXTRA_ASCII_DIGIT. Though it'd be unfortunate that \d would not mean the same thing as [[:digit:]], that might be better than the alternative of having [[:digit:]] mean something different with -P than without -P.

It sounds like this ship has already sailed. At best we can now try to repair it.

For now I installed the attached documentation patch. I left the grep code alone as we are so close to a release.

From 8d3afeebcc2bdf2e8fd4ed1c5256e54be95f36a1 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 29 Apr 2023 23:41:14 -0700
Subject: [PATCH] doc: improve doc for -P '\d'
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This follows up to Carlo Marcelo Arenas Belón’s email
<https://lists.gnu.org/r/grep-devel/2023-04/msg00017.html>
that proposed changing the code too.  These patches change
only the documentation since we’re so near a release.
* NEWS: Be less optimistic about the fix for -P '\d',
and warn that behavior is likely to change again.
* doc/grep.texi (grep Programs): Be less specific about -P \d
behavior, since it’s still in flux.  Warn about mismatching
Unicode versions, or disagreements about obscure constructs.
---
 NEWS          | 14 ++++++++------
 doc/grep.texi | 13 +++++--------
 2 files changed, 13 insertions(+), 14 deletions(-)