Re: bug#60690: [PATCH v2] grep: correctly identify utf-8 characters with \{b, w} in -P

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Jan 9, 2023, 19:51 UTC
Message-ID: <230109.865ydf1mdu.gmgdl@evledraar.gmail.com>
In-Reply-To: <d6814350-10a3-55c0-68da-7e691976cd45@cs.ucla.edu>

On Mon, Jan 09 2023, Paul Eggert wrote:

Show 8 quoted lines

> On 1/9/23 03:35, Ævar Arnfjörð Bjarmason wrote:
>
>> You almost never want "everything Unicode considers a digit", and if you
>> do using e.g. \p{Nd} instead of \d would be better in terms of
>> expressing your intent.
>
> For GNU grep, PCRE2_UCP is needed because of examples like what
> Gro-Tsen and Karl Petterssen supplied.

[For reference, referring to this Twitter thread: https://twitter.com/gro_tsen/status/1610972356972875777]

Those examples compared -E and -P. I think it's correct that UCP brings the behavior closer to -E, but it's also different in various ways.

E.g. on emacs.git (which I've been finding to be quite a nice test case)
a comparison of the two, with "git grep" because I found it easier to
test, but GNU grep will presumably find the same for those files:
	
	for c in b s w
	do
		for pfx in '' '(*UCP)'
		do
			echo "$pfx/$c:" &&
			diff -u <(git -P grep -E "\\$c") <(git -P grep -P "$pfx\\$c") | wc -l
		done
	done

Yields:

	/b:
	155781
	(*UCP)/b:
	46035
	/s:
	0
	(*UCP)/s:
	0
	/w:
	142468
	(*UCP)/w:
	9706

So the output still differs, and some of those differences may or may not be wanted.

> If there's some diagreement
> about how \d should behave with UTF-8 data the GNU grep hackers should
> let the Perl community decide that; that is, GNU grep can simply
> follow PCRE2's lead.

PCRE2 tends to follow Perl, I'm mainly trying to point out here that it isn't a-priory clear how "let Perl decide" is supposed to map to the of a "grep"-like utility, since the Perl behavior is inherently tied up with knowing the encoding of the target data.

For GNU grep and "git grep" that's more of an all-or-nothing with locales, although in this case being as close as possible to -E is probably more correct than not.

Show 10 quoted lines

>> 	$ diff <(git -P grep -P '\d+') <(git -P grep -P '(*UCP)\d')
>> 	53360a53361,53362
>> 	> git-gui/po/ja.po:"- 第１行: 何をしたか、を１行で要約。\n"
>> 	> git-gui/po/ja.po:"- 第２行: 空白\n"
>
> Although I don't speak Japanese I have dealt with quite a bit of
> Japanese text in a previous job, and personally I would prefer \d to
> match those two lines as they do contain digits. So to me this
> particular case is not a good argument that git grep should not match
> those lines.

I'm mainly raising the backwards compatibility concern, which GNU grep and git grep may or may not want to handle differently, but let's at least be aware of the various edge cases.

For \b I think it mostly does the right thing.

For \w and \d in particular I'm mainly noting that yes, sometimes you want to match [0-9], and sometimes you'd want to match Japanese numbers, but you rarely (or at least I haven't) want to match everything Unicode considers X, unless you're doing some self-reflection on Unicode itself.

E.g. for \d it's at least (up from just 10):

	$ perl -CO -wE 'for (1..2**20) { say chr if chr =~ /\d/ }'|wc -l
	650

For \w you similarly go from ~60 to ~130k:

	$ perl -CO -wE 'for (1..2**24) { say chr if chr =~ /\w/ }'|wc -l
	134564

If all you're doing is matching either ASCII or Japanese text and you want "locale-aware numbers" it might do the wrong thing.

But I've found it to be too promiscuous when casting a wider net, which is the usual use-case with 'grep".

> Of course other people might prefer differently, and there are cases
> where I want to match only ASCII digits. I've learned in the past to
> use [0-9] for that. I hope PCRE2 never changes [0-9] to match anything
> but ASCII digits when searching UTF-8 text.

I think that'll never change.