{"thread":{"id":"57118","subject":"[PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns","startedAt":"2021-12-18T19:50:28Z","lastAt":"2022-02-17T22:56:56Z","messageCount":17,"participants":["René Scharfe","Ævar Arnfjörð Bjarmason","Junio C Hamano","SZEDER Gábor"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"444429","messageId":"5fa6962e-3c1c-6dbc-f6d7-589151a9baec@web.de","threadId":"57118","inReplyTo":null,"subject":"[PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-12-18T19:50:02Z","receivedAt":"2021-12-18T19:50:28Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"compile_pcre2_pattern() currently uses the option PCRE2_UTF only for\npatterns with non-ASCII characters.  Patterns with ASCII wildcards can\nmatch non-ASCII strings, though.  Without that option PCRE2 mishandles\nUTF-8 input, though -- it matches parts of multi-byte characters.  Fix\nthat by using PCRE2_UTF even for ASCII-only patterns.\n\nThis is a remake of the reverted ae39ba431a (grep/pcre2: fix an edge\ncase concerning ascii patterns and UTF-8 data, 2021-10-15).  The change\nto the condition and the test are simplified and more targeted.\n\nOriginal-patch-by: Hamza Mahfooz <someguy@effective-light.com>\nSigned-off-by: René Scharfe <l.s.r@web.de>\n---\n grep.c                          | 2 +-\n t/t7812-grep-icase-non-ascii.sh | 6 ++++++\n 2 files changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex fe847a0111..5badb6d851 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -382,7 +382,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \t\t}\n \t\toptions |= PCRE2_CASELESS;\n \t}\n-\tif (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern) &&\n+\tif (!opt->ignore_locale && is_utf8_locale() &&\n \t    !(!opt->ignore_case && (p->fixed || p->is_fixed)))\n \t\toptions |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);\n\ndiff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh\nindex e5d1e4ea68..ca3f24f807 100755\n--- a/t/t7812-grep-icase-non-ascii.sh\n+++ b/t/t7812-grep-icase-non-ascii.sh\n@@ -123,4 +123,10 @@ test_expect_success GETTEXT_LOCALE,LIBPCRE2,PCRE2_MATCH_INVALID_UTF 'PCRE v2: gr\n \ttest_cmp invalid-0xe5 actual\n '\n\n+test_expect_success GETTEXT_LOCALE,LIBPCRE2 'PCRE v2: grep non-literal ASCII from UTF-8' '\n+\tgit grep --perl-regexp -h -o -e ll. file >actual &&\n+\techo \"lló\" >expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n--\n2.34.0\n"},{"id":"444430","messageId":"ba503995-866d-fc80-4e38-b4dfd0e5c5bc@web.de","threadId":"57118","inReplyTo":"5fa6962e-3c1c-6dbc-f6d7-589151a9baec@web.de","subject":"[PATCH 2/2] grep/pcre2: factor out literal variable","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-12-18T19:53:15Z","receivedAt":"2021-12-18T19:53:32Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Patterns that contain no wildcards and don't have to be case-folded are\nliteral.  Give this condition a name to increase the readability of the\nboolean expression for enabling the option PCRE2_UTF.\n\nSigned-off-by: René Scharfe <l.s.r@web.de>\n---\n grep.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 5badb6d851..2b6ac3205d 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -362,6 +362,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \tint jitret;\n \tint patinforet;\n \tsize_t jitsizearg;\n+\tint literal = !opt->ignore_case && (p->fixed || p->is_fixed);\n\n \t/*\n \t * Call pcre2_general_context_create() before calling any\n@@ -382,8 +383,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \t\t}\n \t\toptions |= PCRE2_CASELESS;\n \t}\n-\tif (!opt->ignore_locale && is_utf8_locale() &&\n-\t    !(!opt->ignore_case && (p->fixed || p->is_fixed)))\n+\tif (!opt->ignore_locale && is_utf8_locale() && !literal)\n \t\toptions |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);\n\n #ifdef GIT_PCRE2_VERSION_10_36_OR_HIGHER\n--\n2.34.0\n\n"},{"id":"444435","messageId":"211219.86o85cwfje.gmgdl@evledraar.gmail.com","threadId":"57118","inReplyTo":"ba503995-866d-fc80-4e38-b4dfd0e5c5bc@web.de","subject":"Re: [PATCH 2/2] grep/pcre2: factor out literal variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-19T19:37:37Z","receivedAt":"2021-12-19T19:39:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Dec 18 2021, René Scharfe wrote:\n\n> Patterns that contain no wildcards and don't have to be case-folded are\n> literal.  Give this condition a name to increase the readability of the\n> boolean expression for enabling the option PCRE2_UTF.\n>\n> Signed-off-by: René Scharfe <l.s.r@web.de>\n> ---\n>  grep.c | 4 ++--\n>  1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/grep.c b/grep.c\n> index 5badb6d851..2b6ac3205d 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -362,6 +362,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n>  \tint jitret;\n>  \tint patinforet;\n>  \tsize_t jitsizearg;\n> +\tint literal = !opt->ignore_case && (p->fixed || p->is_fixed);\n>\n>  \t/*\n>  \t * Call pcre2_general_context_create() before calling any\n> @@ -382,8 +383,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n>  \t\t}\n>  \t\toptions |= PCRE2_CASELESS;\n>  \t}\n> -\tif (!opt->ignore_locale && is_utf8_locale() &&\n> -\t    !(!opt->ignore_case && (p->fixed || p->is_fixed)))\n> +\tif (!opt->ignore_locale && is_utf8_locale() && !literal)\n>  \t\toptions |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);\n>\n>  #ifdef GIT_PCRE2_VERSION_10_36_OR_HIGHER\n\nI think for this and 1/2 it would be really nice to pick up a version of\nHamza's CI changes:\nhttps://lore.kernel.org/git/20211118084143.279174-2-someguy@effective-light.com/\n\nAside: Not needed for this change, but I wonder if we could benefit minutely\nfrom:\n\n    #ifdef PCRE2_LITERAL\n    options |= PCRE2_LITERAL;\n    #endif\n\nIt'll save PCRE2 the small effort of finding that we've got no metacharacters.\n"},{"id":"444536","messageId":"xmqqv8zjuhpo.fsf@gitster.g","threadId":"57118","inReplyTo":"ba503995-866d-fc80-4e38-b4dfd0e5c5bc@web.de","subject":"Re: [PATCH 2/2] grep/pcre2: factor out literal variable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-20T20:47:15Z","receivedAt":"2021-12-20T20:47:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n> Patterns that contain no wildcards and don't have to be case-folded are\n> literal.  Give this condition a name to increase the readability of the\n> boolean expression for enabling the option PCRE2_UTF.\n\nMakes sense.  Quite a lot.\n\nThanks.\n"},{"id":"444537","messageId":"xmqqr1a7uhgk.fsf@gitster.g","threadId":"57118","inReplyTo":"211219.86o85cwfje.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] grep/pcre2: factor out literal variable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-20T20:52:43Z","receivedAt":"2021-12-20T20:52:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n> I think for this and 1/2 it would be really nice to pick up a version of\n> Hamza's CI changes:\n> https://lore.kernel.org/git/20211118084143.279174-2-someguy@effective-light.com/\n\nAre there so many incompatible versions of pcre2 library whose usage\nare subtly different that we need to protect ourselves with multiple\nvariants in CI from breakages?\n\nI doubt pcre2 library was _that_ bad.\n\nAdding a special task that builds with the minimum version we\nsupport may not be too bad, but the library should be stable enough\nto allow us to declare it sufficient to test the most common version\nwith the most common build options in our ordinary build job(s).\n"},{"id":"444538","messageId":"xmqqmtkvuhfm.fsf@gitster.g","threadId":"57118","inReplyTo":"211219.86o85cwfje.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] grep/pcre2: factor out literal variable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-20T20:53:17Z","receivedAt":"2021-12-20T20:53:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n> Aside: Not needed for this change, but I wonder if we could benefit minutely\n> from:\n>\n>     #ifdef PCRE2_LITERAL\n>     options |= PCRE2_LITERAL;\n>     #endif\n>\n> It'll save PCRE2 the small effort of finding that we've got no metacharacters.\n\nAfter the dust settles, it would be a good addition in a separate\npatch.\n\nThanks.\n"},{"id":"444548","messageId":"211220.865yrjszg4.gmgdl@evledraar.gmail.com","threadId":"57118","inReplyTo":"xmqqr1a7uhgk.fsf@gitster.g","subject":"Re: [PATCH 2/2] grep/pcre2: factor out literal variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-20T22:03:55Z","receivedAt":"2021-12-20T22:07:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Dec 20 2021, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n>\n>> I think for this and 1/2 it would be really nice to pick up a version of\n>> Hamza's CI changes:\n>> https://lore.kernel.org/git/20211118084143.279174-2-someguy@effective-light.com/\n>\n> Are there so many incompatible versions of pcre2 library whose usage\n> are subtly different that we need to protect ourselves with multiple\n> variants in CI from breakages?\n>\n> I doubt pcre2 library was _that_ bad.\n\nIt's really not, but:\n\n * We have an optional >=10.34 feature we use (albeit trivial)\n * We have an optional >=10.36 feature we use (major, and directly related)\n * We might be targeting JIT or not, and the error handling isn't the same (known PCRE caveat)\n * We might be targeting a PCRE that knows about Unicode, or not\n * We use it in a mode where we might feed UTF-8 invalid data into the UTF-8 mode\n\nAny update to the relevant code really needs to test the combination of\nthose, so it's a perfect target for CI to make that less tedious.\n\n> Adding a special task that builds with the minimum version we\n> support may not be too bad, but the library should be stable enough\n> to allow us to declare it sufficient to test the most common version\n> with the most common build options in our ordinary build job(s).\n\nThat's a nice idea, but not the reality of the situation. Unless we're\nwilling to bump the version requirement & insist on JIT && Unicode\nsupport before using it.\n\nThe CI itself should be realtively cheap, and just runs the few tests\nthat would spot any breakages with the above.\n"},{"id":"447282","messageId":"20220129172542.GB2581@szeder.dev","threadId":"57118","inReplyTo":"5fa6962e-3c1c-6dbc-f6d7-589151a9baec@web.de","subject":"[v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-01-29T17:25:42Z","receivedAt":"2022-01-29T17:25:49Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Sat, Dec 18, 2021 at 08:50:02PM +0100, René Scharfe wrote:\n> compile_pcre2_pattern() currently uses the option PCRE2_UTF only for\n> patterns with non-ASCII characters.  Patterns with ASCII wildcards can\n> match non-ASCII strings, though.  Without that option PCRE2 mishandles\n> UTF-8 input, though -- it matches parts of multi-byte characters.  Fix\n> that by using PCRE2_UTF even for ASCII-only patterns.\n> \n> This is a remake of the reverted ae39ba431a (grep/pcre2: fix an edge\n> case concerning ascii patterns and UTF-8 data, 2021-10-15).  The change\n> to the condition and the test are simplified and more targeted.\n> \n> Original-patch-by: Hamza Mahfooz <someguy@effective-light.com>\n> Signed-off-by: René Scharfe <l.s.r@web.de>\n> ---\n>  grep.c                          | 2 +-\n>  t/t7812-grep-icase-non-ascii.sh | 6 ++++++\n>  2 files changed, 7 insertions(+), 1 deletion(-)\n> \n> diff --git a/grep.c b/grep.c\n> index fe847a0111..5badb6d851 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -382,7 +382,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n>  \t\t}\n>  \t\toptions |= PCRE2_CASELESS;\n>  \t}\n> -\tif (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern) &&\n> +\tif (!opt->ignore_locale && is_utf8_locale() &&\n>  \t    !(!opt->ignore_case && (p->fixed || p->is_fixed)))\n>  \t\toptions |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);\n> \n\nI tried to use 'git grep -P' for the first time ever, and it hung\nright away, spinning all CPUs at 100%.  I could narrow it down, both\nthe complexity of the pattern and the size of the input, see the test\nbelow, and it bisects to this patch.\n\n\n  ---   >8   ---\n\n#!/bin/sh\n\ntest_description='test'\n\n. ./test-lib.sh\n\ntest_expect_success PCRE 'test' '\n\t# LC_ALL=C works\n\tLC_ALL=en_US.UTF-8 &&\n\tcat >ascii <<-\\EOF &&\n\tfoo\n\t bar\n\t baz\n\tEOF\n\tcat >utf8 <<-\\EOF &&\n\tfoo\n\t bar\n\t báz\n\tEOF\n\tgit add ascii utf8 &&\n\n\t# These all work as expected:\n\tgit grep --threads=1 -P \" \" ascii &&\n\tgit grep --threads=1 -P \"^ \" ascii &&\n\tgit grep --threads=1 -P \"\\s\" ascii &&\n\tgit grep --threads=1 -P \"^\\s\" ascii &&\n\tgit grep --threads=1 -P \" \" utf8 &&\n\tgit grep --threads=1 -P \"^ \" utf8 &&\n\tgit grep --threads=1 -P \"\\s\" utf8 &&\n\n\t# This hangs (but it does work with basic and extended regexp):\n\tgit grep --threads=1 -P \"^\\s\" utf8\n'\n\ntest_done\n"},{"id":"447307","messageId":"dca59178-6e9b-315b-06ee-8e3201aa391c@web.de","threadId":"57118","inReplyTo":"20220129172542.GB2581@szeder.dev","subject":"Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-01-30T07:55:02Z","receivedAt":"2022-01-30T07:55:20Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 29.01.22 um 18:25 schrieb SZEDER Gábor:\n> On Sat, Dec 18, 2021 at 08:50:02PM +0100, René Scharfe wrote:\n>> compile_pcre2_pattern() currently uses the option PCRE2_UTF only for\n>> patterns with non-ASCII characters.  Patterns with ASCII wildcards can\n>> match non-ASCII strings, though.  Without that option PCRE2 mishandles\n>> UTF-8 input, though -- it matches parts of multi-byte characters.  Fix\n>> that by using PCRE2_UTF even for ASCII-only patterns.\n>>\n>> This is a remake of the reverted ae39ba431a (grep/pcre2: fix an edge\n>> case concerning ascii patterns and UTF-8 data, 2021-10-15).  The change\n>> to the condition and the test are simplified and more targeted.\n>>\n>> Original-patch-by: Hamza Mahfooz <someguy@effective-light.com>\n>> Signed-off-by: René Scharfe <l.s.r@web.de>\n>> ---\n>>  grep.c                          | 2 +-\n>>  t/t7812-grep-icase-non-ascii.sh | 6 ++++++\n>>  2 files changed, 7 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/grep.c b/grep.c\n>> index fe847a0111..5badb6d851 100644\n>> --- a/grep.c\n>> +++ b/grep.c\n>> @@ -382,7 +382,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n>>  \t\t}\n>>  \t\toptions |= PCRE2_CASELESS;\n>>  \t}\n>> -\tif (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern) &&\n>> +\tif (!opt->ignore_locale && is_utf8_locale() &&\n>>  \t    !(!opt->ignore_case && (p->fixed || p->is_fixed)))\n>>  \t\toptions |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);\n>>\n>\n> I tried to use 'git grep -P' for the first time ever, and it hung\n> right away, spinning all CPUs at 100%.  I could narrow it down, both\n> the complexity of the pattern and the size of the input, see the test\n> below, and it bisects to this patch.\n>\n>\n>   ---   >8   ---\n>\n> #!/bin/sh\n>\n> test_description='test'\n>\n> . ./test-lib.sh\n>\n> test_expect_success PCRE 'test' '\n> \t# LC_ALL=C works\n> \tLC_ALL=en_US.UTF-8 &&\n> \tcat >ascii <<-\\EOF &&\n> \tfoo\n> \t bar\n> \t baz\n> \tEOF\n> \tcat >utf8 <<-\\EOF &&\n> \tfoo\n> \t bar\n> \t báz\n> \tEOF\n> \tgit add ascii utf8 &&\n>\n> \t# These all work as expected:\n> \tgit grep --threads=1 -P \" \" ascii &&\n> \tgit grep --threads=1 -P \"^ \" ascii &&\n> \tgit grep --threads=1 -P \"\\s\" ascii &&\n> \tgit grep --threads=1 -P \"^\\s\" ascii &&\n> \tgit grep --threads=1 -P \" \" utf8 &&\n> \tgit grep --threads=1 -P \"^ \" utf8 &&\n> \tgit grep --threads=1 -P \"\\s\" utf8 &&\n>\n> \t# This hangs (but it does work with basic and extended regexp):\n> \tgit grep --threads=1 -P \"^\\s\" utf8\n> '\n>\n> test_done\n\nI get the following result and no hang with PCRE2 10.39:\n\n   utf8: bar\n   utf8: báz\n\ne0c6029 (Fix inifinite loop when a single byte newline is searched in\nJIT., 2020-05-29) [1] sounds like it might have fixed it.  It's part of\nversion 10.36.\n\nDo you still get the error when you disable JIT, i.e. when you use the\npattern \"(*NO_JIT)^\\s\" instead?\n\nRené\n\n\n[1] https://github.com/PhilipHazel/pcre2/commit/e0c6029a62db9c2161941ecdf459205382d4d379\n"},{"id":"447309","messageId":"20220130090422.GA4769@szeder.dev","threadId":"57118","inReplyTo":"dca59178-6e9b-315b-06ee-8e3201aa391c@web.de","subject":"Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-01-30T09:04:22Z","receivedAt":"2022-01-30T09:04:58Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Sun, Jan 30, 2022 at 08:55:02AM +0100, René Scharfe wrote:\n> Am 29.01.22 um 18:25 schrieb SZEDER Gábor:\n> > On Sat, Dec 18, 2021 at 08:50:02PM +0100, René Scharfe wrote:\n> >> compile_pcre2_pattern() currently uses the option PCRE2_UTF only for\n> >> patterns with non-ASCII characters.  Patterns with ASCII wildcards can\n> >> match non-ASCII strings, though.  Without that option PCRE2 mishandles\n> >> UTF-8 input, though -- it matches parts of multi-byte characters.  Fix\n> >> that by using PCRE2_UTF even for ASCII-only patterns.\n> >>\n> >> This is a remake of the reverted ae39ba431a (grep/pcre2: fix an edge\n> >> case concerning ascii patterns and UTF-8 data, 2021-10-15).  The change\n> >> to the condition and the test are simplified and more targeted.\n> >>\n> >> Original-patch-by: Hamza Mahfooz <someguy@effective-light.com>\n> >> Signed-off-by: René Scharfe <l.s.r@web.de>\n> >> ---\n> >>  grep.c                          | 2 +-\n> >>  t/t7812-grep-icase-non-ascii.sh | 6 ++++++\n> >>  2 files changed, 7 insertions(+), 1 deletion(-)\n> >>\n> >> diff --git a/grep.c b/grep.c\n> >> index fe847a0111..5badb6d851 100644\n> >> --- a/grep.c\n> >> +++ b/grep.c\n> >> @@ -382,7 +382,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n> >>  \t\t}\n> >>  \t\toptions |= PCRE2_CASELESS;\n> >>  \t}\n> >> -\tif (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern) &&\n> >> +\tif (!opt->ignore_locale && is_utf8_locale() &&\n> >>  \t    !(!opt->ignore_case && (p->fixed || p->is_fixed)))\n> >>  \t\toptions |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);\n> >>\n> >\n> > I tried to use 'git grep -P' for the first time ever, and it hung\n> > right away, spinning all CPUs at 100%.  I could narrow it down, both\n> > the complexity of the pattern and the size of the input, see the test\n> > below, and it bisects to this patch.\n> >\n> >\n> >   ---   >8   ---\n> >\n> > #!/bin/sh\n> >\n> > test_description='test'\n> >\n> > . ./test-lib.sh\n> >\n> > test_expect_success PCRE 'test' '\n> > \t# LC_ALL=C works\n> > \tLC_ALL=en_US.UTF-8 &&\n> > \tcat >ascii <<-\\EOF &&\n> > \tfoo\n> > \t bar\n> > \t baz\n> > \tEOF\n> > \tcat >utf8 <<-\\EOF &&\n> > \tfoo\n> > \t bar\n> > \t báz\n> > \tEOF\n> > \tgit add ascii utf8 &&\n> >\n> > \t# These all work as expected:\n> > \tgit grep --threads=1 -P \" \" ascii &&\n> > \tgit grep --threads=1 -P \"^ \" ascii &&\n> > \tgit grep --threads=1 -P \"\\s\" ascii &&\n> > \tgit grep --threads=1 -P \"^\\s\" ascii &&\n> > \tgit grep --threads=1 -P \" \" utf8 &&\n> > \tgit grep --threads=1 -P \"^ \" utf8 &&\n> > \tgit grep --threads=1 -P \"\\s\" utf8 &&\n> >\n> > \t# This hangs (but it does work with basic and extended regexp):\n> > \tgit grep --threads=1 -P \"^\\s\" utf8\n> > '\n> >\n> > test_done\n> \n> I get the following result and no hang with PCRE2 10.39:\n> \n>    utf8: bar\n>    utf8: báz\n> \n> e0c6029 (Fix inifinite loop when a single byte newline is searched in\n> JIT., 2020-05-29) [1] sounds like it might have fixed it.  It's part of\n> version 10.36.\n\nI saw this hang on two Ubuntu 20.04 based boxes, which predate that\nfix you mention only by a month or two, and apparently the almost two\nyears since then was not enough for this fix to trickle down into\nupdated 20.04 pcre packages, because:\n\n> Do you still get the error when you disable JIT, i.e. when you use the\n> pattern \"(*NO_JIT)^\\s\" instead?\n\nNo, with this pattern it works as expected.\n\nSo is there a more convenient way to disable PCRE JIT in Git?  FWIW,\n(non-git) 'grep -P' works with the same patterns.\n\n> René\n> \n> \n> [1] https://github.com/PhilipHazel/pcre2/commit/e0c6029a62db9c2161941ecdf459205382d4d379\n"},{"id":"447314","messageId":"b74f781c-548b-5254-d3d1-fc1873c70e10@web.de","threadId":"57118","inReplyTo":"20220130090422.GA4769@szeder.dev","subject":"Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-01-30T13:32:46Z","receivedAt":"2022-01-30T13:33:16Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 30.01.22 um 10:04 schrieb SZEDER Gábor:\n> On Sun, Jan 30, 2022 at 08:55:02AM +0100, René Scharfe wrote:\n>> e0c6029 (Fix inifinite loop when a single byte newline is searched in\n>> JIT., 2020-05-29) [1] sounds like it might have fixed it.  It's part of\n>> version 10.36.\n>\n> I saw this hang on two Ubuntu 20.04 based boxes, which predate that\n> fix you mention only by a month or two, and apparently the almost two\n> years since then was not enough for this fix to trickle down into\n> updated 20.04 pcre packages, because:\n>\n>> Do you still get the error when you disable JIT, i.e. when you use the\n>> pattern \"(*NO_JIT)^\\s\" instead?\n>\n> No, with this pattern it works as expected.\n>\n> So is there a more convenient way to disable PCRE JIT in Git?  FWIW,\n> (non-git) 'grep -P' works with the same patterns.\n\nI don't know a better way.  We could do it automatically, though:\n\n--- >8 ---\nSubject: [PATCH] grep: disable JIT on PCRE2 before 10.36 to avoid endless loop\n\nCommit e0c6029 (Fix inifinite loop when a single byte newline is\nsearched in JIT., 2020-05-29) of PCRE2 adds the following point to its\nChangeLog for version 10.36:\n\n  2. Fix inifinite loop when a single byte newline is searched in JIT when\n  invalid utf8 mode is enabled.\n\nAvoid that bug on older versions (which are still reportedly found in\nthe wild) by disabling the JIT when handling UTF-8.\n\nReported-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: René Scharfe <l.s.r@web.de>\n---\nNot sure how to test it.  Killing git grep after a second or so seems a\nbit clumsy.  timeout(1) from GNU coreutils at least allows doing that\nfrom the shell, but it's not a standard tool.  Perhaps we need a new\ntest helper for that purpose?\n\n grep.c | 8 ++++++++\n 1 file changed, 8 insertions(+)\n\ndiff --git a/grep.c b/grep.c\nindex 7bb0360869..16629a2301 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -406,6 +406,14 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \t}\n\n \tpcre2_config(PCRE2_CONFIG_JIT, &p->pcre2_jit_on);\n+#ifndef GIT_PCRE2_VERSION_10_36_OR_HIGHER\n+\t/*\n+\t * Work around the bug fixed by e0c6029 (Fix inifinite loop when a\n+\t * single byte newline is searched in JIT., 2020-05-29).\n+\t */\n+\tif (options & PCRE2_MATCH_INVALID_UTF)\n+\t\tp->pcre2_jit_on = 0;\n+#endif\n \tif (p->pcre2_jit_on) {\n \t\tjitret = pcre2_jit_compile(p->pcre2_pattern, PCRE2_JIT_COMPLETE);\n \t\tif (jitret)\n--\n2.35.0\n"},{"id":"447360","messageId":"220131.868ruvbovo.gmgdl@evledraar.gmail.com","threadId":"57118","inReplyTo":"b74f781c-548b-5254-d3d1-fc1873c70e10@web.de","subject":"Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-01-31T21:01:12Z","receivedAt":"2022-01-31T21:02:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jan 30 2022, René Scharfe wrote:\n\n> Am 30.01.22 um 10:04 schrieb SZEDER Gábor:\n>> On Sun, Jan 30, 2022 at 08:55:02AM +0100, René Scharfe wrote:\n>>> e0c6029 (Fix inifinite loop when a single byte newline is searched in\n>>> JIT., 2020-05-29) [1] sounds like it might have fixed it.  It's part of\n>>> version 10.36.\n>>\n>> I saw this hang on two Ubuntu 20.04 based boxes, which predate that\n>> fix you mention only by a month or two, and apparently the almost two\n>> years since then was not enough for this fix to trickle down into\n>> updated 20.04 pcre packages, because:\n>>\n>>> Do you still get the error when you disable JIT, i.e. when you use the\n>>> pattern \"(*NO_JIT)^\\s\" instead?\n>>\n>> No, with this pattern it works as expected.\n>>\n>> So is there a more convenient way to disable PCRE JIT in Git?  FWIW,\n>> (non-git) 'grep -P' works with the same patterns.\n>\n> I don't know a better way.  We could do it automatically, though:\n>\n> --- >8 ---\n> Subject: [PATCH] grep: disable JIT on PCRE2 before 10.36 to avoid endless loop\n>\n> Commit e0c6029 (Fix inifinite loop when a single byte newline is\n> searched in JIT., 2020-05-29) of PCRE2 adds the following point to its\n> ChangeLog for version 10.36:\n>\n>   2. Fix inifinite loop when a single byte newline is searched in JIT when\n>   invalid utf8 mode is enabled.\n>\n> Avoid that bug on older versions (which are still reportedly found in\n> the wild) by disabling the JIT when handling UTF-8.\n>\n> Reported-by: SZEDER Gábor <szeder.dev@gmail.com>\n> Signed-off-by: René Scharfe <l.s.r@web.de>\n> ---\n> Not sure how to test it.  Killing git grep after a second or so seems a\n> bit clumsy.  timeout(1) from GNU coreutils at least allows doing that\n> from the shell, but it's not a standard tool.  Perhaps we need a new\n> test helper for that purpose?\n>\n>  grep.c | 8 ++++++++\n>  1 file changed, 8 insertions(+)\n>\n> diff --git a/grep.c b/grep.c\n> index 7bb0360869..16629a2301 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -406,6 +406,14 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n>  \t}\n>\n>  \tpcre2_config(PCRE2_CONFIG_JIT, &p->pcre2_jit_on);\n> +#ifndef GIT_PCRE2_VERSION_10_36_OR_HIGHER\n> +\t/*\n> +\t * Work around the bug fixed by e0c6029 (Fix inifinite loop when a\n\nBetter to quote this as PhilipHazel/pcre2@e0c6029 or something, i.e. to\nindicate that it's not git.git's commit.\n\n> +\t * single byte newline is searched in JIT., 2020-05-29).\n> +\t */\n> +\tif (options & PCRE2_MATCH_INVALID_UTF)\n> +\t\tp->pcre2_jit_on = 0;\n\nIt seems rather heavy-hande, but I can't think of a better way to deal\nwith this, i.e. if we selectively use JIT on older versions, surely we\nrun into the match-bytes-but-want-chars bug you were fixing.\n\n> +#endif\n>  \tif (p->pcre2_jit_on) {\n>  \t\tjitret = pcre2_jit_compile(p->pcre2_pattern, PCRE2_JIT_COMPLETE);\n>  \t\tif (jitret)\n\n"},{"id":"447825","messageId":"1425e548-c1bb-3cff-d75d-c3760abb6129@web.de","threadId":"57118","inReplyTo":"220131.868ruvbovo.gmgdl@evledraar.gmail.com","subject":"Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-02-05T17:00:02Z","receivedAt":"2022-02-05T17:00:19Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 31.01.22 um 22:01 schrieb Ævar Arnfjörð Bjarmason:\n>\n> On Sun, Jan 30 2022, René Scharfe wrote:\n>\n>> Am 30.01.22 um 10:04 schrieb SZEDER Gábor:\n>>> On Sun, Jan 30, 2022 at 08:55:02AM +0100, René Scharfe wrote:\n>>>> e0c6029 (Fix inifinite loop when a single byte newline is searched in\n>>>> JIT., 2020-05-29) [1] sounds like it might have fixed it.  It's part of\n>>>> version 10.36.\n>>>\n>>> I saw this hang on two Ubuntu 20.04 based boxes, which predate that\n>>> fix you mention only by a month or two, and apparently the almost two\n>>> years since then was not enough for this fix to trickle down into\n>>> updated 20.04 pcre packages, because:\n>>>\n>>>> Do you still get the error when you disable JIT, i.e. when you use the\n>>>> pattern \"(*NO_JIT)^\\s\" instead?\n>>>\n>>> No, with this pattern it works as expected.\n>>>\n>>> So is there a more convenient way to disable PCRE JIT in Git?  FWIW,\n>>> (non-git) 'grep -P' works with the same patterns.\n>>\n>> I don't know a better way.  We could do it automatically, though:\n>>\n>> --- >8 ---\n>> Subject: [PATCH] grep: disable JIT on PCRE2 before 10.36 to avoid endless loop\n>>\n>> Commit e0c6029 (Fix inifinite loop when a single byte newline is\n>> searched in JIT., 2020-05-29) of PCRE2 adds the following point to its\n>> ChangeLog for version 10.36:\n>>\n>>   2. Fix inifinite loop when a single byte newline is searched in JIT when\n>>   invalid utf8 mode is enabled.\n>>\n>> Avoid that bug on older versions (which are still reportedly found in\n>> the wild) by disabling the JIT when handling UTF-8.\n>>\n>> Reported-by: SZEDER Gábor <szeder.dev@gmail.com>\n>> Signed-off-by: René Scharfe <l.s.r@web.de>\n>> ---\n>> Not sure how to test it.  Killing git grep after a second or so seems a\n>> bit clumsy.  timeout(1) from GNU coreutils at least allows doing that\n>> from the shell, but it's not a standard tool.  Perhaps we need a new\n>> test helper for that purpose?\n\nhttps://mywiki.wooledge.org/BashFAQ/068 offers a Perl-based Shell\nfunction or aborting a program if it takes too long:\n\n   doalarm() { perl -e 'alarm shift; exec @ARGV' -- \"$@\"; }\n\nIt doesn't waste time when the program finishes faster and seems to work\nfine with git grep.\n\nI can't actually test the effectiveness of the patch because PCRE2's\nJIT doesn't work on my development machine at all (Apple M1), as I just\ndiscovered. :-/  While we know that disabling JIT helps, we didn't\nactually determine, yet, if e0c6029 (Fix inifinite loop when a single\nbyte newline is searched in JIT., 2020-05-29) really fixes the \"^\\s\"\nbug.\n\nSo I have to abandon this patch, unfortunately.  Any volunteer to pick\nit up?\n\n>>\n>>  grep.c | 8 ++++++++\n>>  1 file changed, 8 insertions(+)\n>>\n>> diff --git a/grep.c b/grep.c\n>> index 7bb0360869..16629a2301 100644\n>> --- a/grep.c\n>> +++ b/grep.c\n>> @@ -406,6 +406,14 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n>>  \t}\n>>\n>>  \tpcre2_config(PCRE2_CONFIG_JIT, &p->pcre2_jit_on);\n>> +#ifndef GIT_PCRE2_VERSION_10_36_OR_HIGHER\n>> +\t/*\n>> +\t * Work around the bug fixed by e0c6029 (Fix inifinite loop when a\n>\n> Better to quote this as PhilipHazel/pcre2@e0c6029 or something, i.e. to\n> indicate that it's not git.git's commit.\n\nThe context should suffice for a human reader to understand that it's a\nPCRE2 about a bug fix, but I can see some program turning hex strings\ninto repo-local links getting confused.  Is there a standard cross-repo\ncitation format?  Using a Github link might be the most practical way\nfor the near term, I suspect.\n\n>\n>> +\t * single byte newline is searched in JIT., 2020-05-29).\n>> +\t */\n>> +\tif (options & PCRE2_MATCH_INVALID_UTF)\n>> +\t\tp->pcre2_jit_on = 0;\n>\n> It seems rather heavy-hande, but I can't think of a better way to deal\n> with this, i.e. if we selectively use JIT on older versions, surely we\n> run into the match-bytes-but-want-chars bug you were fixing.\n>\n>> +#endif\n>>  \tif (p->pcre2_jit_on) {\n>>  \t\tjitret = pcre2_jit_compile(p->pcre2_pattern, PCRE2_JIT_COMPLETE);\n>>  \t\tif (jitret)\n>\n"},{"id":"447834","messageId":"20220206100816.GA1936@szeder.dev","threadId":"57118","inReplyTo":"1425e548-c1bb-3cff-d75d-c3760abb6129@web.de","subject":"Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-02-06T10:08:16Z","receivedAt":"2022-02-06T10:08:27Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Sat, Feb 05, 2022 at 06:00:02PM +0100, René Scharfe wrote:\n> >> --- >8 ---\n> >> Subject: [PATCH] grep: disable JIT on PCRE2 before 10.36 to avoid endless loop\n> >>\n> >> Commit e0c6029 (Fix inifinite loop when a single byte newline is\n> >> searched in JIT., 2020-05-29) of PCRE2 adds the following point to its\n> >> ChangeLog for version 10.36:\n> >>\n> >>   2. Fix inifinite loop when a single byte newline is searched in JIT when\n> >>   invalid utf8 mode is enabled.\n> >>\n> >> Avoid that bug on older versions (which are still reportedly found in\n> >> the wild) by disabling the JIT when handling UTF-8.\n> >>\n> >> Reported-by: SZEDER Gábor <szeder.dev@gmail.com>\n> >> Signed-off-by: René Scharfe <l.s.r@web.de>\n> >> ---\n> >> Not sure how to test it.  Killing git grep after a second or so seems a\n> >> bit clumsy.  timeout(1) from GNU coreutils at least allows doing that\n> >> from the shell, but it's not a standard tool.  Perhaps we need a new\n> >> test helper for that purpose?\n> \n> https://mywiki.wooledge.org/BashFAQ/068 offers a Perl-based Shell\n> function or aborting a program if it takes too long:\n> \n>    doalarm() { perl -e 'alarm shift; exec @ARGV' -- \"$@\"; }\n> \n> It doesn't waste time when the program finishes faster and seems to work\n> fine with git grep.\n> \n> I can't actually test the effectiveness of the patch because PCRE2's\n> JIT doesn't work on my development machine at all (Apple M1), as I just\n> discovered. :-/  While we know that disabling JIT helps, we didn't\n> actually determine, yet, if e0c6029 (Fix inifinite loop when a single\n> byte newline is searched in JIT., 2020-05-29) really fixes the \"^\\s\"\n> bug.\n> \n> So I have to abandon this patch, unfortunately.  Any volunteer to pick\n> it up?\n\nFWIW, I built Git with your patch and USE_LIBPCRE2=YesPlease and run\nthe test suite, and it succeeded.  Though I can't judge how much is\nthis actually worth.\n\n"},{"id":"448321","messageId":"220212.86zgmvx13i.gmgdl@evledraar.gmail.com","threadId":"57118","inReplyTo":"1425e548-c1bb-3cff-d75d-c3760abb6129@web.de","subject":"Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-12T20:46:33Z","receivedAt":"2022-02-12T20:48:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Feb 05 2022, René Scharfe wrote:\n\n> Am 31.01.22 um 22:01 schrieb Ævar Arnfjörð Bjarmason:\n>>\n>> On Sun, Jan 30 2022, René Scharfe wrote:\n>>\n>>> Am 30.01.22 um 10:04 schrieb SZEDER Gábor:\n>>>> On Sun, Jan 30, 2022 at 08:55:02AM +0100, René Scharfe wrote:\n>>>>> e0c6029 (Fix inifinite loop when a single byte newline is searched in\n>>>>> JIT., 2020-05-29) [1] sounds like it might have fixed it.  It's part of\n>>>>> version 10.36.\n>>>>\n>>>> I saw this hang on two Ubuntu 20.04 based boxes, which predate that\n>>>> fix you mention only by a month or two, and apparently the almost two\n>>>> years since then was not enough for this fix to trickle down into\n>>>> updated 20.04 pcre packages, because:\n>>>>\n>>>>> Do you still get the error when you disable JIT, i.e. when you use the\n>>>>> pattern \"(*NO_JIT)^\\s\" instead?\n>>>>\n>>>> No, with this pattern it works as expected.\n>>>>\n>>>> So is there a more convenient way to disable PCRE JIT in Git?  FWIW,\n>>>> (non-git) 'grep -P' works with the same patterns.\n>>>\n>>> I don't know a better way.  We could do it automatically, though:\n>>>\n>>> --- >8 ---\n>>> Subject: [PATCH] grep: disable JIT on PCRE2 before 10.36 to avoid endless loop\n>>>\n>>> Commit e0c6029 (Fix inifinite loop when a single byte newline is\n>>> searched in JIT., 2020-05-29) of PCRE2 adds the following point to its\n>>> ChangeLog for version 10.36:\n>>>\n>>>   2. Fix inifinite loop when a single byte newline is searched in JIT when\n>>>   invalid utf8 mode is enabled.\n>>>\n>>> Avoid that bug on older versions (which are still reportedly found in\n>>> the wild) by disabling the JIT when handling UTF-8.\n>>>\n>>> Reported-by: SZEDER Gábor <szeder.dev@gmail.com>\n>>> Signed-off-by: René Scharfe <l.s.r@web.de>\n>>> ---\n>>> Not sure how to test it.  Killing git grep after a second or so seems a\n>>> bit clumsy.  timeout(1) from GNU coreutils at least allows doing that\n>>> from the shell, but it's not a standard tool.  Perhaps we need a new\n>>> test helper for that purpose?\n>\n> https://mywiki.wooledge.org/BashFAQ/068 offers a Perl-based Shell\n> function or aborting a program if it takes too long:\n>\n>    doalarm() { perl -e 'alarm shift; exec @ARGV' -- \"$@\"; }\n>\n> It doesn't waste time when the program finishes faster and seems to work\n> fine with git grep.\n>\n> I can't actually test the effectiveness of the patch because PCRE2's\n> JIT doesn't work on my development machine at all (Apple M1), as I just\n> discovered. :-/  While we know that disabling JIT helps, we didn't\n> actually determine, yet, if e0c6029 (Fix inifinite loop when a single\n> byte newline is searched in JIT., 2020-05-29) really fixes the \"^\\s\"\n> bug.\n>\n> So I have to abandon this patch, unfortunately.  Any volunteer to pick\n> it up?\n\nWe can test it in CI, and have a proposed patch from Hamza Mahfooz to do\nso. See\nhttps://lore.kernel.org/git/211220.865yrjszg4.gmgdl@evledraar.gmail.com/\n\nThere's been some minor changes to the main.yml since then, but I think\nyou should be able to just pick that patch up, adjust it, apply whatever\nchanges you want to test on top, and push it to github.\n\n"},{"id":"448736","messageId":"4e391e2e-6561-3c2e-0306-c860a37356bc@web.de","threadId":"57118","inReplyTo":"220212.86zgmvx13i.gmgdl@evledraar.gmail.com","subject":"Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2022-02-17T21:14:29Z","receivedAt":"2022-02-17T21:14:45Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 12.02.22 um 21:46 schrieb Ævar Arnfjörð Bjarmason:\n>\n> On Sat, Feb 05 2022, René Scharfe wrote:\n>\n>>\n>> I can't actually test the effectiveness of the patch because PCRE2's\n>> JIT doesn't work on my development machine at all (Apple M1), as I just\n>> discovered. :-/  While we know that disabling JIT helps, we didn't\n>> actually determine, yet, if e0c6029 (Fix inifinite loop when a single\n>> byte newline is searched in JIT., 2020-05-29) really fixes the \"^\\s\"\n>> bug.\n>>\n>> So I have to abandon this patch, unfortunately.  Any volunteer to pick\n>> it up?\n>\n> We can test it in CI, and have a proposed patch from Hamza Mahfooz to do\n> so. See\n> https://lore.kernel.org/git/211220.865yrjszg4.gmgdl@evledraar.gmail.com/\n>\n> There's been some minor changes to the main.yml since then, but I think\n> you should be able to just pick that patch up, adjust it, apply whatever\n> changes you want to test on top, and push it to github.\n\nGood idea!  Except the \"just\" is not justified, I feel.  I learned that\n\n  - t7810 fails with PCRE2 built with --disable-unicode because it uses\n    \\p{...} unconditionally, and that's not supported without Unicode\n    support -- no idea how to detect that and skip those tests except\n    by trying and maybe looking for the note that \"this version of PCRE2\n    does not have support for \\P, \\p, or \\X\", which somehow feels iffy,\n\n  - PCRE2 10.35 doesn't build on Ubuntu x64 without adding -mshstk to\n    CFLAGS, and that's the version I wanted to test,\n\n  - many of the Unicode related tests require Islandic language support,\n    and \"sudo apt-get -y install `check-language-support -l is`\"\n    installs it,\n\n  - the condition for our workaround for bug 2642 is reversed,\n\n  - with that fixed I can't trigger the endless loop.\n\nSo perhaps that's the only fix we need here -- or perhaps I got\nconfused by the multitude of options.\n\n--- >8 ---\nSubject: [PATCH] grep: fix triggering PCRE2_NO_START_OPTIMIZE workaround\n\nPCRE2 bug 2642 was fixed in version 10.36.  Our 95ca1f987e (grep/pcre2:\nbetter support invalid UTF-8 haystacks, 2021-01-24) worked around it on\nolder versions by setting the flag PCRE2_NO_START_OPTIMIZE.  797c359978\n(grep/pcre2: use compile-time PCREv2 version test, 2021-02-18) switched\nit around to set the flag on 10.36 and higher instead, while it claimed\nto use \"the same test done at compile-time\".\n\nSwitch the condition back to apply the workaround on PCRE2 versions\n_before_ 10.36.\n\nSigned-off-by: René Scharfe <l.s.r@web.de>\n---\n grep.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/grep.c b/grep.c\nindex 5bec7fd793..ef34d764f9 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -386,7 +386,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \tif (!opt->ignore_locale && is_utf8_locale() && !literal)\n \t\toptions |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);\n\n-#ifdef GIT_PCRE2_VERSION_10_36_OR_HIGHER\n+#ifndef GIT_PCRE2_VERSION_10_36_OR_HIGHER\n \t/* Work around https://bugs.exim.org/show_bug.cgi?id=2642 fixed in 10.36 */\n \tif (PCRE2_MATCH_INVALID_UTF && options & (PCRE2_UTF | PCRE2_CASELESS))\n \t\toptions |= PCRE2_NO_START_OPTIMIZE;\n--\n2.35.1\n"},{"id":"448746","messageId":"xmqq1r016r0t.fsf@gitster.g","threadId":"57118","inReplyTo":"4e391e2e-6561-3c2e-0306-c860a37356bc@web.de","subject":"Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-17T22:56:50Z","receivedAt":"2022-02-17T22:56:56Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n> Subject: [PATCH] grep: fix triggering PCRE2_NO_START_OPTIMIZE workaround\n>\n> PCRE2 bug 2642 was fixed in version 10.36.  Our 95ca1f987e (grep/pcre2:\n> better support invalid UTF-8 haystacks, 2021-01-24) worked around it on\n> older versions by setting the flag PCRE2_NO_START_OPTIMIZE.  797c359978\n> (grep/pcre2: use compile-time PCREv2 version test, 2021-02-18) switched\n> it around to set the flag on 10.36 and higher instead, while it claimed\n> to use \"the same test done at compile-time\".\n>\n> Switch the condition back to apply the workaround on PCRE2 versions\n> _before_ 10.36.\n>\n> Signed-off-by: René Scharfe <l.s.r@web.de>\n> ---\n>  grep.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/grep.c b/grep.c\n> index 5bec7fd793..ef34d764f9 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -386,7 +386,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n>  \tif (!opt->ignore_locale && is_utf8_locale() && !literal)\n>  \t\toptions |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);\n>\n> -#ifdef GIT_PCRE2_VERSION_10_36_OR_HIGHER\n> +#ifndef GIT_PCRE2_VERSION_10_36_OR_HIGHER\n>  \t/* Work around https://bugs.exim.org/show_bug.cgi?id=2642 fixed in 10.36 */\n\nOh, that's embarrassing.  The #ifdef and the comment sit next to\neach other and they make contradicting statement.\n\n>  \tif (PCRE2_MATCH_INVALID_UTF && options & (PCRE2_UTF | PCRE2_CASELESS))\n>  \t\toptions |= PCRE2_NO_START_OPTIMIZE;\n> --\n> 2.35.1\n"}]}