git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH] t7812: expect failure for grep -i with invalid UTF-8 data

From
Todd Zullinger <tmz@pobox.com>
Date
Nov 30, 2019, 00:46 UTC
Message-ID
<20191130004653.8794-1-tmz@pobox.com>
In-Reply-To
<87blsyl32c.fsf_-_@igel.home>

When the 'grep with invalid UTF-8 data' tests were added/adjusted in 8a5999838e (grep: stess test PCRE v2 on invalid UTF-8 data, 2019-07-26) and 870eea8166 (grep: do not enter PCRE2_UTF mode on fixed matching, 2019-07-26) they lacked a redirect which caused them to falsely succeed on most architectures. They failed on big-endian arches where the test never reached the portion which was missing the redirect.

A recent patch add the missing redirect and exposed the fact that the 'PCRE v2: grep non-ASCII from invalid UTF-8 data with -i' test fails on all architectures.

Based on the final paragraph in in 870eea8166:
    When grepping a non-ASCII fixed string. This is a more general problem
    that's hard to fix, but we can at least fix the most common case of
    grepping for a fixed string without "-i". I can't think of a reason
    for why we'd turn on PCRE2_UTF when matching byte-for-byte like that.

it seems that we don't expect that the case-insensitive grep will succeed. Adjust the test to reflect that expectation.

Signed-off-by: Todd Zullinger <tmz@pobox.com>
---
Hi,
Andreas Schwab wrote:
Show 5 quoted lines
> Two tests in t7812 were missing redirects, failing to actually test the
> produced output.
> 
> Fixes: 8a5999838e ("grep: stess test PCRE v2 on invalid UTF-8 data")
> Signed-off-by: Andreas Schwab <schwab@linux-m68k.org>

Nice catch on these missing redirects. While testing the 2.24.0 rc's this test failed only on big-endian arches (like s390x in Fedora). It was briefly mentioned at the time in <20191020002648.GZ10893@pobox.com>.

After applying the fix to add the missing redirects, the test now fails on all architectures. It seems like that is the intended state and we simply need to adjust the final test with `grep -i` accordingly.

Of course, it's possible that I've misunderstood Ævar's intentions in 870eea8166 (grep: do not enter PCRE2_UTF mode on fixed matching, 2019-07-26) or otherwise have poorly explained the reasoning in my commit message.

We don't appear to test PCRE (neither v1 nor v2) in our CI builds. I added libpcre2-dev and set USE_LIBPCRE to test this in travis. (Whether we want to enable that by default is worth discussing.)

Doing so confirms that the tests (incorrectly) pass without the missing redirects and fail when the redirects are added. Inverting the expectation of test_cmp in the final `grep -i` test results in a successful test run.

with PCRE2: https://travis-ci.org/tmzullinger/git/jobs/618733182

with PCRE2 and 'add missing redirects': https://travis-ci.org/tmzullinger/git/jobs/618723277

with PCRE2, 'add missing redirects' and this patch: https://travis-ci.org/tmzullinger/git/jobs/618796963

This still doesn't fix the failure on s390x.  There
    test_might_fail git grep -hi "Æ" invalid-0x80 >actual

fails and leaves nothing in 'actual' which causes the test to fail at the following

    test_cmp expected actual

line. If we suspect the test might fail, it seems like we need to account for that and not expect that 'expect' will match 'actual' as we currently do.

 t/t7812-grep-icase-non-ascii.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/t/t7812-grep-icase-non-ascii.sh b/t/t7812-grep-icase-non-ascii.sh
index c4528432e5..03dba6685a 100755
--- a/t/t7812-grep-icase-non-ascii.sh
+++ b/t/t7812-grep-icase-non-ascii.sh
@@ -76,9 +76,12 @@ test_expect_success GETTEXT_LOCALE,LIBPCRE2 'PCRE v2: grep non-ASCII from invali
 
 test_expect_success GETTEXT_LOCALE,LIBPCRE2 'PCRE v2: grep non-ASCII from invalid UTF-8 data with -i' '
 	test_might_fail git grep -hi "Æ" invalid-0x80 >actual &&
-	test_cmp expected actual &&
+	if test -s actual
+	then
+	    test_cmp expected actual
+	fi &&
 	test_must_fail git grep -hi "(*NO_JIT)Æ" invalid-0x80 >actual &&
-	test_cmp expected actual
+	! test_cmp expected actual
 '
 
 test_done
-- 
2.24.0
Previous: Jeff KingNext: Andreas Schwab
Message 26 of 70 in “grep: use custom JIT stack with pcre2”
  1. grep: use custom JIT stack with pcre2Carlo Marcelo Arenas Belón, Jul 21, 2019
  2. 0/3 grep: PCRE JIT fixesÆvar Arnfjörð Bjarmason, Jul 24, 2019
  3. Junio C HamanoJul 24, 2019
  4. Ævar Arnfjörð BjarmasonJul 24, 2019
  5. 1/8 grep: remove overly paranoid BUG(...) codeÆvar Arnfjörð Bjarmason, Jul 26, 2019
  6. 2/8 grep: stop "using" a custom JIT stack with PCRE v2Ævar Arnfjörð Bjarmason, Jul 26, 2019
  7. Carlo ArenasJul 29, 2019
  8. 3/8 grep: stop using a custom JIT stack with PCRE v1Ævar Arnfjörð Bjarmason, Jul 26, 2019
  9. Carlo ArenasJul 29, 2019
  10. 4/8 grep: consistently use "p->fixed" in compile_regexp()Ævar Arnfjörð Bjarmason, Jul 26, 2019
  11. Carlo ArenasJul 29, 2019
  12. Ævar Arnfjörð BjarmasonJul 29, 2019
  13. Ævar Arnfjörð BjarmasonJul 29, 2019
  14. Junio C HamanoJul 29, 2019
  15. 5/8 grep: create a "is_fixed" member in "grep_pat"Ævar Arnfjörð Bjarmason, Jul 26, 2019
  16. 7/8 grep: do not enter PCRE2_UTF mode on fixed matchingÆvar Arnfjörð Bjarmason, Jul 26, 2019
  17. Junio C HamanoJul 26, 2019
  18. 6/8 grep: stess test PCRE v2 on invalid UTF-8 dataÆvar Arnfjörð Bjarmason, Jul 26, 2019
  19. Junio C HamanoJul 26, 2019
  20. Ævar Arnfjörð BjarmasonJul 26, 2019
  21. Carlo ArenasJul 29, 2019
  22. t7812: add missing redirectsAndreas Schwab, Nov 26, 2019
  23. Johannes SchindelinNov 26, 2019
  24. Andreas SchwabNov 26, 2019
  25. Jeff KingNov 27, 2019
  26. t7812: expect failure for grep -i with invalid UTF-8 dataTodd Zullinger, Nov 30, 2019
  27. Andreas SchwabNov 30, 2019
  28. Junio C HamanoDec 1, 2019
  29. Andreas SchwabDec 1, 2019
  30. Todd ZullingerDec 1, 2019
  31. Junio C HamanoDec 2, 2019
  32. 0/8 grep: PCRE JIT fixes + ab/no-kwset fixÆvar Arnfjörð Bjarmason, Jul 26, 2019
  33. Junio C HamanoJul 26, 2019
  34. Ævar Arnfjörð BjarmasonJul 29, 2019
  35. Junio C HamanoJul 29, 2019
  36. 8/8 grep: optimistically use PCRE2_MATCH_INVALID_UTFÆvar Arnfjörð Bjarmason, Jul 26, 2019
  37. Junio C HamanoJul 26, 2019
  38. Ævar Arnfjörð BjarmasonJul 26, 2019
  39. Ævar Arnfjörð BjarmasonJul 26, 2019
  40. 0/4 grep: better support invalid UTF-8 haystacksÆvar Arnfjörð Bjarmason, Jan 24, 2021
  41. 1/2 grep/pcre2 tests: don't rely on invalid UTF-8 data testÆvar Arnfjörð Bjarmason, Jan 24, 2021
  42. 2/2 grep/pcre2: better support invalid UTF-8 haystacksÆvar Arnfjörð Bjarmason, Jan 24, 2021
  43. Ramsay JonesJan 24, 2021
  44. Ramsay JonesJan 24, 2021
  45. Ævar Arnfjörð BjarmasonJan 24, 2021
  46. Ramsay JonesJan 24, 2021
  47. Ævar Arnfjörð BjarmasonJan 24, 2021
  48. 0/2 grep: better support invalid UTF-8 haystacksÆvar Arnfjörð Bjarmason, Jan 24, 2021
  49. 0/2 grep: better support invalid UTF-8 haystacksÆvar Arnfjörð Bjarmason, Jan 24, 2021
  50. 1/2 grep/pcre2 tests: don't rely on invalid UTF-8 data testÆvar Arnfjörð Bjarmason, Jan 24, 2021
  51. 2/2 grep/pcre2: better support invalid UTF-8 haystacksÆvar Arnfjörð Bjarmason, Jan 24, 2021
  52. 1/4 grep/pcre2 tests: don't rely on invalid UTF-8 data testÆvar Arnfjörð Bjarmason, Jan 24, 2021
  53. 4/4 grep/pcre2: better support invalid UTF-8 haystacksÆvar Arnfjörð Bjarmason, Jan 24, 2021
  54. 3/4 grep/pcre2: further simplify boolean spaghettiÆvar Arnfjörð Bjarmason, Jan 24, 2021
  55. 2/4 grep/pcre2: simplify boolean spaghettiÆvar Arnfjörð Bjarmason, Jan 24, 2021
  56. Junio C HamanoJan 24, 2021
  57. Johannes SixtJan 24, 2021
  58. 2/3 grep: stop "using" a custom JIT stack with PCRE v2Ævar Arnfjörð Bjarmason, Jul 24, 2019
  59. Junio C HamanoJul 24, 2019
  60. Ævar Arnfjörð BjarmasonJul 24, 2019
  61. Carlo ArenasJul 25, 2019
  62. 1/3 grep: remove overly paranoid BUG(...) codeÆvar Arnfjörð Bjarmason, Jul 24, 2019
  63. 3/3 grep: stop using a custom JIT stack with PCRE v1Ævar Arnfjörð Bjarmason, Jul 24, 2019
  64. Carlo ArenasJul 26, 2019
  65. Ævar Arnfjörð BjarmasonJul 26, 2019
  66. Carlo ArenasJul 26, 2019
  67. Ævar Arnfjörð BjarmasonJul 26, 2019
  68. 0/2 PCRE1 cleanupCarlo Marcelo Arenas Belón, Jul 26, 2019
  69. 1/2 grep: make sure NO_LIBPCRE1_JIT disable JIT in PCRE1Carlo Marcelo Arenas Belón, Jul 26, 2019
  70. 2/2 grep: refactor and simplify PCRE1 supportCarlo Marcelo Arenas Belón, Jul 26, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.