git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 13/31] grep: prepare for testing binary regexes containing rx metacharacters

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
May 25, 2017, 19:45 UTC
Message-ID
<20170525194535.9324-14-avarab@gmail.com>
In-Reply-To
<20170525194535.9324-1-avarab@gmail.com>

Add setup code needed for testing regexes that contain both binary data and regex metacharacters.

The POSIX regcomp() function inherently can't support that, because it takes a \0-delimited char *, but other regex engines APIs like PCRE v2 take a pattern/length pair, and are thus able to handle \0s in patterns as well as any other character.

When kwset was imported in commit 9eceddeec6 ("Use kwset in grep", 2011-08-21) this limitation was fixed, but at the expense of introducing the undocumented limitation that any pattern containing \0 implicitly becomes a fixed match (equivalent to -F having been provided).

That's not something we'd like to keep in the future. The inability to match patterns containing \0 is a leaky implementation detail.

So add tests as a first step towards changing that. In order to test that \0-patterns can properly match as regexes the test string needs to have some regex metacharacters in it.

There were other blind spots in the tests. The code around kwset specially handles case-insensitive & non-ASCII data, but there were no tests for this.

Fix all of that by amending the text being matched to contain both regex metacharacters & non-ASCII data.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t7008-grep-binary.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/t/t7008-grep-binary.sh b/t/t7008-grep-binary.sh
index df93d8e44c..20370d6e0c 100755
--- a/t/t7008-grep-binary.sh
+++ b/t/t7008-grep-binary.sh
@@ -28,7 +28,7 @@ nul_match () {
 }
 
 test_expect_success 'setup' "
-	echo 'binaryQfile' | q_to_nul >a &&
+	echo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&
 	git add a &&
 	git commit -m.
 "
@@ -162,7 +162,7 @@ test_expect_success 'grep does not honor textconv' '
 '
 
 test_expect_success 'grep --textconv honors textconv' '
-	echo "a:binaryQfile" >expect &&
+	echo "a:binaryQfileQm[*]cQ*æQð" >expect &&
 	git grep --textconv Qfile >actual &&
 	test_cmp expect actual
 '
@@ -172,7 +172,7 @@ test_expect_success 'grep --no-textconv does not honor textconv' '
 '
 
 test_expect_success 'grep --textconv blob honors textconv' '
-	echo "HEAD:a:binaryQfile" >expect &&
+	echo "HEAD:a:binaryQfileQm[*]cQ*æQð" >expect &&
 	git grep --textconv Qfile HEAD:a >actual &&
 	test_cmp expect actual
 '
-- 
2.13.0.303.g4ebf302169
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 14 of 32 in “Easy to review grep & pre-PCRE changes”
  1. 00/31 Easy to review grep & pre-PCRE changesÆvar Arnfjörð Bjarmason, May 25, 2017
  2. 01/31 Makefile & configure: reword inaccurate comment about PCREÆvar Arnfjörð Bjarmason, May 25, 2017
  3. 02/31 grep & rev-list doc: stop promising libpcre for --perl-regexpÆvar Arnfjörð Bjarmason, May 25, 2017
  4. 04/31 log: add exhaustive tests for pattern style options & configÆvar Arnfjörð Bjarmason, May 25, 2017
  5. 03/31 test-lib: rename the LIBPCRE prerequisite to PCREÆvar Arnfjörð Bjarmason, May 25, 2017
  6. 05/31 log: make --regexp-ignore-case work with --perl-regexpÆvar Arnfjörð Bjarmason, May 25, 2017
  7. 07/31 grep: add a test for backreferences in PCRE patternsÆvar Arnfjörð Bjarmason, May 25, 2017
  8. 08/31 grep: change non-ASCII -i test to stop using --debugÆvar Arnfjörð Bjarmason, May 25, 2017
  9. 06/31 grep: add a test asserting that --perl-regexp dies when !PCREÆvar Arnfjörð Bjarmason, May 25, 2017
  10. 09/31 grep: add tests for --threads=N and grep.threadsÆvar Arnfjörð Bjarmason, May 25, 2017
  11. 10/31 grep: amend submodule recursion test for regex engine testingÆvar Arnfjörð Bjarmason, May 25, 2017
  12. 11/31 grep: add tests for grep pattern types being passed to submodulesÆvar Arnfjörð Bjarmason, May 25, 2017
  13. 12/31 grep: add a test helper function for less verbose -f \0 testsÆvar Arnfjörð Bjarmason, May 25, 2017
  14. 13/31 grep: prepare for testing binary regexes containing rx metacharactersÆvar Arnfjörð Bjarmason, May 25, 2017
  15. 14/31 grep: add tests to fix blind spots with \0 patternsÆvar Arnfjörð Bjarmason, May 25, 2017
  16. 16/31 perf: emit progress output when unpacking & buildingÆvar Arnfjörð Bjarmason, May 25, 2017
  17. 18/31 perf: add a comparison test of grep regex engines with -FÆvar Arnfjörð Bjarmason, May 25, 2017
  18. 17/31 perf: add a comparison test of grep regex enginesÆvar Arnfjörð Bjarmason, May 25, 2017
  19. 15/31 perf: add a GIT_PERF_MAKE_COMMAND for when *_MAKE_OPTS won't doÆvar Arnfjörð Bjarmason, May 25, 2017
  20. 19/31 perf: add a comparison test of log --grep regex enginesÆvar Arnfjörð Bjarmason, May 25, 2017
  21. 20/31 perf: add a comparison test of log --grep regex engines with -FÆvar Arnfjörð Bjarmason, May 25, 2017
  22. 24/31 grep: change the internal PCRE macro names to be PCRE1Ævar Arnfjörð Bjarmason, May 25, 2017
  23. 23/31 grep: factor test for \0 in grep patterns into a functionÆvar Arnfjörð Bjarmason, May 25, 2017
  24. 21/31 grep: catch a missing enum in switch statementÆvar Arnfjörð Bjarmason, May 25, 2017
  25. 22/31 grep: remove redundant regflags assignmentsÆvar Arnfjörð Bjarmason, May 25, 2017
  26. 28/31 pack-objects & index-pack: add test for --threads warningÆvar Arnfjörð Bjarmason, May 25, 2017
  27. 30/31 grep: given --threads with NO_PTHREADS=YesPlease, warnÆvar Arnfjörð Bjarmason, May 25, 2017
  28. 25/31 grep: change internal *pcre* variable & function names to be *pcre1*Ævar Arnfjörð Bjarmason, May 25, 2017
  29. 31/31 grep: assert that threading is enabled when calling grep_{lock,unlock}Ævar Arnfjörð Bjarmason, May 25, 2017
  30. 29/31 pack-objects: fix buggy warning about threadsÆvar Arnfjörð Bjarmason, May 25, 2017
  31. 27/31 test-lib: add a PTHREADS prerequisiteÆvar Arnfjörð Bjarmason, May 25, 2017
  32. 26/31 grep: move is_fixed() earlier to avoid forward declarationÆvar Arnfjörð Bjarmason, May 25, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.