git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 13/29] grep: add tests to fix blind spots with \0 patterns

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
May 13, 2017, 23:14 UTC
Message-ID
<20170513231509.7834-14-avarab@gmail.com>
In-Reply-To
<20170513231509.7834-1-avarab@gmail.com>

Address a big blind spot in the tests for patterns containing \0. The is_fixed() function considers any string that contains \0 fixed, even if it contains regular expression metacharacters, those patterns are currently matched with kwset.

Before this change removing that memchr(s, 0, len) check from is_fixed() wouldn't change the result of any of the tests, since regcomp() will happily match the part before the \0.

Furthermore, the kwset path is dependent on whether the the -i flag is on, and whether the pattern has any non-ASCII characters, but none of this was tested for.

See the a previous commit in this series ("grep: add tests to fix blind spots with \0 patterns", 2017-04-21) for further details & rationale.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t7008-grep-binary.sh | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)
diff --git a/t/t7008-grep-binary.sh b/t/t7008-grep-binary.sh
index e7754c3946..079395ba54 100755
--- a/t/t7008-grep-binary.sh
+++ b/t/t7008-grep-binary.sh
@@ -22,6 +22,18 @@ nul_match () {
 			printf '$pattern' | q_to_nul >f &&
 			test_must_fail git grep -f f $flags a
 		"
+	elif test "$status" = T1
+	then
+		test_expect_failure "git grep -f f $flags '$pattern_human' a" "
+			printf '$pattern' | q_to_nul >f &&
+			git grep -f f $flags a
+		"
+	elif test "$status" = T0
+	then
+		test_expect_failure "git grep -f f $flags '$pattern_human' a" "
+			printf '$pattern' | q_to_nul >f &&
+			test_must_fail git grep -f f $flags a
+		"
 	else
 		test_expect_success "PANIC: Test framework error. Unknown status $status" 'false'
 	fi
@@ -98,6 +110,65 @@ nul_match 1 '-Fi' 'YQf'
 nul_match 0 '-Fi' 'YQx'
 nul_match 1 '' 'yQf'
 nul_match 0 '' 'yQx'
+nul_match 1 '' 'æQð'
+nul_match 1 '-F' 'eQm[*]c'
+nul_match 1 '-Fi' 'EQM[*]C'
+
+# Regex patterns that would match but shouldn't with -F
+nul_match 0 '-F' 'yQ[f]'
+nul_match 0 '-F' '[y]Qf'
+nul_match 0 '-Fi' 'YQ[F]'
+nul_match 0 '-Fi' '[Y]QF'
+nul_match 0 '-F' 'æQ[ð]'
+nul_match 0 '-F' '[æ]Qð'
+nul_match 0 '-Fi' 'ÆQ[Ð]'
+nul_match 0 '-Fi' '[Æ]QÐ'
+
+# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \0
+# patterns case-insensitively.
+nul_match T1 '-i' 'ÆQÐ'
+
+# \0 implicitly disables regexes. This is an undocumented internal
+# limitation.
+nul_match T1 '' 'yQ[f]'
+nul_match T1 '' '[y]Qf'
+nul_match T1 '-i' 'YQ[F]'
+nul_match T1 '-i' '[Y]Qf'
+nul_match T1 '' 'æQ[ð]'
+nul_match T1 '' '[æ]Qð'
+nul_match T1 '-i' 'ÆQ[Ð]'
+
+# ... because of \0 implicitly disabling regexes regexes that
+# should/shouldn't match don't do the right thing.
+nul_match T1 '' 'eQm.*cQ'
+nul_match T1 '-i' 'EQM.*cQ'
+nul_match T0 '' 'eQm[*]c'
+nul_match T0 '-i' 'EQM[*]C'
+
+# Due to the REG_STARTEND extension when kwset() is disabled on -i &
+# non-ASCII the string will be matched in its entirety, but the
+# pattern will be cut off at the first \0.
+nul_match 0 '-i' 'NOMATCHQð'
+nul_match T0 '-i' '[Æ]QNOMATCH'
+nul_match T0 '-i' '[æ]QNOMATCH'
+# Matches, but for the wrong reasons, just stops at [æ]
+nul_match 1 '-i' '[Æ]Qð'
+nul_match 1 '-i' '[æ]Qð'
+
+# Ensure that the matcher doesn't regress to something that stops at
+# \0
+nul_match 0 '-F' 'yQ[f]'
+nul_match 0 '-Fi' 'YQ[F]'
+nul_match 0 '' 'yQNOMATCH'
+nul_match 0 '' 'QNOMATCH'
+nul_match 0 '-i' 'YQNOMATCH'
+nul_match 0 '-i' 'QNOMATCH'
+nul_match 0 '-F' 'æQ[ð]'
+nul_match 0 '-Fi' 'ÆQ[Ð]'
+nul_match 0 '' 'yQNÓMATCH'
+nul_match 0 '' 'QNÓMATCH'
+nul_match 0 '-i' 'YQNÓMATCH'
+nul_match 0 '-i' 'QNÓMATCH'
 
 test_expect_success 'grep respects binary diff attribute' '
 	echo text >t &&

-- 
2.11.0
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 17 of 42 in “Easy to review grep & pre-PCRE changes”
  1. 00/29 Easy to review grep & pre-PCRE changesÆvar Arnfjörð Bjarmason, May 13, 2017
  2. 01/29 Makefile & configure: reword inaccurate comment about PCREÆvar Arnfjörð Bjarmason, May 13, 2017
  3. 02/29 grep & rev-list doc: stop promising libpcre for --perl-regexpÆvar Arnfjörð Bjarmason, May 13, 2017
  4. 04/29 log: add exhaustive tests for pattern style options & configÆvar Arnfjörð Bjarmason, May 13, 2017
  5. Junio C HamanoMay 15, 2017
  6. Ævar Arnfjörð BjarmasonMay 15, 2017
  7. Junio C HamanoMay 16, 2017
  8. 05/29 grep: add a test asserting that --perl-regexp dies when !PCREÆvar Arnfjörð Bjarmason, May 13, 2017
  9. 03/29 test-lib: rename the LIBPCRE prerequisite to PCREÆvar Arnfjörð Bjarmason, May 13, 2017
  10. 06/29 grep: add a test for backreferences in PCRE patternsÆvar Arnfjörð Bjarmason, May 13, 2017
  11. 07/29 grep: change non-ASCII -i test to stop using --debugÆvar Arnfjörð Bjarmason, May 13, 2017
  12. 08/29 grep: add tests for --threads=N and grep.threadsÆvar Arnfjörð Bjarmason, May 13, 2017
  13. 09/29 grep: amend submodule recursion test for regex engine testingÆvar Arnfjörð Bjarmason, May 13, 2017
  14. 10/29 grep: add tests for grep pattern types being passed to submodulesÆvar Arnfjörð Bjarmason, May 13, 2017
  15. 11/29 grep: add a test helper function for less verbose -f \0 testsÆvar Arnfjörð Bjarmason, May 13, 2017
  16. 12/29 grep: prepare for testing binary regexes containing rx metacharactersÆvar Arnfjörð Bjarmason, May 13, 2017
  17. 13/29 grep: add tests to fix blind spots with \0 patternsÆvar Arnfjörð Bjarmason, May 13, 2017
  18. 14/29 perf: add a GIT_PERF_MAKE_COMMAND for when *_MAKE_OPTS won't doÆvar Arnfjörð Bjarmason, May 13, 2017
  19. 15/29 perf: emit progress output when unpacking & buildingÆvar Arnfjörð Bjarmason, May 13, 2017
  20. 16/29 perf: add a performance comparison test of grep -G, -E and -PÆvar Arnfjörð Bjarmason, May 13, 2017
  21. 18/29 grep: catch a missing enum in switch statementÆvar Arnfjörð Bjarmason, May 13, 2017
  22. Junio C HamanoMay 15, 2017
  23. Ævar Arnfjörð BjarmasonMay 15, 2017
  24. 17/29 perf: add a performance comparison of fixed-string grepÆvar Arnfjörð Bjarmason, May 13, 2017
  25. 20/29 grep: remove redundant `regflags &= ~REG_EXTENDED` assignmentsÆvar Arnfjörð Bjarmason, May 13, 2017
  26. Junio C HamanoMay 15, 2017
  27. Ævar Arnfjörð BjarmasonMay 15, 2017
  28. 19/29 grep: remove redundant regflags assignment under PCREÆvar Arnfjörð Bjarmason, May 13, 2017
  29. 21/29 grep: factor test for \0 in grep patterns into a functionÆvar Arnfjörð Bjarmason, May 13, 2017
  30. Junio C HamanoMay 15, 2017
  31. Ævar Arnfjörð BjarmasonMay 15, 2017
  32. 22/29 grep: change the internal PCRE macro names to be PCRE1Ævar Arnfjörð Bjarmason, May 13, 2017
  33. 23/29 grep: change internal *pcre* variable & function names to be *pcre1*Ævar Arnfjörð Bjarmason, May 13, 2017
  34. 25/29 test-lib: add a PTHREADS prerequisiteÆvar Arnfjörð Bjarmason, May 13, 2017
  35. 26/29 pack-objects & index-pack: add test for --threads warningÆvar Arnfjörð Bjarmason, May 13, 2017
  36. 24/29 grep: move is_fixed() earlier to avoid forward declarationÆvar Arnfjörð Bjarmason, May 13, 2017
  37. 27/29 pack-objects: fix buggy warning about threadsÆvar Arnfjörð Bjarmason, May 13, 2017
  38. Junio C HamanoMay 15, 2017
  39. Ævar Arnfjörð BjarmasonMay 15, 2017
  40. 28/29 grep: given --threads with NO_PTHREADS=YesPlease, warnÆvar Arnfjörð Bjarmason, May 13, 2017
  41. 29/29 grep: assert that threading is enabled when calling grep_{lock,unlock}Ævar Arnfjörð Bjarmason, May 13, 2017
  42. Junio C HamanoMay 15, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.