[PATCH v3 18/30] perf: add a comparison test of grep regex engines with -F

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: May 20, 2017, 21:42 UTC
Message-ID: <20170520214233.7183-19-avarab@gmail.com>
In-Reply-To: <20170520214233.7183-1-avarab@gmail.com>

Add a performance comparison test which compares both case-sensitive & case-insensitive fixed-string grep, as well as non-ASCII case-sensitive & case-insensitive grep.

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux ./run p7821-grep-engines-fixed.sh
    [...]
    Test                             this tree
    ------------------------------------------------
    7821.1: fixed grep int           0.61(1.72+0.65)
    7821.2: basic grep int           0.69(1.72+0.53)
    7821.3: extended grep int        0.60(1.72+0.54)
    7821.4: perl grep int            0.65(1.65+0.64)
    7821.6: fixed grep uncommon      0.25(0.53+0.48)
    7821.7: basic grep uncommon      0.26(0.57+0.46)
    7821.8: extended grep uncommon   0.25(0.52+0.51)
    7821.9: perl grep uncommon       0.26(0.56+0.48)
    7821.11: fixed grep æ            0.40(1.26+0.44)
    7821.12: basic grep æ            0.40(1.28+0.43)
    7821.13: extended grep æ         0.39(1.28+0.44)
    7821.14: perl grep æ             0.39(1.29+0.44)

This test needs to be run with GIT_PERF_7821_GREP_OPTS=' -i' to avoid going through the same kwset.[ch] codepath, see the "Even when -F..." comment in grep.c:

    $ GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_7821_GREP_OPTS=' -i' ./run p7821-grep-engines-fixed.sh
    [...]
    Test                                this tree
    ---------------------------------------------------
    7821.1: fixed grep -i int           1.55(1.86+0.66)
    7821.2: basic grep -i int           0.66(1.97+0.54)
    7821.3: extended grep -i int        0.72(1.88+0.62)
    7821.4: perl grep -i int            0.75(1.93+0.57)
    7821.6: fixed grep -i uncommon      0.27(0.52+0.54)
    7821.7: basic grep -i uncommon      0.25(0.58+0.44)
    7821.8: extended grep -i uncommon   0.26(0.62+0.43)
    7821.9: perl grep -i uncommon       0.26(0.55+0.53)
    7821.11: fixed grep -i æ            0.32(0.87+0.46)
    7821.12: basic grep -i æ            0.30(0.90+0.41)
    7821.13: extended grep -i æ         0.32(0.92+0.41)
    7821.14: perl grep -i æ             0.29(0.71+0.53)

I'm planning to make that not be the case, this performance test gives a baseline for comparing performance before & after any such change.

See commit ("perf: add a comparison test of grep regex engines", 2017-04-19) for details on the machine the above test run was executed on.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/perf/p7821-grep-engines-fixed.sh | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100755 t/perf/p7821-grep-engines-fixed.sh

diff --git a/t/perf/p7821-grep-engines-fixed.sh b/t/perf/p7821-grep-engines-fixed.sh
new file mode 100755
index 0000000000..d935194ecf
--- /dev/null
+++ b/t/perf/p7821-grep-engines-fixed.sh
@@ -0,0 +1,32 @@
+#!/bin/sh
+
+test_description="Comparison of git-grep's regex engines with -F
+
+Set GIT_PERF_7821_GREP_OPTS in the environment to pass options to
+git-grep. Make sure to include a leading space,
+e.g. GIT_PERF_7821_GREP_OPTS=' -w'. See p7820-grep-engines.sh for more
+options to try.
+"
+
+. ./perf-lib.sh
+
+test_perf_large_repo
+test_checkout_worktree
+
+for args in 'int' 'uncommon' 'æ'
+do
+	for engine in fixed basic extended perl
+	do
+		test_perf "$engine grep$GIT_PERF_7821_GREP_OPTS $args" "
+			git -c grep.patternType=$engine grep$GIT_PERF_7821_GREP_OPTS $args >'out.$engine.$args' || :
+		"
+	done
+
+	test_expect_success "assert that all engines found the same for$GIT_PERF_7821_GREP_OPTS $args" "
+		test_cmp 'out.fixed.$args' 'out.basic.$args' &&
+		test_cmp 'out.fixed.$args' 'out.extended.$args' &&
+		test_cmp 'out.fixed.$args' 'out.perl.$args'
+	"
+done
+
+test_done

-- 
2.13.0.303.g4ebf302169