git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 4/7] grep: add support for the PCRE v1 JIT API

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
May 23, 2017, 19:24 UTC
Message-ID
<20170523192453.14172-5-avarab@gmail.com>
In-Reply-To
<20170523192453.14172-1-avarab@gmail.com>

Change the grep PCRE v1 code to use JIT when available. When PCRE support was initially added in commit 63e7e9d8b6 ("git-grep: Learn PCRE", 2011-05-09) PCRE had no JIT support, it was integrated into 8.20 released on 2011-10-21.

Enabling JIT support usually improves performance by more than 40%. The pattern compilation times are relatively slower, but those relative numbers are tiny, and are easily made back in all but the most trivial cases of grep. Detailed benchmarks & overview of compilation times is at: http://sljit.sourceforge.net/pcre.html

With this change the difference in a t/perf/p7820-grep-engines.sh run is, with just the /perl/ tests shown:

    $ GIT_PERF_REPEAT_COUNT=30 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 USE_LIBPCRE=YesPlease CC=~/perl5/installed/bin/gcc NO_R_TO_GCC_LINKER=YesPlease CFLAGS=-O3 LIBPCREDIR=/home/avar/g/pcre/inst LDFLAGS=-Wl,-rpath,/home/avar/g/pcre/inst/lib' ./run HEAD~ HEAD p7820-grep-engines.sh
    Test                                           HEAD~             HEAD
    ---------------------------------------------------------------------------------------
    7820.3: perl grep 'how.to'                      0.35(1.11+0.43)   0.23(0.42+0.46) -34.3%
    7820.7: perl grep '^how to'                     0.64(2.71+0.36)   0.27(0.66+0.44) -57.8%
    7820.11: perl grep '[how] to'                   0.63(2.51+0.42)   0.33(0.98+0.39) -47.6%
    7820.15: perl grep '(e.t[^ ]*|v.ry) rare'       1.17(5.61+0.35)   0.34(1.08+0.46) -70.9%
    7820.19: perl grep 'm(ú|u)lt.b(æ|y)te'          0.43(1.52+0.44)   0.30(0.88+0.42) -30.2%

The conditional support for JIT is implemented as suggested in the pcrejit(3) man page. E.g. defining PCRE_STUDY_JIT_COMPILE to 0 if it's not present.

The implementation is relatively verbose because even if PCRE_CONFIG_JIT is defined only a call to pcre_config() can determine if the JIT is available, and if so the faster pcre_jit_exec() function should be called instead of pcre_exec(), and a different (but not complimentary!) function needs to be called to free pcre1_extra_info.

There's no graceful fallback if pcre_jit_stack_alloc() fails under PCRE_CONFIG_JIT, instead the program will simply abort. I don't think this is worth handling gracefully, it'll only fail in cases where malloc() doesn't work, in which case we're screwed anyway.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 grep.c | 34 +++++++++++++++++++++++++++++++++-
 grep.h |  6 ++++++
 2 files changed, 39 insertions(+), 1 deletion(-)
diff --git a/grep.c b/grep.c
index 1157529115..49e9aed457 100644
--- a/grep.c
+++ b/grep.c
@@ -351,6 +351,9 @@ static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)
 	const char *error;
 	int erroffset;
 	int options = PCRE_MULTILINE;
+#ifdef PCRE_CONFIG_JIT
+	int canjit;
+#endif
 
 	if (opt->ignore_case) {
 		if (has_non_ascii(p->pattern))
@@ -365,9 +368,20 @@ static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)
 	if (!p->pcre1_regexp)
 		compile_regexp_failed(p, error);
 
-	p->pcre1_extra_info = pcre_study(p->pcre1_regexp, 0, &error);
+	p->pcre1_extra_info = pcre_study(p->pcre1_regexp, PCRE_STUDY_JIT_COMPILE, &error);
 	if (!p->pcre1_extra_info && error)
 		die("%s", error);
+
+#ifdef PCRE_CONFIG_JIT
+	pcre_config(PCRE_CONFIG_JIT, &canjit);
+	if (canjit == 1) {
+		p->pcre1_jit_stack = pcre_jit_stack_alloc(1, 1024 * 1024);
+		if (!p->pcre1_jit_stack)
+			die("BUG: Couldn't allocate PCRE JIT stack");
+		pcre_assign_jit_stack(p->pcre1_extra_info, NULL, p->pcre1_jit_stack);
+		p->pcre1_jit_on = 1;
+	}
+#endif
 }
 
 static int pcre1match(struct grep_pat *p, const char *line, const char *eol,
@@ -378,8 +392,17 @@ static int pcre1match(struct grep_pat *p, const char *line, const char *eol,
 	if (eflags & REG_NOTBOL)
 		flags |= PCRE_NOTBOL;
 
+#ifdef PCRE_CONFIG_JIT
+	if (p->pcre1_jit_on)
+		ret = pcre_jit_exec(p->pcre1_regexp, p->pcre1_extra_info, line,
+				    eol - line, 0, flags, ovector,
+				    ARRAY_SIZE(ovector), p->pcre1_jit_stack);
+	else
+#endif
+	/* PCRE_CONFIG_JIT !p->pcre1_jit_on else branch */
 	ret = pcre_exec(p->pcre1_regexp, p->pcre1_extra_info, line, eol - line,
 			0, flags, ovector, ARRAY_SIZE(ovector));
+
 	if (ret < 0 && ret != PCRE_ERROR_NOMATCH)
 		die("pcre_exec failed with error code %d", ret);
 	if (ret > 0) {
@@ -394,7 +417,16 @@ static int pcre1match(struct grep_pat *p, const char *line, const char *eol,
 static void free_pcre1_regexp(struct grep_pat *p)
 {
 	pcre_free(p->pcre1_regexp);
+
+#ifdef PCRE_CONFIG_JIT
+	if (p->pcre1_jit_on) {
+		pcre_free_study(p->pcre1_extra_info);
+		pcre_jit_stack_free(p->pcre1_jit_stack);
+	} else
+#endif
+	/* PCRE_CONFIG_JIT !p->pcre1_jit_on else branch */
 	pcre_free(p->pcre1_extra_info);
+
 	pcre_free((void *)p->pcre1_tables);
 }
 #else /* !USE_LIBPCRE1 */
diff --git a/grep.h b/grep.h
index 38ac82b638..14f47189f9 100644
--- a/grep.h
+++ b/grep.h
@@ -3,9 +3,13 @@
 #include "color.h"
 #ifdef USE_LIBPCRE1
 #include <pcre.h>
+#ifndef PCRE_STUDY_JIT_COMPILE
+#define PCRE_STUDY_JIT_COMPILE 0
+#endif
 #else
 typedef int pcre;
 typedef int pcre_extra;
+typedef int pcre_jit_stack;
 #endif
 #include "kwset.h"
 #include "thread-utils.h"
@@ -48,7 +52,9 @@ struct grep_pat {
 	regex_t regexp;
 	pcre *pcre1_regexp;
 	pcre_extra *pcre1_extra_info;
+	pcre_jit_stack *pcre1_jit_stack;
 	const unsigned char *pcre1_tables;
+	int pcre1_jit_on;
 	kwset_t kws;
 	unsigned fixed:1;
 	unsigned ignore_case:1;

-- 
2.13.0.303.g4ebf302169
Previous: Junio C HamanoNext: Junio C Hamano
Message 47 of 58 in “Easy to review grep & pre-PCRE changes”
  1. 00/30 Easy to review grep & pre-PCRE changesÆvar Arnfjörð Bjarmason, May 20, 2017
  2. 01/30 Makefile & configure: reword inaccurate comment about PCREÆvar Arnfjörð Bjarmason, May 20, 2017
  3. 02/30 grep & rev-list doc: stop promising libpcre for --perl-regexpÆvar Arnfjörð Bjarmason, May 20, 2017
  4. 03/30 test-lib: rename the LIBPCRE prerequisite to PCREÆvar Arnfjörð Bjarmason, May 20, 2017
  5. 04/30 log: add exhaustive tests for pattern style options & configÆvar Arnfjörð Bjarmason, May 20, 2017
  6. 07/30 grep: add a test for backreferences in PCRE patternsÆvar Arnfjörð Bjarmason, May 20, 2017
  7. 05/30 log: make --regexp-ignore-case work with --perl-regexpÆvar Arnfjörð Bjarmason, May 20, 2017
  8. Junio C HamanoMay 20, 2017
  9. Ævar Arnfjörð BjarmasonMay 21, 2017
  10. Junio C HamanoMay 22, 2017
  11. 06/30 grep: add a test asserting that --perl-regexp dies when !PCREÆvar Arnfjörð Bjarmason, May 20, 2017
  12. 08/30 grep: change non-ASCII -i test to stop using --debugÆvar Arnfjörð Bjarmason, May 20, 2017
  13. 12/30 grep: add a test helper function for less verbose -f \0 testsÆvar Arnfjörð Bjarmason, May 20, 2017
  14. 10/30 grep: amend submodule recursion test for regex engine testingÆvar Arnfjörð Bjarmason, May 20, 2017
  15. 09/30 grep: add tests for --threads=N and grep.threadsÆvar Arnfjörð Bjarmason, May 20, 2017
  16. 13/30 grep: prepare for testing binary regexes containing rx metacharactersÆvar Arnfjörð Bjarmason, May 20, 2017
  17. 11/30 grep: add tests for grep pattern types being passed to submodulesÆvar Arnfjörð Bjarmason, May 20, 2017
  18. 14/30 grep: add tests to fix blind spots with \0 patternsÆvar Arnfjörð Bjarmason, May 20, 2017
  19. 15/30 perf: add a GIT_PERF_MAKE_COMMAND for when *_MAKE_OPTS won't doÆvar Arnfjörð Bjarmason, May 20, 2017
  20. Junio C HamanoMay 20, 2017
  21. Ævar Arnfjörð BjarmasonMay 21, 2017
  22. 18/30 perf: add a comparison test of grep regex engines with -FÆvar Arnfjörð Bjarmason, May 20, 2017
  23. 20/30 grep: catch a missing enum in switch statementÆvar Arnfjörð Bjarmason, May 20, 2017
  24. 17/30 perf: add a comparison test of grep regex enginesÆvar Arnfjörð Bjarmason, May 20, 2017
  25. 19/30 perf: add a comparison test of log --grep regex enginesÆvar Arnfjörð Bjarmason, May 20, 2017
  26. 16/30 perf: emit progress output when unpacking & buildingÆvar Arnfjörð Bjarmason, May 20, 2017
  27. 21/30 grep: remove redundant regflags assignmentsÆvar Arnfjörð Bjarmason, May 20, 2017
  28. 22/30 grep: factor test for \0 in grep patterns into a functionÆvar Arnfjörð Bjarmason, May 20, 2017
  29. Brandon WilliamsMay 23, 2017
  30. 25/30 grep: move is_fixed() earlier to avoid forward declarationÆvar Arnfjörð Bjarmason, May 20, 2017
  31. 23/30 grep: change the internal PCRE macro names to be PCRE1Ævar Arnfjörð Bjarmason, May 20, 2017
  32. 24/30 grep: change internal *pcre* variable & function names to be *pcre1*Ævar Arnfjörð Bjarmason, May 20, 2017
  33. 26/30 test-lib: add a PTHREADS prerequisiteÆvar Arnfjörð Bjarmason, May 20, 2017
  34. 28/30 pack-objects: fix buggy warning about threadsÆvar Arnfjörð Bjarmason, May 20, 2017
  35. 30/30 grep: assert that threading is enabled when calling grep_{lock,unlock}Ævar Arnfjörð Bjarmason, May 20, 2017
  36. 29/30 grep: given --threads with NO_PTHREADS=YesPlease, warnÆvar Arnfjörð Bjarmason, May 20, 2017
  37. 27/30 pack-objects & index-pack: add test for --threads warningÆvar Arnfjörð Bjarmason, May 20, 2017
  38. Junio C HamanoMay 20, 2017
  39. 0/7 PCRE v2, PCRE v1 JIT, log -P & fixesÆvar Arnfjörð Bjarmason, May 23, 2017
  40. 3/7 log: add -P as a synonym for --perl-regexpÆvar Arnfjörð Bjarmason, May 23, 2017
  41. 5/7 grep: un-break building with PCRE < 8.32Ævar Arnfjörð Bjarmason, May 23, 2017
  42. Junio C HamanoMay 24, 2017
  43. Junio C HamanoMay 24, 2017
  44. 6/7 grep: un-break building with PCRE < 8.20Ævar Arnfjörð Bjarmason, May 23, 2017
  45. 2/7 grep: skip pthreads overhead when using one threadÆvar Arnfjörð Bjarmason, May 23, 2017
  46. Junio C HamanoMay 24, 2017
  47. 4/7 grep: add support for the PCRE v1 JIT APIÆvar Arnfjörð Bjarmason, May 23, 2017
  48. Junio C HamanoMay 24, 2017
  49. Ævar Arnfjörð BjarmasonMay 24, 2017
  50. 7/7 grep: add support for PCRE v2Ævar Arnfjörð Bjarmason, May 23, 2017
  51. Junio C HamanoMay 24, 2017
  52. Ævar Arnfjörð BjarmasonMay 25, 2017
  53. 1/7 grep: don't redundantly compile throwaway patterns under threadingÆvar Arnfjörð Bjarmason, May 23, 2017
  54. Junio C HamanoMay 24, 2017
  55. Ævar Arnfjörð BjarmasonMay 25, 2017
  56. Junio C HamanoMay 26, 2017
  57. Ævar Arnfjörð BjarmasonMay 26, 2017
  58. Junio C HamanoMay 26, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.