{"thread":{"id":"46046","subject":"[PATCH v3 0/7] PCRE v2, PCRE v1 JIT, log -P & fixes","startedAt":"2017-05-25T20:05:49Z","lastAt":"2017-05-25T20:06:25Z","messageCount":8,"participants":["Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":3,"patchTotal":7},"messages":[{"id":"320762","messageId":"20170525200528.22037-1-avarab@gmail.com","threadId":"46046","inReplyTo":null,"subject":"[PATCH v3 0/7] PCRE v2, PCRE v1 JIT, log -P & fixes","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-25T20:05:21Z","receivedAt":"2017-05-25T20:05:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"See <20170523192453.14172-1-avarab@gmail.com> for v2\n(https://public-inbox.org/git/20170523192453.14172-1-avarab@gmail.com/).\n\nThis on top of \"[PATCH v4 00/31] Easy to review grep & pre-PCRE\nchanges\".\n\nThis hopefully addresses all the comments I had on v2. Comments below:\n\nÆvar Arnfjörð Bjarmason (7):\n  grep: don't redundantly compile throwaway patterns under threading\n  grep: skip pthreads overhead when using one thread\n  log: add -P as a synonym for --perl-regexp\n\nNo changes.\n\n  grep: add support for the PCRE v1 JIT API\n\n * Simplify logic around canjit variable.\n\n * s/BUG: // for things that aren't internal bugs (but keep it for one\n   that's an actual bug if it happens...)\n\n * Simplify syntax around `#ifdef PCRE_CONFIG_JIT` blocks.\n\n  grep: un-break building with PCRE < 8.32\n\n * Reword commit message for less confusion.\n\n * Macro name: s/GIT_PCRE1_CAN_DO_MODERN_JIT/GIT_PCRE1_USE_JIT/g\n\n  grep: un-break building with PCRE < 8.20\n\nNo changes.\n\n  grep: add support for PCRE v2\n\n * Makefile: Clarify comment about what LIBPCREDIR=* does now.\n\n * Makefile: One use of USE_LIBPCRE is now USE_LIBPCRE1. It's\n   logically the same, but less confusing now.\n\n * grep.c: Fix up comment syntax\n\n * grep.c: Same canjit & s/BUG: // bug fixes as noted for PCRE v1.\n\n Documentation/rev-list-options.txt |   1 +\n Makefile                           |  36 +++++--\n builtin/grep.c                     |  16 +++-\n configure.ac                       |  77 ++++++++++++---\n grep.c                             | 187 ++++++++++++++++++++++++++++++++++++-\n grep.h                             |  31 ++++++\n revision.c                         |   2 +-\n t/t4202-log.sh                     |  12 +++\n t/test-lib.sh                      |   2 +-\n 9 files changed, 335 insertions(+), 29 deletions(-)\n\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320763","messageId":"20170525200528.22037-2-avarab@gmail.com","threadId":"46046","inReplyTo":"20170525200528.22037-1-avarab@gmail.com","subject":"[PATCH v3 1/7] grep: don't redundantly compile throwaway patterns under threading","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-25T20:05:22Z","receivedAt":"2017-05-25T20:05:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the pattern compilation logic under threading so that grep\ndoesn't compile a pattern it never ends up using on the non-threaded\ncode path, only to compile it again N times for N threads which will\neach use their own copy, ignoring the initially compiled pattern.\n\nThis redundant compilation dates back to the initial introduction of\nthe threaded grep in commit 5b594f457a (\"Threaded grep\",\n2010-01-25).\n\nThere was never any reason for doing this redundant work other than an\noversight in the initial commit. Jeff King suggested on-list in\n<20170414212325.fefrl3qdjigwyitd@sigill.intra.peff.net> that this\nmight be needed to check the pattern for sanity before threaded\nexecution commences.\n\nThat's not the case. The pattern is compiled under threading in\nstart_threads() before any concurrent execution has started by calling\npthread_create(), so if the pattern contains an error we still do the\nright thing. I.e. die with one error before any threaded execution has\ncommenced, instead of e.g. spewing out an error for each N threads,\nwhich could be a regression a change like this might inadvertently\nintroduce.\n\nThis change is not meant as an optimization, any performance gains\nfrom this are in the hundreds to thousands of nanoseconds at most. If\nwe wanted more performance here we could just re-use the compiled\npatterns in multiple threads (regcomp(3) is thread-safe), or partially\nre-use them and the associated structures in the case of later PCRE\nJIT changes.\n\nRather, it's just to make the code easier to reason about. It's\nconfusing to debug this under threading & non-threading when the\nthreading codepaths redundantly compile a pattern which is never used.\n\nThe reason the patterns are recompiled is as a side-effect of\nduplicating the whole grep_opt structure, which is not thread safe,\nwritable, and munged during execution. The grep_opt structure then\npoints to the grep_pat structure where pattern or patterns are stored.\n\nI looked into e.g. splitting the API into some \"do & alloc threadsafe\nstuff\", \"spawn thread\", \"do and alloc non-threadsafe stuff\", but the\nexecution time of grep_opt_dup() & pattern compilation is trivial\ncompared to actually executing the grep, so there was no point. Even\nwith the more expensive JIT changes to follow the most expensive PCRE\npatterns take something like 0.0X milliseconds to compile at most[1].\n\nThe undocumented --debug mode added in commit 17bf35a3c7 (\"grep: teach\n--debug option to dump the parse tree\", 2012-09-13) still works\nproperly with this change. It only emits debugging info during pattern\ncompilation, which is now dumped by the pattern compiled just before\nthe first thread is started.\n\n1. http://sljit.sourceforge.net/pcre.html\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n builtin/grep.c | 14 +++++++++++---\n 1 file changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex b1095362fb..12e62fcbf3 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -224,7 +224,8 @@ static void start_threads(struct grep_opt *opt)\n \t\tint err;\n \t\tstruct grep_opt *o = grep_opt_dup(opt);\n \t\to->output = strbuf_out;\n-\t\to->debug = 0;\n+\t\tif (i)\n+\t\t\to->debug = 0;\n \t\tcompile_grep_patterns(o);\n \t\terr = pthread_create(&threads[i], NULL, run, o);\n \n@@ -1167,8 +1168,6 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tif (!opt.fixed && opt.ignore_case)\n \t\topt.regflags |= REG_ICASE;\n \n-\tcompile_grep_patterns(&opt);\n-\n \t/*\n \t * We have to find \"--\" in a separate pass, because its presence\n \t * influences how we will parse arguments that come before it.\n@@ -1245,6 +1244,15 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tnum_threads = 0;\n #endif\n \n+\tif (!num_threads)\n+\t\t/*\n+\t\t * The compiled patterns on the main path are only\n+\t\t * used when not using threading. Otherwise\n+\t\t * start_threads() below calls compile_grep_patterns()\n+\t\t * for each thread.\n+\t\t */\n+\t\tcompile_grep_patterns(&opt);\n+\n #ifndef NO_PTHREADS\n \tif (num_threads) {\n \t\tif (!(opt.name_only || opt.unmatch_name_only || opt.count)\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320764","messageId":"20170525200528.22037-3-avarab@gmail.com","threadId":"46046","inReplyTo":"20170525200528.22037-1-avarab@gmail.com","subject":"[PATCH v3 2/7] grep: skip pthreads overhead when using one thread","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-25T20:05:23Z","receivedAt":"2017-05-25T20:05:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Skip the administrative overhead of using pthreads when only using one\nthread. Instead take the non-threaded path which would be taken under\nNO_PTHREADS.\n\nThe threading support was initially added in commit\n5b594f457a (\"Threaded grep\", 2010-01-25) with a hardcoded compile-time\nnumber of 8 threads. Later the number of threads was made configurable\nin commit 89f09dd34e (\"grep: add --threads=<num> option and\ngrep.threads configuration\", 2015-12-15).\n\nThat change did not add any special handling for --threads=1. Now we\ntake a slightly faster path by skipping thread handling entirely when\n1 thread is requested.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n builtin/grep.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 12e62fcbf3..bd008cb100 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -1238,6 +1238,8 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tnum_threads = GREP_NUM_THREADS_DEFAULT;\n \telse if (num_threads < 0)\n \t\tdie(_(\"invalid number of threads specified (%d)\"), num_threads);\n+\tif (num_threads == 1)\n+\t\tnum_threads = 0;\n #else\n \tif (num_threads)\n \t\twarning(_(\"no threads support, ignoring --threads\"));\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320765","messageId":"20170525200528.22037-4-avarab@gmail.com","threadId":"46046","inReplyTo":"20170525200528.22037-1-avarab@gmail.com","subject":"[PATCH v3 3/7] log: add -P as a synonym for --perl-regexp","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-25T20:05:24Z","receivedAt":"2017-05-25T20:05:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a short -P option as a synonym for the longer --perl-regexp, for\nconsistency with the options the corresponding grep invocations\naccept.\n\nThis was intentionally omitted in commit 727b6fc3ed (\"log --grep:\naccept --basic-regexp and --perl-regexp\", 2012-10-03) for unspecified\nfuture use.\n\nMake it consistent with \"grep\" rather than to keep it open for future\nuse, and to avoid the confusion of -P meaning different things for\ngrep & log, as is the case with the -G option.\n\nAs noted in the aforementioned commit the --basic-regexp option can't\nhave a corresponding -G argument, as the log command already uses that\nfor -G<regex>.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n Documentation/rev-list-options.txt |  1 +\n revision.c                         |  2 +-\n t/t4202-log.sh                     | 12 ++++++++++++\n 3 files changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex a46f70c2b1..9c44eae55d 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -91,6 +91,7 @@ endif::git-rev-list[]\n \tConsider the limiting patterns to be fixed strings (don't interpret\n \tpattern as a regular expression).\n \n+-P::\n --perl-regexp::\n \tConsider the limiting patterns to be Perl-compatible regular\n \texpressions.\ndiff --git a/revision.c b/revision.c\nindex 4883cdd2d0..60329da1bd 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1996,7 +1996,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t\tDIFF_OPT_SET(&revs->diffopt, PICKAXE_IGNORE_CASE);\n \t} else if (!strcmp(arg, \"--fixed-strings\") || !strcmp(arg, \"-F\")) {\n \t\trevs->grep_filter.pattern_type_option = GREP_PATTERN_TYPE_FIXED;\n-\t} else if (!strcmp(arg, \"--perl-regexp\")) {\n+\t} else if (!strcmp(arg, \"--perl-regexp\") || !strcmp(arg, \"-P\")) {\n \t\trevs->grep_filter.pattern_type_option = GREP_PATTERN_TYPE_PCRE;\n \t} else if (!strcmp(arg, \"--all-match\")) {\n \t\trevs->grep_filter.all_match = 1;\ndiff --git a/t/t4202-log.sh b/t/t4202-log.sh\nindex c44c4337f8..e6050faa14 100755\n--- a/t/t4202-log.sh\n+++ b/t/t4202-log.sh\n@@ -404,8 +404,20 @@ test_expect_success 'log with various grep.patternType configurations & command-\n \t\t\t--grep=\"(1|2)\" >actual.fixed.short-arg &&\n \t\tgit log --pretty=tformat:%s -E \\\n \t\t\t--grep=\"\\|2\" >actual.extended.short-arg &&\n+\t\tif test_have_prereq PCRE\n+\t\tthen\n+\t\t\tgit log --pretty=tformat:%s -P \\\n+\t\t\t\t--grep=\"[\\d]\\|\" >actual.perl.short-arg\n+\t\telse\n+\t\t\ttest_must_fail git log -P \\\n+\t\t\t\t--grep=\"[\\d]\\|\"\n+\t\tfi &&\n \t\ttest_cmp expect.fixed actual.fixed.short-arg &&\n \t\ttest_cmp expect.extended actual.extended.short-arg &&\n+\t\tif test_have_prereq PCRE\n+\t\tthen\n+\t\t\ttest_cmp expect.perl actual.perl.short-arg\n+\t\tfi &&\n \n \t\tgit log --pretty=tformat:%s --fixed-strings \\\n \t\t\t--grep=\"(1|2)\" >actual.fixed.long-arg &&\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320766","messageId":"20170525200528.22037-6-avarab@gmail.com","threadId":"46046","inReplyTo":"20170525200528.22037-1-avarab@gmail.com","subject":"[PATCH v3 5/7] grep: un-break building with PCRE < 8.32","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-25T20:05:26Z","receivedAt":"2017-05-25T20:06:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend my change earlier in this series (\"grep: add support for the\nPCRE v1 JIT API\", 2017-04-11) to un-break the build on PCRE v1\nversions earlier than 8.32.\n\nThe JIT support was added in version 8.20 released on 2011-10-21, but\nit wasn't until 8.32 released on 2012-11-30 that the fast code path to\nuse the JIT via pcre_jit_exec() was added[1] (see also [2]).\n\nThis means that versions 8.20 through 8.31 could still use the JIT,\nbut supporting it on those versions would add to the already verbose\nmacro soup around JIT support it, and I don't expect that the use-case\nof compiling a brand new git against a 5 year old PCRE is particularly\ncommon, and if someone does that they can just get the existing\npre-JIT slow codepath.\n\nSo just take the easy way out and disable the JIT on any version older\nthan 8.32.\n\nThe reason this change isn't part of the initial change PCRE JIT\nsupport is to have a cleaner history showing which parts of the\nimplementation are only used for ancient PCRE versions. This also\nmakes it easier to revert this change if we ever decide to stop\nsupporting those old versions.\n\n1. http://www.pcre.org/original/changelog.txt (\"28. Introducing a\n   native interface for JIT. Through this interface, the\n   compiled[...]\")\n2. https://bugs.exim.org/show_bug.cgi?id=2121\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n grep.c | 6 +++---\n grep.h | 5 +++++\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex ffe95995ee..19fa67c34c 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -369,7 +369,7 @@ static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \tif (!p->pcre1_extra_info && error)\n \t\tdie(\"%s\", error);\n \n-#ifdef PCRE_CONFIG_JIT\n+#ifdef GIT_PCRE1_USE_JIT\n \tpcre_config(PCRE_CONFIG_JIT, &p->pcre1_jit_on);\n \tif (p->pcre1_jit_on == 1) {\n \t\tp->pcre1_jit_stack = pcre_jit_stack_alloc(1, 1024 * 1024);\n@@ -391,7 +391,7 @@ static int pcre1match(struct grep_pat *p, const char *line, const char *eol,\n \tif (eflags & REG_NOTBOL)\n \t\tflags |= PCRE_NOTBOL;\n \n-#ifdef PCRE_CONFIG_JIT\n+#ifdef GIT_PCRE1_USE_JIT\n \tif (p->pcre1_jit_on) {\n \t\tret = pcre_jit_exec(p->pcre1_regexp, p->pcre1_extra_info, line,\n \t\t\t\t    eol - line, 0, flags, ovector,\n@@ -418,7 +418,7 @@ static int pcre1match(struct grep_pat *p, const char *line, const char *eol,\n static void free_pcre1_regexp(struct grep_pat *p)\n {\n \tpcre_free(p->pcre1_regexp);\n-#ifdef PCRE_CONFIG_JIT\n+#ifdef GIT_PCRE1_USE_JIT\n \tif (p->pcre1_jit_on) {\n \t\tpcre_free_study(p->pcre1_extra_info);\n \t\tpcre_jit_stack_free(p->pcre1_jit_stack);\ndiff --git a/grep.h b/grep.h\nindex 14f47189f9..3b948d9155 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -3,6 +3,11 @@\n #include \"color.h\"\n #ifdef USE_LIBPCRE1\n #include <pcre.h>\n+#ifdef PCRE_CONFIG_JIT\n+#if PCRE_MAJOR >= 8 && PCRE_MINOR >= 32\n+#define GIT_PCRE1_USE_JIT\n+#endif\n+#endif\n #ifndef PCRE_STUDY_JIT_COMPILE\n #define PCRE_STUDY_JIT_COMPILE 0\n #endif\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320767","messageId":"20170525200528.22037-8-avarab@gmail.com","threadId":"46046","inReplyTo":"20170525200528.22037-1-avarab@gmail.com","subject":"[PATCH v3 7/7] grep: add support for PCRE v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-25T20:05:28Z","receivedAt":"2017-05-25T20:06:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add support for v2 of the PCRE API. This is a new major version of\nPCRE that came out in early 2015[1].\n\nThe regular expression syntax is the same, but while the API is\nsimilar, pretty much every function is either renamed or takes\ndifferent arguments. Thus using it via entirely new functions makes\nsense, as opposed to trying to e.g. have one compile_pcre_pattern()\nthat would call either PCRE v1 or v2 functions.\n\nGit can now be compiled with either USE_LIBPCRE1=YesPlease or\nUSE_LIBPCRE2=YesPlease, with USE_LIBPCRE=YesPlease currently being a\nsynonym for the former. Providing both is a compile-time error.\n\nWith earlier patches to enable JIT for PCRE v1 the performance of the\nrelease versions of both libraries is almost exactly the same, with\nPCRE v2 being around 1% slower.\n\nHowever after I reported this to the pcre-dev mailing list[2] I got a\nlot of help with the API use from Zoltán Herczeg, he subsequently\noptimized some of the JIT functionality in v2 of the library.\n\nRunning the p7820-grep-engines.sh performance test against the latest\nSubversion trunk of both, with both them and git compiled as -O3, and\nthe test run against linux.git, gives the following results. Just the\n/perl/ tests shown:\n\n    $ GIT_PERF_REPEAT_COUNT=30 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_COMMAND='grep -q LIBPCRE2 Makefile && make -j8 USE_LIBPCRE2=YesPlease CC=~/perl5/installed/bin/gcc NO_R_TO_GCC_LINKER=YesPlease CFLAGS=-O3 LIBPCREDIR=/home/avar/g/pcre2/inst LDFLAGS=-Wl,-rpath,/home/avar/g/pcre2/inst/lib || make -j8 USE_LIBPCRE=YesPlease CC=~/perl5/installed/bin/gcc NO_R_TO_GCC_LINKER=YesPlease CFLAGS=-O3 LIBPCREDIR=/home/avar/g/pcre/inst LDFLAGS=-Wl,-rpath,/home/avar/g/pcre/inst/lib' ./run HEAD~2 HEAD~ HEAD p7820-grep-engines.sh\n    [...]\n    Test                                           HEAD~2            HEAD~                    HEAD\n    ----------------------------------------------------------------------------------------------------------------\n    7820.3: perl grep 'how.to'                      0.22(0.40+0.48)   0.22(0.31+0.58) +0.0%   0.22(0.26+0.59) +0.0%\n    7820.7: perl grep '^how to'                     0.27(0.62+0.50)   0.28(0.60+0.50) +3.7%   0.22(0.25+0.60) -18.5%\n    7820.11: perl grep '[how] to'                   0.33(0.92+0.47)   0.33(0.94+0.45) +0.0%   0.25(0.42+0.51) -24.2%\n    7820.15: perl grep '(e.t[^ ]*|v.ry) rare'       0.35(1.08+0.46)   0.35(1.12+0.41) +0.0%   0.25(0.52+0.50) -28.6%\n    7820.19: perl grep 'm(ú|u)lt.b(æ|y)te'          0.30(0.78+0.51)   0.30(0.86+0.42) +0.0%   0.25(0.29+0.54) -16.7%\n\nSee commit (\"perf: add a comparison test of grep regex engines\",\n2017-04-19) for details on the machine the above test run was executed\non.\n\nHere HEAD~2 is git with PCRE v1 without JIT, HEAD~ is PCRE v1 with\nJIT, and HEAD is PCRE v2 (also with JIT). See previous commits of mine\nmentioning p7820-grep-engines.sh for more details on the test setup.\n\nFor ease of readability, a different run just of HEAD~ (PCRE v1 with\nJIT v.s. PCRE v2), again with just the /perl/ tests shown:\n\n    Test                                           HEAD~             HEAD\n    ---------------------------------------------------------------------------------------\n    7820.3: perl grep 'how.to'                      0.23(0.41+0.47)   0.23(0.26+0.59) +0.0%\n    7820.7: perl grep '^how to'                     0.27(0.64+0.47)   0.23(0.28+0.56) -14.8%\n    7820.11: perl grep '[how] to'                   0.34(0.95+0.44)   0.25(0.38+0.56) -26.5%\n    7820.15: perl grep '(e.t[^ ]*|v.ry) rare'       0.34(1.07+0.46)   0.24(0.52+0.49) -29.4%\n    7820.19: perl grep 'm(ú|u)lt.b(æ|y)te'          0.30(0.81+0.46)   0.22(0.33+0.54) -26.7%\n\nI.e. the two are either neck-to-neck, but PCRE v2 usually pulls ahead,\nwhen it does it's around 20% faster.\n\nA brief note on thread safety: As noted in pcre2api(3) & pcre2jit(3)\nthe compiled pattern can be shared between threads, but not some of\nthe JIT context, however the grep threading support does all pattern &\nJIT compilation in separate threads, so this code doesn't need to\nconcern itself with thread safety.\n\nSee commit 63e7e9d8b6 (\"git-grep: Learn PCRE\", 2011-05-09) for the\ninitial addition of PCRE v1. This change follows some of the same\npatterns it did (and which were discussed on list at the time),\ne.g. mocking up types with typedef instead of ifdef-ing them out when\nUSE_LIBPCRE2 isn't defined. This adds some trivial memory use to the\nprogram, but makes the code look nicer.\n\n1. https://lists.exim.org/lurker/message/20150105.162835.0666407a.en.html\n2. https://lists.exim.org/lurker/thread/20170419.172322.833ee099.en.html\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n Makefile      |  36 ++++++++++----\n configure.ac  |  77 +++++++++++++++++++++++++-----\n grep.c        | 147 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n grep.h        |  17 +++++++\n t/test-lib.sh |   2 +-\n 5 files changed, 258 insertions(+), 21 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex a79274e5e6..e5a26f77a5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -29,8 +29,15 @@ all::\n # Perl-compatible regular expressions instead of standard or extended\n # POSIX regular expressions.\n #\n-# Define LIBPCREDIR=/foo/bar if your libpcre header and library files are in\n-# /foo/bar/include and /foo/bar/lib directories.\n+# Currently USE_LIBPCRE is a synonym for USE_LIBPCRE1, define\n+# USE_LIBPCRE2 instead if you'd like to use version 2 of the PCRE\n+# library. The USE_LIBPCRE flag will likely be changed to mean v2 by\n+# default in future releases.\n+#\n+# Define LIBPCREDIR=/foo/bar if your PCRE header and library files are\n+# in /foo/bar/include and /foo/bar/lib directories. Which version of\n+# PCRE this points to determined by the USE_LIBPCRE1 and USE_LIBPCRE2\n+# variables.\n #\n # Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.\n #\n@@ -1087,15 +1094,27 @@ ifdef NO_LIBGEN_H\n \tCOMPAT_OBJS += compat/basename.o\n endif\n \n-ifdef USE_LIBPCRE\n-\tBASIC_CFLAGS += -DUSE_LIBPCRE1\n-\tifdef LIBPCREDIR\n-\t\tBASIC_CFLAGS += -I$(LIBPCREDIR)/include\n-\t\tEXTLIBS += -L$(LIBPCREDIR)/$(lib) $(CC_LD_DYNPATH)$(LIBPCREDIR)/$(lib)\n+USE_LIBPCRE1 ?= $(USE_LIBPCRE)\n+\n+ifneq (,$(USE_LIBPCRE1))\n+\tifdef USE_LIBPCRE2\n+$(error Only set USE_LIBPCRE1 (or its alias USE_LIBPCRE) or USE_LIBPCRE2, not both!)\n \tendif\n+\n+\tBASIC_CFLAGS += -DUSE_LIBPCRE1\n \tEXTLIBS += -lpcre\n endif\n \n+ifdef USE_LIBPCRE2\n+\tBASIC_CFLAGS += -DUSE_LIBPCRE2\n+\tEXTLIBS += -lpcre2-8\n+endif\n+\n+ifdef LIBPCREDIR\n+\tBASIC_CFLAGS += -I$(LIBPCREDIR)/include\n+\tEXTLIBS += -L$(LIBPCREDIR)/$(lib) $(CC_LD_DYNPATH)$(LIBPCREDIR)/$(lib)\n+endif\n+\n ifdef HAVE_ALLOCA_H\n \tBASIC_CFLAGS += -DHAVE_ALLOCA_H\n endif\n@@ -2240,7 +2259,8 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo TAR=\\''$(subst ','\\'',$(subst ','\\'',$(TAR)))'\\' >>$@+\n \t@echo NO_CURL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_CURL)))'\\' >>$@+\n \t@echo NO_EXPAT=\\''$(subst ','\\'',$(subst ','\\'',$(NO_EXPAT)))'\\' >>$@+\n-\t@echo USE_LIBPCRE1=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE)))'\\' >>$@+\n+\t@echo USE_LIBPCRE1=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE1)))'\\' >>$@+\n+\t@echo USE_LIBPCRE2=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE2)))'\\' >>$@+\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PTHREADS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PTHREADS)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\ndiff --git a/configure.ac b/configure.ac\nindex deeb968daa..11d083fbe0 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -255,21 +255,61 @@ GIT_PARSE_WITH([openssl]))\n # Perl-compatible regular expressions instead of standard or extended\n # POSIX regular expressions.\n #\n-# Define LIBPCREDIR=/foo/bar if your libpcre header and library files are in\n+# Currently USE_LIBPCRE is a synonym for USE_LIBPCRE1, define\n+# USE_LIBPCRE2 instead if you'd like to use version 2 of the PCRE\n+# library. The USE_LIBPCRE flag will likely be changed to mean v2 by\n+# default in future releases.\n+#\n+# Define LIBPCREDIR=/foo/bar if your PCRE header and library files are in\n # /foo/bar/include and /foo/bar/lib directories.\n #\n AC_ARG_WITH(libpcre,\n-AS_HELP_STRING([--with-libpcre],[support Perl-compatible regexes (default is NO)])\n+AS_HELP_STRING([--with-libpcre],[synonym for --with-libpcre1]),\n+    if test \"$withval\" = \"no\"; then\n+\tUSE_LIBPCRE1=\n+    elif test \"$withval\" = \"yes\"; then\n+\tUSE_LIBPCRE1=YesPlease\n+    else\n+\tUSE_LIBPCRE1=YesPlease\n+\tLIBPCREDIR=$withval\n+\tAC_MSG_NOTICE([Setting LIBPCREDIR to $LIBPCREDIR])\n+        dnl USE_LIBPCRE1 can still be modified below, so don't substitute\n+        dnl it yet.\n+\tGIT_CONF_SUBST([LIBPCREDIR])\n+    fi)\n+\n+AC_ARG_WITH(libpcre1,\n+AS_HELP_STRING([--with-libpcre1],[support Perl-compatible regexes via libpcre1 (default is NO)])\n+AS_HELP_STRING([],           [ARG can be also prefix for libpcre library and headers]),\n+    if test \"$withval\" = \"no\"; then\n+\tUSE_LIBPCRE1=\n+    elif test \"$withval\" = \"yes\"; then\n+\tUSE_LIBPCRE1=YesPlease\n+    else\n+\tUSE_LIBPCRE1=YesPlease\n+\tLIBPCREDIR=$withval\n+\tAC_MSG_NOTICE([Setting LIBPCREDIR to $LIBPCREDIR])\n+        dnl USE_LIBPCRE1 can still be modified below, so don't substitute\n+        dnl it yet.\n+\tGIT_CONF_SUBST([LIBPCREDIR])\n+    fi)\n+\n+AC_ARG_WITH(libpcre2,\n+AS_HELP_STRING([--with-libpcre2],[support Perl-compatible regexes via libpcre2 (default is NO)])\n AS_HELP_STRING([],           [ARG can be also prefix for libpcre library and headers]),\n+    if test -n \"$USE_LIBPCRE1\"; then\n+        AC_MSG_ERROR([Only supply one of --with-libpcre1 or --with-libpcre2!])\n+    fi\n+\n     if test \"$withval\" = \"no\"; then\n-\tUSE_LIBPCRE=\n+\tUSE_LIBPCRE2=\n     elif test \"$withval\" = \"yes\"; then\n-\tUSE_LIBPCRE=YesPlease\n+\tUSE_LIBPCRE2=YesPlease\n     else\n-\tUSE_LIBPCRE=YesPlease\n+\tUSE_LIBPCRE2=YesPlease\n \tLIBPCREDIR=$withval\n \tAC_MSG_NOTICE([Setting LIBPCREDIR to $LIBPCREDIR])\n-        dnl USE_LIBPCRE can still be modified below, so don't substitute\n+        dnl USE_LIBPCRE2 can still be modified below, so don't substitute\n         dnl it yet.\n \tGIT_CONF_SUBST([LIBPCREDIR])\n     fi)\n@@ -501,13 +541,11 @@ GIT_CONF_SUBST([NEEDS_SSL_WITH_CRYPTO])\n GIT_CONF_SUBST([NO_OPENSSL])\n \n #\n-# Define USE_LIBPCRE if you have and want to use libpcre. Various\n-# commands such as log and grep offer runtime options to use\n-# Perl-compatible regular expressions instead of standard or extended\n-# POSIX regular expressions.\n+# Handle the USE_LIBPCRE1 and USE_LIBPCRE2 options potentially set\n+# above.\n #\n \n-if test -n \"$USE_LIBPCRE\"; then\n+if test -n \"$USE_LIBPCRE1\"; then\n \n GIT_STASH_FLAGS($LIBPCREDIR)\n \n@@ -517,7 +555,22 @@ AC_CHECK_LIB([pcre], [pcre_version],\n \n GIT_UNSTASH_FLAGS($LIBPCREDIR)\n \n-GIT_CONF_SUBST([USE_LIBPCRE])\n+GIT_CONF_SUBST([USE_LIBPCRE1])\n+\n+fi\n+\n+\n+if test -n \"$USE_LIBPCRE2\"; then\n+\n+GIT_STASH_FLAGS($LIBPCREDIR)\n+\n+AC_CHECK_LIB([pcre2-8], [pcre2_config_8],\n+[USE_LIBPCRE2=YesPlease],\n+[USE_LIBPCRE2=])\n+\n+GIT_UNSTASH_FLAGS($LIBPCREDIR)\n+\n+GIT_CONF_SUBST([USE_LIBPCRE2])\n \n fi\n \ndiff --git a/grep.c b/grep.c\nindex 19fa67c34c..e4c1ead104 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -179,22 +179,37 @@ static void grep_set_pattern_type_option(enum grep_pattern_type pattern_type, st\n \tcase GREP_PATTERN_TYPE_BRE:\n \t\topt->fixed = 0;\n \t\topt->pcre1 = 0;\n+\t\topt->pcre2 = 0;\n \t\tbreak;\n \n \tcase GREP_PATTERN_TYPE_ERE:\n \t\topt->fixed = 0;\n \t\topt->pcre1 = 0;\n+\t\topt->pcre2 = 0;\n \t\topt->regflags |= REG_EXTENDED;\n \t\tbreak;\n \n \tcase GREP_PATTERN_TYPE_FIXED:\n \t\topt->fixed = 1;\n \t\topt->pcre1 = 0;\n+\t\topt->pcre2 = 0;\n \t\tbreak;\n \n \tcase GREP_PATTERN_TYPE_PCRE:\n \t\topt->fixed = 0;\n+#ifdef USE_LIBPCRE2\n+\t\topt->pcre1 = 0;\n+\t\topt->pcre2 = 1;\n+#else\n+\t\t/*\n+\t\t * It's important that pcre1 always be assigned to\n+\t\t * even when there's no USE_LIBPCRE* defined. We still\n+\t\t * call the PCRE stub function, it just dies with\n+\t\t * \"cannot use Perl-compatible regexes[...]\".\n+\t\t */\n \t\topt->pcre1 = 1;\n+\t\topt->pcre2 = 0;\n+#endif\n \t\tbreak;\n \t}\n }\n@@ -446,6 +461,129 @@ static void free_pcre1_regexp(struct grep_pat *p)\n }\n #endif /* !USE_LIBPCRE1 */\n \n+#ifdef USE_LIBPCRE2\n+static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt)\n+{\n+\tint error;\n+\tPCRE2_UCHAR errbuf[256];\n+\tPCRE2_SIZE erroffset;\n+\tint options = PCRE2_MULTILINE;\n+\tconst uint8_t *character_tables = NULL;\n+\tint jitret;\n+\n+\tassert(opt->pcre2);\n+\n+\tp->pcre2_compile_context = NULL;\n+\n+\tif (opt->ignore_case) {\n+\t\tif (has_non_ascii(p->pattern)) {\n+\t\t\tcharacter_tables = pcre2_maketables(NULL);\n+\t\t\tp->pcre2_compile_context = pcre2_compile_context_create(NULL);\n+\t\t\tpcre2_set_character_tables(p->pcre2_compile_context, character_tables);\n+\t\t}\n+\t\toptions |= PCRE2_CASELESS;\n+\t}\n+\tif (is_utf8_locale() && has_non_ascii(p->pattern))\n+\t\toptions |= PCRE2_UTF;\n+\n+\tp->pcre2_pattern = pcre2_compile((PCRE2_SPTR)p->pattern,\n+\t\t\t\t\t p->patternlen, options, &error, &erroffset,\n+\t\t\t\t\t p->pcre2_compile_context);\n+\n+\tif (p->pcre2_pattern) {\n+\t\tp->pcre2_match_data = pcre2_match_data_create_from_pattern(p->pcre2_pattern, NULL);\n+\t\tif (!p->pcre2_match_data)\n+\t\t\tdie(\"Couldn't allocate PCRE2 match data\");\n+\t} else {\n+\t\tpcre2_get_error_message(error, errbuf, sizeof(errbuf));\n+\t\tcompile_regexp_failed(p, (const char *)&errbuf);\n+\t}\n+\n+\tpcre2_config(PCRE2_CONFIG_JIT, &p->pcre2_jit_on);\n+\tif (p->pcre2_jit_on == 1) {\n+\t\tjitret = pcre2_jit_compile(p->pcre2_pattern, PCRE2_JIT_COMPLETE);\n+\t\tif (!jitret)\n+\t\t\tp->pcre2_jit_on = 1;\n+\t\telse\n+\t\t\tdie(\"Couldn't JIT the PCRE2 pattern '%s', got '%d'\\n\", p->pattern, jitret);\n+\t\tp->pcre2_jit_stack = pcre2_jit_stack_create(1, 1024 * 1024, NULL);\n+\t\tif (!p->pcre2_jit_stack)\n+\t\t\tdie(\"Couldn't allocate PCRE2 JIT stack\");\n+\t\tp->pcre2_match_context = pcre2_match_context_create(NULL);\n+\t\tif (!p->pcre2_jit_stack)\n+\t\t\tdie(\"Couldn't allocate PCRE2 match context\");\n+\t\tpcre2_jit_stack_assign(p->pcre2_match_context, NULL, p->pcre2_jit_stack);\n+\t} else if (p->pcre2_jit_on != 0) {\n+\t\tdie(\"BUG: The pcre2_jit_on variable should be 0 or 1, not %d\",\n+\t\t    p->pcre1_jit_on);\n+\t}\n+}\n+\n+static int pcre2match(struct grep_pat *p, const char *line, const char *eol,\n+\t\tregmatch_t *match, int eflags)\n+{\n+\tint ret, flags = 0;\n+\tPCRE2_SIZE *ovector;\n+\tPCRE2_UCHAR errbuf[256];\n+\n+\tif (eflags & REG_NOTBOL)\n+\t\tflags |= PCRE2_NOTBOL;\n+\n+\tif (p->pcre2_jit_on)\n+\t\tret = pcre2_jit_match(p->pcre2_pattern, (unsigned char *)line,\n+\t\t\t\t      eol - line, 0, flags, p->pcre2_match_data,\n+\t\t\t\t      NULL);\n+\telse\n+\t\tret = pcre2_match(p->pcre2_pattern, (unsigned char *)line,\n+\t\t\t\t  eol - line, 0, flags, p->pcre2_match_data,\n+\t\t\t\t  NULL);\n+\n+\tif (ret < 0 && ret != PCRE2_ERROR_NOMATCH) {\n+\t\tpcre2_get_error_message(ret, errbuf, sizeof(errbuf));\n+\t\tdie(\"%s failed with error code %d: %s\",\n+\t\t    (p->pcre2_jit_on ? \"pcre2_jit_match\" : \"pcre2_match\"), ret,\n+\t\t    errbuf);\n+\t}\n+\tif (ret > 0) {\n+\t\tovector = pcre2_get_ovector_pointer(p->pcre2_match_data);\n+\t\tret = 0;\n+\t\tmatch->rm_so = (int)ovector[0];\n+\t\tmatch->rm_eo = (int)ovector[1];\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static void free_pcre2_pattern(struct grep_pat *p)\n+{\n+\tpcre2_compile_context_free(p->pcre2_compile_context);\n+\tpcre2_code_free(p->pcre2_pattern);\n+\tpcre2_match_data_free(p->pcre2_match_data);\n+\tpcre2_jit_stack_free(p->pcre2_jit_stack);\n+\tpcre2_match_context_free(p->pcre2_match_context);\n+}\n+#else /* !USE_LIBPCRE2 */\n+static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt)\n+{\n+\t/*\n+\t * Unreachable until USE_LIBPCRE2 becomes synonymous with\n+\t * USE_LIBPCRE. See the sibling comment in\n+\t * grep_set_pattern_type_option().\n+\t */\n+\tdie(\"cannot use Perl-compatible regexes when not compiled with USE_LIBPCRE\");\n+}\n+\n+static int pcre2match(struct grep_pat *p, const char *line, const char *eol,\n+\t\tregmatch_t *match, int eflags)\n+{\n+\treturn 1;\n+}\n+\n+static void free_pcre2_pattern(struct grep_pat *p)\n+{\n+}\n+#endif /* !USE_LIBPCRE2 */\n+\n static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -511,6 +649,11 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\treturn;\n \t}\n \n+\tif (opt->pcre2) {\n+\t\tcompile_pcre2_pattern(p, opt);\n+\t\treturn;\n+\t}\n+\n \tif (opt->pcre1) {\n \t\tcompile_pcre1_regexp(p, opt);\n \t\treturn;\n@@ -870,6 +1013,8 @@ void free_grep_patterns(struct grep_opt *opt)\n \t\t\t\tkwsfree(p->kws);\n \t\t\telse if (p->pcre1_regexp)\n \t\t\t\tfree_pcre1_regexp(p);\n+\t\t\telse if (p->pcre2_pattern)\n+\t\t\t\tfree_pcre2_pattern(p);\n \t\t\telse\n \t\t\t\tregfree(&p->regexp);\n \t\t\tfree(p->pattern);\n@@ -950,6 +1095,8 @@ static int patmatch(struct grep_pat *p, char *line, char *eol,\n \t\thit = !fixmatch(p, line, eol, match);\n \telse if (p->pcre1_regexp)\n \t\thit = !pcre1match(p, line, eol, match, eflags);\n+\telse if (p->pcre2_pattern)\n+\t\thit = !pcre2match(p, line, eol, match, eflags);\n \telse\n \t\thit = !regexec_buf(&p->regexp, line, eol - line, 1, match,\n \t\t\t\t   eflags);\ndiff --git a/grep.h b/grep.h\nindex ce90969736..246f6695c1 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -19,6 +19,16 @@ typedef int pcre;\n typedef int pcre_extra;\n typedef int pcre_jit_stack;\n #endif\n+#ifdef USE_LIBPCRE2\n+#define PCRE2_CODE_UNIT_WIDTH 8\n+#include <pcre2.h>\n+#else\n+typedef int pcre2_code;\n+typedef int pcre2_match_data;\n+typedef int pcre2_compile_context;\n+typedef int pcre2_match_context;\n+typedef int pcre2_jit_stack;\n+#endif\n #include \"kwset.h\"\n #include \"thread-utils.h\"\n #include \"userdiff.h\"\n@@ -63,6 +73,12 @@ struct grep_pat {\n \tpcre_jit_stack *pcre1_jit_stack;\n \tconst unsigned char *pcre1_tables;\n \tint pcre1_jit_on;\n+\tpcre2_code *pcre2_pattern;\n+\tpcre2_match_data *pcre2_match_data;\n+\tpcre2_compile_context *pcre2_compile_context;\n+\tpcre2_match_context *pcre2_match_context;\n+\tpcre2_jit_stack *pcre2_jit_stack;\n+\tuint32_t pcre2_jit_on;\n \tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\n@@ -126,6 +142,7 @@ struct grep_opt {\n \tint extended;\n \tint use_reflog_filter;\n \tint pcre1;\n+\tint pcre2;\n \tint relative;\n \tint pathname;\n \tint null_following_name;\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 43529451f9..f5da636bea 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1015,7 +1015,7 @@ esac\n test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PTHREADS\" && test_set_prereq PTHREADS\n test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n-test -n \"$USE_LIBPCRE1\" && test_set_prereq PCRE\n+test -n \"$USE_LIBPCRE1$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n \n # Can we rely on git's output in the C locale?\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320768","messageId":"20170525200528.22037-5-avarab@gmail.com","threadId":"46046","inReplyTo":"20170525200528.22037-1-avarab@gmail.com","subject":"[PATCH v3 4/7] grep: add support for the PCRE v1 JIT API","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-25T20:05:25Z","receivedAt":"2017-05-25T20:06:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the grep PCRE v1 code to use JIT when available. When PCRE\nsupport was initially added in commit 63e7e9d8b6 (\"git-grep: Learn\nPCRE\", 2011-05-09) PCRE had no JIT support, it was integrated into\n8.20 released on 2011-10-21.\n\nEnabling JIT support usually improves performance by more than\n40%. The pattern compilation times are relatively slower, but those\nrelative numbers are tiny, and are easily made back in all but the\nmost trivial cases of grep. Detailed benchmarks & overview of\ncompilation times is at: http://sljit.sourceforge.net/pcre.html\n\nWith this change the difference in a t/perf/p7820-grep-engines.sh run\nis, with just the /perl/ tests shown:\n\n    $ GIT_PERF_REPEAT_COUNT=30 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 USE_LIBPCRE=YesPlease CC=~/perl5/installed/bin/gcc NO_R_TO_GCC_LINKER=YesPlease CFLAGS=-O3 LIBPCREDIR=/home/avar/g/pcre/inst LDFLAGS=-Wl,-rpath,/home/avar/g/pcre/inst/lib' ./run HEAD~ HEAD p7820-grep-engines.sh\n    Test                                           HEAD~             HEAD\n    ---------------------------------------------------------------------------------------\n    7820.3: perl grep 'how.to'                      0.35(1.11+0.43)   0.23(0.42+0.46) -34.3%\n    7820.7: perl grep '^how to'                     0.64(2.71+0.36)   0.27(0.66+0.44) -57.8%\n    7820.11: perl grep '[how] to'                   0.63(2.51+0.42)   0.33(0.98+0.39) -47.6%\n    7820.15: perl grep '(e.t[^ ]*|v.ry) rare'       1.17(5.61+0.35)   0.34(1.08+0.46) -70.9%\n    7820.19: perl grep 'm(ú|u)lt.b(æ|y)te'          0.43(1.52+0.44)   0.30(0.88+0.42) -30.2%\n\nThe conditional support for JIT is implemented as suggested in the\npcrejit(3) man page. E.g. defining PCRE_STUDY_JIT_COMPILE to 0 if it's\nnot present.\n\nThe implementation is relatively verbose because even if\nPCRE_CONFIG_JIT is defined only a call to pcre_config() can determine\nif the JIT is available, and if so the faster pcre_jit_exec() function\nshould be called instead of pcre_exec(), and a different (but not\ncomplimentary!) function needs to be called to free pcre1_extra_info.\n\nThere's no graceful fallback if pcre_jit_stack_alloc() fails under\nPCRE_CONFIG_JIT, instead the program will simply abort. I don't think\nthis is worth handling gracefully, it'll only fail in cases where\nmalloc() doesn't work, in which case we're screwed anyway.\n\nThat there's no assignment of `p->pcre1_jit_on = 0` when\nPCRE_CONFIG_JIT isn't defined isn't a bug. The create_grep_pat()\nfunction allocates the grep_pat allocates it with calloc(), so it's\nguaranteed to be 0 when PCRE_CONFIG_JIT isn't defined.\n\nI you're bisecting and find this change, check that your PCRE isn't\nolder than 8.32. This change intentionally broke really old versions\nof PCRE, but that's fixed in follow-up commits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n grep.c | 40 ++++++++++++++++++++++++++++++++++++----\n grep.h |  6 ++++++\n 2 files changed, 42 insertions(+), 4 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex d03d424e5c..ffe95995ee 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -365,9 +365,22 @@ static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \tif (!p->pcre1_regexp)\n \t\tcompile_regexp_failed(p, error);\n \n-\tp->pcre1_extra_info = pcre_study(p->pcre1_regexp, 0, &error);\n+\tp->pcre1_extra_info = pcre_study(p->pcre1_regexp, PCRE_STUDY_JIT_COMPILE, &error);\n \tif (!p->pcre1_extra_info && error)\n \t\tdie(\"%s\", error);\n+\n+#ifdef PCRE_CONFIG_JIT\n+\tpcre_config(PCRE_CONFIG_JIT, &p->pcre1_jit_on);\n+\tif (p->pcre1_jit_on == 1) {\n+\t\tp->pcre1_jit_stack = pcre_jit_stack_alloc(1, 1024 * 1024);\n+\t\tif (!p->pcre1_jit_stack)\n+\t\t\tdie(\"Couldn't allocate PCRE JIT stack\");\n+\t\tpcre_assign_jit_stack(p->pcre1_extra_info, NULL, p->pcre1_jit_stack);\n+\t} else if (p->pcre1_jit_on != 0) {\n+\t\tdie(\"BUG: The pcre1_jit_on variable should be 0 or 1, not %d\",\n+\t\t    p->pcre1_jit_on);\n+\t}\n+#endif\n }\n \n static int pcre1match(struct grep_pat *p, const char *line, const char *eol,\n@@ -378,8 +391,19 @@ static int pcre1match(struct grep_pat *p, const char *line, const char *eol,\n \tif (eflags & REG_NOTBOL)\n \t\tflags |= PCRE_NOTBOL;\n \n-\tret = pcre_exec(p->pcre1_regexp, p->pcre1_extra_info, line, eol - line,\n-\t\t\t0, flags, ovector, ARRAY_SIZE(ovector));\n+#ifdef PCRE_CONFIG_JIT\n+\tif (p->pcre1_jit_on) {\n+\t\tret = pcre_jit_exec(p->pcre1_regexp, p->pcre1_extra_info, line,\n+\t\t\t\t    eol - line, 0, flags, ovector,\n+\t\t\t\t    ARRAY_SIZE(ovector), p->pcre1_jit_stack);\n+\t} else\n+#endif\n+\t{\n+\t\tret = pcre_exec(p->pcre1_regexp, p->pcre1_extra_info, line,\n+\t\t\t\teol - line, 0, flags, ovector,\n+\t\t\t\tARRAY_SIZE(ovector));\n+\t}\n+\n \tif (ret < 0 && ret != PCRE_ERROR_NOMATCH)\n \t\tdie(\"pcre_exec failed with error code %d\", ret);\n \tif (ret > 0) {\n@@ -394,7 +418,15 @@ static int pcre1match(struct grep_pat *p, const char *line, const char *eol,\n static void free_pcre1_regexp(struct grep_pat *p)\n {\n \tpcre_free(p->pcre1_regexp);\n-\tpcre_free(p->pcre1_extra_info);\n+#ifdef PCRE_CONFIG_JIT\n+\tif (p->pcre1_jit_on) {\n+\t\tpcre_free_study(p->pcre1_extra_info);\n+\t\tpcre_jit_stack_free(p->pcre1_jit_stack);\n+\t} else\n+#endif\n+\t{\n+\t\tpcre_free(p->pcre1_extra_info);\n+\t}\n \tpcre_free((void *)p->pcre1_tables);\n }\n #else /* !USE_LIBPCRE1 */\ndiff --git a/grep.h b/grep.h\nindex 38ac82b638..14f47189f9 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -3,9 +3,13 @@\n #include \"color.h\"\n #ifdef USE_LIBPCRE1\n #include <pcre.h>\n+#ifndef PCRE_STUDY_JIT_COMPILE\n+#define PCRE_STUDY_JIT_COMPILE 0\n+#endif\n #else\n typedef int pcre;\n typedef int pcre_extra;\n+typedef int pcre_jit_stack;\n #endif\n #include \"kwset.h\"\n #include \"thread-utils.h\"\n@@ -48,7 +52,9 @@ struct grep_pat {\n \tregex_t regexp;\n \tpcre *pcre1_regexp;\n \tpcre_extra *pcre1_extra_info;\n+\tpcre_jit_stack *pcre1_jit_stack;\n \tconst unsigned char *pcre1_tables;\n+\tint pcre1_jit_on;\n \tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\n-- \n2.13.0.303.g4ebf302169\n\n"},{"id":"320769","messageId":"20170525200528.22037-7-avarab@gmail.com","threadId":"46046","inReplyTo":"20170525200528.22037-1-avarab@gmail.com","subject":"[PATCH v3 6/7] grep: un-break building with PCRE < 8.20","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-25T20:05:27Z","receivedAt":"2017-05-25T20:06:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend my change earlier in this series (\"grep: add support for the\nPCRE v1 JIT API\", 2017-04-11) to un-break the build on PCRE v1\nversions earlier than 8.20.\n\nThe 8.20 release was the first release to have JIT & pcre_jit_stack in\nthe headers, so a mock type needs to be provided for it on those\nreleases.\n\nNow git should compile with all PCRE versions that it supported before\nmy JIT change.\n\nI've tested it as far back as version 7.5 released on 2008-01-10, once\nI got down to version 7.0 it wouldn't build anymore with GCC 7.1.1,\nand I couldn't be bothered to anything older than 7.5 as I'm confident\nthat if the build breaks on those older versions it's not because of\nmy JIT change.\n\nSee the \"un-break\" change in this series (\"grep: un-break building\nwith PCRE < 8.32\", 2017-05-10) for why this isn't squashed into the\nmain PCRE JIT commit.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n grep.h | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/grep.h b/grep.h\nindex 3b948d9155..ce90969736 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -11,6 +11,9 @@\n #ifndef PCRE_STUDY_JIT_COMPILE\n #define PCRE_STUDY_JIT_COMPILE 0\n #endif\n+#if PCRE_MAJOR <= 8 && PCRE_MINOR < 20\n+typedef int pcre_jit_stack;\n+#endif\n #else\n typedef int pcre;\n typedef int pcre_extra;\n-- \n2.13.0.303.g4ebf302169\n\n"}]}