{"thread":{"id":"45937","subject":"[PATCH/RFC 0/6] Speed up git-grep by using PCRE v2 as a backend","startedAt":"2017-05-11T17:51:35Z","lastAt":"2017-05-11T17:52:22Z","messageCount":6,"participants":["Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"319426","messageId":"20170511175115.648-1-avarab@gmail.com","threadId":"45937","inReplyTo":null,"subject":"[PATCH/RFC 0/6] Speed up git-grep by using PCRE v2 as a backend","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-11T17:51:09Z","receivedAt":"2017-05-11T17:51:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Thought I'd send this to the list too. This is first of my WIP\npost-PCRE v2 inclusion series's.\n\nIn addition to the huge speed improvements for grep -P noted in the\nculmination of that series[1], this speeds up all other types of grep\ninvocations (fixed string & POSIX basic/extended) by using an\nexperimental PCRE API to translate those patterns to PCRE syntax.\n\nFixed string grep is sped up by ~15-50%, and any greps containing\nregexes by 40-70%, with 50% seeming to be the average for most normal\npatterns.\n\nIt isn't ready for the reasons noted in the last patch in the series,\nand currently brings most of PCRE into git in compat/pcre2 since it\nuses an experimental API.\n\nThe 5/6 patch is pretty much ready though and works on stock PCRE, it\nfixes TODO tests for patterns that contain a \\0, and enables regex\nmetacharacters in such patterns (right now they're all implicitly\nfixed strings), see the discussion in that patch for some of the\ncaveats.\n\nThat patch will most likely be dropped by the list, it can be\nretrieved from https://github.com/avar/git\navar/grep-and-pcre-and-more, or the whole series viewed at\nhttps://github.com/git/git/compare/master...avar:avar/grep-and-pcre-and-more.\n\n1. <20170511170142.15934-8-avarab@gmail.com>\n   (https://public-inbox.org/git/20170511170142.15934-8-avarab@gmail.com/)\n\nÆvar Arnfjörð Bjarmason (6):\n  Makefile & compat/pcre2: add ability to build an embedded PCRE\n  Makefile & compat/pcre2: add dependency on pcre2_convert.c\n  compat/pcre2: import pcre2 from svn trunk\n  test-lib: add LIBPCRE1 & LIBPCRE2 prerequisites\n  grep: support regex patterns containing \\0 via PCRE v2\n  grep: use PCRE v2 under the hood for -G & -E for amazing speedup\n\n Makefile                                           |    53 +\n compat/pcre2/get-pcre2.sh                          |    68 +\n compat/pcre2/src/pcre2.h                           |   832 ++\n compat/pcre2/src/pcre2_auto_possess.c              |  1291 ++\n compat/pcre2/src/pcre2_chartables.c                |     1 +\n compat/pcre2/src/pcre2_chartables.c.dist           |   198 +\n compat/pcre2/src/pcre2_compile.c                   |  9626 +++++++++++++++\n compat/pcre2/src/pcre2_config.c                    |   222 +\n compat/pcre2/src/pcre2_context.c                   |   450 +\n compat/pcre2/src/pcre2_convert.c                   |   724 ++\n compat/pcre2/src/pcre2_error.c                     |   327 +\n compat/pcre2/src/pcre2_find_bracket.c              |   218 +\n compat/pcre2/src/pcre2_internal.h                  |  1967 +++\n compat/pcre2/src/pcre2_intmodedep.h                |   884 ++\n compat/pcre2/src/pcre2_jit_compile.c               | 12307 +++++++++++++++++++\n compat/pcre2/src/pcre2_jit_match.c                 |   189 +\n compat/pcre2/src/pcre2_jit_misc.c                  |   227 +\n compat/pcre2/src/pcre2_maketables.c                |   157 +\n compat/pcre2/src/pcre2_match.c                     |  6826 ++++++++++\n compat/pcre2/src/pcre2_match_data.c                |   147 +\n compat/pcre2/src/pcre2_newline.c                   |   243 +\n compat/pcre2/src/pcre2_ord2utf.c                   |   120 +\n compat/pcre2/src/pcre2_string_utils.c              |   201 +\n compat/pcre2/src/pcre2_study.c                     |  1624 +++\n compat/pcre2/src/pcre2_tables.c                    |   765 ++\n compat/pcre2/src/pcre2_ucd.c                       |  3761 ++++++\n compat/pcre2/src/pcre2_ucp.h                       |   268 +\n compat/pcre2/src/pcre2_valid_utf.c                 |   398 +\n compat/pcre2/src/pcre2_xclass.c                    |   271 +\n compat/pcre2/src/sljit/sljitConfig.h               |   145 +\n compat/pcre2/src/sljit/sljitConfigInternal.h       |   725 ++\n compat/pcre2/src/sljit/sljitExecAllocator.c        |   312 +\n compat/pcre2/src/sljit/sljitLir.c                  |  2224 ++++\n compat/pcre2/src/sljit/sljitLir.h                  |  1392 +++\n compat/pcre2/src/sljit/sljitNativeARM_32.c         |  2326 ++++\n compat/pcre2/src/sljit/sljitNativeARM_64.c         |  2104 ++++\n compat/pcre2/src/sljit/sljitNativeARM_T2_32.c      |  1987 +++\n compat/pcre2/src/sljit/sljitNativeMIPS_32.c        |   437 +\n compat/pcre2/src/sljit/sljitNativeMIPS_64.c        |   539 +\n compat/pcre2/src/sljit/sljitNativeMIPS_common.c    |  2110 ++++\n compat/pcre2/src/sljit/sljitNativePPC_32.c         |   276 +\n compat/pcre2/src/sljit/sljitNativePPC_64.c         |   447 +\n compat/pcre2/src/sljit/sljitNativePPC_common.c     |  2421 ++++\n compat/pcre2/src/sljit/sljitNativeSPARC_32.c       |   165 +\n compat/pcre2/src/sljit/sljitNativeSPARC_common.c   |  1471 +++\n compat/pcre2/src/sljit/sljitNativeTILEGX-encoder.c | 10159 +++++++++++++++\n compat/pcre2/src/sljit/sljitNativeTILEGX_64.c      |  2555 ++++\n compat/pcre2/src/sljit/sljitNativeX86_32.c         |   602 +\n compat/pcre2/src/sljit/sljitNativeX86_64.c         |   742 ++\n compat/pcre2/src/sljit/sljitNativeX86_common.c     |  2921 +++++\n compat/pcre2/src/sljit/sljitProtExecAllocator.c    |   421 +\n compat/pcre2/src/sljit/sljitUtils.c                |   334 +\n grep.c                                             |    73 +-\n grep.h                                             |     5 +\n t/README                                           |    18 +\n t/t7008-grep-binary.sh                             |    87 +-\n t/test-lib.sh                                      |     3 +\n 57 files changed, 81335 insertions(+), 31 deletions(-)\n create mode 100755 compat/pcre2/get-pcre2.sh\n create mode 100644 compat/pcre2/src/pcre2.h\n create mode 100644 compat/pcre2/src/pcre2_auto_possess.c\n create mode 120000 compat/pcre2/src/pcre2_chartables.c\n create mode 100644 compat/pcre2/src/pcre2_chartables.c.dist\n create mode 100644 compat/pcre2/src/pcre2_compile.c\n create mode 100644 compat/pcre2/src/pcre2_config.c\n create mode 100644 compat/pcre2/src/pcre2_context.c\n create mode 100644 compat/pcre2/src/pcre2_convert.c\n create mode 100644 compat/pcre2/src/pcre2_error.c\n create mode 100644 compat/pcre2/src/pcre2_find_bracket.c\n create mode 100644 compat/pcre2/src/pcre2_internal.h\n create mode 100644 compat/pcre2/src/pcre2_intmodedep.h\n create mode 100644 compat/pcre2/src/pcre2_jit_compile.c\n create mode 100644 compat/pcre2/src/pcre2_jit_match.c\n create mode 100644 compat/pcre2/src/pcre2_jit_misc.c\n create mode 100644 compat/pcre2/src/pcre2_maketables.c\n create mode 100644 compat/pcre2/src/pcre2_match.c\n create mode 100644 compat/pcre2/src/pcre2_match_data.c\n create mode 100644 compat/pcre2/src/pcre2_newline.c\n create mode 100644 compat/pcre2/src/pcre2_ord2utf.c\n create mode 100644 compat/pcre2/src/pcre2_string_utils.c\n create mode 100644 compat/pcre2/src/pcre2_study.c\n create mode 100644 compat/pcre2/src/pcre2_tables.c\n create mode 100644 compat/pcre2/src/pcre2_ucd.c\n create mode 100644 compat/pcre2/src/pcre2_ucp.h\n create mode 100644 compat/pcre2/src/pcre2_valid_utf.c\n create mode 100644 compat/pcre2/src/pcre2_xclass.c\n create mode 100644 compat/pcre2/src/sljit/sljitConfig.h\n create mode 100644 compat/pcre2/src/sljit/sljitConfigInternal.h\n create mode 100644 compat/pcre2/src/sljit/sljitExecAllocator.c\n create mode 100644 compat/pcre2/src/sljit/sljitLir.c\n create mode 100644 compat/pcre2/src/sljit/sljitLir.h\n create mode 100644 compat/pcre2/src/sljit/sljitNativeARM_32.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeARM_64.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeARM_T2_32.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeMIPS_32.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeMIPS_64.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeMIPS_common.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativePPC_32.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativePPC_64.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativePPC_common.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeSPARC_32.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeSPARC_common.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeTILEGX-encoder.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeTILEGX_64.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeX86_32.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeX86_64.c\n create mode 100644 compat/pcre2/src/sljit/sljitNativeX86_common.c\n create mode 100644 compat/pcre2/src/sljit/sljitProtExecAllocator.c\n create mode 100644 compat/pcre2/src/sljit/sljitUtils.c\n\n-- \n2.11.0\n\n"},{"id":"319427","messageId":"20170511175115.648-2-avarab@gmail.com","threadId":"45937","inReplyTo":"20170511175115.648-1-avarab@gmail.com","subject":"[PATCH/RFC 1/6] Makefile & compat/pcre2: add ability to build an embedded PCRE","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-11T17:51:10Z","receivedAt":"2017-05-11T17:51:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a USE_LIBPCRE2_BUNDLED=YesIHaveNoPackagedVersion flag to the\nMakefile which'll use the PCRE v2 shipped in compat/pcre2 instead of\ntrying to find it via -lpcre2-8 on the installed system.\n\nAs covered in a previous commits (\"grep: add support for PCRE v2\",\n2017-04-08) there are major benefits to using a bleeding edge PCRE v2,\nbut more importantly I'd like to experiment with making PCRE a\nmandatory dependency to power various internal features of grep/log\nwithout the user being aware that they're using the library under the\nhood, similar to how we use kwset now for fixed-string searches.\n\nImposing that hard dependency on everyone using git would bother a lot\nof people, whereas if git itself ships PCRE it's no more bothersome\nthan the code using kwset, i.e. it can be invisible to the builder &\nuser, and allow git to target newer PCRE APIs without worrying about\nversioning.\n\nSee [1] for a mostly one-sided pcre-dev mailing list thread discussing\nhow embed the library.\n\nImplementation details:\n\n * I configured PCRE v2 with ./configure --enable-jit --enable-utf\n\n * It sets a lot of -DHAVE_* but these are used by the subset of the\n   files I copied, many are either unused or only used by pcre2test.c\n   which isn't brought in by the script.\n\n * These -DHAVE_* flags are something we have already by default &\n   assume in other git.git code, so it should be fine to define it.\n\n * -DPCRE2_CODE_UNIT_WIDTH=8 only compiles the functions linking to\n   -lpcre2-8 would have gotten us.\n\n * All the limits / sizes are the PCRE defaults, the\n   MATCH_LIMIT_RECURSION define is a synonym for MATCH_LIMIT_DEPTH in\n   older versions, it allows building against older (currently\n   release) versions of the library.\n\n * -DNEWLINE_DEFAULT=2 means only \\n is recognized as a newline. This\n    corresponds to the --enable-newline-is-lf option. It's also\n    possible to set this to CR, CRLF, any of CR, LF, or CRLF, or any\n    Unicode newline character being recognized as \\n.\n\n    This *might* have to be customized on Windows, but I think the\n    grep machinery always splits on newlines for us already, so this\n    probably works on Windows as-is, but needs testing.\n\n1. https://lists.exim.org/lurker/thread/20170507.223619.fbee8f00.en.html\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n Makefile                  | 52 ++++++++++++++++++++++++++++++++++++\n compat/pcre2/get-pcre2.sh | 67 +++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 119 insertions(+)\n create mode 100755 compat/pcre2/get-pcre2.sh\n\ndiff --git a/Makefile b/Makefile\nindex d77ca4c1a5..b18867196e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -34,6 +34,11 @@ all::\n # library. The USE_LIBPCRE flag will likely be changed to mean v2 by\n # default in future releases.\n #\n+# Define USE_LIBPCRE2_BUNDLED=YesIHaveNoPackagedVersion in addition to\n+# USE_LIBPCRE2=YesPlease if you'd like to use a copy of PCRE version 2\n+# bunded with Git. This is for setups where getting a hold of a\n+# packaged PCRE is inconvenient.\n+#\n # Define LIBPCREDIR=/foo/bar if your PCRE header and library files are in\n # /foo/bar/include and /foo/bar/lib directories.\n #\n@@ -1105,8 +1110,10 @@ endif\n \n ifdef USE_LIBPCRE2\n \tBASIC_CFLAGS += -DUSE_LIBPCRE2\n+ifndef USE_LIBPCRE2_BUNDLED\n \tEXTLIBS += -lpcre2-8\n endif\n+endif\n \n ifdef LIBPCREDIR\n \tBASIC_CFLAGS += -I$(LIBPCREDIR)/include\n@@ -1505,6 +1512,50 @@ ifdef NO_REGEX\n \tCOMPAT_CFLAGS += -Icompat/regex\n \tCOMPAT_OBJS += compat/regex/regex.o\n endif\n+ifdef USE_LIBPCRE2_BUNDLED\n+ifndef USE_LIBPCRE2\n+$(error please set USE_LIBPCRE2=YesPlease when setting \\\n+USE_LIBPCRE2_BUNDLED=$(USE_LIBPCRE2_BUNDLED))\n+endif\n+\tCOMPAT_CFLAGS += \\\n+\t\t-Icompat/pcre2/src \\\n+\t\t-DHAVE_BCOPY=1 \\\n+\t\t-DHAVE_INTTYPES_H=1 \\\n+\t\t-DHAVE_MEMMOVE=1 \\\n+\t\t-DHAVE_STDINT_H=1 \\\n+\t\t-DPCRE2_CODE_UNIT_WIDTH=8 \\\n+\t\t-DLINK_SIZE=2 \\\n+\t\t-DHEAP_LIMIT=20000000 \\\n+\t\t-DMATCH_LIMIT=10000000 \\\n+\t\t-DMATCH_LIMIT_DEPTH=10000000 \\\n+\t\t-DMATCH_LIMIT_RECURSION=10000000 \\\n+\t\t-DMAX_NAME_COUNT=10000 \\\n+\t\t-DMAX_NAME_SIZE=32 \\\n+\t\t-DPARENS_NEST_LIMIT=250 \\\n+\t\t-DNEWLINE_DEFAULT=2 \\\n+\t\t-DSUPPORT_JIT \\\n+\t\t-DSUPPORT_UNICODE\n+\tCOMPAT_OBJS += \\\n+\t\tcompat/pcre2/src/pcre2_auto_possess.o \\\n+\t\tcompat/pcre2/src/pcre2_chartables.o \\\n+\t\tcompat/pcre2/src/pcre2_compile.o \\\n+\t\tcompat/pcre2/src/pcre2_config.o \\\n+\t\tcompat/pcre2/src/pcre2_context.o \\\n+\t\tcompat/pcre2/src/pcre2_error.o \\\n+\t\tcompat/pcre2/src/pcre2_find_bracket.o \\\n+\t\tcompat/pcre2/src/pcre2_jit_compile.o \\\n+\t\tcompat/pcre2/src/pcre2_maketables.o \\\n+\t\tcompat/pcre2/src/pcre2_match.o \\\n+\t\tcompat/pcre2/src/pcre2_match_data.o \\\n+\t\tcompat/pcre2/src/pcre2_newline.o \\\n+\t\tcompat/pcre2/src/pcre2_ord2utf.o \\\n+\t\tcompat/pcre2/src/pcre2_string_utils.o \\\n+\t\tcompat/pcre2/src/pcre2_study.o \\\n+\t\tcompat/pcre2/src/pcre2_tables.o \\\n+\t\tcompat/pcre2/src/pcre2_ucd.o \\\n+\t\tcompat/pcre2/src/pcre2_valid_utf.o \\\n+\t\tcompat/pcre2/src/pcre2_xclass.o\n+endif\n ifdef NATIVE_CRLF\n \tBASIC_CFLAGS += -DNATIVE_CRLF\n endif\n@@ -2259,6 +2310,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_EXPAT=\\''$(subst ','\\'',$(subst ','\\'',$(NO_EXPAT)))'\\' >>$@+\n \t@echo USE_LIBPCRE1=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE)))'\\' >>$@+\n \t@echo USE_LIBPCRE2=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE2)))'\\' >>$@+\n+\t@echo USE_LIBPCRE2_BUNDLED=\\''$(subst ','\\'',$(subst ','\\'',$(USE_LIBPCRE2_BUNDLED)))'\\' >>$@+\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PTHREADS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PTHREADS)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\ndiff --git a/compat/pcre2/get-pcre2.sh b/compat/pcre2/get-pcre2.sh\nnew file mode 100755\nindex 0000000000..f1796cb518\n--- /dev/null\n+++ b/compat/pcre2/get-pcre2.sh\n@@ -0,0 +1,67 @@\n+#!/bin/sh -e\n+\n+# Usage:\n+# ./get-pcre2.sh '' 'trunk'\n+# ./get-pcre2.sh '' 'tags/pcre2-10.23'\n+# ./get-pcre2.sh ~/g/pcre2 ''\n+\n+srcdir=$1\n+version=$2\n+if test -z \"$version\"\n+then\n+\tversion=\"tags/pcre2-10.23\"\n+fi\n+\n+echo Getting PCRE v2 version $version\n+rm -rfv src\n+mkdir src src/sljit\n+\n+for srcfile in \\\n+\tpcre2.h \\\n+\tpcre2_internal.h \\\n+\tpcre2_intmodedep.h \\\n+\tpcre2_ucp.h \\\n+\tpcre2_auto_possess.c \\\n+\tpcre2_chartables.c.dist \\\n+\tpcre2_compile.c \\\n+\tpcre2_config.c \\\n+\tpcre2_context.c \\\n+\tpcre2_error.c \\\n+\tpcre2_find_bracket.c \\\n+\tpcre2_jit_compile.c \\\n+\tpcre2_jit_match.c \\\n+\tpcre2_jit_misc.c \\\n+\tpcre2_maketables.c \\\n+\tpcre2_match.c \\\n+\tpcre2_match_data.c \\\n+\tpcre2_newline.c \\\n+\tpcre2_ord2utf.c \\\n+\tpcre2_string_utils.c \\\n+\tpcre2_study.c \\\n+\tpcre2_tables.c \\\n+\tpcre2_ucd.c \\\n+\tpcre2_valid_utf.c \\\n+\tpcre2_xclass.c\n+do\n+\tif test -z \"$srcdir\"\n+\tthen\n+\t\tsvn cat svn://vcs.exim.org/pcre2/code/$version/src/$srcfile >src/$srcfile\n+\telse\n+\t\tcp \"$srcdir/src/$srcfile\" src/$srcfile\n+\tfi\n+\twc -l src/$srcfile\n+done\n+\n+(cd src && ln -sf pcre2_chartables.c.dist pcre2_chartables.c)\n+\n+if test -z \"$srcdir\"\n+then\n+\tfor srcfile in $(svn ls svn://vcs.exim.org/pcre2/code/tags/pcre2-10.23/src/sljit)\n+\tdo\n+\t\tsvn cat svn://vcs.exim.org/pcre2/code/$version/src/sljit/$srcfile >src/sljit/$srcfile\n+\t\twc -l src/sljit/$srcfile\n+\tdone\n+else\n+\tcp -R \"$srcdir/src/sljit\" src/\n+\twc -l src/sljit/*\n+fi\n-- \n2.11.0\n\n"},{"id":"319428","messageId":"20170511175115.648-3-avarab@gmail.com","threadId":"45937","inReplyTo":"20170511175115.648-1-avarab@gmail.com","subject":"[PATCH/RFC 2/6] Makefile & compat/pcre2: add dependency on pcre2_convert.c","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-11T17:51:11Z","receivedAt":"2017-05-11T17:51:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a dependency on the experimental pcre2_convert.c. This only exists\nin svn trunk of pcre2 currently, and allows for converting POSIX\nbasic/extended & glob patterns to patterns accepted by PCRE[1][2].\n\n1. https://bugs.exim.org/show_bug.cgi?id=2106\n2. https://bugs.exim.org/show_bug.cgi?id=2107\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n Makefile                  | 1 +\n compat/pcre2/get-pcre2.sh | 1 +\n 2 files changed, 2 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex b18867196e..e437fa011c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1541,6 +1541,7 @@ endif\n \t\tcompat/pcre2/src/pcre2_compile.o \\\n \t\tcompat/pcre2/src/pcre2_config.o \\\n \t\tcompat/pcre2/src/pcre2_context.o \\\n+\t\tcompat/pcre2/src/pcre2_convert.o \\\n \t\tcompat/pcre2/src/pcre2_error.o \\\n \t\tcompat/pcre2/src/pcre2_find_bracket.o \\\n \t\tcompat/pcre2/src/pcre2_jit_compile.o \\\ndiff --git a/compat/pcre2/get-pcre2.sh b/compat/pcre2/get-pcre2.sh\nindex f1796cb518..7679fba8e4 100755\n--- a/compat/pcre2/get-pcre2.sh\n+++ b/compat/pcre2/get-pcre2.sh\n@@ -26,6 +26,7 @@ for srcfile in \\\n \tpcre2_compile.c \\\n \tpcre2_config.c \\\n \tpcre2_context.c \\\n+\tpcre2_convert.c \\\n \tpcre2_error.c \\\n \tpcre2_find_bracket.c \\\n \tpcre2_jit_compile.c \\\n-- \n2.11.0\n\n"},{"id":"319430","messageId":"20170511175115.648-5-avarab@gmail.com","threadId":"45937","inReplyTo":"20170511175115.648-1-avarab@gmail.com","subject":"[PATCH/RFC 4/6] test-lib: add LIBPCRE1 & LIBPCRE2 prerequisites","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-11T17:51:13Z","receivedAt":"2017-05-11T17:51:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add LIBPCRE1 and LIBPCRE2 prerequisites which are true when git is\ncompiled with USE_LIBPCRE1=YesPlease or USE_LIBPCRE2=YesPlease,\nrespectively.\n\nThere are various edge cases or version-specific features that need to\nbe tested for.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/README      | 12 ++++++++++++\n t/test-lib.sh |  2 ++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/t/README b/t/README\nindex 2f95860369..1ff612ca65 100644\n--- a/t/README\n+++ b/t/README\n@@ -808,6 +808,18 @@ use these, and \"test_set_prereq\" for how to define your own.\n    Git was compiled with support for PCRE. Wrap any tests\n    that use git-grep --perl-regexp or git-grep -P in these.\n \n+ - LIBPCRE1\n+\n+   Git was compiled with PCRE v1 support via\n+   USE_LIBPCRE1=YesPlease. Wrap any PCRE using tests that for some\n+   reason need v1 of the PCRE library instead of v2 in these.\n+\n+ - LIBPCRE2\n+\n+   Git was compiled with PCRE v2 support via\n+   USE_LIBPCRE2=YesPlease. Wrap any PCRE using tests that for some\n+   reason need v2 of the PCRE library instead of v1 in these.\n+\n  - CASE_INSENSITIVE_FS\n \n    Test is run on a case insensitive file system.\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 44d4679384..13ed81dc16 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1012,6 +1012,8 @@ test -z \"$NO_PERL\" && test_set_prereq PERL\n test -z \"$NO_PTHREADS\" && test_set_prereq PTHREADS\n test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE1$USE_LIBPCRE2\" && test_set_prereq PCRE\n+test -n \"$USE_LIBPCRE1\" && test_set_prereq LIBPCRE1\n+test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n \n # Can we rely on git's output in the C locale?\n-- \n2.11.0\n\n"},{"id":"319431","messageId":"20170511175115.648-7-avarab@gmail.com","threadId":"45937","inReplyTo":"20170511175115.648-1-avarab@gmail.com","subject":"[PATCH/RFC 6/6] grep: use PCRE v2 under the hood for -G & -E for amazing speedup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-11T17:51:15Z","receivedAt":"2017-05-11T17:52:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the underlying engine powering POSIX basic & extended patterns\nto be PCRE v2 under the hood.\n\nThis relies on an experimental SVN-trunk only PCRE v2 API which Philip\nHazel (the PCRE maintainer) wrote up in response to a feature request\nI filed1[1].\n\nThis allows us to use pcre2_pattern_convert() to power all grep regex\nmatches by converting the POSIX patterns into PCRE syntax before\ncompiling them.\n\nDue to PCRE generally being faster than POSIX, but most importantly\ndue to its JIT feature (where available) this speeds up grep by\na *lot*.\n\nThe improvements to the \"perl\" tests are already a part of this\nseries, but all the other benchmarks show improvements made by this\nchange alone:\n\n    $ GIT_PERF_REPEAT_COUNT=30 GIT_PERF_LARGE_REPO=~/g/linux [see note #2 for the GIT_PERF_MAKE_COMMAND ...]\n    [...]\n    Test                                           v2.13.0             HEAD\n    -----------------------------------------------------------------------------------------\n    7810.1: grep worktree, cheap regex             0.19(0.39+0.52)     0.19(0.26+0.56) +0.0%\n    7810.2: grep worktree, expensive regex         5.11(29.75+0.33)    1.07(5.36+0.34) -79.1%\n    7810.3: grep --cached, cheap regex             2.91(2.77+0.12)     2.85(2.78+0.06) -2.1%\n    7810.4: grep --cached, expensive regex         21.00(20.89+0.08)   6.25(6.18+0.06) -70.2%\n    7820.1: basic grep how.to                      0.32(1.20+0.43)     0.19(0.26+0.56) -40.6%\n    7820.2: extended grep how.to                   0.32(1.12+0.51)     0.19(0.22+0.60) -40.6%\n    7820.3: perl grep how.to                       0.31(1.11+0.45)     0.19(0.30+0.54) -38.7%\n    7820.5: basic grep ^how to                     0.31(1.09+0.51)     0.19(0.24+0.57) -38.7%\n    7820.6: extended grep ^how to                  0.31(1.14+0.46)     0.19(0.29+0.52) -38.7%\n    7820.7: perl grep ^how to                      0.57(2.63+0.38)     0.19(0.25+0.56) -66.7%\n    7820.9: basic grep [how] to                    0.49(2.19+0.36)     0.22(0.36+0.54) -55.1%\n    7820.10: extended grep [how] to                0.49(2.16+0.41)     0.22(0.41+0.50) -55.1%\n    7820.11: perl grep [how] to                    0.57(2.55+0.40)     0.22(0.35+0.55) -61.4%\n    7820.13: basic grep \\(e.t[^ ]*\\|v.ry\\) rare    0.65(3.18+0.38)     0.22(0.44+0.52) -66.2%\n    7820.14: extended grep (e.t[^ ]*|v.ry) rare    0.65(3.17+0.40)     0.21(0.47+0.52) -67.7%\n    7820.15: perl grep (e.t[^ ]*|v.ry) rare        1.05(5.64+0.34)     0.22(0.46+0.53) -79.0%\n    7820.17: basic grep m\\(ú\\|u\\)ult.b\\(æ\\|y\\)te   0.33(1.33+0.38)     0.19(0.31+0.51) -42.4%\n    7820.18: extended grep m(ú|u)ult.b(æ|y)te      0.33(1.27+0.44)     0.19(0.32+0.50) -42.4%\n    7820.19: perl grep m(ú|u)ult.b(æ|y)te          0.37(1.58+0.40)     0.19(0.30+0.53) -48.6%\n    7821.1: fixed grep int                         0.53(1.70+0.60)     0.43(1.13+0.66) -18.9%\n    7821.2: basic grep int                         0.55(1.62+0.59)     0.46(1.08+0.64) -16.4%\n    7821.3: extended grep int                      0.54(1.65+0.59)     0.45(1.17+0.56) -16.7%\n    7821.4: perl grep int                          0.54(1.63+0.62)     0.46(1.12+0.60) -14.8%\n    7821.6: fixed grep -i int                      0.58(1.93+0.54)     0.48(1.40+0.52) -17.2%\n    7821.7: basic grep -i int                      0.83(1.91+0.60)     0.57(1.23+0.67) -31.3%\n    7821.8: extended grep -i int                   0.59(1.80+0.66)     0.48(1.33+0.59) -18.6%\n    7821.9: perl grep -i int                       0.61(1.91+0.56)     0.52(1.28+0.63) -14.8%\n    7821.11: fixed grep æ                          0.34(1.25+0.45)     0.19(0.29+0.51) -44.1%\n    7821.12: basic grep æ                          0.34(1.26+0.43)     0.19(0.28+0.53) -44.1%\n    7821.13: extended grep æ                       0.34(1.22+0.48)     0.19(0.29+0.53) -44.1%\n    7821.14: perl grep æ                           0.34(1.30+0.41)     0.19(0.26+0.57) -44.1%\n    7821.16: fixed grep -i æ                       0.27(0.88+0.46)     0.19(0.30+0.51) -29.6%\n    7821.17: basic grep -i æ                       0.27(0.88+0.44)     0.19(0.27+0.54) -29.6%\n    7821.18: extended grep -i æ                    0.27(0.90+0.42)     0.19(0.22+0.59) -29.6%\n    7821.19: perl grep -i æ                        0.25(0.74+0.51)     0.18(0.27+0.58) -28.0%\n    7821.1: fixed grep int                         0.53(1.70+0.60)     0.43(1.13+0.66) -18.9%\n    7821.2: basic grep int                         0.55(1.62+0.59)     0.46(1.08+0.64) -16.4%\n    7821.3: extended grep int                      0.54(1.65+0.59)     0.45(1.17+0.56) -16.7%\n    7821.4: perl grep int                          0.54(1.63+0.62)     0.46(1.12+0.60) -14.8%\n    7821.6: fixed grep -i int                      0.58(1.93+0.54)     0.48(1.40+0.52) -17.2%\n    7821.7: basic grep -i int                      0.83(1.91+0.60)     0.57(1.23+0.67) -31.3%\n    7821.8: extended grep -i int                   0.59(1.80+0.66)     0.48(1.33+0.59) -18.6%\n    7821.9: perl grep -i int                       0.61(1.91+0.56)     0.52(1.28+0.63) -14.8%\n    7821.11: fixed grep æ                          0.34(1.25+0.45)     0.19(0.29+0.51) -44.1%\n    7821.12: basic grep æ                          0.34(1.26+0.43)     0.19(0.28+0.53) -44.1%\n    7821.13: extended grep æ                       0.34(1.22+0.48)     0.19(0.29+0.53) -44.1%\n    7821.14: perl grep æ                           0.34(1.30+0.41)     0.19(0.26+0.57) -44.1%\n    7821.16: fixed grep -i æ                       0.27(0.88+0.46)     0.19(0.30+0.51) -29.6%\n    7821.17: basic grep -i æ                       0.27(0.88+0.44)     0.19(0.27+0.54) -29.6%\n    7821.18: extended grep -i æ                    0.27(0.90+0.42)     0.19(0.22+0.59) -29.6%\n    7821.19: perl grep -i æ                        0.25(0.74+0.51)     0.18(0.27+0.58) -28.0%\n\nCaveats & other things to mention:\n\n * This will expose PCRE v2 (as opposed to C library reg(comp|exec))to\n   the network via gitweb in its default configuration. See\n   <CACBZZX6V8qbnrZAdhRvPthy5Z91iEG8rrJ=Sf9tdkOt52M9j1Q@mail.gmail.com>\n   for a discussion of security & other caveats related to that.\n\n * I'm checking for PCRE2_CONVERT_POSIX_BASIC to enable this, but the\n   experimental API of pcre2_pattern_convert() may change before it\n   makes it into a release.\n\n   If we think this patch is awesome enough to get into a git release\n   regardless, it should be guarded by some other method so we don't\n   rudely tie upstream PCRE to this API least they break git versions\n   in the wild.\n\n * One way to do to that would be to guard this via the\n   USE_LIBPCRE2_BUNDLED flag, but see the above E-Mail thread for\n   concerns about shipping an embedded PCRE, and for ways that could\n   be made OK.\n\n * We could ship some copy of just the logic in\n   pcre2_pattern_convert() & use the system PCRE instead. I haven't\n   tried splitting it off from the PCRE codebase, and don't know how\n   hard that would be.\n\n * There are outstanding bugs in the pcre2_pattern_convert()\n   function. Grepping with -G and -E for all ASCII characters from\n   1..127 both \"$char\" and \"\\\\$char\" will produce numerous\n   differences. These are mostly obscure cases, I'm working out fixes\n   to those with Philip.\n\n1. https://bugs.exim.org/show_bug.cgi?id=2106\n\n2. GIT_PERF_MAKE_COMMAND='grep -q LIBPCRE2 Makefile && make -j8 USE_LIBPCRE2=YesPlease USE_LIBPCRE2_BUNDLED=Y CC=~/perl5/installed/bin/gcc NO_R_TO_GCC_LINKER=YesPlease CFLAGS=-O3 || make -j8 USE_LIBPCRE=YesPlease CC=~/perl5/installed/bin/gcc NO_R_TO_GCC_LINKER=YesPlease CFLAGS=-O3 LIBPCREDIR=/home/avar/g/pcre/inst LDFLAGS=-Wl,-rpath,/home/avar/g/pcre/inst/lib' ./run v2.13.0 HEAD -- p*grep*\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n grep.c                 | 97 +++++++++++++++++++++++++++++++++++---------------\n grep.h                 |  5 +++\n t/README               |  6 ++++\n t/t7008-grep-binary.sh | 13 +++++--\n t/test-lib.sh          |  1 +\n 5 files changed, 90 insertions(+), 32 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 5db614cf80..0f6ee709c5 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -472,8 +472,48 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \tconst uint8_t *character_tables = NULL;\n \tuint32_t canjit;\n \tint jitret;\n+\tint icase = opt->regflags & REG_ICASE || p->ignore_case;\n+\tPCRE2_SPTR pattern = (PCRE2_SPTR)p->pattern;\n+\tPCRE2_SIZE length = p->patternlen;\n+\tint copied_pattern = 0;\n+\tstruct strbuf pattern_sb = STRBUF_INIT;\n+#ifdef PCRE2_CONVERT_POSIX_BASIC\n+\tint convret;\n+\tPCRE2_UCHAR *convpatbuf = NULL;\n+\tPCRE2_SIZE convpatlen;\n+\tint converted_pattern = 0;\n+#endif\n \n-\tassert(opt->pcre2);\n+\tif (opt->fixed || has_null(p->pattern, p->patternlen) || is_fixed(p->pattern, p->patternlen)) {\n+\t\tif (icase)\n+\t\t\tstrbuf_add(&pattern_sb, \"(?i)\", 4);\n+\t\tif (opt->fixed)\n+\t\t\tstrbuf_add(&pattern_sb, \"\\\\Q\", 2);\n+\t\tstrbuf_add(&pattern_sb, p->pattern, p->patternlen);\n+\t\tif (opt->fixed)\n+\t\t\tstrbuf_add(&pattern_sb, \"\\\\E\", 2);\n+\n+\t\tpattern = (PCRE2_SPTR)pattern_sb.buf;\n+\t\tlength = pattern_sb.len;\n+\t\tcopied_pattern = 1;\n+\t} else if (opt->pcre2_posix_emulation) {\n+#ifdef PCRE2_CONVERT_POSIX_BASIC\n+\t\tconvret = pcre2_pattern_convert(pattern, length,\n+\t\t\t\t\t       (opt->regflags & REG_EXTENDED\n+\t\t\t\t\t\t? PCRE2_CONVERT_POSIX_EXTENDED\n+\t\t\t\t\t\t: PCRE2_CONVERT_POSIX_BASIC),\n+\t\t\t\t\t       &convpatbuf, &convpatlen, NULL);\n+\t\tif (convret != 0) {\n+\t\t\tfprintf(stderr, \"oh noes\\n\");\n+\t\t\tpcre2_get_error_message(convret, errbuf, sizeof(errbuf));\n+\t\t\tcompile_regexp_failed(p, (const char *)&errbuf);\n+\t\t}\n+\t\tpattern = convpatbuf;\n+\t\tlength = convpatlen;\n+\t\tconverted_pattern = 1;\n+#endif\n+\t} else\n+\t\tassert(opt->pcre2);\n \n \tp->pcre2_compile_context = NULL;\n \n@@ -488,11 +528,16 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \tif (is_utf8_locale() && has_non_ascii(p->pattern))\n \t\toptions |= PCRE2_UTF;\n \n-\tp->pcre2_pattern = pcre2_compile((PCRE2_SPTR)p->pattern,\n-\t\t\t\t\t p->patternlen, options, &error, &erroffset,\n-\t\t\t\t\t p->pcre2_compile_context);\n+\tp->pcre2_pattern = pcre2_compile(pattern, length, options, &error,\n+\t\t\t\t\t &erroffset, p->pcre2_compile_context);\n \n \tif (p->pcre2_pattern) {\n+\t\tif (copied_pattern)\n+\t\t\tstrbuf_release(&pattern_sb);\n+#ifdef PCRE2_CONVERT_POSIX_BASIC\n+\t\tif (converted_pattern)\n+\t\t\tpcre2_converted_pattern_free(convpatbuf);\n+#endif\n \t\tp->pcre2_match_data = pcre2_match_data_create_from_pattern(p->pcre2_pattern, NULL);\n \t\tif (!p->pcre2_match_data)\n \t\t\tdie(\"BUG: Couldn't allocate PCRE2 match data\");\n@@ -580,7 +625,6 @@ static int pcre2match(struct grep_pat *p, const char *line, const char *eol,\n static void free_pcre2_pattern(struct grep_pat *p)\n {\n }\n-#endif /* !USE_LIBPCRE2 */\n \n static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n@@ -602,41 +646,21 @@ static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tcompile_regexp_failed(p, errbuf);\n \t}\n }\n+#endif /* !USE_LIBPCRE2 */\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n+#ifndef USE_LIBPCRE2\n \tint icase, ascii_only;\n+#endif\n \tint err;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n+#ifndef USE_LIBPCRE2\n \ticase\t       = opt->regflags & REG_ICASE || p->ignore_case;\n \tascii_only     = !has_non_ascii(p->pattern);\n \n-#ifdef USE_LIBPCRE2\n-\tif (has_null(p->pattern, p->patternlen)) {\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\tif (icase)\n-\t\t\tstrbuf_add(&sb, \"(?i)\", 4);\n-\t\tif (opt->fixed)\n-\t\t\tstrbuf_add(&sb, \"\\\\Q\", 2);\t\t\n-\t\tstrbuf_add(&sb, p->pattern, p->patternlen);\n-\t\tif (opt->fixed)\n-\t\t\tstrbuf_add(&sb, \"\\\\E\", 2);\n-\n-\t\tp->pattern = sb.buf;\n-\t\tp->patternlen = sb.len;\n-\n-\t\t/* FIXME: Check in compile_pcre2_pattern() that we're\n-\t\t * using basic rx using !opt->pcre2 && <something>\n-\t\t */\n-\t\topt->pcre2 = 1;\n-\n-\t\tcompile_pcre2_pattern(p, opt);\n-\t\treturn;\n-\t}\n-#endif\n-\n \t/*\n \t * Even when -F (fixed) asks us to do a non-regexp search, we\n \t * may not be able to correctly case-fold when -i\n@@ -668,12 +692,26 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tcompile_fixed_regexp(p, opt);\n \t\treturn;\n \t}\n+#endif\n \n \tif (opt->pcre2) {\n \t\tcompile_pcre2_pattern(p, opt);\n \t\treturn;\n \t}\n \n+#ifdef USE_LIBPCRE2\n+\tif (opt->fixed || has_null(p->pattern, p->patternlen) || is_fixed(p->pattern, p->patternlen)) {\n+\t\tcompile_pcre2_pattern(p, opt);\n+\t\treturn;\n+\t}\n+\n+#ifdef PCRE2_CONVERT_POSIX_BASIC\n+\topt->pcre2_posix_emulation = 1;\n+\tcompile_pcre2_pattern(p, opt);\n+\treturn;\n+#endif\n+#endif\n+\n \tif (opt->pcre1) {\n \t\tcompile_pcre1_regexp(p, opt);\n \t\treturn;\n@@ -686,6 +724,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tregfree(&p->regexp);\n \t\tcompile_regexp_failed(p, errbuf);\n \t}\n+\treturn;\n }\n \n static struct grep_expr *compile_pattern_or(struct grep_pat **);\ndiff --git a/grep.h b/grep.h\nindex b40afc2e2f..39897489e4 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -29,6 +29,9 @@ typedef int pcre2_compile_context;\n typedef int pcre2_match_context;\n typedef int pcre2_jit_stack;\n #endif\n+#ifndef PCRE2_CONVERT_POSIX_EXTENDED\n+typedef int pcre2_convert_context;\n+#endif\n #include \"kwset.h\"\n #include \"thread-utils.h\"\n #include \"userdiff.h\"\n@@ -73,6 +76,7 @@ struct grep_pat {\n \tpcre_jit_stack *pcre1_jit_stack;\n \tconst unsigned char *pcre1_tables;\n \tint pcre1_jit_on;\n+\tpcre2_convert_context *pcre2_convert_context;\n \tpcre2_code *pcre2_pattern;\n \tpcre2_match_data *pcre2_match_data;\n \tpcre2_compile_context *pcre2_compile_context;\n@@ -143,6 +147,7 @@ struct grep_opt {\n \tint use_reflog_filter;\n \tint pcre1;\n \tint pcre2;\n+\tint pcre2_posix_emulation;\n \tint relative;\n \tint pathname;\n \tint null_following_name;\ndiff --git a/t/README b/t/README\nindex 1ff612ca65..0dbf5373a2 100644\n--- a/t/README\n+++ b/t/README\n@@ -820,6 +820,12 @@ use these, and \"test_set_prereq\" for how to define your own.\n    USE_LIBPCRE2=YesPlease. Wrap any PCRE using tests that for some\n    reason need v2 of the PCRE library instead of v1 in these.\n \n+ - LIBPCRE2_BUNDLED\n+\n+   Git was compiled with the bundled PCRE v2 support via\n+   USE_LIBPCRE2=YesPlease &\n+   USE_LIBPCRE2_BUNDLED=IWantPatternConvertAwesomeSauce.\n+\n  - CASE_INSENSITIVE_FS\n \n    Test is run on a case insensitive file system.\ndiff --git a/t/t7008-grep-binary.sh b/t/t7008-grep-binary.sh\nindex fc86ed5fce..d9de5c986c 100755\n--- a/t/t7008-grep-binary.sh\n+++ b/t/t7008-grep-binary.sh\n@@ -100,9 +100,16 @@ test_expect_success 'git grep ile a' '\n \tgit grep ile a\n '\n \n-test_expect_failure 'git grep .fi a' '\n-\tgit grep .fi a\n-'\n+if test_have_prereq LIBPCRE2_BUNDLED\n+then\n+\ttest_expect_success 'git grep .fi a' '\n+\t\tgit grep .fi a\n+\t'\n+else\n+\ttest_expect_failure 'git grep .fi a' '\n+\t\tgit grep .fi a\n+\t'\n+fi\n \n nul_match 1 '-F' 'yQf'\n nul_match 0 '-F' 'yQx'\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 13ed81dc16..71760e01a0 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1014,6 +1014,7 @@ test -z \"$NO_PYTHON\" && test_set_prereq PYTHON\n test -n \"$USE_LIBPCRE1$USE_LIBPCRE2\" && test_set_prereq PCRE\n test -n \"$USE_LIBPCRE1\" && test_set_prereq LIBPCRE1\n test -n \"$USE_LIBPCRE2\" && test_set_prereq LIBPCRE2\n+test -n \"$USE_LIBPCRE2_BUNDLED\" && test_set_prereq LIBPCRE2_BUNDLED\n test -z \"$NO_GETTEXT\" && test_set_prereq GETTEXT\n \n # Can we rely on git's output in the C locale?\n-- \n2.11.0\n\n"},{"id":"319432","messageId":"20170511175115.648-6-avarab@gmail.com","threadId":"45937","inReplyTo":"20170511175115.648-1-avarab@gmail.com","subject":"[PATCH/RFC 5/6] grep: support regex patterns containing \\0 via PCRE v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-05-11T17:51:14Z","receivedAt":"2017-05-11T17:52:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Support regex patterns with embedded \\0's, as an earlier commit[1]\nnotes this was previously impossible due to an internal limitation.\n\nBefore this change any regex metacharacters in patterns containing \\0\nwere silently ignored and the pattern matched as if it were a\n--fixed-strings pattern.\n\nNow these patterns will be matched with PCRE instead, which supports\ncombining regex metacharacters with patterns containing \\0.\n\nA side-effect of this change is that these patterns which previously\nwould be considered --fixed-strings patterns regardless of the engine\nrequested now all implicitly become --perl-regexp instead.\n\nA subsequent change introduces a POSIX to PCRE syntax converter, and\ncould be used to be 100% truthful to our documentation by using POSIX\nbasic syntax (which we haven't been in quite some time with kwset).\n\nBut due to a chicken & egg issue with this change being easier to\nimplement stand-alone first, the subsequent change depending on a SVN\ntrunk version of PCRE, but most importantly I don't think anyone will\nmind this change, so I'm leaving it as it is.\n\nThis implementation is faster than the previous kwset implementation,\nbut I haven't bothered to come up with a \\0-specific fixed-string\nperformance test.\n\nSee the next change in this series for a change which optionally\nexpands the PCRE v2 use to use it for all fixed-string patterns, the\nperformance tests for those will be applicable to these patterns as\nwell, since PCRE matches \\0 like any other character.\n\n1. (\"grep: factor test for \\0 in grep patterns into a function\",\n   2017-05-08)\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n grep.c                 | 24 ++++++++++++++++\n t/t7008-grep-binary.sh | 74 ++++++++++++++++++++++++++++++++++----------------\n 2 files changed, 75 insertions(+), 23 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 2ff4e253ff..5db614cf80 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -613,6 +613,30 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \ticase\t       = opt->regflags & REG_ICASE || p->ignore_case;\n \tascii_only     = !has_non_ascii(p->pattern);\n \n+#ifdef USE_LIBPCRE2\n+\tif (has_null(p->pattern, p->patternlen)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tif (icase)\n+\t\t\tstrbuf_add(&sb, \"(?i)\", 4);\n+\t\tif (opt->fixed)\n+\t\t\tstrbuf_add(&sb, \"\\\\Q\", 2);\t\t\n+\t\tstrbuf_add(&sb, p->pattern, p->patternlen);\n+\t\tif (opt->fixed)\n+\t\t\tstrbuf_add(&sb, \"\\\\E\", 2);\n+\n+\t\tp->pattern = sb.buf;\n+\t\tp->patternlen = sb.len;\n+\n+\t\t/* FIXME: Check in compile_pcre2_pattern() that we're\n+\t\t * using basic rx using !opt->pcre2 && <something>\n+\t\t */\n+\t\topt->pcre2 = 1;\n+\n+\t\tcompile_pcre2_pattern(p, opt);\n+\t\treturn;\n+\t}\n+#endif\n+\n \t/*\n \t * Even when -F (fixed) asks us to do a non-regexp search, we\n \t * may not be able to correctly case-fold when -i\ndiff --git a/t/t7008-grep-binary.sh b/t/t7008-grep-binary.sh\nindex ba3db06501..fc86ed5fce 100755\n--- a/t/t7008-grep-binary.sh\n+++ b/t/t7008-grep-binary.sh\n@@ -124,35 +124,63 @@ nul_match 0 '-F' '[æ]Qð'\n nul_match 0 '-Fi' 'ÆQ[Ð]'\n nul_match 0 '-Fi' '[Æ]QÐ'\n \n-# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n-# patterns case-insensitively.\n-nul_match T1 '-i' 'ÆQÐ'\n-\n-# \\0 implicitly disables regexes. This is an undocumented internal\n-# limitation.\n-nul_match T1 '' 'yQ[f]'\n-nul_match T1 '' '[y]Qf'\n-nul_match T1 '-i' 'YQ[F]'\n-nul_match T1 '-i' '[Y]Qf'\n-nul_match T1 '' 'æQ[ð]'\n-nul_match T1 '' '[æ]Qð'\n-nul_match T1 '-i' 'ÆQ[Ð]'\n-\n-# ... because of \\0 implicitly disabling regexes regexes that\n-# should/shouldn't match don't do the right thing.\n-nul_match T1 '' 'eQm.*cQ'\n-nul_match T1 '-i' 'EQM.*cQ'\n-nul_match T0 '' 'eQm[*]c'\n-nul_match T0 '-i' 'EQM[*]C'\n+if test_have_prereq LIBPCRE2\n+then\n+\t# Regex patterns that should match without -F\n+\tnul_match 1 '' 'yQ[f]'\n+\tnul_match 1 '' '[y]Qf'\n+\tnul_match 1 '-i' 'YQ[F]'\n+\tnul_match 1 '-i' '[Y]Qf'\n+\tnul_match 1 '' 'æQ[ð]'\n+\tnul_match 1 '' '[æ]Qð'\n+\tnul_match 0 '-i' '[Æ]Qð'\n+\tnul_match 1 '' 'eQm.*cQ'\n+\tnul_match 1 '-i' 'EQM.*cQ'\n+\tnul_match 0 '' 'eQm[*]c'\n+\tnul_match 0 '-i' 'EQM[*]C'\n+\n+\t# These should also match, but don't due to some heisenbug,\n+\t# they succeed when run manually!\n+\tnul_match T1 '-i' 'ÆQÐ'\n+\tnul_match T1 '-i' 'ÆQ[Ð]'\n+else\n+\t# \\0 implicitly disables regexes. This is an undocumented\n+\t# internal limitation.\n+\tnul_match T1 '' 'yQ[f]'\n+\tnul_match T1 '' '[y]Qf'\n+\tnul_match T1 '-i' 'YQ[F]'\n+\tnul_match T1 '-i' '[Y]Qf'\n+\tnul_match T1 '' 'æQ[ð]'\n+\tnul_match T1 '' '[æ]Qð'\n+\tnul_match T1 '-i' 'ÆQ[Ð]'\n+\n+\t# ... because of \\0 implicitly disabling regexes regexes that\n+\t# should/shouldn't match don't do the right thing.\n+\tnul_match T1 '' 'eQm.*cQ'\n+\tnul_match T1 '-i' 'EQM.*cQ'\n+\tnul_match T0 '' 'eQm[*]c'\n+\tnul_match T0 '-i' 'EQM[*]C'\n+fi\n \n # Due to the REG_STARTEND extension when kwset() is disabled on -i &\n # non-ASCII the string will be matched in its entirety, but the\n # pattern will be cut off at the first \\0.\n nul_match 0 '-i' 'NOMATCHQð'\n-nul_match T0 '-i' '[Æ]QNOMATCH'\n-nul_match T0 '-i' '[æ]QNOMATCH'\n+if test_have_prereq LIBPCRE2\n+then\n+\tnul_match 0 '-i' '[Æ]QNOMATCH'\n+\tnul_match 0 '-i' '[æ]QNOMATCH'\n+else\n+\tnul_match T0 '-i' '[Æ]QNOMATCH'\n+\tnul_match T0 '-i' '[æ]QNOMATCH'\n+fi\n # Matches, but for the wrong reasons, just stops at [æ]\n-nul_match 1 '-i' '[Æ]Qð'\n+if test_have_prereq LIBPCRE2\n+then\n+\tnul_match T1 '-i' '[Æ]Qð'\n+else\n+\tnul_match 1 '-i' '[Æ]Qð'\n+fi\n nul_match 1 '-i' '[æ]Qð'\n \n # Ensure that the matcher doesn't regress to something that stops at\n-- \n2.11.0\n\n"}]}