{"thread":{"id":"51300","subject":"[PATCH 4/4] config: avoid calling `labs()` on too-large data type","startedAt":"2019-06-13T15:26:07Z","lastAt":"2019-07-03T11:27:14Z","messageCount":90,"participants":["Johannes Schindelin via GitGitGadget","Junio C Hamano","SZEDER Gábor","Jeff King","Ramsay Jones","Ævar Arnfjörð Bjarmason","Carlo Arenas","brian m. carlson","René Scharfe","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"377121","messageId":"9613c88849e53e00e1f7ade49c6c9056309ef0b9.1560426581.git.gitgitgadget@gmail.com","threadId":"51300","inReplyTo":"pull.265.git.gitgitgadget@gmail.com","subject":"[PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-13T11:49:47Z","receivedAt":"2019-06-13T15:26:07Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe `labs()` function operates, as the initial `l` suggests, on `long`\nparameters. However, in `config.c` we tried to use it on values of type\n`intmax_t`.\n\nThis problem was found by GCC v9.x.\n\nTo fix it, let's just \"unroll\" the function (i.e. negate the value if it\nis negative).\n\nSigned-off-by: Johannes Schindelin \n---\n config.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 296a6d9cc4..01c6e9df23 100644\n--- a/config.c\n+++ b/config.c\n@@ -869,9 +869,9 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n \t\t\terrno = EINVAL;\n \t\t\treturn 0;\n \t\t}\n-\t\tuval = labs(val);\n+\t\tuval = val < 0 ? -val : val;\n \t\tuval *= factor;\n-\t\tif (uval > max || labs(val) > uval) {\n+\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n \t\t\terrno = ERANGE;\n \t\t\treturn 0;\n \t\t}\n-- \ngitgitgadget\n"},{"id":"377122","messageId":"d0ae652801c3ae343cc0c6b29c4444282d6c2d4a.1560426581.git.gitgitgadget@gmail.com","threadId":"51300","inReplyTo":"pull.265.git.gitgitgadget@gmail.com","subject":"[PATCH 3/4] winansi: simplify loading the GetCurrentConsoleFontEx() function","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-13T11:49:46Z","receivedAt":"2019-06-13T15:26:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWe introduced helper macros to simplify loading functions dynamically.\nMight just as well use them.\n\nThis also side-steps a compiler warning when building with GCC v8.x: it\nwould complain about casting between incompatible function pointers.\n\nSigned-off-by: Johannes Schindelin \n---\n compat/winansi.c | 14 +++++---------\n 1 file changed, 5 insertions(+), 9 deletions(-)\n\ndiff --git a/compat/winansi.c b/compat/winansi.c\nindex f4f08237f9..a29d34ef44 100644\n--- a/compat/winansi.c\n+++ b/compat/winansi.c\n@@ -7,6 +7,7 @@\n #include \n #include \n #include \"win32.h\"\n+#include \"win32/lazyload.h\"\n \n static int fd_is_interactive[3] = { 0, 0, 0 };\n #define FD_CONSOLE 0x1\n@@ -41,26 +42,21 @@ typedef struct _CONSOLE_FONT_INFOEX {\n #endif\n #endif\n \n-typedef BOOL (WINAPI *PGETCURRENTCONSOLEFONTEX)(HANDLE, BOOL,\n-\t\tPCONSOLE_FONT_INFOEX);\n-\n static void warn_if_raster_font(void)\n {\n \tDWORD fontFamily = 0;\n-\tPGETCURRENTCONSOLEFONTEX pGetCurrentConsoleFontEx;\n+\tDECLARE_PROC_ADDR(kernel32.dll, BOOL, GetCurrentConsoleFontEx,\n+\t\t\tHANDLE, BOOL, PCONSOLE_FONT_INFOEX);\n \n \t/* don't bother if output was ascii only */\n \tif (!non_ascii_used)\n \t\treturn;\n \n \t/* GetCurrentConsoleFontEx is available since Vista */\n-\tpGetCurrentConsoleFontEx = (PGETCURRENTCONSOLEFONTEX) GetProcAddress(\n-\t\t\tGetModuleHandle(\"kernel32.dll\"),\n-\t\t\t\"GetCurrentConsoleFontEx\");\n-\tif (pGetCurrentConsoleFontEx) {\n+\tif (INIT_PROC_ADDR(GetCurrentConsoleFontEx)) {\n \t\tCONSOLE_FONT_INFOEX cfi;\n \t\tcfi.cbSize = sizeof(cfi);\n-\t\tif (pGetCurrentConsoleFontEx(console, 0, &cfi))\n+\t\tif (GetCurrentConsoleFontEx(console, 0, &cfi))\n \t\t\tfontFamily = cfi.FontFamily;\n \t} else {\n \t\t/* pre-Vista: check default console font in registry */\n-- \ngitgitgadget\n\n"},{"id":"377123","messageId":"pull.265.git.gitgitgadget@gmail.com","threadId":"51300","inReplyTo":null,"subject":"[PATCH 0/4] Support building with GCC v8.x/v9.x","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-13T11:49:43Z","receivedAt":"2019-06-13T15:26:11Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"I noticed a while ago that I could not build Git's master in Git for\nWindows' SDK when using GCC v8.x. This became a much less pressing problem\nwhen I discovered a serious bug that would not let us compile with ASLR/DEP\nenabled (the resulting executables would just throw segmentation faults left\nand right), so I put these patches on the backburner for a while.\n\nBut now GCC v8.x is fixed, and MSYS2 even switched to GCC v9.x (which found\nyet another problem that I fix in patch 4/4 of this patch series). So it is\ntime to get Git's master ready.\n\nJohannes Schindelin (4):\n poll (mingw): allow compiling with GCC 8 and DEVELOPER=1\n kwset: allow building with GCC 8\n winansi: simplify loading the GetCurrentConsoleFontEx() function\n config: avoid calling `labs()` on too-large data type\n\n compat/poll/poll.c | 2 +-\n compat/winansi.c | 14 +++++---------\n config.c | 4 ++--\n kwset.c | 8 +++++++-\n 4 files changed, 15 insertions(+), 13 deletions(-)\n\n\nbase-commit: b697d92f56511e804b8ba20ccbe7bdc85dc66810\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-265%2Fdscho%2Fgcc-8-and-9-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-265/dscho/gcc-8-and-9-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/265\n-- \ngitgitgadget\n"},{"id":"377124","messageId":"fde98c32199a6570c721dbef7795e278b36224db.1560426581.git.gitgitgadget@gmail.com","threadId":"51300","inReplyTo":"pull.265.git.gitgitgadget@gmail.com","subject":"[PATCH 1/4] poll (mingw): allow compiling with GCC 8 and DEVELOPER=1","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-13T11:49:44Z","receivedAt":"2019-06-13T15:26:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe return type of the `GetProcAddress()` function is `FARPROC` which\nevaluates to `long long int (*)()`, i.e. it cannot be cast to the\ncorrect function signature by GCC 8.\n\nTo work around that, we first cast to `void *` and go on with our merry\nlives.\n\nSigned-off-by: Johannes Schindelin \n---\n compat/poll/poll.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/compat/poll/poll.c b/compat/poll/poll.c\nindex 4459408c7d..8b07edb0fe 100644\n--- a/compat/poll/poll.c\n+++ b/compat/poll/poll.c\n@@ -149,7 +149,7 @@ win32_compute_revents (HANDLE h, int *p_sought)\n case FILE_TYPE_PIPE:\n if (!once_only)\n \t{\n-\t NtQueryInformationFile = (PNtQueryInformationFile)\n+\t NtQueryInformationFile = (PNtQueryInformationFile)(void (*)(void))\n \t GetProcAddress (GetModuleHandle (\"ntdll.dll\"),\n \t\t\t \"NtQueryInformationFile\");\n \t once_only = TRUE;\n-- \ngitgitgadget\n\n"},{"id":"377125","messageId":"c1527a71672056859a4613f2318bcbfce31e8b50.1560426581.git.gitgitgadget@gmail.com","threadId":"51300","inReplyTo":"pull.265.git.gitgitgadget@gmail.com","subject":"[PATCH 2/4] kwset: allow building with GCC 8","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-13T11:49:45Z","receivedAt":"2019-06-13T15:26:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe kwset functionality makes use of the obstack code, which expects to\nbe handed a function that can allocate large chunks of data. It expects\nthat function to accept a `size` parameter of type `long`.\n\nThis upsets GCC 8 on Windows, because `long` does not have the same\nbit size as `size_t` there.\n\nNow, the proper thing to do would be to switch to `size_t`. But this\nwould make us deviate from the \"upstream\" code even further, making it\nhard to synchronize with newer versions, and also it would be quite\ninvolved because that `long` type is so invasive in that code.\n\nLet's punt, and instead provide a super small wrapper around\n`xmalloc()`.\n\nSigned-off-by: Johannes Schindelin \n---\n kwset.c | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/kwset.c b/kwset.c\nindex 4fb6455aca..efc2ff41bc 100644\n--- a/kwset.c\n+++ b/kwset.c\n@@ -38,7 +38,13 @@\n #include \"compat/obstack.h\"\n \n #define NCHAR (UCHAR_MAX + 1)\n-#define obstack_chunk_alloc xmalloc\n+/* adapter for `xmalloc()`, which takes `size_t`, not `long` */\n+static void *obstack_chunk_alloc(long size)\n+{\n+\tif (size < 0)\n+\t\tBUG(\"Cannot allocate a negative amount: %ld\", size);\n+\treturn xmalloc(size);\n+}\n #define obstack_chunk_free free\n \n #define U(c) ((unsigned char) (c))\n-- \ngitgitgadget\n\n"},{"id":"377130","messageId":"xmqqmuilwjm6.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"c1527a71672056859a4613f2318bcbfce31e8b50.1560426581.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/4] kwset: allow building with GCC 8","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-13T16:11:29Z","receivedAt":"2019-06-13T16:11:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> The kwset functionality makes use of the obstack code, which expects to\n> be handed a function that can allocate large chunks of data. It expects\n> that function to accept a `size` parameter of type `long`.\n>\n> This upsets GCC 8 on Windows, because `long` does not have the same\n> bit size as `size_t` there.\n>\n> Now, the proper thing to do would be to switch to `size_t`. But this\n> would make us deviate from the \"upstream\" code even further, making it\n> hard to synchronize with newer versions, and also it would be quite\n> involved because that `long` type is so invasive in that code.\n>\n> Let's punt, and instead provide a super small wrapper around\n> `xmalloc()`.\n\nYay.\n\nThe above description makes it sound as if this patch is an ugly\nworkaround, but I think this is \"the proper thing\" to do, as long as\nthe use of obstack stuff in this context is meant to allocate less\nthan MAX_LONG bytes at a time, even if long is somtimes smaller than\nsize_t.\n\nThanks.\n\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> kwset.c | 8 +++++++-\n> 1 file changed, 7 insertions(+), 1 deletion(-)\n>\n> diff --git a/kwset.c b/kwset.c\n> index 4fb6455aca..efc2ff41bc 100644\n> --- a/kwset.c\n> +++ b/kwset.c\n> @@ -38,7 +38,13 @@\n> #include \"compat/obstack.h\"\n> \n> #define NCHAR (UCHAR_MAX + 1)\n> -#define obstack_chunk_alloc xmalloc\n> +/* adapter for `xmalloc()`, which takes `size_t`, not `long` */\n> +static void *obstack_chunk_alloc(long size)\n> +{\n> +\tif (size < 0)\n> +\t\tBUG(\"Cannot allocate a negative amount: %ld\", size);\n> +\treturn xmalloc(size);\n> +}\n> #define obstack_chunk_free free\n> \n> #define U(c) ((unsigned char) (c))\n"},{"id":"377131","messageId":"xmqqimt9wji2.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"9613c88849e53e00e1f7ade49c6c9056309ef0b9.1560426581.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-13T16:13:57Z","receivedAt":"2019-06-13T16:14:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> The `labs()` function operates, as the initial `l` suggests, on `long`\n> parameters. However, in `config.c` we tried to use it on values of type\n> `intmax_t`.\n>\n> This problem was found by GCC v9.x.\n>\n> To fix it, let's just \"unroll\" the function (i.e. negate the value if it\n> is negative).\n\nThanks. Obviously correct.\n\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> config.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/config.c b/config.c\n> index 296a6d9cc4..01c6e9df23 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -869,9 +869,9 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n> \t\t\terrno = EINVAL;\n> \t\t\treturn 0;\n> \t\t}\n> -\t\tuval = labs(val);\n> +\t\tuval = val < 0 ? -val : val;\n> \t\tuval *= factor;\n> -\t\tif (uval > max || labs(val) > uval) {\n> +\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n> \t\t\terrno = ERANGE;\n> \t\t\treturn 0;\n> \t\t}\n"},{"id":"377223","messageId":"20190614095308.GG31952@szeder.dev","threadId":"51300","inReplyTo":"c1527a71672056859a4613f2318bcbfce31e8b50.1560426581.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/4] kwset: allow building with GCC 8","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-14T09:53:08Z","receivedAt":"2019-06-14T09:53:15Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> Subject: Re: [PATCH 2/4] kwset: allow building with GCC 8\n\nThe subject could benefit from a \"on Windows\" at the end; 'kwset' and\ncompat/obstack can be build with GCC 8 and 9 just fine on some other\nplatforms.\n\nOn Thu, Jun 13, 2019 at 04:49:45AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin \n> \n> The kwset functionality makes use of the obstack code, which expects to\n> be handed a function that can allocate large chunks of data. It expects\n> that function to accept a `size` parameter of type `long`.\n> \n> This upsets GCC 8 on Windows, because `long` does not have the same\n> bit size as `size_t` there.\n> \n> Now, the proper thing to do would be to switch to `size_t`. But this\n> would make us deviate from the \"upstream\" code even further,\n\nThis is not entirely true: upstream already uses 'size_t', so the\nswitch would actually bring our copy closer to upstream.\n\nBut look out for the patches that I'll send out in a minute...\n\n> making it\n> hard to synchronize with newer versions, and also it would be quite\n> involved because that `long` type is so invasive in that code.\n> \n> Let's punt, and instead provide a super small wrapper around\n> `xmalloc()`.\n> \n> Signed-off-by: Johannes Schindelin \n> ---\n> kwset.c | 8 +++++++-\n> 1 file changed, 7 insertions(+), 1 deletion(-)\n> \n> diff --git a/kwset.c b/kwset.c\n> index 4fb6455aca..efc2ff41bc 100644\n> --- a/kwset.c\n> +++ b/kwset.c\n> @@ -38,7 +38,13 @@\n> #include \"compat/obstack.h\"\n> \n> #define NCHAR (UCHAR_MAX + 1)\n> -#define obstack_chunk_alloc xmalloc\n> +/* adapter for `xmalloc()`, which takes `size_t`, not `long` */\n> +static void *obstack_chunk_alloc(long size)\n> +{\n> +\tif (size < 0)\n> +\t\tBUG(\"Cannot allocate a negative amount: %ld\", size);\n> +\treturn xmalloc(size);\n> +}\n> #define obstack_chunk_free free\n> \n> #define U(c) ((unsigned char) (c))\n> -- \n> gitgitgadget\n> \n"},{"id":"377224","messageId":"20190614100059.13540-1-szeder.dev@gmail.com","threadId":"51300","inReplyTo":"20190614095308.GG31952@szeder.dev","subject":"[RFC/PATCH v1 0/4] compat/obstack: update from upstream","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-14T10:00:55Z","receivedAt":"2019-06-14T10:01:08Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Update 'compat/obstack.{c,h}' from upstream, because they already use\n'size_t' instead of 'long' in places that might eventually end up as\nan argument to malloc(), which might solve build errors with GCC 8 on\nWindows.\n\nThe first patch just imports from upstream and doesn't modify anything\nat all, and, consequently, it can't be compiled because of a screenful\nor two of errors. This is bad for future bisects, of course.\n\nOTOH, adding all the necessary build fixes right away makes review\nharder...\n\nI'm not sure how to deal with this situation, so here is a series with\nthe fixes in separate patches for review, for now. If there's an\nagreement that this is the direction to take, then I'll squash in the\nfixes in the first patch and touch up the resulting commit message.\n\n\nRamsay, could you please run sparse on top of these patch series to\nmake sure that I caught and converted all \"0 instead of NULL\" usages\nin the last patch? Thanks.\n\n\nAnd here is an all-green build of these patches on Travis CI:\n\n https://travis-ci.org/szeder/git/builds/545645247\n\n(and one bonus patch on top to deal with some Homebrew nonsense)\n\nSZEDER Gábor (4):\n compat/obstack: update from upstream\n SQUASH??? compat/obstack: fix portability issues\n SQUASH??? compat/obstack: fix build errors with Clang\n compat/obstack: fix some sparse warnings\n\n compat/obstack.c | 356 ++++++++------------\n compat/obstack.h | 832 ++++++++++++++++++++++++-----------------------\n 2 files changed, 572 insertions(+), 616 deletions(-)\n\n-- \n2.22.0.589.g5bd7971b91\n\n"},{"id":"377225","messageId":"20190614100059.13540-3-szeder.dev@gmail.com","threadId":"51300","inReplyTo":"20190614100059.13540-1-szeder.dev@gmail.com","subject":"[PATCH v1 2/4] SQUASH??? compat/obstack: fix portability issues","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-14T10:00:57Z","receivedAt":"2019-06-14T10:01:09Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"This is sort-of a cherry-pick of d190a0875f (obstack: Fix portability\nissues, 2011-08-28), which is necessary to make 'compat/obstack.c'\ncompile again. Only \"sort-of a cherry-pick\", because the divergence\nbetween upstream and our copy was just too bit and I gave up on the\nconflict resolution, and instead made the still necessary/applicable\nedits in the spirit of d190a0875f by hand.\n\nWith this patch 'compat/obstack.c' can be compiled with GCC both on\nLinux and on macOS (well, at least in our 'osx-gcc' build job on\nTravis CI), but, alas, not with Clang.\n\nSigned-off-by: SZEDER Gábor \n---\n compat/obstack.c | 29 ++++-------------------------\n 1 file changed, 4 insertions(+), 25 deletions(-)\n\ndiff --git a/compat/obstack.c b/compat/obstack.c\nindex 6949111e4d..17fa95d46c 100644\n--- a/compat/obstack.c\n+++ b/compat/obstack.c\n@@ -16,13 +16,9 @@\n License along with the GNU C Library; if not, see\n . */\n \n-\n-#ifdef _LIBC\n-# include \n-#else\n-# include \n-# include \"obstack.h\"\n-#endif\n+#include \"git-compat-util.h\"\n+#include \n+#include \"obstack.h\"\n \n /* NOTE BEFORE MODIFYING THIS FILE: _OBSTACK_INTERFACE_VERSION in\n obstack.h must be incremented whenever callers compiled using an old\n@@ -305,23 +301,6 @@ _obstack_memory_used (struct obstack *h)\n /* Define the error handler. */\n # include \n \n-/* Exit value used when 'print_and_abort' is used. */\n-# ifdef _LIBC\n-int obstack_exit_failure = EXIT_FAILURE;\n-# else\n-# include \"exitfail.h\"\n-# define obstack_exit_failure exit_failure\n-# endif\n-\n-# ifdef _LIBC\n-# include \n-# else\n-# include \"gettext.h\"\n-# endif\n-# ifndef _\n-# define _(msgid) gettext (msgid)\n-# endif\n-\n # ifdef _LIBC\n # include \n # endif\n@@ -339,7 +318,7 @@ print_and_abort (void)\n # else\n fprintf (stderr, \"%s\\n\", _(\"memory exhausted\"));\n # endif\n- exit (obstack_exit_failure);\n+ exit (1);\n }\n \n /* The functions allocating more room by calling 'obstack_chunk_alloc'\n-- \n2.22.0.589.g5bd7971b91\n\n"},{"id":"377226","messageId":"20190614100059.13540-5-szeder.dev@gmail.com","threadId":"51300","inReplyTo":"20190614100059.13540-1-szeder.dev@gmail.com","subject":"[PATCH v1 4/4] compat/obstack: fix some sparse warnings","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-14T10:00:59Z","receivedAt":"2019-06-14T10:01:12Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"'compat/obstack.c' occasionally assigns/compares a plain 0 to a\npointer, which triggers sparse warnings. Use NULL instead.\n\nThis is basically a cherry-pick of 3254310863 (obstack.c: Fix some\nsparse warnings, 2011-09-11) on top of the just updated code from\nupstream.\n\nSigned-off-by: SZEDER Gábor \n---\n compat/obstack.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/compat/obstack.c b/compat/obstack.c\nindex 6ef8cecb8a..5fff087cd3 100644\n--- a/compat/obstack.c\n+++ b/compat/obstack.c\n@@ -135,7 +135,7 @@ _obstack_begin_worker (struct obstack *h,\n h->next_free = h->object_base = __PTR_ALIGN ((char *) chunk, chunk->contents,\n alignment - 1);\n h->chunk_limit = chunk->limit = (char *) chunk + h->chunk_size;\n- chunk->prev = 0;\n+ chunk->prev = NULL;\n /* The initial chunk now contains no empty object. */\n h->maybe_empty_object = 0;\n h->alloc_failed = 0;\n@@ -178,7 +178,7 @@ void\n _obstack_newchunk (struct obstack *h, _OBSTACK_SIZE_T length)\n {\n struct _obstack_chunk *old_chunk = h->chunk;\n- struct _obstack_chunk *new_chunk = 0;\n+ struct _obstack_chunk *new_chunk = NULL;\n size_t obj_size = h->next_free - h->object_base;\n char *object_base;\n \n@@ -243,12 +243,12 @@ _obstack_allocated_p (struct obstack *h, void *obj)\n /* We use >= rather than > since the object cannot be exactly at\n the beginning of the chunk but might be an empty object exactly\n at the end of an adjacent chunk. */\n- while (lp != 0 && ((void *) lp >= obj || (void *) (lp)->limit < obj))\n+ while (lp != NULL && ((void *) lp >= obj || (void *) (lp)->limit < obj))\n {\n plp = lp->prev;\n lp = plp;\n }\n- return lp != 0;\n+ return lp != NULL;\n }\n \n /* Free objects in obstack H, including OBJ and everything allocate\n@@ -264,7 +264,7 @@ _obstack_free (struct obstack *h, void *obj)\n /* We use >= because there cannot be an object at the beginning of a chunk.\n But there can be an empty object at that address\n at the end of another chunk. */\n- while (lp != 0 && ((void *) lp >= obj || (void *) (lp)->limit < obj))\n+ while (lp != NULL && ((void *) lp >= obj || (void *) (lp)->limit < obj))\n {\n plp = lp->prev;\n call_freefun (h, lp);\n@@ -279,7 +279,7 @@ _obstack_free (struct obstack *h, void *obj)\n h->chunk_limit = lp->limit;\n h->chunk = lp;\n }\n- else if (obj != 0)\n+ else if (obj != NULL)\n /* obj is not in any of the chunks! */\n abort ();\n }\n@@ -290,7 +290,7 @@ _obstack_memory_used (struct obstack *h)\n struct _obstack_chunk *lp;\n _OBSTACK_SIZE_T nbytes = 0;\n \n- for (lp = h->chunk; lp != 0; lp = lp->prev)\n+ for (lp = h->chunk; lp != NULL; lp = lp->prev)\n {\n nbytes += lp->limit - (char *) lp;\n }\n-- \n2.22.0.589.g5bd7971b91\n\n"},{"id":"377227","messageId":"20190614100059.13540-4-szeder.dev@gmail.com","threadId":"51300","inReplyTo":"20190614100059.13540-1-szeder.dev@gmail.com","subject":"[PATCH v1 3/4] SQUASH??? compat/obstack: fix build errors with Clang","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-14T10:00:58Z","receivedAt":"2019-06-14T10:01:13Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Compiling 'compat/obstack.c' with Clang on Linux and macOS fails with\ndifferent errors:\n\nOn Linux:\n\n CC compat/obstack.o\n compat/obstack.c:330:31: error: incompatible pointer types initializing\n 'void (*)(void) __attribute__((noreturn))' with an expression of type\n 'void (void)' [-Werror,-Wincompatible-pointer-types]\n __attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)\n ^\n\nRemove '__attribute_noreturn__' from the function's declaration and\ndefinition to resolve this build error.\n\nOn macOS:\n\n compat/obstack.h:223:3: error: expected function body after function declarator\n __attribute_pure__;\n ^\n compat/obstack.h:151:29: note: expanded from macro '__attribute_pure__'\n # define __attribute_pure__ _GL_ATTRIBUTE_PURE\n\nRemove '__attribute_pure__' to resolve this build error.\n\nWith this patch it's now possible to compile 'compat/obstack.c' both\nwith GCC and Clang on both on Linux and macOS.\n\nSigned-off-by: SZEDER Gábor \n---\n compat/obstack.c | 4 ++--\n compat/obstack.h | 5 ++---\n 2 files changed, 4 insertions(+), 5 deletions(-)\n\ndiff --git a/compat/obstack.c b/compat/obstack.c\nindex 17fa95d46c..6ef8cecb8a 100644\n--- a/compat/obstack.c\n+++ b/compat/obstack.c\n@@ -231,7 +231,7 @@ _obstack_newchunk (struct obstack *h, _OBSTACK_SIZE_T length)\n \n /* Suppress -Wmissing-prototypes warning. We don't want to declare this in\n obstack.h because it is just for debugging. */\n-int _obstack_allocated_p (struct obstack *h, void *obj) __attribute_pure__;\n+int _obstack_allocated_p (struct obstack *h, void *obj);\n \n int\n _obstack_allocated_p (struct obstack *h, void *obj)\n@@ -327,7 +327,7 @@ print_and_abort (void)\n abort gracefully or use longjump - but shouldn't return. This\n variable by default points to the internal function\n 'print_and_abort'. */\n-__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)\n+void (*obstack_alloc_failed_handler) (void)\n = print_and_abort;\n # endif /* !_OBSTACK_NO_ERROR_HANDLER */\n #endif /* !_OBSTACK_ELIDE_CODE */\ndiff --git a/compat/obstack.h b/compat/obstack.h\nindex 811de588a4..f8f9625121 100644\n--- a/compat/obstack.h\n+++ b/compat/obstack.h\n@@ -219,15 +219,14 @@ extern int _obstack_begin_1 (struct obstack *,\n _OBSTACK_SIZE_T, _OBSTACK_SIZE_T,\n void *(*) (void *, size_t),\n void (*) (void *, void *), void *);\n-extern _OBSTACK_SIZE_T _obstack_memory_used (struct obstack *)\n- __attribute_pure__;\n+extern _OBSTACK_SIZE_T _obstack_memory_used (struct obstack *);\n \n \n /* Error handler called when 'obstack_chunk_alloc' failed to allocate\n more memory. This can be set to a user defined function which\n should either abort gracefully or use longjump - but shouldn't\n return. The default action is to print a message and abort. */\n-extern __attribute_noreturn__ void (*obstack_alloc_failed_handler) (void);\n+extern void (*obstack_alloc_failed_handler) (void);\n \n /* Exit value used when 'print_and_abort' is used. */\n extern int obstack_exit_failure;\n-- \n2.22.0.589.g5bd7971b91\n\n"},{"id":"377228","messageId":"20190614100059.13540-2-szeder.dev@gmail.com","threadId":"51300","inReplyTo":"20190614100059.13540-1-szeder.dev@gmail.com","subject":"[PATCH v1 1/4] compat/obstack: update from upstream","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-14T10:00:56Z","receivedAt":"2019-06-14T10:01:13Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"!! This does not compile !!\n\nUpdate 'compat/obstack.{c,h}' from commit\n5905d8ca9945f0d60ff40eb6cfa42afc0199ab8f in\nhttps://git.savannah.gnu.org/git/gnulib.git\n\nWe have made a couple of changes to our copy of 'compat/obstack.{c,h}'\nsince it was introduced in e831171d67 (Add obstack.[ch] from EGLIBC\n2.10, 2011-08-21), and in the meantime some of those issues have been\naddressed in upstream as well [1]. Furthermore, upstream fixed one\nbig issue that we still suffer from, namely our copy still uses type\n'long' to specify the size of the chunk of memory to allocate while\nxmalloc() expects type 'size_t', which triggers compiler errors with\nGCC 8 on Windows, where these two data types are of different size;\nupstream has been using 'size_t' for quite some time now. Making the\nconversion from 'long' to 'size_t' in our copy just doesn't worth it,\nhence the update from upstream.\n\n[1] In particular the following changes have been adressed in\n upstream:\n 764473d257 (compat/obstack: fix -Wcast-function-type warnings,\n 2019-01-17)\n 484257925f (Replace Free Software Foundation address in license\n notices, 2017-11-07)\n 7323513d28 (obstack: fix spelling of similar, 2013-04-12)\n\nSigned-off-by: SZEDER Gábor \n---\n compat/obstack.c | 399 ++++++++++------------\n compat/obstack.h | 835 +++++++++++++++++++++++++----------------------\n 2 files changed, 606 insertions(+), 628 deletions(-)\n\ndiff --git a/compat/obstack.c b/compat/obstack.c\nindex 27cd5c1ea1..6949111e4d 100644\n--- a/compat/obstack.c\n+++ b/compat/obstack.c\n@@ -1,6 +1,5 @@\n /* obstack.c - subroutines used implicitly by object stack macros\n- Copyright (C) 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1996, 1997, 1998,\n- 1999, 2000, 2001, 2002, 2003, 2004, 2005 Free Software Foundation, Inc.\n+ Copyright (C) 1988-2019 Free Software Foundation, Inc.\n This file is part of the GNU C Library.\n \n The GNU C Library is free software; you can redistribute it and/or\n@@ -15,16 +14,19 @@\n \n You should have received a copy of the GNU Lesser General Public\n License along with the GNU C Library; if not, see\n- . */\n+ . */\n \n-#include \"git-compat-util.h\"\n-#include \n-#include \"obstack.h\"\n \n-/* NOTE BEFORE MODIFYING THIS FILE: This version number must be\n- incremented whenever callers compiled using an old obstack.h can no\n- longer properly call the functions in this obstack.c. */\n-#define OBSTACK_INTERFACE_VERSION 1\n+#ifdef _LIBC\n+# include \n+#else\n+# include \n+# include \"obstack.h\"\n+#endif\n+\n+/* NOTE BEFORE MODIFYING THIS FILE: _OBSTACK_INTERFACE_VERSION in\n+ obstack.h must be incremented whenever callers compiled using an old\n+ obstack.h can no longer properly call the functions in this file. */\n \n /* Comment out all this code if we are using the GNU C Library, and are not\n actually compiling the library itself, and the installed library\n@@ -32,113 +34,82 @@\n C Library, but also included in many other GNU distributions. Compiling\n and linking in this code is a waste when using the GNU C library\n (especially if it is a shared library). Rather than having every GNU\n- program understand `configure --with-gnu-libc' and omit the object\n+ program understand 'configure --with-gnu-libc' and omit the object\n files, it is simpler to just do this in the source for each such file. */\n-\n-#include \t\t/* Random thing to get __GNU_LIBRARY__. */\n #if !defined _LIBC && defined __GNU_LIBRARY__ && __GNU_LIBRARY__ > 1\n # include \n-# if _GNU_OBSTACK_INTERFACE_VERSION == OBSTACK_INTERFACE_VERSION\n-# define ELIDE_CODE\n+# if (_GNU_OBSTACK_INTERFACE_VERSION == _OBSTACK_INTERFACE_VERSION\t \\\n+ || (_GNU_OBSTACK_INTERFACE_VERSION == 1\t\t\t\t \\\n+ && _OBSTACK_INTERFACE_VERSION == 2\t\t\t\t \\\n+ && defined SIZEOF_INT && defined SIZEOF_SIZE_T\t\t \\\n+ && SIZEOF_INT == SIZEOF_SIZE_T))\n+# define _OBSTACK_ELIDE_CODE\n # endif\n #endif\n \n-#include \n-\n-#ifndef ELIDE_CODE\n-\n-\n-# if HAVE_INTTYPES_H\n-# include \n+#ifndef _OBSTACK_ELIDE_CODE\n+/* If GCC, or if an oddball (testing?) host that #defines __alignof__,\n+ use the already-supplied __alignof__. Otherwise, this must be Gnulib\n+ (as glibc assumes GCC); defer to Gnulib's alignof_type. */\n+# if !defined __GNUC__ && !defined __alignof__\n+# include \n+# define __alignof__(type) alignof_type (type)\n # endif\n-# if HAVE_STDINT_H || defined _LIBC\n-# include \n+# include \n+# include \n+\n+# ifndef MAX\n+# define MAX(a,b) ((a) > (b) ? (a) : (b))\n # endif\n \n /* Determine default alignment. */\n-union fooround\n-{\n- uintmax_t i;\n- long double d;\n- void *p;\n-};\n-struct fooalign\n-{\n- char c;\n- union fooround u;\n-};\n+\n /* If malloc were really smart, it would round addresses to DEFAULT_ALIGNMENT.\n But in fact it might be less smart and round addresses to as much as\n- DEFAULT_ROUNDING. So we prepare for it to do that. */\n-enum\n- {\n- DEFAULT_ALIGNMENT = offsetof (struct fooalign, u),\n- DEFAULT_ROUNDING = sizeof (union fooround)\n- };\n-\n-/* When we copy a long block of data, this is the unit to do it with.\n- On some machines, copying successive ints does not work;\n- in such a case, redefine COPYING_UNIT to `long' (if that works)\n- or `char' as a last resort. */\n-# ifndef COPYING_UNIT\n-# define COPYING_UNIT int\n-# endif\n+ DEFAULT_ROUNDING. So we prepare for it to do that.\n+\n+ DEFAULT_ALIGNMENT cannot be an enum constant; see gnulib's alignof.h. */\n+#define DEFAULT_ALIGNMENT MAX (__alignof__ (long double),\t\t \\\n+ MAX (__alignof__ (uintmax_t),\t\t \\\n+ __alignof__ (void *)))\n+#define DEFAULT_ROUNDING MAX (sizeof (long double),\t\t\t \\\n+ MAX (sizeof (uintmax_t),\t\t\t \\\n+ sizeof (void *)))\n+\n+/* Call functions with either the traditional malloc/free calling\n+ interface, or the mmalloc/mfree interface (that adds an extra first\n+ argument), based on the value of use_extra_arg. */\n+\n+static void *\n+call_chunkfun (struct obstack *h, size_t size)\n+{\n+ if (h->use_extra_arg)\n+ return h->chunkfun.extra (h->extra_arg, size);\n+ else\n+ return h->chunkfun.plain (size);\n+}\n \n+static void\n+call_freefun (struct obstack *h, void *old_chunk)\n+{\n+ if (h->use_extra_arg)\n+ h->freefun.extra (h->extra_arg, old_chunk);\n+ else\n+ h->freefun.plain (old_chunk);\n+}\n \n-/* The functions allocating more room by calling `obstack_chunk_alloc'\n- jump to the handler pointed to by `obstack_alloc_failed_handler'.\n- This can be set to a user defined function which should either\n- abort gracefully or use longjump - but shouldn't return. This\n- variable by default points to the internal function\n- `print_and_abort'. */\n-static void print_and_abort (void);\n-void (*obstack_alloc_failed_handler) (void) = print_and_abort;\n-\n-# ifdef _LIBC\n-# if SHLIB_COMPAT (libc, GLIBC_2_0, GLIBC_2_3_4)\n-/* A looong time ago (before 1994, anyway; we're not sure) this global variable\n- was used by non-GNU-C macros to avoid multiple evaluation. The GNU C\n- library still exports it because somebody might use it. */\n-struct obstack *_obstack_compat;\n-compat_symbol (libc, _obstack_compat, _obstack, GLIBC_2_0);\n-# endif\n-# endif\n \n-/* Define a macro that either calls functions with the traditional malloc/free\n- calling interface, or calls functions with the mmalloc/mfree interface\n- (that adds an extra first argument), based on the state of use_extra_arg.\n- For free, do not use ?:, since some compilers, like the MIPS compilers,\n- do not allow (expr) ? void : void. */\n-\n-# define CALL_CHUNKFUN(h, size) \\\n- (((h) -> use_extra_arg) \\\n- ? (*(h)->chunkfun.extra) ((h)->extra_arg, (size)) \\\n- : (*(h)->chunkfun.plain) ((size)))\n-\n-# define CALL_FREEFUN(h, old_chunk) \\\n- do { \\\n- if ((h) -> use_extra_arg) \\\n- (*(h)->freefun.extra) ((h)->extra_arg, (old_chunk)); \\\n- else \\\n- (*(h)->freefun.plain) ((old_chunk)); \\\n- } while (0)\n-\n-\f\n /* Initialize an obstack H for use. Specify chunk size SIZE (0 means default).\n Objects start on multiples of ALIGNMENT (0 means use default).\n- CHUNKFUN is the function to use to allocate chunks,\n- and FREEFUN the function to free them.\n \n Return nonzero if successful, calls obstack_alloc_failed_handler if\n allocation fails. */\n \n-int\n-_obstack_begin (struct obstack *h,\n-\t\tint size, int alignment,\n-\t\tvoid *(*chunkfun) (long),\n-\t\tvoid (*freefun) (void *))\n+static int\n+_obstack_begin_worker (struct obstack *h,\n+ _OBSTACK_SIZE_T size, _OBSTACK_SIZE_T alignment)\n {\n- register struct _obstack_chunk *chunk; /* points to new chunk */\n+ struct _obstack_chunk *chunk; /* points to new chunk */\n \n if (alignment == 0)\n alignment = DEFAULT_ALIGNMENT;\n@@ -146,33 +117,29 @@ _obstack_begin (struct obstack *h,\n /* Default size is what GNU malloc can fit in a 4096-byte block. */\n {\n /* 12 is sizeof (mhead) and 4 is EXTRA from GNU malloc.\n-\t Use the values for range checking, because if range checking is off,\n-\t the extra bytes won't be missed terribly, but if range checking is on\n-\t and we used a larger request, a whole extra 4096 bytes would be\n-\t allocated.\n+ Use the values for range checking, because if range checking is off,\n+ the extra bytes won't be missed terribly, but if range checking is on\n+ and we used a larger request, a whole extra 4096 bytes would be\n+ allocated.\n \n-\t These number are irrelevant to the new GNU malloc. I suspect it is\n-\t less sensitive to the size of the request. */\n+ These number are irrelevant to the new GNU malloc. I suspect it is\n+ less sensitive to the size of the request. */\n int extra = ((((12 + DEFAULT_ROUNDING - 1) & ~(DEFAULT_ROUNDING - 1))\n-\t\t + 4 + DEFAULT_ROUNDING - 1)\n-\t\t & ~(DEFAULT_ROUNDING - 1));\n+ + 4 + DEFAULT_ROUNDING - 1)\n+ & ~(DEFAULT_ROUNDING - 1));\n size = 4096 - extra;\n }\n \n- h->chunkfun.plain = chunkfun;\n- h->freefun.plain = freefun;\n h->chunk_size = size;\n h->alignment_mask = alignment - 1;\n- h->use_extra_arg = 0;\n \n- chunk = h->chunk = CALL_CHUNKFUN (h, h -> chunk_size);\n+ chunk = h->chunk = call_chunkfun (h, h->chunk_size);\n if (!chunk)\n (*obstack_alloc_failed_handler) ();\n h->next_free = h->object_base = __PTR_ALIGN ((char *) chunk, chunk->contents,\n-\t\t\t\t\t alignment - 1);\n- h->chunk_limit = chunk->limit\n- = (char *) chunk + h->chunk_size;\n- chunk->prev = NULL;\n+ alignment - 1);\n+ h->chunk_limit = chunk->limit = (char *) chunk + h->chunk_size;\n+ chunk->prev = 0;\n /* The initial chunk now contains no empty object. */\n h->maybe_empty_object = 0;\n h->alloc_failed = 0;\n@@ -180,52 +147,29 @@ _obstack_begin (struct obstack *h,\n }\n \n int\n-_obstack_begin_1 (struct obstack *h, int size, int alignment,\n-\t\t void *(*chunkfun) (void *, long),\n-\t\t void (*freefun) (void *, void *),\n-\t\t void *arg)\n+_obstack_begin (struct obstack *h,\n+ _OBSTACK_SIZE_T size, _OBSTACK_SIZE_T alignment,\n+ void *(*chunkfun) (size_t),\n+ void (*freefun) (void *))\n {\n- register struct _obstack_chunk *chunk; /* points to new chunk */\n-\n- if (alignment == 0)\n- alignment = DEFAULT_ALIGNMENT;\n- if (size == 0)\n- /* Default size is what GNU malloc can fit in a 4096-byte block. */\n- {\n- /* 12 is sizeof (mhead) and 4 is EXTRA from GNU malloc.\n-\t Use the values for range checking, because if range checking is off,\n-\t the extra bytes won't be missed terribly, but if range checking is on\n-\t and we used a larger request, a whole extra 4096 bytes would be\n-\t allocated.\n-\n-\t These number are irrelevant to the new GNU malloc. I suspect it is\n-\t less sensitive to the size of the request. */\n- int extra = ((((12 + DEFAULT_ROUNDING - 1) & ~(DEFAULT_ROUNDING - 1))\n-\t\t + 4 + DEFAULT_ROUNDING - 1)\n-\t\t & ~(DEFAULT_ROUNDING - 1));\n- size = 4096 - extra;\n- }\n-\n- h->chunkfun.extra = (struct _obstack_chunk * (*)(void *,long)) chunkfun;\n- h->freefun.extra = (void (*) (void *, struct _obstack_chunk *)) freefun;\n+ h->chunkfun.plain = chunkfun;\n+ h->freefun.plain = freefun;\n+ h->use_extra_arg = 0;\n+ return _obstack_begin_worker (h, size, alignment);\n+}\n \n- h->chunk_size = size;\n- h->alignment_mask = alignment - 1;\n+int\n+_obstack_begin_1 (struct obstack *h,\n+ _OBSTACK_SIZE_T size, _OBSTACK_SIZE_T alignment,\n+ void *(*chunkfun) (void *, size_t),\n+ void (*freefun) (void *, void *),\n+ void *arg)\n+{\n+ h->chunkfun.extra = chunkfun;\n+ h->freefun.extra = freefun;\n h->extra_arg = arg;\n h->use_extra_arg = 1;\n-\n- chunk = h->chunk = CALL_CHUNKFUN (h, h -> chunk_size);\n- if (!chunk)\n- (*obstack_alloc_failed_handler) ();\n- h->next_free = h->object_base = __PTR_ALIGN ((char *) chunk, chunk->contents,\n-\t\t\t\t\t alignment - 1);\n- h->chunk_limit = chunk->limit\n- = (char *) chunk + h->chunk_size;\n- chunk->prev = NULL;\n- /* The initial chunk now contains no empty object. */\n- h->maybe_empty_object = 0;\n- h->alloc_failed = 0;\n- return 1;\n+ return _obstack_begin_worker (h, size, alignment);\n }\n \n /* Allocate a new current chunk for the obstack *H\n@@ -235,25 +179,27 @@ _obstack_begin_1 (struct obstack *h, int size, int alignment,\n to the beginning of the new one. */\n \n void\n-_obstack_newchunk (struct obstack *h, int length)\n+_obstack_newchunk (struct obstack *h, _OBSTACK_SIZE_T length)\n {\n- register struct _obstack_chunk *old_chunk = h->chunk;\n- register struct _obstack_chunk *new_chunk;\n- register long\tnew_size;\n- register long obj_size = h->next_free - h->object_base;\n- register long i;\n- long already;\n+ struct _obstack_chunk *old_chunk = h->chunk;\n+ struct _obstack_chunk *new_chunk = 0;\n+ size_t obj_size = h->next_free - h->object_base;\n char *object_base;\n \n /* Compute size for new chunk. */\n- new_size = (obj_size + length) + (obj_size >> 3) + h->alignment_mask + 100;\n+ size_t sum1 = obj_size + length;\n+ size_t sum2 = sum1 + h->alignment_mask;\n+ size_t new_size = sum2 + (obj_size >> 3) + 100;\n+ if (new_size < sum2)\n+ new_size = sum2;\n if (new_size < h->chunk_size)\n new_size = h->chunk_size;\n \n /* Allocate and initialize the new chunk. */\n- new_chunk = CALL_CHUNKFUN (h, new_size);\n+ if (obj_size <= sum1 && sum1 <= sum2)\n+ new_chunk = call_chunkfun (h, new_size);\n if (!new_chunk)\n- (*obstack_alloc_failed_handler) ();\n+ (*obstack_alloc_failed_handler)();\n h->chunk = new_chunk;\n new_chunk->prev = old_chunk;\n new_chunk->limit = h->chunk_limit = (char *) new_chunk + new_size;\n@@ -262,36 +208,19 @@ _obstack_newchunk (struct obstack *h, int length)\n object_base =\n __PTR_ALIGN ((char *) new_chunk, new_chunk->contents, h->alignment_mask);\n \n- /* Move the existing object to the new chunk.\n- Word at a time is fast and is safe if the object\n- is sufficiently aligned. */\n- if (h->alignment_mask + 1 >= DEFAULT_ALIGNMENT)\n- {\n- for (i = obj_size / sizeof (COPYING_UNIT) - 1;\n-\t i >= 0; i--)\n-\t((COPYING_UNIT *)object_base)[i]\n-\t = ((COPYING_UNIT *)h->object_base)[i];\n- /* We used to copy the odd few remaining bytes as one extra COPYING_UNIT,\n-\t but that can cross a page boundary on a machine\n-\t which does not do strict alignment for COPYING_UNITS. */\n- already = obj_size / sizeof (COPYING_UNIT) * sizeof (COPYING_UNIT);\n- }\n- else\n- already = 0;\n- /* Copy remaining bytes one by one. */\n- for (i = already; i < obj_size; i++)\n- object_base[i] = h->object_base[i];\n+ /* Move the existing object to the new chunk. */\n+ memcpy (object_base, h->object_base, obj_size);\n \n /* If the object just copied was the only data in OLD_CHUNK,\n free that chunk and remove it from the chain.\n But not if that chunk might contain an empty object. */\n- if (! h->maybe_empty_object\n+ if (!h->maybe_empty_object\n && (h->object_base\n-\t == __PTR_ALIGN ((char *) old_chunk, old_chunk->contents,\n-\t\t\t h->alignment_mask)))\n+ == __PTR_ALIGN ((char *) old_chunk, old_chunk->contents,\n+ h->alignment_mask)))\n {\n new_chunk->prev = old_chunk->prev;\n- CALL_FREEFUN (h, old_chunk);\n+ call_freefun (h, old_chunk);\n }\n \n h->object_base = object_base;\n@@ -299,9 +228,6 @@ _obstack_newchunk (struct obstack *h, int length)\n /* The new chunk certainly contains no empty object yet. */\n h->maybe_empty_object = 0;\n }\n-# ifdef _LIBC\n-libc_hidden_def (_obstack_newchunk)\n-# endif\n \n /* Return nonzero if object OBJ has been allocated from obstack H.\n This is here for debugging.\n@@ -309,48 +235,46 @@ libc_hidden_def (_obstack_newchunk)\n \n /* Suppress -Wmissing-prototypes warning. We don't want to declare this in\n obstack.h because it is just for debugging. */\n-int _obstack_allocated_p (struct obstack *h, void *obj);\n+int _obstack_allocated_p (struct obstack *h, void *obj) __attribute_pure__;\n \n int\n _obstack_allocated_p (struct obstack *h, void *obj)\n {\n- register struct _obstack_chunk *lp;\t/* below addr of any objects in this chunk */\n- register struct _obstack_chunk *plp;\t/* point to previous chunk if any */\n+ struct _obstack_chunk *lp; /* below addr of any objects in this chunk */\n+ struct _obstack_chunk *plp; /* point to previous chunk if any */\n \n lp = (h)->chunk;\n /* We use >= rather than > since the object cannot be exactly at\n the beginning of the chunk but might be an empty object exactly\n at the end of an adjacent chunk. */\n- while (lp != NULL && ((void *) lp >= obj || (void *) (lp)->limit < obj))\n+ while (lp != 0 && ((void *) lp >= obj || (void *) (lp)->limit < obj))\n {\n plp = lp->prev;\n lp = plp;\n }\n- return lp != NULL;\n+ return lp != 0;\n }\n-\f\n+\n /* Free objects in obstack H, including OBJ and everything allocate\n more recently than OBJ. If OBJ is zero, free everything in H. */\n \n-# undef obstack_free\n-\n void\n-obstack_free (struct obstack *h, void *obj)\n+_obstack_free (struct obstack *h, void *obj)\n {\n- register struct _obstack_chunk *lp;\t/* below addr of any objects in this chunk */\n- register struct _obstack_chunk *plp;\t/* point to previous chunk if any */\n+ struct _obstack_chunk *lp; /* below addr of any objects in this chunk */\n+ struct _obstack_chunk *plp; /* point to previous chunk if any */\n \n lp = h->chunk;\n /* We use >= because there cannot be an object at the beginning of a chunk.\n But there can be an empty object at that address\n at the end of another chunk. */\n- while (lp != NULL && ((void *) lp >= obj || (void *) (lp)->limit < obj))\n+ while (lp != 0 && ((void *) lp >= obj || (void *) (lp)->limit < obj))\n {\n plp = lp->prev;\n- CALL_FREEFUN (h, lp);\n+ call_freefun (h, lp);\n lp = plp;\n /* If we switch chunks, we can't tell whether the new current\n-\t chunk contains an empty object, so assume that it may. */\n+ chunk contains an empty object, so assume that it may. */\n h->maybe_empty_object = 1;\n }\n if (lp)\n@@ -359,42 +283,50 @@ obstack_free (struct obstack *h, void *obj)\n h->chunk_limit = lp->limit;\n h->chunk = lp;\n }\n- else if (obj != NULL)\n+ else if (obj != 0)\n /* obj is not in any of the chunks! */\n abort ();\n }\n \n-# ifdef _LIBC\n-/* Older versions of libc used a function _obstack_free intended to be\n- called by non-GCC compilers. */\n-strong_alias (obstack_free, _obstack_free)\n-# endif\n-\f\n-int\n+_OBSTACK_SIZE_T\n _obstack_memory_used (struct obstack *h)\n {\n- register struct _obstack_chunk* lp;\n- register int nbytes = 0;\n+ struct _obstack_chunk *lp;\n+ _OBSTACK_SIZE_T nbytes = 0;\n \n- for (lp = h->chunk; lp != NULL; lp = lp->prev)\n+ for (lp = h->chunk; lp != 0; lp = lp->prev)\n {\n nbytes += lp->limit - (char *) lp;\n }\n return nbytes;\n }\n-\f\n-# ifdef _LIBC\n-# include \n-# endif\n \n-# ifndef __attribute__\n-/* This feature is available in gcc versions 2.5 and later. */\n-# if __GNUC__ < 2 || (__GNUC__ == 2 && __GNUC_MINOR__ < 5)\n-# define __attribute__(Spec) /* empty */\n+# ifndef _OBSTACK_NO_ERROR_HANDLER\n+/* Define the error handler. */\n+# include \n+\n+/* Exit value used when 'print_and_abort' is used. */\n+# ifdef _LIBC\n+int obstack_exit_failure = EXIT_FAILURE;\n+# else\n+# include \"exitfail.h\"\n+# define obstack_exit_failure exit_failure\n # endif\n-# endif\n \n-static void\n+# ifdef _LIBC\n+# include \n+# else\n+# include \"gettext.h\"\n+# endif\n+# ifndef _\n+# define _(msgid) gettext (msgid)\n+# endif\n+\n+# ifdef _LIBC\n+# include \n+# endif\n+\n+static _Noreturn void\n print_and_abort (void)\n {\n /* Don't change any of these strings. Yes, it would be possible to add\n@@ -402,12 +334,21 @@ print_and_abort (void)\n happen because the \"memory exhausted\" message appears in other places\n like this and the translation should be reused instead of creating\n a very similar string which requires a separate translation. */\n-# ifdef _LIBC\n+# ifdef _LIBC\n (void) __fxprintf (NULL, \"%s\\n\", _(\"memory exhausted\"));\n-# else\n+# else\n fprintf (stderr, \"%s\\n\", _(\"memory exhausted\"));\n-# endif\n- exit (1);\n+# endif\n+ exit (obstack_exit_failure);\n }\n \n-#endif\t/* !ELIDE_CODE */\n+/* The functions allocating more room by calling 'obstack_chunk_alloc'\n+ jump to the handler pointed to by 'obstack_alloc_failed_handler'.\n+ This can be set to a user defined function which should either\n+ abort gracefully or use longjump - but shouldn't return. This\n+ variable by default points to the internal function\n+ 'print_and_abort'. */\n+__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)\n+ = print_and_abort;\n+# endif /* !_OBSTACK_NO_ERROR_HANDLER */\n+#endif /* !_OBSTACK_ELIDE_CODE */\ndiff --git a/compat/obstack.h b/compat/obstack.h\nindex ced94d0118..811de588a4 100644\n--- a/compat/obstack.h\n+++ b/compat/obstack.h\n@@ -1,6 +1,5 @@\n /* obstack.h - object stack macros\n- Copyright (C) 1988-1994,1996-1999,2003,2004,2005,2009\n-\tFree Software Foundation, Inc.\n+ Copyright (C) 1988-2019 Free Software Foundation, Inc.\n This file is part of the GNU C Library.\n \n The GNU C Library is free software; you can redistribute it and/or\n@@ -15,89 +14,89 @@\n \n You should have received a copy of the GNU Lesser General Public\n License along with the GNU C Library; if not, see\n- . */\n+ . */\n \n /* Summary:\n \n-All the apparent functions defined here are macros. The idea\n-is that you would use these pre-tested macros to solve a\n-very specific set of problems, and they would run fast.\n-Caution: no side-effects in arguments please!! They may be\n-evaluated MANY times!!\n-\n-These macros operate a stack of objects. Each object starts life\n-small, and may grow to maturity. (Consider building a word syllable\n-by syllable.) An object can move while it is growing. Once it has\n-been \"finished\" it never changes address again. So the \"top of the\n-stack\" is typically an immature growing object, while the rest of the\n-stack is of mature, fixed size and fixed address objects.\n-\n-These routines grab large chunks of memory, using a function you\n-supply, called `obstack_chunk_alloc'. On occasion, they free chunks,\n-by calling `obstack_chunk_free'. You must define them and declare\n-them before using any obstack macros.\n-\n-Each independent stack is represented by a `struct obstack'.\n-Each of the obstack macros expects a pointer to such a structure\n-as the first argument.\n-\n-One motivation for this package is the problem of growing char strings\n-in symbol tables. Unless you are \"fascist pig with a read-only mind\"\n---Gosper's immortal quote from HAKMEM item 154, out of context--you\n-would not like to put any arbitrary upper limit on the length of your\n-symbols.\n-\n-In practice this often means you will build many short symbols and a\n-few long symbols. At the time you are reading a symbol you don't know\n-how long it is. One traditional method is to read a symbol into a\n-buffer, realloc()ating the buffer every time you try to read a symbol\n-that is longer than the buffer. This is beaut, but you still will\n-want to copy the symbol from the buffer to a more permanent\n-symbol-table entry say about half the time.\n-\n-With obstacks, you can work differently. Use one obstack for all symbol\n-names. As you read a symbol, grow the name in the obstack gradually.\n-When the name is complete, finalize it. Then, if the symbol exists already,\n-free the newly read name.\n-\n-The way we do this is to take a large chunk, allocating memory from\n-low addresses. When you want to build a symbol in the chunk you just\n-add chars above the current \"high water mark\" in the chunk. When you\n-have finished adding chars, because you got to the end of the symbol,\n-you know how long the chars are, and you can create a new object.\n-Mostly the chars will not burst over the highest address of the chunk,\n-because you would typically expect a chunk to be (say) 100 times as\n-long as an average object.\n-\n-In case that isn't clear, when we have enough chars to make up\n-the object, THEY ARE ALREADY CONTIGUOUS IN THE CHUNK (guaranteed)\n-so we just point to it where it lies. No moving of chars is\n-needed and this is the second win: potentially long strings need\n-never be explicitly shuffled. Once an object is formed, it does not\n-change its address during its lifetime.\n-\n-When the chars burst over a chunk boundary, we allocate a larger\n-chunk, and then copy the partly formed object from the end of the old\n-chunk to the beginning of the new larger chunk. We then carry on\n-accreting characters to the end of the object as we normally would.\n-\n-A special macro is provided to add a single char at a time to a\n-growing object. This allows the use of register variables, which\n-break the ordinary 'growth' macro.\n-\n-Summary:\n-\tWe allocate large chunks.\n-\tWe carve out one object at a time from the current chunk.\n-\tOnce carved, an object never moves.\n-\tWe are free to append data of any size to the currently\n-\t growing object.\n-\tExactly one object is growing in an obstack at any one time.\n-\tYou can run one obstack per control block.\n-\tYou may have as many control blocks as you dare.\n-\tBecause of the way we do it, you can `unwind' an obstack\n-\t back to a previous state. (You may remove objects much\n-\t as you would with a stack.)\n-*/\n+ All the apparent functions defined here are macros. The idea\n+ is that you would use these pre-tested macros to solve a\n+ very specific set of problems, and they would run fast.\n+ Caution: no side-effects in arguments please!! They may be\n+ evaluated MANY times!!\n+\n+ These macros operate a stack of objects. Each object starts life\n+ small, and may grow to maturity. (Consider building a word syllable\n+ by syllable.) An object can move while it is growing. Once it has\n+ been \"finished\" it never changes address again. So the \"top of the\n+ stack\" is typically an immature growing object, while the rest of the\n+ stack is of mature, fixed size and fixed address objects.\n+\n+ These routines grab large chunks of memory, using a function you\n+ supply, called 'obstack_chunk_alloc'. On occasion, they free chunks,\n+ by calling 'obstack_chunk_free'. You must define them and declare\n+ them before using any obstack macros.\n+\n+ Each independent stack is represented by a 'struct obstack'.\n+ Each of the obstack macros expects a pointer to such a structure\n+ as the first argument.\n+\n+ One motivation for this package is the problem of growing char strings\n+ in symbol tables. Unless you are \"fascist pig with a read-only mind\"\n+ --Gosper's immortal quote from HAKMEM item 154, out of context--you\n+ would not like to put any arbitrary upper limit on the length of your\n+ symbols.\n+\n+ In practice this often means you will build many short symbols and a\n+ few long symbols. At the time you are reading a symbol you don't know\n+ how long it is. One traditional method is to read a symbol into a\n+ buffer, realloc()ating the buffer every time you try to read a symbol\n+ that is longer than the buffer. This is beaut, but you still will\n+ want to copy the symbol from the buffer to a more permanent\n+ symbol-table entry say about half the time.\n+\n+ With obstacks, you can work differently. Use one obstack for all symbol\n+ names. As you read a symbol, grow the name in the obstack gradually.\n+ When the name is complete, finalize it. Then, if the symbol exists already,\n+ free the newly read name.\n+\n+ The way we do this is to take a large chunk, allocating memory from\n+ low addresses. When you want to build a symbol in the chunk you just\n+ add chars above the current \"high water mark\" in the chunk. When you\n+ have finished adding chars, because you got to the end of the symbol,\n+ you know how long the chars are, and you can create a new object.\n+ Mostly the chars will not burst over the highest address of the chunk,\n+ because you would typically expect a chunk to be (say) 100 times as\n+ long as an average object.\n+\n+ In case that isn't clear, when we have enough chars to make up\n+ the object, THEY ARE ALREADY CONTIGUOUS IN THE CHUNK (guaranteed)\n+ so we just point to it where it lies. No moving of chars is\n+ needed and this is the second win: potentially long strings need\n+ never be explicitly shuffled. Once an object is formed, it does not\n+ change its address during its lifetime.\n+\n+ When the chars burst over a chunk boundary, we allocate a larger\n+ chunk, and then copy the partly formed object from the end of the old\n+ chunk to the beginning of the new larger chunk. We then carry on\n+ accreting characters to the end of the object as we normally would.\n+\n+ A special macro is provided to add a single char at a time to a\n+ growing object. This allows the use of register variables, which\n+ break the ordinary 'growth' macro.\n+\n+ Summary:\n+ We allocate large chunks.\n+ We carve out one object at a time from the current chunk.\n+ Once carved, an object never moves.\n+ We are free to append data of any size to the currently\n+ growing object.\n+ Exactly one object is growing in an obstack at any one time.\n+ You can run one obstack per control block.\n+ You may have as many control blocks as you dare.\n+ Because of the way we do it, you can \"unwind\" an obstack\n+ back to a previous state. (You may remove objects much\n+ as you would with a stack.)\n+ */\n \n \n /* Don't do the contents of this file more than once. */\n@@ -105,20 +104,30 @@ break the ordinary 'growth' macro.\n #ifndef _OBSTACK_H\n #define _OBSTACK_H 1\n \n-#ifdef __cplusplus\n-extern \"C\" {\n+#ifndef _OBSTACK_INTERFACE_VERSION\n+# define _OBSTACK_INTERFACE_VERSION 2\n+#endif\n+\n+#include /* For size_t and ptrdiff_t. */\n+#include /* For __GNU_LIBRARY__, and memcpy. */\n+\n+#if __STDC_VERSION__ < 199901L || defined __HP_cc\n+# define __FLEXIBLE_ARRAY_MEMBER 1\n+#else\n+# define __FLEXIBLE_ARRAY_MEMBER\n #endif\n-\f\n-/* We need the type of a pointer subtraction. If __PTRDIFF_TYPE__ is\n- defined, as with GNU C, use that; that way we don't pollute the\n- namespace with 's symbols. Otherwise, include \n- and use ptrdiff_t. */\n-\n-#ifdef __PTRDIFF_TYPE__\n-# define PTR_INT_TYPE __PTRDIFF_TYPE__\n+\n+#if _OBSTACK_INTERFACE_VERSION == 1\n+/* For binary compatibility with obstack version 1, which used \"int\"\n+ and \"long\" for these two types. */\n+# define _OBSTACK_SIZE_T unsigned int\n+# define _CHUNK_SIZE_T unsigned long\n+# define _OBSTACK_CAST(type, expr) ((type) (expr))\n #else\n-# include \n-# define PTR_INT_TYPE ptrdiff_t\n+/* Version 2 with sane types, especially for 64-bit hosts. */\n+# define _OBSTACK_SIZE_T size_t\n+# define _CHUNK_SIZE_T size_t\n+# define _OBSTACK_CAST(type, expr) (expr)\n #endif\n \n /* If B is the base of an object addressed by P, return the result of\n@@ -127,78 +136,102 @@ extern \"C\" {\n \n #define __BPTR_ALIGN(B, P, A) ((B) + (((P) - (B) + (A)) & ~(A)))\n \n-/* Similar to _BPTR_ALIGN (B, P, A), except optimize the common case\n+/* Similar to __BPTR_ALIGN (B, P, A), except optimize the common case\n where pointers can be converted to integers, aligned as integers,\n- and converted back again. If PTR_INT_TYPE is narrower than a\n+ and converted back again. If ptrdiff_t is narrower than a\n pointer (e.g., the AS/400), play it safe and compute the alignment\n relative to B. Otherwise, use the faster strategy of computing the\n alignment relative to 0. */\n \n-#define __PTR_ALIGN(B, P, A)\t\t\t\t\t\t \\\n- __BPTR_ALIGN (sizeof (PTR_INT_TYPE) < sizeof (void *) ? (B) : (char *) 0, \\\n-\t\tP, A)\n+#define __PTR_ALIGN(B, P, A)\t\t\t\t\t\t \\\n+ __BPTR_ALIGN (sizeof (ptrdiff_t) < sizeof (void *) ? (B) : (char *) 0, \\\n+ P, A)\n+\n+#ifndef __attribute_pure__\n+# define __attribute_pure__ _GL_ATTRIBUTE_PURE\n+#endif\n+\n+/* Not the same as _Noreturn, since it also works with function pointers. */\n+#ifndef __attribute_noreturn__\n+# if 2 < __GNUC__ + (8 <= __GNUC_MINOR__) || 0x5110 <= __SUNPRO_C\n+# define __attribute_noreturn__ __attribute__ ((__noreturn__))\n+# else\n+# define __attribute_noreturn__\n+# endif\n+#endif\n \n-#include \n+#ifdef __cplusplus\n+extern \"C\" {\n+#endif\n \n-struct _obstack_chunk\t\t/* Lives at front of each chunk. */\n+struct _obstack_chunk /* Lives at front of each chunk. */\n {\n- char *limit;\t\t\t/* 1 past end of this chunk */\n- struct _obstack_chunk *prev;\t/* address of prior chunk or NULL */\n- char\tcontents[4];\t\t/* objects begin here */\n+ char *limit; /* 1 past end of this chunk */\n+ struct _obstack_chunk *prev; /* address of prior chunk or NULL */\n+ char contents[__FLEXIBLE_ARRAY_MEMBER]; /* objects begin here */\n };\n \n-struct obstack\t\t/* control current object in current chunk */\n+struct obstack /* control current object in current chunk */\n {\n- long\tchunk_size;\t\t/* preferred size to allocate chunks in */\n- struct _obstack_chunk *chunk;\t/* address of current struct obstack_chunk */\n- char\t*object_base;\t\t/* address of object we are building */\n- char\t*next_free;\t\t/* where to add next char to current object */\n- char\t*chunk_limit;\t\t/* address of char after current chunk */\n+ _CHUNK_SIZE_T chunk_size; /* preferred size to allocate chunks in */\n+ struct _obstack_chunk *chunk; /* address of current struct obstack_chunk */\n+ char *object_base; /* address of object we are building */\n+ char *next_free; /* where to add next char to current object */\n+ char *chunk_limit; /* address of char after current chunk */\n+ union\n+ {\n+ _OBSTACK_SIZE_T i;\n+ void *p;\n+ } temp; /* Temporary for some macros. */\n+ _OBSTACK_SIZE_T alignment_mask; /* Mask of alignment for each object. */\n+\n+ /* These prototypes vary based on 'use_extra_arg'. */\n union\n {\n- PTR_INT_TYPE tempint;\n- void *tempptr;\n- } temp;\t\t\t/* Temporary for some macros. */\n- int alignment_mask;\t\t/* Mask of alignment for each object. */\n- /* These prototypes vary based on `use_extra_arg'. */\n- union {\n- void *(*plain) (long);\n- struct _obstack_chunk *(*extra) (void *, long);\n+ void *(*plain) (size_t);\n+ void *(*extra) (void *, size_t);\n } chunkfun;\n- union {\n+ union\n+ {\n void (*plain) (void *);\n- void (*extra) (void *, struct _obstack_chunk *);\n+ void (*extra) (void *, void *);\n } freefun;\n- void *extra_arg;\t\t/* first arg for chunk alloc/dealloc funcs */\n- unsigned use_extra_arg:1;\t/* chunk alloc/dealloc funcs take extra arg */\n- unsigned maybe_empty_object:1;/* There is a possibility that the current\n-\t\t\t\t chunk contains a zero-length object. This\n-\t\t\t\t prevents freeing the chunk if we allocate\n-\t\t\t\t a bigger chunk to replace it. */\n- unsigned alloc_failed:1;\t/* No longer used, as we now call the failed\n-\t\t\t\t handler on error, but retained for binary\n-\t\t\t\t compatibility. */\n+\n+ void *extra_arg; /* first arg for chunk alloc/dealloc funcs */\n+ unsigned use_extra_arg : 1; /* chunk alloc/dealloc funcs take extra arg */\n+ unsigned maybe_empty_object : 1; /* There is a possibility that the current\n+ chunk contains a zero-length object. This\n+ prevents freeing the chunk if we allocate\n+ a bigger chunk to replace it. */\n+ unsigned alloc_failed : 1; /* No longer used, as we now call the failed\n+ handler on error, but retained for binary\n+ compatibility. */\n };\n \n /* Declare the external functions we use; they are in obstack.c. */\n \n-extern void _obstack_newchunk (struct obstack *, int);\n-extern int _obstack_begin (struct obstack *, int, int,\n-\t\t\t void *(*) (long), void (*) (void *));\n-extern int _obstack_begin_1 (struct obstack *, int, int,\n-\t\t\t void *(*) (void *, long),\n-\t\t\t void (*) (void *, void *), void *);\n-extern int _obstack_memory_used (struct obstack *);\n+extern void _obstack_newchunk (struct obstack *, _OBSTACK_SIZE_T);\n+extern void _obstack_free (struct obstack *, void *);\n+extern int _obstack_begin (struct obstack *,\n+ _OBSTACK_SIZE_T, _OBSTACK_SIZE_T,\n+ void *(*) (size_t), void (*) (void *));\n+extern int _obstack_begin_1 (struct obstack *,\n+ _OBSTACK_SIZE_T, _OBSTACK_SIZE_T,\n+ void *(*) (void *, size_t),\n+ void (*) (void *, void *), void *);\n+extern _OBSTACK_SIZE_T _obstack_memory_used (struct obstack *)\n+ __attribute_pure__;\n \n-void obstack_free (struct obstack *, void *);\n \n-\f\n-/* Error handler called when `obstack_chunk_alloc' failed to allocate\n+/* Error handler called when 'obstack_chunk_alloc' failed to allocate\n more memory. This can be set to a user defined function which\n should either abort gracefully or use longjump - but shouldn't\n return. The default action is to print a message and abort. */\n-extern void (*obstack_alloc_failed_handler) (void);\n-\f\n+extern __attribute_noreturn__ void (*obstack_alloc_failed_handler) (void);\n+\n+/* Exit value used when 'print_and_abort' is used. */\n+extern int obstack_exit_failure;\n+\n /* Pointer to beginning of object being allocated or to be allocated next.\n Note that this might not be the final address of the object\n because a new chunk might be needed to hold the final size. */\n@@ -211,210 +244,210 @@ extern void (*obstack_alloc_failed_handler) (void);\n \n /* Pointer to next byte not yet allocated in current chunk. */\n \n-#define obstack_next_free(h)\t((h)->next_free)\n+#define obstack_next_free(h) ((void *) (h)->next_free)\n \n /* Mask specifying low bits that should be clear in address of an object. */\n \n #define obstack_alignment_mask(h) ((h)->alignment_mask)\n \n /* To prevent prototype warnings provide complete argument list. */\n-#define obstack_init(h)\t\t\t\t\t\t\\\n- _obstack_begin ((h), 0, 0,\t\t\t\t\t\\\n-\t\t (void *(*) (long)) obstack_chunk_alloc,\t\\\n-\t\t (void (*) (void *)) obstack_chunk_free)\n+#define obstack_init(h)\t\t\t\t\t\t\t \\\n+ _obstack_begin ((h), 0, 0,\t\t\t\t\t\t \\\n+ _OBSTACK_CAST (void *(*) (size_t), obstack_chunk_alloc), \\\n+ _OBSTACK_CAST (void (*) (void *), obstack_chunk_free))\n \n-#define obstack_begin(h, size)\t\t\t\t\t\\\n- _obstack_begin ((h), (size), 0,\t\t\t\t\\\n-\t\t (void *(*) (long)) obstack_chunk_alloc,\t\\\n-\t\t (void (*) (void *)) obstack_chunk_free)\n+#define obstack_begin(h, size)\t\t\t\t\t\t \\\n+ _obstack_begin ((h), (size), 0,\t\t\t\t\t \\\n+ _OBSTACK_CAST (void *(*) (size_t), obstack_chunk_alloc), \\\n+ _OBSTACK_CAST (void (*) (void *), obstack_chunk_free))\n \n-#define obstack_specify_allocation(h, size, alignment, chunkfun, freefun) \\\n- _obstack_begin ((h), (size), (alignment),\t\t\t\t \\\n-\t\t (void *(*) (long)) (chunkfun),\t\t\t \\\n-\t\t (void (*) (void *)) (freefun))\n+#define obstack_specify_allocation(h, size, alignment, chunkfun, freefun) \\\n+ _obstack_begin ((h), (size), (alignment),\t\t\t\t \\\n+ _OBSTACK_CAST (void *(*) (size_t), chunkfun),\t\t \\\n+ _OBSTACK_CAST (void (*) (void *), freefun))\n \n #define obstack_specify_allocation_with_arg(h, size, alignment, chunkfun, freefun, arg) \\\n- _obstack_begin_1 ((h), (size), (alignment),\t\t\t\t\\\n-\t\t (void *(*) (void *, long)) (chunkfun),\t\t\\\n-\t\t (void (*) (void *, void *)) (freefun), (arg))\n+ _obstack_begin_1 ((h), (size), (alignment),\t\t\t\t \\\n+ _OBSTACK_CAST (void *(*) (void *, size_t), chunkfun), \\\n+ _OBSTACK_CAST (void (*) (void *, void *), freefun), arg)\n \n-#define obstack_chunkfun(h, newchunkfun) \\\n- ((h)->chunkfun.extra = (struct _obstack_chunk *(*)(void *, long)) (newchunkfun))\n+#define obstack_chunkfun(h, newchunkfun)\t\t\t\t \\\n+ ((void) ((h)->chunkfun.extra = (void *(*) (void *, size_t)) (newchunkfun)))\n \n-#define obstack_freefun(h, newfreefun) \\\n- ((h)->freefun.extra = (void (*)(void *, struct _obstack_chunk *)) (newfreefun))\n+#define obstack_freefun(h, newfreefun)\t\t\t\t\t \\\n+ ((void) ((h)->freefun.extra = (void *(*) (void *, void *)) (newfreefun)))\n \n-#define obstack_1grow_fast(h,achar) (*((h)->next_free)++ = (achar))\n+#define obstack_1grow_fast(h, achar) ((void) (*((h)->next_free)++ = (achar)))\n \n-#define obstack_blank_fast(h,n) ((h)->next_free += (n))\n+#define obstack_blank_fast(h, n) ((void) ((h)->next_free += (n)))\n \n #define obstack_memory_used(h) _obstack_memory_used (h)\n-\f\n-#if defined __GNUC__ && defined __STDC__ && __STDC__\n-/* NextStep 2.0 cc is really gcc 1.93 but it defines __GNUC__ = 2 and\n- does not implement __extension__. But that compiler doesn't define\n- __GNUC_MINOR__. */\n-# if __GNUC__ < 2 || (__NeXT__ && !__GNUC_MINOR__)\n+\n+#if defined __GNUC__\n+# if !defined __GNUC_MINOR__ || __GNUC__ * 1000 + __GNUC_MINOR__ < 2008\n # define __extension__\n # endif\n \n /* For GNU C, if not -traditional,\n we can define these macros to compute all args only once\n without using a global variable.\n- Also, we can avoid using the `temp' slot, to make faster code. */\n-\n-# define obstack_object_size(OBSTACK)\t\t\t\t\t\\\n- __extension__\t\t\t\t\t\t\t\t\\\n- ({ struct obstack const *__o = (OBSTACK);\t\t\t\t\\\n- (unsigned) (__o->next_free - __o->object_base); })\n-\n-# define obstack_room(OBSTACK)\t\t\t\t\t\t\\\n- __extension__\t\t\t\t\t\t\t\t\\\n- ({ struct obstack const *__o = (OBSTACK);\t\t\t\t\\\n- (unsigned) (__o->chunk_limit - __o->next_free); })\n-\n-# define obstack_make_room(OBSTACK,length)\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o = (OBSTACK);\t\t\t\t\t\\\n- int __len = (length);\t\t\t\t\t\t\\\n- if (__o->chunk_limit - __o->next_free < __len)\t\t\t\\\n- _obstack_newchunk (__o, __len);\t\t\t\t\t\\\n- (void) 0; })\n-\n-# define obstack_empty_p(OBSTACK)\t\t\t\t\t\\\n- __extension__\t\t\t\t\t\t\t\t\\\n- ({ struct obstack const *__o = (OBSTACK);\t\t\t\t\\\n- (__o->chunk->prev == 0\t\t\t\t\t\t\\\n- && __o->next_free == __PTR_ALIGN ((char *) __o->chunk,\t\t\\\n-\t\t\t\t\t__o->chunk->contents,\t\t\\\n-\t\t\t\t\t__o->alignment_mask)); })\n-\n-# define obstack_grow(OBSTACK,where,length)\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o = (OBSTACK);\t\t\t\t\t\\\n- int __len = (length);\t\t\t\t\t\t\\\n- if (__o->next_free + __len > __o->chunk_limit)\t\t\t\\\n- _obstack_newchunk (__o, __len);\t\t\t\t\t\\\n- memcpy (__o->next_free, where, __len);\t\t\t\t\\\n- __o->next_free += __len;\t\t\t\t\t\t\\\n- (void) 0; })\n-\n-# define obstack_grow0(OBSTACK,where,length)\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o = (OBSTACK);\t\t\t\t\t\\\n- int __len = (length);\t\t\t\t\t\t\\\n- if (__o->next_free + __len + 1 > __o->chunk_limit)\t\t\t\\\n- _obstack_newchunk (__o, __len + 1);\t\t\t\t\\\n- memcpy (__o->next_free, where, __len);\t\t\t\t\\\n- __o->next_free += __len;\t\t\t\t\t\t\\\n- *(__o->next_free)++ = 0;\t\t\t\t\t\t\\\n- (void) 0; })\n-\n-# define obstack_1grow(OBSTACK,datum)\t\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o = (OBSTACK);\t\t\t\t\t\\\n- if (__o->next_free + 1 > __o->chunk_limit)\t\t\t\t\\\n- _obstack_newchunk (__o, 1);\t\t\t\t\t\\\n- obstack_1grow_fast (__o, datum);\t\t\t\t\t\\\n- (void) 0; })\n+ Also, we can avoid using the 'temp' slot, to make faster code. */\n+\n+# define obstack_object_size(OBSTACK)\t\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack const *__o = (OBSTACK);\t\t\t\t \\\n+ (_OBSTACK_SIZE_T) (__o->next_free - __o->object_base); })\n+\n+/* The local variable is named __o1 to avoid a shadowed variable\n+ warning when invoked from other obstack macros. */\n+# define obstack_room(OBSTACK)\t\t\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack const *__o1 = (OBSTACK);\t\t\t\t \\\n+ (_OBSTACK_SIZE_T) (__o1->chunk_limit - __o1->next_free); })\n+\n+# define obstack_make_room(OBSTACK, length)\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o = (OBSTACK);\t\t\t\t\t \\\n+ _OBSTACK_SIZE_T __len = (length);\t\t\t\t \\\n+ if (obstack_room (__o) < __len)\t\t\t\t\t \\\n+ _obstack_newchunk (__o, __len);\t\t\t\t \\\n+ (void) 0; })\n+\n+# define obstack_empty_p(OBSTACK)\t\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack const *__o = (OBSTACK);\t\t\t\t \\\n+ (__o->chunk->prev == 0\t\t\t\t\t\t \\\n+ && __o->next_free == __PTR_ALIGN ((char *) __o->chunk,\t\t \\\n+ __o->chunk->contents,\t\t \\\n+ __o->alignment_mask)); })\n+\n+# define obstack_grow(OBSTACK, where, length)\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o = (OBSTACK);\t\t\t\t\t \\\n+ _OBSTACK_SIZE_T __len = (length);\t\t\t\t \\\n+ if (obstack_room (__o) < __len)\t\t\t\t\t \\\n+ _obstack_newchunk (__o, __len);\t\t\t\t \\\n+ memcpy (__o->next_free, where, __len);\t\t\t\t \\\n+ __o->next_free += __len;\t\t\t\t\t\t \\\n+ (void) 0; })\n+\n+# define obstack_grow0(OBSTACK, where, length)\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o = (OBSTACK);\t\t\t\t\t \\\n+ _OBSTACK_SIZE_T __len = (length);\t\t\t\t \\\n+ if (obstack_room (__o) < __len + 1)\t\t\t\t \\\n+ _obstack_newchunk (__o, __len + 1);\t\t\t\t \\\n+ memcpy (__o->next_free, where, __len);\t\t\t\t \\\n+ __o->next_free += __len;\t\t\t\t\t\t \\\n+ *(__o->next_free)++ = 0;\t\t\t\t\t\t \\\n+ (void) 0; })\n+\n+# define obstack_1grow(OBSTACK, datum)\t\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o = (OBSTACK);\t\t\t\t\t \\\n+ if (obstack_room (__o) < 1)\t\t\t\t\t \\\n+ _obstack_newchunk (__o, 1);\t\t\t\t\t \\\n+ obstack_1grow_fast (__o, datum); })\n \n /* These assume that the obstack alignment is good enough for pointers\n or ints, and that the data added so far to the current object\n shares that much alignment. */\n \n-# define obstack_ptr_grow(OBSTACK,datum)\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o = (OBSTACK);\t\t\t\t\t\\\n- if (__o->next_free + sizeof (void *) > __o->chunk_limit)\t\t\\\n- _obstack_newchunk (__o, sizeof (void *));\t\t\t\t\\\n- obstack_ptr_grow_fast (__o, datum); })\t\t\t\t\\\n-\n-# define obstack_int_grow(OBSTACK,datum)\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o = (OBSTACK);\t\t\t\t\t\\\n- if (__o->next_free + sizeof (int) > __o->chunk_limit)\t\t\\\n- _obstack_newchunk (__o, sizeof (int));\t\t\t\t\\\n- obstack_int_grow_fast (__o, datum); })\n-\n-# define obstack_ptr_grow_fast(OBSTACK,aptr)\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o1 = (OBSTACK);\t\t\t\t\t\\\n- *(const void **) __o1->next_free = (aptr);\t\t\t\t\\\n- __o1->next_free += sizeof (const void *);\t\t\t\t\\\n- (void) 0; })\n-\n-# define obstack_int_grow_fast(OBSTACK,aint)\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o1 = (OBSTACK);\t\t\t\t\t\\\n- *(int *) __o1->next_free = (aint);\t\t\t\t\t\\\n- __o1->next_free += sizeof (int);\t\t\t\t\t\\\n- (void) 0; })\n-\n-# define obstack_blank(OBSTACK,length)\t\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o = (OBSTACK);\t\t\t\t\t\\\n- int __len = (length);\t\t\t\t\t\t\\\n- if (__o->chunk_limit - __o->next_free < __len)\t\t\t\\\n- _obstack_newchunk (__o, __len);\t\t\t\t\t\\\n- obstack_blank_fast (__o, __len);\t\t\t\t\t\\\n- (void) 0; })\n-\n-# define obstack_alloc(OBSTACK,length)\t\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__h = (OBSTACK);\t\t\t\t\t\\\n- obstack_blank (__h, (length));\t\t\t\t\t\\\n- obstack_finish (__h); })\n-\n-# define obstack_copy(OBSTACK,where,length)\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__h = (OBSTACK);\t\t\t\t\t\\\n- obstack_grow (__h, (where), (length));\t\t\t\t\\\n- obstack_finish (__h); })\n-\n-# define obstack_copy0(OBSTACK,where,length)\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__h = (OBSTACK);\t\t\t\t\t\\\n- obstack_grow0 (__h, (where), (length));\t\t\t\t\\\n- obstack_finish (__h); })\n-\n-/* The local variable is named __o1 to avoid a name conflict\n- when obstack_blank is called. */\n-# define obstack_finish(OBSTACK)\t\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o1 = (OBSTACK);\t\t\t\t\t\\\n- void *__value = (void *) __o1->object_base;\t\t\t\t\\\n- if (__o1->next_free == __value)\t\t\t\t\t\\\n- __o1->maybe_empty_object = 1;\t\t\t\t\t\\\n- __o1->next_free\t\t\t\t\t\t\t\\\n- = __PTR_ALIGN (__o1->object_base, __o1->next_free,\t\t\t\\\n-\t\t __o1->alignment_mask);\t\t\t\t\\\n- if (__o1->next_free - (char *)__o1->chunk\t\t\t\t\\\n- > __o1->chunk_limit - (char *)__o1->chunk)\t\t\t\\\n- __o1->next_free = __o1->chunk_limit;\t\t\t\t\\\n- __o1->object_base = __o1->next_free;\t\t\t\t\t\\\n- __value; })\n-\n-# define obstack_free(OBSTACK, OBJ)\t\t\t\t\t\\\n-__extension__\t\t\t\t\t\t\t\t\\\n-({ struct obstack *__o = (OBSTACK);\t\t\t\t\t\\\n- void *__obj = (OBJ);\t\t\t\t\t\t\t\\\n- if (__obj > (void *)__o->chunk && __obj < (void *)__o->chunk_limit) \\\n- __o->next_free = __o->object_base = (char *)__obj;\t\t\t\\\n- else (obstack_free) (__o, __obj); })\n-\f\n-#else /* not __GNUC__ or not __STDC__ */\n-\n-# define obstack_object_size(h) \\\n- (unsigned) ((h)->next_free - (h)->object_base)\n-\n-# define obstack_room(h)\t\t\\\n- (unsigned) ((h)->chunk_limit - (h)->next_free)\n-\n-# define obstack_empty_p(h) \\\n- ((h)->chunk->prev == 0\t\t\t\t\t\t\t\\\n- && (h)->next_free == __PTR_ALIGN ((char *) (h)->chunk,\t\t\\\n-\t\t\t\t (h)->chunk->contents,\t\t\\\n-\t\t\t\t (h)->alignment_mask))\n+# define obstack_ptr_grow(OBSTACK, datum)\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o = (OBSTACK);\t\t\t\t\t \\\n+ if (obstack_room (__o) < sizeof (void *))\t\t\t \\\n+ _obstack_newchunk (__o, sizeof (void *));\t\t\t \\\n+ obstack_ptr_grow_fast (__o, datum); })\n+\n+# define obstack_int_grow(OBSTACK, datum)\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o = (OBSTACK);\t\t\t\t\t \\\n+ if (obstack_room (__o) < sizeof (int))\t\t\t\t \\\n+ _obstack_newchunk (__o, sizeof (int));\t\t\t\t \\\n+ obstack_int_grow_fast (__o, datum); })\n+\n+# define obstack_ptr_grow_fast(OBSTACK, aptr)\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o1 = (OBSTACK);\t\t\t\t \\\n+ void *__p1 = __o1->next_free;\t\t\t\t\t \\\n+ *(const void **) __p1 = (aptr);\t\t\t\t\t \\\n+ __o1->next_free += sizeof (const void *);\t\t\t \\\n+ (void) 0; })\n+\n+# define obstack_int_grow_fast(OBSTACK, aint)\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o1 = (OBSTACK);\t\t\t\t \\\n+ void *__p1 = __o1->next_free;\t\t\t\t\t \\\n+ *(int *) __p1 = (aint);\t\t\t\t\t\t \\\n+ __o1->next_free += sizeof (int);\t\t\t\t\t \\\n+ (void) 0; })\n+\n+# define obstack_blank(OBSTACK, length)\t\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o = (OBSTACK);\t\t\t\t\t \\\n+ _OBSTACK_SIZE_T __len = (length);\t\t\t\t \\\n+ if (obstack_room (__o) < __len)\t\t\t\t\t \\\n+ _obstack_newchunk (__o, __len);\t\t\t\t \\\n+ obstack_blank_fast (__o, __len); })\n+\n+# define obstack_alloc(OBSTACK, length)\t\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__h = (OBSTACK);\t\t\t\t\t \\\n+ obstack_blank (__h, (length));\t\t\t\t\t \\\n+ obstack_finish (__h); })\n+\n+# define obstack_copy(OBSTACK, where, length)\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__h = (OBSTACK);\t\t\t\t\t \\\n+ obstack_grow (__h, (where), (length));\t\t\t\t \\\n+ obstack_finish (__h); })\n+\n+# define obstack_copy0(OBSTACK, where, length)\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__h = (OBSTACK);\t\t\t\t\t \\\n+ obstack_grow0 (__h, (where), (length));\t\t\t\t \\\n+ obstack_finish (__h); })\n+\n+/* The local variable is named __o1 to avoid a shadowed variable\n+ warning when invoked from other obstack macros, typically obstack_free. */\n+# define obstack_finish(OBSTACK)\t\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o1 = (OBSTACK);\t\t\t\t \\\n+ void *__value = (void *) __o1->object_base;\t\t\t \\\n+ if (__o1->next_free == __value)\t\t\t\t\t \\\n+ __o1->maybe_empty_object = 1;\t\t\t\t\t \\\n+ __o1->next_free\t\t\t\t\t\t\t \\\n+ = __PTR_ALIGN (__o1->object_base, __o1->next_free,\t\t \\\n+ __o1->alignment_mask);\t\t\t\t \\\n+ if ((size_t) (__o1->next_free - (char *) __o1->chunk)\t\t \\\n+ > (size_t) (__o1->chunk_limit - (char *) __o1->chunk))\t \\\n+ __o1->next_free = __o1->chunk_limit;\t\t\t\t \\\n+ __o1->object_base = __o1->next_free;\t\t\t\t \\\n+ __value; })\n+\n+# define obstack_free(OBSTACK, OBJ)\t\t\t\t\t \\\n+ __extension__\t\t\t\t\t\t\t\t \\\n+ ({ struct obstack *__o = (OBSTACK);\t\t\t\t\t \\\n+ void *__obj = (void *) (OBJ);\t\t\t\t\t \\\n+ if (__obj > (void *) __o->chunk && __obj < (void *) __o->chunk_limit) \\\n+ __o->next_free = __o->object_base = (char *) __obj;\t\t \\\n+ else\t\t\t\t\t\t\t\t \\\n+ _obstack_free (__o, __obj); })\n+\n+#else /* not __GNUC__ */\n+\n+# define obstack_object_size(h)\t\t\t\t\t\t \\\n+ ((_OBSTACK_SIZE_T) ((h)->next_free - (h)->object_base))\n+\n+# define obstack_room(h)\t\t\t\t\t\t \\\n+ ((_OBSTACK_SIZE_T) ((h)->chunk_limit - (h)->next_free))\n+\n+# define obstack_empty_p(h)\t\t\t\t\t\t \\\n+ ((h)->chunk->prev == 0\t\t\t\t\t\t \\\n+ && (h)->next_free == __PTR_ALIGN ((char *) (h)->chunk,\t\t \\\n+ (h)->chunk->contents,\t\t \\\n+ (h)->alignment_mask))\n \n /* Note that the call to _obstack_newchunk is enclosed in (..., 0)\n so that we can avoid having void expressions\n@@ -422,88 +455,92 @@ __extension__\t\t\t\t\t\t\t\t\\\n Casting the third operand to void was tried before,\n but some compilers won't accept it. */\n \n-# define obstack_make_room(h,length)\t\t\t\t\t\\\n-( (h)->temp.tempint = (length),\t\t\t\t\t\t\\\n- (((h)->next_free + (h)->temp.tempint > (h)->chunk_limit)\t\t\\\n- ? (_obstack_newchunk ((h), (h)->temp.tempint), 0) : 0))\n-\n-# define obstack_grow(h,where,length)\t\t\t\t\t\\\n-( (h)->temp.tempint = (length),\t\t\t\t\t\t\\\n- (((h)->next_free + (h)->temp.tempint > (h)->chunk_limit)\t\t\\\n- ? (_obstack_newchunk ((h), (h)->temp.tempint), 0) : 0),\t\t\\\n- memcpy ((h)->next_free, where, (h)->temp.tempint),\t\t\t\\\n- (h)->next_free += (h)->temp.tempint)\n-\n-# define obstack_grow0(h,where,length)\t\t\t\t\t\\\n-( (h)->temp.tempint = (length),\t\t\t\t\t\t\\\n- (((h)->next_free + (h)->temp.tempint + 1 > (h)->chunk_limit)\t\t\\\n- ? (_obstack_newchunk ((h), (h)->temp.tempint + 1), 0) : 0),\t\t\\\n- memcpy ((h)->next_free, where, (h)->temp.tempint),\t\t\t\\\n- (h)->next_free += (h)->temp.tempint,\t\t\t\t\t\\\n- *((h)->next_free)++ = 0)\n-\n-# define obstack_1grow(h,datum)\t\t\t\t\t\t\\\n-( (((h)->next_free + 1 > (h)->chunk_limit)\t\t\t\t\\\n- ? (_obstack_newchunk ((h), 1), 0) : 0),\t\t\t\t\\\n- obstack_1grow_fast (h, datum))\n-\n-# define obstack_ptr_grow(h,datum)\t\t\t\t\t\\\n-( (((h)->next_free + sizeof (char *) > (h)->chunk_limit)\t\t\\\n- ? (_obstack_newchunk ((h), sizeof (char *)), 0) : 0),\t\t\\\n- obstack_ptr_grow_fast (h, datum))\n-\n-# define obstack_int_grow(h,datum)\t\t\t\t\t\\\n-( (((h)->next_free + sizeof (int) > (h)->chunk_limit)\t\t\t\\\n- ? (_obstack_newchunk ((h), sizeof (int)), 0) : 0),\t\t\t\\\n- obstack_int_grow_fast (h, datum))\n-\n-# define obstack_ptr_grow_fast(h,aptr)\t\t\t\t\t\\\n- (((const void **) ((h)->next_free += sizeof (void *)))[-1] = (aptr))\n-\n-# define obstack_int_grow_fast(h,aint)\t\t\t\t\t\\\n- (((int *) ((h)->next_free += sizeof (int)))[-1] = (aint))\n-\n-# define obstack_blank(h,length)\t\t\t\t\t\\\n-( (h)->temp.tempint = (length),\t\t\t\t\t\t\\\n- (((h)->chunk_limit - (h)->next_free < (h)->temp.tempint)\t\t\\\n- ? (_obstack_newchunk ((h), (h)->temp.tempint), 0) : 0),\t\t\\\n- obstack_blank_fast (h, (h)->temp.tempint))\n-\n-# define obstack_alloc(h,length)\t\t\t\t\t\\\n- (obstack_blank ((h), (length)), obstack_finish ((h)))\n-\n-# define obstack_copy(h,where,length)\t\t\t\t\t\\\n- (obstack_grow ((h), (where), (length)), obstack_finish ((h)))\n-\n-# define obstack_copy0(h,where,length)\t\t\t\t\t\\\n- (obstack_grow0 ((h), (where), (length)), obstack_finish ((h)))\n-\n-# define obstack_finish(h)\t\t\t\t\t\t\\\n-( ((h)->next_free == (h)->object_base\t\t\t\t\t\\\n- ? (((h)->maybe_empty_object = 1), 0)\t\t\t\t\t\\\n- : 0),\t\t\t\t\t\t\t\t\\\n- (h)->temp.tempptr = (h)->object_base,\t\t\t\t\t\\\n- (h)->next_free\t\t\t\t\t\t\t\\\n- = __PTR_ALIGN ((h)->object_base, (h)->next_free,\t\t\t\\\n-\t\t (h)->alignment_mask),\t\t\t\t\\\n- (((h)->next_free - (char *) (h)->chunk\t\t\t\t\\\n- > (h)->chunk_limit - (char *) (h)->chunk)\t\t\t\t\\\n- ? ((h)->next_free = (h)->chunk_limit) : 0),\t\t\t\t\\\n- (h)->object_base = (h)->next_free,\t\t\t\t\t\\\n- (h)->temp.tempptr)\n-\n-# define obstack_free(h,obj)\t\t\t\t\t\t\\\n-( (h)->temp.tempint = (char *) (obj) - (char *) (h)->chunk,\t\t\\\n- ((((h)->temp.tempint > 0\t\t\t\t\t\t\\\n- && (h)->temp.tempint < (h)->chunk_limit - (char *) (h)->chunk))\t\\\n- ? (int) ((h)->next_free = (h)->object_base\t\t\t\t\\\n-\t = (h)->temp.tempint + (char *) (h)->chunk)\t\t\t\\\n- : (((obstack_free) ((h), (h)->temp.tempint + (char *) (h)->chunk), 0), 0)))\n-\n-#endif /* not __GNUC__ or not __STDC__ */\n+# define obstack_make_room(h, length)\t\t\t\t\t \\\n+ ((h)->temp.i = (length),\t\t\t\t\t\t \\\n+ ((obstack_room (h) < (h)->temp.i)\t\t\t\t\t \\\n+ ? (_obstack_newchunk (h, (h)->temp.i), 0) : 0),\t\t\t \\\n+ (void) 0)\n+\n+# define obstack_grow(h, where, length)\t\t\t\t\t \\\n+ ((h)->temp.i = (length),\t\t\t\t\t\t \\\n+ ((obstack_room (h) < (h)->temp.i)\t\t\t\t\t \\\n+ ? (_obstack_newchunk ((h), (h)->temp.i), 0) : 0),\t\t\t \\\n+ memcpy ((h)->next_free, where, (h)->temp.i),\t\t\t\t \\\n+ (h)->next_free += (h)->temp.i,\t\t\t\t\t \\\n+ (void) 0)\n+\n+# define obstack_grow0(h, where, length)\t\t\t\t \\\n+ ((h)->temp.i = (length),\t\t\t\t\t\t \\\n+ ((obstack_room (h) < (h)->temp.i + 1)\t\t\t\t \\\n+ ? (_obstack_newchunk ((h), (h)->temp.i + 1), 0) : 0),\t\t \\\n+ memcpy ((h)->next_free, where, (h)->temp.i),\t\t\t\t \\\n+ (h)->next_free += (h)->temp.i,\t\t\t\t\t \\\n+ *((h)->next_free)++ = 0,\t\t\t\t\t\t \\\n+ (void) 0)\n+\n+# define obstack_1grow(h, datum)\t\t\t\t\t \\\n+ (((obstack_room (h) < 1)\t\t\t\t\t\t \\\n+ ? (_obstack_newchunk ((h), 1), 0) : 0),\t\t\t\t \\\n+ obstack_1grow_fast (h, datum))\n+\n+# define obstack_ptr_grow(h, datum)\t\t\t\t\t \\\n+ (((obstack_room (h) < sizeof (char *))\t\t\t\t \\\n+ ? (_obstack_newchunk ((h), sizeof (char *)), 0) : 0),\t\t \\\n+ obstack_ptr_grow_fast (h, datum))\n+\n+# define obstack_int_grow(h, datum)\t\t\t\t\t \\\n+ (((obstack_room (h) < sizeof (int))\t\t\t\t\t \\\n+ ? (_obstack_newchunk ((h), sizeof (int)), 0) : 0),\t\t\t \\\n+ obstack_int_grow_fast (h, datum))\n+\n+# define obstack_ptr_grow_fast(h, aptr)\t\t\t\t\t \\\n+ (((const void **) ((h)->next_free += sizeof (void *)))[-1] = (aptr),\t \\\n+ (void) 0)\n+\n+# define obstack_int_grow_fast(h, aint)\t\t\t\t\t \\\n+ (((int *) ((h)->next_free += sizeof (int)))[-1] = (aint),\t\t \\\n+ (void) 0)\n+\n+# define obstack_blank(h, length)\t\t\t\t\t \\\n+ ((h)->temp.i = (length),\t\t\t\t\t\t \\\n+ ((obstack_room (h) < (h)->temp.i)\t\t\t\t\t \\\n+ ? (_obstack_newchunk ((h), (h)->temp.i), 0) : 0),\t\t\t \\\n+ obstack_blank_fast (h, (h)->temp.i))\n+\n+# define obstack_alloc(h, length)\t\t\t\t\t \\\n+ (obstack_blank ((h), (length)), obstack_finish ((h)))\n+\n+# define obstack_copy(h, where, length)\t\t\t\t\t \\\n+ (obstack_grow ((h), (where), (length)), obstack_finish ((h)))\n+\n+# define obstack_copy0(h, where, length)\t\t\t\t \\\n+ (obstack_grow0 ((h), (where), (length)), obstack_finish ((h)))\n+\n+# define obstack_finish(h)\t\t\t\t\t\t \\\n+ (((h)->next_free == (h)->object_base\t\t\t\t\t \\\n+ ? (((h)->maybe_empty_object = 1), 0)\t\t\t\t \\\n+ : 0),\t\t\t\t\t\t\t\t \\\n+ (h)->temp.p = (h)->object_base,\t\t\t\t\t \\\n+ (h)->next_free\t\t\t\t\t\t\t \\\n+ = __PTR_ALIGN ((h)->object_base, (h)->next_free,\t\t\t \\\n+ (h)->alignment_mask),\t\t\t\t \\\n+ (((size_t) ((h)->next_free - (char *) (h)->chunk)\t\t\t \\\n+ > (size_t) ((h)->chunk_limit - (char *) (h)->chunk))\t\t \\\n+ ? ((h)->next_free = (h)->chunk_limit) : 0),\t\t\t\t \\\n+ (h)->object_base = (h)->next_free,\t\t\t\t\t \\\n+ (h)->temp.p)\n+\n+# define obstack_free(h, obj)\t\t\t\t\t\t \\\n+ ((h)->temp.p = (void *) (obj),\t\t\t\t\t \\\n+ (((h)->temp.p > (void *) (h)->chunk\t\t\t\t\t \\\n+ && (h)->temp.p < (void *) (h)->chunk_limit)\t\t\t \\\n+ ? (void) ((h)->next_free = (h)->object_base = (char *) (h)->temp.p) \\\n+ : _obstack_free ((h), (h)->temp.p)))\n+\n+#endif /* not __GNUC__ */\n \n #ifdef __cplusplus\n-}\t/* C++ */\n+} /* C++ */\n #endif\n \n-#endif /* obstack.h */\n+#endif /* _OBSTACK_H */\n-- \n2.22.0.589.g5bd7971b91\n\n"},{"id":"377244","messageId":"xmqq4l4suovx.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"20190614095308.GG31952@szeder.dev","subject":"Re: [PATCH 2/4] kwset: allow building with GCC 8","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-14T16:12:50Z","receivedAt":"2019-06-14T16:12:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n>> Now, the proper thing to do would be to switch to `size_t`. But this\n>> would make us deviate from the \"upstream\" code even further,\n>\n> This is not entirely true: upstream already uses 'size_t', so the\n> switch would actually bring our copy closer to upstream.\n\nAh, earlier I said that within the context how kwset uses obstack,\nit is perfectly proper to fix it like the patch in question did, but\nthe upstream already using size_t changes the picture quite a bit.\n\nI do not mind updating our copy of obstack, but make sure you pick\nthe version with license compatible with ours.\n\nThanks.\n"},{"id":"377247","messageId":"20190614175700.GA313@sigill.intra.peff.net","threadId":"51300","inReplyTo":"20190614100059.13540-1-szeder.dev@gmail.com","subject":"Re: [RFC/PATCH v1 0/4] compat/obstack: update from upstream","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-06-14T17:57:00Z","receivedAt":"2019-06-14T17:57:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 14, 2019 at 12:00:55PM +0200, SZEDER Gábor wrote:\n\n> Update 'compat/obstack.{c,h}' from upstream, because they already use\n> 'size_t' instead of 'long' in places that might eventually end up as\n> an argument to malloc(), which might solve build errors with GCC 8 on\n> Windows.\n> \n> The first patch just imports from upstream and doesn't modify anything\n> at all, and, consequently, it can't be compiled because of a screenful\n> or two of errors. This is bad for future bisects, of course.\n> \n> OTOH, adding all the necessary build fixes right away makes review\n> harder...\n\nOne thing about your approach that makes it hard to review is that the\nfirst commit obliterates all of our local changes, and then you have to\nre-apply them individually. Looking at \"git log\" there aren't very many\nin this case, so it's pretty easy to be sure you got them all (in some\ncases this can be particularly nasty if the changes were themselves part\nof conflict resolution, and so you have to pick them out of a merge).\n\nI think a flow that better matches \"what really happened\" is to do more\nof a vendor-branch approach: have a line of history that represents the\nupstream changes (each one obliterating the last), and then periodically\nmerge that into our fork.\n\nThat can even retain bisectability as long as the commits along the\nvendor branch don't actually try to build the code. Unfortunately our\ninitial import does try to build, so I think it already wrecks\nbisectability, but we can undo that now. So e.g.,:\n\n # start at e831171d67 (Add obstack.[ch] from EGLIBC 2.10, 2011-08-21)\n git checkout -b upstream-obstack e831171d67\n\n # undo build changes to restore bisection; ideally this would have\n # been done back then, but it's too late now\n sed -i /obstack/d Makefile\n git commit -am 'strip out obstack building'\n\n # but of course in our merged version we want that back, so let's\n # do a noop merge to represent that.\n git checkout master ;# or whatever feature branch you're working on\n git merge -s ours upstream-obstack\n\n # and now with a sane vendor branch established, we can proceed to do\n # a real update there\n git checkout upstream-obstack\n cp /path/to/obstack.[ch] compat/\n git commit -am 'update obstack'\n\n # and now we are free to merge that in, getting a real 3-way merge\n # between our changes and what happened upstream.\n git checkout master\n git merge upstream-obstack\n\nNow, if you try this you may find that the conflicts are pretty horrid.\nAnd the result may end up way less readable than your cherry-picks (and\nharder to resolve in the first place). I claim only that:\n\n 1. This represents in the history graph more clearly the actual\n sequence of events.\n\n 2. Its saves you from digging up the set of changes that have been\n applied since our last upstream import.\n\nSo in this case the way you did it may well be the best way. But I offer\nit as an alternative. :)\n\n-Peff\n"},{"id":"377248","messageId":"xmqqy324t4g0.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"20190614100059.13540-1-szeder.dev@gmail.com","subject":"Re: [RFC/PATCH v1 0/4] compat/obstack: update from upstream","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-14T18:19:43Z","receivedAt":"2019-06-14T18:19:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n> And here is an all-green build of these patches on Travis CI:\n>\n> https://travis-ci.org/szeder/git/builds/545645247\n>\n> (and one bonus patch on top to deal with some Homebrew nonsense)\n\nIs this the one that making all of the jobs pass in the above\noutput, including the mac gcc one. It would be wonderful to have it\nseparately and fast-tracked ;-)\n\n-- >8 --\nFrom: SZEDER Gábor \nDate: Wed, 3 Apr 2019 02:49:47 +0200\nSubject: [PATCH] ci: make Homebrew's operations faster\n\n---\n ci/install-dependencies.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7f6acdd803..f804b40ddd 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -34,7 +34,7 @@ linux-clang|linux-gcc)\n \tpopd\n \t;;\n osx-clang|osx-gcc)\n-\tbrew update >/dev/null\n+\texport HOMEBREW_NO_INSTALL_CLEANUP=1 HOMEBREW_NO_AUTO_UPDATE=1\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n \ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n-- \n2.22.0-68-g0aae918dd9\n\n"},{"id":"377255","messageId":"bd955bb3-626e-345d-550e-562027c2f7e6@ramsayjones.plus.com","threadId":"51300","inReplyTo":"20190614100059.13540-1-szeder.dev@gmail.com","subject":"Re: [RFC/PATCH v1 0/4] compat/obstack: update from upstream","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2019-06-14T20:30:20Z","receivedAt":"2019-06-14T20:33:51Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 14/06/2019 11:00, SZEDER Gábor wrote:\n> Update 'compat/obstack.{c,h}' from upstream, because they already use\n> 'size_t' instead of 'long' in places that might eventually end up as\n> an argument to malloc(), which might solve build errors with GCC 8 on\n> Windows.\n> \n> The first patch just imports from upstream and doesn't modify anything\n> at all, and, consequently, it can't be compiled because of a screenful\n> or two of errors. This is bad for future bisects, of course.\n> \n> OTOH, adding all the necessary build fixes right away makes review\n> harder...\n> \n> I'm not sure how to deal with this situation, so here is a series with\n> the fixes in separate patches for review, for now. If there's an\n> agreement that this is the direction to take, then I'll squash in the\n> fixes in the first patch and touch up the resulting commit message.\n> \n> \n> Ramsay, could you please run sparse on top of these patch series to\n> make sure that I caught and converted all \"0 instead of NULL\" usages\n> in the last patch? Thanks.\n\nI applied your patches to current master (@0aae918dd9) and, since\nyou dropped the final hunk of commit 3254310863 (\"obstack.c: Fix\nsome sparse warnings\", 2011-09-11), sparse complains, thus:\n\n $ diff sp-out sp-out1\n 27a28,30\n > compat/obstack.c:331:5: warning: incorrect type in initializer (different modifiers)\n > compat/obstack.c:331:5: expected void ( *[addressable] [toplevel] obstack_alloc_failed_handler )( ... )\n > compat/obstack.c:331:5: got void ( [noreturn] * )( ... )\n $ \n\nSo, yes you did catch all \"using plain integer as NULL pointer\"\nwarnings! :-D\n\nThanks.\n\nATB,\nRamsay Jones\n"},{"id":"377259","messageId":"b99ab7c9-63e3-6f2d-979b-d24a2670f9d5@ramsayjones.plus.com","threadId":"51300","inReplyTo":"bd955bb3-626e-345d-550e-562027c2f7e6@ramsayjones.plus.com","subject":"Re: [RFC/PATCH v1 0/4] compat/obstack: update from upstream","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2019-06-14T21:24:39Z","receivedAt":"2019-06-14T21:24:43Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 14/06/2019 21:30, Ramsay Jones wrote:\n> \n> \n> On 14/06/2019 11:00, SZEDER Gábor wrote:\n>> Update 'compat/obstack.{c,h}' from upstream, because they already use\n>> 'size_t' instead of 'long' in places that might eventually end up as\n>> an argument to malloc(), which might solve build errors with GCC 8 on\n>> Windows.\n>>\n>> The first patch just imports from upstream and doesn't modify anything\n>> at all, and, consequently, it can't be compiled because of a screenful\n>> or two of errors. This is bad for future bisects, of course.\n>>\n>> OTOH, adding all the necessary build fixes right away makes review\n>> harder...\n>>\n>> I'm not sure how to deal with this situation, so here is a series with\n>> the fixes in separate patches for review, for now. If there's an\n>> agreement that this is the direction to take, then I'll squash in the\n>> fixes in the first patch and touch up the resulting commit message.\n>>\n>>\n>> Ramsay, could you please run sparse on top of these patch series to\n>> make sure that I caught and converted all \"0 instead of NULL\" usages\n>> in the last patch? Thanks.\n> \n> I applied your patches to current master (@0aae918dd9) and, since\n> you dropped the final hunk of commit 3254310863 (\"obstack.c: Fix\n> some sparse warnings\", 2011-09-11), sparse complains, thus:\n> \n> $ diff sp-out sp-out1\n> 27a28,30\n> > compat/obstack.c:331:5: warning: incorrect type in initializer (different modifiers)\n> > compat/obstack.c:331:5: expected void ( *[addressable] [toplevel] obstack_alloc_failed_handler )( ... )\n> > compat/obstack.c:331:5: got void ( [noreturn] * )( ... )\n> $ \n> \n> So, yes you did catch all \"using plain integer as NULL pointer\"\n> warnings! :-D\n\nSorry for being a bit slow here, but I just realized that\nI should not have seen that on Linux (and should have tested\non cygwin), because the obstack code gets elided on Linux ...\n\nOh, wait:\n\n $ diff sc sc1\n 3a4,7\n > compat/obstack.o\t- _obstack_allocated_p\n > compat/obstack.o\t- obstack_alloc_failed_handler\n > compat/obstack.o\t- _obstack_begin_1\n > compat/obstack.o\t- _obstack_memory_used\n $ \n\nHmm, so on master, this code is totally elided on Linux, but\nthat is no longer the case with your patches applied. I guess\nyou need to look at the definition of the {_OBSTACK_}ELIDE_CODE\npreprocessor variable(s).\n\nHTH.\n\nATB,\nRamsay Jones\n\n"},{"id":"377270","messageId":"87wohn95vb.fsf@evledraar.gmail.com","threadId":"51300","inReplyTo":"c1527a71672056859a4613f2318bcbfce31e8b50.1560426581.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/4] kwset: allow building with GCC 8","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-14T22:09:12Z","receivedAt":"2019-06-14T22:09:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jun 13 2019, Johannes Schindelin via GitGitGadget wrote:\n\n> From: Johannes Schindelin \n>\n> The kwset functionality makes use of the obstack code, which expects to\n> be handed a function that can allocate large chunks of data. It expects\n> that function to accept a `size` parameter of type `long`.\n>\n> This upsets GCC 8 on Windows, because `long` does not have the same\n> bit size as `size_t` there.\n>\n> Now, the proper thing to do would be to switch to `size_t`. But this\n> would make us deviate from the \"upstream\" code even further, making it\n> hard to synchronize with newer versions, and also it would be quite\n> involved because that `long` type is so invasive in that code.\n\nAlso because we'd need to switch git.git to GPLv3, as noted at the top\nof the file we've grabbed the last GPLv2 version of this code.\n\nThat or convince these authors of GNU grep to dual-license their\ncontributions, assuming their code isn't derived from something else (I\ndidn't check):\n\n grep.git $ git shortlog -sn e7ac713d.. -- '*kwset*'\n 25 Jim Meyering\n 25 Paul Eggert\n 8 Norihiro Tanaka\n 2 Paolo Bonzini\n 1 Karl Berry\n 1 Tony Abou-Assaleh\n 1 Yuliy Pisetsky\n"},{"id":"377271","messageId":"87v9x793qi.fsf@evledraar.gmail.com","threadId":"51300","inReplyTo":"c1527a71672056859a4613f2318bcbfce31e8b50.1560426581.git.gitgitgadget@gmail.com","subject":"Can we just get rid of kwset & obstack in favor of optimistically using PCRE v2 JIT?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-14T22:55:17Z","receivedAt":"2019-06-14T22:55:24Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jun 13 2019, Johannes Schindelin via GitGitGadget wrote:\n\n> The kwset functionality makes use of the obstack code, which expects to\n> be handed a function that can allocate large chunks of data. It expects\n> that function to accept a `size` parameter of type `long`.\n>\n> This upsets GCC 8 on Windows, because `long` does not have the same\n> bit size as `size_t` there.\n>\n> Now, the proper thing to do would be to switch to `size_t`. But this\n> would make us deviate from the \"upstream\" code even further, making it\n> hard to synchronize with newer versions, and also it would be quite\n> involved because that `long` type is so invasive in that code.\n>\n> Let's punt, and instead provide a super small wrapper around\n> `xmalloc()`.\n\nI have a WIP patches from 2017 that do $subject that I can dig up, but\nthought I'd gauge interest first.\n\nRight now the grep code & pickaxe machinery will detect fixed strings\nand use kwset() as an optimization.\n\nBack when kwset was added in 9eceddeec6 (\"Use kwset in grep\",\n2011-08-21) this helped, but now doing this for grep with a PCRE pattern\nis actually counterproductive for performance. On top of current\n`master`:\n\n @@ -368 +368 @@ static int is_fixed(const char *s, size_t len)\n - return 1;\n + return 0;\n\nAnd running p7821-grep-engines-fixed.sh[1] (which is in git.git, and is\nas far as I got with this) we get:\n\n Test HEAD~ HEAD\n -------------------------------------------------------------------------\n 7821.1: fixed grep int 0.48(1.59+0.63) 0.48(1.53+0.68) +0.0%\n 7821.2: basic grep int 0.55(1.64+0.51) 0.72(2.97+0.54) +30.9%\n 7821.3: extended grep int 0.65(1.63+0.54) 0.77(2.92+0.60) +18.5%\n 7821.4: perl grep int 1.01(1.62+0.55) 0.36(0.97+0.58) -64.4%\n 7821.6: fixed grep uncommon 0.18(0.51+0.45) 0.18(0.51+0.46) +0.0%\n 7821.7: basic grep uncommon 0.18(0.50+0.46) 0.30(1.36+0.33) +66.7%\n 7821.8: extended grep uncommon 0.18(0.45+0.52) 0.28(1.37+0.37) +55.6%\n 7821.9: perl grep uncommon 0.18(0.52+0.45) 0.16(0.28+0.54) -11.1%\n 7821.11: fixed grep æ 0.31(1.28+0.39) 0.31(1.24+0.43) +0.0%\n 7821.12: basic grep æ 0.30(1.29+0.38) 0.22(0.85+0.36) -26.7%\n 7821.13: extended grep æ 0.30(1.26+0.40) 0.22(0.78+0.45) -26.7%\n 7821.14: perl grep æ 0.30(1.33+0.34) 0.16(0.25+0.56) -46.7%\n\nSo what this means on my Debian box is that when we use PCRE with JIT\nand just get out of its way and let it do its own fixed string matching\nit's up to ~65% faster than the kwset() path.\n\nThe usual case of just feeding the fixed pattern to glibc's regex\nfunction is slower, although as seen there when you grep for a\nrarely-occurring non-ASCII string glibc does better now (the perils of\nusing ancient last-version-to-use-GPLv2 snapshots...).\n\nSo my plan for this (which I partially implemented) was to have a series\nwhere if we have a fixed string and have PCRE v2 we'd use it instead of\nkwset() for fixed strings.\n\nIt seems most packagers build with PCRE v2 now (CC:\ngit-packagers@). I.e. we'd keep something like compile_fixed_regexp()\n(and as an aside just use PCRE's \\Q...\\E instead of our own escaping).\n\nWe'd have performance regression for platforms that use kwset() now but\ndon't build pcre2, or where pcre2 jit doesn't work. Does anyone care?\n\nThis would allow us to just \"git rm\" kwset.[ch] compat/obstack.[ch],\nwhich is ~2k lines of tricky code, 1/2 of which we're currently doomed\nto maintain a bitrotting version of due to license incompatibilities\nwith upstream[2].\n\nAs an aside there's other code in grep.c that we could similarly remove\nin favor of optimistic PCRE v2 use, e.g. the -w case can be replaced by\n\\b\\b, but I found that less promising[3]. We can also get a huge\nperformance win for BRE and ERE patterns by using PCRE v2 with a\ntranslation layer for those under the hood[4], but various solvable\nbackwards compatible headaches[5] related to that are why I got lost in\nthe weeds back in 2017 and didn't finish this.\n\nBut just the s/kwset/pcre2/ case is easy enough.\n\n1. Via:\n\n GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_7821_GREP_OPTS='' GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE2=YesPlease' ./run HEAD~ HEAD -- p7821-grep-engines-fixed.sh\n2. https://public-inbox.org/git/87wohn95vb.fsf@evledraar.gmail.com/\n3. https://github.com/avar/git/commit/49ca92e799\n4. https://github.com/avar/git/commit/a3cc090344\n5. https://github.com/avar/git/commit/7dd367eb37\n"},{"id":"377274","messageId":"87tvcr92m2.fsf@evledraar.gmail.com","threadId":"51300","inReplyTo":"87v9x793qi.fsf@evledraar.gmail.com","subject":"Re: Can we just get rid of kwset & obstack in favor of optimistically using PCRE v2 JIT?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-14T23:19:33Z","receivedAt":"2019-06-14T23:19:39Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Jun 15 2019, Ævar Arnfjörð Bjarmason wrote:\n\n> On Thu, Jun 13 2019, Johannes Schindelin via GitGitGadget wrote:\n>\n>> The kwset functionality makes use of the obstack code, which expects to\n>> be handed a function that can allocate large chunks of data. It expects\n>> that function to accept a `size` parameter of type `long`.\n>>\n>> This upsets GCC 8 on Windows, because `long` does not have the same\n>> bit size as `size_t` there.\n>>\n>> Now, the proper thing to do would be to switch to `size_t`. But this\n>> would make us deviate from the \"upstream\" code even further, making it\n>> hard to synchronize with newer versions, and also it would be quite\n>> involved because that `long` type is so invasive in that code.\n>>\n>> Let's punt, and instead provide a super small wrapper around\n>> `xmalloc()`.\n>\n> I have a WIP patches from 2017 that do $subject that I can dig up, but\n> thought I'd gauge interest first.\n>\n> Right now the grep code & pickaxe machinery will detect fixed strings\n> and use kwset() as an optimization.\n>\n> Back when kwset was added in 9eceddeec6 (\"Use kwset in grep\",\n> 2011-08-21) this helped, but now doing this for grep with a PCRE pattern\n> is actually counterproductive for performance. On top of current\n> `master`:\n>\n> @@ -368 +368 @@ static int is_fixed(const char *s, size_t len)\n> - return 1;\n> + return 0;\n>\n> And running p7821-grep-engines-fixed.sh[1] (which is in git.git, and is\n> as far as I got with this) we get:\n>\n> Test HEAD~ HEAD\n> -------------------------------------------------------------------------\n> 7821.1: fixed grep int 0.48(1.59+0.63) 0.48(1.53+0.68) +0.0%\n> 7821.2: basic grep int 0.55(1.64+0.51) 0.72(2.97+0.54) +30.9%\n> 7821.3: extended grep int 0.65(1.63+0.54) 0.77(2.92+0.60) +18.5%\n> 7821.4: perl grep int 1.01(1.62+0.55) 0.36(0.97+0.58) -64.4%\n> 7821.6: fixed grep uncommon 0.18(0.51+0.45) 0.18(0.51+0.46) +0.0%\n> 7821.7: basic grep uncommon 0.18(0.50+0.46) 0.30(1.36+0.33) +66.7%\n> 7821.8: extended grep uncommon 0.18(0.45+0.52) 0.28(1.37+0.37) +55.6%\n> 7821.9: perl grep uncommon 0.18(0.52+0.45) 0.16(0.28+0.54) -11.1%\n> 7821.11: fixed grep æ 0.31(1.28+0.39) 0.31(1.24+0.43) +0.0%\n> 7821.12: basic grep æ 0.30(1.29+0.38) 0.22(0.85+0.36) -26.7%\n> 7821.13: extended grep æ 0.30(1.26+0.40) 0.22(0.78+0.45) -26.7%\n> 7821.14: perl grep æ 0.30(1.33+0.34) 0.16(0.25+0.56) -46.7%\n>\n> So what this means on my Debian box is that when we use PCRE with JIT\n> and just get out of its way and let it do its own fixed string matching\n> it's up to ~65% faster than the kwset() path.\n>\n> The usual case of just feeding the fixed pattern to glibc's regex\n> function is slower, although as seen there when you grep for a\n> rarely-occurring non-ASCII string glibc does better now (the perils of\n> using ancient last-version-to-use-GPLv2 snapshots...).\n>\n> So my plan for this (which I partially implemented) was to have a series\n> where if we have a fixed string and have PCRE v2 we'd use it instead of\n> kwset() for fixed strings.\n>\n> It seems most packagers build with PCRE v2 now (CC:\n> git-packagers@). I.e. we'd keep something like compile_fixed_regexp()\n> (and as an aside just use PCRE's \\Q...\\E instead of our own escaping).\n>\n> We'd have performance regression for platforms that use kwset() now but\n> don't build pcre2, or where pcre2 jit doesn't work. Does anyone care?\n>\n> This would allow us to just \"git rm\" kwset.[ch] compat/obstack.[ch],\n> which is ~2k lines of tricky code, 1/2 of which we're currently doomed\n> to maintain a bitrotting version of due to license incompatibilities\n> with upstream[2].\n>\n> As an aside there's other code in grep.c that we could similarly remove\n> in favor of optimistic PCRE v2 use, e.g. the -w case can be replaced by\n> \\b\\b, but I found that less promising[3]. We can also get a huge\n> performance win for BRE and ERE patterns by using PCRE v2 with a\n> translation layer for those under the hood[4], but various solvable\n> backwards compatible headaches[5] related to that are why I got lost in\n> the weeds back in 2017 and didn't finish this.\n>\n> But just the s/kwset/pcre2/ case is easy enough.\n\n...small correction, we currently hard-rely on kwset() for any pattern\ncontaining a \\0 for \"git-grep\" (these can only by supplied via the -f\n option), this means that any pattern containing a \\0\nis implicitly fixed, unless kwset() doesn't like it (-i and non-ASCII),\nwhat a mess.\n\nSince we hard depend on REG_STARTEND since 2f8952250a (\"regex: add\nregexec_buf() that can work on a non NUL-terminated string\", 2016-09-21)\nwe should just fix that while we're at it. It's a backwards-incompatible\nchange, but I doubt anyone is relying on our undocumented behavior of\nimplicitly considering grep patterns with \\0 in them always fixed.\n\n\n> 1. Via:\n>\n> GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_7821_GREP_OPTS='' GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE2=YesPlease' ./run HEAD~ HEAD -- p7821-grep-engines-fixed.sh\n> 2. https://public-inbox.org/git/87wohn95vb.fsf@evledraar.gmail.com/\n> 3. https://github.com/avar/git/commit/49ca92e799\n> 4. https://github.com/avar/git/commit/a3cc090344\n> 5. https://github.com/avar/git/commit/7dd367eb37\n"},{"id":"377295","messageId":"CAPUEspjKxQFiRgmfb2SuR_xpVu4=MN66kGEeBK1pHdBgXQbv7Q@mail.gmail.com","threadId":"51300","inReplyTo":"87v9x793qi.fsf@evledraar.gmail.com","subject":"Re: Can we just get rid of kwset & obstack in favor of optimistically using PCRE v2 JIT?","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2019-06-15T09:01:32Z","receivedAt":"2019-06-15T09:01:45Z","isPatch":false,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"> It seems most packagers build with PCRE v2 now (CC:\n> git-packagers@). I.e. we'd keep something like compile_fixed_regexp()\n> (and as an aside just use PCRE's \\Q...\\E instead of our own escaping).\n\nOpenBSD does PCRE v1 without JIT but HardenedBSD does and therefore\nsegfaults when calling: `git grep -P`.\n\nA fix probably based on the old proposed patchset[1] will be needed in this case\nwith more urgency\n\nCarlo\n\n[1] https://public-inbox.org/git/20181209230024.43444-1-carenas@gmail.com/\n"},{"id":"377307","messageId":"20190615191514.GD8616@genre.crustytoothpaste.net","threadId":"51300","inReplyTo":"87v9x793qi.fsf@evledraar.gmail.com","subject":"Re: Can we just get rid of kwset & obstack in favor of optimistically using PCRE v2 JIT?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2019-06-15T19:15:14Z","receivedAt":"2019-06-15T19:15:22Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2019-06-14 at 22:55:17, Ævar Arnfjörð Bjarmason wrote:\n> It seems most packagers build with PCRE v2 now (CC:\n> git-packagers@). I.e. we'd keep something like compile_fixed_regexp()\n> (and as an aside just use PCRE's \\Q...\\E instead of our own escaping).\n> \n> We'd have performance regression for platforms that use kwset() now but\n> don't build pcre2, or where pcre2 jit doesn't work. Does anyone care?\n\nI know that there are people shipping newer versions of Git using CentOS\n6, which IIRC doesn't ship PCRE 2[0]. Since having to ship your own PCRE\nis a security maintenance nightmare, it's probably best to leave this at\nleast compatible with non-PCRE 2 systems until November 2020. At that\npoint, I'm happy to drop support for it.\n\nIf it would work but just be slower with PCRE 1, I'm not too terribly\nconcerned. Let that be an incentive to users to upgrade.\n\nAlso, as Carlos pointed out, not all platforms will have the JIT support\nfunctional, such as OpenBSD, NetBSD, and PaX Linux systems. That may be\nmore of a blocker than the CentOS issue, especially since people run PaX\nkernels with standard distros.\n\n[0] I'm not certain because CentOS 6 Docker images segfault on newer\nkernels and I'm too lazy to download a live CD image for testing.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"377308","messageId":"87r27u8pie.fsf@evledraar.gmail.com","threadId":"51300","inReplyTo":"20190615191514.GD8616@genre.crustytoothpaste.net","subject":"Re: Can we just get rid of kwset & obstack in favor of optimistically using PCRE v2 JIT?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-15T22:14:49Z","receivedAt":"2019-06-15T22:14:54Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Jun 15 2019, brian m. carlson wrote:\n\n> On 2019-06-14 at 22:55:17, Ævar Arnfjörð Bjarmason wrote:\n>> It seems most packagers build with PCRE v2 now (CC:\n>> git-packagers@). I.e. we'd keep something like compile_fixed_regexp()\n>> (and as an aside just use PCRE's \\Q...\\E instead of our own escaping).\n>>\n>> We'd have performance regression for platforms that use kwset() now but\n>> don't build pcre2, or where pcre2 jit doesn't work. Does anyone care?\n>\n> I know that there are people shipping newer versions of Git using CentOS\n> 6, which IIRC doesn't ship PCRE 2[0]. Since having to ship your own PCRE\n> is a security maintenance nightmare, it's probably best to leave this at\n> least compatible with non-PCRE 2 systems until November 2020. At that\n> point, I'm happy to drop support for it.\n>\n> If it would work but just be slower with PCRE 1, I'm not too terribly\n> concerned. Let that be an incentive to users to upgrade.\n\nNot just PCRE, but if you don't have PCRE at all things would still work\nperfectly fine.\n\nI.e. all we're talking about is how to treat this internal\noptimization. If we'd never imported kwset we'd still be perfectly\ncapable of searching for fixed strings with grep/pickaxe, it would have\njust been slower.\n\nSo platforms that don't have PCRE at all would be slowed down by\nsomething like what the benchmark in my\nhttps://public-inbox.org/git/87v9x793qi.fsf@evledraar.gmail.com/\nupthread shows.\n\nOr not, maybe their C library POSIX regcomp()/regexec() is faster.\n\nPlatforms that do have PCRE would be faster than they are now, and we\ncould stop shipping this kwset code.\n\nThe *only* case where what I've outlined above isn't true is cases where\nthe pattern being matched has a \\0. See my 966be95549 (\"grep: add tests\nto fix blind spots with \\0 patterns\", 2017-05-20) for how that behaves.\n\nThere our current behavior is IMNSHO insane, and is certainly\nundocumented and unreliable (i.e. it behaves differently if you\ne.g. have non-ASCII along with \\0 in the pattern, none of this is\ndocumented).\n\nHaving poked a bit at that I think the only sane thing there is to just\noutright die unless you have PCRE v2, which is the only backend we have\nthat has any hope of handling that sanely.\n\n> Also, as Carlos pointed out, not all platforms will have the JIT support\n> functional, such as OpenBSD, NetBSD, and PaX Linux systems. That may be\n> more of a blocker than the CentOS issue, especially since people run PaX\n> kernels with standard distros.\n>\n> [0] I'm not certain because CentOS 6 Docker images segfault on newer\n> kernels and I'm too lazy to download a live CD image for testing.\n"},{"id":"377310","messageId":"03a68560-8058-7436-6edb-38e276a37190@web.de","threadId":"51300","inReplyTo":"9613c88849e53e00e1f7ade49c6c9056309ef0b9.1560426581.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-16T06:48:02Z","receivedAt":"2019-06-16T06:48:19Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 13.06.19 um 13:49 schrieb Johannes Schindelin via GitGitGadget:\n> From: Johannes Schindelin \n>\n> The `labs()` function operates, as the initial `l` suggests, on `long`\n> parameters. However, in `config.c` we tried to use it on values of type\n> `intmax_t`.\n>\n> This problem was found by GCC v9.x.\n>\n> To fix it, let's just \"unroll\" the function (i.e. negate the value if it\n> is negative).\n\nThere's also imaxabs(3).\n\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> config.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/config.c b/config.c\n> index 296a6d9cc4..01c6e9df23 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -869,9 +869,9 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n> \t\t\terrno = EINVAL;\n> \t\t\treturn 0;\n> \t\t}\n> -\t\tuval = labs(val);\n> +\t\tuval = val < 0 ? -val : val;\n> \t\tuval *= factor;\n> -\t\tif (uval > max || labs(val) > uval) {\n> +\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n> \t\t\terrno = ERANGE;\n> \t\t\treturn 0;\n> \t\t}\n\nSo this check uses unsigned arithmetic to find out if the multiplication\noverflows, right? Let's say value is \"4G\", then val will be 4 and\nfactor will be 2^30. Multiplying the two yields 2^32. On a 32-bit\nsystem this will wrap around to 0, so that's what we get for uval there.\nThe range check will then pass unless max is negative, which is wrong.\n\nThis behavior was not introduced by your patch, of course.\n\nWe could fix it by using the macro unsigned_mult_overflows():\n\n\t\tuval = imaxabs(val);\n\t\tif (unsigned_mult_overflows(uval, factor) ||\n\t\t uval * factor > max) {\n\t\t\terrno = ERANGE;\n\t\t\treturn 0;\n\t\t}\n\nRené\n"},{"id":"377317","messageId":"f22599d4-148f-212c-282c-790f5bde1706@web.de","threadId":"51300","inReplyTo":"03a68560-8058-7436-6edb-38e276a37190@web.de","subject":"Re: [PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-16T08:24:17Z","receivedAt":"2019-06-16T08:24:25Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 16.06.19 um 08:48 schrieb René Scharfe:\n> Am 13.06.19 um 13:49 schrieb Johannes Schindelin via GitGitGadget:\n>> From: Johannes Schindelin \n>>\n>> The `labs()` function operates, as the initial `l` suggests, on `long`\n>> parameters. However, in `config.c` we tried to use it on values of type\n>> `intmax_t`.\n>>\n>> This problem was found by GCC v9.x.\n>>\n>> To fix it, let's just \"unroll\" the function (i.e. negate the value if it\n>> is negative).\n>\n> There's also imaxabs(3).\n>\n>>\n>> Signed-off-by: Johannes Schindelin \n>> ---\n>> config.c | 4 ++--\n>> 1 file changed, 2 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/config.c b/config.c\n>> index 296a6d9cc4..01c6e9df23 100644\n>> --- a/config.c\n>> +++ b/config.c\n>> @@ -869,9 +869,9 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n>> \t\t\terrno = EINVAL;\n>> \t\t\treturn 0;\n>> \t\t}\n>> -\t\tuval = labs(val);\n>> +\t\tuval = val < 0 ? -val : val;\n>> \t\tuval *= factor;\n>> -\t\tif (uval > max || labs(val) > uval) {\n>> +\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n>> \t\t\terrno = ERANGE;\n>> \t\t\treturn 0;\n>> \t\t}\n>\n> So this check uses unsigned arithmetic to find out if the multiplication\n> overflows, right? Let's say value is \"4G\", then val will be 4 and\n> factor will be 2^30. Multiplying the two yields 2^32. On a 32-bit\n> system this will wrap around to 0, so that's what we get for uval there.\n> The range check will then pass unless max is negative, which is wrong.\n\nNo, this example is wrong. (I need to remember to take baby steps while\ncarrying numbers. o_O)\n\nSo value = \"5G\", then val = 5 and factor = 2^30. After multiplication\nuval = 2^32 + 2^30, on 32-bit machines this is 2^30 due to wrap-around.\nCorrect so far?\n\nIf uval is 2^30, then it's smaller than 2^31, so will pass a check\nagainst INT_MAX. val is 5, which is smaller than 2^30, so will pass the\nsecond check as well. Makes sense?\n\nThat would mean \"5G\" will overflow on a 32-bit machine, but we won't\ndetect it.\n\nRené\n"},{"id":"377319","messageId":"90ef3797-78a6-f6d9-443b-387c0ab7cbe7@web.de","threadId":"51300","inReplyTo":"f22599d4-148f-212c-282c-790f5bde1706@web.de","subject":"Re: [PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-16T14:01:23Z","receivedAt":"2019-06-16T14:01:44Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 16.06.19 um 10:24 schrieb René Scharfe:\n> Am 16.06.19 um 08:48 schrieb René Scharfe:\n>> Am 13.06.19 um 13:49 schrieb Johannes Schindelin via GitGitGadget:\n>>> From: Johannes Schindelin \n>>>\n>>> The `labs()` function operates, as the initial `l` suggests, on `long`\n>>> parameters. However, in `config.c` we tried to use it on values of type\n>>> `intmax_t`.\n>>>\n>>> This problem was found by GCC v9.x.\n>>>\n>>> To fix it, let's just \"unroll\" the function (i.e. negate the value if it\n>>> is negative).\n>>\n>> There's also imaxabs(3).\n>>\n>>>\n>>> Signed-off-by: Johannes Schindelin \n>>> ---\n>>> config.c | 4 ++--\n>>> 1 file changed, 2 insertions(+), 2 deletions(-)\n>>>\n>>> diff --git a/config.c b/config.c\n>>> index 296a6d9cc4..01c6e9df23 100644\n>>> --- a/config.c\n>>> +++ b/config.c\n>>> @@ -869,9 +869,9 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n>>> \t\t\terrno = EINVAL;\n>>> \t\t\treturn 0;\n>>> \t\t}\n>>> -\t\tuval = labs(val);\n>>> +\t\tuval = val < 0 ? -val : val;\n>>> \t\tuval *= factor;\n>>> -\t\tif (uval > max || labs(val) > uval) {\n>>> +\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n>>> \t\t\terrno = ERANGE;\n>>> \t\t\treturn 0;\n>>> \t\t}\n>>\n>> So this check uses unsigned arithmetic to find out if the multiplication\n>> overflows, right? Let's say value is \"4G\", then val will be 4 and\n>> factor will be 2^30. Multiplying the two yields 2^32. On a 32-bit\n>> system this will wrap around to 0, so that's what we get for uval there.\n>> The range check will then pass unless max is negative, which is wrong.\n>\n> No, this example is wrong. (I need to remember to take baby steps while\n> carrying numbers. o_O)\n>\n> So value = \"5G\", then val = 5 and factor = 2^30. After multiplication\n> uval = 2^32 + 2^30, on 32-bit machines this is 2^30 due to wrap-around.\n\nYeah, except that in the real world uintmax_t is 8 bytes wide\neverywhere, even on x86 and ARM. So the code should be fine as-is. It\nwould be in trouble if we introduced bigger units, like T for 2^40 etc.,\nthough.\n\nAnyway, the code would be easier to read and ready for any units if it\nused unsigned_mult_overflows; would have saved me time spent painfully\nwading through the math at least. (Hopefully that's just my problem,\nthough.)\n\nRené\n"},{"id":"377336","messageId":"xmqq7e9ltbe2.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"90ef3797-78a6-f6d9-443b-387c0ab7cbe7@web.de","subject":"Re: [PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-16T22:26:29Z","receivedAt":"2019-06-16T22:26:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n>>>> To fix it, let's just \"unroll\" the function (i.e. negate the value if it\n>>>> is negative).\n>>>\n>>> There's also imaxabs(3).\n\nThat may be true, but seeing that some platforms wants to see\nintmax_t defined in the compat/ layer, I suspect we cannot avoid\nhaving a copy of unrolled implementation somewhere in our code.\n\n>>>> +\t\tuval = val < 0 ? -val : val;\n>>>> \t\tuval *= factor;\n>>>> -\t\tif (uval > max || labs(val) > uval) {\n>>>> +\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n>>>> \t\t\terrno = ERANGE;\n>>>> \t\t\treturn 0;\n>>>> \t\t}\n>>>\n>>> So this check uses unsigned arithmetic to find out if the multiplication\n>>> overflows, right?...\n>> No, this example is wrong...\n> ...\n> Anyway, the code would be easier to read and ready for any units if it\n> used unsigned_mult_overflows; would have saved me time spent painfully\n> wading through the math at least.\n\nA patch to use unsigned_mult_overflows() here, on top of the\n\"unrolled imaxabs\" patch we reviewed, would be good to tie a loose\nend.\n\nThanks.\n\n"},{"id":"377363","messageId":"20190617182621.GI31952@szeder.dev","threadId":"51300","inReplyTo":"xmqq4l4suovx.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 2/4] kwset: allow building with GCC 8","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-17T18:26:21Z","receivedAt":"2019-06-17T18:26:28Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Jun 14, 2019 at 09:12:50AM -0700, Junio C Hamano wrote:\n> SZEDER Gábor writes:\n> \n> >> Now, the proper thing to do would be to switch to `size_t`. But this\n> >> would make us deviate from the \"upstream\" code even further,\n> >\n> > This is not entirely true: upstream already uses 'size_t', so the\n> > switch would actually bring our copy closer to upstream.\n> \n> Ah, earlier I said that within the context how kwset uses obstack,\n> it is perfectly proper to fix it like the patch in question did, but\n> the upstream already using size_t changes the picture quite a bit.\n> \n> I do not mind updating our copy of obstack, but make sure you pick\n> the version with license compatible with ours.\n\nThe licensing of obstack.{c,h} didn't change, it's still \"GNU Lesser\nGeneral Public License as published by the Free Software Foundation;\neither version 2.1 of the License, or (at your option) any later\nversion\"\n\nNote how the first patch updating these files makes only superficial\nchanges to their license notices:\n\n diff --git a/compat/obstack.h b/compat/obstack.h\n index ced94d0118..811de588a4 100644\n --- a/compat/obstack.h\n +++ b/compat/obstack.h\n @@ -1,6 +1,5 @@\n /* obstack.h - object stack macros\n - Copyright (C) 1988-1994,1996-1999,2003,2004,2005,2009\n - Free Software Foundation, Inc.\n + Copyright (C) 1988-2019 Free Software Foundation, Inc.\n This file is part of the GNU C Library.\n \n The GNU C Library is free software; you can redistribute it and/or\n @@ -15,89 +14,89 @@\n \n You should have received a copy of the GNU Lesser General Public\n License along with the GNU C Library; if not, see\n - . */\n + . */\n \n\nBut I rather like Ævar's idea of simply getting rid of them :)\n\n"},{"id":"377364","messageId":"20190617183628.GJ31952@szeder.dev","threadId":"51300","inReplyTo":"bd955bb3-626e-345d-550e-562027c2f7e6@ramsayjones.plus.com","subject":"Re: [RFC/PATCH v1 0/4] compat/obstack: update from upstream","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-17T18:36:28Z","receivedAt":"2019-06-17T18:36:35Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Jun 14, 2019 at 09:30:20PM +0100, Ramsay Jones wrote:\n> \n> \n> On 14/06/2019 11:00, SZEDER Gábor wrote:\n> > Update 'compat/obstack.{c,h}' from upstream, because they already use\n> > 'size_t' instead of 'long' in places that might eventually end up as\n> > an argument to malloc(), which might solve build errors with GCC 8 on\n> > Windows.\n> > \n> > The first patch just imports from upstream and doesn't modify anything\n> > at all, and, consequently, it can't be compiled because of a screenful\n> > or two of errors. This is bad for future bisects, of course.\n> > \n> > OTOH, adding all the necessary build fixes right away makes review\n> > harder...\n> > \n> > I'm not sure how to deal with this situation, so here is a series with\n> > the fixes in separate patches for review, for now. If there's an\n> > agreement that this is the direction to take, then I'll squash in the\n> > fixes in the first patch and touch up the resulting commit message.\n> > \n> > \n> > Ramsay, could you please run sparse on top of these patch series to\n> > make sure that I caught and converted all \"0 instead of NULL\" usages\n> > in the last patch? Thanks.\n> \n> I applied your patches to current master (@0aae918dd9) and, since\n> you dropped the final hunk of commit 3254310863 (\"obstack.c: Fix\n> some sparse warnings\", 2011-09-11), sparse complains, thus:\n\nOh, indeed. 3254310863 removed that \"__attribute__ ((noreturn))\" from\nthe function's definition, but nowadays upstream writes that as\n\"static _Noreturn void print_and_abort (void)\", and I didn't realize\nthat this _Noreturn is the same thing.\n\n> $ diff sp-out sp-out1\n> 27a28,30\n> > compat/obstack.c:331:5: warning: incorrect type in initializer (different modifiers)\n> > compat/obstack.c:331:5: expected void ( *[addressable] [toplevel] obstack_alloc_failed_handler )( ... )\n> > compat/obstack.c:331:5: got void ( [noreturn] * )( ... )\n> $ \n> \n> So, yes you did catch all \"using plain integer as NULL pointer\"\n> warnings! :-D\n\nHeh :)\n\nAnyway, I won't do anything for the time being, in the hope that we\ncan get on board with removing kwset/obstack...\n\n"},{"id":"377647","messageId":"20190620103530.GA15700@sigill.intra.peff.net","threadId":"51300","inReplyTo":"87tvcr92m2.fsf@evledraar.gmail.com","subject":"Re: Can we just get rid of kwset & obstack in favor of optimistically using PCRE v2 JIT?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-06-20T10:35:31Z","receivedAt":"2019-06-20T10:35:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jun 15, 2019 at 01:19:33AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> ...small correction, we currently hard-rely on kwset() for any pattern\n> containing a \\0 for \"git-grep\" (these can only by supplied via the -f\n> option), this means that any pattern containing a \\0\n> is implicitly fixed, unless kwset() doesn't like it (-i and non-ASCII),\n> what a mess.\n> \n> Since we hard depend on REG_STARTEND since 2f8952250a (\"regex: add\n> regexec_buf() that can work on a non NUL-terminated string\", 2016-09-21)\n> we should just fix that while we're at it. It's a backwards-incompatible\n> change, but I doubt anyone is relying on our undocumented behavior of\n> implicitly considering grep patterns with \\0 in them always fixed.\n\nThat's only for NULs in the haystack, though. I don't think there's a\nway to have a NUL in the pattern with regcomp(), since it takes a\nNUL-terminated string.\n\nI do agree with you that treating it like a fixed string is somewhat\ninsane. We're probably better off to die.\n\nIn general, your plan to get rid of kwset sounds like a good path. It\nwould be a slight regression for somebody who is truly feeding a\nfixed-string pattern with a NUL in it, on a system without pcre. Right\nnow that works (via kwset), and if we would start feeding fixed strings\nto regcomp() then obviously that won't work. I guess we could go back to\nusing memmem as a fallback, which is what it looks like we used before\n9eceddeec6 (Use kwset in grep, 2011-08-21).\n\nSeems like a code path that would get exercised approximately never,\nthough.\n\n-Peff\n"},{"id":"377693","messageId":"ebe4176d-1b51-fc82-7bce-713d098dbcf3@web.de","threadId":"51300","inReplyTo":"xmqq7e9ltbe2.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-20T19:58:37Z","receivedAt":"2019-06-20T19:58:49Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 17.06.19 um 00:26 schrieb Junio C Hamano:\n> René Scharfe writes:\n>\n>>>>> To fix it, let's just \"unroll\" the function (i.e. negate the value if it\n>>>>> is negative).\n>>>>\n>>>> There's also imaxabs(3).\n>\n> That may be true, but seeing that some platforms wants to see\n> intmax_t defined in the compat/ layer, I suspect we cannot avoid\n> having a copy of unrolled implementation somewhere in our code.\n\nRight. And if we later decide to give it a name then we could ask\nCoccinelle to find the places that need converting with a semantic\npatch like this:\n\n\t@@\n\tintmax_t i;\n\t@@\n\t- i < 0 ? -i : i\n\t+ imaxabs(i)\n\nSide note: I was surprised to find that I added those labs(3) calls, in\n83915ba521 (\"use labs() for variables of type long instead of abs()\",\n2014-11-15). Sloppy. :-/\n\n> A patch to use unsigned_mult_overflows() here, on top of the\n> \"unrolled imaxabs\" patch we reviewed, would be good to tie a loose\n> end.\n\nHow about this?\n\n-- >8 --\nSubject: [PATCH] config: simplify unit suffix handling\n\nparse_unit_factor() checks if a K, M or G is present after a number and\nmultiplies it by 2^10, 2^20 or 2^30, respectively. One of its callers\nchecks if the result is smaller than the number alone to detect\noverflows. The other one passes 1 as the number and does multiplication\nand overflow check itself in a similar manner.\n\nThis works, but is inconsistent, and it would break if we added support\nfor a bigger unit factor. E.g. 16777217T expands to 2^64 + 2^40, which\nis too big for a 64-bit number. Modulo 2^64 we get 2^40 == 1TB, which\nis bigger than the raw number 16777217 == 2^24 + 1, so the overflow\nwould go undetected by that method.\n\nMove the multiplication out of parse_unit_factor() and rename it to\nget_unit_factor() to signify its reduced functionality. This partially\nreverts c8deb5a146 (\"Improve error messages when int/long cannot be\nparsed from config\", 2007-12-25), but keeps the improved error messages.\nUse a return value of 0 to signal an invalid suffix.\n\nAnd use unsigned_mult_overflows to check for an overflow *before* doing\nthe actual multiplication, which is simpler and can deal with larger\nunit factors.\n\nSigned-off-by: Rene Scharfe \n---\nPatch generated with --function-context for easier reviewing.\n\n config.c | 39 ++++++++++++++++++---------------------\n 1 file changed, 18 insertions(+), 21 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 01c6e9df23..61a8bbb5cd 100644\n--- a/config.c\n+++ b/config.c\n@@ -834,51 +834,46 @@ static int git_parse_source(config_fn_t fn, void *data,\n \treturn error_return;\n }\n\n-static int parse_unit_factor(const char *end, uintmax_t *val)\n+static uintmax_t get_unit_factor(const char *end)\n {\n \tif (!*end)\n \t\treturn 1;\n-\telse if (!strcasecmp(end, \"k\")) {\n-\t\t*val *= 1024;\n-\t\treturn 1;\n-\t}\n-\telse if (!strcasecmp(end, \"m\")) {\n-\t\t*val *= 1024 * 1024;\n-\t\treturn 1;\n-\t}\n-\telse if (!strcasecmp(end, \"g\")) {\n-\t\t*val *= 1024 * 1024 * 1024;\n-\t\treturn 1;\n-\t}\n+\tif (!strcasecmp(end, \"k\"))\n+\t\treturn 1024;\n+\tif (!strcasecmp(end, \"m\"))\n+\t\treturn 1024 * 1024;\n+\tif (!strcasecmp(end, \"g\"))\n+\t\treturn 1024 * 1024 * 1024;\n \treturn 0;\n }\n\n static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n {\n \tif (value && *value) {\n \t\tchar *end;\n \t\tintmax_t val;\n \t\tuintmax_t uval;\n-\t\tuintmax_t factor = 1;\n+\t\tuintmax_t factor;\n\n \t\terrno = 0;\n \t\tval = strtoimax(value, &end, 0);\n \t\tif (errno == ERANGE)\n \t\t\treturn 0;\n-\t\tif (!parse_unit_factor(end, &factor)) {\n+\t\tfactor = get_unit_factor(end);\n+\t\tif (!factor) {\n \t\t\terrno = EINVAL;\n \t\t\treturn 0;\n \t\t}\n \t\tuval = val < 0 ? -val : val;\n-\t\tuval *= factor;\n-\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n+\t\tif (unsigned_mult_overflows(factor, uval) ||\n+\t\t factor * uval > max) {\n \t\t\terrno = ERANGE;\n \t\t\treturn 0;\n \t\t}\n \t\tval *= factor;\n \t\t*ret = val;\n \t\treturn 1;\n \t}\n \terrno = EINVAL;\n \treturn 0;\n }\n@@ -886,26 +881,28 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n {\n \tif (value && *value) {\n \t\tchar *end;\n \t\tuintmax_t val;\n-\t\tuintmax_t oldval;\n+\t\tuintmax_t factor;\n\n \t\terrno = 0;\n \t\tval = strtoumax(value, &end, 0);\n \t\tif (errno == ERANGE)\n \t\t\treturn 0;\n-\t\toldval = val;\n-\t\tif (!parse_unit_factor(end, &val)) {\n+\t\tfactor = get_unit_factor(end);\n+\t\tif (!factor) {\n \t\t\terrno = EINVAL;\n \t\t\treturn 0;\n \t\t}\n-\t\tif (val > max || oldval > val) {\n+\t\tif (unsigned_mult_overflows(factor, val) ||\n+\t\t factor * val > max) {\n \t\t\terrno = ERANGE;\n \t\t\treturn 0;\n \t\t}\n+\t\tval *= factor;\n \t\t*ret = val;\n \t\treturn 1;\n \t}\n \terrno = EINVAL;\n \treturn 0;\n }\n--\n2.22.0\n"},{"id":"377703","messageId":"xmqqtvckklsu.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"ebe4176d-1b51-fc82-7bce-713d098dbcf3@web.de","subject":"Re: [PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-20T21:07:45Z","receivedAt":"2019-06-20T21:07:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> How about this?\n\nSounds sensible.\n\n> -- >8 --\n> Subject: [PATCH] config: simplify unit suffix handling\n>\n> parse_unit_factor() checks if a K, M or G is present after a number and\n> multiplies it by 2^10, 2^20 or 2^30, respectively. One of its callers\n> checks if the result is smaller than the number alone to detect\n> overflows. The other one passes 1 as the number and does multiplication\n> and overflow check itself in a similar manner.\n>\n> This works, but is inconsistent, and it would break if we added support\n> for a bigger unit factor. E.g. 16777217T expands to 2^64 + 2^40, which\n> is too big for a 64-bit number. Modulo 2^64 we get 2^40 == 1TB, which\n> is bigger than the raw number 16777217 == 2^24 + 1, so the overflow\n> would go undetected by that method.\n>\n> Move the multiplication out of parse_unit_factor() and rename it to\n> get_unit_factor() to signify its reduced functionality. This partially\n> reverts c8deb5a146 (\"Improve error messages when int/long cannot be\n> parsed from config\", 2007-12-25), but keeps the improved error messages.\n> Use a return value of 0 to signal an invalid suffix.\n>\n> And use unsigned_mult_overflows to check for an overflow *before* doing\n> the actual multiplication, which is simpler and can deal with larger\n> unit factors.\n>\n> Signed-off-by: Rene Scharfe \n> ---\n> Patch generated with --function-context for easier reviewing.\n>\n> config.c | 39 ++++++++++++++++++---------------------\n> 1 file changed, 18 insertions(+), 21 deletions(-)\n>\n> diff --git a/config.c b/config.c\n> index 01c6e9df23..61a8bbb5cd 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -834,51 +834,46 @@ static int git_parse_source(config_fn_t fn, void *data,\n> \treturn error_return;\n> }\n>\n> -static int parse_unit_factor(const char *end, uintmax_t *val)\n> +static uintmax_t get_unit_factor(const char *end)\n> {\n> \tif (!*end)\n> \t\treturn 1;\n> -\telse if (!strcasecmp(end, \"k\")) {\n> -\t\t*val *= 1024;\n> -\t\treturn 1;\n> -\t}\n> -\telse if (!strcasecmp(end, \"m\")) {\n> -\t\t*val *= 1024 * 1024;\n> -\t\treturn 1;\n> -\t}\n> -\telse if (!strcasecmp(end, \"g\")) {\n> -\t\t*val *= 1024 * 1024 * 1024;\n> -\t\treturn 1;\n> -\t}\n> +\tif (!strcasecmp(end, \"k\"))\n> +\t\treturn 1024;\n> +\tif (!strcasecmp(end, \"m\"))\n> +\t\treturn 1024 * 1024;\n> +\tif (!strcasecmp(end, \"g\"))\n> +\t\treturn 1024 * 1024 * 1024;\n> \treturn 0;\n> }\n>\n> static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n> {\n> \tif (value && *value) {\n> \t\tchar *end;\n> \t\tintmax_t val;\n> \t\tuintmax_t uval;\n> -\t\tuintmax_t factor = 1;\n> +\t\tuintmax_t factor;\n>\n> \t\terrno = 0;\n> \t\tval = strtoimax(value, &end, 0);\n> \t\tif (errno == ERANGE)\n> \t\t\treturn 0;\n> -\t\tif (!parse_unit_factor(end, &factor)) {\n> +\t\tfactor = get_unit_factor(end);\n> +\t\tif (!factor) {\n> \t\t\terrno = EINVAL;\n> \t\t\treturn 0;\n> \t\t}\n> \t\tuval = val < 0 ? -val : val;\n> -\t\tuval *= factor;\n> -\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n> +\t\tif (unsigned_mult_overflows(factor, uval) ||\n> +\t\t factor * uval > max) {\n> \t\t\terrno = ERANGE;\n> \t\t\treturn 0;\n> \t\t}\n> \t\tval *= factor;\n> \t\t*ret = val;\n> \t\treturn 1;\n> \t}\n> \terrno = EINVAL;\n> \treturn 0;\n> }\n> @@ -886,26 +881,28 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n> static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n> {\n> \tif (value && *value) {\n> \t\tchar *end;\n> \t\tuintmax_t val;\n> -\t\tuintmax_t oldval;\n> +\t\tuintmax_t factor;\n>\n> \t\terrno = 0;\n> \t\tval = strtoumax(value, &end, 0);\n> \t\tif (errno == ERANGE)\n> \t\t\treturn 0;\n> -\t\toldval = val;\n> -\t\tif (!parse_unit_factor(end, &val)) {\n> +\t\tfactor = get_unit_factor(end);\n> +\t\tif (!factor) {\n> \t\t\terrno = EINVAL;\n> \t\t\treturn 0;\n> \t\t}\n> -\t\tif (val > max || oldval > val) {\n> +\t\tif (unsigned_mult_overflows(factor, val) ||\n> +\t\t factor * val > max) {\n> \t\t\terrno = ERANGE;\n> \t\t\treturn 0;\n> \t\t}\n> +\t\tval *= factor;\n> \t\t*ret = val;\n> \t\treturn 1;\n> \t}\n> \terrno = EINVAL;\n> \treturn 0;\n> }\n> --\n> 2.22.0\n"},{"id":"377761","messageId":"nycvar.QRO.7.76.6.1906212024010.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"ebe4176d-1b51-fc82-7bce-713d098dbcf3@web.de","subject":"Re: [PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-21T18:35:21Z","receivedAt":"2019-06-21T18:35:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi René,\n\nOn Thu, 20 Jun 2019, René Scharfe wrote:\n\n> Subject: [PATCH] config: simplify unit suffix handling\n>\n> parse_unit_factor() checks if a K, M or G is present after a number and\n> multiplies it by 2^10, 2^20 or 2^30, respectively. One of its callers\n> checks if the result is smaller than the number alone to detect\n> overflows. The other one passes 1 as the number and does multiplication\n> and overflow check itself in a similar manner.\n>\n> This works, but is inconsistent, and it would break if we added support\n> for a bigger unit factor. E.g. 16777217T expands to 2^64 + 2^40, which\n> is too big for a 64-bit number. Modulo 2^64 we get 2^40 == 1TB, which\n> is bigger than the raw number 16777217 == 2^24 + 1, so the overflow\n> would go undetected by that method.\n>\n> Move the multiplication out of parse_unit_factor() and rename it to\n> get_unit_factor() to signify its reduced functionality. This partially\n\nI do not necessarily think that the name `get_unit_factor()` is better,\ngiven that we still parse the unit factor. I'd vote for keeping the\noriginal name.\n\nHowever, what _does_ make sense is to change that function to _really_\nonly parse the unit factor. That is, I would keep the exact signature, I\njust would not multiply `*val` by the unit factor, I would overwrite it by\nthe unit factor instead.\n\nAt least that is what I would have expected, reading the name\n`parse_unit_factor()`.\n\n> reverts c8deb5a146 (\"Improve error messages when int/long cannot be\n> parsed from config\", 2007-12-25), but keeps the improved error messages.\n> Use a return value of 0 to signal an invalid suffix.\n\nThis comment should probably become a code comment above the function.\n\n> And use unsigned_mult_overflows to check for an overflow *before* doing\n> the actual multiplication, which is simpler and can deal with larger\n> unit factors.\n\nMakes sense.\n\n> Signed-off-by: Rene Scharfe \n\nWhat, no accent?\n\n> ---\n> Patch generated with --function-context for easier reviewing.\n\nOoh, ooh, I did not know that flag. Neat!\n\n> diff --git a/config.c b/config.c\n> index 01c6e9df23..61a8bbb5cd 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -834,51 +834,46 @@ static int git_parse_source(config_fn_t fn, void *data,\n> \treturn error_return;\n> }\n>\n> -static int parse_unit_factor(const char *end, uintmax_t *val)\n> +static uintmax_t get_unit_factor(const char *end)\n\nIt has been a historical wart that the parameter was called `end`. Maybe\nthat could be fixed, \"while at it\"?\n\nAnd as I said earlier, I do not see the need to change the signature\n(including the function name) at all.\n\n> {\n> \tif (!*end)\n> \t\treturn 1;\n> -\telse if (!strcasecmp(end, \"k\")) {\n> -\t\t*val *= 1024;\n> -\t\treturn 1;\n> -\t}\n> -\telse if (!strcasecmp(end, \"m\")) {\n> -\t\t*val *= 1024 * 1024;\n> -\t\treturn 1;\n> -\t}\n> -\telse if (!strcasecmp(end, \"g\")) {\n> -\t\t*val *= 1024 * 1024 * 1024;\n> -\t\treturn 1;\n> -\t}\n> +\tif (!strcasecmp(end, \"k\"))\n> +\t\treturn 1024;\n> +\tif (!strcasecmp(end, \"m\"))\n> +\t\treturn 1024 * 1024;\n> +\tif (!strcasecmp(end, \"g\"))\n> +\t\treturn 1024 * 1024 * 1024;\n> \treturn 0;\n> }\n>\n> static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n> {\n> \tif (value && *value) {\n> \t\tchar *end;\n> \t\tintmax_t val;\n> \t\tuintmax_t uval;\n> -\t\tuintmax_t factor = 1;\n> +\t\tuintmax_t factor;\n\nI'd keep this change, but...\n\n>\n> \t\terrno = 0;\n> \t\tval = strtoimax(value, &end, 0);\n> \t\tif (errno == ERANGE)\n> \t\t\treturn 0;\n> -\t\tif (!parse_unit_factor(end, &factor)) {\n> +\t\tfactor = get_unit_factor(end);\n> +\t\tif (!factor) {\n\n... drop this change, and...\n\n> \t\t\terrno = EINVAL;\n> \t\t\treturn 0;\n> \t\t}\n> \t\tuval = val < 0 ? -val : val;\n> -\t\tuval *= factor;\n> -\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n> +\t\tif (unsigned_mult_overflows(factor, uval) ||\n> +\t\t factor * uval > max) {\n\n... again keep this change.\n\n> \t\t\terrno = ERANGE;\n> \t\t\treturn 0;\n> \t\t}\n> \t\tval *= factor;\n> \t\t*ret = val;\n> \t\treturn 1;\n> \t}\n> \terrno = EINVAL;\n> \treturn 0;\n> }\n> @@ -886,26 +881,28 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n> static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n> {\n> \tif (value && *value) {\n> \t\tchar *end;\n> \t\tuintmax_t val;\n> -\t\tuintmax_t oldval;\n> +\t\tuintmax_t factor;\n\nGood.\n\n>\n> \t\terrno = 0;\n> \t\tval = strtoumax(value, &end, 0);\n> \t\tif (errno == ERANGE)\n> \t\t\treturn 0;\n> -\t\toldval = val;\n> -\t\tif (!parse_unit_factor(end, &val)) {\n> +\t\tfactor = get_unit_factor(end);\n> +\t\tif (!factor) {\n\nAgain, here I would strongly suggest the less intrusive change (with a\nmore intuitive outcome):\n\n-\t\toldval = val;\n-\t\tif (!parse_unit_factor(end, &val)) {\n+\t\tif (!parse_unit_factor(end, &factor)) {\n\n> \t\t\terrno = EINVAL;\n> \t\t\treturn 0;\n> \t\t}\n> -\t\tif (val > max || oldval > val) {\n> +\t\tif (unsigned_mult_overflows(factor, val) ||\n> +\t\t factor * val > max) {\n\nAnd this is obviously a good change again.\n\n> \t\t\terrno = ERANGE;\n> \t\t\treturn 0;\n> \t\t}\n> +\t\tval *= factor;\n\nAs is this.\n\nThanks for working on this!\nDscho\n\n> \t\t*ret = val;\n> \t\treturn 1;\n> \t}\n> \terrno = EINVAL;\n> \treturn 0;\n> }\n> --\n> 2.22.0\n>\n"},{"id":"377807","messageId":"0dd77efa-a8f5-9f3c-6057-d6fddfc23563@web.de","threadId":"51300","inReplyTo":"nycvar.QRO.7.76.6.1906212024010.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 4/4] config: avoid calling `labs()` on too-large data type","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-22T10:03:24Z","receivedAt":"2019-06-22T10:03:35Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 21.06.19 um 20:35 schrieb Johannes Schindelin:\n> Hi René,\n>\n> On Thu, 20 Jun 2019, René Scharfe wrote:\n>\n>> Subject: [PATCH] config: simplify unit suffix handling\n>>\n>> parse_unit_factor() checks if a K, M or G is present after a number and\n>> multiplies it by 2^10, 2^20 or 2^30, respectively. One of its callers\n>> checks if the result is smaller than the number alone to detect\n>> overflows. The other one passes 1 as the number and does multiplication\n>> and overflow check itself in a similar manner.\n>>\n>> This works, but is inconsistent, and it would break if we added support\n>> for a bigger unit factor. E.g. 16777217T expands to 2^64 + 2^40, which\n>> is too big for a 64-bit number. Modulo 2^64 we get 2^40 == 1TB, which\n>> is bigger than the raw number 16777217 == 2^24 + 1, so the overflow\n>> would go undetected by that method.\n>>\n>> Move the multiplication out of parse_unit_factor() and rename it to\n>> get_unit_factor() to signify its reduced functionality. This partially\n>\n> I do not necessarily think that the name `get_unit_factor()` is better,\n> given that we still parse the unit factor. I'd vote for keeping the\n> original name.\n\nget_unit_factor() is the original name from before c8deb5a146.\n\n> However, what _does_ make sense is to change that function to _really_\n> only parse the unit factor. That is, I would keep the exact signature, I\n> just would not multiply `*val` by the unit factor, I would overwrite it by\n> the unit factor instead.\n\nSo the patch is too big. Its narrative of \"let's restore the original\ncode, but keep the good features added since\" is not carrying the\nweight of its many changes.\n\n> At least that is what I would have expected, reading the name\n> `parse_unit_factor()`.\n\nHence the renaming. :)\n\nWhen I read parse_unit_factor() without any context then I expect it to\nwork in the middle of a string, telling the caller how many characters\nwere recognized. It would then be usable with different units, e.g.\nfor \"17KB\" just as well as for \"100Mbps\".\n\nWe don't need such a generic function here, of course.\n\n>> reverts c8deb5a146 (\"Improve error messages when int/long cannot be\n>> parsed from config\", 2007-12-25), but keeps the improved error messages.\n>> Use a return value of 0 to signal an invalid suffix.\n>\n> This comment should probably become a code comment above the function.\n\nYou mean just the last sentence, right? For an exported function I'd\nagree, but for this short helper I'm not so sure and would rather not\nbother the reader with easily inferable facts.\n\n>> Signed-off-by: Rene Scharfe \n>\n> What, no accent?\n\nI prefer a recognizable simplified version to a butchered one. Perhaps\nthe world is ready for Unicode now? I still get weirdly transformed\ncharacters on letters and parcels, so I'm cautious. Testing the waters\nwith the sender name setting in my MUA for some time now..\n\n>> diff --git a/config.c b/config.c\n>> index 01c6e9df23..61a8bbb5cd 100644\n>> --- a/config.c\n>> +++ b/config.c\n>> @@ -834,51 +834,46 @@ static int git_parse_source(config_fn_t fn, void *data,\n>> \treturn error_return;\n>> }\n>>\n>> -static int parse_unit_factor(const char *end, uintmax_t *val)\n>> +static uintmax_t get_unit_factor(const char *end)\n>\n> It has been a historical wart that the parameter was called `end`. Maybe\n> that could be fixed, \"while at it\"?\n\nI was tempted to do that, and am a bit proud of having resisted that\none. I try to avoid \"what at it\" these days -- if it's worth doing\nthat other thing then it can live in its own patch.\n\nBut the name \"end\" is arguably good, as it signifies that the function\nonly works with unit factors at the end of strings.\n\n>>\n>> \t\terrno = 0;\n>> \t\tval = strtoumax(value, &end, 0);\n>> \t\tif (errno == ERANGE)\n>> \t\t\treturn 0;\n>> -\t\toldval = val;\n>> -\t\tif (!parse_unit_factor(end, &val)) {\n>> +\t\tfactor = get_unit_factor(end);\n>> +\t\tif (!factor) {\n>\n> Again, here I would strongly suggest the less intrusive change (with a\n> more intuitive outcome):\n>\n> -\t\toldval = val;\n> -\t\tif (!parse_unit_factor(end, &val)) {\n> +\t\tif (!parse_unit_factor(end, &factor)) {\n>\n>> \t\t\terrno = EINVAL;\n>> \t\t\treturn 0;\n>> \t\t}\n>> -\t\tif (val > max || oldval > val) {\n>> +\t\tif (unsigned_mult_overflows(factor, val) ||\n>> +\t\t factor * val > max) {\n\nI'll split that out, then we can discuss it separately.\n\nRené\n"},{"id":"377808","messageId":"7a580777-2ccc-6097-477a-e66eb28d5f9b@web.de","threadId":"51300","inReplyTo":"xmqq7e9ltbe2.fsf@gitster-ct.c.googlers.com","subject":"[PATCH v2 1/3] config: use unsigned_mult_overflows to check for overflows","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-22T10:03:30Z","receivedAt":"2019-06-22T10:03:40Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"parse_unit_factor() checks if a K, M or G is present after a number and\nmultiplies it by 2^10, 2^20 or 2^30, respectively. One of its callers\nchecks if the result is smaller than the number alone to detect\noverflows. The other one passes 1 as the number and does multiplication\nand overflow check itself in a similar manner.\n\nThis works, but is inconsistent, and it would break if we added support\nfor a bigger unit factor. E.g. 16777217T is 2^64 + 2^40, i.e. too big\nfor a 64-bit number. Modulo 2^64 we get 2^40 == 1TB, which is bigger\nthan the raw number 16777217 == 2^24 + 1, so the overflow would go\nundetected by that method.\n\nLet both callers pass 1 and handle overflow check and multiplication\nthemselves. Do the check before the multiplication, using\nunsigned_mult_overflows, which is simpler and can deal with larger unit\nfactors.\n\nSigned-off-by: Rene Scharfe \n---\nPatch generated with --function-context for easier review (e.g. to see\nwhy we can stop updating uval in place).\n\n config.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 01c6e9df23..3c00369ba8 100644\n--- a/config.c\n+++ b/config.c\n@@ -856,29 +856,29 @@ static int parse_unit_factor(const char *end, uintmax_t *val)\n static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n {\n \tif (value && *value) {\n \t\tchar *end;\n \t\tintmax_t val;\n \t\tuintmax_t uval;\n \t\tuintmax_t factor = 1;\n\n \t\terrno = 0;\n \t\tval = strtoimax(value, &end, 0);\n \t\tif (errno == ERANGE)\n \t\t\treturn 0;\n \t\tif (!parse_unit_factor(end, &factor)) {\n \t\t\terrno = EINVAL;\n \t\t\treturn 0;\n \t\t}\n \t\tuval = val < 0 ? -val : val;\n-\t\tuval *= factor;\n-\t\tif (uval > max || (val < 0 ? -val : val) > uval) {\n+\t\tif (unsigned_mult_overflows(factor, uval) ||\n+\t\t factor * uval > max) {\n \t\t\terrno = ERANGE;\n \t\t\treturn 0;\n \t\t}\n \t\tval *= factor;\n \t\t*ret = val;\n \t\treturn 1;\n \t}\n \terrno = EINVAL;\n \treturn 0;\n }\n@@ -886,26 +886,27 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n {\n \tif (value && *value) {\n \t\tchar *end;\n \t\tuintmax_t val;\n-\t\tuintmax_t oldval;\n+\t\tuintmax_t factor = 1;\n\n \t\terrno = 0;\n \t\tval = strtoumax(value, &end, 0);\n \t\tif (errno == ERANGE)\n \t\t\treturn 0;\n-\t\toldval = val;\n-\t\tif (!parse_unit_factor(end, &val)) {\n+\t\tif (!parse_unit_factor(end, &factor)) {\n \t\t\terrno = EINVAL;\n \t\t\treturn 0;\n \t\t}\n-\t\tif (val > max || oldval > val) {\n+\t\tif (unsigned_mult_overflows(factor, val) ||\n+\t\t factor * val > max) {\n \t\t\terrno = ERANGE;\n \t\t\treturn 0;\n \t\t}\n+\t\tval *= factor;\n \t\t*ret = val;\n \t\treturn 1;\n \t}\n \terrno = EINVAL;\n \treturn 0;\n }\n--\n2.22.0\n"},{"id":"377809","messageId":"20bc9ec5-6e69-57aa-7347-4b7eaa8c8c1f@web.de","threadId":"51300","inReplyTo":"xmqq7e9ltbe2.fsf@gitster-ct.c.googlers.com","subject":"[PATCH v2 2/3] config: don't multiply in parse_unit_factor()","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-22T10:03:36Z","receivedAt":"2019-06-22T10:03:42Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"parse_unit_factor() multiplies the number that is passed to it with the\nvalue of a recognized unit factor (K, M or G for 2^10, 2^20 and 2^30,\nrespectively). All callers pass in 1 as a number, though, which allows\nthem to check the actual multiplication for overflow before they are\ndoing it themselves.\n\nIgnore the passed in number and don't multiply, as this feature of\nparse_unit_factor() is not used anymore. Rename the output parameter to\nreflect that it's not about the end result anymore, but just about the\nunit factor.\n\nSuggested-by: Johannes Schindelin \nSigned-off-by: Rene Scharfe \n---\n config.c | 16 +++++++++-------\n 1 file changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 3c00369ba8..a8bd1d821e 100644\n--- a/config.c\n+++ b/config.c\n@@ -834,20 +834,22 @@ static int git_parse_source(config_fn_t fn, void *data,\n \treturn error_return;\n }\n\n-static int parse_unit_factor(const char *end, uintmax_t *val)\n+static int parse_unit_factor(const char *end, uintmax_t *factor)\n {\n-\tif (!*end)\n+\tif (!*end) {\n+\t\t*factor = 1;\n \t\treturn 1;\n+\t}\n \telse if (!strcasecmp(end, \"k\")) {\n-\t\t*val *= 1024;\n+\t\t*factor = 1024;\n \t\treturn 1;\n \t}\n \telse if (!strcasecmp(end, \"m\")) {\n-\t\t*val *= 1024 * 1024;\n+\t\t*factor = 1024 * 1024;\n \t\treturn 1;\n \t}\n \telse if (!strcasecmp(end, \"g\")) {\n-\t\t*val *= 1024 * 1024 * 1024;\n+\t\t*factor = 1024 * 1024 * 1024;\n \t\treturn 1;\n \t}\n \treturn 0;\n@@ -859,7 +861,7 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n \t\tchar *end;\n \t\tintmax_t val;\n \t\tuintmax_t uval;\n-\t\tuintmax_t factor = 1;\n+\t\tuintmax_t factor;\n\n \t\terrno = 0;\n \t\tval = strtoimax(value, &end, 0);\n@@ -888,7 +890,7 @@ static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n \tif (value && *value) {\n \t\tchar *end;\n \t\tuintmax_t val;\n-\t\tuintmax_t factor = 1;\n+\t\tuintmax_t factor;\n\n \t\terrno = 0;\n \t\tval = strtoumax(value, &end, 0);\n--\n2.22.0\n"},{"id":"377810","messageId":"7a0ea32c-f480-4aa9-d258-68a62373d916@web.de","threadId":"51300","inReplyTo":"xmqq7e9ltbe2.fsf@gitster-ct.c.googlers.com","subject":"[PATCH v2 3/3] config: simplify parsing of unit factors","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-22T10:03:40Z","receivedAt":"2019-06-22T10:03:47Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Just return the value of the factor or zero for unrecognized strings\ninstead of using an output reference and a separate return value to\nindicate success. This is shorter and simpler.\n\nIt basically reverts that function to before c8deb5a146 (\"Improve error\nmessages when int/long cannot be parsed from config\", 2007-12-25), while\nkeeping the better messages, so restore its old name, get_unit_factor(),\nas well.\n\nSigned-off-by: Rene Scharfe \n---\nChange from v1: The \"else\" is kept in each branch, even though it's not\nneeded, to match the original code from before c8deb5a146. Other than\nthat this series arrives at the same end result. Patch 3 can be\ndropped easily if it's not convincing.\n\n config.c | 30 ++++++++++++------------------\n 1 file changed, 12 insertions(+), 18 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex a8bd1d821e..26196bdccf 100644\n--- a/config.c\n+++ b/config.c\n@@ -834,24 +834,16 @@ static int git_parse_source(config_fn_t fn, void *data,\n \treturn error_return;\n }\n\n-static int parse_unit_factor(const char *end, uintmax_t *factor)\n+static uintmax_t get_unit_factor(const char *end)\n {\n-\tif (!*end) {\n-\t\t*factor = 1;\n+\tif (!*end)\n \t\treturn 1;\n-\t}\n-\telse if (!strcasecmp(end, \"k\")) {\n-\t\t*factor = 1024;\n-\t\treturn 1;\n-\t}\n-\telse if (!strcasecmp(end, \"m\")) {\n-\t\t*factor = 1024 * 1024;\n-\t\treturn 1;\n-\t}\n-\telse if (!strcasecmp(end, \"g\")) {\n-\t\t*factor = 1024 * 1024 * 1024;\n-\t\treturn 1;\n-\t}\n+\telse if (!strcasecmp(end, \"k\"))\n+\t\treturn 1024;\n+\telse if (!strcasecmp(end, \"m\"))\n+\t\treturn 1024 * 1024;\n+\telse if (!strcasecmp(end, \"g\"))\n+\t\treturn 1024 * 1024 * 1024;\n \treturn 0;\n }\n\n@@ -867,7 +859,8 @@ static int git_parse_signed(const char *value, intmax_t *ret, intmax_t max)\n \t\tval = strtoimax(value, &end, 0);\n \t\tif (errno == ERANGE)\n \t\t\treturn 0;\n-\t\tif (!parse_unit_factor(end, &factor)) {\n+\t\tfactor = get_unit_factor(end);\n+\t\tif (!factor) {\n \t\t\terrno = EINVAL;\n \t\t\treturn 0;\n \t\t}\n@@ -896,7 +889,8 @@ static int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max)\n \t\tval = strtoumax(value, &end, 0);\n \t\tif (errno == ERANGE)\n \t\t\treturn 0;\n-\t\tif (!parse_unit_factor(end, &factor)) {\n+\t\tfactor = get_unit_factor(end);\n+\t\tif (!factor) {\n \t\t\terrno = EINVAL;\n \t\t\treturn 0;\n \t\t}\n--\n2.22.0\n"},{"id":"378045","messageId":"20190626000329.32475-1-avarab@gmail.com","threadId":"51300","inReplyTo":"87r27u8pie.fsf@evledraar.gmail.com","subject":"[RFC/PATCH 0/7] grep: move from kwset to optional PCRE v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-26T00:03:22Z","receivedAt":"2019-06-26T00:03:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This speeds things up a lot, but as shown in the patches & tests\nchanged modifies the behavior where we have \\0 in *patterns* (only\npossible with 'grep -f ').\n\nI'd like to go down this route because it makes dropping kwset a lot\neasier, and I don't think bending over backwards to support these \\0\npatterns is worth it.\n\nBut maybe others disagree, so I wanted to send what I had before I\ntried tackling the pickaxe code. There I figured I'd just make -G's\nERE be a PCRE if we had the PCRE v2 backend, since unlike \"grep\"'s\ndefault BRE the ERE syntax is mostly a subset of PCRE, but again\nothers might thing that's too aggressive and would prefer to keep the\ndistinction, only using PCRE there in place of our current use of\nkwset.\n\nÆvar Arnfjörð Bjarmason (7):\n grep: inline the return value of a function call used only once\n grep tests: move \"grep binary\" alongside the rest\n grep tests: move binary pattern tests into their own file\n grep: make the behavior for \\0 in patterns sane\n grep: drop support for \\0 in --fixed-strings \n grep: remove the kwset optimization\n grep: use PCRE v2 for optimized fixed-string search\n\n Documentation/git-grep.txt | 17 +++\n grep.c | 103 ++++++--------\n grep.h | 2 -\n ...a1.sh => t7008-filter-branch-null-sha1.sh} | 0\n ...08-grep-binary.sh => t7815-grep-binary.sh} | 101 --------------\n t/t7816-grep-binary-pattern.sh | 127 ++++++++++++++++++\n 6 files changed, 183 insertions(+), 167 deletions(-)\n rename t/{t7009-filter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} (100%)\n rename t/{t7008-grep-binary.sh => t7815-grep-binary.sh} (55%)\n create mode 100755 t/t7816-grep-binary-pattern.sh\n\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378046","messageId":"20190626000329.32475-2-avarab@gmail.com","threadId":"51300","inReplyTo":"87r27u8pie.fsf@evledraar.gmail.com","subject":"[RFC/PATCH 1/7] grep: inline the return value of a function call used only once","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-26T00:03:23Z","receivedAt":"2019-06-26T00:03:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Since e944d9d932 (\"grep: rewrite an if/else condition to avoid\nduplicate expression\", 2016-06-25) the \"ascii_only\" variable has only\nbeen used once in compile_regexp(), let's just inline it there.\n\nThis makes the code easier to read, and might make it marginally\nfaster depending on compiler optimizations.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex f7c3a5803e..d3e6111c46 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -650,13 +650,11 @@ static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n-\tint ascii_only;\n \tint err;\n \tint regflags = REG_NEWLINE;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n-\tascii_only = !has_non_ascii(p->pattern);\n \n \t/*\n \t * Even when -F (fixed) asks us to do a non-regexp search, we\n@@ -673,7 +671,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tif (opt->fixed ||\n \t has_null(p->pattern, p->patternlen) ||\n \t is_fixed(p->pattern, p->patternlen))\n-\t\tp->fixed = !p->ignore_case || ascii_only;\n+\t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n \n \tif (p->fixed) {\n \t\tp->kws = kwsalloc(p->ignore_case ? tolower_trans_tbl : NULL);\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378047","messageId":"20190626000329.32475-3-avarab@gmail.com","threadId":"51300","inReplyTo":"87r27u8pie.fsf@evledraar.gmail.com","subject":"[RFC/PATCH 2/7] grep tests: move \"grep binary\" alongside the rest","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-26T00:03:24Z","receivedAt":"2019-06-26T00:04:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the \"grep binary\" test case added in aca20dd558 (\"grep: add test\nscript for binary file handling\", 2010-05-22) so that it lives\nalongside the rest of the \"grep\" tests in t781*. This would have left\na gap in the t/700* namespace, so move a \"filter-branch\" test down,\nleaving the \"t7010-setup.sh\" test as the next one after that.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ...ilter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} | 0\n t/{t7008-grep-binary.sh => t7815-grep-binary.sh} | 0\n 2 files changed, 0 insertions(+), 0 deletions(-)\n rename t/{t7009-filter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} (100%)\n rename t/{t7008-grep-binary.sh => t7815-grep-binary.sh} (100%)\n\ndiff --git a/t/t7009-filter-branch-null-sha1.sh b/t/t7008-filter-branch-null-sha1.sh\nsimilarity index 100%\nrename from t/t7009-filter-branch-null-sha1.sh\nrename to t/t7008-filter-branch-null-sha1.sh\ndiff --git a/t/t7008-grep-binary.sh b/t/t7815-grep-binary.sh\nsimilarity index 100%\nrename from t/t7008-grep-binary.sh\nrename to t/t7815-grep-binary.sh\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378048","messageId":"20190626000329.32475-5-avarab@gmail.com","threadId":"51300","inReplyTo":"87r27u8pie.fsf@evledraar.gmail.com","subject":"[RFC/PATCH 4/7] grep: make the behavior for \\0 in patterns sane","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-26T00:03:26Z","receivedAt":"2019-06-26T00:04:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The behavior of \"grep\" when patterns contained \"\\0\" has always been\nhaphazard, and has served the vagaries of the implementation more than\nanything else. A \"\\0\" in a pattern can only be provided via \"-f\n\", and since pickaxe (log search) has no such flag \"\\0\" in\npatterns has only ever been supported by \"grep\".\n\nSince 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) patterns containing\n\"\\0\" were considered fixed. In 966be95549 (\"grep: add tests to fix\nblind spots with \\0 patterns\", 2017-05-20) I added tests for this\nbehavior.\n\nChange the behavior to do the obvious thing, i.e. don't silently\ndiscard a regex pattern and make it implicitly fixed just because it\ncontains a \\0. Instead die if e.g. --basic-regexp is combined with\nsuch a pattern.\n\nThis is desired because from a user's point of view it's the obvious\nthing to do. Whether we support BRE/ERE/Perl syntax is different from\nwhether our implementation is limited by C-strings. These patterns are\nobscure enough that I think this behavior change is OK, especially\nsince we never documented the old behavior.\n\nDoing this also makes it easier to replace the kwset backend with\nsomething else, since we'll no longer strictly need it for anything we\ncan't easily use another fixed-string backend for.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/git-grep.txt | 17 ++++\n grep.c | 23 ++---\n t/t7816-grep-binary-pattern.sh | 159 ++++++++++++++++++---------------\n 3 files changed, 110 insertions(+), 89 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 2d27969057..c89fb569e3 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -271,6 +271,23 @@ providing this option will cause it to die.\n \n -f ::\n \tRead patterns from , one per line.\n++\n+Passing the pattern via allows for providing a search pattern\n+containing a \\0.\n++\n+Not all pattern types support patterns containing \\0. Git will error\n+out if a given pattern type can't support such a pattern. The\n+`--perl-regexp` pattern type when compiled against the PCRE v2 backend\n+has the widest support for these types of patterns.\n++\n+In versions of Git before 2.23.0 patterns containing \\0 would be\n+silently considered fixed. This was never documented, there were also\n+odd and undocumented interactions between e.g. non-ASCII patterns\n+containing \\0 and `--ignore-case`.\n++\n+In future versions we may learn to support patterns containing \\0 for\n+more search backends, until then we'll die when the pattern type in\n+question doesn't support them.\n \n -e::\n \tThe next parameter is the pattern. This option has to be\ndiff --git a/grep.c b/grep.c\nindex d3e6111c46..261bd3a342 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -368,18 +368,6 @@ static int is_fixed(const char *s, size_t len)\n \treturn 1;\n }\n \n-static int has_null(const char *s, size_t len)\n-{\n-\t/*\n-\t * regcomp cannot accept patterns with NULs so when using it\n-\t * we consider any pattern containing a NUL fixed.\n-\t */\n-\tif (memchr(s, 0, len))\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n #ifdef USE_LIBPCRE1\n static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n {\n@@ -668,9 +656,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t * simple string match using kws. p->fixed tells us if we\n \t * want to use kws.\n \t */\n-\tif (opt->fixed ||\n-\t has_null(p->pattern, p->patternlen) ||\n-\t is_fixed(p->pattern, p->patternlen))\n+\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n \n \tif (p->fixed) {\n@@ -678,7 +664,12 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n \t\tkwsprep(p->kws);\n \t\treturn;\n-\t} else if (opt->fixed) {\n+\t}\n+\n+\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n+\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n+\n+\tif (opt->fixed) {\n \t\t/*\n \t\t * We come here when the pattern has the non-ascii\n \t\t * characters we cannot case-fold, and asked to\ndiff --git a/t/t7816-grep-binary-pattern.sh b/t/t7816-grep-binary-pattern.sh\nindex 4060dbd679..9e09bd5d6a 100755\n--- a/t/t7816-grep-binary-pattern.sh\n+++ b/t/t7816-grep-binary-pattern.sh\n@@ -2,113 +2,126 @@\n \n test_description='git grep with a binary pattern files'\n \n-. ./test-lib.sh\n+. ./lib-gettext.sh\n \n-nul_match () {\n+nul_match_internal () {\n \tmatches=$1\n-\tflags=$2\n-\tpattern=$3\n+\tprereqs=$2\n+\tlc_all=$3\n+\textra_flags=$4\n+\tflags=$5\n+\tpattern=$6\n \tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n \n \tif test \"$matches\" = 1\n \tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\ttest_expect_success $prereqs \"LC_ALL='$lc_all' git grep $extra_flags -f f $flags '$pattern_human' a\" \"\n \t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n+\t\t\tLC_ALL='$lc_all' git grep $extra_flags -f f $flags a\n \t\t\"\n \telif test \"$matches\" = 0\n \tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\ttest_expect_success $prereqs \"LC_ALL='$lc_all' git grep $extra_flags -f f $flags '$pattern_human' a\" \"\n+\t\t\t>stderr &&\n \t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\ttest_must_fail env LC_ALL=\\\"$lc_all\\\" git grep $extra_flags -f f $flags a 2>stderr &&\n+\t\t\ttest_i18ngrep ! 'This is only supported with -P under PCRE v2' stderr\n \t\t\"\n-\telif test \"$matches\" = T1\n+\telif test \"$matches\" = P\n \tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\ttest_expect_success $prereqs \"error, PCRE v2 only: LC_ALL='$lc_all' git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\t>stderr &&\n \t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = T0\n-\tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\ttest_must_fail env LC_ALL=\\\"$lc_all\\\" git grep -f f $flags a 2>stderr &&\n+\t\t\ttest_i18ngrep 'This is only supported with -P under PCRE v2' stderr\n \t\t\"\n \telse\n \t\ttest_expect_success \"PANIC: Test framework error. Unknown matches value $matches\" 'false'\n \tfi\n }\n \n+nul_match () {\n+\tmatches=$1\n+\tmatches_pcre2=$2\n+\tmatches_pcre2_locale=$3\n+\tflags=$4\n+\tpattern=$5\n+\tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n+\n+\tnul_match_internal \"$matches\" \"\" \"C\" \"\" \"$flags\" \"$pattern\"\n+\tnul_match_internal \"$matches_pcre2\" \"LIBPCRE2\" \"C\" \"-P\" \"$flags\" \"$pattern\"\n+\tnul_match_internal \"$matches_pcre2_locale\" \"LIBPCRE2,GETTEXT_LOCALE\" \"$is_IS_locale\" \"-P\" \"$flags\" \"$pattern\"\n+}\n+\n test_expect_success 'setup' \"\n \techo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&\n \tgit add a &&\n \tgit commit -m.\n \"\n \n-nul_match 1 '-F' 'yQf'\n-nul_match 0 '-F' 'yQx'\n-nul_match 1 '-Fi' 'YQf'\n-nul_match 0 '-Fi' 'YQx'\n-nul_match 1 '' 'yQf'\n-nul_match 0 '' 'yQx'\n-nul_match 1 '' 'æQð'\n-nul_match 1 '-F' 'eQm[*]c'\n-nul_match 1 '-Fi' 'EQM[*]C'\n+# Simple fixed-string matching that can use kwset (no -i && non-ASCII)\n+nul_match 1 1 1 '-F' 'yQf'\n+nul_match 0 0 0 '-F' 'yQx'\n+nul_match 1 1 1 '-Fi' 'YQf'\n+nul_match 0 0 0 '-Fi' 'YQx'\n+nul_match 1 1 1 '' 'yQf'\n+nul_match 0 0 0 '' 'yQx'\n+nul_match 1 1 1 '' 'æQð'\n+nul_match 1 1 1 '-F' 'eQm[*]c'\n+nul_match 1 1 1 '-Fi' 'EQM[*]C'\n \n # Regex patterns that would match but shouldn't with -F\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-F' '[y]Qf'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '-Fi' '[Y]QF'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-F' '[æ]Qð'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '-Fi' '[Æ]QÐ'\n+nul_match 0 0 0 '-F' 'yQ[f]'\n+nul_match 0 0 0 '-F' '[y]Qf'\n+nul_match 0 0 0 '-Fi' 'YQ[F]'\n+nul_match 0 0 0 '-Fi' '[Y]QF'\n+nul_match 0 0 0 '-F' 'æQ[ð]'\n+nul_match 0 0 0 '-F' '[æ]Qð'\n \n-# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n-# patterns case-insensitively.\n-nul_match T1 '-i' 'ÆQÐ'\n+# The -F kwset codepath can't handle -i && non-ASCII...\n+nul_match P 1 1 '-i' '[æ]Qð'\n \n-# \\0 implicitly disables regexes. This is an undocumented internal\n-# limitation.\n-nul_match T1 '' 'yQ[f]'\n-nul_match T1 '' '[y]Qf'\n-nul_match T1 '-i' 'YQ[F]'\n-nul_match T1 '-i' '[Y]Qf'\n-nul_match T1 '' 'æQ[ð]'\n-nul_match T1 '' '[æ]Qð'\n-nul_match T1 '-i' 'ÆQ[Ð]'\n+# ...PCRE v2 only matches non-ASCII with -i casefolding under UTF-8\n+# semantics\n+nul_match P P P '-Fi' 'ÆQ[Ð]'\n+nul_match P 0 1 '-i' 'ÆQ[Ð]'\n+nul_match P 0 1 '-i' '[Æ]QÐ'\n+nul_match P 0 1 '-i' '[Æ]Qð'\n+nul_match P 0 1 '-i' 'ÆQÐ'\n \n-# ... because of \\0 implicitly disabling regexes regexes that\n-# should/shouldn't match don't do the right thing.\n-nul_match T1 '' 'eQm.*cQ'\n-nul_match T1 '-i' 'EQM.*cQ'\n-nul_match T0 '' 'eQm[*]c'\n-nul_match T0 '-i' 'EQM[*]C'\n+# \\0 in regexes can only work with -P & PCRE v2\n+nul_match P 1 1 '' 'yQ[f]'\n+nul_match P 1 1 '' '[y]Qf'\n+nul_match P 1 1 '-i' 'YQ[F]'\n+nul_match P 1 1 '-i' '[Y]Qf'\n+nul_match P 1 1 '' 'æQ[ð]'\n+nul_match P 1 1 '' '[æ]Qð'\n+nul_match P 0 1 '-i' 'ÆQ[Ð]'\n+nul_match P 1 1 '' 'eQm.*cQ'\n+nul_match P 1 1 '-i' 'EQM.*cQ'\n+nul_match P 0 0 '' 'eQm[*]c'\n+nul_match P 0 0 '-i' 'EQM[*]C'\n \n-# Due to the REG_STARTEND extension when kwset() is disabled on -i &\n-# non-ASCII the string will be matched in its entirety, but the\n-# pattern will be cut off at the first \\0.\n-nul_match 0 '-i' 'NOMATCHQð'\n-nul_match T0 '-i' '[Æ]QNOMATCH'\n-nul_match T0 '-i' '[æ]QNOMATCH'\n-# Matches, but for the wrong reasons, just stops at [æ]\n-nul_match 1 '-i' '[Æ]Qð'\n-nul_match 1 '-i' '[æ]Qð'\n+# Assert that we're using REG_STARTEND and the pattern doesn't match\n+# just because it's cut off at the first \\0.\n+nul_match 0 0 0 '-i' 'NOMATCHQð'\n+nul_match P 0 0 '-i' '[Æ]QNOMATCH'\n+nul_match P 0 0 '-i' '[æ]QNOMATCH'\n \n # Ensure that the matcher doesn't regress to something that stops at\n # \\0\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '' 'yQNOMATCH'\n-nul_match 0 '' 'QNOMATCH'\n-nul_match 0 '-i' 'YQNOMATCH'\n-nul_match 0 '-i' 'QNOMATCH'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '' 'yQNÓMATCH'\n-nul_match 0 '' 'QNÓMATCH'\n-nul_match 0 '-i' 'YQNÓMATCH'\n-nul_match 0 '-i' 'QNÓMATCH'\n+nul_match 0 0 0 '-F' 'yQ[f]'\n+nul_match 0 0 0 '-Fi' 'YQ[F]'\n+nul_match 0 0 0 '' 'yQNOMATCH'\n+nul_match 0 0 0 '' 'QNOMATCH'\n+nul_match 0 0 0 '-i' 'YQNOMATCH'\n+nul_match 0 0 0 '-i' 'QNOMATCH'\n+nul_match 0 0 0 '-F' 'æQ[ð]'\n+nul_match P P P '-Fi' 'ÆQ[Ð]'\n+nul_match P 0 1 '-i' 'ÆQ[Ð]'\n+nul_match 0 0 0 '' 'yQNÓMATCH'\n+nul_match 0 0 0 '' 'QNÓMATCH'\n+nul_match 0 0 0 '-i' 'YQNÓMATCH'\n+nul_match 0 0 0 '-i' 'QNÓMATCH'\n \n test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378049","messageId":"20190626000329.32475-4-avarab@gmail.com","threadId":"51300","inReplyTo":"87r27u8pie.fsf@evledraar.gmail.com","subject":"[RFC/PATCH 3/7] grep tests: move binary pattern tests into their own file","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-26T00:03:25Z","receivedAt":"2019-06-26T00:04:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the tests for \"-f \" where \"\" contains a \"\\0\" pattern\ninto their own file. I added most of these tests in 966be95549 (\"grep:\nadd tests to fix blind spots with \\0 patterns\", 2017-05-20).\n\nWhether a regex engine supports matching binary content is very\ndifferent from whether it matches binary patterns. Since\n2f8952250a (\"regex: add regexec_buf() that can work on a non\nNUL-terminated string\", 2016-09-21) we've required REG_STARTEND of our\nregex engines so we can match binary content, but only the PCRE v2\nengine can sensibly match binary patterns.\n\nSince 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) we've been punting\npatterns containing \"\\0\" and considering them fixed, except in cases\nwhere \"--ignore-case\" is provided and they're non-ASCII, see\n5c1ebcca4d (\"grep/icase: avoid kwsset on literal non-ascii strings\",\n2016-06-25). Subsequent commits will change this behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t7815-grep-binary.sh | 101 -----------------------------\n t/t7816-grep-binary-pattern.sh | 114 +++++++++++++++++++++++++++++++++\n 2 files changed, 114 insertions(+), 101 deletions(-)\n create mode 100755 t/t7816-grep-binary-pattern.sh\n\ndiff --git a/t/t7815-grep-binary.sh b/t/t7815-grep-binary.sh\nindex 2d87c49b75..90ebb64f46 100755\n--- a/t/t7815-grep-binary.sh\n+++ b/t/t7815-grep-binary.sh\n@@ -4,41 +4,6 @@ test_description='git grep in binary files'\n \n . ./test-lib.sh\n \n-nul_match () {\n-\tmatches=$1\n-\tflags=$2\n-\tpattern=$3\n-\tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n-\n-\tif test \"$matches\" = 1\n-\tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = 0\n-\tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = T1\n-\tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = T0\n-\tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n-\t\t\"\n-\telse\n-\t\ttest_expect_success \"PANIC: Test framework error. Unknown matches value $matches\" 'false'\n-\tfi\n-}\n-\n test_expect_success 'setup' \"\n \techo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&\n \tgit add a &&\n@@ -102,72 +67,6 @@ test_expect_failure 'git grep .fi a' '\n \tgit grep .fi a\n '\n \n-nul_match 1 '-F' 'yQf'\n-nul_match 0 '-F' 'yQx'\n-nul_match 1 '-Fi' 'YQf'\n-nul_match 0 '-Fi' 'YQx'\n-nul_match 1 '' 'yQf'\n-nul_match 0 '' 'yQx'\n-nul_match 1 '' 'æQð'\n-nul_match 1 '-F' 'eQm[*]c'\n-nul_match 1 '-Fi' 'EQM[*]C'\n-\n-# Regex patterns that would match but shouldn't with -F\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-F' '[y]Qf'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '-Fi' '[Y]QF'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-F' '[æ]Qð'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '-Fi' '[Æ]QÐ'\n-\n-# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n-# patterns case-insensitively.\n-nul_match T1 '-i' 'ÆQÐ'\n-\n-# \\0 implicitly disables regexes. This is an undocumented internal\n-# limitation.\n-nul_match T1 '' 'yQ[f]'\n-nul_match T1 '' '[y]Qf'\n-nul_match T1 '-i' 'YQ[F]'\n-nul_match T1 '-i' '[Y]Qf'\n-nul_match T1 '' 'æQ[ð]'\n-nul_match T1 '' '[æ]Qð'\n-nul_match T1 '-i' 'ÆQ[Ð]'\n-\n-# ... because of \\0 implicitly disabling regexes regexes that\n-# should/shouldn't match don't do the right thing.\n-nul_match T1 '' 'eQm.*cQ'\n-nul_match T1 '-i' 'EQM.*cQ'\n-nul_match T0 '' 'eQm[*]c'\n-nul_match T0 '-i' 'EQM[*]C'\n-\n-# Due to the REG_STARTEND extension when kwset() is disabled on -i &\n-# non-ASCII the string will be matched in its entirety, but the\n-# pattern will be cut off at the first \\0.\n-nul_match 0 '-i' 'NOMATCHQð'\n-nul_match T0 '-i' '[Æ]QNOMATCH'\n-nul_match T0 '-i' '[æ]QNOMATCH'\n-# Matches, but for the wrong reasons, just stops at [æ]\n-nul_match 1 '-i' '[Æ]Qð'\n-nul_match 1 '-i' '[æ]Qð'\n-\n-# Ensure that the matcher doesn't regress to something that stops at\n-# \\0\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '' 'yQNOMATCH'\n-nul_match 0 '' 'QNOMATCH'\n-nul_match 0 '-i' 'YQNOMATCH'\n-nul_match 0 '-i' 'QNOMATCH'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '' 'yQNÓMATCH'\n-nul_match 0 '' 'QNÓMATCH'\n-nul_match 0 '-i' 'YQNÓMATCH'\n-nul_match 0 '-i' 'QNÓMATCH'\n-\n test_expect_success 'grep respects binary diff attribute' '\n \techo text >t &&\n \tgit add t &&\ndiff --git a/t/t7816-grep-binary-pattern.sh b/t/t7816-grep-binary-pattern.sh\nnew file mode 100755\nindex 0000000000..4060dbd679\n--- /dev/null\n+++ b/t/t7816-grep-binary-pattern.sh\n@@ -0,0 +1,114 @@\n+#!/bin/sh\n+\n+test_description='git grep with a binary pattern files'\n+\n+. ./test-lib.sh\n+\n+nul_match () {\n+\tmatches=$1\n+\tflags=$2\n+\tpattern=$3\n+\tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n+\n+\tif test \"$matches\" = 1\n+\tthen\n+\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\tgit grep -f f $flags a\n+\t\t\"\n+\telif test \"$matches\" = 0\n+\tthen\n+\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\"\n+\telif test \"$matches\" = T1\n+\tthen\n+\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\tgit grep -f f $flags a\n+\t\t\"\n+\telif test \"$matches\" = T0\n+\tthen\n+\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\"\n+\telse\n+\t\ttest_expect_success \"PANIC: Test framework error. Unknown matches value $matches\" 'false'\n+\tfi\n+}\n+\n+test_expect_success 'setup' \"\n+\techo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&\n+\tgit add a &&\n+\tgit commit -m.\n+\"\n+\n+nul_match 1 '-F' 'yQf'\n+nul_match 0 '-F' 'yQx'\n+nul_match 1 '-Fi' 'YQf'\n+nul_match 0 '-Fi' 'YQx'\n+nul_match 1 '' 'yQf'\n+nul_match 0 '' 'yQx'\n+nul_match 1 '' 'æQð'\n+nul_match 1 '-F' 'eQm[*]c'\n+nul_match 1 '-Fi' 'EQM[*]C'\n+\n+# Regex patterns that would match but shouldn't with -F\n+nul_match 0 '-F' 'yQ[f]'\n+nul_match 0 '-F' '[y]Qf'\n+nul_match 0 '-Fi' 'YQ[F]'\n+nul_match 0 '-Fi' '[Y]QF'\n+nul_match 0 '-F' 'æQ[ð]'\n+nul_match 0 '-F' '[æ]Qð'\n+nul_match 0 '-Fi' 'ÆQ[Ð]'\n+nul_match 0 '-Fi' '[Æ]QÐ'\n+\n+# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n+# patterns case-insensitively.\n+nul_match T1 '-i' 'ÆQÐ'\n+\n+# \\0 implicitly disables regexes. This is an undocumented internal\n+# limitation.\n+nul_match T1 '' 'yQ[f]'\n+nul_match T1 '' '[y]Qf'\n+nul_match T1 '-i' 'YQ[F]'\n+nul_match T1 '-i' '[Y]Qf'\n+nul_match T1 '' 'æQ[ð]'\n+nul_match T1 '' '[æ]Qð'\n+nul_match T1 '-i' 'ÆQ[Ð]'\n+\n+# ... because of \\0 implicitly disabling regexes regexes that\n+# should/shouldn't match don't do the right thing.\n+nul_match T1 '' 'eQm.*cQ'\n+nul_match T1 '-i' 'EQM.*cQ'\n+nul_match T0 '' 'eQm[*]c'\n+nul_match T0 '-i' 'EQM[*]C'\n+\n+# Due to the REG_STARTEND extension when kwset() is disabled on -i &\n+# non-ASCII the string will be matched in its entirety, but the\n+# pattern will be cut off at the first \\0.\n+nul_match 0 '-i' 'NOMATCHQð'\n+nul_match T0 '-i' '[Æ]QNOMATCH'\n+nul_match T0 '-i' '[æ]QNOMATCH'\n+# Matches, but for the wrong reasons, just stops at [æ]\n+nul_match 1 '-i' '[Æ]Qð'\n+nul_match 1 '-i' '[æ]Qð'\n+\n+# Ensure that the matcher doesn't regress to something that stops at\n+# \\0\n+nul_match 0 '-F' 'yQ[f]'\n+nul_match 0 '-Fi' 'YQ[F]'\n+nul_match 0 '' 'yQNOMATCH'\n+nul_match 0 '' 'QNOMATCH'\n+nul_match 0 '-i' 'YQNOMATCH'\n+nul_match 0 '-i' 'QNOMATCH'\n+nul_match 0 '-F' 'æQ[ð]'\n+nul_match 0 '-Fi' 'ÆQ[Ð]'\n+nul_match 0 '' 'yQNÓMATCH'\n+nul_match 0 '' 'QNÓMATCH'\n+nul_match 0 '-i' 'YQNÓMATCH'\n+nul_match 0 '-i' 'QNÓMATCH'\n+\n+test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378050","messageId":"20190626000329.32475-6-avarab@gmail.com","threadId":"51300","inReplyTo":"87r27u8pie.fsf@evledraar.gmail.com","subject":"[RFC/PATCH 5/7] grep: drop support for \\0 in --fixed-strings ","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-26T00:03:27Z","receivedAt":"2019-06-26T00:04:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change \"-f \" to not support patterns with \"\\0\" in them under\n--fixed-strings, we'll now only support these under --perl-regexp with\nPCRE v2.\n\nA previous change to Documentation/git-grep.txt changed the\ndescription of \"-f \" to be vague enough as to not promise that\nthis would work, and by dropping support for this we make it a whole\nlot easier to move away from the kwset backend, which a subsequent\nchange will try to do.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 6 +--\n t/t7816-grep-binary-pattern.sh | 82 +++++++++++++++++-----------------\n 2 files changed, 44 insertions(+), 44 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 261bd3a342..14570c7ac1 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -644,6 +644,9 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \n+\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n+\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n+\n \t/*\n \t * Even when -F (fixed) asks us to do a non-regexp search, we\n \t * may not be able to correctly case-fold when -i\n@@ -666,9 +669,6 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\treturn;\n \t}\n \n-\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n-\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n-\n \tif (opt->fixed) {\n \t\t/*\n \t\t * We come here when the pattern has the non-ascii\ndiff --git a/t/t7816-grep-binary-pattern.sh b/t/t7816-grep-binary-pattern.sh\nindex 9e09bd5d6a..60bab291e4 100755\n--- a/t/t7816-grep-binary-pattern.sh\n+++ b/t/t7816-grep-binary-pattern.sh\n@@ -60,23 +60,23 @@ test_expect_success 'setup' \"\n \"\n \n # Simple fixed-string matching that can use kwset (no -i && non-ASCII)\n-nul_match 1 1 1 '-F' 'yQf'\n-nul_match 0 0 0 '-F' 'yQx'\n-nul_match 1 1 1 '-Fi' 'YQf'\n-nul_match 0 0 0 '-Fi' 'YQx'\n-nul_match 1 1 1 '' 'yQf'\n-nul_match 0 0 0 '' 'yQx'\n-nul_match 1 1 1 '' 'æQð'\n-nul_match 1 1 1 '-F' 'eQm[*]c'\n-nul_match 1 1 1 '-Fi' 'EQM[*]C'\n+nul_match P P P '-F' 'yQf'\n+nul_match P P P '-F' 'yQx'\n+nul_match P P P '-Fi' 'YQf'\n+nul_match P P P '-Fi' 'YQx'\n+nul_match P P 1 '' 'yQf'\n+nul_match P P 0 '' 'yQx'\n+nul_match P P 1 '' 'æQð'\n+nul_match P P P '-F' 'eQm[*]c'\n+nul_match P P P '-Fi' 'EQM[*]C'\n \n # Regex patterns that would match but shouldn't with -F\n-nul_match 0 0 0 '-F' 'yQ[f]'\n-nul_match 0 0 0 '-F' '[y]Qf'\n-nul_match 0 0 0 '-Fi' 'YQ[F]'\n-nul_match 0 0 0 '-Fi' '[Y]QF'\n-nul_match 0 0 0 '-F' 'æQ[ð]'\n-nul_match 0 0 0 '-F' '[æ]Qð'\n+nul_match P P P '-F' 'yQ[f]'\n+nul_match P P P '-F' '[y]Qf'\n+nul_match P P P '-Fi' 'YQ[F]'\n+nul_match P P P '-Fi' '[Y]QF'\n+nul_match P P P '-F' 'æQ[ð]'\n+nul_match P P P '-F' '[æ]Qð'\n \n # The -F kwset codepath can't handle -i && non-ASCII...\n nul_match P 1 1 '-i' '[æ]Qð'\n@@ -90,38 +90,38 @@ nul_match P 0 1 '-i' '[Æ]Qð'\n nul_match P 0 1 '-i' 'ÆQÐ'\n \n # \\0 in regexes can only work with -P & PCRE v2\n-nul_match P 1 1 '' 'yQ[f]'\n-nul_match P 1 1 '' '[y]Qf'\n-nul_match P 1 1 '-i' 'YQ[F]'\n-nul_match P 1 1 '-i' '[Y]Qf'\n-nul_match P 1 1 '' 'æQ[ð]'\n-nul_match P 1 1 '' '[æ]Qð'\n-nul_match P 0 1 '-i' 'ÆQ[Ð]'\n-nul_match P 1 1 '' 'eQm.*cQ'\n-nul_match P 1 1 '-i' 'EQM.*cQ'\n-nul_match P 0 0 '' 'eQm[*]c'\n-nul_match P 0 0 '-i' 'EQM[*]C'\n+nul_match P P 1 '' 'yQ[f]'\n+nul_match P P 1 '' '[y]Qf'\n+nul_match P P 1 '-i' 'YQ[F]'\n+nul_match P P 1 '-i' '[Y]Qf'\n+nul_match P P 1 '' 'æQ[ð]'\n+nul_match P P 1 '' '[æ]Qð'\n+nul_match P P 1 '-i' 'ÆQ[Ð]'\n+nul_match P P 1 '' 'eQm.*cQ'\n+nul_match P P 1 '-i' 'EQM.*cQ'\n+nul_match P P 0 '' 'eQm[*]c'\n+nul_match P P 0 '-i' 'EQM[*]C'\n \n # Assert that we're using REG_STARTEND and the pattern doesn't match\n # just because it's cut off at the first \\0.\n-nul_match 0 0 0 '-i' 'NOMATCHQð'\n-nul_match P 0 0 '-i' '[Æ]QNOMATCH'\n-nul_match P 0 0 '-i' '[æ]QNOMATCH'\n+nul_match P P 0 '-i' 'NOMATCHQð'\n+nul_match P P 0 '-i' '[Æ]QNOMATCH'\n+nul_match P P 0 '-i' '[æ]QNOMATCH'\n \n # Ensure that the matcher doesn't regress to something that stops at\n # \\0\n-nul_match 0 0 0 '-F' 'yQ[f]'\n-nul_match 0 0 0 '-Fi' 'YQ[F]'\n-nul_match 0 0 0 '' 'yQNOMATCH'\n-nul_match 0 0 0 '' 'QNOMATCH'\n-nul_match 0 0 0 '-i' 'YQNOMATCH'\n-nul_match 0 0 0 '-i' 'QNOMATCH'\n-nul_match 0 0 0 '-F' 'æQ[ð]'\n+nul_match P P P '-F' 'yQ[f]'\n+nul_match P P P '-Fi' 'YQ[F]'\n+nul_match P P 0 '' 'yQNOMATCH'\n+nul_match P P 0 '' 'QNOMATCH'\n+nul_match P P 0 '-i' 'YQNOMATCH'\n+nul_match P P 0 '-i' 'QNOMATCH'\n+nul_match P P P '-F' 'æQ[ð]'\n nul_match P P P '-Fi' 'ÆQ[Ð]'\n-nul_match P 0 1 '-i' 'ÆQ[Ð]'\n-nul_match 0 0 0 '' 'yQNÓMATCH'\n-nul_match 0 0 0 '' 'QNÓMATCH'\n-nul_match 0 0 0 '-i' 'YQNÓMATCH'\n-nul_match 0 0 0 '-i' 'QNÓMATCH'\n+nul_match P P 1 '-i' 'ÆQ[Ð]'\n+nul_match P P 0 '' 'yQNÓMATCH'\n+nul_match P P 0 '' 'QNÓMATCH'\n+nul_match P P 0 '-i' 'YQNÓMATCH'\n+nul_match P P 0 '-i' 'QNÓMATCH'\n \n test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378051","messageId":"20190626000329.32475-7-avarab@gmail.com","threadId":"51300","inReplyTo":"87r27u8pie.fsf@evledraar.gmail.com","subject":"[RFC/PATCH 6/7] grep: remove the kwset optimization","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-26T00:03:28Z","receivedAt":"2019-06-26T00:04:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A later change will replace this optimization with a different one,\nbut as removing it and running the tests demonstrates no grep\nsemantics depend on this backend anymore.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 63 +++-------------------------------------------------------\n grep.h | 2 --\n 2 files changed, 3 insertions(+), 62 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 14570c7ac1..4716217837 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -356,18 +356,6 @@ static NORETURN void compile_regexp_failed(const struct grep_pat *p,\n \tdie(\"%s'%s': %s\", where, p->pattern, error);\n }\n \n-static int is_fixed(const char *s, size_t len)\n-{\n-\tsize_t i;\n-\n-\tfor (i = 0; i < len; i++) {\n-\t\tif (is_regex_special(s[i]))\n-\t\t\treturn 0;\n-\t}\n-\n-\treturn 1;\n-}\n-\n #ifdef USE_LIBPCRE1\n static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n {\n@@ -643,38 +631,12 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n+\tp->fixed = opt->fixed;\n \n \tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n \t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n \n-\t/*\n-\t * Even when -F (fixed) asks us to do a non-regexp search, we\n-\t * may not be able to correctly case-fold when -i\n-\t * (ignore-case) is asked (in which case, we'll synthesize a\n-\t * regexp to match the pattern that matches regexp special\n-\t * characters literally, while ignoring case differences). On\n-\t * the other hand, even without -F, if the pattern does not\n-\t * have any regexp special characters and there is no need for\n-\t * case-folding search, we can internally turn it into a\n-\t * simple string match using kws. p->fixed tells us if we\n-\t * want to use kws.\n-\t */\n-\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n-\t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n-\n-\tif (p->fixed) {\n-\t\tp->kws = kwsalloc(p->ignore_case ? tolower_trans_tbl : NULL);\n-\t\tkwsincr(p->kws, p->pattern, p->patternlen);\n-\t\tkwsprep(p->kws);\n-\t\treturn;\n-\t}\n-\n \tif (opt->fixed) {\n-\t\t/*\n-\t\t * We come here when the pattern has the non-ascii\n-\t\t * characters we cannot case-fold, and asked to\n-\t\t * ignore-case.\n-\t\t */\n \t\tcompile_fixed_regexp(p, opt);\n \t\treturn;\n \t}\n@@ -1042,9 +1004,7 @@ void free_grep_patterns(struct grep_opt *opt)\n \t\tcase GREP_PATTERN: /* atom */\n \t\tcase GREP_PATTERN_HEAD:\n \t\tcase GREP_PATTERN_BODY:\n-\t\t\tif (p->kws)\n-\t\t\t\tkwsfree(p->kws);\n-\t\t\telse if (p->pcre1_regexp)\n+\t\t\tif (p->pcre1_regexp)\n \t\t\t\tfree_pcre1_regexp(p);\n \t\t\telse if (p->pcre2_pattern)\n \t\t\t\tfree_pcre2_pattern(p);\n@@ -1104,29 +1064,12 @@ static void show_name(struct grep_opt *opt, const char *name)\n \topt->output(opt, opt->null_following_name ? \"\\0\" : \"\\n\", 1);\n }\n \n-static int fixmatch(struct grep_pat *p, char *line, char *eol,\n-\t\t regmatch_t *match)\n-{\n-\tstruct kwsmatch kwsm;\n-\tsize_t offset = kwsexec(p->kws, line, eol - line, &kwsm);\n-\tif (offset == -1) {\n-\t\tmatch->rm_so = match->rm_eo = -1;\n-\t\treturn REG_NOMATCH;\n-\t} else {\n-\t\tmatch->rm_so = offset;\n-\t\tmatch->rm_eo = match->rm_so + kwsm.size[0];\n-\t\treturn 0;\n-\t}\n-}\n-\n static int patmatch(struct grep_pat *p, char *line, char *eol,\n \t\t regmatch_t *match, int eflags)\n {\n \tint hit;\n \n-\tif (p->fixed)\n-\t\thit = !fixmatch(p, line, eol, match);\n-\telse if (p->pcre1_regexp)\n+\tif (p->pcre1_regexp)\n \t\thit = !pcre1match(p, line, eol, match, eflags);\n \telse if (p->pcre2_pattern)\n \t\thit = !pcre2match(p, line, eol, match, eflags);\ndiff --git a/grep.h b/grep.h\nindex 1875880f37..90ca435aad 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -32,7 +32,6 @@ typedef int pcre2_compile_context;\n typedef int pcre2_match_context;\n typedef int pcre2_jit_stack;\n #endif\n-#include \"kwset.h\"\n #include \"thread-utils.h\"\n #include \"userdiff.h\"\n \n@@ -97,7 +96,6 @@ struct grep_pat {\n \tpcre2_match_context *pcre2_match_context;\n \tpcre2_jit_stack *pcre2_jit_stack;\n \tuint32_t pcre2_jit_on;\n-\tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\n \tunsigned word_regexp:1;\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378052","messageId":"20190626000329.32475-8-avarab@gmail.com","threadId":"51300","inReplyTo":"87r27u8pie.fsf@evledraar.gmail.com","subject":"[RFC/PATCH 7/7] grep: use PCRE v2 for optimized fixed-string search","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-26T00:03:29Z","receivedAt":"2019-06-26T00:04:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Bring back optimized fixed-string search for \"grep\", this time with\nPCRE v2 as an optional backend. As noted in [1] with kwset we were\nslower than PCRE v1 and v2 JIT with the kwset backend, so that\noptimization was counterproductive.\n\nThis brings back the optimization for \"-F\", without changing the\nsemantics of \"\\0\" in patterns. As seen in previous commits in this\nseries we could support it now, but I'd rather just leave that\nedge-case aside so the tests don't need to do one thing or the other\ndepending on what --fixed-strings backend we're using.\n\nI could also support the v1 backend here, but that would make the code\nmore complex, and I'd rather aim for simplicity here and in future\nchanges to the diffcore. We're not going to have someone who\nabsolutely must have faster search, but for whom building PCRE v2\nisn't acceptable.\n\n1. https://public-inbox.org/git/87v9x793qi.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 47 +++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 45 insertions(+), 2 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 4716217837..6b75d5be68 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -356,6 +356,18 @@ static NORETURN void compile_regexp_failed(const struct grep_pat *p,\n \tdie(\"%s'%s': %s\", where, p->pattern, error);\n }\n \n+static int is_fixed(const char *s, size_t len)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (is_regex_special(s[i]))\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n #ifdef USE_LIBPCRE1\n static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n {\n@@ -602,7 +614,6 @@ static int pcre2match(struct grep_pat *p, const char *line, const char *eol,\n static void free_pcre2_pattern(struct grep_pat *p)\n {\n }\n-#endif /* !USE_LIBPCRE2 */\n \n static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n@@ -623,11 +634,13 @@ static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tcompile_regexp_failed(p, errbuf);\n \t}\n }\n+#endif /* !USE_LIBPCRE2 */\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tint err;\n \tint regflags = REG_NEWLINE;\n+\tint pat_is_fixed;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n@@ -636,8 +649,38 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n \t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n \n-\tif (opt->fixed) {\n+\tpat_is_fixed = is_fixed(p->pattern, p->patternlen);\n+\tif (opt->fixed || pat_is_fixed) {\n+#ifdef USE_LIBPCRE2\n+\t\topt->pcre2 = 1;\n+\t\tif (pat_is_fixed) {\n+\t\t\tcompile_pcre2_pattern(p, opt);\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * E.g. t7811-grep-open.sh relies on the\n+\t\t\t * pattern being restored, and unfortunately\n+\t\t\t * there's no PCRE compile flag for \"this is\n+\t\t\t * fixed\", so we need to munge it to\n+\t\t\t * \"\\Q\\E\".\n+\t\t\t */\n+\t\t\tchar *old_pattern = p->pattern;\n+\t\t\tsize_t old_patternlen = p->patternlen;\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\t\tstrbuf_add(&sb, \"\\\\Q\", 2);\n+\t\t\tstrbuf_add(&sb, p->pattern, p->patternlen);\n+\t\t\tstrbuf_add(&sb, \"\\\\E\", 2);\n+\n+\t\t\tp->pattern = sb.buf;\n+\t\t\tp->patternlen = sb.len;\n+\t\t\tcompile_pcre2_pattern(p, opt);\n+\t\t\tp->pattern = old_pattern;\n+\t\t\tp->patternlen = old_patternlen;\n+\t\t\tstrbuf_release(&sb);\n+\t\t}\n+#else\n \t\tcompile_fixed_regexp(p, opt);\n+#endif\n \t\treturn;\n \t}\n \n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378063","messageId":"nycvar.QRO.7.76.6.1906261601270.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"Re: [RFC/PATCH 0/7] grep: move from kwset to optional PCRE v2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-26T14:02:33Z","receivedAt":"2019-06-26T14:02:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Wed, 26 Jun 2019, Ævar Arnfjörð Bjarmason wrote:\n\n> This speeds things up a lot, but as shown in the patches & tests\n> changed modifies the behavior where we have \\0 in *patterns* (only\n> possible with 'grep -f ').\n\nI agree that it is not worth a lot to care about NULs in search patterns.\n\nSo I am in favor of the goal of this patch series.\n\nCiao,\nDscho\n"},{"id":"378064","messageId":"nycvar.QRO.7.76.6.1906261603380.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"20190626000329.32475-3-avarab@gmail.com","subject":"Re: [RFC/PATCH 2/7] grep tests: move \"grep binary\" alongside the rest","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-26T14:05:41Z","receivedAt":"2019-06-26T14:05:38Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Wed, 26 Jun 2019, Ævar Arnfjörð Bjarmason wrote:\n\n> Move the \"grep binary\" test case added in aca20dd558 (\"grep: add test\n> script for binary file handling\", 2010-05-22) so that it lives\n> alongside the rest of the \"grep\" tests in t781*. This would have left\n> a gap in the t/700* namespace, so move a \"filter-branch\" test down,\n> leaving the \"t7010-setup.sh\" test as the next one after that.\n\nI would be totally fine with having gaps.\n\nCiao,\nDscho\n"},{"id":"378065","messageId":"nycvar.QRO.7.76.6.1906261610310.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"20190626000329.32475-8-avarab@gmail.com","subject":"Re: [RFC/PATCH 7/7] grep: use PCRE v2 for optimized fixed-string search","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-26T14:13:45Z","receivedAt":"2019-06-26T14:15:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Wed, 26 Jun 2019, Ævar Arnfjörð Bjarmason wrote:\n\n> Bring back optimized fixed-string search for \"grep\", this time with\n> PCRE v2 as an optional backend. As noted in [1] with kwset we were\n> slower than PCRE v1 and v2 JIT with the kwset backend, so that\n> optimization was counterproductive.\n>\n> This brings back the optimization for \"-F\", without changing the\n> semantics of \"\\0\" in patterns. As seen in previous commits in this\n> series we could support it now, but I'd rather just leave that\n> edge-case aside so the tests don't need to do one thing or the other\n> depending on what --fixed-strings backend we're using.\n\nNice. Very, very nice.\n\n> I could also support the v1 backend here, but that would make the code\n> more complex, and I'd rather aim for simplicity here and in future\n> changes to the diffcore. We're not going to have someone who\n> absolutely must have faster search, but for whom building PCRE v2\n> isn't acceptable.\n\nI could not agree more.\n\n> diff --git a/grep.c b/grep.c\n> index 4716217837..6b75d5be68 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -356,6 +356,18 @@ static NORETURN void compile_regexp_failed(const struct grep_pat *p,\n> \tdie(\"%s'%s': %s\", where, p->pattern, error);\n> }\n>\n> +static int is_fixed(const char *s, size_t len)\n> +{\n> +\tsize_t i;\n> +\n> +\tfor (i = 0; i < len; i++) {\n> +\t\tif (is_regex_special(s[i]))\n> +\t\t\treturn 0;\n> +\t}\n> +\n> +\treturn 1;\n> +}\n> +\n> #ifdef USE_LIBPCRE1\n> static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n> {\n> @@ -602,7 +614,6 @@ static int pcre2match(struct grep_pat *p, const char *line, const char *eol,\n> static void free_pcre2_pattern(struct grep_pat *p)\n> {\n> }\n> -#endif /* !USE_LIBPCRE2 */\n\nHuh? Removing an `#endif` without removing the corresponding `#if`?\n\n... but...\n\n> static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n> {\n> @@ -623,11 +634,13 @@ static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n> \t\tcompile_regexp_failed(p, errbuf);\n> \t}\n> }\n> +#endif /* !USE_LIBPCRE2 */\n\nAh hah!\n\nIf we would not have plenty of exercise for the PCRE2 build options, I\nwould be worried. But AFAICT the CI build includes this all the time, so\nwe're fine.\n\n> static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n> {\n> \tint err;\n> \tint regflags = REG_NEWLINE;\n> +\tint pat_is_fixed;\n>\n> \tp->word_regexp = opt->word_regexp;\n> \tp->ignore_case = opt->ignore_case;\n> @@ -636,8 +649,38 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n> \tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n> \t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n>\n> -\tif (opt->fixed) {\n> +\tpat_is_fixed = is_fixed(p->pattern, p->patternlen);\n> +\tif (opt->fixed || pat_is_fixed) {\n> +#ifdef USE_LIBPCRE2\n> +\t\topt->pcre2 = 1;\n> +\t\tif (pat_is_fixed) {\n> +\t\t\tcompile_pcre2_pattern(p, opt);\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * E.g. t7811-grep-open.sh relies on the\n> +\t\t\t * pattern being restored, and unfortunately\n> +\t\t\t * there's no PCRE compile flag for \"this is\n> +\t\t\t * fixed\", so we need to munge it to\n> +\t\t\t * \"\\Q\\E\".\n> +\t\t\t */\n> +\t\t\tchar *old_pattern = p->pattern;\n> +\t\t\tsize_t old_patternlen = p->patternlen;\n> +\t\t\tstruct strbuf sb = STRBUF_INIT;\n> +\n> +\t\t\tstrbuf_add(&sb, \"\\\\Q\", 2);\n> +\t\t\tstrbuf_add(&sb, p->pattern, p->patternlen);\n> +\t\t\tstrbuf_add(&sb, \"\\\\E\", 2);\n> +\n> +\t\t\tp->pattern = sb.buf;\n> +\t\t\tp->patternlen = sb.len;\n> +\t\t\tcompile_pcre2_pattern(p, opt);\n> +\t\t\tp->pattern = old_pattern;\n> +\t\t\tp->patternlen = old_patternlen;\n> +\t\t\tstrbuf_release(&sb);\n> +\t\t}\n> +#else\n> \t\tcompile_fixed_regexp(p, opt);\n> +#endif\n\nIt might be a bit easier to read if the shorter clause came first.\n\nOther than that: what a nice read. I should save reviewing all your patch\nseries for just-before-bed time.\n\nThanks,\nDscho\n\n> \t\treturn;\n> \t}\n>\n> --\n> 2.22.0.455.g172b71a6c5\n>\n>\n"},{"id":"378067","messageId":"xmqqftnwe32i.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"20190626000329.32475-6-avarab@gmail.com","subject":"Re: [RFC/PATCH 5/7] grep: drop support for \\0 in --fixed-strings ","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-26T16:14:45Z","receivedAt":"2019-06-26T16:14:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Change \"-f \" to not support patterns with \"\\0\" in them under\n> --fixed-strings, we'll now only support these under --perl-regexp with\n> PCRE v2.\n>\n> A previous change to Documentation/git-grep.txt changed the\n> description of \"-f \" to be vague enough as to not promise that\n> this would work, and by dropping support for this we make it a whole\n> lot easier to move away from the kwset backend, which a subsequent\n> change will try to do.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n\nThis step, together with all others, looks sensibly justified to me\nonly when I wear tinted glasses that make me pretend that the final\ngoal is to promote pcre backend, which is much more important than\nserving the current users. When I remove the glasses, it smells\nmore like making excuses.\n\nBut as we saw discussed in the previous thread, I too think it is OK\nto make 'Nobody would notice the updated behaviour of NUL in the\npatterns' our working assumption and see if anybody screams---after\nall we have to start somewhere to make progress.\n\nA very good thing about this series is that it does *not* add a new\nfeature that people would miss, even if it went straight to 'master'\nand to the next release. All it does is to optimize differently and\nchanging the behaviour we assume nobody depends on. \n\nIt is easy enough to revert the whole thing if it turns out to be\nproblematic even in the worst case, and nobody would notice ;-)\n"},{"id":"378074","messageId":"xmqqsgrwcj0h.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"20190626000329.32475-3-avarab@gmail.com","subject":"Re: [RFC/PATCH 2/7] grep tests: move \"grep binary\" alongside the rest","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-26T18:13:18Z","receivedAt":"2019-06-26T18:13:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Move the \"grep binary\" test case added in aca20dd558 (\"grep: add test\n> script for binary file handling\", 2010-05-22) so that it lives\n> alongside the rest of the \"grep\" tests in t781*. This would have left\n> a gap in the t/700* namespace, so move a \"filter-branch\" test down,\n> leaving the \"t7010-setup.sh\" test as the next one after that.\n\nA gap here and there is fine.\n\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> ...ilter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} | 0\n> t/{t7008-grep-binary.sh => t7815-grep-binary.sh} | 0\n> 2 files changed, 0 insertions(+), 0 deletions(-)\n> rename t/{t7009-filter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} (100%)\n> rename t/{t7008-grep-binary.sh => t7815-grep-binary.sh} (100%)\n>\n> diff --git a/t/t7009-filter-branch-null-sha1.sh b/t/t7008-filter-branch-null-sha1.sh\n> similarity index 100%\n> rename from t/t7009-filter-branch-null-sha1.sh\n> rename to t/t7008-filter-branch-null-sha1.sh\n> diff --git a/t/t7008-grep-binary.sh b/t/t7815-grep-binary.sh\n> similarity index 100%\n> rename from t/t7008-grep-binary.sh\n> rename to t/t7815-grep-binary.sh\n"},{"id":"378075","messageId":"xmqqo92kchjm.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"nycvar.QRO.7.76.6.1906261610310.44@tvgsbejvaqbjf.bet","subject":"Re: [RFC/PATCH 7/7] grep: use PCRE v2 for optimized fixed-string search","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-26T18:45:01Z","receivedAt":"2019-06-26T18:45:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> ...\n> Ah hah!\n>\n> If we would not have plenty of exercise for the PCRE2 build options, I\n> would be worried. But AFAICT the CI build includes this all the time, so\n> we're fine.\n\nWell, I'd feel safer if it were not \"all the time\", i.e. we know we\nare testing both sides of the coin.\n"},{"id":"378109","messageId":"20190627020259.GB9224@genre.crustytoothpaste.net","threadId":"51300","inReplyTo":"20190626000329.32475-5-avarab@gmail.com","subject":"Re: [RFC/PATCH 4/7] grep: make the behavior for \\0 in patterns sane","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2019-06-27T02:03:00Z","receivedAt":"2019-06-27T02:03:08Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2019-06-26 at 00:03:26, Ævar Arnfjörð Bjarmason wrote:\n> diff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\n> index 2d27969057..c89fb569e3 100644\n> --- a/Documentation/git-grep.txt\n> +++ b/Documentation/git-grep.txt\n> @@ -271,6 +271,23 @@ providing this option will cause it to die.\n> \n> -f ::\n> \tRead patterns from , one per line.\n> ++\n> +Passing the pattern via allows for providing a search pattern\n> +containing a \\0.\n\nIn this case, I think it's easier if we write this as \"NUL\" or \"NUL\nbyte\", since I think you mean a literal byte with value 0 and not the\nliteral string \"\\0\". I certainly find myself a bit confused, at least,\nand I expect others will as well.\n\n> diff --git a/grep.c b/grep.c\n> index d3e6111c46..261bd3a342 100644\n> --- a/grep.c\n> +++ b/grep.c\n> @@ -368,18 +368,6 @@ static int is_fixed(const char *s, size_t len)\n> \treturn 1;\n> }\n> \n> -static int has_null(const char *s, size_t len)\n> -{\n> -\t/*\n> -\t * regcomp cannot accept patterns with NULs so when using it\n> -\t * we consider any pattern containing a NUL fixed.\n> -\t */\n> -\tif (memchr(s, 0, len))\n> -\t\treturn 1;\n> -\n> -\treturn 0;\n> -}\n> -\n> #ifdef USE_LIBPCRE1\n> static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n> {\n> @@ -668,9 +656,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n> \t * simple string match using kws. p->fixed tells us if we\n> \t * want to use kws.\n> \t */\n> -\tif (opt->fixed ||\n> -\t has_null(p->pattern, p->patternlen) ||\n> -\t is_fixed(p->pattern, p->patternlen))\n> +\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n> \t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n> \n> \tif (p->fixed) {\n> @@ -678,7 +664,12 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n> \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n> \t\tkwsprep(p->kws);\n> \t\treturn;\n> -\t} else if (opt->fixed) {\n> +\t}\n> +\n> +\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n> +\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n\nWe probably want to write this as \"NUL\" as well.\n\nOtherwise, I'm okay with this change. I didn't expect Git to handle\nliteral NULs in patterns and I'm surprised that it ever worked.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"378133","messageId":"nycvar.QRO.7.76.6.1906271113090.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"nycvar.QRO.7.76.6.1906261601270.44@tvgsbejvaqbjf.bet","subject":"Re: [RFC/PATCH 0/7] grep: move from kwset to optional PCRE v2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-27T09:16:02Z","receivedAt":"2019-06-27T09:16:02Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Wed, 26 Jun 2019, Johannes Schindelin wrote:\n\n> On Wed, 26 Jun 2019, Ævar Arnfjörð Bjarmason wrote:\n>\n> > This speeds things up a lot, but as shown in the patches & tests\n> > changed modifies the behavior where we have \\0 in *patterns* (only\n> > possible with 'grep -f ').\n>\n> I agree that it is not worth a lot to care about NULs in search patterns.\n>\n> So I am in favor of the goal of this patch series.\n\nThere seems to be a Windows-specific test failure:\nhttps://dev.azure.com/gitgitgadget/git/_build/results?buildId=11535&view=ms.vss-test-web.build-test-results-tab&runId=28232&resultId=101315&paneView=debug\n\nThe output is this:\n\n-- snip --\nnot ok 5 - log --grep does not find non-reencoded values (latin1)\n\nexpecting success:\n\tgit log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e >actual\n&&\n\ttest_must_be_empty actual\n\n++ git log --encoding=ISO-8859-1 --format=%s --grep=é\nfatal: pcre2_match failed with error code -8: UTF-8 error: byte 2 top bits\nnot 0x80\n-- snap --\n\nAny quick ideas? (I _could_ imagine that it is yet another case of passing\nnon-UTF-8-encoded stuff via command-line vs via file, which does not work\non Windows.)\n\nCiao,\nDscho\n"},{"id":"378145","messageId":"nycvar.QRO.7.76.6.1906271129370.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"xmqqo92kchjm.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC/PATCH 7/7] grep: use PCRE v2 for optimized fixed-string search","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-27T09:31:58Z","receivedAt":"2019-06-27T09:31:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 26 Jun 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > ...\n> > Ah hah!\n> >\n> > If we would not have plenty of exercise for the PCRE2 build options, I\n> > would be worried. But AFAICT the CI build includes this all the time, so\n> > we're fine.\n>\n> Well, I'd feel safer if it were not \"all the time\", i.e. we know we\n> are testing both sides of the coin.\n\nAFAIR at least the Linux32 job is built without PCRE2 by default. I might\nbe wrong on that, though...\n\nIn any case, the upcoming MSVC support for our Azure Pipeline _will_ build\nwithout PCRE2, then we will have that axis covered.\n\nCiao,\nDscho\n"},{"id":"378167","messageId":"871rzf8034.fsf@evledraar.gmail.com","threadId":"51300","inReplyTo":"nycvar.QRO.7.76.6.1906271113090.44@tvgsbejvaqbjf.bet","subject":"Re: [RFC/PATCH 0/7] grep: move from kwset to optional PCRE v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T16:27:59Z","receivedAt":"2019-06-27T16:28:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jun 27 2019, Johannes Schindelin wrote:\n\n> Hi Ævar,\n>\n> On Wed, 26 Jun 2019, Johannes Schindelin wrote:\n>\n>> On Wed, 26 Jun 2019, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> > This speeds things up a lot, but as shown in the patches & tests\n>> > changed modifies the behavior where we have \\0 in *patterns* (only\n>> > possible with 'grep -f ').\n>>\n>> I agree that it is not worth a lot to care about NULs in search patterns.\n>>\n>> So I am in favor of the goal of this patch series.\n>\n> There seems to be a Windows-specific test failure:\n> https://dev.azure.com/gitgitgadget/git/_build/results?buildId=11535&view=ms.vss-test-web.build-test-results-tab&runId=28232&resultId=101315&paneView=debug\n>\n> The output is this:\n>\n> -- snip --\n> not ok 5 - log --grep does not find non-reencoded values (latin1)\n>\n> expecting success:\n> \tgit log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e >actual\n> &&\n> \ttest_must_be_empty actual\n>\n> ++ git log --encoding=ISO-8859-1 --format=%s --grep=é\n> fatal: pcre2_match failed with error code -8: UTF-8 error: byte 2 top bits\n> not 0x80\n> -- snap --\n>\n> Any quick ideas? (I _could_ imagine that it is yet another case of passing\n> non-UTF-8-encoded stuff via command-line vs via file, which does not work\n> on Windows.)\n\nThis is an existing issue that my patches just happen to uncover. I'm\nworking on a v2 which'll fix it.\n"},{"id":"378182","messageId":"nycvar.QRO.7.76.6.1906272012470.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"871rzf8034.fsf@evledraar.gmail.com","subject":"Re: [RFC/PATCH 0/7] grep: move from kwset to optional PCRE v2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-27T18:21:06Z","receivedAt":"2019-06-27T18:21:04Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Thu, 27 Jun 2019, Ævar Arnfjörð Bjarmason wrote:\n\n>\n> On Thu, Jun 27 2019, Johannes Schindelin wrote:\n>\n> > On Wed, 26 Jun 2019, Johannes Schindelin wrote:\n> >\n> >> On Wed, 26 Jun 2019, Ævar Arnfjörð Bjarmason wrote:\n> >>\n> >> > This speeds things up a lot, but as shown in the patches & tests\n> >> > changed modifies the behavior where we have \\0 in *patterns* (only\n> >> > possible with 'grep -f ').\n> >>\n> >> I agree that it is not worth a lot to care about NULs in search patterns.\n> >>\n> >> So I am in favor of the goal of this patch series.\n> >\n> > There seems to be a Windows-specific test failure:\n> > https://dev.azure.com/gitgitgadget/git/_build/results?buildId=11535&view=ms.vss-test-web.build-test-results-tab&runId=28232&resultId=101315&paneView=debug\n> >\n> > The output is this:\n> >\n> > -- snip --\n> > not ok 5 - log --grep does not find non-reencoded values (latin1)\n> >\n> > expecting success:\n> > \tgit log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e >actual\n> > &&\n> > \ttest_must_be_empty actual\n> >\n> > ++ git log --encoding=ISO-8859-1 --format=%s --grep=é\n> > fatal: pcre2_match failed with error code -8: UTF-8 error: byte 2 top bits\n> > not 0x80\n> > -- snap --\n> >\n> > Any quick ideas? (I _could_ imagine that it is yet another case of passing\n> > non-UTF-8-encoded stuff via command-line vs via file, which does not work\n> > on Windows.)\n>\n> This is an existing issue that my patches just happen to uncover. I'm\n> working on a v2 which'll fix it.\n\nI just found yet another problem. When using a libpcre2 _without_ JIT\nsupport, I get this:\n\n-- snip --\n$ sh t4210-log-i18n.sh -i -V -x || tail -20\ntest-results/t4210-log-i18n.out\nok 1 - create commits in different encodings\nok 2 - log --grep searches in log output encoding (utf8)\nok 3 # skip log --grep searches in log output encoding (latin1) (missing !MINGW)\nok 4 # skip log --grep does not find non-reencoded values (utf8) (missing !MINGW)\nnot ok 5 - log --grep does not find non-reencoded values (latin1)\n#\n# git log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e\n# >actual &&\n# test_must_be_empty actual\n#\nok 3 # skip log --grep searches in log output encoding (latin1) (missing !MINGW)\n\nskipping test: log --grep does not find non-reencoded values (utf8)\n git log --encoding=utf8 --format=%s --grep=$latin1_e >actual &&\n test_must_be_empty actual\n\nok 4 # skip log --grep does not find non-reencoded values (utf8) (missing !MINGW)\n\nexpecting success:\n git log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e >actual &&\n test_must_be_empty actual\n\n++ git log --encoding=ISO-8859-1 --format=%s --grep=é\nfatal: pcre2_match failed with error code -8: UTF-8 error: byte 2 top bits not 0x80\nerror: last command exited with $?=128\nnot ok 5 - log --grep does not find non-reencoded values (latin1)\n#\n# git log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e\n# >actual &&\n# test_must_be_empty actual\n#\n-- snap --\n\nThis is actually a correct error, as we specifically feed non-UTF-8 text\nto PCRE2, but we do turn on the PCRE2_UTF flag.\n\nFunnily enough, this error only occurs when `pcre2_jit_on == 0`, i.e. when\nwe hit the code path that calls `pcre2_match()`. When the alternative code\npath is used, `pcre2_jit_match()` is called, and it does _not_ print that\nerror.\n\nWhatever the bug in libpcre2 that causes the JIT code path to fail on the\nUnicode validation, it points to the problem in this code in\n`compile_pcre2_pattern()`:\n\n-- snip --\n if (is_utf8_locale() && has_non_ascii(p->pattern))\n options |= PCRE2_UTF;\n-- snap --\n\nIt only asks whether there is a non-ASCII character in pattern, but we\nnever bother to see whether the haystack is also encoded in UTF-8. In this\ncase, it is not...\n\nCiao,\nDscho\n\nP.S.: Yes, yes, I know, we should run PCRE2 with JIT, and I am trying to\nfigure out why it is not enabled on Windows when I specifically asked\n`./configure` to enable it... Investigating now.\n\n-\n"},{"id":"378185","messageId":"nycvar.QRO.7.76.6.1906272042350.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"nycvar.QRO.7.76.6.1906271129370.44@tvgsbejvaqbjf.bet","subject":"Re: [RFC/PATCH 7/7] grep: use PCRE v2 for optimized fixed-string search","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-27T18:45:47Z","receivedAt":"2019-06-27T18:45:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 27 Jun 2019, Johannes Schindelin wrote:\n\n> On Wed, 26 Jun 2019, Junio C Hamano wrote:\n>\n> > Johannes Schindelin writes:\n> >\n> > > ...\n> > > Ah hah!\n> > >\n> > > If we would not have plenty of exercise for the PCRE2 build options, I\n> > > would be worried. But AFAICT the CI build includes this all the time, so\n> > > we're fine.\n> >\n> > Well, I'd feel safer if it were not \"all the time\", i.e. we know we\n> > are testing both sides of the coin.\n>\n> AFAIR at least the Linux32 job is built without PCRE2 by default. I might\n> be wrong on that, though...\n\nActually, it seems that _all_ of the Linux builds in our Azure Pipeline\ncompile without pcre2. It seems you have to pass `USE_LIBPCRE2=1` to\n`make`, and we do not do that in `ci/run-build-and-tests.sh` nor in\n`azure-pipelines.yml`. I do not even see that for the macOS builds.\n\nSo we got PCRE2 covered only in the Windows build, it seems.\n\nCiao,\nDscho\n"},{"id":"378190","messageId":"xmqqv9wqalv2.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"nycvar.QRO.7.76.6.1906272042350.44@tvgsbejvaqbjf.bet","subject":"Re: [RFC/PATCH 7/7] grep: use PCRE v2 for optimized fixed-string search","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-27T19:06:57Z","receivedAt":"2019-06-27T19:07:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> > > If we would not have plenty of exercise for the PCRE2 build options, I\n>> > > would be worried. But AFAICT the CI build includes this all the time, so\n>> > > we're fine.\n>> >\n>> > Well, I'd feel safer if it were not \"all the time\", i.e. we know we\n>> > are testing both sides of the coin.\n>>\n>> AFAIR at least the Linux32 job is built without PCRE2 by default. I might\n>> be wrong on that, though...\n>\n> Actually, it seems that _all_ of the Linux builds in our Azure Pipeline\n> compile without pcre2. It seems you have to pass `USE_LIBPCRE2=1` to\n> `make`, and we do not do that in `ci/run-build-and-tests.sh` nor in\n> `azure-pipelines.yml`. I do not even see that for the macOS builds.\n>\n> So we got PCRE2 covered only in the Windows build, it seems.\n\nOK, it sounds like we have sufficient coverage on both fronts.\nGood.\n"},{"id":"378224","messageId":"20190627233912.7117-2-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 1/9] log tests: test regex backends in \"--encode=\" tests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:04Z","receivedAt":"2019-06-27T23:39:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the tests added in 04deccda11 (\"log: re-encode commit messages\nbefore grepping\", 2013-02-11) to test the regex backends. Those tests\nnever worked as advertised, due to the is_fixed() optimization in\ngrep.c (which was in place at the time), and the needle in the tests\nbeing a fixed string.\n\nWe'd thus always use the \"fixed\" backend during the tests, which would\nuse the kwset() backend. This backend liberally accepts any garbage\ninput, so invalid encodings would be silently accepted.\n\nIn a follow-up commit we'll fix this bug, this test just demonstrates\nthe existing issue.\n\nIn practice this issue happened on Windows, see [1], but due to the\nstructure of the existing tests & how liberal the kwset code is about\ngarbage we missed this.\n\nCover this blind spot by testing all our regex engines. The PCRE\nbackend will spot these invalid encodings. It's possible that this\ntest breaks the \"basic\" and \"extended\" backends on some systems that\nare more anal than glibc about the encoding of locale issues with\nPOSIX functions that I can remember, but PCRE is more careful about\nthe validation.\n\n1. https://public-inbox.org/git/nycvar.QRO.7.76.6.1906271113090.44@tvgsbejvaqbjf.bet/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t4210-log-i18n.sh | 41 ++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 40 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t4210-log-i18n.sh b/t/t4210-log-i18n.sh\nindex 7c519436ef..86d22c1d4c 100755\n--- a/t/t4210-log-i18n.sh\n+++ b/t/t4210-log-i18n.sh\n@@ -1,12 +1,15 @@\n #!/bin/sh\n \n test_description='test log with i18n features'\n-. ./test-lib.sh\n+. ./lib-gettext.sh\n \n # two forms of é\n utf8_e=$(printf '\\303\\251')\n latin1_e=$(printf '\\351')\n \n+# invalid UTF-8\n+invalid_e=$(printf '\\303\\50)') # \")\" at end to close opening \"(\"\n+\n test_expect_success 'create commits in different encodings' '\n \ttest_tick &&\n \tcat >msg <<-EOF &&\n@@ -53,4 +56,40 @@ test_expect_success 'log --grep does not find non-reencoded values (latin1)' '\n \ttest_must_be_empty actual\n '\n \n+for engine in fixed basic extended perl\n+do\n+\tprereq=\n+\tresult=success\n+\tif test $engine = \"perl\"\n+\tthen\n+\t\tresult=failure\n+\t\tprereq=\"PCRE\"\n+\telse\n+\t\tprereq=\"\"\n+\tfi\n+\tforce_regex=\n+\tif test $engine != \"fixed\"\n+\tthen\n+\t force_regex=.*\n+\tfi\n+\ttest_expect_$result GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n+\t\tcat >expect <<-\\EOF &&\n+\t\tlatin1\n+\t\tutf8\n+\t\tEOF\n+\t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$latin1_e\\\" >actual &&\n+\t\ttest_cmp expect actual\n+\t\"\n+\n+\ttest_expect_success GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n+\t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$utf8_e\\\" >actual &&\n+\t\ttest_must_be_empty actual\n+\t\"\n+\n+\ttest_expect_$result GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not die on invalid UTF-8 value (latin1 + locale + invalid needle)\" \"\n+\t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$invalid_e\\\" >actual &&\n+\t\ttest_must_be_empty actual\n+\t\"\n+done\n+\n test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378225","messageId":"20190627233912.7117-1-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 0/9] grep: move from kwset to optional PCRE v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:03Z","receivedAt":"2019-06-27T23:39:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A non-RFC since it seem people like this approach.\n\nThis should fix the test failure noted by Johannes, there's two new\npatches at the start of this series. They address a bug that was there\nfor a long time, but I happened to trip over since PCRE is more strict\nabout UTF-8 validation than kwset (which doesn't care at all).\n\nI also added performance numbers to the relevant commit messages, took\nbrian's suggestion of saying \"NUL-byte\" instead of \"\\0\", and did some\nother copyediting of my own.\n\nThe rest of the code changes are all just comments & rewording of\npreviously added comments.\n\nÆvar Arnfjörð Bjarmason (9):\n log tests: test regex backends in \"--encode=\" tests\n grep: don't use PCRE2?_UTF8 with \"log --encoding=\"\n grep: inline the return value of a function call used only once\n grep tests: move \"grep binary\" alongside the rest\n grep tests: move binary pattern tests into their own file\n grep: make the behavior for NUL-byte in patterns sane\n grep: drop support for \\0 in --fixed-strings \n grep: remove the kwset optimization\n grep: use PCRE v2 for optimized fixed-string search\n\n Documentation/git-grep.txt | 17 +++\n grep.c | 115 +++++++---------\n grep.h | 3 +-\n revision.c | 3 +\n t/t4210-log-i18n.sh | 39 +++++-\n ...a1.sh => t7008-filter-branch-null-sha1.sh} | 0\n ...08-grep-binary.sh => t7815-grep-binary.sh} | 101 --------------\n t/t7816-grep-binary-pattern.sh | 127 ++++++++++++++++++\n 8 files changed, 233 insertions(+), 172 deletions(-)\n rename t/{t7009-filter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} (100%)\n rename t/{t7008-grep-binary.sh => t7815-grep-binary.sh} (55%)\n create mode 100755 t/t7816-grep-binary-pattern.sh\n\nRange-diff:\n -: ---------- > 1: cfc01f49d3 log tests: test regex backends in \"--encode=\" tests\n -: ---------- > 2: 4b59eb32f0 grep: don't use PCRE2?_UTF8 with \"log --encoding=\"\n 1: ad55d3be7e = 3: cc4d3b50d5 grep: inline the return value of a function call used only once\n 2: 650bcc8582 = 4: d9b29bdd89 grep tests: move \"grep binary\" alongside the rest\n 3: ef10a8820d ! 5: f85614f435 grep tests: move binary pattern tests into their own file\n @@ -2,9 +2,10 @@\n \n grep tests: move binary pattern tests into their own file\n \n - Move the tests for \"-f \" where \"\" contains a \"\\0\" pattern\n - into their own file. I added most of these tests in 966be95549 (\"grep:\n - add tests to fix blind spots with \\0 patterns\", 2017-05-20).\n + Move the tests for \"-f \" where \"\" contains a NUL byte\n + pattern into their own file. I added most of these tests in\n + 966be95549 (\"grep: add tests to fix blind spots with \\0 patterns\",\n + 2017-05-20).\n \n Whether a regex engine supports matching binary content is very\n different from whether it matches binary patterns. Since\n @@ -14,8 +15,8 @@\n engine can sensibly match binary patterns.\n \n Since 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) we've been punting\n - patterns containing \"\\0\" and considering them fixed, except in cases\n - where \"--ignore-case\" is provided and they're non-ASCII, see\n + patterns containing NUL-byte and considering them fixed, except in\n + cases where \"--ignore-case\" is provided and they're non-ASCII, see\n 5c1ebcca4d (\"grep/icase: avoid kwsset on literal non-ascii strings\",\n 2016-06-25). Subsequent commits will change this behavior.\n \n 4: 03e5637efc ! 6: 90afca8707 grep: make the behavior for \\0 in patterns sane\n @@ -1,12 +1,13 @@\n Author: Ævar Arnfjörð Bjarmason \n \n - grep: make the behavior for \\0 in patterns sane\n + grep: make the behavior for NUL-byte in patterns sane\n \n - The behavior of \"grep\" when patterns contained \"\\0\" has always been\n - haphazard, and has served the vagaries of the implementation more than\n - anything else. A \"\\0\" in a pattern can only be provided via \"-f\n - \", and since pickaxe (log search) has no such flag \"\\0\" in\n - patterns has only ever been supported by \"grep\".\n + The behavior of \"grep\" when patterns contained a NUL-byte has always\n + been haphazard, and has served the vagaries of the implementation more\n + than anything else. A pattern containing a NUL-byte can only be\n + provided via \"-f \". Since pickaxe (log search) has no such flag\n + the NUL-byte in patterns has only ever been supported by \"grep\" (and\n + not \"log --grep\").\n \n Since 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) patterns containing\n \"\\0\" were considered fixed. In 966be95549 (\"grep: add tests to fix\n @@ -14,9 +15,9 @@\n behavior.\n \n Change the behavior to do the obvious thing, i.e. don't silently\n - discard a regex pattern and make it implicitly fixed just because it\n - contains a \\0. Instead die if e.g. --basic-regexp is combined with\n - such a pattern.\n + discard a regex pattern and make it implicitly fixed just because they\n + contain a NUL-byte. Instead die if the backend in question can't\n + handle them, e.g. --basic-regexp is combined with such a pattern.\n \n This is desired because from a user's point of view it's the obvious\n thing to do. Whether we support BRE/ERE/Perl syntax is different from\n 5: b9aad3ec1c ! 7: 526b925fdc grep: drop support for \\0 in --fixed-strings \n @@ -2,15 +2,14 @@\n \n grep: drop support for \\0 in --fixed-strings \n \n - Change \"-f \" to not support patterns with \"\\0\" in them under\n - --fixed-strings, we'll now only support these under --perl-regexp with\n - PCRE v2.\n + Change \"-f \" to not support patterns with a NUL-byte in them\n + under --fixed-strings. We'll now only support these under\n + \"--perl-regexp\" with PCRE v2.\n \n - A previous change to Documentation/git-grep.txt changed the\n - description of \"-f \" to be vague enough as to not promise that\n - this would work, and by dropping support for this we make it a whole\n - lot easier to move away from the kwset backend, which a subsequent\n - change will try to do.\n + A previous change to grep's documentation changed the description of\n + \"-f \" to be vague enough as to not promise that this would work.\n + By dropping support for this we make it a whole lot easier to move\n + away from the kwset backend, which we'll do in a subsequent change.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n 6: 3587be009a ! 8: 14269bb295 grep: remove the kwset optimization\n @@ -2,9 +2,99 @@\n \n grep: remove the kwset optimization\n \n - A later change will replace this optimization with a different one,\n - but as removing it and running the tests demonstrates no grep\n - semantics depend on this backend anymore.\n + A later change will replace this optimization with optimistic use of\n + PCRE v2. I'm completely removing it as an intermediate step, as\n + opposed to replacing it with PCRE v2, to demonstrate that no grep\n + semantics depend on this (or any other) optimization for the fixed\n + backend anymore.\n +\n + For now this is mostly (but not entirely) a performance regression, as\n + shown by this hacky one-liner:\n +\n + for opt in '' ' -i'\n + do\n + GIT_PERF_7821_GREP_OPTS=$opt GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE=YesPlease' ./run origin/master HEAD -- p7821-grep-engines-fixed.sh\n + done &&\n + for opt in '' ' -i'\n + do GIT_PERF_4221_LOG_OPTS=$opt GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE=YesPlease' ./run origin/master HEAD -- p4221-log-grep-engines-fixed.sh\n + done\n +\n + Which produces:\n +\n + plain grep:\n +\n + Test origin/master HEAD\n + -------------------------------------------------------------------------\n + 7821.1: fixed grep int 0.55(1.60+0.63) 0.82(3.11+0.51) +49.1%\n + 7821.2: basic grep int 0.62(1.68+0.49) 0.85(3.02+0.52) +37.1%\n + 7821.3: extended grep int 0.61(1.63+0.53) 0.91(3.09+0.44) +49.2%\n + 7821.4: perl grep int 0.55(1.60+0.57) 0.41(0.93+0.57) -25.5%\n + 7821.6: fixed grep uncommon 0.20(0.50+0.44) 0.35(1.27+0.42) +75.0%\n + 7821.7: basic grep uncommon 0.20(0.49+0.45) 0.35(1.29+0.41) +75.0%\n + 7821.8: extended grep uncommon 0.20(0.45+0.48) 0.35(1.25+0.44) +75.0%\n + 7821.9: perl grep uncommon 0.20(0.53+0.41) 0.16(0.24+0.49) -20.0%\n + 7821.11: fixed grep æ 0.35(1.27+0.40) 0.25(0.82+0.39) -28.6%\n + 7821.12: basic grep æ 0.35(1.28+0.38) 0.25(0.75+0.44) -28.6%\n + 7821.13: extended grep æ 0.36(1.21+0.46) 0.25(0.86+0.35) -30.6%\n + 7821.14: perl grep æ 0.35(1.33+0.34) 0.16(0.26+0.47) -54.3%\n +\n + grep with -i:\n +\n + Test origin/master HEAD\n + -----------------------------------------------------------------------------\n + 7821.1: fixed grep -i int 0.61(1.84+0.64) 1.11(4.12+0.64) +82.0%\n + 7821.2: basic grep -i int 0.72(1.86+0.57) 1.15(4.48+0.49) +59.7%\n + 7821.3: extended grep -i int 0.94(1.83+0.60) 1.53(4.12+0.58) +62.8%\n + 7821.4: perl grep -i int 0.66(1.82+0.59) 0.55(1.08+0.58) -16.7%\n + 7821.6: fixed grep -i uncommon 0.21(0.51+0.44) 0.44(1.74+0.34) +109.5%\n + 7821.7: basic grep -i uncommon 0.21(0.55+0.41) 0.44(1.72+0.40) +109.5%\n + 7821.8: extended grep -i uncommon 0.21(0.57+0.39) 0.42(1.64+0.45) +100.0%\n + 7821.9: perl grep -i uncommon 0.21(0.48+0.48) 0.17(0.30+0.45) -19.0%\n + 7821.11: fixed grep -i æ 0.25(0.73+0.45) 0.25(0.75+0.45) +0.0%\n + 7821.12: basic grep -i æ 0.25(0.71+0.49) 0.26(0.77+0.44) +4.0%\n + 7821.13: extended grep -i æ 0.25(0.75+0.44) 0.25(0.74+0.46) +0.0%\n + 7821.14: perl grep -i æ 0.17(0.26+0.48) 0.16(0.20+0.52) -5.9%\n +\n + plain log:\n +\n + Test origin/master HEAD\n + ---------------------------------------------------------------------------------\n + 4221.1: fixed log --grep='int' 7.31(7.06+0.21) 8.11(7.85+0.20) +10.9%\n + 4221.2: basic log --grep='int' 7.30(6.94+0.27) 8.16(7.89+0.19) +11.8%\n + 4221.3: extended log --grep='int' 7.34(7.05+0.21) 8.08(7.76+0.25) +10.1%\n + 4221.4: perl log --grep='int' 7.27(6.94+0.24) 7.05(6.76+0.25) -3.0%\n + 4221.6: fixed log --grep='uncommon' 6.97(6.62+0.32) 7.86(7.51+0.30) +12.8%\n + 4221.7: basic log --grep='uncommon' 7.05(6.69+0.29) 7.89(7.60+0.28) +11.9%\n + 4221.8: extended log --grep='uncommon' 6.89(6.56+0.32) 7.99(7.66+0.24) +16.0%\n + 4221.9: perl log --grep='uncommon' 7.02(6.66+0.33) 6.97(6.54+0.36) -0.7%\n + 4221.11: fixed log --grep='æ' 7.37(7.03+0.33) 7.67(7.30+0.31) +4.1%\n + 4221.12: basic log --grep='æ' 7.41(7.00+0.31) 7.60(7.28+0.26) +2.6%\n + 4221.13: extended log --grep='æ' 7.35(6.96+0.38) 7.73(7.31+0.34) +5.2%\n + 4221.14: perl log --grep='æ' 7.43(7.10+0.32) 6.95(6.61+0.27) -6.5%\n +\n + log with -i:\n +\n + Test origin/master HEAD\n + ------------------------------------------------------------------------------------\n + 4221.1: fixed log -i --grep='int' 7.40(7.05+0.23) 8.66(8.38+0.20) +17.0%\n + 4221.2: basic log -i --grep='int' 7.39(7.09+0.23) 8.67(8.39+0.20) +17.3%\n + 4221.3: extended log -i --grep='int' 7.29(6.99+0.26) 8.69(8.31+0.26) +19.2%\n + 4221.4: perl log -i --grep='int' 7.42(7.16+0.21) 7.14(6.80+0.24) -3.8%\n + 4221.6: fixed log -i --grep='uncommon' 6.94(6.58+0.35) 8.43(8.04+0.30) +21.5%\n + 4221.7: basic log -i --grep='uncommon' 6.95(6.62+0.31) 8.34(7.93+0.32) +20.0%\n + 4221.8: extended log -i --grep='uncommon' 7.06(6.75+0.25) 8.32(7.98+0.31) +17.8%\n + 4221.9: perl log -i --grep='uncommon' 6.96(6.69+0.26) 7.04(6.64+0.32) +1.1%\n + 4221.11: fixed log -i --grep='æ' 7.92(7.55+0.33) 7.86(7.44+0.34) -0.8%\n + 4221.12: basic log -i --grep='æ' 7.88(7.49+0.32) 7.84(7.46+0.34) -0.5%\n + 4221.13: extended log -i --grep='æ' 7.91(7.51+0.32) 7.87(7.48+0.32) -0.5%\n + 4221.14: perl log -i --grep='æ' 7.01(6.59+0.35) 6.99(6.64+0.28) -0.3%\n +\n + Some of those, as noted in [1] are because PCRE is faster at finding\n + fixed strings. This looks bad for some engines, but in the next change\n + we'll optimistically use PCRE v2 for all of these, so it'll look\n + better.\n +\n + 1. https://public-inbox.org/git/87v9x793qi.fsf@evledraar.gmail.com/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n 7: 5bc25c03b8 ! 9: c0fd75d102 grep: use PCRE v2 for optimized fixed-string search\n @@ -7,19 +7,95 @@\n slower than PCRE v1 and v2 JIT with the kwset backend, so that\n optimization was counterproductive.\n \n - This brings back the optimization for \"-F\", without changing the\n - semantics of \"\\0\" in patterns. As seen in previous commits in this\n - series we could support it now, but I'd rather just leave that\n - edge-case aside so the tests don't need to do one thing or the other\n - depending on what --fixed-strings backend we're using.\n -\n - I could also support the v1 backend here, but that would make the code\n - more complex, and I'd rather aim for simplicity here and in future\n + This brings back the optimization for \"--fixed-strings\", without\n + changing the semantics of having a NUL-byte in patterns. As seen in\n + previous commits in this series we could support it now, but I'd\n + rather just leave that edge-case aside so we don't have one behavior\n + or the other depending what \"--fixed-strings\" backend we're using. It\n + makes the behavior harder to understand and document, and makes tests\n + for the different backends more painful.\n +\n + I could also support the PCRE v1 backend here, but that would make the\n + code more complex. I'd rather aim for simplicity here and in future\n changes to the diffcore. We're not going to have someone who\n absolutely must have faster search, but for whom building PCRE v2\n isn't acceptable.\n \n - 1. https://public-inbox.org/git/87v9x793qi.fsf@evledraar.gmail.com/\n + The difference between this series of commits and the current \"master\"\n + is, using the same t/perf commands shown in the last commit:\n +\n + plain grep:\n +\n + Test origin/master HEAD\n + -------------------------------------------------------------------------\n + 7821.1: fixed grep int 0.55(1.67+0.56) 0.41(0.98+0.60) -25.5%\n + 7821.2: basic grep int 0.58(1.65+0.52) 0.41(0.96+0.57) -29.3%\n + 7821.3: extended grep int 0.57(1.66+0.49) 0.42(0.93+0.60) -26.3%\n + 7821.4: perl grep int 0.54(1.67+0.50) 0.43(0.88+0.65) -20.4%\n + 7821.6: fixed grep uncommon 0.21(0.52+0.42) 0.16(0.24+0.51) -23.8%\n + 7821.7: basic grep uncommon 0.20(0.49+0.45) 0.17(0.28+0.47) -15.0%\n + 7821.8: extended grep uncommon 0.20(0.54+0.39) 0.16(0.25+0.50) -20.0%\n + 7821.9: perl grep uncommon 0.20(0.58+0.36) 0.16(0.23+0.50) -20.0%\n + 7821.11: fixed grep æ 0.35(1.24+0.43) 0.16(0.23+0.50) -54.3%\n + 7821.12: basic grep æ 0.36(1.29+0.38) 0.16(0.20+0.54) -55.6%\n + 7821.13: extended grep æ 0.35(1.23+0.44) 0.16(0.24+0.50) -54.3%\n + 7821.14: perl grep æ 0.35(1.33+0.34) 0.16(0.28+0.46) -54.3%\n +\n + grep with -i:\n +\n + Test origin/master HEAD\n + ----------------------------------------------------------------------------\n + 7821.1: fixed grep -i int 0.62(1.81+0.70) 0.47(1.11+0.64) -24.2%\n + 7821.2: basic grep -i int 0.67(1.90+0.53) 0.46(1.07+0.62) -31.3%\n + 7821.3: extended grep -i int 0.62(1.92+0.53) 0.53(1.12+0.58) -14.5%\n + 7821.4: perl grep -i int 0.66(1.85+0.58) 0.45(1.10+0.59) -31.8%\n + 7821.6: fixed grep -i uncommon 0.21(0.54+0.43) 0.17(0.20+0.55) -19.0%\n + 7821.7: basic grep -i uncommon 0.20(0.52+0.45) 0.17(0.29+0.48) -15.0%\n + 7821.8: extended grep -i uncommon 0.21(0.52+0.44) 0.17(0.26+0.50) -19.0%\n + 7821.9: perl grep -i uncommon 0.21(0.53+0.44) 0.17(0.20+0.56) -19.0%\n + 7821.11: fixed grep -i æ 0.26(0.79+0.44) 0.16(0.29+0.46) -38.5%\n + 7821.12: basic grep -i æ 0.26(0.79+0.42) 0.16(0.20+0.54) -38.5%\n + 7821.13: extended grep -i æ 0.26(0.84+0.39) 0.16(0.24+0.50) -38.5%\n + 7821.14: perl grep -i æ 0.16(0.24+0.49) 0.17(0.25+0.51) +6.3%\n +\n + plain log:\n +\n + Test origin/master HEAD\n + --------------------------------------------------------------------------------\n + 4221.1: fixed log --grep='int' 7.24(6.95+0.28) 7.20(6.95+0.18) -0.6%\n + 4221.2: basic log --grep='int' 7.31(6.97+0.22) 7.20(6.93+0.21) -1.5%\n + 4221.3: extended log --grep='int' 7.37(7.04+0.24) 7.22(6.91+0.25) -2.0%\n + 4221.4: perl log --grep='int' 7.31(7.04+0.21) 7.19(6.89+0.21) -1.6%\n + 4221.6: fixed log --grep='uncommon' 6.93(6.59+0.32) 7.04(6.66+0.37) +1.6%\n + 4221.7: basic log --grep='uncommon' 6.92(6.58+0.29) 7.08(6.75+0.29) +2.3%\n + 4221.8: extended log --grep='uncommon' 6.92(6.55+0.31) 7.00(6.68+0.31) +1.2%\n + 4221.9: perl log --grep='uncommon' 7.03(6.59+0.33) 7.12(6.73+0.34) +1.3%\n + 4221.11: fixed log --grep='æ' 7.41(7.08+0.28) 7.05(6.76+0.29) -4.9%\n + 4221.12: basic log --grep='æ' 7.39(6.99+0.33) 7.00(6.68+0.25) -5.3%\n + 4221.13: extended log --grep='æ' 7.34(7.00+0.25) 7.15(6.81+0.31) -2.6%\n + 4221.14: perl log --grep='æ' 7.43(7.13+0.26) 7.01(6.60+0.36) -5.7%\n +\n + log with -i:\n +\n + Test origin/master HEAD\n + ------------------------------------------------------------------------------------\n + 4221.1: fixed log -i --grep='int' 7.31(7.07+0.24) 7.23(7.00+0.22) -1.1%\n + 4221.2: basic log -i --grep='int' 7.40(7.08+0.28) 7.19(6.92+0.20) -2.8%\n + 4221.3: extended log -i --grep='int' 7.43(7.13+0.25) 7.27(6.99+0.21) -2.2%\n + 4221.4: perl log -i --grep='int' 7.34(7.10+0.24) 7.10(6.90+0.19) -3.3%\n + 4221.6: fixed log -i --grep='uncommon' 7.07(6.71+0.32) 7.11(6.77+0.28) +0.6%\n + 4221.7: basic log -i --grep='uncommon' 6.99(6.64+0.28) 7.12(6.69+0.38) +1.9%\n + 4221.8: extended log -i --grep='uncommon' 7.11(6.74+0.32) 7.10(6.77+0.27) -0.1%\n + 4221.9: perl log -i --grep='uncommon' 6.98(6.60+0.29) 7.05(6.64+0.34) +1.0%\n + 4221.11: fixed log -i --grep='æ' 7.85(7.45+0.34) 7.03(6.68+0.32) -10.4%\n + 4221.12: basic log -i --grep='æ' 7.87(7.49+0.29) 7.06(6.69+0.31) -10.3%\n + 4221.13: extended log -i --grep='æ' 7.87(7.54+0.31) 7.09(6.69+0.31) -9.9%\n + 4221.14: perl log -i --grep='æ' 7.06(6.77+0.28) 6.91(6.57+0.31) -2.1%\n +\n + So as with e05b027627 (\"grep: use PCRE v2 for optimized fixed-string\n + search\", 2019-06-26) there's a huge improvement in performance for\n + \"grep\", but in \"log\" most of our time is spent elsewhere, so we don't\n + notice it that much.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ -81,15 +157,19 @@\n +\t\t} else {\n +\t\t\t/*\n +\t\t\t * E.g. t7811-grep-open.sh relies on the\n -+\t\t\t * pattern being restored, and unfortunately\n -+\t\t\t * there's no PCRE compile flag for \"this is\n -+\t\t\t * fixed\", so we need to munge it to\n -+\t\t\t * \"\\Q\\E\".\n ++\t\t\t * pattern being restored.\n +\t\t\t */\n +\t\t\tchar *old_pattern = p->pattern;\n +\t\t\tsize_t old_patternlen = p->patternlen;\n +\t\t\tstruct strbuf sb = STRBUF_INIT;\n +\n ++\t\t\t/*\n ++\t\t\t * There is the PCRE2_LITERAL flag, but it's\n ++\t\t\t * only in PCRE v2 10.30 and later. Needing to\n ++\t\t\t * ifdef our way around that and dealing with\n ++\t\t\t * it + PCRE2_MULTILINE being an error is more\n ++\t\t\t * complex than just quoting this ourselves.\n ++\t\t\t*/\n +\t\t\tstrbuf_add(&sb, \"\\\\Q\", 2);\n +\t\t\tstrbuf_add(&sb, p->pattern, p->patternlen);\n +\t\t\tstrbuf_add(&sb, \"\\\\E\", 2);\n @@ -101,9 +181,9 @@\n +\t\t\tp->patternlen = old_patternlen;\n +\t\t\tstrbuf_release(&sb);\n +\t\t}\n -+#else\n ++#else /* !USE_LIBPCRE2 */\n \t\tcompile_fixed_regexp(p, opt);\n -+#endif\n ++#endif /* !USE_LIBPCRE2 */\n \t\treturn;\n \t}\n \n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378226","messageId":"20190627233912.7117-3-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 2/9] grep: don't use PCRE2?_UTF8 with \"log --encoding=\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:05Z","receivedAt":"2019-06-27T23:39:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug introduced in 18547aacf5 (\"grep/pcre: support utf-8\",\n2016-06-25) that was missed due to a blindspot in our tests, as\ndiscussed in the previous commit. I then blindly copied the same bug\nin 94da9193a6 (\"grep: add support for PCRE v2\", 2017-06-01) when\nadding the PCRE v2 code.\n\nWe should not tell PCRE that we're processing UTF-8 just because we're\ndealing with non-ASCII. In the case of e.g. \"log --encoding=<...>\"\nunder is_utf8_locale() the haystack might be in ISO-8859-1, and the\nneedle might be in a non-UTF-8 encoding.\n\nMaybe we should be more strict here and die earlier? Should we also be\nconverting the needle to the encoding in question, and failing if it's\nnot a string that's valid in that encoding? Maybe.\n\nBut for now matching this as non-UTF8 at least has some hope of\nproducing sensible results, since we know that our default heuristic\nof assuming the text to be matched is in the user locale encoding\nisn't true when we've explicitly encoded it to be in a different\nencoding.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 8 ++++----\n grep.h | 1 +\n revision.c | 3 +++\n t/t4210-log-i18n.sh | 6 ++----\n 4 files changed, 10 insertions(+), 8 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex f7c3a5803e..1de4ab49c0 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -388,11 +388,11 @@ static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \tint options = PCRE_MULTILINE;\n \n \tif (opt->ignore_case) {\n-\t\tif (has_non_ascii(p->pattern))\n+\t\tif (!opt->ignore_locale && has_non_ascii(p->pattern))\n \t\t\tp->pcre1_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n \t}\n-\tif (is_utf8_locale() && has_non_ascii(p->pattern))\n+\tif (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern))\n \t\toptions |= PCRE_UTF8;\n \n \tp->pcre1_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n@@ -498,14 +498,14 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \tp->pcre2_compile_context = NULL;\n \n \tif (opt->ignore_case) {\n-\t\tif (has_non_ascii(p->pattern)) {\n+\t\tif (!opt->ignore_locale && has_non_ascii(p->pattern)) {\n \t\t\tcharacter_tables = pcre2_maketables(NULL);\n \t\t\tp->pcre2_compile_context = pcre2_compile_context_create(NULL);\n \t\t\tpcre2_set_character_tables(p->pcre2_compile_context, character_tables);\n \t\t}\n \t\toptions |= PCRE2_CASELESS;\n \t}\n-\tif (is_utf8_locale() && has_non_ascii(p->pattern))\n+\tif (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern))\n \t\toptions |= PCRE2_UTF;\n \n \tp->pcre2_pattern = pcre2_compile((PCRE2_SPTR)p->pattern,\ndiff --git a/grep.h b/grep.h\nindex 1875880f37..4bb8a79d93 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -173,6 +173,7 @@ struct grep_opt {\n \tint funcbody;\n \tint extended_regexp_option;\n \tint pattern_type_option;\n+\tint ignore_locale;\n \tchar colors[NR_GREP_COLORS][COLOR_MAXLEN];\n \tunsigned pre_context;\n \tunsigned post_context;\ndiff --git a/revision.c b/revision.c\nindex 621feb9df7..a842fb158a 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -28,6 +28,7 @@\n #include \"commit-graph.h\"\n #include \"prio-queue.h\"\n #include \"hashmap.h\"\n+#include \"utf8.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2655,6 +2656,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \n \tgrep_commit_pattern_type(GREP_PATTERN_TYPE_UNSPECIFIED,\n \t\t\t\t &revs->grep_filter);\n+\tif (!is_encoding_utf8(get_log_output_encoding()))\n+\t\trevs->grep_filter.ignore_locale = 1;\n \tcompile_grep_patterns(&revs->grep_filter);\n \n \tif (revs->reverse && revs->reflog_info)\ndiff --git a/t/t4210-log-i18n.sh b/t/t4210-log-i18n.sh\nindex 86d22c1d4c..515bcb7ce1 100755\n--- a/t/t4210-log-i18n.sh\n+++ b/t/t4210-log-i18n.sh\n@@ -59,10 +59,8 @@ test_expect_success 'log --grep does not find non-reencoded values (latin1)' '\n for engine in fixed basic extended perl\n do\n \tprereq=\n-\tresult=success\n \tif test $engine = \"perl\"\n \tthen\n-\t\tresult=failure\n \t\tprereq=\"PCRE\"\n \telse\n \t\tprereq=\"\"\n@@ -72,7 +70,7 @@ do\n \tthen\n \t force_regex=.*\n \tfi\n-\ttest_expect_$result GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n+\ttest_expect_success GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n \t\tcat >expect <<-\\EOF &&\n \t\tlatin1\n \t\tutf8\n@@ -86,7 +84,7 @@ do\n \t\ttest_must_be_empty actual\n \t\"\n \n-\ttest_expect_$result GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not die on invalid UTF-8 value (latin1 + locale + invalid needle)\" \"\n+\ttest_expect_success GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not die on invalid UTF-8 value (latin1 + locale + invalid needle)\" \"\n \t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$invalid_e\\\" >actual &&\n \t\ttest_must_be_empty actual\n \t\"\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378227","messageId":"20190627233912.7117-4-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 3/9] grep: inline the return value of a function call used only once","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:06Z","receivedAt":"2019-06-27T23:39:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Since e944d9d932 (\"grep: rewrite an if/else condition to avoid\nduplicate expression\", 2016-06-25) the \"ascii_only\" variable has only\nbeen used once in compile_regexp(), let's just inline it there.\n\nThis makes the code easier to read, and might make it marginally\nfaster depending on compiler optimizations.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1de4ab49c0..4e8d0645a8 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -650,13 +650,11 @@ static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n-\tint ascii_only;\n \tint err;\n \tint regflags = REG_NEWLINE;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n-\tascii_only = !has_non_ascii(p->pattern);\n \n \t/*\n \t * Even when -F (fixed) asks us to do a non-regexp search, we\n@@ -673,7 +671,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tif (opt->fixed ||\n \t has_null(p->pattern, p->patternlen) ||\n \t is_fixed(p->pattern, p->patternlen))\n-\t\tp->fixed = !p->ignore_case || ascii_only;\n+\t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n \n \tif (p->fixed) {\n \t\tp->kws = kwsalloc(p->ignore_case ? tolower_trans_tbl : NULL);\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378228","messageId":"20190627233912.7117-5-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 4/9] grep tests: move \"grep binary\" alongside the rest","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:07Z","receivedAt":"2019-06-27T23:39:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the \"grep binary\" test case added in aca20dd558 (\"grep: add test\nscript for binary file handling\", 2010-05-22) so that it lives\nalongside the rest of the \"grep\" tests in t781*. This would have left\na gap in the t/700* namespace, so move a \"filter-branch\" test down,\nleaving the \"t7010-setup.sh\" test as the next one after that.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ...ilter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} | 0\n t/{t7008-grep-binary.sh => t7815-grep-binary.sh} | 0\n 2 files changed, 0 insertions(+), 0 deletions(-)\n rename t/{t7009-filter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} (100%)\n rename t/{t7008-grep-binary.sh => t7815-grep-binary.sh} (100%)\n\ndiff --git a/t/t7009-filter-branch-null-sha1.sh b/t/t7008-filter-branch-null-sha1.sh\nsimilarity index 100%\nrename from t/t7009-filter-branch-null-sha1.sh\nrename to t/t7008-filter-branch-null-sha1.sh\ndiff --git a/t/t7008-grep-binary.sh b/t/t7815-grep-binary.sh\nsimilarity index 100%\nrename from t/t7008-grep-binary.sh\nrename to t/t7815-grep-binary.sh\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378229","messageId":"20190627233912.7117-6-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 5/9] grep tests: move binary pattern tests into their own file","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:08Z","receivedAt":"2019-06-27T23:39:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the tests for \"-f \" where \"\" contains a NUL byte\npattern into their own file. I added most of these tests in\n966be95549 (\"grep: add tests to fix blind spots with \\0 patterns\",\n2017-05-20).\n\nWhether a regex engine supports matching binary content is very\ndifferent from whether it matches binary patterns. Since\n2f8952250a (\"regex: add regexec_buf() that can work on a non\nNUL-terminated string\", 2016-09-21) we've required REG_STARTEND of our\nregex engines so we can match binary content, but only the PCRE v2\nengine can sensibly match binary patterns.\n\nSince 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) we've been punting\npatterns containing NUL-byte and considering them fixed, except in\ncases where \"--ignore-case\" is provided and they're non-ASCII, see\n5c1ebcca4d (\"grep/icase: avoid kwsset on literal non-ascii strings\",\n2016-06-25). Subsequent commits will change this behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t7815-grep-binary.sh | 101 -----------------------------\n t/t7816-grep-binary-pattern.sh | 114 +++++++++++++++++++++++++++++++++\n 2 files changed, 114 insertions(+), 101 deletions(-)\n create mode 100755 t/t7816-grep-binary-pattern.sh\n\ndiff --git a/t/t7815-grep-binary.sh b/t/t7815-grep-binary.sh\nindex 2d87c49b75..90ebb64f46 100755\n--- a/t/t7815-grep-binary.sh\n+++ b/t/t7815-grep-binary.sh\n@@ -4,41 +4,6 @@ test_description='git grep in binary files'\n \n . ./test-lib.sh\n \n-nul_match () {\n-\tmatches=$1\n-\tflags=$2\n-\tpattern=$3\n-\tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n-\n-\tif test \"$matches\" = 1\n-\tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = 0\n-\tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = T1\n-\tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = T0\n-\tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n-\t\t\"\n-\telse\n-\t\ttest_expect_success \"PANIC: Test framework error. Unknown matches value $matches\" 'false'\n-\tfi\n-}\n-\n test_expect_success 'setup' \"\n \techo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&\n \tgit add a &&\n@@ -102,72 +67,6 @@ test_expect_failure 'git grep .fi a' '\n \tgit grep .fi a\n '\n \n-nul_match 1 '-F' 'yQf'\n-nul_match 0 '-F' 'yQx'\n-nul_match 1 '-Fi' 'YQf'\n-nul_match 0 '-Fi' 'YQx'\n-nul_match 1 '' 'yQf'\n-nul_match 0 '' 'yQx'\n-nul_match 1 '' 'æQð'\n-nul_match 1 '-F' 'eQm[*]c'\n-nul_match 1 '-Fi' 'EQM[*]C'\n-\n-# Regex patterns that would match but shouldn't with -F\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-F' '[y]Qf'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '-Fi' '[Y]QF'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-F' '[æ]Qð'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '-Fi' '[Æ]QÐ'\n-\n-# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n-# patterns case-insensitively.\n-nul_match T1 '-i' 'ÆQÐ'\n-\n-# \\0 implicitly disables regexes. This is an undocumented internal\n-# limitation.\n-nul_match T1 '' 'yQ[f]'\n-nul_match T1 '' '[y]Qf'\n-nul_match T1 '-i' 'YQ[F]'\n-nul_match T1 '-i' '[Y]Qf'\n-nul_match T1 '' 'æQ[ð]'\n-nul_match T1 '' '[æ]Qð'\n-nul_match T1 '-i' 'ÆQ[Ð]'\n-\n-# ... because of \\0 implicitly disabling regexes regexes that\n-# should/shouldn't match don't do the right thing.\n-nul_match T1 '' 'eQm.*cQ'\n-nul_match T1 '-i' 'EQM.*cQ'\n-nul_match T0 '' 'eQm[*]c'\n-nul_match T0 '-i' 'EQM[*]C'\n-\n-# Due to the REG_STARTEND extension when kwset() is disabled on -i &\n-# non-ASCII the string will be matched in its entirety, but the\n-# pattern will be cut off at the first \\0.\n-nul_match 0 '-i' 'NOMATCHQð'\n-nul_match T0 '-i' '[Æ]QNOMATCH'\n-nul_match T0 '-i' '[æ]QNOMATCH'\n-# Matches, but for the wrong reasons, just stops at [æ]\n-nul_match 1 '-i' '[Æ]Qð'\n-nul_match 1 '-i' '[æ]Qð'\n-\n-# Ensure that the matcher doesn't regress to something that stops at\n-# \\0\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '' 'yQNOMATCH'\n-nul_match 0 '' 'QNOMATCH'\n-nul_match 0 '-i' 'YQNOMATCH'\n-nul_match 0 '-i' 'QNOMATCH'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '' 'yQNÓMATCH'\n-nul_match 0 '' 'QNÓMATCH'\n-nul_match 0 '-i' 'YQNÓMATCH'\n-nul_match 0 '-i' 'QNÓMATCH'\n-\n test_expect_success 'grep respects binary diff attribute' '\n \techo text >t &&\n \tgit add t &&\ndiff --git a/t/t7816-grep-binary-pattern.sh b/t/t7816-grep-binary-pattern.sh\nnew file mode 100755\nindex 0000000000..4060dbd679\n--- /dev/null\n+++ b/t/t7816-grep-binary-pattern.sh\n@@ -0,0 +1,114 @@\n+#!/bin/sh\n+\n+test_description='git grep with a binary pattern files'\n+\n+. ./test-lib.sh\n+\n+nul_match () {\n+\tmatches=$1\n+\tflags=$2\n+\tpattern=$3\n+\tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n+\n+\tif test \"$matches\" = 1\n+\tthen\n+\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\tgit grep -f f $flags a\n+\t\t\"\n+\telif test \"$matches\" = 0\n+\tthen\n+\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\"\n+\telif test \"$matches\" = T1\n+\tthen\n+\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\tgit grep -f f $flags a\n+\t\t\"\n+\telif test \"$matches\" = T0\n+\tthen\n+\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\"\n+\telse\n+\t\ttest_expect_success \"PANIC: Test framework error. Unknown matches value $matches\" 'false'\n+\tfi\n+}\n+\n+test_expect_success 'setup' \"\n+\techo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&\n+\tgit add a &&\n+\tgit commit -m.\n+\"\n+\n+nul_match 1 '-F' 'yQf'\n+nul_match 0 '-F' 'yQx'\n+nul_match 1 '-Fi' 'YQf'\n+nul_match 0 '-Fi' 'YQx'\n+nul_match 1 '' 'yQf'\n+nul_match 0 '' 'yQx'\n+nul_match 1 '' 'æQð'\n+nul_match 1 '-F' 'eQm[*]c'\n+nul_match 1 '-Fi' 'EQM[*]C'\n+\n+# Regex patterns that would match but shouldn't with -F\n+nul_match 0 '-F' 'yQ[f]'\n+nul_match 0 '-F' '[y]Qf'\n+nul_match 0 '-Fi' 'YQ[F]'\n+nul_match 0 '-Fi' '[Y]QF'\n+nul_match 0 '-F' 'æQ[ð]'\n+nul_match 0 '-F' '[æ]Qð'\n+nul_match 0 '-Fi' 'ÆQ[Ð]'\n+nul_match 0 '-Fi' '[Æ]QÐ'\n+\n+# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n+# patterns case-insensitively.\n+nul_match T1 '-i' 'ÆQÐ'\n+\n+# \\0 implicitly disables regexes. This is an undocumented internal\n+# limitation.\n+nul_match T1 '' 'yQ[f]'\n+nul_match T1 '' '[y]Qf'\n+nul_match T1 '-i' 'YQ[F]'\n+nul_match T1 '-i' '[Y]Qf'\n+nul_match T1 '' 'æQ[ð]'\n+nul_match T1 '' '[æ]Qð'\n+nul_match T1 '-i' 'ÆQ[Ð]'\n+\n+# ... because of \\0 implicitly disabling regexes regexes that\n+# should/shouldn't match don't do the right thing.\n+nul_match T1 '' 'eQm.*cQ'\n+nul_match T1 '-i' 'EQM.*cQ'\n+nul_match T0 '' 'eQm[*]c'\n+nul_match T0 '-i' 'EQM[*]C'\n+\n+# Due to the REG_STARTEND extension when kwset() is disabled on -i &\n+# non-ASCII the string will be matched in its entirety, but the\n+# pattern will be cut off at the first \\0.\n+nul_match 0 '-i' 'NOMATCHQð'\n+nul_match T0 '-i' '[Æ]QNOMATCH'\n+nul_match T0 '-i' '[æ]QNOMATCH'\n+# Matches, but for the wrong reasons, just stops at [æ]\n+nul_match 1 '-i' '[Æ]Qð'\n+nul_match 1 '-i' '[æ]Qð'\n+\n+# Ensure that the matcher doesn't regress to something that stops at\n+# \\0\n+nul_match 0 '-F' 'yQ[f]'\n+nul_match 0 '-Fi' 'YQ[F]'\n+nul_match 0 '' 'yQNOMATCH'\n+nul_match 0 '' 'QNOMATCH'\n+nul_match 0 '-i' 'YQNOMATCH'\n+nul_match 0 '-i' 'QNOMATCH'\n+nul_match 0 '-F' 'æQ[ð]'\n+nul_match 0 '-Fi' 'ÆQ[Ð]'\n+nul_match 0 '' 'yQNÓMATCH'\n+nul_match 0 '' 'QNÓMATCH'\n+nul_match 0 '-i' 'YQNÓMATCH'\n+nul_match 0 '-i' 'QNÓMATCH'\n+\n+test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378230","messageId":"20190627233912.7117-7-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 6/9] grep: make the behavior for NUL-byte in patterns sane","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:09Z","receivedAt":"2019-06-27T23:39:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The behavior of \"grep\" when patterns contained a NUL-byte has always\nbeen haphazard, and has served the vagaries of the implementation more\nthan anything else. A pattern containing a NUL-byte can only be\nprovided via \"-f \". Since pickaxe (log search) has no such flag\nthe NUL-byte in patterns has only ever been supported by \"grep\" (and\nnot \"log --grep\").\n\nSince 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) patterns containing\n\"\\0\" were considered fixed. In 966be95549 (\"grep: add tests to fix\nblind spots with \\0 patterns\", 2017-05-20) I added tests for this\nbehavior.\n\nChange the behavior to do the obvious thing, i.e. don't silently\ndiscard a regex pattern and make it implicitly fixed just because they\ncontain a NUL-byte. Instead die if the backend in question can't\nhandle them, e.g. --basic-regexp is combined with such a pattern.\n\nThis is desired because from a user's point of view it's the obvious\nthing to do. Whether we support BRE/ERE/Perl syntax is different from\nwhether our implementation is limited by C-strings. These patterns are\nobscure enough that I think this behavior change is OK, especially\nsince we never documented the old behavior.\n\nDoing this also makes it easier to replace the kwset backend with\nsomething else, since we'll no longer strictly need it for anything we\ncan't easily use another fixed-string backend for.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/git-grep.txt | 17 ++++\n grep.c | 23 ++---\n t/t7816-grep-binary-pattern.sh | 159 ++++++++++++++++++---------------\n 3 files changed, 110 insertions(+), 89 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 2d27969057..c89fb569e3 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -271,6 +271,23 @@ providing this option will cause it to die.\n \n -f ::\n \tRead patterns from , one per line.\n++\n+Passing the pattern via allows for providing a search pattern\n+containing a \\0.\n++\n+Not all pattern types support patterns containing \\0. Git will error\n+out if a given pattern type can't support such a pattern. The\n+`--perl-regexp` pattern type when compiled against the PCRE v2 backend\n+has the widest support for these types of patterns.\n++\n+In versions of Git before 2.23.0 patterns containing \\0 would be\n+silently considered fixed. This was never documented, there were also\n+odd and undocumented interactions between e.g. non-ASCII patterns\n+containing \\0 and `--ignore-case`.\n++\n+In future versions we may learn to support patterns containing \\0 for\n+more search backends, until then we'll die when the pattern type in\n+question doesn't support them.\n \n -e::\n \tThe next parameter is the pattern. This option has to be\ndiff --git a/grep.c b/grep.c\nindex 4e8d0645a8..d6603bc950 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -368,18 +368,6 @@ static int is_fixed(const char *s, size_t len)\n \treturn 1;\n }\n \n-static int has_null(const char *s, size_t len)\n-{\n-\t/*\n-\t * regcomp cannot accept patterns with NULs so when using it\n-\t * we consider any pattern containing a NUL fixed.\n-\t */\n-\tif (memchr(s, 0, len))\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n #ifdef USE_LIBPCRE1\n static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n {\n@@ -668,9 +656,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t * simple string match using kws. p->fixed tells us if we\n \t * want to use kws.\n \t */\n-\tif (opt->fixed ||\n-\t has_null(p->pattern, p->patternlen) ||\n-\t is_fixed(p->pattern, p->patternlen))\n+\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n \n \tif (p->fixed) {\n@@ -678,7 +664,12 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n \t\tkwsprep(p->kws);\n \t\treturn;\n-\t} else if (opt->fixed) {\n+\t}\n+\n+\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n+\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n+\n+\tif (opt->fixed) {\n \t\t/*\n \t\t * We come here when the pattern has the non-ascii\n \t\t * characters we cannot case-fold, and asked to\ndiff --git a/t/t7816-grep-binary-pattern.sh b/t/t7816-grep-binary-pattern.sh\nindex 4060dbd679..9e09bd5d6a 100755\n--- a/t/t7816-grep-binary-pattern.sh\n+++ b/t/t7816-grep-binary-pattern.sh\n@@ -2,113 +2,126 @@\n \n test_description='git grep with a binary pattern files'\n \n-. ./test-lib.sh\n+. ./lib-gettext.sh\n \n-nul_match () {\n+nul_match_internal () {\n \tmatches=$1\n-\tflags=$2\n-\tpattern=$3\n+\tprereqs=$2\n+\tlc_all=$3\n+\textra_flags=$4\n+\tflags=$5\n+\tpattern=$6\n \tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n \n \tif test \"$matches\" = 1\n \tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\ttest_expect_success $prereqs \"LC_ALL='$lc_all' git grep $extra_flags -f f $flags '$pattern_human' a\" \"\n \t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n+\t\t\tLC_ALL='$lc_all' git grep $extra_flags -f f $flags a\n \t\t\"\n \telif test \"$matches\" = 0\n \tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\ttest_expect_success $prereqs \"LC_ALL='$lc_all' git grep $extra_flags -f f $flags '$pattern_human' a\" \"\n+\t\t\t>stderr &&\n \t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\ttest_must_fail env LC_ALL=\\\"$lc_all\\\" git grep $extra_flags -f f $flags a 2>stderr &&\n+\t\t\ttest_i18ngrep ! 'This is only supported with -P under PCRE v2' stderr\n \t\t\"\n-\telif test \"$matches\" = T1\n+\telif test \"$matches\" = P\n \tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\ttest_expect_success $prereqs \"error, PCRE v2 only: LC_ALL='$lc_all' git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\t>stderr &&\n \t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = T0\n-\tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\ttest_must_fail env LC_ALL=\\\"$lc_all\\\" git grep -f f $flags a 2>stderr &&\n+\t\t\ttest_i18ngrep 'This is only supported with -P under PCRE v2' stderr\n \t\t\"\n \telse\n \t\ttest_expect_success \"PANIC: Test framework error. Unknown matches value $matches\" 'false'\n \tfi\n }\n \n+nul_match () {\n+\tmatches=$1\n+\tmatches_pcre2=$2\n+\tmatches_pcre2_locale=$3\n+\tflags=$4\n+\tpattern=$5\n+\tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n+\n+\tnul_match_internal \"$matches\" \"\" \"C\" \"\" \"$flags\" \"$pattern\"\n+\tnul_match_internal \"$matches_pcre2\" \"LIBPCRE2\" \"C\" \"-P\" \"$flags\" \"$pattern\"\n+\tnul_match_internal \"$matches_pcre2_locale\" \"LIBPCRE2,GETTEXT_LOCALE\" \"$is_IS_locale\" \"-P\" \"$flags\" \"$pattern\"\n+}\n+\n test_expect_success 'setup' \"\n \techo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&\n \tgit add a &&\n \tgit commit -m.\n \"\n \n-nul_match 1 '-F' 'yQf'\n-nul_match 0 '-F' 'yQx'\n-nul_match 1 '-Fi' 'YQf'\n-nul_match 0 '-Fi' 'YQx'\n-nul_match 1 '' 'yQf'\n-nul_match 0 '' 'yQx'\n-nul_match 1 '' 'æQð'\n-nul_match 1 '-F' 'eQm[*]c'\n-nul_match 1 '-Fi' 'EQM[*]C'\n+# Simple fixed-string matching that can use kwset (no -i && non-ASCII)\n+nul_match 1 1 1 '-F' 'yQf'\n+nul_match 0 0 0 '-F' 'yQx'\n+nul_match 1 1 1 '-Fi' 'YQf'\n+nul_match 0 0 0 '-Fi' 'YQx'\n+nul_match 1 1 1 '' 'yQf'\n+nul_match 0 0 0 '' 'yQx'\n+nul_match 1 1 1 '' 'æQð'\n+nul_match 1 1 1 '-F' 'eQm[*]c'\n+nul_match 1 1 1 '-Fi' 'EQM[*]C'\n \n # Regex patterns that would match but shouldn't with -F\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-F' '[y]Qf'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '-Fi' '[Y]QF'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-F' '[æ]Qð'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '-Fi' '[Æ]QÐ'\n+nul_match 0 0 0 '-F' 'yQ[f]'\n+nul_match 0 0 0 '-F' '[y]Qf'\n+nul_match 0 0 0 '-Fi' 'YQ[F]'\n+nul_match 0 0 0 '-Fi' '[Y]QF'\n+nul_match 0 0 0 '-F' 'æQ[ð]'\n+nul_match 0 0 0 '-F' '[æ]Qð'\n \n-# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n-# patterns case-insensitively.\n-nul_match T1 '-i' 'ÆQÐ'\n+# The -F kwset codepath can't handle -i && non-ASCII...\n+nul_match P 1 1 '-i' '[æ]Qð'\n \n-# \\0 implicitly disables regexes. This is an undocumented internal\n-# limitation.\n-nul_match T1 '' 'yQ[f]'\n-nul_match T1 '' '[y]Qf'\n-nul_match T1 '-i' 'YQ[F]'\n-nul_match T1 '-i' '[Y]Qf'\n-nul_match T1 '' 'æQ[ð]'\n-nul_match T1 '' '[æ]Qð'\n-nul_match T1 '-i' 'ÆQ[Ð]'\n+# ...PCRE v2 only matches non-ASCII with -i casefolding under UTF-8\n+# semantics\n+nul_match P P P '-Fi' 'ÆQ[Ð]'\n+nul_match P 0 1 '-i' 'ÆQ[Ð]'\n+nul_match P 0 1 '-i' '[Æ]QÐ'\n+nul_match P 0 1 '-i' '[Æ]Qð'\n+nul_match P 0 1 '-i' 'ÆQÐ'\n \n-# ... because of \\0 implicitly disabling regexes regexes that\n-# should/shouldn't match don't do the right thing.\n-nul_match T1 '' 'eQm.*cQ'\n-nul_match T1 '-i' 'EQM.*cQ'\n-nul_match T0 '' 'eQm[*]c'\n-nul_match T0 '-i' 'EQM[*]C'\n+# \\0 in regexes can only work with -P & PCRE v2\n+nul_match P 1 1 '' 'yQ[f]'\n+nul_match P 1 1 '' '[y]Qf'\n+nul_match P 1 1 '-i' 'YQ[F]'\n+nul_match P 1 1 '-i' '[Y]Qf'\n+nul_match P 1 1 '' 'æQ[ð]'\n+nul_match P 1 1 '' '[æ]Qð'\n+nul_match P 0 1 '-i' 'ÆQ[Ð]'\n+nul_match P 1 1 '' 'eQm.*cQ'\n+nul_match P 1 1 '-i' 'EQM.*cQ'\n+nul_match P 0 0 '' 'eQm[*]c'\n+nul_match P 0 0 '-i' 'EQM[*]C'\n \n-# Due to the REG_STARTEND extension when kwset() is disabled on -i &\n-# non-ASCII the string will be matched in its entirety, but the\n-# pattern will be cut off at the first \\0.\n-nul_match 0 '-i' 'NOMATCHQð'\n-nul_match T0 '-i' '[Æ]QNOMATCH'\n-nul_match T0 '-i' '[æ]QNOMATCH'\n-# Matches, but for the wrong reasons, just stops at [æ]\n-nul_match 1 '-i' '[Æ]Qð'\n-nul_match 1 '-i' '[æ]Qð'\n+# Assert that we're using REG_STARTEND and the pattern doesn't match\n+# just because it's cut off at the first \\0.\n+nul_match 0 0 0 '-i' 'NOMATCHQð'\n+nul_match P 0 0 '-i' '[Æ]QNOMATCH'\n+nul_match P 0 0 '-i' '[æ]QNOMATCH'\n \n # Ensure that the matcher doesn't regress to something that stops at\n # \\0\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '' 'yQNOMATCH'\n-nul_match 0 '' 'QNOMATCH'\n-nul_match 0 '-i' 'YQNOMATCH'\n-nul_match 0 '-i' 'QNOMATCH'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '' 'yQNÓMATCH'\n-nul_match 0 '' 'QNÓMATCH'\n-nul_match 0 '-i' 'YQNÓMATCH'\n-nul_match 0 '-i' 'QNÓMATCH'\n+nul_match 0 0 0 '-F' 'yQ[f]'\n+nul_match 0 0 0 '-Fi' 'YQ[F]'\n+nul_match 0 0 0 '' 'yQNOMATCH'\n+nul_match 0 0 0 '' 'QNOMATCH'\n+nul_match 0 0 0 '-i' 'YQNOMATCH'\n+nul_match 0 0 0 '-i' 'QNOMATCH'\n+nul_match 0 0 0 '-F' 'æQ[ð]'\n+nul_match P P P '-Fi' 'ÆQ[Ð]'\n+nul_match P 0 1 '-i' 'ÆQ[Ð]'\n+nul_match 0 0 0 '' 'yQNÓMATCH'\n+nul_match 0 0 0 '' 'QNÓMATCH'\n+nul_match 0 0 0 '-i' 'YQNÓMATCH'\n+nul_match 0 0 0 '-i' 'QNÓMATCH'\n \n test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378231","messageId":"20190627233912.7117-8-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 7/9] grep: drop support for \\0 in --fixed-strings ","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:10Z","receivedAt":"2019-06-27T23:39:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change \"-f \" to not support patterns with a NUL-byte in them\nunder --fixed-strings. We'll now only support these under\n\"--perl-regexp\" with PCRE v2.\n\nA previous change to grep's documentation changed the description of\n\"-f \" to be vague enough as to not promise that this would work.\nBy dropping support for this we make it a whole lot easier to move\naway from the kwset backend, which we'll do in a subsequent change.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 6 +--\n t/t7816-grep-binary-pattern.sh | 82 +++++++++++++++++-----------------\n 2 files changed, 44 insertions(+), 44 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex d6603bc950..8d0fff316c 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -644,6 +644,9 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \n+\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n+\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n+\n \t/*\n \t * Even when -F (fixed) asks us to do a non-regexp search, we\n \t * may not be able to correctly case-fold when -i\n@@ -666,9 +669,6 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\treturn;\n \t}\n \n-\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n-\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n-\n \tif (opt->fixed) {\n \t\t/*\n \t\t * We come here when the pattern has the non-ascii\ndiff --git a/t/t7816-grep-binary-pattern.sh b/t/t7816-grep-binary-pattern.sh\nindex 9e09bd5d6a..60bab291e4 100755\n--- a/t/t7816-grep-binary-pattern.sh\n+++ b/t/t7816-grep-binary-pattern.sh\n@@ -60,23 +60,23 @@ test_expect_success 'setup' \"\n \"\n \n # Simple fixed-string matching that can use kwset (no -i && non-ASCII)\n-nul_match 1 1 1 '-F' 'yQf'\n-nul_match 0 0 0 '-F' 'yQx'\n-nul_match 1 1 1 '-Fi' 'YQf'\n-nul_match 0 0 0 '-Fi' 'YQx'\n-nul_match 1 1 1 '' 'yQf'\n-nul_match 0 0 0 '' 'yQx'\n-nul_match 1 1 1 '' 'æQð'\n-nul_match 1 1 1 '-F' 'eQm[*]c'\n-nul_match 1 1 1 '-Fi' 'EQM[*]C'\n+nul_match P P P '-F' 'yQf'\n+nul_match P P P '-F' 'yQx'\n+nul_match P P P '-Fi' 'YQf'\n+nul_match P P P '-Fi' 'YQx'\n+nul_match P P 1 '' 'yQf'\n+nul_match P P 0 '' 'yQx'\n+nul_match P P 1 '' 'æQð'\n+nul_match P P P '-F' 'eQm[*]c'\n+nul_match P P P '-Fi' 'EQM[*]C'\n \n # Regex patterns that would match but shouldn't with -F\n-nul_match 0 0 0 '-F' 'yQ[f]'\n-nul_match 0 0 0 '-F' '[y]Qf'\n-nul_match 0 0 0 '-Fi' 'YQ[F]'\n-nul_match 0 0 0 '-Fi' '[Y]QF'\n-nul_match 0 0 0 '-F' 'æQ[ð]'\n-nul_match 0 0 0 '-F' '[æ]Qð'\n+nul_match P P P '-F' 'yQ[f]'\n+nul_match P P P '-F' '[y]Qf'\n+nul_match P P P '-Fi' 'YQ[F]'\n+nul_match P P P '-Fi' '[Y]QF'\n+nul_match P P P '-F' 'æQ[ð]'\n+nul_match P P P '-F' '[æ]Qð'\n \n # The -F kwset codepath can't handle -i && non-ASCII...\n nul_match P 1 1 '-i' '[æ]Qð'\n@@ -90,38 +90,38 @@ nul_match P 0 1 '-i' '[Æ]Qð'\n nul_match P 0 1 '-i' 'ÆQÐ'\n \n # \\0 in regexes can only work with -P & PCRE v2\n-nul_match P 1 1 '' 'yQ[f]'\n-nul_match P 1 1 '' '[y]Qf'\n-nul_match P 1 1 '-i' 'YQ[F]'\n-nul_match P 1 1 '-i' '[Y]Qf'\n-nul_match P 1 1 '' 'æQ[ð]'\n-nul_match P 1 1 '' '[æ]Qð'\n-nul_match P 0 1 '-i' 'ÆQ[Ð]'\n-nul_match P 1 1 '' 'eQm.*cQ'\n-nul_match P 1 1 '-i' 'EQM.*cQ'\n-nul_match P 0 0 '' 'eQm[*]c'\n-nul_match P 0 0 '-i' 'EQM[*]C'\n+nul_match P P 1 '' 'yQ[f]'\n+nul_match P P 1 '' '[y]Qf'\n+nul_match P P 1 '-i' 'YQ[F]'\n+nul_match P P 1 '-i' '[Y]Qf'\n+nul_match P P 1 '' 'æQ[ð]'\n+nul_match P P 1 '' '[æ]Qð'\n+nul_match P P 1 '-i' 'ÆQ[Ð]'\n+nul_match P P 1 '' 'eQm.*cQ'\n+nul_match P P 1 '-i' 'EQM.*cQ'\n+nul_match P P 0 '' 'eQm[*]c'\n+nul_match P P 0 '-i' 'EQM[*]C'\n \n # Assert that we're using REG_STARTEND and the pattern doesn't match\n # just because it's cut off at the first \\0.\n-nul_match 0 0 0 '-i' 'NOMATCHQð'\n-nul_match P 0 0 '-i' '[Æ]QNOMATCH'\n-nul_match P 0 0 '-i' '[æ]QNOMATCH'\n+nul_match P P 0 '-i' 'NOMATCHQð'\n+nul_match P P 0 '-i' '[Æ]QNOMATCH'\n+nul_match P P 0 '-i' '[æ]QNOMATCH'\n \n # Ensure that the matcher doesn't regress to something that stops at\n # \\0\n-nul_match 0 0 0 '-F' 'yQ[f]'\n-nul_match 0 0 0 '-Fi' 'YQ[F]'\n-nul_match 0 0 0 '' 'yQNOMATCH'\n-nul_match 0 0 0 '' 'QNOMATCH'\n-nul_match 0 0 0 '-i' 'YQNOMATCH'\n-nul_match 0 0 0 '-i' 'QNOMATCH'\n-nul_match 0 0 0 '-F' 'æQ[ð]'\n+nul_match P P P '-F' 'yQ[f]'\n+nul_match P P P '-Fi' 'YQ[F]'\n+nul_match P P 0 '' 'yQNOMATCH'\n+nul_match P P 0 '' 'QNOMATCH'\n+nul_match P P 0 '-i' 'YQNOMATCH'\n+nul_match P P 0 '-i' 'QNOMATCH'\n+nul_match P P P '-F' 'æQ[ð]'\n nul_match P P P '-Fi' 'ÆQ[Ð]'\n-nul_match P 0 1 '-i' 'ÆQ[Ð]'\n-nul_match 0 0 0 '' 'yQNÓMATCH'\n-nul_match 0 0 0 '' 'QNÓMATCH'\n-nul_match 0 0 0 '-i' 'YQNÓMATCH'\n-nul_match 0 0 0 '-i' 'QNÓMATCH'\n+nul_match P P 1 '-i' 'ÆQ[Ð]'\n+nul_match P P 0 '' 'yQNÓMATCH'\n+nul_match P P 0 '' 'QNÓMATCH'\n+nul_match P P 0 '-i' 'YQNÓMATCH'\n+nul_match P P 0 '-i' 'QNÓMATCH'\n \n test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378232","messageId":"20190627233912.7117-9-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 8/9] grep: remove the kwset optimization","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:11Z","receivedAt":"2019-06-27T23:39:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A later change will replace this optimization with optimistic use of\nPCRE v2. I'm completely removing it as an intermediate step, as\nopposed to replacing it with PCRE v2, to demonstrate that no grep\nsemantics depend on this (or any other) optimization for the fixed\nbackend anymore.\n\nFor now this is mostly (but not entirely) a performance regression, as\nshown by this hacky one-liner:\n\n for opt in '' ' -i'\n do\n GIT_PERF_7821_GREP_OPTS=$opt GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE=YesPlease' ./run origin/master HEAD -- p7821-grep-engines-fixed.sh\n done &&\n for opt in '' ' -i'\n do GIT_PERF_4221_LOG_OPTS=$opt GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE=YesPlease' ./run origin/master HEAD -- p4221-log-grep-engines-fixed.sh\n done\n\nWhich produces:\n\nplain grep:\n\n Test origin/master HEAD\n -------------------------------------------------------------------------\n 7821.1: fixed grep int 0.55(1.60+0.63) 0.82(3.11+0.51) +49.1%\n 7821.2: basic grep int 0.62(1.68+0.49) 0.85(3.02+0.52) +37.1%\n 7821.3: extended grep int 0.61(1.63+0.53) 0.91(3.09+0.44) +49.2%\n 7821.4: perl grep int 0.55(1.60+0.57) 0.41(0.93+0.57) -25.5%\n 7821.6: fixed grep uncommon 0.20(0.50+0.44) 0.35(1.27+0.42) +75.0%\n 7821.7: basic grep uncommon 0.20(0.49+0.45) 0.35(1.29+0.41) +75.0%\n 7821.8: extended grep uncommon 0.20(0.45+0.48) 0.35(1.25+0.44) +75.0%\n 7821.9: perl grep uncommon 0.20(0.53+0.41) 0.16(0.24+0.49) -20.0%\n 7821.11: fixed grep æ 0.35(1.27+0.40) 0.25(0.82+0.39) -28.6%\n 7821.12: basic grep æ 0.35(1.28+0.38) 0.25(0.75+0.44) -28.6%\n 7821.13: extended grep æ 0.36(1.21+0.46) 0.25(0.86+0.35) -30.6%\n 7821.14: perl grep æ 0.35(1.33+0.34) 0.16(0.26+0.47) -54.3%\n\ngrep with -i:\n\n Test origin/master HEAD\n -----------------------------------------------------------------------------\n 7821.1: fixed grep -i int 0.61(1.84+0.64) 1.11(4.12+0.64) +82.0%\n 7821.2: basic grep -i int 0.72(1.86+0.57) 1.15(4.48+0.49) +59.7%\n 7821.3: extended grep -i int 0.94(1.83+0.60) 1.53(4.12+0.58) +62.8%\n 7821.4: perl grep -i int 0.66(1.82+0.59) 0.55(1.08+0.58) -16.7%\n 7821.6: fixed grep -i uncommon 0.21(0.51+0.44) 0.44(1.74+0.34) +109.5%\n 7821.7: basic grep -i uncommon 0.21(0.55+0.41) 0.44(1.72+0.40) +109.5%\n 7821.8: extended grep -i uncommon 0.21(0.57+0.39) 0.42(1.64+0.45) +100.0%\n 7821.9: perl grep -i uncommon 0.21(0.48+0.48) 0.17(0.30+0.45) -19.0%\n 7821.11: fixed grep -i æ 0.25(0.73+0.45) 0.25(0.75+0.45) +0.0%\n 7821.12: basic grep -i æ 0.25(0.71+0.49) 0.26(0.77+0.44) +4.0%\n 7821.13: extended grep -i æ 0.25(0.75+0.44) 0.25(0.74+0.46) +0.0%\n 7821.14: perl grep -i æ 0.17(0.26+0.48) 0.16(0.20+0.52) -5.9%\n\nplain log:\n\n Test origin/master HEAD\n ---------------------------------------------------------------------------------\n 4221.1: fixed log --grep='int' 7.31(7.06+0.21) 8.11(7.85+0.20) +10.9%\n 4221.2: basic log --grep='int' 7.30(6.94+0.27) 8.16(7.89+0.19) +11.8%\n 4221.3: extended log --grep='int' 7.34(7.05+0.21) 8.08(7.76+0.25) +10.1%\n 4221.4: perl log --grep='int' 7.27(6.94+0.24) 7.05(6.76+0.25) -3.0%\n 4221.6: fixed log --grep='uncommon' 6.97(6.62+0.32) 7.86(7.51+0.30) +12.8%\n 4221.7: basic log --grep='uncommon' 7.05(6.69+0.29) 7.89(7.60+0.28) +11.9%\n 4221.8: extended log --grep='uncommon' 6.89(6.56+0.32) 7.99(7.66+0.24) +16.0%\n 4221.9: perl log --grep='uncommon' 7.02(6.66+0.33) 6.97(6.54+0.36) -0.7%\n 4221.11: fixed log --grep='æ' 7.37(7.03+0.33) 7.67(7.30+0.31) +4.1%\n 4221.12: basic log --grep='æ' 7.41(7.00+0.31) 7.60(7.28+0.26) +2.6%\n 4221.13: extended log --grep='æ' 7.35(6.96+0.38) 7.73(7.31+0.34) +5.2%\n 4221.14: perl log --grep='æ' 7.43(7.10+0.32) 6.95(6.61+0.27) -6.5%\n\nlog with -i:\n\n Test origin/master HEAD\n ------------------------------------------------------------------------------------\n 4221.1: fixed log -i --grep='int' 7.40(7.05+0.23) 8.66(8.38+0.20) +17.0%\n 4221.2: basic log -i --grep='int' 7.39(7.09+0.23) 8.67(8.39+0.20) +17.3%\n 4221.3: extended log -i --grep='int' 7.29(6.99+0.26) 8.69(8.31+0.26) +19.2%\n 4221.4: perl log -i --grep='int' 7.42(7.16+0.21) 7.14(6.80+0.24) -3.8%\n 4221.6: fixed log -i --grep='uncommon' 6.94(6.58+0.35) 8.43(8.04+0.30) +21.5%\n 4221.7: basic log -i --grep='uncommon' 6.95(6.62+0.31) 8.34(7.93+0.32) +20.0%\n 4221.8: extended log -i --grep='uncommon' 7.06(6.75+0.25) 8.32(7.98+0.31) +17.8%\n 4221.9: perl log -i --grep='uncommon' 6.96(6.69+0.26) 7.04(6.64+0.32) +1.1%\n 4221.11: fixed log -i --grep='æ' 7.92(7.55+0.33) 7.86(7.44+0.34) -0.8%\n 4221.12: basic log -i --grep='æ' 7.88(7.49+0.32) 7.84(7.46+0.34) -0.5%\n 4221.13: extended log -i --grep='æ' 7.91(7.51+0.32) 7.87(7.48+0.32) -0.5%\n 4221.14: perl log -i --grep='æ' 7.01(6.59+0.35) 6.99(6.64+0.28) -0.3%\n\nSome of those, as noted in [1] are because PCRE is faster at finding\nfixed strings. This looks bad for some engines, but in the next change\nwe'll optimistically use PCRE v2 for all of these, so it'll look\nbetter.\n\n1. https://public-inbox.org/git/87v9x793qi.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 63 +++-------------------------------------------------------\n grep.h | 2 --\n 2 files changed, 3 insertions(+), 62 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 8d0fff316c..4468519d5c 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -356,18 +356,6 @@ static NORETURN void compile_regexp_failed(const struct grep_pat *p,\n \tdie(\"%s'%s': %s\", where, p->pattern, error);\n }\n \n-static int is_fixed(const char *s, size_t len)\n-{\n-\tsize_t i;\n-\n-\tfor (i = 0; i < len; i++) {\n-\t\tif (is_regex_special(s[i]))\n-\t\t\treturn 0;\n-\t}\n-\n-\treturn 1;\n-}\n-\n #ifdef USE_LIBPCRE1\n static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n {\n@@ -643,38 +631,12 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n+\tp->fixed = opt->fixed;\n \n \tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n \t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n \n-\t/*\n-\t * Even when -F (fixed) asks us to do a non-regexp search, we\n-\t * may not be able to correctly case-fold when -i\n-\t * (ignore-case) is asked (in which case, we'll synthesize a\n-\t * regexp to match the pattern that matches regexp special\n-\t * characters literally, while ignoring case differences). On\n-\t * the other hand, even without -F, if the pattern does not\n-\t * have any regexp special characters and there is no need for\n-\t * case-folding search, we can internally turn it into a\n-\t * simple string match using kws. p->fixed tells us if we\n-\t * want to use kws.\n-\t */\n-\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n-\t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n-\n-\tif (p->fixed) {\n-\t\tp->kws = kwsalloc(p->ignore_case ? tolower_trans_tbl : NULL);\n-\t\tkwsincr(p->kws, p->pattern, p->patternlen);\n-\t\tkwsprep(p->kws);\n-\t\treturn;\n-\t}\n-\n \tif (opt->fixed) {\n-\t\t/*\n-\t\t * We come here when the pattern has the non-ascii\n-\t\t * characters we cannot case-fold, and asked to\n-\t\t * ignore-case.\n-\t\t */\n \t\tcompile_fixed_regexp(p, opt);\n \t\treturn;\n \t}\n@@ -1042,9 +1004,7 @@ void free_grep_patterns(struct grep_opt *opt)\n \t\tcase GREP_PATTERN: /* atom */\n \t\tcase GREP_PATTERN_HEAD:\n \t\tcase GREP_PATTERN_BODY:\n-\t\t\tif (p->kws)\n-\t\t\t\tkwsfree(p->kws);\n-\t\t\telse if (p->pcre1_regexp)\n+\t\t\tif (p->pcre1_regexp)\n \t\t\t\tfree_pcre1_regexp(p);\n \t\t\telse if (p->pcre2_pattern)\n \t\t\t\tfree_pcre2_pattern(p);\n@@ -1104,29 +1064,12 @@ static void show_name(struct grep_opt *opt, const char *name)\n \topt->output(opt, opt->null_following_name ? \"\\0\" : \"\\n\", 1);\n }\n \n-static int fixmatch(struct grep_pat *p, char *line, char *eol,\n-\t\t regmatch_t *match)\n-{\n-\tstruct kwsmatch kwsm;\n-\tsize_t offset = kwsexec(p->kws, line, eol - line, &kwsm);\n-\tif (offset == -1) {\n-\t\tmatch->rm_so = match->rm_eo = -1;\n-\t\treturn REG_NOMATCH;\n-\t} else {\n-\t\tmatch->rm_so = offset;\n-\t\tmatch->rm_eo = match->rm_so + kwsm.size[0];\n-\t\treturn 0;\n-\t}\n-}\n-\n static int patmatch(struct grep_pat *p, char *line, char *eol,\n \t\t regmatch_t *match, int eflags)\n {\n \tint hit;\n \n-\tif (p->fixed)\n-\t\thit = !fixmatch(p, line, eol, match);\n-\telse if (p->pcre1_regexp)\n+\tif (p->pcre1_regexp)\n \t\thit = !pcre1match(p, line, eol, match, eflags);\n \telse if (p->pcre2_pattern)\n \t\thit = !pcre2match(p, line, eol, match, eflags);\ndiff --git a/grep.h b/grep.h\nindex 4bb8a79d93..d35a137fcb 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -32,7 +32,6 @@ typedef int pcre2_compile_context;\n typedef int pcre2_match_context;\n typedef int pcre2_jit_stack;\n #endif\n-#include \"kwset.h\"\n #include \"thread-utils.h\"\n #include \"userdiff.h\"\n \n@@ -97,7 +96,6 @@ struct grep_pat {\n \tpcre2_match_context *pcre2_match_context;\n \tpcre2_jit_stack *pcre2_jit_stack;\n \tuint32_t pcre2_jit_on;\n-\tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\n \tunsigned word_regexp:1;\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378233","messageId":"20190627233912.7117-10-avarab@gmail.com","threadId":"51300","inReplyTo":"20190626000329.32475-1-avarab@gmail.com","subject":"[PATCH v2 9/9] grep: use PCRE v2 for optimized fixed-string search","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-27T23:39:12Z","receivedAt":"2019-06-27T23:39:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Bring back optimized fixed-string search for \"grep\", this time with\nPCRE v2 as an optional backend. As noted in [1] with kwset we were\nslower than PCRE v1 and v2 JIT with the kwset backend, so that\noptimization was counterproductive.\n\nThis brings back the optimization for \"--fixed-strings\", without\nchanging the semantics of having a NUL-byte in patterns. As seen in\nprevious commits in this series we could support it now, but I'd\nrather just leave that edge-case aside so we don't have one behavior\nor the other depending what \"--fixed-strings\" backend we're using. It\nmakes the behavior harder to understand and document, and makes tests\nfor the different backends more painful.\n\nI could also support the PCRE v1 backend here, but that would make the\ncode more complex. I'd rather aim for simplicity here and in future\nchanges to the diffcore. We're not going to have someone who\nabsolutely must have faster search, but for whom building PCRE v2\nisn't acceptable.\n\nThe difference between this series of commits and the current \"master\"\nis, using the same t/perf commands shown in the last commit:\n\nplain grep:\n\n Test origin/master HEAD\n -------------------------------------------------------------------------\n 7821.1: fixed grep int 0.55(1.67+0.56) 0.41(0.98+0.60) -25.5%\n 7821.2: basic grep int 0.58(1.65+0.52) 0.41(0.96+0.57) -29.3%\n 7821.3: extended grep int 0.57(1.66+0.49) 0.42(0.93+0.60) -26.3%\n 7821.4: perl grep int 0.54(1.67+0.50) 0.43(0.88+0.65) -20.4%\n 7821.6: fixed grep uncommon 0.21(0.52+0.42) 0.16(0.24+0.51) -23.8%\n 7821.7: basic grep uncommon 0.20(0.49+0.45) 0.17(0.28+0.47) -15.0%\n 7821.8: extended grep uncommon 0.20(0.54+0.39) 0.16(0.25+0.50) -20.0%\n 7821.9: perl grep uncommon 0.20(0.58+0.36) 0.16(0.23+0.50) -20.0%\n 7821.11: fixed grep æ 0.35(1.24+0.43) 0.16(0.23+0.50) -54.3%\n 7821.12: basic grep æ 0.36(1.29+0.38) 0.16(0.20+0.54) -55.6%\n 7821.13: extended grep æ 0.35(1.23+0.44) 0.16(0.24+0.50) -54.3%\n 7821.14: perl grep æ 0.35(1.33+0.34) 0.16(0.28+0.46) -54.3%\n\ngrep with -i:\n\n Test origin/master HEAD\n ----------------------------------------------------------------------------\n 7821.1: fixed grep -i int 0.62(1.81+0.70) 0.47(1.11+0.64) -24.2%\n 7821.2: basic grep -i int 0.67(1.90+0.53) 0.46(1.07+0.62) -31.3%\n 7821.3: extended grep -i int 0.62(1.92+0.53) 0.53(1.12+0.58) -14.5%\n 7821.4: perl grep -i int 0.66(1.85+0.58) 0.45(1.10+0.59) -31.8%\n 7821.6: fixed grep -i uncommon 0.21(0.54+0.43) 0.17(0.20+0.55) -19.0%\n 7821.7: basic grep -i uncommon 0.20(0.52+0.45) 0.17(0.29+0.48) -15.0%\n 7821.8: extended grep -i uncommon 0.21(0.52+0.44) 0.17(0.26+0.50) -19.0%\n 7821.9: perl grep -i uncommon 0.21(0.53+0.44) 0.17(0.20+0.56) -19.0%\n 7821.11: fixed grep -i æ 0.26(0.79+0.44) 0.16(0.29+0.46) -38.5%\n 7821.12: basic grep -i æ 0.26(0.79+0.42) 0.16(0.20+0.54) -38.5%\n 7821.13: extended grep -i æ 0.26(0.84+0.39) 0.16(0.24+0.50) -38.5%\n 7821.14: perl grep -i æ 0.16(0.24+0.49) 0.17(0.25+0.51) +6.3%\n\nplain log:\n\n Test origin/master HEAD\n --------------------------------------------------------------------------------\n 4221.1: fixed log --grep='int' 7.24(6.95+0.28) 7.20(6.95+0.18) -0.6%\n 4221.2: basic log --grep='int' 7.31(6.97+0.22) 7.20(6.93+0.21) -1.5%\n 4221.3: extended log --grep='int' 7.37(7.04+0.24) 7.22(6.91+0.25) -2.0%\n 4221.4: perl log --grep='int' 7.31(7.04+0.21) 7.19(6.89+0.21) -1.6%\n 4221.6: fixed log --grep='uncommon' 6.93(6.59+0.32) 7.04(6.66+0.37) +1.6%\n 4221.7: basic log --grep='uncommon' 6.92(6.58+0.29) 7.08(6.75+0.29) +2.3%\n 4221.8: extended log --grep='uncommon' 6.92(6.55+0.31) 7.00(6.68+0.31) +1.2%\n 4221.9: perl log --grep='uncommon' 7.03(6.59+0.33) 7.12(6.73+0.34) +1.3%\n 4221.11: fixed log --grep='æ' 7.41(7.08+0.28) 7.05(6.76+0.29) -4.9%\n 4221.12: basic log --grep='æ' 7.39(6.99+0.33) 7.00(6.68+0.25) -5.3%\n 4221.13: extended log --grep='æ' 7.34(7.00+0.25) 7.15(6.81+0.31) -2.6%\n 4221.14: perl log --grep='æ' 7.43(7.13+0.26) 7.01(6.60+0.36) -5.7%\n\nlog with -i:\n\n Test origin/master HEAD\n ------------------------------------------------------------------------------------\n 4221.1: fixed log -i --grep='int' 7.31(7.07+0.24) 7.23(7.00+0.22) -1.1%\n 4221.2: basic log -i --grep='int' 7.40(7.08+0.28) 7.19(6.92+0.20) -2.8%\n 4221.3: extended log -i --grep='int' 7.43(7.13+0.25) 7.27(6.99+0.21) -2.2%\n 4221.4: perl log -i --grep='int' 7.34(7.10+0.24) 7.10(6.90+0.19) -3.3%\n 4221.6: fixed log -i --grep='uncommon' 7.07(6.71+0.32) 7.11(6.77+0.28) +0.6%\n 4221.7: basic log -i --grep='uncommon' 6.99(6.64+0.28) 7.12(6.69+0.38) +1.9%\n 4221.8: extended log -i --grep='uncommon' 7.11(6.74+0.32) 7.10(6.77+0.27) -0.1%\n 4221.9: perl log -i --grep='uncommon' 6.98(6.60+0.29) 7.05(6.64+0.34) +1.0%\n 4221.11: fixed log -i --grep='æ' 7.85(7.45+0.34) 7.03(6.68+0.32) -10.4%\n 4221.12: basic log -i --grep='æ' 7.87(7.49+0.29) 7.06(6.69+0.31) -10.3%\n 4221.13: extended log -i --grep='æ' 7.87(7.54+0.31) 7.09(6.69+0.31) -9.9%\n 4221.14: perl log -i --grep='æ' 7.06(6.77+0.28) 6.91(6.57+0.31) -2.1%\n\nSo as with e05b027627 (\"grep: use PCRE v2 for optimized fixed-string\nsearch\", 2019-06-26) there's a huge improvement in performance for\n\"grep\", but in \"log\" most of our time is spent elsewhere, so we don't\nnotice it that much.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 49 insertions(+), 2 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 4468519d5c..fc0ed73ef3 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -356,6 +356,18 @@ static NORETURN void compile_regexp_failed(const struct grep_pat *p,\n \tdie(\"%s'%s': %s\", where, p->pattern, error);\n }\n \n+static int is_fixed(const char *s, size_t len)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (is_regex_special(s[i]))\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n #ifdef USE_LIBPCRE1\n static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n {\n@@ -602,7 +614,6 @@ static int pcre2match(struct grep_pat *p, const char *line, const char *eol,\n static void free_pcre2_pattern(struct grep_pat *p)\n {\n }\n-#endif /* !USE_LIBPCRE2 */\n \n static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n@@ -623,11 +634,13 @@ static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tcompile_regexp_failed(p, errbuf);\n \t}\n }\n+#endif /* !USE_LIBPCRE2 */\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tint err;\n \tint regflags = REG_NEWLINE;\n+\tint pat_is_fixed;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n@@ -636,8 +649,42 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n \t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n \n-\tif (opt->fixed) {\n+\tpat_is_fixed = is_fixed(p->pattern, p->patternlen);\n+\tif (opt->fixed || pat_is_fixed) {\n+#ifdef USE_LIBPCRE2\n+\t\topt->pcre2 = 1;\n+\t\tif (pat_is_fixed) {\n+\t\t\tcompile_pcre2_pattern(p, opt);\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * E.g. t7811-grep-open.sh relies on the\n+\t\t\t * pattern being restored.\n+\t\t\t */\n+\t\t\tchar *old_pattern = p->pattern;\n+\t\t\tsize_t old_patternlen = p->patternlen;\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\t\t/*\n+\t\t\t * There is the PCRE2_LITERAL flag, but it's\n+\t\t\t * only in PCRE v2 10.30 and later. Needing to\n+\t\t\t * ifdef our way around that and dealing with\n+\t\t\t * it + PCRE2_MULTILINE being an error is more\n+\t\t\t * complex than just quoting this ourselves.\n+\t\t\t*/\n+\t\t\tstrbuf_add(&sb, \"\\\\Q\", 2);\n+\t\t\tstrbuf_add(&sb, p->pattern, p->patternlen);\n+\t\t\tstrbuf_add(&sb, \"\\\\E\", 2);\n+\n+\t\t\tp->pattern = sb.buf;\n+\t\t\tp->patternlen = sb.len;\n+\t\t\tcompile_pcre2_pattern(p, opt);\n+\t\t\tp->pattern = old_pattern;\n+\t\t\tp->patternlen = old_patternlen;\n+\t\t\tstrbuf_release(&sb);\n+\t\t}\n+#else /* !USE_LIBPCRE2 */\n \t\tcompile_fixed_regexp(p, opt);\n+#endif /* !USE_LIBPCRE2 */\n \t\treturn;\n \t}\n \n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378242","messageId":"87y31m6un7.fsf@evledraar.gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"Re: [PATCH v2 0/9] grep: move from kwset to optional PCRE v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-06-28T07:23:08Z","receivedAt":"2019-06-28T07:23:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jun 28 2019, Ævar Arnfjörð Bjarmason wrote:\n\n> A non-RFC since it seem people like this approach.\n>\n> This should fix the test failure noted by Johannes, there's two new\n> patches at the start of this series. They address a bug that was there\n> for a long time, but I happened to trip over since PCRE is more strict\n> about UTF-8 validation than kwset (which doesn't care at all).\n>\n> I also added performance numbers to the relevant commit messages, took\n> brian's suggestion of saying \"NUL-byte\" instead of \"\\0\", and did some\n> other copyediting of my own.\n>\n> The rest of the code changes are all just comments & rewording of\n> previously added comments.\n\nJunio. I thought I'd submit this in before your merge to \"next\", but I\nsee that happened. Are you OK with rewinding it for this (& maybe\nsomething else) or should I submit a v3 rebased on \"next\"?\n\nI'd really prefer the improved commit messages with performance numbers,\nand thought I'd have time to work on those details since it was an\nRFC/PATCH :)\n\n> Ævar Arnfjörð Bjarmason (9):\n> log tests: test regex backends in \"--encode=\" tests\n> grep: don't use PCRE2?_UTF8 with \"log --encoding=\"\n> grep: inline the return value of a function call used only once\n> grep tests: move \"grep binary\" alongside the rest\n> grep tests: move binary pattern tests into their own file\n> grep: make the behavior for NUL-byte in patterns sane\n> grep: drop support for \\0 in --fixed-strings \n> grep: remove the kwset optimization\n> grep: use PCRE v2 for optimized fixed-string search\n>\n> Documentation/git-grep.txt | 17 +++\n> grep.c | 115 +++++++---------\n> grep.h | 3 +-\n> revision.c | 3 +\n> t/t4210-log-i18n.sh | 39 +++++-\n> ...a1.sh => t7008-filter-branch-null-sha1.sh} | 0\n> ...08-grep-binary.sh => t7815-grep-binary.sh} | 101 --------------\n> t/t7816-grep-binary-pattern.sh | 127 ++++++++++++++++++\n> 8 files changed, 233 insertions(+), 172 deletions(-)\n> rename t/{t7009-filter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} (100%)\n> rename t/{t7008-grep-binary.sh => t7815-grep-binary.sh} (55%)\n> create mode 100755 t/t7816-grep-binary-pattern.sh\n>\n> Range-diff:\n> -: ---------- > 1: cfc01f49d3 log tests: test regex backends in \"--encode=\" tests\n> -: ---------- > 2: 4b59eb32f0 grep: don't use PCRE2?_UTF8 with \"log --encoding=\"\n> 1: ad55d3be7e = 3: cc4d3b50d5 grep: inline the return value of a function call used only once\n> 2: 650bcc8582 = 4: d9b29bdd89 grep tests: move \"grep binary\" alongside the rest\n> 3: ef10a8820d ! 5: f85614f435 grep tests: move binary pattern tests into their own file\n> @@ -2,9 +2,10 @@\n>\n> grep tests: move binary pattern tests into their own file\n>\n> - Move the tests for \"-f \" where \"\" contains a \"\\0\" pattern\n> - into their own file. I added most of these tests in 966be95549 (\"grep:\n> - add tests to fix blind spots with \\0 patterns\", 2017-05-20).\n> + Move the tests for \"-f \" where \"\" contains a NUL byte\n> + pattern into their own file. I added most of these tests in\n> + 966be95549 (\"grep: add tests to fix blind spots with \\0 patterns\",\n> + 2017-05-20).\n>\n> Whether a regex engine supports matching binary content is very\n> different from whether it matches binary patterns. Since\n> @@ -14,8 +15,8 @@\n> engine can sensibly match binary patterns.\n>\n> Since 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) we've been punting\n> - patterns containing \"\\0\" and considering them fixed, except in cases\n> - where \"--ignore-case\" is provided and they're non-ASCII, see\n> + patterns containing NUL-byte and considering them fixed, except in\n> + cases where \"--ignore-case\" is provided and they're non-ASCII, see\n> 5c1ebcca4d (\"grep/icase: avoid kwsset on literal non-ascii strings\",\n> 2016-06-25). Subsequent commits will change this behavior.\n>\n> 4: 03e5637efc ! 6: 90afca8707 grep: make the behavior for \\0 in patterns sane\n> @@ -1,12 +1,13 @@\n> Author: Ævar Arnfjörð Bjarmason \n>\n> - grep: make the behavior for \\0 in patterns sane\n> + grep: make the behavior for NUL-byte in patterns sane\n>\n> - The behavior of \"grep\" when patterns contained \"\\0\" has always been\n> - haphazard, and has served the vagaries of the implementation more than\n> - anything else. A \"\\0\" in a pattern can only be provided via \"-f\n> - \", and since pickaxe (log search) has no such flag \"\\0\" in\n> - patterns has only ever been supported by \"grep\".\n> + The behavior of \"grep\" when patterns contained a NUL-byte has always\n> + been haphazard, and has served the vagaries of the implementation more\n> + than anything else. A pattern containing a NUL-byte can only be\n> + provided via \"-f \". Since pickaxe (log search) has no such flag\n> + the NUL-byte in patterns has only ever been supported by \"grep\" (and\n> + not \"log --grep\").\n>\n> Since 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) patterns containing\n> \"\\0\" were considered fixed. In 966be95549 (\"grep: add tests to fix\n> @@ -14,9 +15,9 @@\n> behavior.\n>\n> Change the behavior to do the obvious thing, i.e. don't silently\n> - discard a regex pattern and make it implicitly fixed just because it\n> - contains a \\0. Instead die if e.g. --basic-regexp is combined with\n> - such a pattern.\n> + discard a regex pattern and make it implicitly fixed just because they\n> + contain a NUL-byte. Instead die if the backend in question can't\n> + handle them, e.g. --basic-regexp is combined with such a pattern.\n>\n> This is desired because from a user's point of view it's the obvious\n> thing to do. Whether we support BRE/ERE/Perl syntax is different from\n> 5: b9aad3ec1c ! 7: 526b925fdc grep: drop support for \\0 in --fixed-strings \n> @@ -2,15 +2,14 @@\n>\n> grep: drop support for \\0 in --fixed-strings \n>\n> - Change \"-f \" to not support patterns with \"\\0\" in them under\n> - --fixed-strings, we'll now only support these under --perl-regexp with\n> - PCRE v2.\n> + Change \"-f \" to not support patterns with a NUL-byte in them\n> + under --fixed-strings. We'll now only support these under\n> + \"--perl-regexp\" with PCRE v2.\n>\n> - A previous change to Documentation/git-grep.txt changed the\n> - description of \"-f \" to be vague enough as to not promise that\n> - this would work, and by dropping support for this we make it a whole\n> - lot easier to move away from the kwset backend, which a subsequent\n> - change will try to do.\n> + A previous change to grep's documentation changed the description of\n> + \"-f \" to be vague enough as to not promise that this would work.\n> + By dropping support for this we make it a whole lot easier to move\n> + away from the kwset backend, which we'll do in a subsequent change.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n>\n> 6: 3587be009a ! 8: 14269bb295 grep: remove the kwset optimization\n> @@ -2,9 +2,99 @@\n>\n> grep: remove the kwset optimization\n>\n> - A later change will replace this optimization with a different one,\n> - but as removing it and running the tests demonstrates no grep\n> - semantics depend on this backend anymore.\n> + A later change will replace this optimization with optimistic use of\n> + PCRE v2. I'm completely removing it as an intermediate step, as\n> + opposed to replacing it with PCRE v2, to demonstrate that no grep\n> + semantics depend on this (or any other) optimization for the fixed\n> + backend anymore.\n> +\n> + For now this is mostly (but not entirely) a performance regression, as\n> + shown by this hacky one-liner:\n> +\n> + for opt in '' ' -i'\n> + do\n> + GIT_PERF_7821_GREP_OPTS=$opt GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE=YesPlease' ./run origin/master HEAD -- p7821-grep-engines-fixed.sh\n> + done &&\n> + for opt in '' ' -i'\n> + do GIT_PERF_4221_LOG_OPTS=$opt GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE=YesPlease' ./run origin/master HEAD -- p4221-log-grep-engines-fixed.sh\n> + done\n> +\n> + Which produces:\n> +\n> + plain grep:\n> +\n> + Test origin/master HEAD\n> + -------------------------------------------------------------------------\n> + 7821.1: fixed grep int 0.55(1.60+0.63) 0.82(3.11+0.51) +49.1%\n> + 7821.2: basic grep int 0.62(1.68+0.49) 0.85(3.02+0.52) +37.1%\n> + 7821.3: extended grep int 0.61(1.63+0.53) 0.91(3.09+0.44) +49.2%\n> + 7821.4: perl grep int 0.55(1.60+0.57) 0.41(0.93+0.57) -25.5%\n> + 7821.6: fixed grep uncommon 0.20(0.50+0.44) 0.35(1.27+0.42) +75.0%\n> + 7821.7: basic grep uncommon 0.20(0.49+0.45) 0.35(1.29+0.41) +75.0%\n> + 7821.8: extended grep uncommon 0.20(0.45+0.48) 0.35(1.25+0.44) +75.0%\n> + 7821.9: perl grep uncommon 0.20(0.53+0.41) 0.16(0.24+0.49) -20.0%\n> + 7821.11: fixed grep æ 0.35(1.27+0.40) 0.25(0.82+0.39) -28.6%\n> + 7821.12: basic grep æ 0.35(1.28+0.38) 0.25(0.75+0.44) -28.6%\n> + 7821.13: extended grep æ 0.36(1.21+0.46) 0.25(0.86+0.35) -30.6%\n> + 7821.14: perl grep æ 0.35(1.33+0.34) 0.16(0.26+0.47) -54.3%\n> +\n> + grep with -i:\n> +\n> + Test origin/master HEAD\n> + -----------------------------------------------------------------------------\n> + 7821.1: fixed grep -i int 0.61(1.84+0.64) 1.11(4.12+0.64) +82.0%\n> + 7821.2: basic grep -i int 0.72(1.86+0.57) 1.15(4.48+0.49) +59.7%\n> + 7821.3: extended grep -i int 0.94(1.83+0.60) 1.53(4.12+0.58) +62.8%\n> + 7821.4: perl grep -i int 0.66(1.82+0.59) 0.55(1.08+0.58) -16.7%\n> + 7821.6: fixed grep -i uncommon 0.21(0.51+0.44) 0.44(1.74+0.34) +109.5%\n> + 7821.7: basic grep -i uncommon 0.21(0.55+0.41) 0.44(1.72+0.40) +109.5%\n> + 7821.8: extended grep -i uncommon 0.21(0.57+0.39) 0.42(1.64+0.45) +100.0%\n> + 7821.9: perl grep -i uncommon 0.21(0.48+0.48) 0.17(0.30+0.45) -19.0%\n> + 7821.11: fixed grep -i æ 0.25(0.73+0.45) 0.25(0.75+0.45) +0.0%\n> + 7821.12: basic grep -i æ 0.25(0.71+0.49) 0.26(0.77+0.44) +4.0%\n> + 7821.13: extended grep -i æ 0.25(0.75+0.44) 0.25(0.74+0.46) +0.0%\n> + 7821.14: perl grep -i æ 0.17(0.26+0.48) 0.16(0.20+0.52) -5.9%\n> +\n> + plain log:\n> +\n> + Test origin/master HEAD\n> + ---------------------------------------------------------------------------------\n> + 4221.1: fixed log --grep='int' 7.31(7.06+0.21) 8.11(7.85+0.20) +10.9%\n> + 4221.2: basic log --grep='int' 7.30(6.94+0.27) 8.16(7.89+0.19) +11.8%\n> + 4221.3: extended log --grep='int' 7.34(7.05+0.21) 8.08(7.76+0.25) +10.1%\n> + 4221.4: perl log --grep='int' 7.27(6.94+0.24) 7.05(6.76+0.25) -3.0%\n> + 4221.6: fixed log --grep='uncommon' 6.97(6.62+0.32) 7.86(7.51+0.30) +12.8%\n> + 4221.7: basic log --grep='uncommon' 7.05(6.69+0.29) 7.89(7.60+0.28) +11.9%\n> + 4221.8: extended log --grep='uncommon' 6.89(6.56+0.32) 7.99(7.66+0.24) +16.0%\n> + 4221.9: perl log --grep='uncommon' 7.02(6.66+0.33) 6.97(6.54+0.36) -0.7%\n> + 4221.11: fixed log --grep='æ' 7.37(7.03+0.33) 7.67(7.30+0.31) +4.1%\n> + 4221.12: basic log --grep='æ' 7.41(7.00+0.31) 7.60(7.28+0.26) +2.6%\n> + 4221.13: extended log --grep='æ' 7.35(6.96+0.38) 7.73(7.31+0.34) +5.2%\n> + 4221.14: perl log --grep='æ' 7.43(7.10+0.32) 6.95(6.61+0.27) -6.5%\n> +\n> + log with -i:\n> +\n> + Test origin/master HEAD\n> + ------------------------------------------------------------------------------------\n> + 4221.1: fixed log -i --grep='int' 7.40(7.05+0.23) 8.66(8.38+0.20) +17.0%\n> + 4221.2: basic log -i --grep='int' 7.39(7.09+0.23) 8.67(8.39+0.20) +17.3%\n> + 4221.3: extended log -i --grep='int' 7.29(6.99+0.26) 8.69(8.31+0.26) +19.2%\n> + 4221.4: perl log -i --grep='int' 7.42(7.16+0.21) 7.14(6.80+0.24) -3.8%\n> + 4221.6: fixed log -i --grep='uncommon' 6.94(6.58+0.35) 8.43(8.04+0.30) +21.5%\n> + 4221.7: basic log -i --grep='uncommon' 6.95(6.62+0.31) 8.34(7.93+0.32) +20.0%\n> + 4221.8: extended log -i --grep='uncommon' 7.06(6.75+0.25) 8.32(7.98+0.31) +17.8%\n> + 4221.9: perl log -i --grep='uncommon' 6.96(6.69+0.26) 7.04(6.64+0.32) +1.1%\n> + 4221.11: fixed log -i --grep='æ' 7.92(7.55+0.33) 7.86(7.44+0.34) -0.8%\n> + 4221.12: basic log -i --grep='æ' 7.88(7.49+0.32) 7.84(7.46+0.34) -0.5%\n> + 4221.13: extended log -i --grep='æ' 7.91(7.51+0.32) 7.87(7.48+0.32) -0.5%\n> + 4221.14: perl log -i --grep='æ' 7.01(6.59+0.35) 6.99(6.64+0.28) -0.3%\n> +\n> + Some of those, as noted in [1] are because PCRE is faster at finding\n> + fixed strings. This looks bad for some engines, but in the next change\n> + we'll optimistically use PCRE v2 for all of these, so it'll look\n> + better.\n> +\n> + 1. https://public-inbox.org/git/87v9x793qi.fsf@evledraar.gmail.com/\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n>\n> 7: 5bc25c03b8 ! 9: c0fd75d102 grep: use PCRE v2 for optimized fixed-string search\n> @@ -7,19 +7,95 @@\n> slower than PCRE v1 and v2 JIT with the kwset backend, so that\n> optimization was counterproductive.\n>\n> - This brings back the optimization for \"-F\", without changing the\n> - semantics of \"\\0\" in patterns. As seen in previous commits in this\n> - series we could support it now, but I'd rather just leave that\n> - edge-case aside so the tests don't need to do one thing or the other\n> - depending on what --fixed-strings backend we're using.\n> -\n> - I could also support the v1 backend here, but that would make the code\n> - more complex, and I'd rather aim for simplicity here and in future\n> + This brings back the optimization for \"--fixed-strings\", without\n> + changing the semantics of having a NUL-byte in patterns. As seen in\n> + previous commits in this series we could support it now, but I'd\n> + rather just leave that edge-case aside so we don't have one behavior\n> + or the other depending what \"--fixed-strings\" backend we're using. It\n> + makes the behavior harder to understand and document, and makes tests\n> + for the different backends more painful.\n> +\n> + I could also support the PCRE v1 backend here, but that would make the\n> + code more complex. I'd rather aim for simplicity here and in future\n> changes to the diffcore. We're not going to have someone who\n> absolutely must have faster search, but for whom building PCRE v2\n> isn't acceptable.\n>\n> - 1. https://public-inbox.org/git/87v9x793qi.fsf@evledraar.gmail.com/\n> + The difference between this series of commits and the current \"master\"\n> + is, using the same t/perf commands shown in the last commit:\n> +\n> + plain grep:\n> +\n> + Test origin/master HEAD\n> + -------------------------------------------------------------------------\n> + 7821.1: fixed grep int 0.55(1.67+0.56) 0.41(0.98+0.60) -25.5%\n> + 7821.2: basic grep int 0.58(1.65+0.52) 0.41(0.96+0.57) -29.3%\n> + 7821.3: extended grep int 0.57(1.66+0.49) 0.42(0.93+0.60) -26.3%\n> + 7821.4: perl grep int 0.54(1.67+0.50) 0.43(0.88+0.65) -20.4%\n> + 7821.6: fixed grep uncommon 0.21(0.52+0.42) 0.16(0.24+0.51) -23.8%\n> + 7821.7: basic grep uncommon 0.20(0.49+0.45) 0.17(0.28+0.47) -15.0%\n> + 7821.8: extended grep uncommon 0.20(0.54+0.39) 0.16(0.25+0.50) -20.0%\n> + 7821.9: perl grep uncommon 0.20(0.58+0.36) 0.16(0.23+0.50) -20.0%\n> + 7821.11: fixed grep æ 0.35(1.24+0.43) 0.16(0.23+0.50) -54.3%\n> + 7821.12: basic grep æ 0.36(1.29+0.38) 0.16(0.20+0.54) -55.6%\n> + 7821.13: extended grep æ 0.35(1.23+0.44) 0.16(0.24+0.50) -54.3%\n> + 7821.14: perl grep æ 0.35(1.33+0.34) 0.16(0.28+0.46) -54.3%\n> +\n> + grep with -i:\n> +\n> + Test origin/master HEAD\n> + ----------------------------------------------------------------------------\n> + 7821.1: fixed grep -i int 0.62(1.81+0.70) 0.47(1.11+0.64) -24.2%\n> + 7821.2: basic grep -i int 0.67(1.90+0.53) 0.46(1.07+0.62) -31.3%\n> + 7821.3: extended grep -i int 0.62(1.92+0.53) 0.53(1.12+0.58) -14.5%\n> + 7821.4: perl grep -i int 0.66(1.85+0.58) 0.45(1.10+0.59) -31.8%\n> + 7821.6: fixed grep -i uncommon 0.21(0.54+0.43) 0.17(0.20+0.55) -19.0%\n> + 7821.7: basic grep -i uncommon 0.20(0.52+0.45) 0.17(0.29+0.48) -15.0%\n> + 7821.8: extended grep -i uncommon 0.21(0.52+0.44) 0.17(0.26+0.50) -19.0%\n> + 7821.9: perl grep -i uncommon 0.21(0.53+0.44) 0.17(0.20+0.56) -19.0%\n> + 7821.11: fixed grep -i æ 0.26(0.79+0.44) 0.16(0.29+0.46) -38.5%\n> + 7821.12: basic grep -i æ 0.26(0.79+0.42) 0.16(0.20+0.54) -38.5%\n> + 7821.13: extended grep -i æ 0.26(0.84+0.39) 0.16(0.24+0.50) -38.5%\n> + 7821.14: perl grep -i æ 0.16(0.24+0.49) 0.17(0.25+0.51) +6.3%\n> +\n> + plain log:\n> +\n> + Test origin/master HEAD\n> + --------------------------------------------------------------------------------\n> + 4221.1: fixed log --grep='int' 7.24(6.95+0.28) 7.20(6.95+0.18) -0.6%\n> + 4221.2: basic log --grep='int' 7.31(6.97+0.22) 7.20(6.93+0.21) -1.5%\n> + 4221.3: extended log --grep='int' 7.37(7.04+0.24) 7.22(6.91+0.25) -2.0%\n> + 4221.4: perl log --grep='int' 7.31(7.04+0.21) 7.19(6.89+0.21) -1.6%\n> + 4221.6: fixed log --grep='uncommon' 6.93(6.59+0.32) 7.04(6.66+0.37) +1.6%\n> + 4221.7: basic log --grep='uncommon' 6.92(6.58+0.29) 7.08(6.75+0.29) +2.3%\n> + 4221.8: extended log --grep='uncommon' 6.92(6.55+0.31) 7.00(6.68+0.31) +1.2%\n> + 4221.9: perl log --grep='uncommon' 7.03(6.59+0.33) 7.12(6.73+0.34) +1.3%\n> + 4221.11: fixed log --grep='æ' 7.41(7.08+0.28) 7.05(6.76+0.29) -4.9%\n> + 4221.12: basic log --grep='æ' 7.39(6.99+0.33) 7.00(6.68+0.25) -5.3%\n> + 4221.13: extended log --grep='æ' 7.34(7.00+0.25) 7.15(6.81+0.31) -2.6%\n> + 4221.14: perl log --grep='æ' 7.43(7.13+0.26) 7.01(6.60+0.36) -5.7%\n> +\n> + log with -i:\n> +\n> + Test origin/master HEAD\n> + ------------------------------------------------------------------------------------\n> + 4221.1: fixed log -i --grep='int' 7.31(7.07+0.24) 7.23(7.00+0.22) -1.1%\n> + 4221.2: basic log -i --grep='int' 7.40(7.08+0.28) 7.19(6.92+0.20) -2.8%\n> + 4221.3: extended log -i --grep='int' 7.43(7.13+0.25) 7.27(6.99+0.21) -2.2%\n> + 4221.4: perl log -i --grep='int' 7.34(7.10+0.24) 7.10(6.90+0.19) -3.3%\n> + 4221.6: fixed log -i --grep='uncommon' 7.07(6.71+0.32) 7.11(6.77+0.28) +0.6%\n> + 4221.7: basic log -i --grep='uncommon' 6.99(6.64+0.28) 7.12(6.69+0.38) +1.9%\n> + 4221.8: extended log -i --grep='uncommon' 7.11(6.74+0.32) 7.10(6.77+0.27) -0.1%\n> + 4221.9: perl log -i --grep='uncommon' 6.98(6.60+0.29) 7.05(6.64+0.34) +1.0%\n> + 4221.11: fixed log -i --grep='æ' 7.85(7.45+0.34) 7.03(6.68+0.32) -10.4%\n> + 4221.12: basic log -i --grep='æ' 7.87(7.49+0.29) 7.06(6.69+0.31) -10.3%\n> + 4221.13: extended log -i --grep='æ' 7.87(7.54+0.31) 7.09(6.69+0.31) -9.9%\n> + 4221.14: perl log -i --grep='æ' 7.06(6.77+0.28) 6.91(6.57+0.31) -2.1%\n> +\n> + So as with e05b027627 (\"grep: use PCRE v2 for optimized fixed-string\n> + search\", 2019-06-26) there's a huge improvement in performance for\n> + \"grep\", but in \"log\" most of our time is spent elsewhere, so we don't\n> + notice it that much.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n>\n> @@ -81,15 +157,19 @@\n> +\t\t} else {\n> +\t\t\t/*\n> +\t\t\t * E.g. t7811-grep-open.sh relies on the\n> -+\t\t\t * pattern being restored, and unfortunately\n> -+\t\t\t * there's no PCRE compile flag for \"this is\n> -+\t\t\t * fixed\", so we need to munge it to\n> -+\t\t\t * \"\\Q\\E\".\n> ++\t\t\t * pattern being restored.\n> +\t\t\t */\n> +\t\t\tchar *old_pattern = p->pattern;\n> +\t\t\tsize_t old_patternlen = p->patternlen;\n> +\t\t\tstruct strbuf sb = STRBUF_INIT;\n> +\n> ++\t\t\t/*\n> ++\t\t\t * There is the PCRE2_LITERAL flag, but it's\n> ++\t\t\t * only in PCRE v2 10.30 and later. Needing to\n> ++\t\t\t * ifdef our way around that and dealing with\n> ++\t\t\t * it + PCRE2_MULTILINE being an error is more\n> ++\t\t\t * complex than just quoting this ourselves.\n> ++\t\t\t*/\n> +\t\t\tstrbuf_add(&sb, \"\\\\Q\", 2);\n> +\t\t\tstrbuf_add(&sb, p->pattern, p->patternlen);\n> +\t\t\tstrbuf_add(&sb, \"\\\\E\", 2);\n> @@ -101,9 +181,9 @@\n> +\t\t\tp->patternlen = old_patternlen;\n> +\t\t\tstrbuf_release(&sb);\n> +\t\t}\n> -+#else\n> ++#else /* !USE_LIBPCRE2 */\n> \t\tcompile_fixed_regexp(p, opt);\n> -+#endif\n> ++#endif /* !USE_LIBPCRE2 */\n> \t\treturn;\n> \t}\n"},{"id":"378257","messageId":"nycvar.QRO.7.76.6.1906281254370.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"xmqqv9wqalv2.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC/PATCH 7/7] grep: use PCRE v2 for optimized fixed-string search","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-28T10:56:33Z","receivedAt":"2019-06-28T10:56:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 27 Jun 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> >> > > If we would not have plenty of exercise for the PCRE2 build\n> >> > > options, I would be worried. But AFAICT the CI build includes\n> >> > > this all the time, so we're fine.\n> >> >\n> >> > Well, I'd feel safer if it were not \"all the time\", i.e. we know we\n> >> > are testing both sides of the coin.\n> >>\n> >> AFAIR at least the Linux32 job is built without PCRE2 by default. I\n> >> might be wrong on that, though...\n> >\n> > Actually, it seems that _all_ of the Linux builds in our Azure Pipeline\n> > compile without pcre2. It seems you have to pass `USE_LIBPCRE2=1` to\n> > `make`, and we do not do that in `ci/run-build-and-tests.sh` nor in\n> > `azure-pipelines.yml`. I do not even see that for the macOS builds.\n> >\n> > So we got PCRE2 covered only in the Windows build, it seems.\n>\n> OK, it sounds like we have sufficient coverage on both fronts.\n\nMaybe not. With the bug I uncovered that is _only_ triggering an error\nmessage if the PCRE2 in question does not support JIT'ed operations, I am\na bit wary now.\n\nBut I cannot really see a reasonable way to add those axes to the CI\nbuilds.\n\nCiao,\nDscho\n"},{"id":"378273","messageId":"xmqqy31l8zcx.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"87y31m6un7.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/9] grep: move from kwset to optional PCRE v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-28T16:10:38Z","receivedAt":"2019-06-28T16:10:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Junio. I thought I'd submit this in before your merge to \"next\", but I\n> see that happened. Are you OK with rewinding it for this (& maybe\n> something else) or should I submit a v3 rebased on \"next\"?\n\nSensible choices are between\n\n (1) reverting 984c7ccbbf (Merge branch 'ab/no-kwset' into next) and\n then replacing the whole topic with this round, or\n\n (2) queuing an incremental series based on 984c7ccbbf^2 (i.e. the\n tip of ab/no-kwset as of today)\n\nI think the former is probably good enough.\n\n"},{"id":"378451","messageId":"20190701212100.27850-1-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 00/10] grep: move from kwset to optional PCRE v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:50Z","receivedAt":"2019-07-01T21:21:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This v3 has a new patch (3/10) that I believe fixes the regression on\nMinGW Johannes noted in\nhttps://public-inbox.org/git/nycvar.QRO.7.76.6.1907011515150.44@tvgsbejvaqbjf.bet/\n\nAs noted in the updated commit message in 10/10 I believe just\nskipping this test & documenting this in a commit message is the least\namount of suck for now. It's really an existing issue with us doing\nnothing sensible when the log/grep haystack encoding doesn't match the\nneedle encoding supplied via the command line.\n\nWe swept that under the carpet with the kwset backend, but PCRE v2\nexposes it.\n\nÆvar Arnfjörð Bjarmason (10):\n log tests: test regex backends in \"--encode=\" tests\n grep: don't use PCRE2?_UTF8 with \"log --encoding=\"\n t4210: skip more command-line encoding tests on MinGW\n grep: inline the return value of a function call used only once\n grep tests: move \"grep binary\" alongside the rest\n grep tests: move binary pattern tests into their own file\n grep: make the behavior for NUL-byte in patterns sane\n grep: drop support for \\0 in --fixed-strings \n grep: remove the kwset optimization\n grep: use PCRE v2 for optimized fixed-string search\n\n Documentation/git-grep.txt | 17 +++\n grep.c | 115 +++++++---------\n grep.h | 3 +-\n revision.c | 3 +\n t/t4210-log-i18n.sh | 41 +++++-\n ...a1.sh => t7008-filter-branch-null-sha1.sh} | 0\n ...08-grep-binary.sh => t7815-grep-binary.sh} | 101 --------------\n t/t7816-grep-binary-pattern.sh | 127 ++++++++++++++++++\n 8 files changed, 234 insertions(+), 173 deletions(-)\n rename t/{t7009-filter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} (100%)\n rename t/{t7008-grep-binary.sh => t7815-grep-binary.sh} (55%)\n create mode 100755 t/t7816-grep-binary-pattern.sh\n\nRange-diff:\n 1: cfc01f49d3 = 1: cfc01f49d3 log tests: test regex backends in \"--encode=\" tests\n 2: 4b59eb32f0 = 2: 4b59eb32f0 grep: don't use PCRE2?_UTF8 with \"log --encoding=\"\n -: ---------- > 3: 676c76afe4 t4210: skip more command-line encoding tests on MinGW\n 3: cc4d3b50d5 = 4: da9b491f70 grep: inline the return value of a function call used only once\n 4: d9b29bdd89 = 5: c42d3268fa grep tests: move \"grep binary\" alongside the rest\n 5: f85614f435 = 6: 36b9c1c541 grep tests: move binary pattern tests into their own file\n 6: 90afca8707 = 7: 3c54e782e6 grep: make the behavior for NUL-byte in patterns sane\n 7: 526b925fdc = 8: 8e5f418189 grep: drop support for \\0 in --fixed-strings \n 8: 14269bb295 = 9: d1cb8319d5 grep: remove the kwset optimization\n 9: c0fd75d102 ! 10: 4de0c82314 grep: use PCRE v2 for optimized fixed-string search\n @@ -15,6 +15,15 @@\n makes the behavior harder to understand and document, and makes tests\n for the different backends more painful.\n \n + This does change the behavior under non-C locales when \"log\"'s\n + \"--encoding\" option is used and the heystack/needle in the\n + content/command-line doesn't have a matching encoding. See the recent\n + change in \"t4210: skip more command-line encoding tests on MinGW\" in\n + this series. I think that's OK. We did nothing sensible before\n + then (just compared raw bytes that had no hope of matching). At least\n + now the user will get some idea why their grep/log never matches in\n + that edge case.\n +\n I could also support the PCRE v1 backend here, but that would make the\n code more complex. I'd rather aim for simplicity here and in future\n changes to the diffcore. We're not going to have someone who\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378452","messageId":"20190701212100.27850-2-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 01/10] log tests: test regex backends in \"--encode=\" tests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:51Z","receivedAt":"2019-07-01T21:21:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improve the tests added in 04deccda11 (\"log: re-encode commit messages\nbefore grepping\", 2013-02-11) to test the regex backends. Those tests\nnever worked as advertised, due to the is_fixed() optimization in\ngrep.c (which was in place at the time), and the needle in the tests\nbeing a fixed string.\n\nWe'd thus always use the \"fixed\" backend during the tests, which would\nuse the kwset() backend. This backend liberally accepts any garbage\ninput, so invalid encodings would be silently accepted.\n\nIn a follow-up commit we'll fix this bug, this test just demonstrates\nthe existing issue.\n\nIn practice this issue happened on Windows, see [1], but due to the\nstructure of the existing tests & how liberal the kwset code is about\ngarbage we missed this.\n\nCover this blind spot by testing all our regex engines. The PCRE\nbackend will spot these invalid encodings. It's possible that this\ntest breaks the \"basic\" and \"extended\" backends on some systems that\nare more anal than glibc about the encoding of locale issues with\nPOSIX functions that I can remember, but PCRE is more careful about\nthe validation.\n\n1. https://public-inbox.org/git/nycvar.QRO.7.76.6.1906271113090.44@tvgsbejvaqbjf.bet/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t4210-log-i18n.sh | 41 ++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 40 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t4210-log-i18n.sh b/t/t4210-log-i18n.sh\nindex 7c519436ef..86d22c1d4c 100755\n--- a/t/t4210-log-i18n.sh\n+++ b/t/t4210-log-i18n.sh\n@@ -1,12 +1,15 @@\n #!/bin/sh\n \n test_description='test log with i18n features'\n-. ./test-lib.sh\n+. ./lib-gettext.sh\n \n # two forms of é\n utf8_e=$(printf '\\303\\251')\n latin1_e=$(printf '\\351')\n \n+# invalid UTF-8\n+invalid_e=$(printf '\\303\\50)') # \")\" at end to close opening \"(\"\n+\n test_expect_success 'create commits in different encodings' '\n \ttest_tick &&\n \tcat >msg <<-EOF &&\n@@ -53,4 +56,40 @@ test_expect_success 'log --grep does not find non-reencoded values (latin1)' '\n \ttest_must_be_empty actual\n '\n \n+for engine in fixed basic extended perl\n+do\n+\tprereq=\n+\tresult=success\n+\tif test $engine = \"perl\"\n+\tthen\n+\t\tresult=failure\n+\t\tprereq=\"PCRE\"\n+\telse\n+\t\tprereq=\"\"\n+\tfi\n+\tforce_regex=\n+\tif test $engine != \"fixed\"\n+\tthen\n+\t force_regex=.*\n+\tfi\n+\ttest_expect_$result GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n+\t\tcat >expect <<-\\EOF &&\n+\t\tlatin1\n+\t\tutf8\n+\t\tEOF\n+\t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$latin1_e\\\" >actual &&\n+\t\ttest_cmp expect actual\n+\t\"\n+\n+\ttest_expect_success GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n+\t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$utf8_e\\\" >actual &&\n+\t\ttest_must_be_empty actual\n+\t\"\n+\n+\ttest_expect_$result GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not die on invalid UTF-8 value (latin1 + locale + invalid needle)\" \"\n+\t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$invalid_e\\\" >actual &&\n+\t\ttest_must_be_empty actual\n+\t\"\n+done\n+\n test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378453","messageId":"20190701212100.27850-3-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 02/10] grep: don't use PCRE2?_UTF8 with \"log --encoding=\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:52Z","receivedAt":"2019-07-01T21:21:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug introduced in 18547aacf5 (\"grep/pcre: support utf-8\",\n2016-06-25) that was missed due to a blindspot in our tests, as\ndiscussed in the previous commit. I then blindly copied the same bug\nin 94da9193a6 (\"grep: add support for PCRE v2\", 2017-06-01) when\nadding the PCRE v2 code.\n\nWe should not tell PCRE that we're processing UTF-8 just because we're\ndealing with non-ASCII. In the case of e.g. \"log --encoding=<...>\"\nunder is_utf8_locale() the haystack might be in ISO-8859-1, and the\nneedle might be in a non-UTF-8 encoding.\n\nMaybe we should be more strict here and die earlier? Should we also be\nconverting the needle to the encoding in question, and failing if it's\nnot a string that's valid in that encoding? Maybe.\n\nBut for now matching this as non-UTF8 at least has some hope of\nproducing sensible results, since we know that our default heuristic\nof assuming the text to be matched is in the user locale encoding\nisn't true when we've explicitly encoded it to be in a different\nencoding.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 8 ++++----\n grep.h | 1 +\n revision.c | 3 +++\n t/t4210-log-i18n.sh | 6 ++----\n 4 files changed, 10 insertions(+), 8 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex f7c3a5803e..1de4ab49c0 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -388,11 +388,11 @@ static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n \tint options = PCRE_MULTILINE;\n \n \tif (opt->ignore_case) {\n-\t\tif (has_non_ascii(p->pattern))\n+\t\tif (!opt->ignore_locale && has_non_ascii(p->pattern))\n \t\t\tp->pcre1_tables = pcre_maketables();\n \t\toptions |= PCRE_CASELESS;\n \t}\n-\tif (is_utf8_locale() && has_non_ascii(p->pattern))\n+\tif (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern))\n \t\toptions |= PCRE_UTF8;\n \n \tp->pcre1_regexp = pcre_compile(p->pattern, options, &error, &erroffset,\n@@ -498,14 +498,14 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt\n \tp->pcre2_compile_context = NULL;\n \n \tif (opt->ignore_case) {\n-\t\tif (has_non_ascii(p->pattern)) {\n+\t\tif (!opt->ignore_locale && has_non_ascii(p->pattern)) {\n \t\t\tcharacter_tables = pcre2_maketables(NULL);\n \t\t\tp->pcre2_compile_context = pcre2_compile_context_create(NULL);\n \t\t\tpcre2_set_character_tables(p->pcre2_compile_context, character_tables);\n \t\t}\n \t\toptions |= PCRE2_CASELESS;\n \t}\n-\tif (is_utf8_locale() && has_non_ascii(p->pattern))\n+\tif (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern))\n \t\toptions |= PCRE2_UTF;\n \n \tp->pcre2_pattern = pcre2_compile((PCRE2_SPTR)p->pattern,\ndiff --git a/grep.h b/grep.h\nindex 1875880f37..4bb8a79d93 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -173,6 +173,7 @@ struct grep_opt {\n \tint funcbody;\n \tint extended_regexp_option;\n \tint pattern_type_option;\n+\tint ignore_locale;\n \tchar colors[NR_GREP_COLORS][COLOR_MAXLEN];\n \tunsigned pre_context;\n \tunsigned post_context;\ndiff --git a/revision.c b/revision.c\nindex 621feb9df7..a842fb158a 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -28,6 +28,7 @@\n #include \"commit-graph.h\"\n #include \"prio-queue.h\"\n #include \"hashmap.h\"\n+#include \"utf8.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2655,6 +2656,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \n \tgrep_commit_pattern_type(GREP_PATTERN_TYPE_UNSPECIFIED,\n \t\t\t\t &revs->grep_filter);\n+\tif (!is_encoding_utf8(get_log_output_encoding()))\n+\t\trevs->grep_filter.ignore_locale = 1;\n \tcompile_grep_patterns(&revs->grep_filter);\n \n \tif (revs->reverse && revs->reflog_info)\ndiff --git a/t/t4210-log-i18n.sh b/t/t4210-log-i18n.sh\nindex 86d22c1d4c..515bcb7ce1 100755\n--- a/t/t4210-log-i18n.sh\n+++ b/t/t4210-log-i18n.sh\n@@ -59,10 +59,8 @@ test_expect_success 'log --grep does not find non-reencoded values (latin1)' '\n for engine in fixed basic extended perl\n do\n \tprereq=\n-\tresult=success\n \tif test $engine = \"perl\"\n \tthen\n-\t\tresult=failure\n \t\tprereq=\"PCRE\"\n \telse\n \t\tprereq=\"\"\n@@ -72,7 +70,7 @@ do\n \tthen\n \t force_regex=.*\n \tfi\n-\ttest_expect_$result GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n+\ttest_expect_success GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n \t\tcat >expect <<-\\EOF &&\n \t\tlatin1\n \t\tutf8\n@@ -86,7 +84,7 @@ do\n \t\ttest_must_be_empty actual\n \t\"\n \n-\ttest_expect_$result GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not die on invalid UTF-8 value (latin1 + locale + invalid needle)\" \"\n+\ttest_expect_success GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not die on invalid UTF-8 value (latin1 + locale + invalid needle)\" \"\n \t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$invalid_e\\\" >actual &&\n \t\ttest_must_be_empty actual\n \t\"\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378454","messageId":"20190701212100.27850-4-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 03/10] t4210: skip more command-line encoding tests on MinGW","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:53Z","receivedAt":"2019-07-01T21:21:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In 5212f91deb (\"t4210: skip command-line encoding tests on mingw\",\n2014-07-17) the positive tests in this file were skipped. That left\nthe negative tests that don't produce a match.\n\nAn upcoming change to migrate the \"fixed\" backend of grep to PCRE v2\nwill cause these \"log\" commands to produce an error instead on\nMinGW. This is because the command-line on that platform implicitly\nhas its encoding changed before being passed to git. See [1].\n\n1. https://public-inbox.org/git/nycvar.QRO.7.76.6.1907011515150.44@tvgsbejvaqbjf.bet/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t4210-log-i18n.sh | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t4210-log-i18n.sh b/t/t4210-log-i18n.sh\nindex 515bcb7ce1..6e61f57f09 100755\n--- a/t/t4210-log-i18n.sh\n+++ b/t/t4210-log-i18n.sh\n@@ -51,7 +51,7 @@ test_expect_success !MINGW 'log --grep does not find non-reencoded values (utf8)\n \ttest_must_be_empty actual\n '\n \n-test_expect_success 'log --grep does not find non-reencoded values (latin1)' '\n+test_expect_success !MINGW 'log --grep does not find non-reencoded values (latin1)' '\n \tgit log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e >actual &&\n \ttest_must_be_empty actual\n '\n@@ -70,7 +70,7 @@ do\n \tthen\n \t force_regex=.*\n \tfi\n-\ttest_expect_success GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n+\ttest_expect_success !MINGW,GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n \t\tcat >expect <<-\\EOF &&\n \t\tlatin1\n \t\tutf8\n@@ -79,12 +79,12 @@ do\n \t\ttest_cmp expect actual\n \t\"\n \n-\ttest_expect_success GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n+\ttest_expect_success !MINGW,GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not find non-reencoded values (latin1 + locale)\" \"\n \t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$utf8_e\\\" >actual &&\n \t\ttest_must_be_empty actual\n \t\"\n \n-\ttest_expect_success GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not die on invalid UTF-8 value (latin1 + locale + invalid needle)\" \"\n+\ttest_expect_success !MINGW,GETTEXT_LOCALE,$prereq \"-c grep.patternType=$engine log --grep does not die on invalid UTF-8 value (latin1 + locale + invalid needle)\" \"\n \t\tLC_ALL=\\\"$is_IS_locale\\\" git -c grep.patternType=$engine log --encoding=ISO-8859-1 --format=%s --grep=\\\"$force_regex$invalid_e\\\" >actual &&\n \t\ttest_must_be_empty actual\n \t\"\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378455","messageId":"20190701212100.27850-5-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 04/10] grep: inline the return value of a function call used only once","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:54Z","receivedAt":"2019-07-01T21:21:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Since e944d9d932 (\"grep: rewrite an if/else condition to avoid\nduplicate expression\", 2016-06-25) the \"ascii_only\" variable has only\nbeen used once in compile_regexp(), let's just inline it there.\n\nThis makes the code easier to read, and might make it marginally\nfaster depending on compiler optimizations.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 4 +---\n 1 file changed, 1 insertion(+), 3 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 1de4ab49c0..4e8d0645a8 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -650,13 +650,11 @@ static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n-\tint ascii_only;\n \tint err;\n \tint regflags = REG_NEWLINE;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n-\tascii_only = !has_non_ascii(p->pattern);\n \n \t/*\n \t * Even when -F (fixed) asks us to do a non-regexp search, we\n@@ -673,7 +671,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tif (opt->fixed ||\n \t has_null(p->pattern, p->patternlen) ||\n \t is_fixed(p->pattern, p->patternlen))\n-\t\tp->fixed = !p->ignore_case || ascii_only;\n+\t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n \n \tif (p->fixed) {\n \t\tp->kws = kwsalloc(p->ignore_case ? tolower_trans_tbl : NULL);\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378456","messageId":"20190701212100.27850-6-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 05/10] grep tests: move \"grep binary\" alongside the rest","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:55Z","receivedAt":"2019-07-01T21:21:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the \"grep binary\" test case added in aca20dd558 (\"grep: add test\nscript for binary file handling\", 2010-05-22) so that it lives\nalongside the rest of the \"grep\" tests in t781*. This would have left\na gap in the t/700* namespace, so move a \"filter-branch\" test down,\nleaving the \"t7010-setup.sh\" test as the next one after that.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ...ilter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} | 0\n t/{t7008-grep-binary.sh => t7815-grep-binary.sh} | 0\n 2 files changed, 0 insertions(+), 0 deletions(-)\n rename t/{t7009-filter-branch-null-sha1.sh => t7008-filter-branch-null-sha1.sh} (100%)\n rename t/{t7008-grep-binary.sh => t7815-grep-binary.sh} (100%)\n\ndiff --git a/t/t7009-filter-branch-null-sha1.sh b/t/t7008-filter-branch-null-sha1.sh\nsimilarity index 100%\nrename from t/t7009-filter-branch-null-sha1.sh\nrename to t/t7008-filter-branch-null-sha1.sh\ndiff --git a/t/t7008-grep-binary.sh b/t/t7815-grep-binary.sh\nsimilarity index 100%\nrename from t/t7008-grep-binary.sh\nrename to t/t7815-grep-binary.sh\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378457","messageId":"20190701212100.27850-9-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 08/10] grep: drop support for \\0 in --fixed-strings ","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:58Z","receivedAt":"2019-07-01T21:21:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change \"-f \" to not support patterns with a NUL-byte in them\nunder --fixed-strings. We'll now only support these under\n\"--perl-regexp\" with PCRE v2.\n\nA previous change to grep's documentation changed the description of\n\"-f \" to be vague enough as to not promise that this would work.\nBy dropping support for this we make it a whole lot easier to move\naway from the kwset backend, which we'll do in a subsequent change.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 6 +--\n t/t7816-grep-binary-pattern.sh | 82 +++++++++++++++++-----------------\n 2 files changed, 44 insertions(+), 44 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex d6603bc950..8d0fff316c 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -644,6 +644,9 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n \n+\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n+\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n+\n \t/*\n \t * Even when -F (fixed) asks us to do a non-regexp search, we\n \t * may not be able to correctly case-fold when -i\n@@ -666,9 +669,6 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\treturn;\n \t}\n \n-\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n-\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n-\n \tif (opt->fixed) {\n \t\t/*\n \t\t * We come here when the pattern has the non-ascii\ndiff --git a/t/t7816-grep-binary-pattern.sh b/t/t7816-grep-binary-pattern.sh\nindex 9e09bd5d6a..60bab291e4 100755\n--- a/t/t7816-grep-binary-pattern.sh\n+++ b/t/t7816-grep-binary-pattern.sh\n@@ -60,23 +60,23 @@ test_expect_success 'setup' \"\n \"\n \n # Simple fixed-string matching that can use kwset (no -i && non-ASCII)\n-nul_match 1 1 1 '-F' 'yQf'\n-nul_match 0 0 0 '-F' 'yQx'\n-nul_match 1 1 1 '-Fi' 'YQf'\n-nul_match 0 0 0 '-Fi' 'YQx'\n-nul_match 1 1 1 '' 'yQf'\n-nul_match 0 0 0 '' 'yQx'\n-nul_match 1 1 1 '' 'æQð'\n-nul_match 1 1 1 '-F' 'eQm[*]c'\n-nul_match 1 1 1 '-Fi' 'EQM[*]C'\n+nul_match P P P '-F' 'yQf'\n+nul_match P P P '-F' 'yQx'\n+nul_match P P P '-Fi' 'YQf'\n+nul_match P P P '-Fi' 'YQx'\n+nul_match P P 1 '' 'yQf'\n+nul_match P P 0 '' 'yQx'\n+nul_match P P 1 '' 'æQð'\n+nul_match P P P '-F' 'eQm[*]c'\n+nul_match P P P '-Fi' 'EQM[*]C'\n \n # Regex patterns that would match but shouldn't with -F\n-nul_match 0 0 0 '-F' 'yQ[f]'\n-nul_match 0 0 0 '-F' '[y]Qf'\n-nul_match 0 0 0 '-Fi' 'YQ[F]'\n-nul_match 0 0 0 '-Fi' '[Y]QF'\n-nul_match 0 0 0 '-F' 'æQ[ð]'\n-nul_match 0 0 0 '-F' '[æ]Qð'\n+nul_match P P P '-F' 'yQ[f]'\n+nul_match P P P '-F' '[y]Qf'\n+nul_match P P P '-Fi' 'YQ[F]'\n+nul_match P P P '-Fi' '[Y]QF'\n+nul_match P P P '-F' 'æQ[ð]'\n+nul_match P P P '-F' '[æ]Qð'\n \n # The -F kwset codepath can't handle -i && non-ASCII...\n nul_match P 1 1 '-i' '[æ]Qð'\n@@ -90,38 +90,38 @@ nul_match P 0 1 '-i' '[Æ]Qð'\n nul_match P 0 1 '-i' 'ÆQÐ'\n \n # \\0 in regexes can only work with -P & PCRE v2\n-nul_match P 1 1 '' 'yQ[f]'\n-nul_match P 1 1 '' '[y]Qf'\n-nul_match P 1 1 '-i' 'YQ[F]'\n-nul_match P 1 1 '-i' '[Y]Qf'\n-nul_match P 1 1 '' 'æQ[ð]'\n-nul_match P 1 1 '' '[æ]Qð'\n-nul_match P 0 1 '-i' 'ÆQ[Ð]'\n-nul_match P 1 1 '' 'eQm.*cQ'\n-nul_match P 1 1 '-i' 'EQM.*cQ'\n-nul_match P 0 0 '' 'eQm[*]c'\n-nul_match P 0 0 '-i' 'EQM[*]C'\n+nul_match P P 1 '' 'yQ[f]'\n+nul_match P P 1 '' '[y]Qf'\n+nul_match P P 1 '-i' 'YQ[F]'\n+nul_match P P 1 '-i' '[Y]Qf'\n+nul_match P P 1 '' 'æQ[ð]'\n+nul_match P P 1 '' '[æ]Qð'\n+nul_match P P 1 '-i' 'ÆQ[Ð]'\n+nul_match P P 1 '' 'eQm.*cQ'\n+nul_match P P 1 '-i' 'EQM.*cQ'\n+nul_match P P 0 '' 'eQm[*]c'\n+nul_match P P 0 '-i' 'EQM[*]C'\n \n # Assert that we're using REG_STARTEND and the pattern doesn't match\n # just because it's cut off at the first \\0.\n-nul_match 0 0 0 '-i' 'NOMATCHQð'\n-nul_match P 0 0 '-i' '[Æ]QNOMATCH'\n-nul_match P 0 0 '-i' '[æ]QNOMATCH'\n+nul_match P P 0 '-i' 'NOMATCHQð'\n+nul_match P P 0 '-i' '[Æ]QNOMATCH'\n+nul_match P P 0 '-i' '[æ]QNOMATCH'\n \n # Ensure that the matcher doesn't regress to something that stops at\n # \\0\n-nul_match 0 0 0 '-F' 'yQ[f]'\n-nul_match 0 0 0 '-Fi' 'YQ[F]'\n-nul_match 0 0 0 '' 'yQNOMATCH'\n-nul_match 0 0 0 '' 'QNOMATCH'\n-nul_match 0 0 0 '-i' 'YQNOMATCH'\n-nul_match 0 0 0 '-i' 'QNOMATCH'\n-nul_match 0 0 0 '-F' 'æQ[ð]'\n+nul_match P P P '-F' 'yQ[f]'\n+nul_match P P P '-Fi' 'YQ[F]'\n+nul_match P P 0 '' 'yQNOMATCH'\n+nul_match P P 0 '' 'QNOMATCH'\n+nul_match P P 0 '-i' 'YQNOMATCH'\n+nul_match P P 0 '-i' 'QNOMATCH'\n+nul_match P P P '-F' 'æQ[ð]'\n nul_match P P P '-Fi' 'ÆQ[Ð]'\n-nul_match P 0 1 '-i' 'ÆQ[Ð]'\n-nul_match 0 0 0 '' 'yQNÓMATCH'\n-nul_match 0 0 0 '' 'QNÓMATCH'\n-nul_match 0 0 0 '-i' 'YQNÓMATCH'\n-nul_match 0 0 0 '-i' 'QNÓMATCH'\n+nul_match P P 1 '-i' 'ÆQ[Ð]'\n+nul_match P P 0 '' 'yQNÓMATCH'\n+nul_match P P 0 '' 'QNÓMATCH'\n+nul_match P P 0 '-i' 'YQNÓMATCH'\n+nul_match P P 0 '-i' 'QNÓMATCH'\n \n test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378458","messageId":"20190701212100.27850-8-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 07/10] grep: make the behavior for NUL-byte in patterns sane","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:57Z","receivedAt":"2019-07-01T21:21:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The behavior of \"grep\" when patterns contained a NUL-byte has always\nbeen haphazard, and has served the vagaries of the implementation more\nthan anything else. A pattern containing a NUL-byte can only be\nprovided via \"-f \". Since pickaxe (log search) has no such flag\nthe NUL-byte in patterns has only ever been supported by \"grep\" (and\nnot \"log --grep\").\n\nSince 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) patterns containing\n\"\\0\" were considered fixed. In 966be95549 (\"grep: add tests to fix\nblind spots with \\0 patterns\", 2017-05-20) I added tests for this\nbehavior.\n\nChange the behavior to do the obvious thing, i.e. don't silently\ndiscard a regex pattern and make it implicitly fixed just because they\ncontain a NUL-byte. Instead die if the backend in question can't\nhandle them, e.g. --basic-regexp is combined with such a pattern.\n\nThis is desired because from a user's point of view it's the obvious\nthing to do. Whether we support BRE/ERE/Perl syntax is different from\nwhether our implementation is limited by C-strings. These patterns are\nobscure enough that I think this behavior change is OK, especially\nsince we never documented the old behavior.\n\nDoing this also makes it easier to replace the kwset backend with\nsomething else, since we'll no longer strictly need it for anything we\ncan't easily use another fixed-string backend for.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/git-grep.txt | 17 ++++\n grep.c | 23 ++---\n t/t7816-grep-binary-pattern.sh | 159 ++++++++++++++++++---------------\n 3 files changed, 110 insertions(+), 89 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 2d27969057..c89fb569e3 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -271,6 +271,23 @@ providing this option will cause it to die.\n \n -f ::\n \tRead patterns from , one per line.\n++\n+Passing the pattern via allows for providing a search pattern\n+containing a \\0.\n++\n+Not all pattern types support patterns containing \\0. Git will error\n+out if a given pattern type can't support such a pattern. The\n+`--perl-regexp` pattern type when compiled against the PCRE v2 backend\n+has the widest support for these types of patterns.\n++\n+In versions of Git before 2.23.0 patterns containing \\0 would be\n+silently considered fixed. This was never documented, there were also\n+odd and undocumented interactions between e.g. non-ASCII patterns\n+containing \\0 and `--ignore-case`.\n++\n+In future versions we may learn to support patterns containing \\0 for\n+more search backends, until then we'll die when the pattern type in\n+question doesn't support them.\n \n -e::\n \tThe next parameter is the pattern. This option has to be\ndiff --git a/grep.c b/grep.c\nindex 4e8d0645a8..d6603bc950 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -368,18 +368,6 @@ static int is_fixed(const char *s, size_t len)\n \treturn 1;\n }\n \n-static int has_null(const char *s, size_t len)\n-{\n-\t/*\n-\t * regcomp cannot accept patterns with NULs so when using it\n-\t * we consider any pattern containing a NUL fixed.\n-\t */\n-\tif (memchr(s, 0, len))\n-\t\treturn 1;\n-\n-\treturn 0;\n-}\n-\n #ifdef USE_LIBPCRE1\n static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n {\n@@ -668,9 +656,7 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t * simple string match using kws. p->fixed tells us if we\n \t * want to use kws.\n \t */\n-\tif (opt->fixed ||\n-\t has_null(p->pattern, p->patternlen) ||\n-\t is_fixed(p->pattern, p->patternlen))\n+\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n \t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n \n \tif (p->fixed) {\n@@ -678,7 +664,12 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tkwsincr(p->kws, p->pattern, p->patternlen);\n \t\tkwsprep(p->kws);\n \t\treturn;\n-\t} else if (opt->fixed) {\n+\t}\n+\n+\tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n+\t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n+\n+\tif (opt->fixed) {\n \t\t/*\n \t\t * We come here when the pattern has the non-ascii\n \t\t * characters we cannot case-fold, and asked to\ndiff --git a/t/t7816-grep-binary-pattern.sh b/t/t7816-grep-binary-pattern.sh\nindex 4060dbd679..9e09bd5d6a 100755\n--- a/t/t7816-grep-binary-pattern.sh\n+++ b/t/t7816-grep-binary-pattern.sh\n@@ -2,113 +2,126 @@\n \n test_description='git grep with a binary pattern files'\n \n-. ./test-lib.sh\n+. ./lib-gettext.sh\n \n-nul_match () {\n+nul_match_internal () {\n \tmatches=$1\n-\tflags=$2\n-\tpattern=$3\n+\tprereqs=$2\n+\tlc_all=$3\n+\textra_flags=$4\n+\tflags=$5\n+\tpattern=$6\n \tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n \n \tif test \"$matches\" = 1\n \tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\ttest_expect_success $prereqs \"LC_ALL='$lc_all' git grep $extra_flags -f f $flags '$pattern_human' a\" \"\n \t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n+\t\t\tLC_ALL='$lc_all' git grep $extra_flags -f f $flags a\n \t\t\"\n \telif test \"$matches\" = 0\n \tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\ttest_expect_success $prereqs \"LC_ALL='$lc_all' git grep $extra_flags -f f $flags '$pattern_human' a\" \"\n+\t\t\t>stderr &&\n \t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\ttest_must_fail env LC_ALL=\\\"$lc_all\\\" git grep $extra_flags -f f $flags a 2>stderr &&\n+\t\t\ttest_i18ngrep ! 'This is only supported with -P under PCRE v2' stderr\n \t\t\"\n-\telif test \"$matches\" = T1\n+\telif test \"$matches\" = P\n \tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\ttest_expect_success $prereqs \"error, PCRE v2 only: LC_ALL='$lc_all' git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\t>stderr &&\n \t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = T0\n-\tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\ttest_must_fail env LC_ALL=\\\"$lc_all\\\" git grep -f f $flags a 2>stderr &&\n+\t\t\ttest_i18ngrep 'This is only supported with -P under PCRE v2' stderr\n \t\t\"\n \telse\n \t\ttest_expect_success \"PANIC: Test framework error. Unknown matches value $matches\" 'false'\n \tfi\n }\n \n+nul_match () {\n+\tmatches=$1\n+\tmatches_pcre2=$2\n+\tmatches_pcre2_locale=$3\n+\tflags=$4\n+\tpattern=$5\n+\tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n+\n+\tnul_match_internal \"$matches\" \"\" \"C\" \"\" \"$flags\" \"$pattern\"\n+\tnul_match_internal \"$matches_pcre2\" \"LIBPCRE2\" \"C\" \"-P\" \"$flags\" \"$pattern\"\n+\tnul_match_internal \"$matches_pcre2_locale\" \"LIBPCRE2,GETTEXT_LOCALE\" \"$is_IS_locale\" \"-P\" \"$flags\" \"$pattern\"\n+}\n+\n test_expect_success 'setup' \"\n \techo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&\n \tgit add a &&\n \tgit commit -m.\n \"\n \n-nul_match 1 '-F' 'yQf'\n-nul_match 0 '-F' 'yQx'\n-nul_match 1 '-Fi' 'YQf'\n-nul_match 0 '-Fi' 'YQx'\n-nul_match 1 '' 'yQf'\n-nul_match 0 '' 'yQx'\n-nul_match 1 '' 'æQð'\n-nul_match 1 '-F' 'eQm[*]c'\n-nul_match 1 '-Fi' 'EQM[*]C'\n+# Simple fixed-string matching that can use kwset (no -i && non-ASCII)\n+nul_match 1 1 1 '-F' 'yQf'\n+nul_match 0 0 0 '-F' 'yQx'\n+nul_match 1 1 1 '-Fi' 'YQf'\n+nul_match 0 0 0 '-Fi' 'YQx'\n+nul_match 1 1 1 '' 'yQf'\n+nul_match 0 0 0 '' 'yQx'\n+nul_match 1 1 1 '' 'æQð'\n+nul_match 1 1 1 '-F' 'eQm[*]c'\n+nul_match 1 1 1 '-Fi' 'EQM[*]C'\n \n # Regex patterns that would match but shouldn't with -F\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-F' '[y]Qf'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '-Fi' '[Y]QF'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-F' '[æ]Qð'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '-Fi' '[Æ]QÐ'\n+nul_match 0 0 0 '-F' 'yQ[f]'\n+nul_match 0 0 0 '-F' '[y]Qf'\n+nul_match 0 0 0 '-Fi' 'YQ[F]'\n+nul_match 0 0 0 '-Fi' '[Y]QF'\n+nul_match 0 0 0 '-F' 'æQ[ð]'\n+nul_match 0 0 0 '-F' '[æ]Qð'\n \n-# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n-# patterns case-insensitively.\n-nul_match T1 '-i' 'ÆQÐ'\n+# The -F kwset codepath can't handle -i && non-ASCII...\n+nul_match P 1 1 '-i' '[æ]Qð'\n \n-# \\0 implicitly disables regexes. This is an undocumented internal\n-# limitation.\n-nul_match T1 '' 'yQ[f]'\n-nul_match T1 '' '[y]Qf'\n-nul_match T1 '-i' 'YQ[F]'\n-nul_match T1 '-i' '[Y]Qf'\n-nul_match T1 '' 'æQ[ð]'\n-nul_match T1 '' '[æ]Qð'\n-nul_match T1 '-i' 'ÆQ[Ð]'\n+# ...PCRE v2 only matches non-ASCII with -i casefolding under UTF-8\n+# semantics\n+nul_match P P P '-Fi' 'ÆQ[Ð]'\n+nul_match P 0 1 '-i' 'ÆQ[Ð]'\n+nul_match P 0 1 '-i' '[Æ]QÐ'\n+nul_match P 0 1 '-i' '[Æ]Qð'\n+nul_match P 0 1 '-i' 'ÆQÐ'\n \n-# ... because of \\0 implicitly disabling regexes regexes that\n-# should/shouldn't match don't do the right thing.\n-nul_match T1 '' 'eQm.*cQ'\n-nul_match T1 '-i' 'EQM.*cQ'\n-nul_match T0 '' 'eQm[*]c'\n-nul_match T0 '-i' 'EQM[*]C'\n+# \\0 in regexes can only work with -P & PCRE v2\n+nul_match P 1 1 '' 'yQ[f]'\n+nul_match P 1 1 '' '[y]Qf'\n+nul_match P 1 1 '-i' 'YQ[F]'\n+nul_match P 1 1 '-i' '[Y]Qf'\n+nul_match P 1 1 '' 'æQ[ð]'\n+nul_match P 1 1 '' '[æ]Qð'\n+nul_match P 0 1 '-i' 'ÆQ[Ð]'\n+nul_match P 1 1 '' 'eQm.*cQ'\n+nul_match P 1 1 '-i' 'EQM.*cQ'\n+nul_match P 0 0 '' 'eQm[*]c'\n+nul_match P 0 0 '-i' 'EQM[*]C'\n \n-# Due to the REG_STARTEND extension when kwset() is disabled on -i &\n-# non-ASCII the string will be matched in its entirety, but the\n-# pattern will be cut off at the first \\0.\n-nul_match 0 '-i' 'NOMATCHQð'\n-nul_match T0 '-i' '[Æ]QNOMATCH'\n-nul_match T0 '-i' '[æ]QNOMATCH'\n-# Matches, but for the wrong reasons, just stops at [æ]\n-nul_match 1 '-i' '[Æ]Qð'\n-nul_match 1 '-i' '[æ]Qð'\n+# Assert that we're using REG_STARTEND and the pattern doesn't match\n+# just because it's cut off at the first \\0.\n+nul_match 0 0 0 '-i' 'NOMATCHQð'\n+nul_match P 0 0 '-i' '[Æ]QNOMATCH'\n+nul_match P 0 0 '-i' '[æ]QNOMATCH'\n \n # Ensure that the matcher doesn't regress to something that stops at\n # \\0\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '' 'yQNOMATCH'\n-nul_match 0 '' 'QNOMATCH'\n-nul_match 0 '-i' 'YQNOMATCH'\n-nul_match 0 '-i' 'QNOMATCH'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '' 'yQNÓMATCH'\n-nul_match 0 '' 'QNÓMATCH'\n-nul_match 0 '-i' 'YQNÓMATCH'\n-nul_match 0 '-i' 'QNÓMATCH'\n+nul_match 0 0 0 '-F' 'yQ[f]'\n+nul_match 0 0 0 '-Fi' 'YQ[F]'\n+nul_match 0 0 0 '' 'yQNOMATCH'\n+nul_match 0 0 0 '' 'QNOMATCH'\n+nul_match 0 0 0 '-i' 'YQNOMATCH'\n+nul_match 0 0 0 '-i' 'QNOMATCH'\n+nul_match 0 0 0 '-F' 'æQ[ð]'\n+nul_match P P P '-Fi' 'ÆQ[Ð]'\n+nul_match P 0 1 '-i' 'ÆQ[Ð]'\n+nul_match 0 0 0 '' 'yQNÓMATCH'\n+nul_match 0 0 0 '' 'QNÓMATCH'\n+nul_match 0 0 0 '-i' 'YQNÓMATCH'\n+nul_match 0 0 0 '-i' 'QNÓMATCH'\n \n test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378459","messageId":"20190701212100.27850-7-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 06/10] grep tests: move binary pattern tests into their own file","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:56Z","receivedAt":"2019-07-01T21:21:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the tests for \"-f \" where \"\" contains a NUL byte\npattern into their own file. I added most of these tests in\n966be95549 (\"grep: add tests to fix blind spots with \\0 patterns\",\n2017-05-20).\n\nWhether a regex engine supports matching binary content is very\ndifferent from whether it matches binary patterns. Since\n2f8952250a (\"regex: add regexec_buf() that can work on a non\nNUL-terminated string\", 2016-09-21) we've required REG_STARTEND of our\nregex engines so we can match binary content, but only the PCRE v2\nengine can sensibly match binary patterns.\n\nSince 9eceddeec6 (\"Use kwset in grep\", 2011-08-21) we've been punting\npatterns containing NUL-byte and considering them fixed, except in\ncases where \"--ignore-case\" is provided and they're non-ASCII, see\n5c1ebcca4d (\"grep/icase: avoid kwsset on literal non-ascii strings\",\n2016-06-25). Subsequent commits will change this behavior.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t7815-grep-binary.sh | 101 -----------------------------\n t/t7816-grep-binary-pattern.sh | 114 +++++++++++++++++++++++++++++++++\n 2 files changed, 114 insertions(+), 101 deletions(-)\n create mode 100755 t/t7816-grep-binary-pattern.sh\n\ndiff --git a/t/t7815-grep-binary.sh b/t/t7815-grep-binary.sh\nindex 2d87c49b75..90ebb64f46 100755\n--- a/t/t7815-grep-binary.sh\n+++ b/t/t7815-grep-binary.sh\n@@ -4,41 +4,6 @@ test_description='git grep in binary files'\n \n . ./test-lib.sh\n \n-nul_match () {\n-\tmatches=$1\n-\tflags=$2\n-\tpattern=$3\n-\tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n-\n-\tif test \"$matches\" = 1\n-\tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = 0\n-\tthen\n-\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = T1\n-\tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\tgit grep -f f $flags a\n-\t\t\"\n-\telif test \"$matches\" = T0\n-\tthen\n-\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n-\t\t\tprintf '$pattern' | q_to_nul >f &&\n-\t\t\ttest_must_fail git grep -f f $flags a\n-\t\t\"\n-\telse\n-\t\ttest_expect_success \"PANIC: Test framework error. Unknown matches value $matches\" 'false'\n-\tfi\n-}\n-\n test_expect_success 'setup' \"\n \techo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&\n \tgit add a &&\n@@ -102,72 +67,6 @@ test_expect_failure 'git grep .fi a' '\n \tgit grep .fi a\n '\n \n-nul_match 1 '-F' 'yQf'\n-nul_match 0 '-F' 'yQx'\n-nul_match 1 '-Fi' 'YQf'\n-nul_match 0 '-Fi' 'YQx'\n-nul_match 1 '' 'yQf'\n-nul_match 0 '' 'yQx'\n-nul_match 1 '' 'æQð'\n-nul_match 1 '-F' 'eQm[*]c'\n-nul_match 1 '-Fi' 'EQM[*]C'\n-\n-# Regex patterns that would match but shouldn't with -F\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-F' '[y]Qf'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '-Fi' '[Y]QF'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-F' '[æ]Qð'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '-Fi' '[Æ]QÐ'\n-\n-# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n-# patterns case-insensitively.\n-nul_match T1 '-i' 'ÆQÐ'\n-\n-# \\0 implicitly disables regexes. This is an undocumented internal\n-# limitation.\n-nul_match T1 '' 'yQ[f]'\n-nul_match T1 '' '[y]Qf'\n-nul_match T1 '-i' 'YQ[F]'\n-nul_match T1 '-i' '[Y]Qf'\n-nul_match T1 '' 'æQ[ð]'\n-nul_match T1 '' '[æ]Qð'\n-nul_match T1 '-i' 'ÆQ[Ð]'\n-\n-# ... because of \\0 implicitly disabling regexes regexes that\n-# should/shouldn't match don't do the right thing.\n-nul_match T1 '' 'eQm.*cQ'\n-nul_match T1 '-i' 'EQM.*cQ'\n-nul_match T0 '' 'eQm[*]c'\n-nul_match T0 '-i' 'EQM[*]C'\n-\n-# Due to the REG_STARTEND extension when kwset() is disabled on -i &\n-# non-ASCII the string will be matched in its entirety, but the\n-# pattern will be cut off at the first \\0.\n-nul_match 0 '-i' 'NOMATCHQð'\n-nul_match T0 '-i' '[Æ]QNOMATCH'\n-nul_match T0 '-i' '[æ]QNOMATCH'\n-# Matches, but for the wrong reasons, just stops at [æ]\n-nul_match 1 '-i' '[Æ]Qð'\n-nul_match 1 '-i' '[æ]Qð'\n-\n-# Ensure that the matcher doesn't regress to something that stops at\n-# \\0\n-nul_match 0 '-F' 'yQ[f]'\n-nul_match 0 '-Fi' 'YQ[F]'\n-nul_match 0 '' 'yQNOMATCH'\n-nul_match 0 '' 'QNOMATCH'\n-nul_match 0 '-i' 'YQNOMATCH'\n-nul_match 0 '-i' 'QNOMATCH'\n-nul_match 0 '-F' 'æQ[ð]'\n-nul_match 0 '-Fi' 'ÆQ[Ð]'\n-nul_match 0 '' 'yQNÓMATCH'\n-nul_match 0 '' 'QNÓMATCH'\n-nul_match 0 '-i' 'YQNÓMATCH'\n-nul_match 0 '-i' 'QNÓMATCH'\n-\n test_expect_success 'grep respects binary diff attribute' '\n \techo text >t &&\n \tgit add t &&\ndiff --git a/t/t7816-grep-binary-pattern.sh b/t/t7816-grep-binary-pattern.sh\nnew file mode 100755\nindex 0000000000..4060dbd679\n--- /dev/null\n+++ b/t/t7816-grep-binary-pattern.sh\n@@ -0,0 +1,114 @@\n+#!/bin/sh\n+\n+test_description='git grep with a binary pattern files'\n+\n+. ./test-lib.sh\n+\n+nul_match () {\n+\tmatches=$1\n+\tflags=$2\n+\tpattern=$3\n+\tpattern_human=$(echo \"$pattern\" | sed 's/Q//g')\n+\n+\tif test \"$matches\" = 1\n+\tthen\n+\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\tgit grep -f f $flags a\n+\t\t\"\n+\telif test \"$matches\" = 0\n+\tthen\n+\t\ttest_expect_success \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\"\n+\telif test \"$matches\" = T1\n+\tthen\n+\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\tgit grep -f f $flags a\n+\t\t\"\n+\telif test \"$matches\" = T0\n+\tthen\n+\t\ttest_expect_failure \"git grep -f f $flags '$pattern_human' a\" \"\n+\t\t\tprintf '$pattern' | q_to_nul >f &&\n+\t\t\ttest_must_fail git grep -f f $flags a\n+\t\t\"\n+\telse\n+\t\ttest_expect_success \"PANIC: Test framework error. Unknown matches value $matches\" 'false'\n+\tfi\n+}\n+\n+test_expect_success 'setup' \"\n+\techo 'binaryQfileQm[*]cQ*æQð' | q_to_nul >a &&\n+\tgit add a &&\n+\tgit commit -m.\n+\"\n+\n+nul_match 1 '-F' 'yQf'\n+nul_match 0 '-F' 'yQx'\n+nul_match 1 '-Fi' 'YQf'\n+nul_match 0 '-Fi' 'YQx'\n+nul_match 1 '' 'yQf'\n+nul_match 0 '' 'yQx'\n+nul_match 1 '' 'æQð'\n+nul_match 1 '-F' 'eQm[*]c'\n+nul_match 1 '-Fi' 'EQM[*]C'\n+\n+# Regex patterns that would match but shouldn't with -F\n+nul_match 0 '-F' 'yQ[f]'\n+nul_match 0 '-F' '[y]Qf'\n+nul_match 0 '-Fi' 'YQ[F]'\n+nul_match 0 '-Fi' '[Y]QF'\n+nul_match 0 '-F' 'æQ[ð]'\n+nul_match 0 '-F' '[æ]Qð'\n+nul_match 0 '-Fi' 'ÆQ[Ð]'\n+nul_match 0 '-Fi' '[Æ]QÐ'\n+\n+# kwset is disabled on -i & non-ASCII. No way to match non-ASCII \\0\n+# patterns case-insensitively.\n+nul_match T1 '-i' 'ÆQÐ'\n+\n+# \\0 implicitly disables regexes. This is an undocumented internal\n+# limitation.\n+nul_match T1 '' 'yQ[f]'\n+nul_match T1 '' '[y]Qf'\n+nul_match T1 '-i' 'YQ[F]'\n+nul_match T1 '-i' '[Y]Qf'\n+nul_match T1 '' 'æQ[ð]'\n+nul_match T1 '' '[æ]Qð'\n+nul_match T1 '-i' 'ÆQ[Ð]'\n+\n+# ... because of \\0 implicitly disabling regexes regexes that\n+# should/shouldn't match don't do the right thing.\n+nul_match T1 '' 'eQm.*cQ'\n+nul_match T1 '-i' 'EQM.*cQ'\n+nul_match T0 '' 'eQm[*]c'\n+nul_match T0 '-i' 'EQM[*]C'\n+\n+# Due to the REG_STARTEND extension when kwset() is disabled on -i &\n+# non-ASCII the string will be matched in its entirety, but the\n+# pattern will be cut off at the first \\0.\n+nul_match 0 '-i' 'NOMATCHQð'\n+nul_match T0 '-i' '[Æ]QNOMATCH'\n+nul_match T0 '-i' '[æ]QNOMATCH'\n+# Matches, but for the wrong reasons, just stops at [æ]\n+nul_match 1 '-i' '[Æ]Qð'\n+nul_match 1 '-i' '[æ]Qð'\n+\n+# Ensure that the matcher doesn't regress to something that stops at\n+# \\0\n+nul_match 0 '-F' 'yQ[f]'\n+nul_match 0 '-Fi' 'YQ[F]'\n+nul_match 0 '' 'yQNOMATCH'\n+nul_match 0 '' 'QNOMATCH'\n+nul_match 0 '-i' 'YQNOMATCH'\n+nul_match 0 '-i' 'QNOMATCH'\n+nul_match 0 '-F' 'æQ[ð]'\n+nul_match 0 '-Fi' 'ÆQ[Ð]'\n+nul_match 0 '' 'yQNÓMATCH'\n+nul_match 0 '' 'QNÓMATCH'\n+nul_match 0 '-i' 'YQNÓMATCH'\n+nul_match 0 '-i' 'QNÓMATCH'\n+\n+test_done\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378460","messageId":"20190701212100.27850-10-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 09/10] grep: remove the kwset optimization","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:20:59Z","receivedAt":"2019-07-01T21:21:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A later change will replace this optimization with optimistic use of\nPCRE v2. I'm completely removing it as an intermediate step, as\nopposed to replacing it with PCRE v2, to demonstrate that no grep\nsemantics depend on this (or any other) optimization for the fixed\nbackend anymore.\n\nFor now this is mostly (but not entirely) a performance regression, as\nshown by this hacky one-liner:\n\n for opt in '' ' -i'\n do\n GIT_PERF_7821_GREP_OPTS=$opt GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE=YesPlease' ./run origin/master HEAD -- p7821-grep-engines-fixed.sh\n done &&\n for opt in '' ' -i'\n do GIT_PERF_4221_LOG_OPTS=$opt GIT_PERF_REPEAT_COUNT=10 GIT_PERF_LARGE_REPO=~/g/linux GIT_PERF_MAKE_OPTS='-j8 CFLAGS=-O3 USE_LIBPCRE=YesPlease' ./run origin/master HEAD -- p4221-log-grep-engines-fixed.sh\n done\n\nWhich produces:\n\nplain grep:\n\n Test origin/master HEAD\n -------------------------------------------------------------------------\n 7821.1: fixed grep int 0.55(1.60+0.63) 0.82(3.11+0.51) +49.1%\n 7821.2: basic grep int 0.62(1.68+0.49) 0.85(3.02+0.52) +37.1%\n 7821.3: extended grep int 0.61(1.63+0.53) 0.91(3.09+0.44) +49.2%\n 7821.4: perl grep int 0.55(1.60+0.57) 0.41(0.93+0.57) -25.5%\n 7821.6: fixed grep uncommon 0.20(0.50+0.44) 0.35(1.27+0.42) +75.0%\n 7821.7: basic grep uncommon 0.20(0.49+0.45) 0.35(1.29+0.41) +75.0%\n 7821.8: extended grep uncommon 0.20(0.45+0.48) 0.35(1.25+0.44) +75.0%\n 7821.9: perl grep uncommon 0.20(0.53+0.41) 0.16(0.24+0.49) -20.0%\n 7821.11: fixed grep æ 0.35(1.27+0.40) 0.25(0.82+0.39) -28.6%\n 7821.12: basic grep æ 0.35(1.28+0.38) 0.25(0.75+0.44) -28.6%\n 7821.13: extended grep æ 0.36(1.21+0.46) 0.25(0.86+0.35) -30.6%\n 7821.14: perl grep æ 0.35(1.33+0.34) 0.16(0.26+0.47) -54.3%\n\ngrep with -i:\n\n Test origin/master HEAD\n -----------------------------------------------------------------------------\n 7821.1: fixed grep -i int 0.61(1.84+0.64) 1.11(4.12+0.64) +82.0%\n 7821.2: basic grep -i int 0.72(1.86+0.57) 1.15(4.48+0.49) +59.7%\n 7821.3: extended grep -i int 0.94(1.83+0.60) 1.53(4.12+0.58) +62.8%\n 7821.4: perl grep -i int 0.66(1.82+0.59) 0.55(1.08+0.58) -16.7%\n 7821.6: fixed grep -i uncommon 0.21(0.51+0.44) 0.44(1.74+0.34) +109.5%\n 7821.7: basic grep -i uncommon 0.21(0.55+0.41) 0.44(1.72+0.40) +109.5%\n 7821.8: extended grep -i uncommon 0.21(0.57+0.39) 0.42(1.64+0.45) +100.0%\n 7821.9: perl grep -i uncommon 0.21(0.48+0.48) 0.17(0.30+0.45) -19.0%\n 7821.11: fixed grep -i æ 0.25(0.73+0.45) 0.25(0.75+0.45) +0.0%\n 7821.12: basic grep -i æ 0.25(0.71+0.49) 0.26(0.77+0.44) +4.0%\n 7821.13: extended grep -i æ 0.25(0.75+0.44) 0.25(0.74+0.46) +0.0%\n 7821.14: perl grep -i æ 0.17(0.26+0.48) 0.16(0.20+0.52) -5.9%\n\nplain log:\n\n Test origin/master HEAD\n ---------------------------------------------------------------------------------\n 4221.1: fixed log --grep='int' 7.31(7.06+0.21) 8.11(7.85+0.20) +10.9%\n 4221.2: basic log --grep='int' 7.30(6.94+0.27) 8.16(7.89+0.19) +11.8%\n 4221.3: extended log --grep='int' 7.34(7.05+0.21) 8.08(7.76+0.25) +10.1%\n 4221.4: perl log --grep='int' 7.27(6.94+0.24) 7.05(6.76+0.25) -3.0%\n 4221.6: fixed log --grep='uncommon' 6.97(6.62+0.32) 7.86(7.51+0.30) +12.8%\n 4221.7: basic log --grep='uncommon' 7.05(6.69+0.29) 7.89(7.60+0.28) +11.9%\n 4221.8: extended log --grep='uncommon' 6.89(6.56+0.32) 7.99(7.66+0.24) +16.0%\n 4221.9: perl log --grep='uncommon' 7.02(6.66+0.33) 6.97(6.54+0.36) -0.7%\n 4221.11: fixed log --grep='æ' 7.37(7.03+0.33) 7.67(7.30+0.31) +4.1%\n 4221.12: basic log --grep='æ' 7.41(7.00+0.31) 7.60(7.28+0.26) +2.6%\n 4221.13: extended log --grep='æ' 7.35(6.96+0.38) 7.73(7.31+0.34) +5.2%\n 4221.14: perl log --grep='æ' 7.43(7.10+0.32) 6.95(6.61+0.27) -6.5%\n\nlog with -i:\n\n Test origin/master HEAD\n ------------------------------------------------------------------------------------\n 4221.1: fixed log -i --grep='int' 7.40(7.05+0.23) 8.66(8.38+0.20) +17.0%\n 4221.2: basic log -i --grep='int' 7.39(7.09+0.23) 8.67(8.39+0.20) +17.3%\n 4221.3: extended log -i --grep='int' 7.29(6.99+0.26) 8.69(8.31+0.26) +19.2%\n 4221.4: perl log -i --grep='int' 7.42(7.16+0.21) 7.14(6.80+0.24) -3.8%\n 4221.6: fixed log -i --grep='uncommon' 6.94(6.58+0.35) 8.43(8.04+0.30) +21.5%\n 4221.7: basic log -i --grep='uncommon' 6.95(6.62+0.31) 8.34(7.93+0.32) +20.0%\n 4221.8: extended log -i --grep='uncommon' 7.06(6.75+0.25) 8.32(7.98+0.31) +17.8%\n 4221.9: perl log -i --grep='uncommon' 6.96(6.69+0.26) 7.04(6.64+0.32) +1.1%\n 4221.11: fixed log -i --grep='æ' 7.92(7.55+0.33) 7.86(7.44+0.34) -0.8%\n 4221.12: basic log -i --grep='æ' 7.88(7.49+0.32) 7.84(7.46+0.34) -0.5%\n 4221.13: extended log -i --grep='æ' 7.91(7.51+0.32) 7.87(7.48+0.32) -0.5%\n 4221.14: perl log -i --grep='æ' 7.01(6.59+0.35) 6.99(6.64+0.28) -0.3%\n\nSome of those, as noted in [1] are because PCRE is faster at finding\nfixed strings. This looks bad for some engines, but in the next change\nwe'll optimistically use PCRE v2 for all of these, so it'll look\nbetter.\n\n1. https://public-inbox.org/git/87v9x793qi.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 63 +++-------------------------------------------------------\n grep.h | 2 --\n 2 files changed, 3 insertions(+), 62 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 8d0fff316c..4468519d5c 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -356,18 +356,6 @@ static NORETURN void compile_regexp_failed(const struct grep_pat *p,\n \tdie(\"%s'%s': %s\", where, p->pattern, error);\n }\n \n-static int is_fixed(const char *s, size_t len)\n-{\n-\tsize_t i;\n-\n-\tfor (i = 0; i < len; i++) {\n-\t\tif (is_regex_special(s[i]))\n-\t\t\treturn 0;\n-\t}\n-\n-\treturn 1;\n-}\n-\n #ifdef USE_LIBPCRE1\n static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n {\n@@ -643,38 +631,12 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n+\tp->fixed = opt->fixed;\n \n \tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n \t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n \n-\t/*\n-\t * Even when -F (fixed) asks us to do a non-regexp search, we\n-\t * may not be able to correctly case-fold when -i\n-\t * (ignore-case) is asked (in which case, we'll synthesize a\n-\t * regexp to match the pattern that matches regexp special\n-\t * characters literally, while ignoring case differences). On\n-\t * the other hand, even without -F, if the pattern does not\n-\t * have any regexp special characters and there is no need for\n-\t * case-folding search, we can internally turn it into a\n-\t * simple string match using kws. p->fixed tells us if we\n-\t * want to use kws.\n-\t */\n-\tif (opt->fixed || is_fixed(p->pattern, p->patternlen))\n-\t\tp->fixed = !p->ignore_case || !has_non_ascii(p->pattern);\n-\n-\tif (p->fixed) {\n-\t\tp->kws = kwsalloc(p->ignore_case ? tolower_trans_tbl : NULL);\n-\t\tkwsincr(p->kws, p->pattern, p->patternlen);\n-\t\tkwsprep(p->kws);\n-\t\treturn;\n-\t}\n-\n \tif (opt->fixed) {\n-\t\t/*\n-\t\t * We come here when the pattern has the non-ascii\n-\t\t * characters we cannot case-fold, and asked to\n-\t\t * ignore-case.\n-\t\t */\n \t\tcompile_fixed_regexp(p, opt);\n \t\treturn;\n \t}\n@@ -1042,9 +1004,7 @@ void free_grep_patterns(struct grep_opt *opt)\n \t\tcase GREP_PATTERN: /* atom */\n \t\tcase GREP_PATTERN_HEAD:\n \t\tcase GREP_PATTERN_BODY:\n-\t\t\tif (p->kws)\n-\t\t\t\tkwsfree(p->kws);\n-\t\t\telse if (p->pcre1_regexp)\n+\t\t\tif (p->pcre1_regexp)\n \t\t\t\tfree_pcre1_regexp(p);\n \t\t\telse if (p->pcre2_pattern)\n \t\t\t\tfree_pcre2_pattern(p);\n@@ -1104,29 +1064,12 @@ static void show_name(struct grep_opt *opt, const char *name)\n \topt->output(opt, opt->null_following_name ? \"\\0\" : \"\\n\", 1);\n }\n \n-static int fixmatch(struct grep_pat *p, char *line, char *eol,\n-\t\t regmatch_t *match)\n-{\n-\tstruct kwsmatch kwsm;\n-\tsize_t offset = kwsexec(p->kws, line, eol - line, &kwsm);\n-\tif (offset == -1) {\n-\t\tmatch->rm_so = match->rm_eo = -1;\n-\t\treturn REG_NOMATCH;\n-\t} else {\n-\t\tmatch->rm_so = offset;\n-\t\tmatch->rm_eo = match->rm_so + kwsm.size[0];\n-\t\treturn 0;\n-\t}\n-}\n-\n static int patmatch(struct grep_pat *p, char *line, char *eol,\n \t\t regmatch_t *match, int eflags)\n {\n \tint hit;\n \n-\tif (p->fixed)\n-\t\thit = !fixmatch(p, line, eol, match);\n-\telse if (p->pcre1_regexp)\n+\tif (p->pcre1_regexp)\n \t\thit = !pcre1match(p, line, eol, match, eflags);\n \telse if (p->pcre2_pattern)\n \t\thit = !pcre2match(p, line, eol, match, eflags);\ndiff --git a/grep.h b/grep.h\nindex 4bb8a79d93..d35a137fcb 100644\n--- a/grep.h\n+++ b/grep.h\n@@ -32,7 +32,6 @@ typedef int pcre2_compile_context;\n typedef int pcre2_match_context;\n typedef int pcre2_jit_stack;\n #endif\n-#include \"kwset.h\"\n #include \"thread-utils.h\"\n #include \"userdiff.h\"\n \n@@ -97,7 +96,6 @@ struct grep_pat {\n \tpcre2_match_context *pcre2_match_context;\n \tpcre2_jit_stack *pcre2_jit_stack;\n \tuint32_t pcre2_jit_on;\n-\tkwset_t kws;\n \tunsigned fixed:1;\n \tunsigned ignore_case:1;\n \tunsigned word_regexp:1;\n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378461","messageId":"20190701212100.27850-11-avarab@gmail.com","threadId":"51300","inReplyTo":"20190627233912.7117-1-avarab@gmail.com","subject":"[PATCH v3 10/10] grep: use PCRE v2 for optimized fixed-string search","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-01T21:21:00Z","receivedAt":"2019-07-01T21:21:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Bring back optimized fixed-string search for \"grep\", this time with\nPCRE v2 as an optional backend. As noted in [1] with kwset we were\nslower than PCRE v1 and v2 JIT with the kwset backend, so that\noptimization was counterproductive.\n\nThis brings back the optimization for \"--fixed-strings\", without\nchanging the semantics of having a NUL-byte in patterns. As seen in\nprevious commits in this series we could support it now, but I'd\nrather just leave that edge-case aside so we don't have one behavior\nor the other depending what \"--fixed-strings\" backend we're using. It\nmakes the behavior harder to understand and document, and makes tests\nfor the different backends more painful.\n\nThis does change the behavior under non-C locales when \"log\"'s\n\"--encoding\" option is used and the heystack/needle in the\ncontent/command-line doesn't have a matching encoding. See the recent\nchange in \"t4210: skip more command-line encoding tests on MinGW\" in\nthis series. I think that's OK. We did nothing sensible before\nthen (just compared raw bytes that had no hope of matching). At least\nnow the user will get some idea why their grep/log never matches in\nthat edge case.\n\nI could also support the PCRE v1 backend here, but that would make the\ncode more complex. I'd rather aim for simplicity here and in future\nchanges to the diffcore. We're not going to have someone who\nabsolutely must have faster search, but for whom building PCRE v2\nisn't acceptable.\n\nThe difference between this series of commits and the current \"master\"\nis, using the same t/perf commands shown in the last commit:\n\nplain grep:\n\n Test origin/master HEAD\n -------------------------------------------------------------------------\n 7821.1: fixed grep int 0.55(1.67+0.56) 0.41(0.98+0.60) -25.5%\n 7821.2: basic grep int 0.58(1.65+0.52) 0.41(0.96+0.57) -29.3%\n 7821.3: extended grep int 0.57(1.66+0.49) 0.42(0.93+0.60) -26.3%\n 7821.4: perl grep int 0.54(1.67+0.50) 0.43(0.88+0.65) -20.4%\n 7821.6: fixed grep uncommon 0.21(0.52+0.42) 0.16(0.24+0.51) -23.8%\n 7821.7: basic grep uncommon 0.20(0.49+0.45) 0.17(0.28+0.47) -15.0%\n 7821.8: extended grep uncommon 0.20(0.54+0.39) 0.16(0.25+0.50) -20.0%\n 7821.9: perl grep uncommon 0.20(0.58+0.36) 0.16(0.23+0.50) -20.0%\n 7821.11: fixed grep æ 0.35(1.24+0.43) 0.16(0.23+0.50) -54.3%\n 7821.12: basic grep æ 0.36(1.29+0.38) 0.16(0.20+0.54) -55.6%\n 7821.13: extended grep æ 0.35(1.23+0.44) 0.16(0.24+0.50) -54.3%\n 7821.14: perl grep æ 0.35(1.33+0.34) 0.16(0.28+0.46) -54.3%\n\ngrep with -i:\n\n Test origin/master HEAD\n ----------------------------------------------------------------------------\n 7821.1: fixed grep -i int 0.62(1.81+0.70) 0.47(1.11+0.64) -24.2%\n 7821.2: basic grep -i int 0.67(1.90+0.53) 0.46(1.07+0.62) -31.3%\n 7821.3: extended grep -i int 0.62(1.92+0.53) 0.53(1.12+0.58) -14.5%\n 7821.4: perl grep -i int 0.66(1.85+0.58) 0.45(1.10+0.59) -31.8%\n 7821.6: fixed grep -i uncommon 0.21(0.54+0.43) 0.17(0.20+0.55) -19.0%\n 7821.7: basic grep -i uncommon 0.20(0.52+0.45) 0.17(0.29+0.48) -15.0%\n 7821.8: extended grep -i uncommon 0.21(0.52+0.44) 0.17(0.26+0.50) -19.0%\n 7821.9: perl grep -i uncommon 0.21(0.53+0.44) 0.17(0.20+0.56) -19.0%\n 7821.11: fixed grep -i æ 0.26(0.79+0.44) 0.16(0.29+0.46) -38.5%\n 7821.12: basic grep -i æ 0.26(0.79+0.42) 0.16(0.20+0.54) -38.5%\n 7821.13: extended grep -i æ 0.26(0.84+0.39) 0.16(0.24+0.50) -38.5%\n 7821.14: perl grep -i æ 0.16(0.24+0.49) 0.17(0.25+0.51) +6.3%\n\nplain log:\n\n Test origin/master HEAD\n --------------------------------------------------------------------------------\n 4221.1: fixed log --grep='int' 7.24(6.95+0.28) 7.20(6.95+0.18) -0.6%\n 4221.2: basic log --grep='int' 7.31(6.97+0.22) 7.20(6.93+0.21) -1.5%\n 4221.3: extended log --grep='int' 7.37(7.04+0.24) 7.22(6.91+0.25) -2.0%\n 4221.4: perl log --grep='int' 7.31(7.04+0.21) 7.19(6.89+0.21) -1.6%\n 4221.6: fixed log --grep='uncommon' 6.93(6.59+0.32) 7.04(6.66+0.37) +1.6%\n 4221.7: basic log --grep='uncommon' 6.92(6.58+0.29) 7.08(6.75+0.29) +2.3%\n 4221.8: extended log --grep='uncommon' 6.92(6.55+0.31) 7.00(6.68+0.31) +1.2%\n 4221.9: perl log --grep='uncommon' 7.03(6.59+0.33) 7.12(6.73+0.34) +1.3%\n 4221.11: fixed log --grep='æ' 7.41(7.08+0.28) 7.05(6.76+0.29) -4.9%\n 4221.12: basic log --grep='æ' 7.39(6.99+0.33) 7.00(6.68+0.25) -5.3%\n 4221.13: extended log --grep='æ' 7.34(7.00+0.25) 7.15(6.81+0.31) -2.6%\n 4221.14: perl log --grep='æ' 7.43(7.13+0.26) 7.01(6.60+0.36) -5.7%\n\nlog with -i:\n\n Test origin/master HEAD\n ------------------------------------------------------------------------------------\n 4221.1: fixed log -i --grep='int' 7.31(7.07+0.24) 7.23(7.00+0.22) -1.1%\n 4221.2: basic log -i --grep='int' 7.40(7.08+0.28) 7.19(6.92+0.20) -2.8%\n 4221.3: extended log -i --grep='int' 7.43(7.13+0.25) 7.27(6.99+0.21) -2.2%\n 4221.4: perl log -i --grep='int' 7.34(7.10+0.24) 7.10(6.90+0.19) -3.3%\n 4221.6: fixed log -i --grep='uncommon' 7.07(6.71+0.32) 7.11(6.77+0.28) +0.6%\n 4221.7: basic log -i --grep='uncommon' 6.99(6.64+0.28) 7.12(6.69+0.38) +1.9%\n 4221.8: extended log -i --grep='uncommon' 7.11(6.74+0.32) 7.10(6.77+0.27) -0.1%\n 4221.9: perl log -i --grep='uncommon' 6.98(6.60+0.29) 7.05(6.64+0.34) +1.0%\n 4221.11: fixed log -i --grep='æ' 7.85(7.45+0.34) 7.03(6.68+0.32) -10.4%\n 4221.12: basic log -i --grep='æ' 7.87(7.49+0.29) 7.06(6.69+0.31) -10.3%\n 4221.13: extended log -i --grep='æ' 7.87(7.54+0.31) 7.09(6.69+0.31) -9.9%\n 4221.14: perl log -i --grep='æ' 7.06(6.77+0.28) 6.91(6.57+0.31) -2.1%\n\nSo as with e05b027627 (\"grep: use PCRE v2 for optimized fixed-string\nsearch\", 2019-06-26) there's a huge improvement in performance for\n\"grep\", but in \"log\" most of our time is spent elsewhere, so we don't\nnotice it that much.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n grep.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 49 insertions(+), 2 deletions(-)\n\ndiff --git a/grep.c b/grep.c\nindex 4468519d5c..fc0ed73ef3 100644\n--- a/grep.c\n+++ b/grep.c\n@@ -356,6 +356,18 @@ static NORETURN void compile_regexp_failed(const struct grep_pat *p,\n \tdie(\"%s'%s': %s\", where, p->pattern, error);\n }\n \n+static int is_fixed(const char *s, size_t len)\n+{\n+\tsize_t i;\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (is_regex_special(s[i]))\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n #ifdef USE_LIBPCRE1\n static void compile_pcre1_regexp(struct grep_pat *p, const struct grep_opt *opt)\n {\n@@ -602,7 +614,6 @@ static int pcre2match(struct grep_pat *p, const char *line, const char *eol,\n static void free_pcre2_pattern(struct grep_pat *p)\n {\n }\n-#endif /* !USE_LIBPCRE2 */\n \n static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n@@ -623,11 +634,13 @@ static void compile_fixed_regexp(struct grep_pat *p, struct grep_opt *opt)\n \t\tcompile_regexp_failed(p, errbuf);\n \t}\n }\n+#endif /* !USE_LIBPCRE2 */\n \n static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n {\n \tint err;\n \tint regflags = REG_NEWLINE;\n+\tint pat_is_fixed;\n \n \tp->word_regexp = opt->word_regexp;\n \tp->ignore_case = opt->ignore_case;\n@@ -636,8 +649,42 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)\n \tif (memchr(p->pattern, 0, p->patternlen) && !opt->pcre2)\n \t\tdie(_(\"given pattern contains NULL byte (via -f ). This is only supported with -P under PCRE v2\"));\n \n-\tif (opt->fixed) {\n+\tpat_is_fixed = is_fixed(p->pattern, p->patternlen);\n+\tif (opt->fixed || pat_is_fixed) {\n+#ifdef USE_LIBPCRE2\n+\t\topt->pcre2 = 1;\n+\t\tif (pat_is_fixed) {\n+\t\t\tcompile_pcre2_pattern(p, opt);\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * E.g. t7811-grep-open.sh relies on the\n+\t\t\t * pattern being restored.\n+\t\t\t */\n+\t\t\tchar *old_pattern = p->pattern;\n+\t\t\tsize_t old_patternlen = p->patternlen;\n+\t\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\t\t/*\n+\t\t\t * There is the PCRE2_LITERAL flag, but it's\n+\t\t\t * only in PCRE v2 10.30 and later. Needing to\n+\t\t\t * ifdef our way around that and dealing with\n+\t\t\t * it + PCRE2_MULTILINE being an error is more\n+\t\t\t * complex than just quoting this ourselves.\n+\t\t\t*/\n+\t\t\tstrbuf_add(&sb, \"\\\\Q\", 2);\n+\t\t\tstrbuf_add(&sb, p->pattern, p->patternlen);\n+\t\t\tstrbuf_add(&sb, \"\\\\E\", 2);\n+\n+\t\t\tp->pattern = sb.buf;\n+\t\t\tp->patternlen = sb.len;\n+\t\t\tcompile_pcre2_pattern(p, opt);\n+\t\t\tp->pattern = old_pattern;\n+\t\t\tp->patternlen = old_patternlen;\n+\t\t\tstrbuf_release(&sb);\n+\t\t}\n+#else /* !USE_LIBPCRE2 */\n \t\tcompile_fixed_regexp(p, opt);\n+#endif /* !USE_LIBPCRE2 */\n \t\treturn;\n \t}\n \n-- \n2.22.0.455.g172b71a6c5\n\n"},{"id":"378463","messageId":"xmqqef395tmi.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"20190701212100.27850-1-avarab@gmail.com","subject":"Re: [PATCH v3 00/10] grep: move from kwset to optional PCRE v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-07-01T21:31:49Z","receivedAt":"2019-07-01T21:31:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> This v3 has a new patch (3/10) that I believe fixes the regression on\n> MinGW Johannes noted in\n> https://public-inbox.org/git/nycvar.QRO.7.76.6.1907011515150.44@tvgsbejvaqbjf.bet/\n>\n> As noted in the updated commit message in 10/10 I believe just\n> skipping this test & documenting this in a commit message is the least\n> amount of suck for now. It's really an existing issue with us doing\n> nothing sensible when the log/grep haystack encoding doesn't match the\n> needle encoding supplied via the command line.\n\nIs that quite the case? If they do not match, not finding the match\nis the right answer, because we are byte-for-byte matching/searching\nIIUC.\n\n> We swept that under the carpet with the kwset backend, but PCRE v2\n> exposes it.\n\nIs it exposing, or just showing the limitation of the rewritten\nimplementation where it cannot do byte-for-byte matching/searching\nas we used to be able to?\n\nWithout having a way to know what encoding is used on the command\nline, there is no sensible way to reencode them to match the\nhaystack encoding (even when it is known), so \"you got to feed the\nstrings in the same encoding, as we are going to match/search\nbyte-for-byte\" is the only sensible way to work, given the design\nspace, I would think.\n\nNot that it is all that useful to be able to match/search\nbyte-for-byte, of course, so I am OK if we punt with these tests,\nbut I'd prefer to see us admit we are punting when we do ;-).\n\n\n\n\n"},{"id":"378489","messageId":"87pnms7kv0.fsf@evledraar.gmail.com","threadId":"51300","inReplyTo":"xmqqef395tmi.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 00/10] grep: move from kwset to optional PCRE v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-07-02T11:10:27Z","receivedAt":"2019-07-02T11:10:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 01 2019, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> This v3 has a new patch (3/10) that I believe fixes the regression on\n>> MinGW Johannes noted in\n>> https://public-inbox.org/git/nycvar.QRO.7.76.6.1907011515150.44@tvgsbejvaqbjf.bet/\n>>\n>> As noted in the updated commit message in 10/10 I believe just\n>> skipping this test & documenting this in a commit message is the least\n>> amount of suck for now. It's really an existing issue with us doing\n>> nothing sensible when the log/grep haystack encoding doesn't match the\n>> needle encoding supplied via the command line.\n>\n> Is that quite the case? If they do not match, not finding the match\n> is the right answer, because we are byte-for-byte matching/searching\n> IIUC.\n>\n>> We swept that under the carpet with the kwset backend, but PCRE v2\n>> exposes it.\n>\n> Is it exposing, or just showing the limitation of the rewritten\n> implementation where it cannot do byte-for-byte matching/searching\n> as we used to be able to?\n>\n> Without having a way to know what encoding is used on the command\n> line, there is no sensible way to reencode them to match the\n> haystack encoding (even when it is known), so \"you got to feed the\n> strings in the same encoding, as we are going to match/search\n> byte-for-byte\" is the only sensible way to work, given the design\n> space, I would think.\n>\n> Not that it is all that useful to be able to match/search\n> byte-for-byte, of course, so I am OK if we punt with these tests,\n> but I'd prefer to see us admit we are punting when we do ;-).\n\nI'm guilty as charged in punting this larger encoding issue. As it\npertains to this patch series it unearths an obscure case I think nobody\ncares about in practice, and I'd like to move on with the \"remove kwset\"\noptimization.\n\nBut I strongly believe that the new behavior with the PCRE v2\noptimization is the only sane thing to do, and to the extent we have\nanything left to do (#leftoverbits) it's that we should modify git more\ngenerally (aside from string searching) to do the same thing where\nappropriate.\n\nRemember, this only happens if the user has set a UTF-8 locale and thus\npromised that they're going to give us UTF-8. We then take that promise\nand make e.g. \"æ\" match \"Æ\" under --ignore-case.\n\nJust falling back on raw byte matching isn't going to cut it, because\nthen \"æ\" won't match \"Æ\" under\n--ignore-case, and there's other cases like that with matching word\nboundaries & other Unicode gotchas.\n\nThe best that can be hoped for at that point is some \"loose UTF-8\"\nmode. I see both perl & GNU grep seem to support that (although I'm sure\nit falls apart at some point). GNU grep will also die in the same way\nthat we now die with --perl-regexp (since it also use PCRE).\n\nI think that's saner, if the user thinks they're feeding us UTF-8 but\nthey're not I think they'd like to know rather than having the string\nmatching library fall back.\n"},{"id":"378493","messageId":"nycvar.QRO.7.76.6.1907021417050.48@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"20190701212100.27850-1-avarab@gmail.com","subject":"Re: [PATCH v3 00/10] grep: move from kwset to optional PCRE v2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-07-02T12:32:56Z","receivedAt":"2019-07-02T12:33:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Mon, 1 Jul 2019, Ævar Arnfjörð Bjarmason wrote:\n\n> This v3 has a new patch (3/10) that I believe fixes the regression on\n> MinGW Johannes noted in\n> https://public-inbox.org/git/nycvar.QRO.7.76.6.1907011515150.44@tvgsbejvaqbjf.bet/\n\nYes.\n\nHowever, I probably failed to mention another breakage, though...:\n\nnot ok 54 - LC_ALL='C' git grep -P -f f -i 'Æ[Ð]' a\n\n expecting success:\n\t\t\t>stderr &&\n\t\t\tprintf 'ÆQ[Ð]' | q_to_nul >f &&\n\t\t\ttest_must_fail env LC_ALL=\"C\" git grep -P -f f -i a 2>stderr &&\n\t\t\ttest_i18ngrep ! 'This is only supported with -P under PCRE v2' stderr\n\n++ printf 'ÆQ[Ð]'\n++ q_to_nul\n++ perl -pe 'y/Q/\\000/'\n++ command /usr/bin/perl -pe 'y/Q/\\000/'\n++ /usr/bin/perl -pe 'y/Q/\\000/'\n++ test_must_fail env LC_ALL=C git grep -P -f f -i a\n++ case \"$1\" in\n++ _test_ok=\n++ env LC_ALL=C git grep -P -f f -i a\nBinary file a matches\n++ exit_code=0\n++ test 0 -eq 0\n++ list_contains '' success\n++ case \",$1,\" in\n++ return 1\n++ echo 'test_must_fail: command succeeded: env LC_ALL=C git grep -P -f f -i a'\ntest_must_fail: command succeeded: env LC_ALL=C git grep -P -f f -i a\n++ return 1\nerror: last command exited with $?=1\n\nThere are three more test cases in that test script that fail similarly. See\nhttps://dev.azure.com/Git-for-Windows/git/_build/results?buildId=38852&view=ms.vss-test-web.build-test-results-tab&runId=1019770&resultId=101368&paneView=debug\n\nI ran out of time to look into this in more detail :-(\n\n> As noted in the updated commit message in 10/10 I believe just\n> skipping this test & documenting this in a commit message is the least\n> amount of suck for now. It's really an existing issue with us doing\n> nothing sensible when the log/grep haystack encoding doesn't match the\n> needle encoding supplied via the command line.\n>\n> We swept that under the carpet with the kwset backend, but PCRE v2\n> exposes it.\n\nPlease note that the problem is _not_ MinGW! The problem is that the\nnon-JIT'ted code path is a lot more stringent than the JIT'ted one. So\nwhat you'd need is a prerequisite that tests whether the PCREv2 in use\nsupports JIT'ted code or not, and skip the test case in the latter one.\n\nOr you fix the code by re-encoding the plain text in UTF-8 if we know that\nit is not UTF-8-encoded but the needle is.\n\nCiao,\nDscho\n"},{"id":"378518","messageId":"xmqqo92c2oru.fsf@gitster-ct.c.googlers.com","threadId":"51300","inReplyTo":"nycvar.QRO.7.76.6.1907021417050.48@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 00/10] grep: move from kwset to optional PCRE v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-07-02T19:57:09Z","receivedAt":"2019-07-02T19:57:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> Please note that the problem is _not_ MinGW! The problem is that the\n> non-JIT'ted code path is a lot more stringent than the JIT'ted one. So\n> what you'd need is a prerequisite that tests whether the PCREv2 in use\n> supports JIT'ted code or not, and skip the test case in the latter one.\n\nHmph, so additional prereq !MINGW may happen to match \"do we use\npcre sans jit?\" but not a right thing to use here?\n\n"},{"id":"378533","messageId":"nycvar.QRO.7.76.6.1907031205110.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"xmqqo92c2oru.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 00/10] grep: move from kwset to optional PCRE v2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-07-03T10:08:33Z","receivedAt":"2019-07-03T10:08:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 2 Jul 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > Please note that the problem is _not_ MinGW! The problem is that the\n> > non-JIT'ted code path is a lot more stringent than the JIT'ted one. So\n> > what you'd need is a prerequisite that tests whether the PCREv2 in use\n> > supports JIT'ted code or not, and skip the test case in the latter one.\n>\n> Hmph, so additional prereq !MINGW may happen to match \"do we use\n> pcre sans jit?\" but not a right thing to use here?\n\nThat's right, the `!MINGW` prereq works by happenstance only, and as soon\nas I find some time to rebuild PCREv2 with JIT support, it will stop doing\nthe right thing.\n\nWhich might happen very soon.\n\nQuite honestly, I'd rather introduce a prerequisite here that specifically\ntests whether the output of a `git grep -P` suggests that it has been fed\nincorrect UTF-8, and skip the test case under that circumstance.\n\nCiao,\nDscho\n"},{"id":"378534","messageId":"nycvar.QRO.7.76.6.1907031213220.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"nycvar.QRO.7.76.6.1907021417050.48@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 00/10] grep: move from kwset to optional PCRE v2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-07-03T10:25:28Z","receivedAt":"2019-07-03T10:25:21Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 2 Jul 2019, Johannes Schindelin wrote:\n\n> [...] I probably failed to mention another breakage, though...:\n>\n> not ok 54 - LC_ALL='C' git grep -P -f f -i 'Æ[Ð]' a\n>\n> expecting success:\n> \t\t\t>stderr &&\n> \t\t\tprintf 'ÆQ[Ð]' | q_to_nul >f &&\n> \t\t\ttest_must_fail env LC_ALL=\"C\" git grep -P -f f -i a 2>stderr &&\n> \t\t\ttest_i18ngrep ! 'This is only supported with -P under PCRE v2' stderr\n>\n> ++ printf 'ÆQ[Ð]'\n> ++ q_to_nul\n> ++ perl -pe 'y/Q/\\000/'\n> ++ command /usr/bin/perl -pe 'y/Q/\\000/'\n> ++ /usr/bin/perl -pe 'y/Q/\\000/'\n> ++ test_must_fail env LC_ALL=C git grep -P -f f -i a\n> ++ case \"$1\" in\n> ++ _test_ok=\n> ++ env LC_ALL=C git grep -P -f f -i a\n> Binary file a matches\n> ++ exit_code=0\n> ++ test 0 -eq 0\n> ++ list_contains '' success\n> ++ case \",$1,\" in\n> ++ return 1\n> ++ echo 'test_must_fail: command succeeded: env LC_ALL=C git grep -P -f f -i a'\n> test_must_fail: command succeeded: env LC_ALL=C git grep -P -f f -i a\n> ++ return 1\n> error: last command exited with $?=1\n>\n> There are three more test cases in that test script that fail similarly. See\n> https://dev.azure.com/Git-for-Windows/git/_build/results?buildId=38852&view=ms.vss-test-web.build-test-results-tab&runId=1019770&resultId=101368&paneView=debug\n>\n> I ran out of time to look into this in more detail :-(\n\nI figured it out. It does not happen with your `ab/no-kwset` patch series\nin isolation, it's only when it is merged into `pu`, and the culprit is\nthe bad interaction with the `js/mingw-use-utf8` branch.\n\nTo fix it, I have a tentative patch:\nhttps://github.com/git-for-windows/git/commit/e561446d\n\nSo I'll head over to that patch series and add more information there.\n\nCiao,\nDscho\n"},{"id":"378539","messageId":"nycvar.QRO.7.76.6.1907031326300.44@tvgsbejvaqbjf.bet","threadId":"51300","inReplyTo":"nycvar.QRO.7.76.6.1907031213220.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 00/10] grep: move from kwset to optional PCRE v2","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-07-03T11:27:22Z","receivedAt":"2019-07-03T11:27:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 3 Jul 2019, Johannes Schindelin wrote:\n\n> Hi,\n>\n> On Tue, 2 Jul 2019, Johannes Schindelin wrote:\n>\n> > [...] I probably failed to mention another breakage, though...:\n> >\n> > not ok 54 - LC_ALL='C' git grep -P -f f -i 'Æ[Ð]' a\n> >\n> > expecting success:\n> > \t\t\t>stderr &&\n> > \t\t\tprintf 'ÆQ[Ð]' | q_to_nul >f &&\n> > \t\t\ttest_must_fail env LC_ALL=\"C\" git grep -P -f f -i a 2>stderr &&\n> > \t\t\ttest_i18ngrep ! 'This is only supported with -P under PCRE v2' stderr\n> >\n> > ++ printf 'ÆQ[Ð]'\n> > ++ q_to_nul\n> > ++ perl -pe 'y/Q/\\000/'\n> > ++ command /usr/bin/perl -pe 'y/Q/\\000/'\n> > ++ /usr/bin/perl -pe 'y/Q/\\000/'\n> > ++ test_must_fail env LC_ALL=C git grep -P -f f -i a\n> > ++ case \"$1\" in\n> > ++ _test_ok=\n> > ++ env LC_ALL=C git grep -P -f f -i a\n> > Binary file a matches\n> > ++ exit_code=0\n> > ++ test 0 -eq 0\n> > ++ list_contains '' success\n> > ++ case \",$1,\" in\n> > ++ return 1\n> > ++ echo 'test_must_fail: command succeeded: env LC_ALL=C git grep -P -f f -i a'\n> > test_must_fail: command succeeded: env LC_ALL=C git grep -P -f f -i a\n> > ++ return 1\n> > error: last command exited with $?=1\n> >\n> > There are three more test cases in that test script that fail similarly. See\n> > https://dev.azure.com/Git-for-Windows/git/_build/results?buildId=38852&view=ms.vss-test-web.build-test-results-tab&runId=1019770&resultId=101368&paneView=debug\n> >\n> > I ran out of time to look into this in more detail :-(\n>\n> I figured it out. It does not happen with your `ab/no-kwset` patch series\n> in isolation, it's only when it is merged into `pu`, and the culprit is\n> the bad interaction with the `js/mingw-use-utf8` branch.\n\nWhoops, it is the `kb/windows-force-utf8` branch instead.\n\nCiao,\nDscho\n"}]}