{"thread":{"id":"59497","subject":"regex compilation error with --color-words","startedAt":"2023-03-29T22:55:20Z","lastAt":"2023-04-07T17:24:11Z","messageCount":13,"participants":["Eric Sunshine","Diomidis Spinellis","René Scharfe","Junio C Hamano","Johannes Sixt","D. Ben Knoble"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"474433","messageId":"CAPig+cSNmws2b7f7aRA2C56kvQYG3w_g+KhYdqhtmf+XhtAMhQ@mail.gmail.com","threadId":"59497","inReplyTo":null,"subject":"regex compilation error with --color-words","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2023-03-29T22:55:05Z","receivedAt":"2023-03-29T22:55:20Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"I'm encountering a failure on macOS High Sierra 10.13.6 when using\n--color-words:\n\n    % git show --color-words HEAD\n    fatal: invalid regular expression:\n[[:alpha:]_'][[:alnum:]_']*|0[xb]?[0-9a-fA-F_]*|[0-9a-fA-F_]+(\\.[0-9a-fA-F_]+)?([eE][-+]?[0-9_]+)?|=>|-[rwxoRWXOezsfdlpSugkbctTBMAC>]|~~|::|&&=|\\|\\|=|//=|\\*\\*=|&&|\\|\\||//|\\+\\+|--|\\*\\*|\\.\\.\\.?|[-+*/%.^&<>=!|]=|=~|!~|<<|<>|<=>|>>|[^[:space:]]|[<C0>-<FF>][<80>-<BF>]+\n\nThis crash happens when viewing the commit I sent to Peff today[1],\nthough it doesn't happen with all commits. The problem bisects to:\n\n    Author: Diomidis Spinellis <dds@aueb.gr>\n    Date:   Fri Aug 26 11:58:15 2022 +0300\n\n    grep: fix multibyte regex handling under macOS\n\n    The commit 29de20504e (Makefile: fix default regex settings on\n    Darwin, 2013-05-11) fixed t0070-fundamental.sh under Darwin (macOS) by\n    adopting Git's regex library.  However, this library is compiled with\n    NO_MBSUPPORT, which causes git-grep to work incorrectly on multibyte\n    (e.g. UTF-8) files.  Current macOS versions pass t0070-fundamental.sh\n    with the native macOS regex library, which also supports multibyte\n    characters.\n\n    Adjust the Makefile to use the native regex library, and call\n    setlocale(3) to set CTYPE according to the user's preference.\n    The setlocale call is required on all platforms, but in platforms\n    supporting gettext(3), setlocale was called as a side-effect of\n    initializing gettext.  Therefore, move the CTYPE setlocale call from\n    gettext.c to common-main.c and the corresponding locale.h include\n    into git-compat-util.h.\n\n    Thanks to the global initialization of CTYPE setlocale, the test-tool\n    regex command now works correctly with supported multibyte regexes, and\n    is used to set the MB_REGEX test prerequisite by assessing a platform's\n    support for them.\n\n    Signed-off-by: Diomidis Spinellis <dds@aueb.gr>\n    Signed-off-by: Junio C Hamano <gitster@pobox.com>\n\nI see that this same commit is also the subject of another bug report\ncurrently being discussed[2], so I've Cc:'d the participants of that\nthread, as well.\n\nAny pointers aimed at getting this resolved would be appreciated.\n\n[1]: https://lore.kernel.org/git/CAPig+cQiOGrDSUc34jHEBp87Rx-dnXNcPcF76bu0SJoOzD+1hw@mail.gmail.com/\n[2]: https://lore.kernel.org/git/MW4PR20MB5517583CBEEF34B1E87CCF1290859@MW4PR20MB5517.namprd20.prod.outlook.com/\n"},{"id":"474448","messageId":"b45bf46f-580a-870c-5293-10ecdf2e56d3@aueb.gr","threadId":"59497","inReplyTo":"CAPig+cSNmws2b7f7aRA2C56kvQYG3w_g+KhYdqhtmf+XhtAMhQ@mail.gmail.com","subject":"Re: regex compilation error with --color-words","fromName":"Diomidis Spinellis","fromEmail":"dds@aueb.gr","sentAt":"2023-03-30T07:55:00Z","receivedAt":"2023-03-30T08:00:47Z","isPatch":false,"sender":{"key":"dds@aueb.gr","avatar":"https://avatars.githubusercontent.com/u/1212716?v=4"},"body":"On 30-Mar-23 1:55, Eric Sunshine wrote:\n> I'm encountering a failure on macOS High Sierra 10.13.6 when using\n> --color-words:\n\nThe built-in word separation regular expression pattern for the Perl \nlanguage fails to work with the macOS regex engine.  The same also \nhappens with the FreeBSD one (tested on 14.0).\n\nThe issue can be replicated through the following sequence of commands.\n\ngit init color-words\ncd color-words\necho '*.pl   diff=perl' >.gitattributes\necho 'print 42;' >t.pl\ngit add t.pl\ngit commit -am Add\ngit show --color-words\n\nStrangely, I haven't been able to reproduce the failure with egrep on \nany of the two platforms.\n\negrep \n'[[:alpha:]_'\\''][[:alnum:]_'\\'']*|0[xb]?[0-9a-fA-F_]*|[0-9a-fA-F_]+(\\.[0-9a-fA-F_]+)?([eE][-+]?[0-9_]+)?|=>|-[rwxoRWXOezsfdlpSugkbctTBMAC>]|~~|::|&&=|\\|\\|=|//=|\\*\\*=|&&|\\|\\||//|\\+\\+|--|\\*\\*|\\.\\.\\.?|[-+*/%.^&<>=!|]=|=~|!~|<<|<>|<=>|>>|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+' \n/dev/null\n"},{"id":"474531","messageId":"b8b3777b-ee6e-d90e-3365-5cb9c9d129fe@web.de","threadId":"59497","inReplyTo":"b45bf46f-580a-870c-5293-10ecdf2e56d3@aueb.gr","subject":"Re: regex compilation error with --color-words","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-03-31T20:44:55Z","receivedAt":"2023-03-31T20:45:25Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 30.03.23 um 09:55 schrieb Diomidis Spinellis:\n> On 30-Mar-23 1:55, Eric Sunshine wrote:\n>> I'm encountering a failure on macOS High Sierra 10.13.6 when using\n>> --color-words:\n>\n> The built-in word separation regular expression pattern for the Perl language fails to work with the macOS regex engine.  The same also happens with the FreeBSD one (tested on 14.0).\n>\n> The issue can be replicated through the following sequence of commands.\n>\n> git init color-words\n> cd color-words\n> echo '*.pl   diff=perl' >.gitattributes\n> echo 'print 42;' >t.pl\n> git add t.pl\n> git commit -am Add\n> git show --color-words\n\nOr in Git's own repo:\n\n   $ git log -p --color-words --no-merges '*.c'\n   Schwerwiegend: invalid regular expression: [a-zA-Z_][a-zA-Z0-9_]*|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*|0[xXbB][0-9a-fA-F]+[lLuU]*|\\.[0-9][0-9]*([Ee][-+]?[0-9]+)?[fFlL]?|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*|<=>|[^[:space:]]|[<C0>-<FF>][<80>-<BF>]+\n   commit 14b9a044798ebb3858a1f1a1377309a3d6054ac8\n   [...]\n\nThe error disappears when localization is turned off:\n\n   $ LANG=C git log -p --color-words --no-merges '*.c' >/dev/null\n   # just finishes without an error\n\nThe issue also vanishes when the \"|[\\xc0-\\xff][\\x80-\\xbf]+\" part is\nremoved that the macros PATTERNS and IPATTERN in userdiff.c append.\n\nSo it seems regcomp(1) on macOS doesn't like invalid Unicode characters\nunless it's in ASCII mode (LANG=C).  664d44ee7f (userdiff: simplify\nword-diff safeguard, 2011-01-11) explains that this part exists to match\na multi-byte UTF-8 character.  With a regcomp(1) that supports\nmulti-byte characters natively they need to be specified differently, I\nguess, perhaps like this \"[^\\x00-\\x7f]\"?\n\n> Strangely, I haven't been able to reproduce the failure with egrep on any of the two platforms.\n>\n> egrep '[[:alpha:]_'\\''][[:alnum:]_'\\'']*|0[xb]?[0-9a-fA-F_]*|[0-9a-fA-F_]+(\\.[0-9a-fA-F_]+)?([eE][-+]?[0-9_]+)?|=>|-[rwxoRWXOezsfdlpSugkbctTBMAC>]|~~|::|&&=|\\|\\|=|//=|\\*\\*=|&&|\\|\\||//|\\+\\+|--|\\*\\*|\\.\\.\\.?|[-+*/%.^&<>=!|]=|=~|!~|<<|<>|<=>|>>|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+' /dev/null\n\nNo idea how to specify non-ASCII bytes in shell or regex.  '\\xNN' does\nnot seem to do the trick.  printf(1) interpretes octal numbers, though:\n\n   $ echo ö | egrep $(printf \"[\\200-\\377]\")\n   egrep: illegal byte sequence\n\n(The regex contains \"illegal bytes\" -- UTF-8 multi-byte sequences cut\nshort; the \"ö\" is OK.)\n\nRené\n"},{"id":"474655","messageId":"c4728fac-bea9-3794-077e-c978d99f46bf@web.de","threadId":"59497","inReplyTo":"b8b3777b-ee6e-d90e-3365-5cb9c9d129fe@web.de","subject":"Re: regex compilation error with --color-words","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-04-02T09:44:30Z","receivedAt":"2023-04-02T09:44:42Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 31.03.23 um 22:44 schrieb René Scharfe:\n> Am 30.03.23 um 09:55 schrieb Diomidis Spinellis:\n>> On 30-Mar-23 1:55, Eric Sunshine wrote:\n>>> I'm encountering a failure on macOS High Sierra 10.13.6 when using\n>>> --color-words:\n>>\n>> The built-in word separation regular expression pattern for the Perl language fails to work with the macOS regex engine.  The same also happens with the FreeBSD one (tested on 14.0).\n>>\n>> The issue can be replicated through the following sequence of commands.\n>>\n>> git init color-words\n>> cd color-words\n>> echo '*.pl   diff=perl' >.gitattributes\n>> echo 'print 42;' >t.pl\n>> git add t.pl\n>> git commit -am Add\n>> git show --color-words\n>\n> Or in Git's own repo:\n>\n>    $ git log -p --color-words --no-merges '*.c'\n>    Schwerwiegend: invalid regular expression: [a-zA-Z_][a-zA-Z0-9_]*|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*|0[xXbB][0-9a-fA-F]+[lLuU]*|\\.[0-9][0-9]*([Ee][-+]?[0-9]+)?[fFlL]?|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*|<=>|[^[:space:]]|[<C0>-<FF>][<80>-<BF>]+\n>    commit 14b9a044798ebb3858a1f1a1377309a3d6054ac8\n>    [...]\n>\n> The error disappears when localization is turned off:\n>\n>    $ LANG=C git log -p --color-words --no-merges '*.c' >/dev/null\n>    # just finishes without an error\n>\n> The issue also vanishes when the \"|[\\xc0-\\xff][\\x80-\\xbf]+\" part is\n> removed that the macros PATTERNS and IPATTERN in userdiff.c append.\n>\n> So it seems regcomp(1) on macOS doesn't like invalid Unicode characters\n> unless it's in ASCII mode (LANG=C).  664d44ee7f (userdiff: simplify\n> word-diff safeguard, 2011-01-11) explains that this part exists to match\n> a multi-byte UTF-8 character.  With a regcomp(1) that supports\n> multi-byte characters natively they need to be specified differently, I\n> guess, perhaps like this \"[^\\x00-\\x7f]\"?\n\nActually we can drop the \"|[\\xc0-\\xff][\\x80-\\xbf]+\" part in that case\nbecause the \"[^[:space:]]\" suffices.  And we probably need to do that at\nruntime because it depends on the locale.  The rather elaborate patch\nbelow does that.  It leaks the truncated word_regex, which isn't that\nbad because it's done only once per run, but certainly untidy.\n\nI suspect/hope this can be done simpler and cleaner after refactoring\nthe userdiff code to allow for runtime assembly of regular expressions.\n\nAnd it's regcomp(3), or rather regexec(3), not regcomp(1).\n\n---\n userdiff.c | 38 ++++++++++++++++++++++++++++++++++++--\n userdiff.h |  2 ++\n 2 files changed, 38 insertions(+), 2 deletions(-)\n\ndiff --git a/userdiff.c b/userdiff.c\nindex 09203fbc35..aa2cd150ba 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -9,6 +9,8 @@ static struct userdiff_driver *drivers;\n static int ndrivers;\n static int drivers_alloc;\n\n+#define OR_MULTI_BYTE_CHAR \"|[\\xc0-\\xff][\\x80-\\xbf]+\"\n+\n #define PATTERNS(lang, rx, wrx) { \\\n \t.name = lang, \\\n \t.binary = -1, \\\n@@ -16,7 +18,9 @@ static int drivers_alloc;\n \t\t.pattern = rx, \\\n \t\t.cflags = REG_EXTENDED, \\\n \t}, \\\n-\t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n+\t.word_regex = wrx \"|[^[:space:]]\" OR_MULTI_BYTE_CHAR, \\\n+\t.is_builtin = 1, \\\n+\t.has_multi_byte_char_fallback = 1, \\\n }\n #define IPATTERN(lang, rx, wrx) { \\\n \t.name = lang, \\\n@@ -25,7 +29,9 @@ static int drivers_alloc;\n \t\t.pattern = rx, \\\n \t\t.cflags = REG_EXTENDED | REG_ICASE, \\\n \t}, \\\n-\t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n+\t.word_regex = wrx \"|[^[:space:]]\" OR_MULTI_BYTE_CHAR, \\\n+\t.is_builtin = 1, \\\n+\t.has_multi_byte_char_fallback = 1, \\\n }\n\n /*\n@@ -330,6 +336,25 @@ static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,\n \treturn 0;\n }\n\n+static int regexec_support_multi_byte_chars(void)\n+{\n+\tstatic const char not_space[] = \"[^[:space:]]\";\n+\tstatic const char utf8_multi_byte_char[] = \"\\xc2\\xa3\";\n+\tregex_t re;\n+\tregmatch_t match;\n+\tstatic int result = -1;\n+\n+\tif (result != -1)\n+\t\treturn result;\n+\tif (regcomp(&re, not_space, REG_EXTENDED))\n+\t\tBUG(\"invalid regular expression: %s\", not_space);\n+\tresult = !regexec(&re, utf8_multi_byte_char, 1, &match, 0) &&\n+\t\tmatch.rm_so == 0 &&\n+\t\tmatch.rm_eo == strlen(utf8_multi_byte_char);\n+\tregfree(&re);\n+\treturn result;\n+}\n+\n static struct userdiff_driver *userdiff_find_by_namelen(const char *name, size_t len)\n {\n \tstruct find_by_namelen_data udcbdata = {\n@@ -337,6 +362,15 @@ static struct userdiff_driver *userdiff_find_by_namelen(const char *name, size_t\n \t\t.len = len,\n \t};\n \tfor_each_userdiff_driver(userdiff_find_by_namelen_cb, &udcbdata);\n+\tif (udcbdata.driver &&\n+\t    udcbdata.driver->is_builtin &&\n+\t    udcbdata.driver->has_multi_byte_char_fallback &&\n+\t    regexec_support_multi_byte_chars()) {\n+\t\tconst char *word_regex = udcbdata.driver->word_regex;\n+\t\tudcbdata.driver->word_regex = xmemdupz(word_regex,\n+\t\t\tstrlen(word_regex) - strlen(OR_MULTI_BYTE_CHAR));\n+\t\tudcbdata.driver->has_multi_byte_char_fallback = 0;\n+\t}\n \treturn udcbdata.driver;\n }\n\ndiff --git a/userdiff.h b/userdiff.h\nindex 24419db697..83f5863d58 100644\n--- a/userdiff.h\n+++ b/userdiff.h\n@@ -21,6 +21,8 @@ struct userdiff_driver {\n \tconst char *textconv;\n \tstruct notes_cache *textconv_cache;\n \tint textconv_want_cache;\n+\tint is_builtin;\n+\tint has_multi_byte_char_fallback;\n };\n enum userdiff_driver_type {\n \tUSERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,\n--\n2.40.0\n"},{"id":"474687","messageId":"xmqq5yad7wv3.fsf@gitster.g","threadId":"59497","inReplyTo":"c4728fac-bea9-3794-077e-c978d99f46bf@web.de","subject":"Re: regex compilation error with --color-words","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-04-03T16:29:20Z","receivedAt":"2023-04-03T16:29:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n> Actually we can drop the \"|[\\xc0-\\xff][\\x80-\\xbf]+\" part in that case\n> because the \"[^[:space:]]\" suffices.  And we probably need to do that at\n> runtime because it depends on the locale.  The rather elaborate patch\n> below does that.  It leaks the truncated word_regex, which isn't that\n> bad because it's done only once per run, but certainly untidy.\n\nSmall ugliness like what we see below is fine in a technology\ndemonostration.\n\n> I suspect/hope this can be done simpler and cleaner after refactoring\n> the userdiff code to allow for runtime assembly of regular expressions.\n\nDo we expect \"does the regcomp(3) and regexec(3) correctly match a\nnon-space multi-byte UTF-8 sequence as expected?\" to be the only\nchoices, do we expect we will choose from only two, and do we expect\nthat the differences between the MB version and fallback version to\nbe the same \"OR_MULTI_BYTE_CHAR may be omitted\"?  For now I think\nit would be reasonable to answer yes to all three.\n\nHow are .is_builtin and .has_multi_byte_char_fallback bits expected\nto be used?  For what kind of files do we expect them to be set\ndifferently?\n\nIn the simplest case, I would imagine that we could do this\n\n \t...\n \tconst char *word_regex;\n+\tconst char *word_regex_wo_mb;\n \tconst char *textconv;\n \t...\n\nin the definition of \"struct userdifif_driver\", use\n\n #define PATTERNS(lang, rx, wrx) { \\\n \t...\n \t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n+\t.word_regex_wo_mb = wrx \"|[^[:space:]]\", \\\n }\n\nand similar for IPATTERN, and make a non-NULL .word.regex_wo_mb\nserve as the .has_multi_byte_char_fallback bit to trigger \"does our\nregex engine do a good job for multi-byte?\"\n\nThanks.\n\n> diff --git a/userdiff.c b/userdiff.c\n> index 09203fbc35..aa2cd150ba 100644\n> --- a/userdiff.c\n> +++ b/userdiff.c\n> @@ -9,6 +9,8 @@ static struct userdiff_driver *drivers;\n>  static int ndrivers;\n>  static int drivers_alloc;\n>\n> +#define OR_MULTI_BYTE_CHAR \"|[\\xc0-\\xff][\\x80-\\xbf]+\"\n> +\n>  #define PATTERNS(lang, rx, wrx) { \\\n>  \t.name = lang, \\\n>  \t.binary = -1, \\\n> @@ -16,7 +18,9 @@ static int drivers_alloc;\n>  \t\t.pattern = rx, \\\n>  \t\t.cflags = REG_EXTENDED, \\\n>  \t}, \\\n> -\t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n> +\t.word_regex = wrx \"|[^[:space:]]\" OR_MULTI_BYTE_CHAR, \\\n> +\t.is_builtin = 1, \\\n> +\t.has_multi_byte_char_fallback = 1, \\\n>  }\n>  #define IPATTERN(lang, rx, wrx) { \\\n>  \t.name = lang, \\\n> @@ -25,7 +29,9 @@ static int drivers_alloc;\n>  \t\t.pattern = rx, \\\n>  \t\t.cflags = REG_EXTENDED | REG_ICASE, \\\n>  \t}, \\\n> -\t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n> +\t.word_regex = wrx \"|[^[:space:]]\" OR_MULTI_BYTE_CHAR, \\\n> +\t.is_builtin = 1, \\\n> +\t.has_multi_byte_char_fallback = 1, \\\n>  }\n>\n>  /*\n> @@ -330,6 +336,25 @@ static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,\n>  \treturn 0;\n>  }\n>\n> +static int regexec_support_multi_byte_chars(void)\n> +{\n> +\tstatic const char not_space[] = \"[^[:space:]]\";\n> +\tstatic const char utf8_multi_byte_char[] = \"\\xc2\\xa3\";\n> +\tregex_t re;\n> +\tregmatch_t match;\n> +\tstatic int result = -1;\n> +\n> +\tif (result != -1)\n> +\t\treturn result;\n> +\tif (regcomp(&re, not_space, REG_EXTENDED))\n> +\t\tBUG(\"invalid regular expression: %s\", not_space);\n> +\tresult = !regexec(&re, utf8_multi_byte_char, 1, &match, 0) &&\n> +\t\tmatch.rm_so == 0 &&\n> +\t\tmatch.rm_eo == strlen(utf8_multi_byte_char);\n> +\tregfree(&re);\n> +\treturn result;\n> +}\n> +\n>  static struct userdiff_driver *userdiff_find_by_namelen(const char *name, size_t len)\n>  {\n>  \tstruct find_by_namelen_data udcbdata = {\n> @@ -337,6 +362,15 @@ static struct userdiff_driver *userdiff_find_by_namelen(const char *name, size_t\n>  \t\t.len = len,\n>  \t};\n>  \tfor_each_userdiff_driver(userdiff_find_by_namelen_cb, &udcbdata);\n> +\tif (udcbdata.driver &&\n> +\t    udcbdata.driver->is_builtin &&\n> +\t    udcbdata.driver->has_multi_byte_char_fallback &&\n> +\t    regexec_support_multi_byte_chars()) {\n> +\t\tconst char *word_regex = udcbdata.driver->word_regex;\n> +\t\tudcbdata.driver->word_regex = xmemdupz(word_regex,\n> +\t\t\tstrlen(word_regex) - strlen(OR_MULTI_BYTE_CHAR));\n> +\t\tudcbdata.driver->has_multi_byte_char_fallback = 0;\n> +\t}\n>  \treturn udcbdata.driver;\n>  }\n>\n> diff --git a/userdiff.h b/userdiff.h\n> index 24419db697..83f5863d58 100644\n> --- a/userdiff.h\n> +++ b/userdiff.h\n> @@ -21,6 +21,8 @@ struct userdiff_driver {\n>  \tconst char *textconv;\n>  \tstruct notes_cache *textconv_cache;\n>  \tint textconv_want_cache;\n> +\tint is_builtin;\n> +\tint has_multi_byte_char_fallback;\n>  };\n>  enum userdiff_driver_type {\n>  \tUSERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,\n> --\n> 2.40.0\n"},{"id":"474711","messageId":"bc6e89c9-d886-c519-85b3-fbc3f4eb5528@web.de","threadId":"59497","inReplyTo":"xmqq5yad7wv3.fsf@gitster.g","subject":"Re: regex compilation error with --color-words","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-04-03T19:32:51Z","receivedAt":"2023-04-03T19:33:10Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 03.04.23 um 18:29 schrieb Junio C Hamano:\n> René Scharfe <l.s.r@web.de> writes:\n>\n>> Actually we can drop the \"|[\\xc0-\\xff][\\x80-\\xbf]+\" part in that case\n>> because the \"[^[:space:]]\" suffices.  And we probably need to do that at\n>> runtime because it depends on the locale.  The rather elaborate patch\n>> below does that.  It leaks the truncated word_regex, which isn't that\n>> bad because it's done only once per run, but certainly untidy.\n>\n> Small ugliness like what we see below is fine in a technology\n> demonostration.\n>\n>> I suspect/hope this can be done simpler and cleaner after refactoring\n>> the userdiff code to allow for runtime assembly of regular expressions.\n>\n> Do we expect \"does the regcomp(3) and regexec(3) correctly match a\n> non-space multi-byte UTF-8 sequence as expected?\" to be the only\n> choices, do we expect we will choose from only two, and do we expect\n> that the differences between the MB version and fallback version to\n> be the same \"OR_MULTI_BYTE_CHAR may be omitted\"?  For now I think\n> it would be reasonable to answer yes to all three.\n>\n> How are .is_builtin and .has_multi_byte_char_fallback bits expected\n> to be used?  For what kind of files do we expect them to be set\n> differently?\n\n.is_builtin is unnecessary here.  It is a remnant of me noticing that we\ndon't add \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\" to user-defined\npatterns, but .has_multi_byte_char_fallback alone suffices.\n\n>\n> In the simplest case, I would imagine that we could do this\n>\n>  \t...\n>  \tconst char *word_regex;\n> +\tconst char *word_regex_wo_mb;\n>  \tconst char *textconv;\n>  \t...\n>\n> in the definition of \"struct userdifif_driver\", use\n>\n>  #define PATTERNS(lang, rx, wrx) { \\\n>  \t...\n>  \t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n> +\t.word_regex_wo_mb = wrx \"|[^[:space:]]\", \\\n\nAh, nice, no allocation or string manipulation at runtime at all, at\nthe small cost of having near-duplicate static strings.\n\n>  }\n>\n> and similar for IPATTERN, and make a non-NULL .word.regex_wo_mb\n> serve as the .has_multi_byte_char_fallback bit to trigger \"does our\n> regex engine do a good job for multi-byte?\"\n>\n> Thanks.\n>\n>> diff --git a/userdiff.c b/userdiff.c\n>> index 09203fbc35..aa2cd150ba 100644\n>> --- a/userdiff.c\n>> +++ b/userdiff.c\n>> @@ -9,6 +9,8 @@ static struct userdiff_driver *drivers;\n>>  static int ndrivers;\n>>  static int drivers_alloc;\n>>\n>> +#define OR_MULTI_BYTE_CHAR \"|[\\xc0-\\xff][\\x80-\\xbf]+\"\n>> +\n>>  #define PATTERNS(lang, rx, wrx) { \\\n>>  \t.name = lang, \\\n>>  \t.binary = -1, \\\n>> @@ -16,7 +18,9 @@ static int drivers_alloc;\n>>  \t\t.pattern = rx, \\\n>>  \t\t.cflags = REG_EXTENDED, \\\n>>  \t}, \\\n>> -\t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n>> +\t.word_regex = wrx \"|[^[:space:]]\" OR_MULTI_BYTE_CHAR, \\\n>> +\t.is_builtin = 1, \\\n>> +\t.has_multi_byte_char_fallback = 1, \\\n>>  }\n>>  #define IPATTERN(lang, rx, wrx) { \\\n>>  \t.name = lang, \\\n>> @@ -25,7 +29,9 @@ static int drivers_alloc;\n>>  \t\t.pattern = rx, \\\n>>  \t\t.cflags = REG_EXTENDED | REG_ICASE, \\\n>>  \t}, \\\n>> -\t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n>> +\t.word_regex = wrx \"|[^[:space:]]\" OR_MULTI_BYTE_CHAR, \\\n>> +\t.is_builtin = 1, \\\n>> +\t.has_multi_byte_char_fallback = 1, \\\n>>  }\n>>\n>>  /*\n>> @@ -330,6 +336,25 @@ static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,\n>>  \treturn 0;\n>>  }\n>>\n>> +static int regexec_support_multi_byte_chars(void)\n>> +{\n>> +\tstatic const char not_space[] = \"[^[:space:]]\";\n>> +\tstatic const char utf8_multi_byte_char[] = \"\\xc2\\xa3\";\n>> +\tregex_t re;\n>> +\tregmatch_t match;\n>> +\tstatic int result = -1;\n>> +\n>> +\tif (result != -1)\n>> +\t\treturn result;\n>> +\tif (regcomp(&re, not_space, REG_EXTENDED))\n>> +\t\tBUG(\"invalid regular expression: %s\", not_space);\n>> +\tresult = !regexec(&re, utf8_multi_byte_char, 1, &match, 0) &&\n>> +\t\tmatch.rm_so == 0 &&\n>> +\t\tmatch.rm_eo == strlen(utf8_multi_byte_char);\n>> +\tregfree(&re);\n>> +\treturn result;\n>> +}\n>> +\n>>  static struct userdiff_driver *userdiff_find_by_namelen(const char *name, size_t len)\n>>  {\n>>  \tstruct find_by_namelen_data udcbdata = {\n>> @@ -337,6 +362,15 @@ static struct userdiff_driver *userdiff_find_by_namelen(const char *name, size_t\n>>  \t\t.len = len,\n>>  \t};\n>>  \tfor_each_userdiff_driver(userdiff_find_by_namelen_cb, &udcbdata);\n>> +\tif (udcbdata.driver &&\n>> +\t    udcbdata.driver->is_builtin &&\n>> +\t    udcbdata.driver->has_multi_byte_char_fallback &&\n>> +\t    regexec_support_multi_byte_chars()) {\n>> +\t\tconst char *word_regex = udcbdata.driver->word_regex;\n>> +\t\tudcbdata.driver->word_regex = xmemdupz(word_regex,\n>> +\t\t\tstrlen(word_regex) - strlen(OR_MULTI_BYTE_CHAR));\n>> +\t\tudcbdata.driver->has_multi_byte_char_fallback = 0;\n>> +\t}\n>>  \treturn udcbdata.driver;\n>>  }\n>>\n>> diff --git a/userdiff.h b/userdiff.h\n>> index 24419db697..83f5863d58 100644\n>> --- a/userdiff.h\n>> +++ b/userdiff.h\n>> @@ -21,6 +21,8 @@ struct userdiff_driver {\n>>  \tconst char *textconv;\n>>  \tstruct notes_cache *textconv_cache;\n>>  \tint textconv_want_cache;\n>> +\tint is_builtin;\n>> +\tint has_multi_byte_char_fallback;\n>>  };\n>>  enum userdiff_driver_type {\n>>  \tUSERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,\n>> --\n>> 2.40.0\n"},{"id":"474950","messageId":"7327ac06-d5da-ec53-543e-78e7729e78bb@web.de","threadId":"59497","inReplyTo":"bc6e89c9-d886-c519-85b3-fbc3f4eb5528@web.de","subject":"[PATCH] userdiff: support regexec(3) with multi-byte support","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-04-06T20:19:11Z","receivedAt":"2023-04-06T20:19:30Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Since 1819ad327b (grep: fix multibyte regex handling under macOS,\n2022-08-26) we use the system library for all regular expression\nmatching on macOS, not just for git grep.  It supports multi-byte\nstrings and rejects invalid multi-byte characters.\n\nThis broke all built-in userdiff word regexes in UTF-8 locales because\nthey all include such invalid bytes in expressions that are intended to\nmatch multi-byte characters without explicit support for that from the\nregex engine.\n\n\"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\" is added to all built-in word\nregexes to match a single non-space or multi-byte character.  The \\xNN\ncharacters are invalid if interpreted as UTF-8 because they have their\nhigh bit set, which indicates they are part of a multi-byte character,\nbut they are surrounded by single-byte characters.\n\nReplace that expression with \"|[^[:space:]]\" if the regex engine\nsupports multi-byte matching, as there is no need to have an explicit\nrange for multi-byte characters then.  Check for that capability at\nruntime, because it depends on the locale and thus on environment\nvariables.  Construct the full replacement expression at build time\nand just switch it in if necessary to avoid string manipulation and\nallocations at runtime.\n\nAdditionally the word regex for tex contains the expression\n\"[a-zA-Z0-9\\x80-\\xff]+\" with a similarly invalid range.  The best\nreplacement with only valid characters that I can come up with is\n\"([a-zA-Z0-9]|[^\\x01-\\x7f])+\".  Unlike the original it matches NUL\ncharacters, though.  Assuming that tex files usually don't contain NUL\nthis should be acceptable.\n\nReported-by: D. Ben Knoble <ben.knoble@gmail.com>\nReported-by: Eric Sunshine <sunshine@sunshineco.com>\nHelped-by: Junio C Hamano <gitster@pobox.com>\nSigned-off-by: René Scharfe <l.s.r@web.de>\n---\n t/t4034-diff-words.sh |  4 ++++\n userdiff.c            | 31 +++++++++++++++++++++++++++++--\n userdiff.h            |  1 +\n 3 files changed, 34 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t4034-diff-words.sh b/t/t4034-diff-words.sh\nindex 15764ee9ac..74586f3813 100755\n--- a/t/t4034-diff-words.sh\n+++ b/t/t4034-diff-words.sh\n@@ -69,6 +69,10 @@ test_language_driver () {\n \t\techo \"* diff='\"$lang\"'\" >.gitattributes &&\n \t\tword_diff --color-words\n \t'\n+\ttest_expect_success \"diff driver '$lang' in Islandic\" '\n+\t\tLANG=is_IS.UTF-8 LANGUAGE=is LC_ALL=\"$is_IS_locale\" \\\n+\t\tword_diff --color-words\n+\t'\n }\n\n test_expect_success setup '\ndiff --git a/userdiff.c b/userdiff.c\nindex 09203fbc35..eaec6ebb5e 100644\n--- a/userdiff.c\n+++ b/userdiff.c\n@@ -17,6 +17,7 @@ static int drivers_alloc;\n \t\t.cflags = REG_EXTENDED, \\\n \t}, \\\n \t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n+\t.word_regex_multi_byte = wrx \"|[^[:space:]]\", \\\n }\n #define IPATTERN(lang, rx, wrx) { \\\n \t.name = lang, \\\n@@ -26,6 +27,7 @@ static int drivers_alloc;\n \t\t.cflags = REG_EXTENDED | REG_ICASE, \\\n \t}, \\\n \t.word_regex = wrx \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\", \\\n+\t.word_regex_multi_byte = wrx \"|[^[:space:]]\", \\\n }\n\n /*\n@@ -294,7 +296,7 @@ PATTERNS(\"scheme\",\n \t /* All other words should be delimited by spaces or parentheses */\n \t \"|([^][)(}{[ \\t])+\"),\n PATTERNS(\"tex\", \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\",\n-\t \"\\\\\\\\[a-zA-Z@]+|\\\\\\\\.|[a-zA-Z0-9\\x80-\\xff]+\"),\n+\t \"\\\\\\\\[a-zA-Z@]+|\\\\\\\\.|([a-zA-Z0-9]|[^\\x01-\\x7f])+\"),\n { \"default\", NULL, NULL, -1, { NULL, 0 } },\n };\n #undef PATTERNS\n@@ -330,6 +332,25 @@ static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,\n \treturn 0;\n }\n\n+static int regexec_supports_multi_byte_chars(void)\n+{\n+\tstatic const char not_space[] = \"[^[:space:]]\";\n+\tstatic const char utf8_multi_byte_char[] = \"\\xc2\\xa3\";\n+\tregex_t re;\n+\tregmatch_t match;\n+\tstatic int result = -1;\n+\n+\tif (result != -1)\n+\t\treturn result;\n+\tif (regcomp(&re, not_space, REG_EXTENDED))\n+\t\tBUG(\"invalid regular expression: %s\", not_space);\n+\tresult = !regexec(&re, utf8_multi_byte_char, 1, &match, 0) &&\n+\t\tmatch.rm_so == 0 &&\n+\t\tmatch.rm_eo == strlen(utf8_multi_byte_char);\n+\tregfree(&re);\n+\treturn result;\n+}\n+\n static struct userdiff_driver *userdiff_find_by_namelen(const char *name, size_t len)\n {\n \tstruct find_by_namelen_data udcbdata = {\n@@ -405,7 +426,13 @@ int userdiff_config(const char *k, const char *v)\n struct userdiff_driver *userdiff_find_by_name(const char *name)\n {\n \tint len = strlen(name);\n-\treturn userdiff_find_by_namelen(name, len);\n+\tstruct userdiff_driver *driver = userdiff_find_by_namelen(name, len);\n+\tif (driver && driver->word_regex_multi_byte) {\n+\t\tif (regexec_supports_multi_byte_chars())\n+\t\t\tdriver->word_regex = driver->word_regex_multi_byte;\n+\t\tdriver->word_regex_multi_byte = NULL;\n+\t}\n+\treturn driver;\n }\n\n struct userdiff_driver *userdiff_find_by_path(struct index_state *istate,\ndiff --git a/userdiff.h b/userdiff.h\nindex 24419db697..d726804c3e 100644\n--- a/userdiff.h\n+++ b/userdiff.h\n@@ -18,6 +18,7 @@ struct userdiff_driver {\n \tint binary;\n \tstruct userdiff_funcname funcname;\n \tconst char *word_regex;\n+\tconst char *word_regex_multi_byte;\n \tconst char *textconv;\n \tstruct notes_cache *textconv_cache;\n \tint textconv_want_cache;\n--\n2.40.0\n\n"},{"id":"474960","messageId":"7fe0aa93-a764-66b0-5015-2f5fbd3901ab@kdbg.org","threadId":"59497","inReplyTo":"7327ac06-d5da-ec53-543e-78e7729e78bb@web.de","subject":"Re: [PATCH] userdiff: support regexec(3) with multi-byte support","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2023-04-06T22:35:41Z","receivedAt":"2023-04-06T23:23:20Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 06.04.23 um 22:19 schrieb René Scharfe:\n> Since 1819ad327b (grep: fix multibyte regex handling under macOS,\n> 2022-08-26) we use the system library for all regular expression\n> matching on macOS, not just for git grep.  It supports multi-byte\n> strings and rejects invalid multi-byte characters.\n> \n> This broke all built-in userdiff word regexes in UTF-8 locales because\n> they all include such invalid bytes in expressions that are intended to\n> match multi-byte characters without explicit support for that from the\n> regex engine.\n> \n> \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\" is added to all built-in word\n> regexes to match a single non-space or multi-byte character.  The \\xNN\n> characters are invalid if interpreted as UTF-8 because they have their\n> high bit set, which indicates they are part of a multi-byte character,\n> but they are surrounded by single-byte characters.\n\nPerhpas the expression should be \"[\\xc4\\x80-\\xf7\\xbf\\xbf\\xbf]+\", i.e.,\nsequences of code points U+0080 to U+10FFFF?\n\n> \n> Replace that expression with \"|[^[:space:]]\" if the regex engine\n> supports multi-byte matching, as there is no need to have an explicit\n> range for multi-byte characters then.\n\nThis is not equivalent. The original treated a sequence of non-ASCII\ncharacters as a word. The new version treats each individual non-space\ncharacter (both ASCII and non-ASCII) as a word.\n\n> Additionally the word regex for tex contains the expression\n> \"[a-zA-Z0-9\\x80-\\xff]+\" with a similarly invalid range.  The best\n> replacement with only valid characters that I can come up with is\n> \"([a-zA-Z0-9]|[^\\x01-\\x7f])+\".  Unlike the original it matches NUL\n> characters, though.  Assuming that tex files usually don't contain NUL\n> this should be acceptable.\n\nThis is acceptable, of course. The replacement range looks sensible.\n\n-- Hannes\n\n"},{"id":"474991","messageId":"39eb2a9f-83e0-449e-1157-152c43d49b48@web.de","threadId":"59497","inReplyTo":"7fe0aa93-a764-66b0-5015-2f5fbd3901ab@kdbg.org","subject":"Re: [PATCH] userdiff: support regexec(3) with multi-byte support","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2023-04-07T07:49:10Z","receivedAt":"2023-04-07T07:49:33Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 07.04.23 um 00:35 schrieb Johannes Sixt:\n> Am 06.04.23 um 22:19 schrieb René Scharfe:\n>> Since 1819ad327b (grep: fix multibyte regex handling under macOS,\n>> 2022-08-26) we use the system library for all regular expression\n>> matching on macOS, not just for git grep.  It supports multi-byte\n>> strings and rejects invalid multi-byte characters.\n>>\n>> This broke all built-in userdiff word regexes in UTF-8 locales because\n>> they all include such invalid bytes in expressions that are intended to\n>> match multi-byte characters without explicit support for that from the\n>> regex engine.\n>>\n>> \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\" is added to all built-in word\n>> regexes to match a single non-space or multi-byte character.  The \\xNN\n>> characters are invalid if interpreted as UTF-8 because they have their\n>> high bit set, which indicates they are part of a multi-byte character,\n>> but they are surrounded by single-byte characters.\n>\n> Perhpas the expression should be \"[\\xc4\\x80-\\xf7\\xbf\\xbf\\xbf]+\", i.e.,\n> sequences of code points U+0080 to U+10FFFF?\n\nregcomp(3) on macOS doesn't like it:\n\nfatal: invalid regular expression: [a-zA-Z_][a-zA-Z0-9_]*|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*|0[xXbB][0-9a-fA-F]+[lLuU]*|\\.[0-9][0-9]*([Ee][-+]?[0-9]+)?[fFlL]?|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*|<=>|[^[:space:]]|[Ā-????]\n\nLooks like it objects to U+10FFFF here; \"[\\xc4\\x80-\\xf3\\xa0\\x80\\x80]\" is\naccepted for example.\n\n\\xc4\\x80 is U+0100, by the way; U+0080 would be \\xc2\\x80.  And\nregcomp(3) doesn't like that either (\"[\\xc2\\x80-\\xf3\\xa0\\x80\\x80]\"):\n\nfatal: invalid regular expression: [a-zA-Z_][a-zA-Z0-9_]*|[0-9][0-9.]*([Ee][-+]?[0-9]+)?[fFlLuU]*|0[xXbB][0-9a-fA-F]+[lLuU]*|\\.[0-9][0-9]*([Ee][-+]?[0-9]+)?[fFlL]?|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*|<=>|[^[:space:]]|[<U+0080>-󠀀]\n\n>> Replace that expression with \"|[^[:space:]]\" if the regex engine\n>> supports multi-byte matching, as there is no need to have an explicit\n>> range for multi-byte characters then.\n>\n> This is not equivalent. The original treated a sequence of non-ASCII\n> characters as a word. The new version treats each individual non-space\n> character (both ASCII and non-ASCII) as a word.\n\nI assume you mean \"The original treated [a single non-space as well as]\na sequence of non-ASCII characters [making up a single multi-byte\ncharacter] as a word.\".  That works as intended by 664d44ee7f (userdiff:\nsimplify word-diff safeguard, 2011-01-11).\n\nThe new one doesn't match multi-byte whitespace anymore.  What other\ndifferences do they have?\n\nRené\n"},{"id":"474992","messageId":"a548ca26-2766-1a95-4928-c22063d5d890@kdbg.org","threadId":"59497","inReplyTo":"39eb2a9f-83e0-449e-1157-152c43d49b48@web.de","subject":"Re: [PATCH] userdiff: support regexec(3) with multi-byte support","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2023-04-07T10:56:00Z","receivedAt":"2023-04-07T10:57:06Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 07.04.23 um 09:49 schrieb René Scharfe:\n> Am 07.04.23 um 00:35 schrieb Johannes Sixt:\n>> This is not equivalent. The original treated a sequence of non-ASCII\n>> characters as a word. The new version treats each individual non-space\n>> character (both ASCII and non-ASCII) as a word.\n> \n> I assume you mean \"The original treated [a single non-space as well as]\n> a sequence of non-ASCII characters [making up a single multi-byte\n> character] as a word.\".  That works as intended by 664d44ee7f (userdiff:\n> simplify word-diff safeguard, 2011-01-11).\n\nI misread the original RE. I thought it would lump multiple multi-byte\ncharacters together into one word, but it does not; sorry for that. It\nlooks like your suggested replacement is behaviorally identical to the\noriginal after all, except perhaps for this one:\n\n> The new one doesn't match multi-byte whitespace anymore.\n\nbut I did not find a reference that confirms it. I don't think we need\nto bend over backwards to keep this compatibility, though.\n\n-- Hannes\n\n"},{"id":"475002","messageId":"CALnO6CDgs+of5KCRRwpmzEoHcqZ4udbHVhNrd63q4fFh_5TwHg@mail.gmail.com","threadId":"59497","inReplyTo":"7327ac06-d5da-ec53-543e-78e7729e78bb@web.de","subject":"Re: [PATCH] userdiff: support regexec(3) with multi-byte support","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-04-07T14:41:07Z","receivedAt":"2023-04-07T14:41:24Z","isPatch":true,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Thu, Apr 6, 2023 at 4:19 PM René Scharfe <l.s.r@web.de> wrote:\n>\n> Since 1819ad327b (grep: fix multibyte regex handling under macOS,\n> 2022-08-26) we use the system library for all regular expression\n> matching on macOS, not just for git grep.  It supports multi-byte\n> strings and rejects invalid multi-byte characters.\n>\n> This broke all built-in userdiff word regexes in UTF-8 locales because\n> they all include such invalid bytes in expressions that are intended to\n> match multi-byte characters without explicit support for that from the\n> regex engine.\n>\n> \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\" is added to all built-in word\n> regexes to match a single non-space or multi-byte character.  The \\xNN\n> characters are invalid if interpreted as UTF-8 because they have their\n> high bit set, which indicates they are part of a multi-byte character,\n> but they are surrounded by single-byte characters.\n>\n> Replace that expression with \"|[^[:space:]]\" if the regex engine\n> supports multi-byte matching, as there is no need to have an explicit\n> range for multi-byte characters then.  Check for that capability at\n> runtime, because it depends on the locale and thus on environment\n> variables.  Construct the full replacement expression at build time\n> and just switch it in if necessary to avoid string manipulation and\n> allocations at runtime.\n>\n> Additionally the word regex for tex contains the expression\n> \"[a-zA-Z0-9\\x80-\\xff]+\" with a similarly invalid range.  The best\n> replacement with only valid characters that I can come up with is\n> \"([a-zA-Z0-9]|[^\\x01-\\x7f])+\".  Unlike the original it matches NUL\n> characters, though.  Assuming that tex files usually don't contain NUL\n> this should be acceptable.\n>\n> Reported-by: D. Ben Knoble <ben.knoble@gmail.com>\n> Reported-by: Eric Sunshine <sunshine@sunshineco.com>\n> Helped-by: Junio C Hamano <gitster@pobox.com>\n> Signed-off-by: René Scharfe <l.s.r@web.de>\n\nI tested the patch locally on top of ae73b2c8f1 and it solved my\nproblem. Seems like there's still some further discussion, though.\n"},{"id":"475005","messageId":"xmqqedovlly3.fsf@gitster.g","threadId":"59497","inReplyTo":"CALnO6CDgs+of5KCRRwpmzEoHcqZ4udbHVhNrd63q4fFh_5TwHg@mail.gmail.com","subject":"Re: [PATCH] userdiff: support regexec(3) with multi-byte support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-04-07T16:02:44Z","receivedAt":"2023-04-07T16:02:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"D. Ben Knoble\" <ben.knoble@gmail.com> writes:\n\n> On Thu, Apr 6, 2023 at 4:19 PM René Scharfe <l.s.r@web.de> wrote:\n>>\n>> Since 1819ad327b (grep: fix multibyte regex handling under macOS,\n>> 2022-08-26) we use the system library for all regular expression\n>> matching on macOS, not just for git grep.  It supports multi-byte\n>> strings and rejects invalid multi-byte characters.\n>> ...\n>> Reported-by: D. Ben Knoble <ben.knoble@gmail.com>\n>> Reported-by: Eric Sunshine <sunshine@sunshineco.com>\n>> Helped-by: Junio C Hamano <gitster@pobox.com>\n>> Signed-off-by: René Scharfe <l.s.r@web.de>\n>\n> I tested the patch locally on top of ae73b2c8f1 and it solved my\n> problem. Seems like there's still some further discussion, though.\n\nThanks very much for reporting and testing.  Also, thanks all for\nworking on this solution.\n\nWill queue.\n\n"},{"id":"475012","messageId":"CAPig+cSmtc3pgdv5+3y1KhLymhjzLY2JsOqN2S__pXb4jGNHpA@mail.gmail.com","threadId":"59497","inReplyTo":"7327ac06-d5da-ec53-543e-78e7729e78bb@web.de","subject":"Re: [PATCH] userdiff: support regexec(3) with multi-byte support","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2023-04-07T17:23:56Z","receivedAt":"2023-04-07T17:24:11Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Apr 6, 2023 at 4:19 PM René Scharfe <l.s.r@web.de> wrote:\n> Since 1819ad327b (grep: fix multibyte regex handling under macOS,\n> 2022-08-26) we use the system library for all regular expression\n> matching on macOS, not just for git grep.  It supports multi-byte\n> strings and rejects invalid multi-byte characters.\n>\n> This broke all built-in userdiff word regexes in UTF-8 locales because\n> they all include such invalid bytes in expressions that are intended to\n> match multi-byte characters without explicit support for that from the\n> regex engine.\n>\n> \"|[^[:space:]]|[\\xc0-\\xff][\\x80-\\xbf]+\" is added to all built-in word\n> regexes to match a single non-space or multi-byte character.  The \\xNN\n> characters are invalid if interpreted as UTF-8 because they have their\n> high bit set, which indicates they are part of a multi-byte character,\n> but they are surrounded by single-byte characters.\n>\n> Replace that expression with \"|[^[:space:]]\" if the regex engine\n> supports multi-byte matching, as there is no need to have an explicit\n> range for multi-byte characters then.  Check for that capability at\n> runtime, because it depends on the locale and thus on environment\n> variables.  Construct the full replacement expression at build time\n> and just switch it in if necessary to avoid string manipulation and\n> allocations at runtime.\n>\n> Reported-by: D. Ben Knoble <ben.knoble@gmail.com>\n> Reported-by: Eric Sunshine <sunshine@sunshineco.com>\n> Helped-by: Junio C Hamano <gitster@pobox.com>\n> Signed-off-by: René Scharfe <l.s.r@web.de>\n\nThank you, René! This patch resolves the problem I was experiencing[1].\n\nI'm happy to have --color-words working again.\n\n[1]: https://lore.kernel.org/git/CAPig+cSNmws2b7f7aRA2C56kvQYG3w_g+KhYdqhtmf+XhtAMhQ@mail.gmail.com/\n"}]}