{"thread":{"id":"59175","subject":"RE: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","startedAt":"2023-02-01T15:19:43Z","lastAt":"2023-02-07T22:27:33Z","messageCount":22,"participants":["D. Ben Knoble","demerphq","Junio C Hamano","Jeff King","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"471246","messageId":"CALnO6CAZtwfGY4SYeOuKqdP9+e_0EYNf4F703DRQB7UUfd_bUg@mail.gmail.com","threadId":"59175","inReplyTo":null,"subject":"RE: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-02-01T15:18:39Z","receivedAt":"2023-02-01T15:19:43Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"I recently updated to git 2.39.1 and noticed today that `git diff\n--word-diff` fails for files with `diff=scheme`. I was able to narrow\nthe failure down to the inclusion of control characters \\xc0, \\xff,\n\\x80, \\xbf by https://github.com/git/git/blob/2fc9e9ca3c7505bc60069f11e7ef09b1aeeee473/userdiff.c#L17\nin the definition of the scheme diff pattern (really, all patterns).\n\nI suspect the commit referenced in the subject, given that it messes\nwith regex handling on macOS.\n\nRelevant environment that I can think of:\n```\n# locale\nLANG=\"fr_FR.UTF-8\"\nLC_COLLATE=\"fr_FR.UTF-8\"\nLC_CTYPE=\"fr_FR.UTF-8\"\nLC_MESSAGES=\"fr_FR.UTF-8\"\nLC_MONETARY=\"fr_FR.UTF-8\"\nLC_NUMERIC=\"fr_FR.UTF-8\"\nLC_TIME=\"fr_FR.UTF-8\"\nLC_ALL=\"fr_FR.UTF-8\"\n```\n\nI'm on macOS 11.7.\n\nFailure (using Zsh to produce the characters; I think there's a Bash\nequivalent):\n```\n# git diff --word-diff --word-diff-regex=$'[\\xc0-\\xff][\\x80-\\xbf]+'\nfatal¬†: invalid regular expression: [¿-ˇ][Ä-ø]+\n```\n(Looks like the output is a bit scrambled; here's the hexdump)\n```\n# !! |& xxd\n00000000: 6661 7461 6cc2 a03a 2069 6e76 616c 6964 fatal..: invalid\n00000010: 2072 6567 756c 6172 2065 7870 7265 7373 regular express\n00000020: 696f 6e3a 205b c02d ff5d 5b80 2dbf 5d2b ion: [.-.][.-.]+\n00000030: 0a .\n```\n\n-- \nD. Ben Knoble\n"},{"id":"471249","messageId":"CANgJU+X_e0owKC3uWPaA_gVP54syF1+MJ-cTn+fjPrNS5LDsMA@mail.gmail.com","threadId":"59175","inReplyTo":"CALnO6CAZtwfGY4SYeOuKqdP9+e_0EYNf4F703DRQB7UUfd_bUg@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2023-02-01T16:09:33Z","receivedAt":"2023-02-01T16:09:50Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"On Wed, 1 Feb 2023 at 16:25, D. Ben Knoble wrote:\n>\n> I recently updated to git 2.39.1 and noticed today that `git diff\n> --word-diff` fails for files with `diff=scheme`. I was able to narrow\n> the failure down to the inclusion of control characters \\xc0, \\xff,\n> \\x80, \\xbf by https://github.com/git/git/blob/2fc9e9ca3c7505bc60069f11e7ef09b1aeeee473/userdiff.c#L17\n> in the definition of the scheme diff pattern (really, all patterns).\n>\n> I suspect the commit referenced in the subject, given that it messes\n> with regex handling on macOS.\n>\n> Relevant environment that I can think of:\n> ```\n> # locale\n> LANG=\"fr_FR.UTF-8\"\n> LC_COLLATE=\"fr_FR.UTF-8\"\n> LC_CTYPE=\"fr_FR.UTF-8\"\n> LC_MESSAGES=\"fr_FR.UTF-8\"\n> LC_MONETARY=\"fr_FR.UTF-8\"\n> LC_NUMERIC=\"fr_FR.UTF-8\"\n> LC_TIME=\"fr_FR.UTF-8\"\n> LC_ALL=\"fr_FR.UTF-8\"\n> ```\n>\n> I'm on macOS 11.7.\n>\n> Failure (using Zsh to produce the characters; I think there's a Bash\n> equivalent):\n> ```\n> # git diff --word-diff --word-diff-regex=$'[\\xc0-\\xff][\\x80-\\xbf]+'\n> fatal¬†: invalid regular expression: [¿-ˇ][Ä-ø]+\n> ```\n\nFWIW that looks pretty weird to me, like the escapes in the charclass\nwere interpolated before being fed to the regex engine. Are you sure\nyou tested the right thing?\n\nYves\n\n\n--\nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"471250","messageId":"CALnO6CDCkuN2XU_AyO66hQSm2ztfpe8Rs_baw_J4uTQZmekREw@mail.gmail.com","threadId":"59175","inReplyTo":"CANgJU+X_e0owKC3uWPaA_gVP54syF1+MJ-cTn+fjPrNS5LDsMA@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-02-01T16:21:57Z","receivedAt":"2023-02-01T16:22:14Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Wed, Feb 1, 2023 at 11:09 AM demerphq wrote:\n> FWIW that looks pretty weird to me, like the escapes in the charclass\n> were interpolated before being fed to the regex engine. Are you sure\n> you tested the right thing?\n\nQuite sure. `git diff --word-diff` fails. This was just a smaller\nexample based on the linked C code.\n\nHere's the output of `git diff --word-diff` (verbatim and dumped):\n\n```\nfatal¬†: invalid regular expression: \\|([^\\\\]*)\\||([^][)(}{[\n])+|[^[:space:]]|[¿-ˇ][Ä-ø]+\n00000000: 6661 7461 6cc2 a03a 2069 6e76 616c 6964 fatal..: invalid\n00000010: 2072 6567 756c 6172 2065 7870 7265 7373 regular express\n00000020: 696f 6e3a 205c 7c28 5b5e 5c5c 5d2a 295c ion: \\|([^\\\\]*)\\\n00000030: 7c7c 285b 5e5d 5b29 287d 7b5b 2009 5d29 ||([^][)(}{[ .])\n00000040: 2b7c 5b5e 5b3a 7370 6163 653a 5d5d 7c5b +|[^[:space:]]|[\n00000050: c02d ff5d 5b80 2dbf 5d2b 0a .-.][.-.]+.\n```\n\n-- \nD. Ben Knoble\n"},{"id":"471258","messageId":"CANgJU+WZR56xG+KL3P053aD_qTh+rBhZ01mqNajg2qRt_+RNMA@mail.gmail.com","threadId":"59175","inReplyTo":"CALnO6CDCkuN2XU_AyO66hQSm2ztfpe8Rs_baw_J4uTQZmekREw@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2023-02-01T18:23:28Z","receivedAt":"2023-02-01T18:23:44Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"On Wed, 1 Feb 2023 at 17:22, D. Ben Knoble wrote:\n>\n> On Wed, Feb 1, 2023 at 11:09 AM demerphq wrote:\n> > FWIW that looks pretty weird to me, like the escapes in the charclass\n> > were interpolated before being fed to the regex engine. Are you sure\n> > you tested the right thing?\n>\n> Quite sure. `git diff --word-diff` fails. This was just a smaller\n> example based on the linked C code.\n>\n> Here's the output of `git diff --word-diff` (verbatim and dumped):\n>\n> ```\n> fatal¬†: invalid regular expression: \\|([^\\\\]*)\\||([^][)(}{[\n> ])+|[^[:space:]]|[¿-ˇ][Ä-ø]+\n> 00000000: 6661 7461 6cc2 a03a 2069 6e76 616c 6964 fatal..: invalid\n> 00000010: 2072 6567 756c 6172 2065 7870 7265 7373 regular express\n> 00000020: 696f 6e3a 205c 7c28 5b5e 5c5c 5d2a 295c ion: \\|([^\\\\]*)\\\n> 00000030: 7c7c 285b 5e5d 5b29 287d 7b5b 2009 5d29 ||([^][)(}{[ .])\n> 00000040: 2b7c 5b5e 5b3a 7370 6163 653a 5d5d 7c5b +|[^[:space:]]|[\n> 00000050: c02d ff5d 5b80 2dbf 5d2b 0a .-.][.-.]+.\n> ```\n\nInteresting. The regex engine seems to be interpolating the \\xC0 in\nsuch a way you arent seeing the real pattern. In the Perl regex engine\nI'd call that a bug (it used to do the same thing before we fixed it\nyears ago[1]). FWIW, this is a valid regex in Perl so i dont think the\npattern is at fault, its something else. I saw some discussion\nrecently that the mac regex engine doesn't play nicely in certain\nways, but i dont recollect the details.\n\nSorry i can't help more. Try searching for EXTENDED and regex and mac.\nMaybe you can find the mail I mean.\n\ncheers,\nyves\n[1] I am one of the maintainers of the perl regex engine.\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"471260","messageId":"xmqqsffpz05o.fsf@gitster.g","threadId":"59175","inReplyTo":"CANgJU+WZR56xG+KL3P053aD_qTh+rBhZ01mqNajg2qRt_+RNMA@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-01T18:54:43Z","receivedAt":"2023-02-01T18:54:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"demerphq writes:\n\n> ... I saw some discussion\n> recently that the mac regex engine doesn't play nicely in certain\n> ways, but i dont recollect the details.\n\nThere was a discussion on BRE having and not having GNU-like\nextension on macOS, in this thread\n\n https://lore.kernel.org/git/26a0d4ca-3d97-ace4-1a1f-92b1ee6715a6@web.de/\n\nThe patch we ended up using avoids touching the behaviour with ERE,\nas REG_ENHANCED on macOS affects REG_EXTENDED, but the issue we were\nlooking at in the thread was about BRE.\n\n"},{"id":"471266","messageId":"CALnO6CAZ_RMirOwQqJyqJrq2dY1w09eV1h=0JFosYSb7XhyQVQ@mail.gmail.com","threadId":"59175","inReplyTo":"xmqqsffpz05o.fsf@gitster.g","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-02-01T21:33:52Z","receivedAt":"2023-02-01T21:34:08Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Wed, Feb 1, 2023 at 1:54 PM Junio C Hamano wrote:\n> There was a discussion on BRE having and not having GNU-like\n> extension on macOS, in this thread\n>\n> https://lore.kernel.org/git/26a0d4ca-3d97-ace4-1a1f-92b1ee6715a6@web.de/\n>\n> The patch we ended up using avoids touching the behaviour with ERE,\n> as REG_ENHANCED on macOS affects REG_EXTENDED, but the issue we were\n> looking at in the thread was about BRE.\n>\n\nAre you suggesting that I try with `-P` and/or `-E`?\n\nCC'ing the original author of\nhttps://github.com/git/git/commit/1819ad327b7a1f19540a819813b70a0e8a7f798f,\nwhich I suspect is the most relevant.\n\n-- \nD. Ben Knoble\n"},{"id":"471267","messageId":"CALnO6CA8ZYfQ7pGMAgupJMhwReZVRzVw7KkfqwAF6i+dCMyzJw@mail.gmail.com","threadId":"59175","inReplyTo":"CALnO6CAZ_RMirOwQqJyqJrq2dY1w09eV1h=0JFosYSb7XhyQVQ@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-02-01T21:34:49Z","receivedAt":"2023-02-01T21:35:06Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Wed, Feb 1, 2023 at 4:33 PM D. Ben Knoble wrote:\n> Are you suggesting that I try with `-P` and/or `-E`?\n\nEr, obviously these don't exist for git-diff.\n\n\n\n-- \nD. Ben Knoble\n"},{"id":"471270","messageId":"xmqq7cx1yqu9.fsf@gitster.g","threadId":"59175","inReplyTo":"CALnO6CAZ_RMirOwQqJyqJrq2dY1w09eV1h=0JFosYSb7XhyQVQ@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-01T22:15:58Z","receivedAt":"2023-02-01T22:16:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"D. Ben Knoble\" writes:\n\n> On Wed, Feb 1, 2023 at 1:54 PM Junio C Hamano wrote:\n>> There was a discussion on BRE having and not having GNU-like\n>> extension on macOS, in this thread\n>>\n>> https://lore.kernel.org/git/26a0d4ca-3d97-ace4-1a1f-92b1ee6715a6@web.de/\n>>\n>> The patch we ended up using avoids touching the behaviour with ERE,\n>> as REG_ENHANCED on macOS affects REG_EXTENDED, but the issue we were\n>> looking at in the thread was about BRE.\n>>\n>\n> Are you suggesting that I try with `-P` and/or `-E`?\n\nNo. I was just giving a concrete URL for what I think demerphq was\nreferring to as \"saw some discussion recently\".\n"},{"id":"471278","messageId":"Y9rv29c0dYUAYx8B@coredump.intra.peff.net","threadId":"59175","inReplyTo":"CANgJU+X_e0owKC3uWPaA_gVP54syF1+MJ-cTn+fjPrNS5LDsMA@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-01T23:03:55Z","receivedAt":"2023-02-01T23:04:00Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 01, 2023 at 05:09:33PM +0100, demerphq wrote:\n\n> > Failure (using Zsh to produce the characters; I think there's a Bash\n> > equivalent):\n> > ```\n> > # git diff --word-diff --word-diff-regex=$'[\\xc0-\\xff][\\x80-\\xbf]+'\n> > fatal¬†: invalid regular expression: [¿-ˇ][Ä-ø]+\n> > ```\n> \n> FWIW that looks pretty weird to me, like the escapes in the charclass\n> were interpolated before being fed to the regex engine. Are you sure\n> you tested the right thing?\n\nI think the point is that he is feeding a raw \\xc0 byte (not the escape\nsequence) to the regex engine, which is bogus UTF8. And the internal\nuserdiff drivers do the same thing. They contain \"[\\xc0-\\xff]\", and\nthose \"\\x\" will be interpolated by the compiler into their actual bytes.\n\nSo the regex engine is complaining that it is getting bytes with high\nbits set, but that are not part of a multi-byte character. I.e., it is\nnot happy to do bytewise matching, but really wants valid UTF8 in the\nexpression.\n\nglibc's regex engine seems OK with this. Try:\n\n git grep $'[\\xc0-\\xff]'\n\nin git.git, and it will find lots of multi-byte characters. But pcre,\nfor example, is not:\n\n $ git grep -P $'[\\xc0-\\xff]'\n fatal: command line, '[-]': UTF-8 error: byte 2 top bits not 0x80\n\nThere you really want to feed the literal escapes (obviously dropping\nthe '$ shell interpolation is a better solution, but for the sake of\nillustration):\n\n git grep -P $'[\\\\xc0-\\\\xff]'\n\nBut I don't think we can rely on the libc BRE supporting \"\\x\" in\ncharacter classes. Glibc certainly doesn't. I'm not sure what the\nportable solution is.\n\n-Peff\n"},{"id":"471368","messageId":"CANgJU+XNLqf0E2+YC8yxtRPVh=mevc3P0eeye2_nx=ULB2iVWw@mail.gmail.com","threadId":"59175","inReplyTo":"Y9rv29c0dYUAYx8B@coredump.intra.peff.net","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2023-02-02T16:22:37Z","receivedAt":"2023-02-02T16:23:26Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"On Thu, 2 Feb 2023 at 00:03, Jeff King wrote:\n>\n> On Wed, Feb 01, 2023 at 05:09:33PM +0100, demerphq wrote:\n>\n> > > Failure (using Zsh to produce the characters; I think there's a Bash\n> > > equivalent):\n> > > ```\n> > > # git diff --word-diff --word-diff-regex=$'[\\xc0-\\xff][\\x80-\\xbf]+'\n> > > fatal¬†: invalid regular expression: [¿-ˇ][Ä-ø]+\n> > > ```\n> >\n> > FWIW that looks pretty weird to me, like the escapes in the charclass\n> > were interpolated before being fed to the regex engine. Are you sure\n> > you tested the right thing?\n>\n> I think the point is that he is feeding a raw \\xc0 byte (not the escape\n> sequence) to the regex engine, which is bogus UTF8. And the internal\n> userdiff drivers do the same thing. They contain \"[\\xc0-\\xff]\", and\n> those \"\\x\" will be interpolated by the compiler into their actual bytes.\n\nThanks, that was the bit that threw me off. I had completely forgotten\nthat C supports \\x escapes :-(. The Perl internals and regex engine is\nwhere I do most of my C hacking and it uses octal exclusively AFAIK.\n(I guess it uses octal because of the \"where does the escape end\"\nproblem that C seems to have with hex escapes). So I had assumed that\nsomething else, or the regex engine itself was interpolating them. I\nappreciate that you took the time to set me straight.\n\n> So the regex engine is complaining that it is getting bytes with high\n> bits set, but that are not part of a multi-byte character. I.e., it is\n> not happy to do bytewise matching, but really wants valid UTF8 in the\n> expression.\n\nYeah, that was my first thought too, but as I said above the hex\nescapes threw me off.\n\n> glibc's regex engine seems OK with this. Try:\n>\n> git grep $'[\\xc0-\\xff]'\n>\n> in git.git, and it will find lots of multi-byte characters. But pcre,\n> for example, is not:\n>\n> $ git grep -P $'[\\xc0-\\xff]'\n> fatal: command line, '[-]': UTF-8 error: byte 2 top bits not 0x80\n\nI expect that has something to do with how you are configuring PCRE,\nand that with a slightly different config it would be fine with this.\n\n> There you really want to feed the literal escapes (obviously dropping\n> the '$ shell interpolation is a better solution, but for the sake of\n> illustration):\n>\n> git grep -P $'[\\\\xc0-\\\\xff]'\n>\n> But I don't think we can rely on the libc BRE supporting \"\\x\" in\n> character classes. Glibc certainly doesn't. I'm not sure what the\n> portable solution is.\n\nI've been lurking watching some of the regex discussion on the list\nand personally I think it is asking for trouble to use \"whatever regex\nengine is traditional in a given environment\" instead of just choosing\na good open source engine and using it consistently everywhere. I\ndon't really buy the arguments I have seen to justify a policy of \"use\nthe standard library version\"; regex engines vary widely in\nperformance and implementation and feature set, and even the really\ngood ones do not entirely agree on every semantic[1], so if you don't\nstandardize you will be forever dealing with bugs related to those\ndifferences.\n\nI think the git project should choose the feature set[2] it thinks are\nimportant, and then choose a regex engine that provides those features\nand is well supported, and then use it consistently everywhere that\ngit needs to do regex based matching. Anything else is asking for\ntrouble at some level or another.\n\nCheers,\nyves\n[1] Leaving aside advanced features, even something as simple as\nalternation can vary by engine. Consider \"foo\"=~/f|fo|foo/. Some\nregex engines will match \"foo\", and some will match \"f\", depending on\nwhether they implement \"longest match\" (as most NFA/DFA engines do),\nor if they implement \"leftmost longest match\" (as Perl and other\nbacktracking engines tend to do).\n\n[2] Personally I think that features like recursive patterns, named\ncapture, negative and positive lookahead and lookbehind and branch\nreset are so useful that it would be wise to choose an engine that\nsupports them, but some might argue for other priorities, performance\nbeing a likely candidate.\n\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"471391","messageId":"CALnO6CA3LL2TbMyvVsgeNgGHr9tGq4-FYR0-RMyJJiMvV3P91w@mail.gmail.com","threadId":"59175","inReplyTo":"Y9rv29c0dYUAYx8B@coredump.intra.peff.net","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-02-02T20:47:28Z","receivedAt":"2023-02-02T20:48:21Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Wed, Feb 1, 2023 at 6:03 PM Jeff King wrote:\n> So the regex engine is complaining that it is getting bytes with high\n> bits set, but that are not part of a multi-byte character. I.e., it is\n> not happy to do bytewise matching, but really wants valid UTF8 in the\n> expression.\n\nI did manage to find that the call to regcomp in diff.c's\ninit_diff_words_data (line 2212 in v2.39.1) is what crashes; I could\nnot step into it with gdb, however.\n\nFurther, the following C program compiles without warnings (except for\nthe unused main parameters):\n```\n#include \n#include \n#include \n#include \n\nint main(int argc, char **argv) {\n regex_t re;\n int ret = regcomp(&re, \"[\\xc0-\\xff][\\x80-\\xbf]+\", REG_EXTENDED |\nREG_NEWLINE);\n /* assert(ret != 0); */\n size_t errbuf_size = regerror(ret, &re, NULL, 0);\n char errbuf[errbuf_size];\n regerror(ret, &re, errbuf, errbuf_size);\n printf(\"%s\\n\", errbuf);\n}\n```\n\n```\n# CFLAGS='-Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes\n-Wold-style-definition -Wshadow -Wpointer-arith -Wcast-qual -pedantic\n-std=c11'\n# cc $CFLAGS regtest.c -o regtest && ./regtest\n*** unknown regexp error code ***\n```\n(the assertion fails because regcomp succeeds!)\n\nSo I can neither find out what's to blame nor what to fix. Here are\nthe linked libraries on macOS (IIUC):\n```\n# otool -L regtest\nregtest:\n/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current\nversion 1311.0.0)\n# otool -L ./git-diff # from v2.39.1 source build today\n./git-diff:\n/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices\n(compatibility version 1.0.0, current version 1141.1.0)\n/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)\n/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)\n/usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version\n12.0.0, current version 12.0.0)\n/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current\nversion 1311.0.0)\n/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation\n(compatibility version 150.0.0, current version 1856.105.0)\n```\n\n-- \nD. Ben Knoble\n"},{"id":"471392","messageId":"CALnO6CD_x5Dhh-x8wq6BGT10Qv7iPzsjZyD8Pun0GCKwKKRczw@mail.gmail.com","threadId":"59175","inReplyTo":"CANgJU+XNLqf0E2+YC8yxtRPVh=mevc3P0eeye2_nx=ULB2iVWw@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-02-02T20:49:45Z","receivedAt":"2023-02-02T20:50:07Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Thu, Feb 2, 2023 at 11:22 AM demerphq wrote:\n> I've been lurking watching some of the regex discussion on the list\n> and personally I think it is asking for trouble to use \"whatever regex\n> engine is traditional in a given environment\" instead of just choosing\n> a good open source engine and using it consistently everywhere. I\n> don't really buy the arguments I have seen to justify a policy of \"use\n> the standard library version\"; regex engines vary widely in\n> performance and implementation and feature set, and even the really\n> good ones do not entirely agree on every semantic[1], so if you don't\n> standardize you will be forever dealing with bugs related to those\n> differences.\n\nMy understanding is that the patch suspected in the subject and linked\nseveral times in this thread _does_ use the native Darwin regex\nlibrary which causes the problem, but I can't reproduce it in a\nsmaller program (see previous mail).\n\n\n-- \nD. Ben Knoble\n"},{"id":"471442","messageId":"Y908f2qaxKeljtj+@coredump.intra.peff.net","threadId":"59175","inReplyTo":"CALnO6CA3LL2TbMyvVsgeNgGHr9tGq4-FYR0-RMyJJiMvV3P91w@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-03T16:55:27Z","receivedAt":"2023-02-03T16:55:31Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 02, 2023 at 03:47:28PM -0500, D. Ben Knoble wrote:\n\n> Further, the following C program compiles without warnings (except for\n> the unused main parameters):\n> ```\n> #include \n> #include \n> #include \n> #include \n> \n> int main(int argc, char **argv) {\n> regex_t re;\n> int ret = regcomp(&re, \"[\\xc0-\\xff][\\x80-\\xbf]+\", REG_EXTENDED |\n> REG_NEWLINE);\n> /* assert(ret != 0); */\n> size_t errbuf_size = regerror(ret, &re, NULL, 0);\n> char errbuf[errbuf_size];\n> regerror(ret, &re, errbuf, errbuf_size);\n> printf(\"%s\\n\", errbuf);\n> }\n> ```\n\nJust a guess, but does calling:\n\n setlocale(LC_CTYPE, \"\");\n\nat the start of the program change things (you'll probably need to also\ninclude locale.h)? I don't have a macos system to test on.\n\n-Peff\n"},{"id":"471443","messageId":"Y9098dyaTtiNk506@coredump.intra.peff.net","threadId":"59175","inReplyTo":"CANgJU+XNLqf0E2+YC8yxtRPVh=mevc3P0eeye2_nx=ULB2iVWw@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-03T17:01:37Z","receivedAt":"2023-02-03T17:01:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 02, 2023 at 05:22:37PM +0100, demerphq wrote:\n\n> I've been lurking watching some of the regex discussion on the list\n> and personally I think it is asking for trouble to use \"whatever regex\n> engine is traditional in a given environment\" instead of just choosing\n> a good open source engine and using it consistently everywhere. I\n> don't really buy the arguments I have seen to justify a policy of \"use\n> the standard library version\"; regex engines vary widely in\n> performance and implementation and feature set, and even the really\n> good ones do not entirely agree on every semantic[1], so if you don't\n> standardize you will be forever dealing with bugs related to those\n> differences.\n\nI think this is a perennial question for portable software: is it better\nto be consistent across platforms (by shipping our own regex engine), or\nconsistent with other programs on the same platform (by using the system\nregex).\n\nI don't have a strong opinion either way. The main concern I'd have is\nhandling dependencies. I like pcre a lot, but I'm not sure that I would\nwant building Git to require pcre on every platform. If there's an\nengine we can ship as a vendored dependency that builds everywhere, that\nhelps. We have the engine imported from gawk in compat/regex. That\n_probably_ builds everywhere (though we don't really know, because any\nplatform that doesn't set NO_REGEX has been happily using the system\nroutines). But it also may not be the best choice; avoiding its\nmulti-byte handling was the reason behind 1819ad327 in the first place.\n\n> I think the git project should choose the feature set[2] it thinks are\n> important, and then choose a regex engine that provides those features\n> and is well supported, and then use it consistently everywhere that\n> git needs to do regex based matching. Anything else is asking for\n> trouble at some level or another.\n\nIMHO the biggest issue here is that the built-in userdiff regexes are\ndoing something a bit questionable, which is embedding high-bit\ncharacters directly into the regex. If we can avoid that, then having\nconsistency in multi-byte handling across platforms becomes a lot less\nimportant.\n\n-Peff\n"},{"id":"471444","messageId":"CALnO6CAfKA2atXwHXGJxnBGJ46EMbjGmFU54mb4FJL2O8ceXyQ@mail.gmail.com","threadId":"59175","inReplyTo":"Y908f2qaxKeljtj+@coredump.intra.peff.net","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-02-03T17:06:55Z","receivedAt":"2023-02-03T17:07:15Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Fri, Feb 3, 2023 at 11:55 AM Jeff King wrote:\n> Just a guess, but does calling:\n>\n> setlocale(LC_CTYPE, \"\");\n>\n> at the start of the program change things (you'll probably need to also\n> include locale.h)?\n\nIndeed, the new output is\n\n illegal byte sequence\n\nFor the following program\n\n #include \n #include \n #include \n #include \n #include \n\n int main(int argc, char **argv) {\n char *loc = setlocale(LC_CTYPE, \"\");\n assert (loc != NULL);\n regex_t re;\n int ret = regcomp(&re, \"[\\xc0-\\xff][\\x80-\\xbf]+\", REG_EXTENDED\n| REG_NEWLINE);\n /* assert(ret != 0); */\n size_t errbuf_size = regerror(ret, &re, NULL, 0);\n char errbuf[errbuf_size];\n regerror(ret, &re, errbuf, errbuf_size);\n printf(\"%s\\n\", errbuf);\n }\n\nMy own locale output, for completion's sake:\n\n LANG=\"fr_FR.UTF-8\"\n LC_COLLATE=\"fr_FR.UTF-8\"\n LC_CTYPE=\"fr_FR.UTF-8\"\n LC_MESSAGES=\"fr_FR.UTF-8\"\n LC_MONETARY=\"fr_FR.UTF-8\"\n LC_NUMERIC=\"fr_FR.UTF-8\"\n LC_TIME=\"fr_FR.UTF-8\"\n LC_ALL=\"fr_FR.UTF-8\"\n\n\n-- \nD. Ben Knoble\n"},{"id":"471464","messageId":"230203.86357mbe8x.gmgdl@evledraar.gmail.com","threadId":"59175","inReplyTo":"Y9098dyaTtiNk506@coredump.intra.peff.net","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-03T21:56:53Z","receivedAt":"2023-02-03T22:01:12Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Feb 03 2023, Jeff King wrote:\n\n> On Thu, Feb 02, 2023 at 05:22:37PM +0100, demerphq wrote:\n>\n>> I've been lurking watching some of the regex discussion on the list\n>> and personally I think it is asking for trouble to use \"whatever regex\n>> engine is traditional in a given environment\" instead of just choosing\n>> a good open source engine and using it consistently everywhere. I\n>> don't really buy the arguments I have seen to justify a policy of \"use\n>> the standard library version\"; regex engines vary widely in\n>> performance and implementation and feature set, and even the really\n>> good ones do not entirely agree on every semantic[1], so if you don't\n>> standardize you will be forever dealing with bugs related to those\n>> differences.\n>\n> I think this is a perennial question for portable software: is it better\n> to be consistent across platforms (by shipping our own regex engine), or\n> consistent with other programs on the same platform (by using the system\n> regex).\n\n*nod*\n\n> I don't have a strong opinion either way. The main concern I'd have is\n> handling dependencies. I like pcre a lot, but I'm not sure that I would\n> want building Git to require pcre on every platform. If there's an\n> engine we can ship as a vendored dependency that builds everywhere, that\n> helps.\n\nWe can just make that fallback engine be PCRE. I submitted patches a\nwhile ago to include a minimal version of it in compat/pcre, as we seem\nto have some allergy to external dependencies:\nhttps://lore.kernel.org/git/20170511175115.648-1-avarab@gmail.com/\n\nIt's ~80k lines instead of compat/regex's ~15k, but it's actually\nmaintained, and would be much easier to upgrade.\n"},{"id":"471479","messageId":"Y94+wVcV8OAFOIuK@coredump.intra.peff.net","threadId":"59175","inReplyTo":"230203.86357mbe8x.gmgdl@evledraar.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-04T11:17:21Z","receivedAt":"2023-02-04T11:17:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 03, 2023 at 10:56:53PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> > I don't have a strong opinion either way. The main concern I'd have is\n> > handling dependencies. I like pcre a lot, but I'm not sure that I would\n> > want building Git to require pcre on every platform. If there's an\n> > engine we can ship as a vendored dependency that builds everywhere, that\n> > helps.\n> \n> We can just make that fallback engine be PCRE. I submitted patches a\n> while ago to include a minimal version of it in compat/pcre, as we seem\n> to have some allergy to external dependencies:\n> https://lore.kernel.org/git/20170511175115.648-1-avarab@gmail.com/\n> \n> It's ~80k lines instead of compat/regex's ~15k, but it's actually\n> maintained, and would be much easier to upgrade.\n\nI'm OK with that if we really think that libpcre will build without\nproblems on every platform that Git does. I don't know if we have any\ndata there. Obviously libpcre builds lots of places, but will we have\nproblems on obscure platforms like NonStop? Part of me wants to not\ncare, but if the value here is saying \"the regex engine is always going\nto be X\", then there is not much point in saying \"the regex engine is\nusually X, but you can't rely on it because sometimes it's not\".\n\n\"Usually\" is enough for helping users quality of life (if we help 99% of\nusers, that is good). It isn't enough for making assumptions in the code\n(like using constructs in userdiff regexes that would break horribly on\nthe other 1% of platforms).\n\n-Peff\n"},{"id":"471481","messageId":"CANgJU+UJ-p54MDmQWYTEsdh_ss-q8TUEcqHFm568Lpfsy2dU+w@mail.gmail.com","threadId":"59175","inReplyTo":"Y9098dyaTtiNk506@coredump.intra.peff.net","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"demerphq","fromEmail":"demerphq@gmail.com","sentAt":"2023-02-04T11:32:09Z","receivedAt":"2023-02-04T11:32:25Z","isPatch":false,"sender":{"key":"demerphq@gmail.com","avatar":null},"body":"On Fri, 3 Feb 2023 at 18:01, Jeff King wrote:\n>\n> On Thu, Feb 02, 2023 at 05:22:37PM +0100, demerphq wrote:\n>\n> > I've been lurking watching some of the regex discussion on the list\n> > and personally I think it is asking for trouble to use \"whatever regex\n> > engine is traditional in a given environment\" instead of just choosing\n> > a good open source engine and using it consistently everywhere. I\n> > don't really buy the arguments I have seen to justify a policy of \"use\n> > the standard library version\"; regex engines vary widely in\n> > performance and implementation and feature set, and even the really\n> > good ones do not entirely agree on every semantic[1], so if you don't\n> > standardize you will be forever dealing with bugs related to those\n> > differences.\n>\n> I think this is a perennial question for portable software: is it better\n> to be consistent across platforms (by shipping our own regex engine), or\n> consistent with other programs on the same platform (by using the system\n> regex).\n\nPersonally I think that while this seems to be an impartial reading of\nthe question at hand I think it frames the debate in a way that has\nthe potential to bias[1] the discussion in favour of a particular\npolicy outcome[2]. It implies that all other things are equal between\nthe two options presented, and frames the question as something that\ncomes down to personal preference between one form of consistency and\nanother.\n\nBut I am not sure that all other things are equal here, at least as\nfar as regex engines go. I think there is evidence that suggests that\ndepending on the system regex engine introduces long term recurring\ncosts that would not be incurred if git chose to link to a specific\nlibrary everywhere. I think it ignores the implications on the wider\necosystem and toolchains. For instance if the behavior of git grep\ndiffers by platform then scripts that might want to use git grep to\nautomate git become less portable, or more expensive to maintain to\nwork around the inconsistencies. It overlooks the costs of training\nhumans (arguably low) versus the costs of training computers (arguably\nhigh). It also assumes that being consistent with other programs on\nthe same platform is inherently beneficial, when that doesn't seem to\nbe clearly established[2]. It also assumes that there are only two\noptions. Maybe there are more. Maybe there is a third or fourth option\nas well. One would be to use a specific library for internal regexes,\nand let the command line use the system library by default. Another\nwould be to make the default engine be one that ships with git, and\nthat users that want \"platform compatibility\" should use an option to\nget it, much as they would with -P to enable PCRE. You mentioned that\nthere is already such an engine, but it isn't well tested. Maybe that\nshould be changed.\n\nI think that if you look at other broadly ported projects there is\nevidence that owning your own dependencies makes a project easier and\ncheaper to port to new platforms. The more platforms you target the\nmore room there is for inconsistencies and the more costs there will\nbe to deal with them. If portability of git is a goal and minimizing\nthe cost of doing so is a secondary goal then I would say that using a\nspecific library will make achieving that goal easier and lower cost\nthan depending on the system libraries. There would be a high initial\ncost to do the switch, and then a low cost in the long run. As far as\nI know Perl is more broadly ported than git, based on the fact that\nwhen we migrated to git a number of the Perl maintainers could not use\ngit on their platforms (Vax comes to mind), and Perl definitely adopts\nthe view that it is better to own your dependencies, and wraps and\nhides system inconsistencies as much as possible to make porting as\neasy as possible. So that is one precedent to consider. No doubt there\nare many other long running projects with precedents in this area.\nWhat does Vi or Vim do? What does Emacs do? Etc.\n\nAnyway, my opinion on these things doesn't matter that much, I am just\na git user who happens to have a passion for regular expressions and\nregular expression engines and I am happily served by the PCRE support\nin my git build. So I can't answer these questions for the project.\nBut I do think the questions that need to be answered are more complex\nand nuanced than deciding which of two forms of consistency is more\nimportant.\n\nThanks for hearing me out.\n\ncheers,\nyves\n[1] PS I do not mean to imply that you are *intending* to bias the\ndiscussion. I think your writing and measured approach indicates\nstrongly that you intend to be impartial, but nevertheless I think\nthis way of framing the question does bias the debate.\n\n[2] My thinking about the framing of this question is probably pretty\nstrongly influenced by a recently released report from the BBC on\njournalistic bias in presenting questions of economic debate. The\nreport presents some interesting perspectives on how framing questions\nand data can bias the discussion even though the person presenting the\ndata or asking the question actually intended to be impartial.\nRegardless of your position on what regex engine git should use I\nthink it is a good read. Especially in this day and age of austerity\npolitics and debt-ceiling debates around the world.\nhttps://www.bbc.co.uk/aboutthebbc/documents/thematic-review-taxation-public-spending-govt-borrowing-debt.pdf\n\n[3] Personally I use git grep with patterns and the -P flag far more\nthan I use grep with anything other than a constant string. These days\nI barely ever use grep, so to me what it does is entirely irrelevant\nto what git does. I suspect I am not alone in this.\n-- \nperl -Mre=debug -e \"/just|another|perl|hacker/\"\n"},{"id":"471539","messageId":"CALnO6CC2w5LUAjXVw7CWKAVobjFMc7M7jm7HuuGT1SO0q5cYbA@mail.gmail.com","threadId":"59175","inReplyTo":"CANgJU+UJ-p54MDmQWYTEsdh_ss-q8TUEcqHFm568Lpfsy2dU+w@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-02-05T19:51:05Z","receivedAt":"2023-02-05T19:51:22Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"On Sat, Feb 4, 2023 at 6:32 AM demerphq wrote:\n> I think that if you look at other broadly ported projects there is\n> evidence that owning your own dependencies makes a project easier and\n> cheaper to port to new platforms. […] No doubt there\n> are many other long running projects with precedents in this area.\n> What does Vi or Vim do? What does Emacs do? Etc.\n\nVi and Vim have their own pattern language descended from Ed. They\nthus have their own implementation for it. (Vim's has at least two\ndifferent \"engines\" powering the same syntax, and tries to\nautomatically pick the best one in case; I'm not sure if Vim compiles\nthe patterns before using them.) The advantage is that, as has been\npointed out, you can rely on Vi or Vim patterns.\n\n…\n\nAny thoughts on some sort of stop-gap measure to fix --word-diff while\nGit decides how to handle the regex engine incompatibilities? How\nimportant is the sequence of bytes at the end of --word-diff regexes\nin userdiff.c?\n\n-- \nD. Ben Knoble\n"},{"id":"471713","messageId":"Y+KWFyjTa8t5d5Q8@coredump.intra.peff.net","threadId":"59175","inReplyTo":"CANgJU+UJ-p54MDmQWYTEsdh_ss-q8TUEcqHFm568Lpfsy2dU+w@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-07T18:19:03Z","receivedAt":"2023-02-07T18:21:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Feb 04, 2023 at 12:32:09PM +0100, demerphq wrote:\n\n> > I think this is a perennial question for portable software: is it better\n> > to be consistent across platforms (by shipping our own regex engine), or\n> > consistent with other programs on the same platform (by using the system\n> > regex).\n> \n> Personally I think that while this seems to be an impartial reading of\n> the question at hand I think it frames the debate in a way that has\n> the potential to bias[1] the discussion in favour of a particular\n> policy outcome[2]. It implies that all other things are equal between\n> the two options presented, and frames the question as something that\n> comes down to personal preference between one form of consistency and\n> another.\n\nSorry, I didn't mean to imply one side or the other, or even that they\nwere equal. I more just meant: you can make arguments either way.\n\n_I_ don't have a strong opinion there, but you have made many of those\narguments in your email. And everything you said was pretty sensible.\n\nI did want to mention one thing, though:\n\n> It also assumes that being consistent with other programs on\n> the same platform is inherently beneficial, when that doesn't seem to\n> be clearly established[2].\n\nThis isn't consistency per se, but one problem that arises when a\nportable program ships its own implementation is that its implementation\nsucks more than the system one. And that was the case with 1819ad327;\nour compat/regex was worse than the system with respect to multi-byte\ncharacters.\n\nBut yes, if your point is: \"well, don't ship a crappy implementation\",\nthen I agree that is a way to mitigate that problem. ;)\n\n-Peff\n"},{"id":"471714","messageId":"Y+KXC5b0qUg2/nxt@coredump.intra.peff.net","threadId":"59175","inReplyTo":"CALnO6CC2w5LUAjXVw7CWKAVobjFMc7M7jm7HuuGT1SO0q5cYbA@mail.gmail.com","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-07T18:23:07Z","receivedAt":"2023-02-07T18:24:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Feb 05, 2023 at 02:51:05PM -0500, D. Ben Knoble wrote:\n\n> Any thoughts on some sort of stop-gap measure to fix --word-diff while\n> Git decides how to handle the regex engine incompatibilities? How\n> important is the sequence of bytes at the end of --word-diff regexes\n> in userdiff.c?\n\nIt comes from 664d44ee7f (userdiff: simplify word-diff safeguard,\n2011-01-11). So presumably we'd want to figure out a way to accomplish\nthe same thing in a portable way. I'm not sure that's possible, though,\nwithout making assumptions about the regex engine.\n\nThere's a little more discussion in the original thread:\n\n https://lore.kernel.org/git/561247.22837.qm@web110707.mail.gq1.yahoo.com/\n\nbut I didn't read it very carefully.\n\nIt may also be possible to play tricks with LC_CTYPE (either not setting\nit, or resetting before/after a regex), but that feels pretty hacky.\n\n-Peff\n"},{"id":"471731","messageId":"CALnO6CCzp0brmCjbNmOKe9E1bcxLzGNrqpfK_JrU=+LXt-DUyQ@mail.gmail.com","threadId":"59175","inReplyTo":"Y+KXC5b0qUg2/nxt@coredump.intra.peff.net","subject":"Re: grep: fix multibyte regex handling under macOS (1819ad327b7a1f19540a819813b70a0e8a7f798f)","fromName":"D. Ben Knoble","fromEmail":"ben.knoble@gmail.com","sentAt":"2023-02-07T22:27:14Z","receivedAt":"2023-02-07T22:27:33Z","isPatch":false,"sender":{"key":"ben.knoble@gmail.com","avatar":"https://avatars.githubusercontent.com/u/22802209?v=4"},"body":"CC'ing Jonathan Nieder\n\nOn Tue, Feb 7, 2023 at 1:23 PM Jeff King wrote:\n>\n> On Sun, Feb 05, 2023 at 02:51:05PM -0500, D. Ben Knoble wrote:\n>\n> > Any thoughts on some sort of stop-gap measure to fix --word-diff while\n> > Git decides how to handle the regex engine incompatibilities? How\n> > important is the sequence of bytes at the end of --word-diff regexes\n> > in userdiff.c?\n>\n> It comes from 664d44ee7f (userdiff: simplify word-diff safeguard,\n> 2011-01-11). So presumably we'd want to figure out a way to accomplish\n> the same thing in a portable way. I'm not sure that's possible, though,\n> without making assumptions about the regex engine.\n\nIf \"use the safeguard portably\" implies \"make assumptions about the\nregex engine,\" that sounds like an argument for Git to ship its own\nengine with exactly the necessary features. If that implementation\nincludes proper locale and UTF-8 support alongside support for the\nhigh-byte character classes, I think we would be all set…\n\nOTOH, perhaps there is a way to express the safeguard character\nclasses portably?\n\nJonathan, can you provide more context for the safeguard? I've read\nthis message several times\n\n> git's diff-words support has a detail that can be a little dangerous:\n> any text not matched by a given language's tokenization pattern is\n> treated as whitespace and changes in such text would go unnoticed.\n> Therefore each of the built-in regexes allows a special token type\n> consisting of a single non-whitespace character [^[:space:]].\n>\n> To make sure UTF-8 sequences remain human readable, the builtin\n> regexes also have a special token type for runs of bytes with the high\n> bit set. In English, non-ASCII characters are usually isolated so\n> this is analogous to the [^[:space:]] pattern, except it matches a\n> single _multibyte_ character despite use of the C locale.\n>\n> Unfortunately it is easy to make typos or forget entirely to include\n> these catch-all token types when adding support for new languages (see\n> v1.7.3.5~16, userdiff: fix typo in ruby and python word regexes,\n> 2010-12-18). Avoid this by including them automatically within the\n> PATTERNS and IPATTERN macros.\n>\n> While at it, change the UTF-8 sequence token type to match exactly one\n> non-ASCII multi-byte character, rather than an arbitrary run of them.\n\nand I can hardly make heads or tails of it.\n\n-- \nD. Ben Knoble\n"}]}