{"thread":{"id":"58365","subject":"[PATCH v4] grep: fix multibyte regex handling under macOS","startedAt":"2022-08-26T08:58:43Z","lastAt":"2022-09-13T20:41:19Z","messageCount":4,"participants":["Diomidis Spinellis","Junio C Hamano"],"isPatch":true,"patchVersion":4,"patchTotal":null},"messages":[{"id":"461985","messageId":"20220826085815.2771102-1-dds@aueb.gr","threadId":"58365","inReplyTo":null,"subject":"[PATCH v4] grep: fix multibyte regex handling under macOS","fromName":"Diomidis Spinellis","fromEmail":"dds@aueb.gr","sentAt":"2022-08-26T08:58:15Z","receivedAt":"2022-08-26T08:58:43Z","isPatch":true,"sender":{"key":"dds@aueb.gr","avatar":"https://avatars.githubusercontent.com/u/1212716?v=4"},"body":"The commit 29de20504e (Makefile: fix default regex settings on\nDarwin, 2013-05-11) fixed t0070-fundamental.sh under Darwin (macOS) by\nadopting Git's regex library. However, this library is compiled with\nNO_MBSUPPORT, which causes git-grep to work incorrectly on multibyte\n(e.g. UTF-8) files. Current macOS versions pass t0070-fundamental.sh\nwith the native macOS regex library, which also supports multibyte\ncharacters.\n\nAdjust the Makefile to use the native regex library, and call\nsetlocale(3) to set CTYPE according to the user's preference.\nThe setlocale call is required on all platforms, but in platforms\nsupporting gettext(3), setlocale was called as a side-effect of\ninitializing gettext. Therefore, move the CTYPE setlocale call from\ngettext.c to common-main.c and the corresponding locale.h include\ninto git-compat-util.h.\n\nThanks to the global initialization of CTYPE setlocale, the test-tool\nregex command now works correctly with supported multibyte regexes, and\nis used to set the MB_REGEX test prerequisite by assessing a platform's\nsupport for them.\n\nSigned-off-by: Diomidis Spinellis \n---\n\n- Fold test script into t7810-grep.sh\n- Simplify the test prerequisite evaluation\n\n Makefile | 2 +-\n common-main.c | 1 +\n gettext.c | 2 --\n git-compat-util.h | 1 +\n t/t7810-grep.sh | 15 +++++++++++++++\n 5 files changed, 18 insertions(+), 3 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 04d0fd1fe6..d1a9825715 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1427,7 +1427,6 @@ ifeq ($(uname_S),Darwin)\n \t\tAPPLE_COMMON_CRYPTO = YesPlease\n \t\tCOMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO\n \tendif\n-\tNO_REGEX = YesPlease\n \tPTHREAD_LIBS =\n endif\n \n@@ -2970,6 +2969,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PTHREADS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PTHREADS)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n+\t@echo NO_REGEX=\\''$(subst ','\\'',$(subst ','\\'',$(NO_REGEX)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\ndiff --git a/common-main.c b/common-main.c\nindex c531372f3f..0a22861f1c 100644\n--- a/common-main.c\n+++ b/common-main.c\n@@ -40,6 +40,7 @@ int main(int argc, const char **argv)\n \n \tgit_resolve_executable_dir(argv[0]);\n \n+\tsetlocale(LC_CTYPE, \"\");\n \tgit_setup_gettext();\n \n \tinitialize_the_repository();\ndiff --git a/gettext.c b/gettext.c\nindex bb5ba1fe7c..f139008d0a 100644\n--- a/gettext.c\n+++ b/gettext.c\n@@ -10,7 +10,6 @@\n #include \"config.h\"\n \n #ifndef NO_GETTEXT\n-#\tinclude \n #\tinclude \n #\tifdef GIT_WINDOWS_NATIVE\n \n@@ -80,7 +79,6 @@ static int test_vsnprintf(const char *fmt, ...)\n \n static void init_gettext_charset(const char *domain)\n {\n-\tsetlocale(LC_CTYPE, \"\");\n \tcharset = locale_charset();\n \tbind_textdomain_codeset(domain, charset);\n \ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 58d7708296..c6fa3c7469 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -212,6 +212,7 @@\n #endif\n #include \n #include \n+#include \n #ifdef NEEDS_SYS_PARAM_H\n #include \n #endif\ndiff --git a/t/t7810-grep.sh b/t/t7810-grep.sh\nindex 6935601171..308a3dc231 100755\n--- a/t/t7810-grep.sh\n+++ b/t/t7810-grep.sh\n@@ -18,6 +18,9 @@ test_invalid_grep_expression() {\n \t'\n }\n \n+LC_ALL=en_US.UTF-8 test-tool regex '^.$' '¿' &&\n+ test_set_prereq MB_REGEX\n+\n cat >hello.c <\n #include \n@@ -87,6 +90,10 @@ test_expect_success setup '\n \t\techo unusual >\"\\\"unusual\\\" pathname\" &&\n \t\techo unusual >\"t/nested \\\"unusual\\\" pathname\"\n \tfi &&\n+\tif test_have_prereq MB_REGEX\n+\tthen\n+\t\techo \"¿\" >reverse-question-mark\n+\tfi &&\n \tgit add . &&\n \ttest_tick &&\n \tgit commit -m initial\n@@ -568,6 +575,14 @@ do\n \t'\n done\n \n+test_expect_success MB_REGEX 'grep exactly one char in single-char multibyte file' '\n+\tLC_ALL=en_US.UTF-8 git grep \"^.$\" reverse-question-mark\n+'\n+\n+test_expect_success MB_REGEX 'grep two chars in single-char multibyte file' '\n+\tLC_ALL=en_US.UTF-8 test_expect_code 1 git grep \"..\" reverse-question-mark\n+'\n+\n cat >expected < writes:\n\n> - Fold test script into t7810-grep.sh\n> - Simplify the test prerequisite evaluation\n\nThe tests certainly look better, and as v3, the build procedure\nsounds OK.\n\n> diff --git a/common-main.c b/common-main.c\n> index c531372f3f..0a22861f1c 100644\n> --- a/common-main.c\n> +++ b/common-main.c\n> @@ -40,6 +40,7 @@ int main(int argc, const char **argv)\n> \n> \tgit_resolve_executable_dir(argv[0]);\n> \n> +\tsetlocale(LC_CTYPE, \"\");\n> \tgit_setup_gettext();\n> \n> \tinitialize_the_repository();\n> ...\n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index 58d7708296..c6fa3c7469 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -212,6 +212,7 @@\n> #endif\n> #include \n> #include \n> +#include \n> #ifdef NEEDS_SYS_PARAM_H\n> #include \n> #endif\n\nI'll let others more familiar with the locale support to comment on\nthese changes. We are unconditionally including now;\nplatforms that lack could set NO_GETTEXT to work it\naround before this change, but that will no longer work.\n\nI do not know if thta is a practical downside to anybody, but it\ncould be a problem.\n\nPerhaps cook this in 'next' and see if anybody screams?\n\nThanks.\n"},{"id":"462983","messageId":"a8ba447f-d087-1c5e-0ce0-a9040ad080d1@aueb.gr","threadId":"58365","inReplyTo":"xmqqzgf389k9.fsf@gitster.g","subject":"Re: [PATCH v4] grep: fix multibyte regex handling under macOS","fromName":"Diomidis Spinellis","fromEmail":"dds@aueb.gr","sentAt":"2022-09-13T18:09:21Z","receivedAt":"2022-09-13T18:40:24Z","isPatch":true,"sender":{"key":"dds@aueb.gr","avatar":"https://avatars.githubusercontent.com/u/1212716?v=4"},"body":"On 13-Sep-22 20:32, Junio C Hamano wrote:\n>> +#include \n>> #ifdef NEEDS_SYS_PARAM_H\n>> #include \n>> #endif\n> \n> I'll let others more familiar with the locale support to comment on\n> these changes. We are unconditionally including now;\n> platforms that lack could set NO_GETTEXT to work it\n> around before this change, but that will no longer work.\n> \n> I do not know if thta is a practical downside to anybody, but it\n> could be a problem.\n> \n> Perhaps cook this in 'next' and see if anybody screams?\n\nI was programming in C before locale.h was introduced, so its \navailability was something that worried me as well. I looked up whether \nwe could/should use a GNU autoconf macro to test for the locale.h \navailability. According to autoconf's documentation \"All hosted \nenvironments that are still of interest for portable code provide all of \nthe headers specified in ISO C90 (as amended in 1995): assert.h, [...] \nlocale.h [...], and wctype.h. Most programs can safely include these \nheaders unconditionally.\" [1]\n\n[1] https://www.gnu.org/software/autoconf/manual/autoconf-2.70/autoconf.html\n"},{"id":"463001","messageId":"xmqqmtb380t2.fsf@gitster.g","threadId":"58365","inReplyTo":"a8ba447f-d087-1c5e-0ce0-a9040ad080d1@aueb.gr","subject":"Re: [PATCH v4] grep: fix multibyte regex handling under macOS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-09-13T20:41:13Z","receivedAt":"2022-09-13T20:41:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Diomidis Spinellis writes:\n\n>> Perhaps cook this in 'next' and see if anybody screams?\n\nLet's do this.\n\n> I was programming in C before locale.h was introduced, so its\n> availability was something that worried me as well. I looked up\n> whether we could/should use a GNU autoconf macro to test for the\n> locale.h availability. According to autoconf's documentation \"All\n> hosted environments that are still of interest for portable code\n> provide all of the headers specified in ISO C90 (as amended in 1995):\n> assert.h, [...] locale.h [...], and wctype.h. Most programs can safely\n> include these headers unconditionally.\" [1]\n\nYeah, but there are folks on minor platforms where full POSIX\ncompliance is iffy, so whatever autoconf's documentation says\nunfortunately does not weigh as much as we wish it does.\n\n\n"}]}