{"thread":{"id":"58357","subject":"[PATCH v3] grep: fix multibyte regex handling under macOS","startedAt":"2022-08-25T08:31:50Z","lastAt":"2022-08-26T00:20:36Z","messageCount":2,"participants":["Diomidis Spinellis","Junio C Hamano"],"isPatch":true,"patchVersion":3,"patchTotal":null},"messages":[{"id":"461937","messageId":"20220825082045.2662893-1-dds@aueb.gr","threadId":"58357","inReplyTo":null,"subject":"[PATCH v3] grep: fix multibyte regex handling under macOS","fromName":"Diomidis Spinellis","fromEmail":"dds@aueb.gr","sentAt":"2022-08-25T08:20:45Z","receivedAt":"2022-08-25T08:31:50Z","isPatch":true,"sender":{"key":"dds@aueb.gr","avatar":"https://avatars.githubusercontent.com/u/1212716?v=4"},"body":"The commit 29de20504e (Makefile: fix default regex settings on\nDarwin, 2013-05-11) fixed t0070-fundamental.sh under Darwin (macOS) by\nadopting Git's regex library.  However, this library is compiled with\nNO_MBSUPPORT, which causes git-grep to work incorrectly on multibyte\n(e.g. UTF-8) files.  Current macOS versions pass t0070-fundamental.sh\nwith the native macOS regex library, which also supports multibyte\ncharacters.\n\nAdjust the Makefile to use the native regex library, and call\nsetlocale(3) to set CTYPE according to the user's preference.\nThe setlocale call is required on all platforms, but in platforms\nsupporting gettext(3), setlocale was called as a side-effect of\ninitializing gettext.  Therefore, move the CTYPE setlocale call from\ngettext.c to common-main.c and the corresponding locale.h include\ninto git-compat-util.h.\n\nTo avoid running the new tests on the few platforms still using the\ncompatibility library, which is compiled without multibyte support,\nstore the corresponding NO_REGEX setting in the GIT-BUILD-OPTIONS file.\nThis makes it available to the test scripts.  Thanks to the global\ninitialization of CTYPE setlocale, the test-tool regex command now\nworks correctly with multibyte regexes and is used to further test\na platform's support for them.\n\nSigned-off-by: Diomidis Spinellis <dds@aueb.gr>\n---\n- Move added setlocale from grep.c and test-regex.c to common-main.c\n- Move added locale.h include from the above files to git-compat-util.h\n- Correct use of test_expect_code 1\n- Have old commit reference in the commit message follow commit\n  referencing conventions\n\n Makefile                  |  2 +-\n common-main.c             |  1 +\n gettext.c                 |  2 --\n git-compat-util.h         |  1 +\n t/t7818-grep-multibyte.sh | 34 ++++++++++++++++++++++++++++++++++\n 5 files changed, 37 insertions(+), 3 deletions(-)\n create mode 100755 t/t7818-grep-multibyte.sh\n\ndiff --git a/Makefile b/Makefile\nindex 04d0fd1fe6..d1a9825715 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1427,7 +1427,6 @@ ifeq ($(uname_S),Darwin)\n \t\tAPPLE_COMMON_CRYPTO = YesPlease\n \t\tCOMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO\n \tendif\n-\tNO_REGEX = YesPlease\n \tPTHREAD_LIBS =\n endif\n \n@@ -2970,6 +2969,7 @@ GIT-BUILD-OPTIONS: FORCE\n \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n \t@echo NO_PTHREADS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PTHREADS)))'\\' >>$@+\n \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n+\t@echo NO_REGEX=\\''$(subst ','\\'',$(subst ','\\'',$(NO_REGEX)))'\\' >>$@+\n \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\ndiff --git a/common-main.c b/common-main.c\nindex c531372f3f..0a22861f1c 100644\n--- a/common-main.c\n+++ b/common-main.c\n@@ -40,6 +40,7 @@ int main(int argc, const char **argv)\n \n \tgit_resolve_executable_dir(argv[0]);\n \n+\tsetlocale(LC_CTYPE, \"\");\n \tgit_setup_gettext();\n \n \tinitialize_the_repository();\ndiff --git a/gettext.c b/gettext.c\nindex bb5ba1fe7c..f139008d0a 100644\n--- a/gettext.c\n+++ b/gettext.c\n@@ -10,7 +10,6 @@\n #include \"config.h\"\n \n #ifndef NO_GETTEXT\n-#\tinclude <locale.h>\n #\tinclude <libintl.h>\n #\tifdef GIT_WINDOWS_NATIVE\n \n@@ -80,7 +79,6 @@ static int test_vsnprintf(const char *fmt, ...)\n \n static void init_gettext_charset(const char *domain)\n {\n-\tsetlocale(LC_CTYPE, \"\");\n \tcharset = locale_charset();\n \tbind_textdomain_codeset(domain, charset);\n \ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 58d7708296..c6fa3c7469 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -212,6 +212,7 @@\n #endif\n #include <errno.h>\n #include <limits.h>\n+#include <locale.h>\n #ifdef NEEDS_SYS_PARAM_H\n #include <sys/param.h>\n #endif\ndiff --git a/t/t7818-grep-multibyte.sh b/t/t7818-grep-multibyte.sh\nnew file mode 100755\nindex 0000000000..a3889f9822\n--- /dev/null\n+++ b/t/t7818-grep-multibyte.sh\n@@ -0,0 +1,34 @@\n+#!/bin/sh\n+\n+test_description='grep multibyte characters'\n+\n+. ./test-lib.sh\n+\n+# Multibyte regex search is only supported with a native regex library\n+# that supports it.\n+# (The supplied compatibility library is compiled with NO_MBSUPPORT.)\n+test -z \"$NO_REGEX\" &&\n+  LC_ALL=en_US.UTF-8 test-tool regex '^.$' '¿' &&\n+  test_set_prereq MB_REGEX\n+\n+if ! test_have_prereq MB_REGEX\n+then\n+  skip_all='multibyte grep tests; Git compiled with NO_REGEX, NO_MBSUPPORT'\n+  test_done\n+fi\n+\n+test_expect_success 'setup' '\n+\ttest_write_lines \"¿\" >file &&\n+\tgit add file &&\n+\tLC_ALL=\"en_US.UTF-8\" &&\n+\texport LC_ALL\n+'\n+test_expect_success 'grep exactly one char in single-char multibyte file' '\n+\tgit grep \"^.$\"\n+'\n+\n+test_expect_success 'grep two chars in single-char multibyte file' '\n+\ttest_expect_code 1 git grep \"..\"\n+'\n+\n+test_done\n-- \n2.30.2\n\n"},{"id":"461973","messageId":"xmqqpmgn26up.fsf@gitster.g","threadId":"58357","inReplyTo":"20220825082045.2662893-1-dds@aueb.gr","subject":"Re: [PATCH v3] grep: fix multibyte regex handling under macOS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-08-26T00:20:30Z","receivedAt":"2022-08-26T00:20:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Diomidis Spinellis <dds@aueb.gr> writes:\n\n> diff --git a/Makefile b/Makefile\n> index 04d0fd1fe6..d1a9825715 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1427,7 +1427,6 @@ ifeq ($(uname_S),Darwin)\n>  \t\tAPPLE_COMMON_CRYPTO = YesPlease\n>  \t\tCOMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO\n>  \tendif\n> -\tNO_REGEX = YesPlease\n>  \tPTHREAD_LIBS =\n>  endif\n>  \n> @@ -2970,6 +2969,7 @@ GIT-BUILD-OPTIONS: FORCE\n>  \t@echo NO_PERL=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PERL)))'\\' >>$@+\n>  \t@echo NO_PTHREADS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PTHREADS)))'\\' >>$@+\n>  \t@echo NO_PYTHON=\\''$(subst ','\\'',$(subst ','\\'',$(NO_PYTHON)))'\\' >>$@+\n> +\t@echo NO_REGEX=\\''$(subst ','\\'',$(subst ','\\'',$(NO_REGEX)))'\\' >>$@+\n>  \t@echo NO_UNIX_SOCKETS=\\''$(subst ','\\'',$(subst ','\\'',$(NO_UNIX_SOCKETS)))'\\' >>$@+\n>  \t@echo PAGER_ENV=\\''$(subst ','\\'',$(subst ','\\'',$(PAGER_ENV)))'\\' >>$@+\n>  \t@echo DC_SHA1=\\''$(subst ','\\'',$(subst ','\\'',$(DC_SHA1)))'\\' >>$@+\n\nBuild part looks good to me.\n\n> diff --git a/common-main.c b/common-main.c\n> index c531372f3f..0a22861f1c 100644\n> --- a/common-main.c\n> +++ b/common-main.c\n> @@ -40,6 +40,7 @@ int main(int argc, const char **argv)\n>  \n>  \tgit_resolve_executable_dir(argv[0]);\n>  \n> +\tsetlocale(LC_CTYPE, \"\");\n>  \tgit_setup_gettext();\n>  \n>  \tinitialize_the_repository();\n> diff --git a/gettext.c b/gettext.c\n> index bb5ba1fe7c..f139008d0a 100644\n> --- a/gettext.c\n> +++ b/gettext.c\n> @@ -10,7 +10,6 @@\n>  #include \"config.h\"\n>  \n>  #ifndef NO_GETTEXT\n> -#\tinclude <locale.h>\n>  #\tinclude <libintl.h>\n>  #\tifdef GIT_WINDOWS_NATIVE\n>  \n> @@ -80,7 +79,6 @@ static int test_vsnprintf(const char *fmt, ...)\n>  \n>  static void init_gettext_charset(const char *domain)\n>  {\n> -\tsetlocale(LC_CTYPE, \"\");\n>  \tcharset = locale_charset();\n>  \tbind_textdomain_codeset(domain, charset);\n> \n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index 58d7708296..c6fa3c7469 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -212,6 +212,7 @@\n>  #endif\n>  #include <errno.h>\n>  #include <limits.h>\n> +#include <locale.h>\n>  #ifdef NEEDS_SYS_PARAM_H\n>  #include <sys/param.h>\n>  #endif\n\nI'll let others more familiar with the locale support to comment on\nthese changes.  We are unconditionally including <locale.h> now;\nbefore platforms that lack locale.h can set NO_GETTEXT but that will\nno longer work as a \"workaround\" for them.  I do not know if thta is\na practical downside to anybody, but it could be a problem.\n\n> diff --git a/t/t7818-grep-multibyte.sh b/t/t7818-grep-multibyte.sh\n> new file mode 100755\n> index 0000000000..a3889f9822\n> --- /dev/null\n> +++ b/t/t7818-grep-multibyte.sh\n\nDo we need a new test script for this?\n\n> @@ -0,0 +1,34 @@\n> +#!/bin/sh\n> +\n> +test_description='grep multibyte characters'\n> +\n> +. ./test-lib.sh\n> +\n> +# Multibyte regex search is only supported with a native regex library\n> +# that supports it.\n> +# (The supplied compatibility library is compiled with NO_MBSUPPORT.)\n\nThis file is not specific to Darwin; \"... with a native regex\nlibrary\" etc. is not something we want to see here.\n\n> +test -z \"$NO_REGEX\" &&\n> +  LC_ALL=en_US.UTF-8 test-tool regex '^.$' '¿' &&\n> +  test_set_prereq MB_REGEX\n\nWe can safely drop 'test -z \"$NO_REGEX\" &&' part here, no?  Even if\nwe omit it, those who built with NO_REGEX would fail \"test-tool\nregex\" step above.  And by omitting $NO_REGEX check, we do not have\nto look for and update the condition when the fallback regex engine\nwe use starts supporting MB_REGEX.\n\n> +if ! test_have_prereq MB_REGEX\n> +then\n> +  skip_all='multibyte grep tests; Git compiled with NO_REGEX, NO_MBSUPPORT'\n> +  test_done\n> +fi\n\nI do not think if we need a single use prereq here.  We can just\nuse whatever condition that is used to set MB_REGEX above and do the\nskip-all thing here.\n\n> +test_expect_success 'setup' '\n> +\ttest_write_lines \"¿\" >file &&\n> +\tgit add file &&\n> +\tLC_ALL=\"en_US.UTF-8\" &&\n> +\texport LC_ALL\n> +'\n\nMissing inter-test blank line.\n\n> +test_expect_success 'grep exactly one char in single-char multibyte file' '\n> +\tgit grep \"^.$\"\n> +'\n\nI am not sure how much value we are getting out of this test, which\nis identical to what we already tested earlier above with \"test-tool\nregex\".\n\n> +test_expect_success 'grep two chars in single-char multibyte file' '\n> +\ttest_expect_code 1 git grep \"..\"\n> +'\n> +\n> +test_done\n"}]}