git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2] t: use user-specific utf-8 locale for testing

From
Torsten Bögershausen <tboegi@web.de>
Date
Jun 6, 2021, 20:06 UTC
Message-ID
<20210606200613.muanikoqgpjsgk66@tb-raspi4>
In-Reply-To
<20210606163316.8630-1-congdanhqx@gmail.com>

This all looks good. Some suggestions about the commit message are inline.

On Sun, Jun 06, 2021 at 11:33:16PM +0700, Đoàn Trần Công Danh wrote:
> In some test-cases, utf-8 locale is required. To find such locale,
> we're using the first available UTF-8 locale that returned by
> "locale -a".

Good explanation. I think that in generaral "utf-8" as a specification/specifier is better written as "UTF-8", with uppercase. "utf-8" or utf8 may be used inside the code, depending on the language.

>
> Despite being required by POSIX, locale(1) is unavailable in some
> systems, e.g. Linux with musl libc.  Some of those systems support
> utf-8 locale out of the box.

This reads a little bit harsh (the first sentence) and it is not fully clear which systems do what (the second sentence). Or are 2 things mentioned - the locale(1) utility and the support of one UTF-8 locale "out of the box" ? Does Linux with musl libs support an UTF-8 locale, but not the locale(1) untility ? Git itself supports many systems, that are not POSIX compliant, strictly speaking. But if avaliable, the functions defined in POSIX are used, whenever available.

Could we write: However, the locale(1) utility is unavailable on some systems, e.g. Linux with musl libc.

Show 5 quoted lines
>
> However, without "locale -a", we can't guess provided UTF-8 locale.
>
> Let's give users of those systems an option to have better test
> coverage.
Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for linux-musl
Show 6 quoted lines
>
> This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to
> prepare_utf8_locale, since we no longer prepare the variable named
> "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE
> instead.  The fallback will be LC_ALL, LANG environment variable,
> or the first utf-8 locale from output of "locale -a", in that order.

rename -> renames, may be drop "This change", like this ? Rename t/lib-git-svn.sh:prepare_a_utf8_locale into prepare_utf8_locale, since we no longer prepare the variable named "a_utf8_locale", but set up a fallback value for GIT_TEST_UTF8_LOCALE instead. The fallback will be LC_ALL, LANG environment variable, or the first UTF-8 locale from output of "locale -a", in that order.

Show 232 quoted lines
>
> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> ---
> Range-diff against v1:
> 1:  d242ce64c4 ! 1:  f299ae2239 t: use user-specific utf-8 locale for testing
>     @@ Commit message
>          Let's give users of those systems an option to have better test
>          coverage.
>
>     +    This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to
>     +    prepare_utf8_locale, since we no longer prepare the variable named
>     +    "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE
>     +    instead.  The fallback will be LC_ALL, LANG environment variable,
>     +    or the first utf-8 locale from output of "locale -a", in that order.
>     +
>          Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
>
>       ## Makefile ##
>     @@ Makefile: all::
>       # with a different indexfile format version.  If it isn't set the index
>       # file format used is index-v[23].
>       #
>     -+# Define GIT_TEST_UTF8_LOCALE to prefered utf-8 locale for testing.
>     -+# If it isn't set, use the first utf-8 locale returned by "locale -a".
>     ++# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
>     ++# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
>     ++# locale returned by "locale -a".
>      +#
>       # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
>       #
>     @@ Makefile: ifdef GIT_TEST_CMP
>       	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
>       ifdef GIT_PERF_REPEAT_COUNT
>
>     + ## ci/lib.sh ##
>     +@@ ci/lib.sh: linux-musl)
>     + 	CC=gcc
>     + 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
>     + 	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
>     ++	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
>     + 	;;
>     + esac
>     +
>     +
>       ## t/lib-git-svn.sh ##
>      @@ t/lib-git-svn.sh: start_svnserve () {
>       		 --listen-host 127.0.0.1 &
>     @@ t/lib-git-svn.sh: start_svnserve () {
>      -}')
>      -	if test -n "$a_utf8_locale"
>      +prepare_utf8_locale () {
>     -+	if test -z "$GIT_TEST_UTF8_LOCALE"
>     ++	if test -n "$GIT_TEST_UTF8_LOCALE"
>     ++	then
>     ++		: test_set_prereq UTF8
>     ++	elif test -n "${LC_ALL:-$LANG}"
>      +	then
>     ++		case "${LC_ALL:-$LANG}" in
>     ++		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
>     ++			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
>     ++			;;
>     ++		esac
>     ++	else
>      +		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
>      +		p
>      +		q
>
>  Makefile                                 |  7 +++++++
>  ci/lib.sh                                |  1 +
>  t/lib-git-svn.sh                         | 24 ++++++++++++++++++------
>  t/t9100-git-svn-basic.sh                 | 14 +++-----------
>  t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
>  t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
>  6 files changed, 34 insertions(+), 22 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index c3565fc0f8..502e0c9a81 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -398,6 +398,10 @@ all::
>  # with a different indexfile format version.  If it isn't set the index
>  # file format used is index-v[23].
>  #
> +# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
> +# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
> +# locale returned by "locale -a".
> +#
>  # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
>  #
>  # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
> @@ -2801,6 +2805,9 @@ ifdef GIT_TEST_CMP
>  endif
>  ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT
>  	@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+
> +endif
> +ifdef GIT_TEST_UTF8_LOCALE
> +	@echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+
>  endif
>  	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
>  ifdef GIT_PERF_REPEAT_COUNT
> diff --git a/ci/lib.sh b/ci/lib.sh
> index d848c036c5..476c3f369f 100755
> --- a/ci/lib.sh
> +++ b/ci/lib.sh
> @@ -229,6 +229,7 @@ linux-musl)
>  	CC=gcc
>  	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
>  	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
> +	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
>  	;;
>  esac
>
> diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
> index 547eb3c31a..83efc17661 100644
> --- a/t/lib-git-svn.sh
> +++ b/t/lib-git-svn.sh
> @@ -121,12 +121,24 @@ start_svnserve () {
>  		 --listen-host 127.0.0.1 &
>  }
>
> -prepare_a_utf8_locale () {
> -	a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
> -	p
> -	q
> -}')
> -	if test -n "$a_utf8_locale"
> +prepare_utf8_locale () {
> +	if test -n "$GIT_TEST_UTF8_LOCALE"
> +	then
> +		: test_set_prereq UTF8
> +	elif test -n "${LC_ALL:-$LANG}"
> +	then
> +		case "${LC_ALL:-$LANG}" in
> +		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
> +			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
> +			;;
> +		esac
> +	else
> +		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
> +		p
> +		q
> +	}')
> +	fi
> +	if test -n "$GIT_TEST_UTF8_LOCALE"
>  	then
>  		test_set_prereq UTF8
>  	else
> diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
> index 1d3fdcc997..d5563ec35f 100755
> --- a/t/t9100-git-svn-basic.sh
> +++ b/t/t9100-git-svn-basic.sh
> @@ -4,21 +4,13 @@
>  #
>
>  test_description='git svn basic tests'
> -GIT_SVN_LC_ALL=${LC_ALL:-$LANG}
>
>  GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
>  export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
>
>  . ./lib-git-svn.sh
>
> -case "$GIT_SVN_LC_ALL" in
> -*.UTF-8)
> -	test_set_prereq UTF8
> -	;;
> -*)
> -	say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
> -	;;
> -esac
> +prepare_utf8_locale
>
>  test_expect_success 'git svn --version works anywhere' '
>  	nongit git svn --version
> @@ -187,8 +179,8 @@ test_expect_success POSIXPERM,SYMLINKS "$name" '
>  	test ! -h "$SVN_TREE"/exec-2.sh &&
>  	test_cmp help "$SVN_TREE"/exec-2.sh'
>
> -name="commit with UTF-8 message: locale: $GIT_SVN_LC_ALL"
> -LC_ALL="$GIT_SVN_LC_ALL"
> +name="commit with UTF-8 message: locale: $GIT_TEST_UTF8_LOCALE"
> +LC_ALL="$GIT_TEST_UTF8_LOCALE"
>  export LC_ALL
>  # This test relies on the previous test, hence requires POSIXPERM,SYMLINKS
>  test_expect_success UTF8,POSIXPERM,SYMLINKS "$name" "
> diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh
> index 9b44a44bc1..743fbe1fe4 100755
> --- a/t/t9115-git-svn-dcommit-funky-renames.sh
> +++ b/t/t9115-git-svn-dcommit-funky-renames.sh
> @@ -93,9 +93,9 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' '
>  # > ... All of the above characters, except for the backslash, are converted
>  # > to special UNICODE characters in the range 0xf000 to 0xf0ff (the
>  # > "Private use area") when creating or accessing files.
> -prepare_a_utf8_locale
> +prepare_utf8_locale
>  test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new file on dcommit' '
> -	LC_ALL=$a_utf8_locale &&
> +	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
>  	export LC_ALL &&
>  	neq=$(printf "\201\202") &&
>  	git config svn.pathnameencoding cp932 &&
> @@ -107,7 +107,7 @@ test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new
>
>  # See the comment on the above test for setting of LC_ALL.
>  test_expect_success !MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 rename on dcommit' '
> -	LC_ALL=$a_utf8_locale &&
> +	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
>  	export LC_ALL &&
>  	inf=$(printf "\201\207") &&
>  	git config svn.pathnameencoding cp932 &&
> diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh
> index 2c213ae654..01e1e8a8f7 100755
> --- a/t/t9129-git-svn-i18n-commitencoding.sh
> +++ b/t/t9129-git-svn-i18n-commitencoding.sh
> @@ -14,12 +14,12 @@ compare_git_head_with () {
>  	test_cmp current "$1"
>  }
>
> -prepare_a_utf8_locale
> +prepare_utf8_locale
>
>  compare_svn_head_with () {
>  	# extract just the log message and strip out committer info.
>  	# don't use --limit here since svn 1.1.x doesn't have it,
> -	LC_ALL="$a_utf8_locale" svn log $(git svn info --url) | perl -w -e '
> +	LC_ALL="$GIT_TEST_UTF8_LOCALE" svn log $(git svn info --url) | perl -w -e '
>  		use bytes;
>  		$/ = ("-"x72) . "\n";
>  		my @x = <STDIN>;
> --
> 2.32.0.rc3.5.gf3d78db977
>
Previous: Đoàn Trần Công DanhNext: Junio C Hamano
Message 8 of 19 in “t: use user-specific utf-8 locale for testing”
  1. t: use user-specific utf-8 locale for testingĐoàn Trần Công Danh, Jun 2, 2021
  2. Taylor BlauJun 2, 2021
  3. Ævar Arnfjörð BjarmasonJun 8, 2021
  4. Jeff KingJun 3, 2021
  5. Bagas SanjayaJun 4, 2021
  6. Đoàn Trần Công DanhJun 4, 2021
  7. t: use user-specific utf-8 locale for testingĐoàn Trần Công Danh, Jun 6, 2021
  8. Torsten BögershausenJun 6, 2021
  9. Junio C HamanoJun 7, 2021
  10. t: use pre-defined utf-8 locale for testing svnĐoàn Trần Công Danh, Jun 7, 2021
  11. Junio C HamanoJun 7, 2021
  12. Torsten BögershausenJun 7, 2021
  13. Đoàn Trần Công DanhJun 7, 2021
  14. Jeff KingJun 8, 2021
  15. Đoàn Trần Công DanhJun 8, 2021
  16. t: use user-specified utf-8 locale for testing svnĐoàn Trần Công Danh, Jun 7, 2021
  17. Jeff KingJun 8, 2021
  18. t: use user-specified utf-8 locale for testing svnĐoàn Trần Công Danh, Jun 8, 2021
  19. Jeff KingJun 8, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.