Re: t0028-working-tree-encoding.sh failing on musl based systems (Alpine Linux)

From: Junio C Hamano <gitster@pobox.com>
Date: Feb 8, 2019, 23:12 UTC
Message-ID: <xmqqd0o1kh7t.fsf@gitster-ct.c.googlers.com>
In-Reply-To: <20190208204219.GF11927@genre.crustytoothpaste.net>

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> +test_lazy_prereq NO_BOM '
> +	printf abc | iconv -f UTF-8 -t UTF-16 &&
> +	test $(wc -c) = 6
> +'

This must be "just for illustration of idea" patch? The pipeline goes to the standard output, and nobody feeds "wc".

But I think I got the idea.

In the real implementation, it probably is a good idea to allow NO_BOM16 and NO_BOM32 be orthogonal.

> +
> +write_utf16 () {
> +	test_have_prereq NO_BOM && printf '\xfe\xff'
> +	iconv -f UTF-8 -t UTF-16

This assumes "iconv -t UTF-16" on the platform gives little endian (with or without BOM), which may not be a good assumption.

If you are forcing the world to be where UTF-16 (no other specificaiton) means LE with BOM, then perhaps doing

	printf '\xfe\xff'; iconv -f UTF-8 -t UTF-16LE

without any lazy prereq may be more explicit and in line with what you did in utf8.c::reencode_string_len() below.

> -	printf "$text" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&
> -	printf "$text" | iconv -f UTF-8 -t UTF-32 >test.utf32.raw &&
> +	printf "$text" | write_utf16 >test.utf16.raw &&
> +	printf "$text" | write_utf32 >test.utf32.raw &&

Show 13 quoted lines

> diff --git a/utf8.c b/utf8.c
> index 83824dc2f4..4aa69cd65b 100644
> --- a/utf8.c
> +++ b/utf8.c
> @@ -568,6 +568,10 @@ char *reencode_string_len(const char *in, size_t insz,
>  		bom_str = utf16_be_bom;
>  		bom_len = sizeof(utf16_be_bom);
>  		out_encoding = "UTF-16BE";
> +	} else if (same_utf_encoding("UTF-16", out_encoding)) {
> +		bom_str = utf16_le_bom;
> +		bom_len = sizeof(utf16_le_bom);
> +		out_encoding = "UTF-16LE";
>  	}

I am not sure what is going on here. When the caller asks for "UTF-16", we do not let the platform implementation of iconv() to pick one of the allowed ones (i.e. BE with BOM, LE with BOM, or BE without BOM) but instead force LE with BOM?

Show 6 quoted lines

>  	conv = iconv_open(out_encoding, in_encoding);
> ------ %< ---------
>
> This passes for me on glibc, but only on a little-endian system. If this
> works for musl folks, then I'll add a config option for those people who
> have UTF-16 without BOM.