From: René Scharfe <l.s.r@web.de>
Date: Tue, 09 Dec 2025 19:35:23 GMT
Subject: Re: t3900 failure on macOS, iconv(3) broken?
Message-ID: <51dc4ca7-61fd-42f7-8e72-a516a870e011@web.de>
In-Reply-To: <20251209163356.GA5762@tb-raspi4>

On 12/9/25 5:33 PM, Torsten Bögershausen wrote:
> On Mon, Dec 08, 2025 at 11:59:11PM +0100, René Scharfe wrote:
>>
>> diff --git a/utf8.c b/utf8.c
>> index 35a0251939..ff0c541fbc 100644
>> --- a/utf8.c
>> +++ b/utf8.c
>> @@ -515,6 +515,19 @@ char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv,
>>  			out = xrealloc(out, outalloc);
>>  			outpos = out + sofar;
>>  			outsz = outalloc - sofar - 1;
>> +#ifdef ICONV_BREAKS
>> +			/*
>> +			 * If iconv(3) messes up piecemeal conversions
>> +			 * then restore the original pointers, sizes,
>> +			 * and converter state, then retry converting
>> +			 * the full string using the reallocated buffer.
>> +			 */
>> +			insz += (char *)cp - in;
>> +			cp = (iconv_ibp)in;
>> +			outpos = out + bom_len;
>> +			outsz = outalloc - bom_len - 1;
>> +			iconv(conv, NULL, NULL, NULL, NULL);
>> +#endif
>>  		}
>>  		else {
>>  			*outpos = '\0';
> 
> 
> I am not sure, if I understand the second call to iconv(NULL....)

It resets the state of the converter, e.g. the current code page of
encodings that have multiple ones.

> Here is a slightly different patch.
> Comments wellcome.
> 
> 
> diff --git a/utf8.c b/utf8.c
> index 35a0251939..b3c1dd2b59 100644
> --- a/utf8.c
> +++ b/utf8.c
> @@ -486,10 +486,11 @@ int utf8_fprintf(FILE *stream, const char *format, ...)
>  char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv,
>  			    size_t bom_len, size_t *outsz_p)
>  {
> -	size_t outsz, outalloc;
> +	size_t outsz, outalloc, originsz;
>  	char *out, *outpos;
>  	iconv_ibp cp;
>  
> +	originsz = insz;
>  	outsz = insz;
>  	outalloc = st_add(outsz, 1 + bom_len); /* for terminating NUL */
>  	out = xmalloc(outalloc);
> @@ -515,6 +516,17 @@ char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv,
>  			out = xrealloc(out, outalloc);
>  			outpos = out + sofar;
>  			outsz = outalloc - sofar - 1;
> +#ifdef __APPLE__
> +			/*
> +			 * Several version of iconv(3) mess up piecemeal conversions.
> +			 * Restore the original pointers, sizes,
> +			 * and converter state, then retry converting
> +			 * the full string using the reallocated buffer.
> +			 */
> +                        insz = originsz;
> +                        outpos = out + bom_len;
> +                        cp = (iconv_ibp)in;

This forgets to reset outsz and the converter state.  With this patch
t0028-working-tree-encoding.sh seems to get stuck in an endless loop.

> +#endif
>  		}
>  		else {
>  			*outpos = '\0';