Really noisy encoding warnings post-v2.33.0

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Oct 9, 2021, 00:58 UTC
Message-ID: <87ily7m1mv.fsf@evledraar.gmail.com>
In-Reply-To: <YSkvNyR4uT52de13@coredump.intra.peff.net>

On Fri, Aug 27 2021, Jeff King wrote:

$subject because I think we should really consider backing this out before it gets to a real release.

I ran into this while testing the grep coloring patch[1] (but it's unrelated). Before this commit e.g.:

    LC_ALL=C ~/g/git/git -P -c i18n.commitEncoding=ascii log --author=Ævar -100|wc -l
    28333

So ~3k lines for my last 100 commits, but then:

    $ LC_ALL=C ~/g/git/git -P -c i18n.commitEncoding=ascii log --author=Ævar -100 2>&1|grep -c ^warning
    299

At first I thought it was spewing warnings for every failed re-encoded line in some cases, because I get hundreds at a time sometimes, but it's because stderr and stdout I/O buffering is different (a common case). Adding a "fflush(stderr)" "fixes" that.

But anyway, I think we've got a lot of users who say *do* want to reencode something from say UTF-8 to latin1, but then might have the occasional non-latin1 representable data. The old behavior of silently falling back is going to be much better for those users, or maybe show one warning at the end or something, if you feel it really needs to be kept.

Show 105 quoted lines

> On Wed, Aug 25, 2021 at 09:31:59AM -0700, Junio C Hamano wrote:
>
>> Jeff King <peff@peff.net> writes:
>> 
>> > We feed the encoding "HTML" to iconv_open(), which of course has no idea
>> > what that is. It's unfortunate, though, that we don't even print a
>> > warning, and instead just quietly leave the text intact. I wonder if we
>> > should do something like:
>> [...]
>> This addition sounds quite sensible to me.
>> 
>> "git log --encoding=bogus" would issue this warning for each and
>> every commit and that may be a bit irritating, but being irritating
>> may be a good characteristic for a warning message that is given to
>> an easily correctable condition.
>> 
>> I originally thought that the warning would be lost to the pager,
>> but apparently I forgot what I did eons ago at 61b80509 (sending
>> errors to stdout under $PAGER, 2008-02-16) ;-).
>
> Here it is polished into a real commit.
>
> -- >8 --
> Subject: [PATCH] logmsg_reencode(): warn when iconv() fails
>
> If the user asks for a pretty-printed commit to be converted (either
> explicitly with --encoding=foo, or implicitly because the commit is
> non-utf8 and we want to convert it), we pass it through iconv(). If that
> fails, we fall back to showing the input verbatim, but don't tell the
> user that the output may be bogus.
>
> Let's add a warning to do so, along with a mention in the documentation
> for --encoding. Two things to note about the implementation:
>
>   - we could produce the warning closer to the call to iconv() in
>     reencode_string_len(), which would let us relay the value of errno.
>     But this is not actually very helpful. reencode_string_len() does
>     not know we are operating on a commit, and indeed does not know that
>     the caller won't produce an error of its own. And the errno values
>     from iconv() are seldom helpful (iconv_open() only ever produces
>     EINVAL; perhaps EILSEQ from iconv() might be illuminating, but it
>     can also return EINVAL for incomplete sequences).
>
>   - if the reason for the failure is that the output charset is not
>     supported, then the user will see this warning for every commit we
>     try to display. That might be ugly and overwhelming, but on the
>     other hand it is making it clear that every one of them has not been
>     converted (and the likely outcome anyway is to re-try the command
>     with a supported output encoding).
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  Documentation/pretty-options.txt | 4 +++-
>  pretty.c                         | 6 +++++-
>  t/t4210-log-i18n.sh              | 7 +++++++
>  3 files changed, 15 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/pretty-options.txt b/Documentation/pretty-options.txt
> index 27ddaf84a1..42b227bc40 100644
> --- a/Documentation/pretty-options.txt
> +++ b/Documentation/pretty-options.txt
> @@ -40,7 +40,9 @@ people using 80-column terminals.
>  	defaults to UTF-8. Note that if an object claims to be encoded
>  	in `X` and we are outputting in `X`, we will output the object
>  	verbatim; this means that invalid sequences in the original
> -	commit may be copied to the output.
> +	commit may be copied to the output. Likewise, if iconv(3) fails
> +	to convert the commit, we will output the original object
> +	verbatim, along with a warning.
>  
>  --expand-tabs=<n>::
>  --expand-tabs::
> diff --git a/pretty.c b/pretty.c
> index 9631529c10..73b5ead509 100644
> --- a/pretty.c
> +++ b/pretty.c
> @@ -671,7 +671,11 @@ const char *repo_logmsg_reencode(struct repository *r,
>  	 * If the re-encoding failed, out might be NULL here; in that
>  	 * case we just return the commit message verbatim.
>  	 */
> -	return out ? out : msg;
> +	if (!out) {
> +		warning("unable to reencode commit to '%s'", output_encoding);
> +		return msg;
> +	}
> +	return out;
>  }
>  
>  static int mailmap_name(const char **email, size_t *email_len,
> diff --git a/t/t4210-log-i18n.sh b/t/t4210-log-i18n.sh
> index d2dfcf164e..0141f36e33 100755
> --- a/t/t4210-log-i18n.sh
> +++ b/t/t4210-log-i18n.sh
> @@ -131,4 +131,11 @@ do
>  	fi
>  done
>  
> +test_expect_success 'log shows warning when conversion fails' '
> +	enc=this-encoding-does-not-exist &&
> +	git log -1 --encoding=$enc 2>err &&
> +	echo "warning: unable to reencode commit to ${SQ}${enc}${SQ}" >expect &&
> +	test_cmp expect err
> +'
> +
>  test_done

*Really* noisy encoding warnings post-v2.33.0

Really noisy encoding warnings post-v2.33.0