Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows

From: Jeff King <peff@peff.net>
Date: Apr 2, 2017, 07:45 UTC
Message-ID: <20170402074522.4qhannjus4ynwx4i@sigill.intra.peff.net>
In-Reply-To: <63eb5546-0dce-2f69-c2f8-1e777b97c532@gmail.com>

On Sat, Apr 01, 2017 at 08:31:27PM +0200, Jakub Narębski wrote:

Show 15 quoted lines

> W dniu 01.04.2017 o 08:08, Jeff King pisze:
> > On Fri, Mar 31, 2017 at 03:24:48PM +0200, Jakub Narębski wrote:
> > 
> >>> I suspect in the normal case that git is doing line-ending conversion,
> >>> but it's suppressed when textconv is in use.
> >>
> >> I would not consider this a bug if not for the fact that there is no ^M
> >> without using iconv as textconv.
> > 
> > I don't think it's a bug, though. You have told Git that you will
> > convert the contents (whatever their format) into the canonical format,
> > but your program to do so includes a CR.
> 
> Well, I have not declared file binary with "binary = true" in diff driver
> definition, isn't it?

I don't think binary has anything to do with it. A textconv filter takes input (binary or not) and delivers a normalized representation to feed to the diff algorithm. There's no further post-processing, and it's the responsibility of the filter to deliver the bytes it wants diffed.

Like I said, I could see an argument for treating the filter output as text to be pre-processed, but that's not how it works (and I don't think it is a good idea to change it now, unless by adding an option to the diff filter).

> P.S. What do you think about Git supporting 'encoding' attribute (or
> 'core.encoding' config) plus 'core.outputEncoding' in-core?

Supporting an "encoding" attribute to normalize file encodings in diffs seems reasonable to me. But it would have to be enabled only for human-readable diffs, as the result could not be applied (so the same as textconv).

I don't think core.outputEncoding is necessarily a good idea. We are not really equipped anything that isn't an ascii superset, as we intermingle the bytes with ascii diff headers (though I think that is true of the commitEncoding stuff; I assume everything breaks horribly if you tried to set that to UTF-16, but I've never tried it).

-Peff