Re: Should the --encoding argument to log/show commands make any guarantees about their output?

From: Jan-Philip Gehrcke <jgehrcke@googlemail.com>
Date: Jun 16, 2015, 09:38 UTC
Message-ID: <557FEEA5.2080006@googlemail.com>
In-Reply-To: <557EFB94.3040104@web.de>

On 15.06.2015 18:21, Torsten Bögershausen wrote:

Show 9 quoted lines

> On 2015-06-15 10.50, Jan-Philip Gehrcke wrote:
>> Let me describe what I think it currently does:
>>
>> The program attempts to re-code a log message, so it follows the chain
>>
>>      raw input -> unicode -> raw output
> Not sure what "raw input/output" means.
> But there is only one reencode step involved, e.g.
> input(8859) -> output(UTF-8)

We surely agree. With "raw" I meant a sequence of bytes, and with "unicode" I meant the intermediate state in the process of re-encoding (which can be thought of as decoding and encoding with a transient intermediate state).

> If the user ignores this warning, how should Git guess the encoding  ?

I entirely appreciate that there is no satisfying solution to this very problem.

Show 5 quoted lines

>> If this step fails (if the entry contains a byte sequence that is invalid in the specified/assumed input codec),
>> the procedure is aborted and the data is dumped as is (obviously without applying the requested output encoding).
>>
>> Is that correct?
> Yes, see above.

Thanks!

Show 13 quoted lines

>> Hence, from my point of view, the rational that git show/log should be able to output *text* information means
>> that they should not emit byte sequences that are invalid in the codec specified via the --encoding argument.
>> In the current situation, the work of dealing with invalid byte sequences is just outsourced to software
>> further below in the tool chain
>> (at some point a replacement character � should be displayed to the user instead of the invalid raw bytes).
>>
>> I am not entirely sure where this discussion should lead to.
> Yes, until someone writes a patch to improve either the documentation or the code,
> nothing will be changed.
>> However, I think that if the behavior of the software will not be changed,
>> then the documentation for the --encoding option should be more precise and
>> clarify what actually happens behind the scenes. What do you think?
> Patches are more than welcome.

I'd be willing to contribute, but of course there must be a discussion and an agreement before that, if there is need to change something at all, and what exactly.

To this discussion I would like to contribute that I am of the opinion that there should be a command line option to make git show/log/friends emit a byte stream that is guaranteed to be valid in a given codec.

That would require detection and treatment of those cases where corrupted text resides in the repository (we cannot prevent it from entering the repository, as discussed). In these cases, one could emit a replacement symbol (e.g. '?') per invalid byte subsequence (this seems to be more established than just swallowing the invalid byte sequence).

What do you think?

I think the --encoding option would have ideal semantics for described behavior.

However, I guess maintaining backwards compatibility is an issue here. On the other hand, I realize that the --encoding option undergoes changes: the docs for git log in release 2.4.3 do not even list the --encoding option anymore. Why is that? I haven't found a corresponding changelog/release notes entry.

Thanks,

Jan-Philip