Should the --encoding argument to log/show commands make any guarantees about their output?

Hello,

I was surprised to see that the output of

     git log --encoding=utf-8 "--format=format:%b"

can contain byte sequences that are invalid in UTF-8. Note: I am using git 2.1.4 and the %b format specifier represents the commit message body.

I have seen this with the Linux git repository and the following test:

     git log --encoding=utf-8 "--format=format:%b" | python2 -c \
         'import sys; [l.decode("utf-8") for l in sys.stdin]'

Soon enough errors like this appears:

     'utf8' codec can't decode byte 0xf6 in position 19

The help message to the --encoding argument reads:

I realize that this message does not give any guarantee about the output of the command, in the sense that --encoding=utf-8 produces valid UTF-8 data in all cases.

However, I wonder what --encoding precisely does and if it has the behavior most users would expect.

Let me describe what I think it currently does:

The program attempts to re-code a log message, so it follows the chain

	raw input -> unicode -> raw output

For the first step, knowledge about the input encoding is required. This is retrieved from the encoding header of the commit object if present or (from the docs) "lack of this header implies that the commit log message is encoded in UTF-8." If this step fails (if the entry contains a byte sequence that is invalid in the specified/assumed input codec), the procedure is aborted and the data is dumped as is (obviously without applying the requested output encoding).

Is that correct?

 From my point of view the most natural abstraction of a log *message* 
is *text*, not bytes. The same is true for author names. If I want to 
build a tool chain on top of log/show, this usually means that I want to 
work with text information. Hence, I want to retrieve text (a sequence 
of code points) from git show/log. Text must be transported in encoded 
form, sure, but it must not contain byte sequences that are invalid in 
this codec. Because otherwise it's just not text anymore.

Hence, from my point of view, the rational that git show/log should be able to output *text* information means that they should not emit byte sequences that are invalid in the codec specified via the --encoding argument. In the current situation, the work of dealing with invalid byte sequences is just outsourced to software further below in the tool chain (at some point a replacement character � should be displayed to the user instead of the invalid raw bytes).

I am not entirely sure where this discussion should lead to. However, I think that if the behavior of the software will not be changed, then the documentation for the --encoding option should be more precise and clarify what actually happens behind the scenes. What do you think?

Cheers,

Jan-Philip Gehrcke