Possible bug: "git log" ignores "--encoding=UTF-8" option if --pretty=format:%e%n%s%n is used

If encoding is forced for git log with "--encoding=UTF-8", it seems to be ineffective for the formatted output. Is it intentional?

Lets consider the following scenario (script is for bash, git 1.5.4.3 and 1.6.0.2):

git init echo initial >test.txt git add test.txt echo -e \\0320\\0242\\0320\\0265\\0321\\0201\\0321\\0202 >msg-UTF-8.txt git commit -F msg-UTF-8.txt echo updated >test.txt git add test.txt echo -e \\0322\\0345\\0361\\0362 >msg-windows-1251.txt git config i18n.commitencoding Windows-1251 git commit -F msg-windows-1251.txt git log --encoding=Windows-1251 >log1.txt git log --encoding=UTF-8 >log2.txt git log --encoding=Windows-1251 --pretty=format:%e%n%s%n >log3.txt git log --encoding=UTF-8 --pretty=format:%e%n%s%n >log4.txt

In the both cases the string is the russian string meaning test in different encoding.

If we compare log1.txt and log2.txt, we will see that the same encoding specified is used for both commits in the log.

If we compare log3.txt and log4.txt, we will see that the same the contents of the file is completely identical. The native encoding is used for each commit. So the first listed commit is encoded using Windows-1251 and the second one using UTF-8. However in the description of the %s %b options nothing is said about which encoding is used and implied behavior is that they are affected by --encoding option.

I suggest documenting that the placeholders %s and %b use native commit encoding and introducing the placeholders %S and %B options that use encoding specified on the command line or the default log encoding.

I also suggest adding %g and %G placeholders (%m placeholder is already occupied) that print the entire commit message instead of just the subject or the body. Currently the tools have to join the entire message from two parts when they are just interested in the entire message.

Regards, Constantine

On Tue, Nov 11, 2008 at 10:12:46PM +0300, Constantine Plotnikov wrote:

I don't actually use any encodings except UTF-8, so maybe there is some subtle reason not to do so that I don't understand, but I would have expected all of the format placeholders to respect any --encoding parameter.

If that is the desired behavior, this should not be too hard to make a patch for:

  1. in pretty_print_commit, move the code path for userformat to just
     after the re-encoding

  2. pass the re-encoded buffer to format_commit_message, where it will
     be put into the context struct

  3. use the re-encoded buffer in parse_commit_header

Maybe it would make a good exercise for somebody who wants to dig into git a little deeper? Volunteers?

This actually annoyed me earlier today. What got me was that '%s%n%n%b' doesn't necessarily give you the exact commit message; if it's a one-liner (i.e., body is blank), then you end up with an extra newline.

Again, this should be a pretty easy exercise to add. Volunteers?

-Peff

[re-adding list to the cc]

On Wed, Nov 12, 2008 at 02:11:46PM +0300, Constantine Plotnikov wrote:

Are there such tools? I assumed they would have complained about this as a bug before writing their own encoding conversion tools. And this is, AFAIK, the first bug report.

I don't mind playing it safe to avoid breaking other people's tools, but I'm also not excited about adding a second, "respect encoding" version of many placeholders (and it's not just %s and %b; I think you would need author and committer names and emails, too).

-Peff

On Wed, Nov 12, 2008 at 2:26 PM, Jeff King <peff@peff.net> wrote:

The reason for the request was that for IDE integration (I'm working on the IDEA plugin), we need to work with past versions of the git as well. However we could write that this is known git bug that will be fixed in some future version and just to show incorrect data in history view when non-UTF-8 encoding is used for a while. I hope that non-UTF-8 encoding for commits is indeed a rare case, so users will not complain much.

BTW for some reason --pretty=raw is affected by encoding option on the command line. And this is a bit surprising as from description of the raw format it looks like it should not be affected, because the re-encoded commit is not "the entire commit exactly as stored in the commit object". Possibly the man page should be updated to clarify this.

Regards, Constantine

"Constantine Plotnikov" <constantine.plotnikov@gmail.com> writes:

Unfortunately, that is what you get for reading from a Porcelain output, which is meant for, and are subject to improvement for, human consumption.

If you want bit-for-bit information, you can always ask "git cat-file".

On Wed, Nov 12, 2008 at 05:38:47PM -0800, Junio C Hamano wrote:

What about "git rev-list --pretty=raw"? Is that also porcelain?

I would be curious to hear your take on our failure to respect --encoding for --pretty=format. Is it a bug to be fixed, or a historical behavior to be maintained?

-Peff

Jeff King <peff@peff.net> writes:

Does it re-encode? I didn't check, but ideally it shouldn't (but I do not care too much either way, to be honest).

I think the fix you outlined was quite reasonable.

On Wed, Nov 12, 2008 at 09:10:26PM -0800, Junio C Hamano wrote:

Yes, it uses the same pretty_print_commit routine as the "log".

One thing I just realized that makes it even more reasonable: we properly munge the encoding header when we _do_ re-encode. So whether we re-encode or not, you will always get the correct encoding for what is being output via "%e". Which means that a tool which handles the current "broken" behavior by re-encoding themselves will trivially handle the new version: the output will just always be in the --encoding specified instead of whatever the original encoding was.

And if there are tools that are not looking at the output encoding (and blindly assuming --encoding works), then they are already broken by the current behavior, and we will be fixing them.

So I think it is safe to "fix" it as I described.

-Peff