{"thread":{"id":"16274","subject":"Possible bug: \"git log\" ignores \"--encoding=UTF-8\" option if --pretty=format:%e%n%s%n is used","startedAt":"2008-11-11T19:12:46Z","lastAt":"2008-11-13T05:48:51Z","messageCount":8,"participants":["Constantine Plotnikov","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"95483","messageId":"85647ef50811111112o5449c12elfc571e46e607cfd0@mail.gmail.com","threadId":"16274","inReplyTo":null,"subject":"Possible bug: \"git log\" ignores \"--encoding=UTF-8\" option if --pretty=format:%e%n%s%n is used","fromName":"Constantine Plotnikov","fromEmail":"constantine.plotnikov@gmail.com","sentAt":"2008-11-11T19:12:46Z","receivedAt":"2008-11-11T19:12:46Z","isPatch":false,"sender":{"key":"constantine.plotnikov@gmail.com","avatar":null},"body":"If encoding is forced for git log with \"--encoding=UTF-8\", it seems to\nbe ineffective for the formatted output. Is it intentional?\n\nLets consider the following scenario (script is for bash, git 1.5.4.3\nand 1.6.0.2):\n\ngit init\necho initial >test.txt\ngit add test.txt\necho -e \\\\0320\\\\0242\\\\0320\\\\0265\\\\0321\\\\0201\\\\0321\\\\0202 >msg-UTF-8.txt\ngit commit -F msg-UTF-8.txt\necho updated >test.txt\ngit add test.txt\necho -e \\\\0322\\\\0345\\\\0361\\\\0362 >msg-windows-1251.txt\ngit config i18n.commitencoding Windows-1251\ngit commit -F msg-windows-1251.txt\ngit log --encoding=Windows-1251 >log1.txt\ngit log --encoding=UTF-8 >log2.txt\ngit log --encoding=Windows-1251 --pretty=format:%e%n%s%n >log3.txt\ngit log --encoding=UTF-8 --pretty=format:%e%n%s%n >log4.txt\n\nIn the both cases the string is the russian string meaning test in\ndifferent encoding.\n\nIf we compare log1.txt and log2.txt, we will see that the same\nencoding specified is used for both commits in the log.\n\nIf we compare log3.txt and log4.txt, we will see that the same the\ncontents of the file is completely identical. The native encoding is\nused for each commit. So the first listed commit\nis encoded using Windows-1251 and the second one using UTF-8. However\nin the description of the %s %b options nothing is said about which\nencoding is used and implied behavior is that they are affected by\n--encoding option.\n\nI suggest documenting that the placeholders %s and %b use native\ncommit encoding and introducing the placeholders %S and %B options\nthat use encoding specified on the command line or the default log\nencoding.\n\nI also suggest adding %g and %G placeholders (%m placeholder is\nalready occupied) that print the entire commit message instead of just\nthe subject or the body. Currently the tools have to join the entire\nmessage from two parts when they are just interested in the entire\nmessage.\n\nRegards,\nConstantine\n"},{"id":"95551","messageId":"20081112104318.GA20120@coredump.intra.peff.net","threadId":"16274","inReplyTo":"85647ef50811111112o5449c12elfc571e46e607cfd0@mail.gmail.com","subject":"Re: Possible bug: \"git log\" ignores \"--encoding=UTF-8\" option if --pretty=format:%e%n%s%n is used","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-11-12T10:43:18Z","receivedAt":"2008-11-12T10:43:18Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 11, 2008 at 10:12:46PM +0300, Constantine Plotnikov wrote:\n\n> is encoded using Windows-1251 and the second one using UTF-8. However\n> in the description of the %s %b options nothing is said about which\n> encoding is used and implied behavior is that they are affected by\n> --encoding option.\n> \n> I suggest documenting that the placeholders %s and %b use native\n> commit encoding and introducing the placeholders %S and %B options\n> that use encoding specified on the command line or the default log\n> encoding.\n\nI don't actually use any encodings except UTF-8, so maybe there is some\nsubtle reason not to do so that I don't understand, but I would have\nexpected all of the format placeholders to respect any --encoding\nparameter.\n\nIf that is the desired behavior, this should not be too hard to make a\npatch for:\n\n  1. in pretty_print_commit, move the code path for userformat to just\n     after the re-encoding\n\n  2. pass the re-encoded buffer to format_commit_message, where it will\n     be put into the context struct\n\n  3. use the re-encoded buffer in parse_commit_header\n\nMaybe it would make a good exercise for somebody who wants to dig into\ngit a little deeper? Volunteers?\n\n> I also suggest adding %g and %G placeholders (%m placeholder is\n> already occupied) that print the entire commit message instead of just\n> the subject or the body. Currently the tools have to join the entire\n> message from two parts when they are just interested in the entire\n> message.\n\nThis actually annoyed me earlier today. What got me was that '%s%n%n%b'\ndoesn't necessarily give you the exact commit message; if it's a\none-liner (i.e., body is blank), then you end up with an extra newline.\n\nAgain, this should be a pretty easy exercise to add. Volunteers?\n\n-Peff\n"},{"id":"95555","messageId":"20081112112654.GA20640@coredump.intra.peff.net","threadId":"16274","inReplyTo":"85647ef50811120311q7bc5451x7c084fd2a7864177@mail.gmail.com","subject":"Re: Possible bug: \"git log\" ignores \"--encoding=UTF-8\" option if --pretty=format:%e%n%s%n is used","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-11-12T11:26:54Z","receivedAt":"2008-11-12T11:26:54Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"[re-adding list to the cc]\n\nOn Wed, Nov 12, 2008 at 02:11:46PM +0300, Constantine Plotnikov wrote:\n\n> > I don't actually use any encodings except UTF-8, so maybe there is some\n> > subtle reason not to do so that I don't understand, but I would have\n> > expected all of the format placeholders to respect any --encoding\n> > parameter.\n> >\n> Even if this is the bug, it would be better to leave the old behavior\n> for backward compatibility reasons and introduce new placeholders.\n> Currently tools have to decode messages according to the commit\n> encoding, and changing behavior of options will break these tools\n> that have implemented workaround for this problem.\n\nAre there such tools? I assumed they would have complained about this as\na bug before writing their own encoding conversion tools. And this is,\nAFAIK, the first bug report.\n\nI don't mind playing it safe to avoid breaking other people's tools, but\nI'm also not excited about adding a second, \"respect encoding\" version\nof many placeholders (and it's not just %s and %b; I think you would\nneed author and committer names and emails, too).\n\n-Peff\n"},{"id":"95559","messageId":"85647ef50811120508j5074f892p1e2a95acfae8c0a8@mail.gmail.com","threadId":"16274","inReplyTo":"20081112112654.GA20640@coredump.intra.peff.net","subject":"Re: Possible bug: \"git log\" ignores \"--encoding=UTF-8\" option if --pretty=format:%e%n%s%n is used","fromName":"Constantine Plotnikov","fromEmail":"constantine.plotnikov@gmail.com","sentAt":"2008-11-12T13:08:38Z","receivedAt":"2008-11-12T13:08:38Z","isPatch":false,"sender":{"key":"constantine.plotnikov@gmail.com","avatar":null},"body":"On Wed, Nov 12, 2008 at 2:26 PM, Jeff King <peff@peff.net> wrote:\n> [re-adding list to the cc]\n>\n> On Wed, Nov 12, 2008 at 02:11:46PM +0300, Constantine Plotnikov wrote:\n>\n>> > I don't actually use any encodings except UTF-8, so maybe there is some\n>> > subtle reason not to do so that I don't understand, but I would have\n>> > expected all of the format placeholders to respect any --encoding\n>> > parameter.\n>> >\n>> Even if this is the bug, it would be better to leave the old behavior\n>> for backward compatibility reasons and introduce new placeholders.\n>> Currently tools have to decode messages according to the commit\n>> encoding, and changing behavior of options will break these tools\n>> that have implemented workaround for this problem.\n>\n> Are there such tools? I assumed they would have complained about this as\n> a bug before writing their own encoding conversion tools. And this is,\n> AFAIK, the first bug report.\n>\n> I don't mind playing it safe to avoid breaking other people's tools, but\n> I'm also not excited about adding a second, \"respect encoding\" version\n> of many placeholders (and it's not just %s and %b; I think you would\n> need author and committer names and emails, too).\n>\nThe reason for the request was that for IDE integration (I'm working\non the IDEA plugin), we need to work with past versions of the git as\nwell. However we could write that this is known git bug that will be\nfixed in some future version and just to show incorrect data in\nhistory view when non-UTF-8 encoding is used for a while. I hope that\nnon-UTF-8 encoding for commits is indeed a rare case, so users will\nnot complain much.\n\nBTW for some reason --pretty=raw is affected by encoding option on the\ncommand line. And this is a bit surprising as from description of the\nraw format it looks like it should not be affected, because the\nre-encoded commit is not \"the entire commit exactly as stored in the\ncommit object\". Possibly the man page should be updated to clarify\nthis.\n\nRegards,\nConstantine\n"},{"id":"95652","messageId":"7vvdusjtg8.fsf@gitster.siamese.dyndns.org","threadId":"16274","inReplyTo":"85647ef50811120508j5074f892p1e2a95acfae8c0a8@mail.gmail.com","subject":"Re: Possible bug: \"git log\" ignores \"--encoding=UTF-8\" option if --pretty=format:%e%n%s%n is used","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-11-13T01:38:47Z","receivedAt":"2008-11-13T01:38:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Constantine Plotnikov\" <constantine.plotnikov@gmail.com> writes:\n\n> BTW for some reason --pretty=raw is affected by encoding option on the\n> command line.\n\nUnfortunately, that is what you get for reading from a Porcelain output,\nwhich is meant for, and are subject to improvement for, human consumption.\n\nIf you want bit-for-bit information, you can always ask \"git cat-file\".\n"},{"id":"95662","messageId":"20081113043454.GA5048@coredump.intra.peff.net","threadId":"16274","inReplyTo":"7vvdusjtg8.fsf@gitster.siamese.dyndns.org","subject":"Re: Possible bug: \"git log\" ignores \"--encoding=UTF-8\" option if --pretty=format:%e%n%s%n is used","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-11-13T04:34:55Z","receivedAt":"2008-11-13T04:34:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 12, 2008 at 05:38:47PM -0800, Junio C Hamano wrote:\n\n> > BTW for some reason --pretty=raw is affected by encoding option on the\n> > command line.\n> \n> Unfortunately, that is what you get for reading from a Porcelain output,\n> which is meant for, and are subject to improvement for, human consumption.\n> \n> If you want bit-for-bit information, you can always ask \"git cat-file\".\n\nWhat about \"git rev-list --pretty=raw\"? Is that also porcelain?\n\nI would be curious to hear your take on our failure to respect\n--encoding for --pretty=format. Is it a bug to be fixed, or a historical\nbehavior to be maintained?\n\n-Peff\n"},{"id":"95663","messageId":"7vod0ki531.fsf@gitster.siamese.dyndns.org","threadId":"16274","inReplyTo":"20081113043454.GA5048@coredump.intra.peff.net","subject":"Re: Possible bug: \"git log\" ignores \"--encoding=UTF-8\" option if --pretty=format:%e%n%s%n is used","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-11-13T05:10:26Z","receivedAt":"2008-11-13T05:10:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> What about \"git rev-list --pretty=raw\"? Is that also porcelain?\n\nDoes it re-encode?  I didn't check, but ideally it shouldn't (but I do not\ncare too much either way, to be honest).\n\n> I would be curious to hear your take on our failure to respect\n> --encoding for --pretty=format. Is it a bug to be fixed, or a historical\n> behavior to be maintained?\n\nI think the fix you outlined was quite reasonable.\n"},{"id":"95666","messageId":"20081113054850.GB5343@coredump.intra.peff.net","threadId":"16274","inReplyTo":"7vod0ki531.fsf@gitster.siamese.dyndns.org","subject":"Re: Possible bug: \"git log\" ignores \"--encoding=UTF-8\" option if --pretty=format:%e%n%s%n is used","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-11-13T05:48:51Z","receivedAt":"2008-11-13T05:48:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 12, 2008 at 09:10:26PM -0800, Junio C Hamano wrote:\n\n> > What about \"git rev-list --pretty=raw\"? Is that also porcelain?\n> \n> Does it re-encode?  I didn't check, but ideally it shouldn't (but I do not\n> care too much either way, to be honest).\n\nYes, it uses the same pretty_print_commit routine as the \"log\".\n\n> > I would be curious to hear your take on our failure to respect\n> > --encoding for --pretty=format. Is it a bug to be fixed, or a historical\n> > behavior to be maintained?\n> \n> I think the fix you outlined was quite reasonable.\n\nOne thing I just realized that makes it even more reasonable: we\nproperly munge the encoding header when we _do_ re-encode. So whether we\nre-encode or not, you will always get the correct encoding for what is\nbeing output via \"%e\". Which means that a tool which handles the current\n\"broken\" behavior by re-encoding themselves will trivially handle the\nnew version: the output will just always be in the --encoding specified\ninstead of whatever the original encoding was.\n\nAnd if there are tools that are not looking at the output encoding (and\nblindly assuming --encoding works), then they are already broken by the\ncurrent behavior, and we will be fixing them.\n\nSo I think it is safe to \"fix\" it as I described.\n\n-Peff\n"}]}