patch, 10 partsRe: [PATCH 01/10] Add a birdview-on-the-source-code section to the user manual

On 2007-05-14 11:21:20 -0400, J. Bruce Fields wrote:

Agh! utf8/latin1 confusion! Your mail is in latin1, but you've used the utf8 byte sequence for my name.

Hmm. Maybe I should keep quiet, so people won't start dropping my name completely just to get rid of my complaints. :-)

On Mon, May 14, 2007 at 08:19:43PM +0200, Karl Hasselström wrote:

No, I appreciate the complaint, I just don't know what to do about it--as far as I can tell, I've chosen utf-8 everywhere I can: my commits are in utf-8, and "locale" run from the shell reports everything as "en_US.UTF-8". But I suspect the problem is on my end somewhere--do I need to do something to make sure mail I send gets a header identifying it as utf-8 and not iso-8859-1? I'll investigate some more tonight if I get the chance; any advice welcomed.

--b.

"J. Bruce Fields" <bfields@fieldses.org> writes:

[...]

Probably except in your mailer's configuration then!

On 2007-05-14 14:39:31 -0400, J. Bruce Fields wrote:

Your mail headers include this:

  Content-Transfer-Encoding: QUOTED-PRINTABLE
  Content-Type: TEXT/PLAIN; charset=ISO-8859-1

but the mail body has this:

  It includes modifications as suggested by J. Bruce Fields, Karl
  Hasselstr=C3=B6m and Daniel Barkalow.

(That's a two-byte sequence for a single character, which indicates utf8 and rules out latin1.)

I guess the program that generates the e-mail (git-format-patch?) thinks it's getting latin1 input, when it's in fact getting utf8 input. This is the exact same error (or rather, the exact same symptom) that's happened once or twice the last week or so.

On Tue, May 15, 2007 at 12:22:00AM -0400, Jeff King wrote:

Hm. Yes, so if I send that patch to myself with git-send-email, I see the same thing as you:

...

...

But the mail I got through the git list yesterday has some odd stuff in it:

Received: from vger.kernel.org ([209.132.176.167])
	by fieldses.org with esmtp (Exim 4.67)
	(envelope-from <git-owner@vger.kernel.org>)
	id 1HncN6-00051C-Mh
	for bfields@fieldses.org; Mon, 14 May 2007 11:22:01 -0400
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1755729AbXENPVe (ORCPT <rfc822;bfields@fieldses.org>);
	Mon, 14 May 2007 11:21:34 -0400
X-Warning: Original message contained 8-bit characters, however during
	   the SMTP transport session the receiving system did not announce
	   capability of receiving 8-bit SMTP (RFC 1651-1653), and as this
	   message does not have MIME headers (RFC 2045-2049) to enable
	   encoding change, we had very little choice.
X-Warning: We ASSUME it is less harmful to add the MIME headers, and
	   convert the text to Quoted-Printable, than not to do so,
	   and to strip the message to 7-bits.. (RFC 1428 Appendix A)
X-Warning: We don't know what character set the user used, thus we had to
	   write these MIME-headers with our local system default value.
MIME-Version: 1.0
Content-Transfer-Encoding: QUOTED-PRINTABLE
Content-Type: TEXT/PLAIN; charset=ISO-8859-1
Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1756250AbXENPVc
	(ORCPT <rfc822;git-outgoing>); Mon, 14 May 2007 11:21:32 -0400
Received: from mail.fieldses.org ([66.93.2.214]:54954 "EHLO fieldses.org"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1755315AbXENPVb (ORCPT <rfc822;git@vger.kernel.org>);
	Mon, 14 May 2007 11:21:31 -0400
Received: from bfields by fieldses.org with local (Exim 4.67)
	(envelope-from <bfields@fieldses.org>)
	id 1HncMb-0004z0-E7; Mon, 14 May 2007 11:21:29 -0400
From:	"J. Bruce Fields" <bfields@citi.umich.edu>
To:	Junio C Hamano <junkio@cox.net>
Cc:	git@vger.kernel.org,
	Johannes Schindelin <Johannes.Schindelin@gmx.de>
Subject: [PATCH 01/10] Add a birdview-on-the-source-code section to the user manual
Date:	Mon, 14 May 2007 11:21:20 -0400
Message-Id: <11791560893572-git-send-email->
X-Mailer: git-send-email 1.5.1.4.19.g69e2
Sender:	git-owner@vger.kernel.org
Precedence: bulk
X-Mailing-List:	git@vger.kernel.org
Status: RO

Any idea how that happened?

--b.

On Tue, May 15, 2007 at 12:50:44AM -0400, J. Bruce Fields wrote:

Interesting. vger is correct in translating, since your mail server does _not_ advertise the 8BITMIME extension (even though exim is 8-bit clean, and could handle it).

However, the content-type is already specified, so it shouldn't need to
rewrite. However, I notice that your original message is missing a
MIME-Version: 1.0 header. My guess is that vger's logic is that without
that header, it can't trust the Content-Type you have provided (and
indeed, not including MIME-Version violates the MIME RFCs, I believe).

I assumed this was a bug in git-send-email, but looking closer, it doesn't put in any mime information at all! So your sending smtp server is adding in the content-type header, but it's failing to add the MIME-Version header, which I think is a bug (I can dig up the RFC reference if you want).

Arguably, git should be generating the full MIME header-set, since it knows what actual encoding the message is in.

-Peff

On Tue, 2007-05-15 at 01:08 -0400, Jeff King wrote:

Exim can advertise the 8BITMIME extension - it's turned off by default:

http://www.exim.org/exim-html-current/doc/html/spec_html/ch14.html#SECTalomo

Jeff

On Tue, May 15, 2007 at 12:57:53AM -0500, Jeffrey C. Ollie wrote:

Yes, although turning it on would just paper over the actual problem, which is that vger is rewritin the content-type header with the wrong charset. It would fix the problem for Bruce, but not for other receivers.

The real problem is (I believe) the lack of the MIME-Version header. I will do a few test messages momentarily (which will unfortunately require me spamming the list a bit).

-Peff

On 2007-05-15 01:08:08 -0400, Jeff King wrote:

You know, this rings a bell. I've discovered that a "MIME-Version: 1.0" is needed before. :-)

"stg mail" used to have the same problem, until it was changed to use the Python e-mail libraries for all that stuff. And since then I haven't had problems with it.

I very much agree.

Karl Hasselström <kha@treskal.com> writes:

If the above statement meand git-send-email by "git" I would very much agree.

On Tue, May 15, 2007 at 01:55:19AM -0700, Junio C Hamano wrote:

OK, the lack of a MIME-Version is clearly the problem, based on Karl's view of the messages I sent. I agree that git-send-email is the right place to implement this (though the weird partial mime headers are actually an artifact of Bruce's MTA).

Unfortunately, I don't think we have the encoding information any more at that point. We can infer how the patch was generated by looking at the git-config, and that should be right 99% of the time (unless the patches were generated with a different config, either from another repo or before some settings were changed).

Junio, can you confirm my understanding that:
  - if i18n.logOutputEncoding is set, then we are definitely in that
    encoding
  - otherwise, if i18n.commitEncoding is set, we should assume commits are
    in that encoding (which is just a guess, since they may have been
    generated on another config, but it's our best guess)
  - otherwise, assume utf-8

If that is OK, I will work up a patch.

Also Junio, it looks like commit 7cbcf4d5 moved parsing of the --encoding parameter into setup_revisions, but it's still being checked for in cmd_log_init. Can you confirm that the latter is now superfluous and can be removed?

-Peff

Jeff King <peff@peff.net> writes:

I do not want to break projects whose members consistently use a single non UTF-8 encoding, and I've been hoping that in such a use case they should not have to set any of these encoding configuration. So in that sense I would be somewhat reluctant to agree with the last one. But I am getting a feeling that it is a losing battle.

On the patch acceptance side, when we do _not_ have encoding information and the input does not look like a valid UTF-8, we assume that the input is latin-1 and convert it to UTF-8, if I recall correctly. If somebody sent you a patch without encoding header, and then you are forwarding that patch, not adding anything ourselves (because we do not know) and let the receiving end to do that conversion is certainly the best; but if we _were_ to add anything I would suspect it would be a better idea to use the same logic to default to latin-1 or UTF-8. East Asian users may want to raise objections here.

I think it is a reasonable compromise to do it the way you outlined. Doing it at patch generation time would fix the ambiguity issues during the step 2, so it might turn out to be necessary to add the encoding header to format-patch output after all, but send-email needs to be able to handle messages that do not have the header anyway, so probably the first step is to do so in send-email.

When we update format-patch, the ambiguity at step 2 would disappear. My gut feeling is that adding an extra header to format-patch output would not break people's workflow nor scripts (I do not think it would break mine, as I either suck in only the body of the message to my MUA or use send-email), but I am not sure.

Thanks for noticing, and I think you are right. The code parses the same input and sets the same global variable the same way.

On Tue, May 15, 2007 at 11:41:01AM -0700, Junio C Hamano wrote:

I think that is a good goal, but I think we have already failed, as git-format-patch generates content-type headers with charset=utf-8 (unless the encoding variables are set up). This code was added last year around this time (cdd406e38).

It looks like this is squelched in the presence of format.headers configuration. However, that still means they have to do _something_ to get it to work right (and I note that the fact that format.headers squelches MIME headers doesn't seem to be documented anywhere...)

As I noted in my other email, it actually _is_ there already. So the MIME-Version fix just keeps the status quo, and we've been doing it this way for a year.

Is it still worth making these guesses in send-email?

Well, I wouldn't have noticed it if you hadn't written git-log -S. :) In case you haven't fixed it yet, here it is in patch form:

-- >8 -- cmd_log_init: remove parsing of --encoding command line parameter

This was moved to the setup_revisions parsing in 7cbcf4d5, so it was never being triggered.

Signed-off-by: Jeff King <peff@peff.net>
---

On Tue, May 15, 2007 at 01:08:08AM -0400, Jeff King wrote:

Nope...

... Yes. But actually, the Content-Type header is from git-format-patch:

$ git format-patch --stdout 12806b^..12806b |head
From 12806b65b0d1faec249002c51b871775dc344a47 Mon Sep 17 00:00:00 2001
From: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Date: Thu, 10 May 2007 12:36:15 +0200
Subject: [PATCH] Add a birdview-on-the-source-code section to the user
manual
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

In http://thread.gmane.org/gmane.comp.version-control.git/42479, a birdview on the source code was requested.

So it's a git-format-patch bug?

--b.

On Tue, May 15, 2007 at 11:24:58AM -0400, J. Bruce Fields wrote:

Ah, interesting. I had checked that, but my test didn't produce those headers. It seems we only produce them if there are non-ascii characters in the commit message (and I just checked with an arbitrary commit).

So really, this (totally untested) one-liner should fix it:

Jeff King <peff@peff.net> writes:

Thanks; I think this is a sane thing to do.

On Tue, May 15, 2007 at 11:42:24AM -0700, Junio C Hamano wrote:

Do you want me to work up a commit message, or do you just want to assemble it from my other discussion?

BTW, I also checked for other places where we generate a content-type. The only other place I found was when we do multipart/mixed (log-tree.c:209), but we correctly generate the MIME-Version header there.

-Peff