threads / discuss / 27644

git imap-send converting my patches to CRLF line endings?

Subject: git imap-send converting my patches to CRLF line endings?

## tl;dr

10 messages between Jun 17, 2011 and Jun 20, 2011.

replies: 9people: 3as markdown or json

Michael Mc Donnell· Jun 17, 2011, 13:35 UTC · lore

Hi

I'm using git imap-send to send patches to wine-patches, and it seems like it converts all my patches to have CRLF line endings?

I can see it when I download the patch from the Gmail drafts folder. Git complains about white space when I apply the downloaded patch. It works fine if I just use git to create the patch and then apply it on a new branch. Is it git imap-send or just Gmail that's the problem?

Is there any way to disable the conversion?

I'm using git 1.7.5.4

Thanks, Michael Mc Donnell

Jeff King· Jun 17, 2011, 14:14 UTC · re: Michael Mc Donnell · lore

Re: git imap-send converting my patches to CRLF line endings?

On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:

> I'm using git imap-send to send patches to wine-patches, and it seems
> like it converts all my patches to have CRLF line endings?

The canonical line ending for mail is CRLF. So yes, it will convert your patch to CRLF for storage. But anything pulling it out of the IMAP folder should convert it back to native line endings.

> I can see it when I download the patch from the Gmail drafts folder.
> Git complains about white space when I apply the downloaded patch. It
> works fine if I just use git to create the patch and then apply it on
> a new branch. Is it git imap-send or just Gmail that's the problem?

How do you download and apply the patch exactly? If you are speaking imap to gmail, generally the client would strip out the CR's from the mail.

-Peff

Michael Mc Donnell· Jun 17, 2011, 14:45 UTC · re: Jeff King · lore

Re: git imap-send converting my patches to CRLF line endings?

On Fri, Jun 17, 2011 at 4:14 PM, Jeff King <peff@peff.net> wrote:

Show 8 quoted lines

> On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:
>
>> I'm using git imap-send to send patches to wine-patches, and it seems
>> like it converts all my patches to have CRLF line endings?
>
> The canonical line ending for mail is CRLF. So yes, it will convert your
> patch to CRLF for storage. But anything pulling it out of the IMAP
> folder should convert it back to native line endings.

Ok, so it's the clients responsibility to convert it back?

Show 8 quoted lines

>> I can see it when I download the patch from the Gmail drafts folder.
>> Git complains about white space when I apply the downloaded patch. It
>> works fine if I just use git to create the patch and then apply it on
>> a new branch. Is it git imap-send or just Gmail that's the problem?
>
> How do you download and apply the patch exactly? If you are speaking
> imap to gmail, generally the client would strip out the CR's from the
> mail.

I'm just downloading it with Chrome.

Steps to reproduce:
1. Upload patch via:
$ git format-patch --stdout --keep-subject --attach origin | git imap-send
2. Open Gmail in Chrome.
3. Open email in drafts folder.
4. Click attachment download link
5. Apply patch on a fresh branch with git apply.

Git complains about the white space, which indicates that the downloaded version has CRLF line endings.

I guess there's not much to do if the fault lies with Gmail?

Thanks for your reply.

Brandon Casey· Jun 17, 2011, 15:08 UTC · re: Michael Mc Donnell · lore

Re: git imap-send converting my patches to CRLF line endings?

On 06/17/2011 09:45 AM, Michael Mc Donnell wrote:

> On Fri, Jun 17, 2011 at 4:14 PM, Jeff King <peff@peff.net> wrote:
>> On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:

Show 13 quoted lines

>> How do you download and apply the patch exactly? If you are speaking
>> imap to gmail, generally the client would strip out the CR's from the
>> mail.
> 
> I'm just downloading it with Chrome.
> 
> Steps to reproduce:
> 1. Upload patch via:
> $ git format-patch --stdout --keep-subject --attach origin | git imap-send
> 2. Open Gmail in Chrome.
> 3. Open email in drafts folder.
> 4. Click attachment download link
> 5. Apply patch on a fresh branch with git apply.

                                        ^^^^^^^^^

Ok, I suspected that. The thing that you download from your gmail drafts folder is an email, not a patch. It may contain many inline patches though. You need to use 'git am' which will extract the patches from the email and apply them.

A word of caution about using imap and gmail:

Unless something has changed recently, and I don't think it has, if you _send_ the email using gmail's web interface, it will add newlines at the 72nd character, and corrupt your patch. So, even though you uploaded the patch using 'git imap-send', you still have to select it from your drafts folder and click "send" from gmail's web interface. So, gmail's imap interface is pretty useless for sending patches. You should be able to use 'git send-email' and configure gmail as your smtp server though.

-Brandon

Brandon Casey· Jun 17, 2011, 15:37 UTC · re: Brandon Casey · lore

Re: git imap-send converting my patches to CRLF line endings?

On 06/17/2011 10:08 AM, Brandon Casey wrote:

Show 13 quoted lines

> On 06/17/2011 09:45 AM, Michael Mc Donnell wrote:
>> On Fri, Jun 17, 2011 at 4:14 PM, Jeff King <peff@peff.net> wrote:
>>> On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:
> 
>>> How do you download and apply the patch exactly? If you are speaking
>>> imap to gmail, generally the client would strip out the CR's from the
>>> mail.
>>
>> I'm just downloading it with Chrome.
>>
>> Steps to reproduce:
>> 1. Upload patch via:
>> $ git format-patch --stdout --keep-subject --attach origin | git imap-send

Wait a second.  You used --attach.

>> 2. Open Gmail in Chrome.
>> 3. Open email in drafts folder.
>> 4. Click attachment download link

Then you downloaded the attachment, which should be a _patch_.

>> 5. Apply patch on a fresh branch with git apply.

Well, scratch what I said before, you were correct in using git apply.

Shouldn't the attachment have it's content preserved exactly? Maybe the fault does belong to gmail.

-Brandon

Jeff King· Jun 17, 2011, 15:50 UTC · re: Brandon Casey · lore

Re: git imap-send converting my patches to CRLF line endings?

On Fri, Jun 17, 2011 at 10:37:54AM -0500, Brandon Casey wrote:

Show 9 quoted lines

> >> $ git format-patch --stdout --keep-subject --attach origin | git imap-send
> 
> Wait a second.  You used --attach.
> 
> >> 2. Open Gmail in Chrome.
> >> 3. Open email in drafts folder.
> >> 4. Click attachment download link
> 
> Then you downloaded the attachment, which should be a _patch_.

Yeah, but if it is text/*, then according to rfc2046, it must be represented with CRLF as the line break. And especially if we are including it unencoded in a message, it is going to need CR's added.

Show 7 quoted lines

> >> 5. Apply patch on a fresh branch with git apply.
> 
> Well, scratch what I said before, you were correct in using
> git apply.
>
> Shouldn't the attachment have it's content preserved exactly?  Maybe
> the fault does belong to gmail.

Is it gmail's fault, or the browser's? If gmail is handing back a text/* content-type, then my reading of rfc2046 is that it should have CRLF line breaks. And it would be the browser's responsibility to convert to native line endings. But that's the MIME spec, and was written with mail in mind; I don't know what's normal for HTTP in these situations. But if the problem is not "strip CR" but "convert to native line endings" (which I think it is), then how could gmail know the user's native line ending preference, anyway?

-Peff

Brandon Casey· Jun 17, 2011, 16:54 UTC · re: Jeff King · lore

Re: git imap-send converting my patches to CRLF line endings?

On 06/17/2011 10:50 AM, Jeff King wrote:

Show 13 quoted lines

> On Fri, Jun 17, 2011 at 10:37:54AM -0500, Brandon Casey wrote:
> 
>>>> $ git format-patch --stdout --keep-subject --attach origin | git imap-send
>>
>> Wait a second.  You used --attach.
>>
>>>> 2. Open Gmail in Chrome.
>>>> 3. Open email in drafts folder.
>>>> 4. Click attachment download link
>>
>> Then you downloaded the attachment, which should be a _patch_.
> 
> Yeah, but if it is text/*,

It is.

Show 20 quoted lines

> then according to rfc2046, it must be
> represented with CRLF as the line break. And especially if we are
> including it unencoded in a message, it is going to need CR's added.
> 
>>>> 5. Apply patch on a fresh branch with git apply.
>>
>> Well, scratch what I said before, you were correct in using
>> git apply.
>>
>> Shouldn't the attachment have it's content preserved exactly?  Maybe
>> the fault does belong to gmail.
> 
> Is it gmail's fault, or the browser's?  If gmail is handing back a
> text/* content-type, then my reading of rfc2046 is that it should have
> CRLF line breaks.  And it would be the browser's responsibility to
> convert to native line endings.  But that's the MIME spec, and was
> written with mail in mind; I don't know what's normal for HTTP in these
> situations. But if the problem is not "strip CR" but "convert to native
> line endings" (which I think it is), then how could gmail know the
> user's native line ending preference, anyway?

So it's the same issue of line ending ambiguity that affects patches sent inline in the body of the email message. What we really want is the _original_ line ending, not necessarily the native line ending of the platform, but since any text/* content returned from or sent to the mail server must have CRLF line endings, it is impossible to determine whether or not the original content really had LF line endings or not. Currently, mailsplit chooses to assume the original line ending was LF, based on the assumption that that's the line ending that most projects use.

There doesn't seem to be any advantage to using --attach then, over just including the patch inline. Maybe attachments should always be base64 encoded? I get the eerie feeling that this topic has already been hashed to death.

-Brandon

Michael Mc Donnell· Jun 20, 2011, 10:40 UTC · re: Brandon Casey · lore

Re: git imap-send converting my patches to CRLF line endings?

On Fri, Jun 17, 2011 at 6:54 PM, Brandon Casey <brandon.casey.ctr@nrlssc.navy.mil> wrote:

Show 52 quoted lines

> On 06/17/2011 10:50 AM, Jeff King wrote:
>> On Fri, Jun 17, 2011 at 10:37:54AM -0500, Brandon Casey wrote:
>>
>>>>> $ git format-patch --stdout --keep-subject --attach origin | git imap-send
>>>
>>> Wait a second.  You used --attach.
>>>
>>>>> 2. Open Gmail in Chrome.
>>>>> 3. Open email in drafts folder.
>>>>> 4. Click attachment download link
>>>
>>> Then you downloaded the attachment, which should be a _patch_.
>>
>> Yeah, but if it is text/*,
>
> It is.
>
>> then according to rfc2046, it must be
>> represented with CRLF as the line break. And especially if we are
>> including it unencoded in a message, it is going to need CR's added.
>>
>>>>> 5. Apply patch on a fresh branch with git apply.
>>>
>>> Well, scratch what I said before, you were correct in using
>>> git apply.
>>>
>>> Shouldn't the attachment have it's content preserved exactly?  Maybe
>>> the fault does belong to gmail.
>>
>> Is it gmail's fault, or the browser's?  If gmail is handing back a
>> text/* content-type, then my reading of rfc2046 is that it should have
>> CRLF line breaks.  And it would be the browser's responsibility to
>> convert to native line endings.  But that's the MIME spec, and was
>> written with mail in mind; I don't know what's normal for HTTP in these
>> situations. But if the problem is not "strip CR" but "convert to native
>> line endings" (which I think it is), then how could gmail know the
>> user's native line ending preference, anyway?
>
> So it's the same issue of line ending ambiguity that affects patches
> sent inline in the body of the email message.  What we really want
> is the _original_ line ending, not necessarily the native line ending
> of the platform, but since any text/* content returned from or sent
> to the mail server must have CRLF line endings, it is impossible to
> determine whether or not the original content really had LF line
> endings or not.  Currently, mailsplit chooses to assume the original
> line ending was LF, based on the assumption that that's the line
> ending that most projects use.
>
> There doesn't seem to be any advantage to using --attach then, over
> just including the patch inline.  Maybe attachments should always be
> base64 encoded?  I get the eerie feeling that this topic has already
> been hashed to death.

It sounds like its a big job to implement all those RFCs correctly. Is there a library that could be used for handling imap?

Brandon Casey· Jun 17, 2011, 14:47 UTC · re: Jeff King · lore

Re: git imap-send converting my patches to CRLF line endings?

On 06/17/2011 09:14 AM, Jeff King wrote:

Show 8 quoted lines

> On Fri, Jun 17, 2011 at 03:35:04PM +0200, Michael Mc Donnell wrote:
> 
>> I'm using git imap-send to send patches to wine-patches, and it seems
>> like it converts all my patches to have CRLF line endings?
> 
> The canonical line ending for mail is CRLF. So yes, it will convert your
> patch to CRLF for storage. But anything pulling it out of the IMAP
> folder should convert it back to native line endings.

Not always. Modern thunderbird (3.1.10, is that modern? I haven't checked), saves mail using CRLF. I don't have access to gmail at the moment, but I'm pretty sure gmail does the same thing, i.e. when you select "view original", and then use your browser to "save as...".

mailsplit was modified to strip CRLF when splitting mail here:

  c2ca1d7 Allow mailsplit (and hence git-am) to handle mails with CRLF line-endings

which should have first appeared in git v1.6.5.

Show 8 quoted lines

>> I can see it when I download the patch from the Gmail drafts folder.
>> Git complains about white space when I apply the downloaded patch. It
>> works fine if I just use git to create the patch and then apply it on
>> a new branch. Is it git imap-send or just Gmail that's the problem?
> 
> How do you download and apply the patch exactly? If you are speaking
> imap to gmail, generally the client would strip out the CR's from the
> mail.

Michael, how are you applying the "email"? Are you using 'git am'? or possibly are you trying to use 'git apply'? You need to use 'git am'.

Also, as I mentioned above, you should be using git more recent than v1.6.5 so that you have a mailsplit that will strip out the CRLF line endings.

-Brandon

Jeff King· Jun 17, 2011, 15:02 UTC · re: Brandon Casey · lore

Re: git imap-send converting my patches to CRLF line endings?

On Fri, Jun 17, 2011 at 09:47:17AM -0500, Brandon Casey wrote:

Show 14 quoted lines

> > The canonical line ending for mail is CRLF. So yes, it will convert your
> > patch to CRLF for storage. But anything pulling it out of the IMAP
> > folder should convert it back to native line endings.
> 
> Not always.  Modern thunderbird (3.1.10, is that modern? I haven't
> checked), saves mail using CRLF.  I don't have access to gmail at the
> moment, but I'm pretty sure gmail does the same thing, i.e. when you
> select "view original", and then use your browser to "save as...".
> 
> mailsplit was modified to strip CRLF when splitting mail here:
> 
>   c2ca1d7 Allow mailsplit (and hence git-am) to handle mails with CRLF line-endings
> 
> which should have first appeared in git v1.6.5.

Ah, I forgot about that. I am used to unix-y tools like mutt. But it is obviously sensible for git to accept canonical mail via am, given that some clients produce it.

Show 6 quoted lines

> > How do you download and apply the patch exactly? If you are speaking
> > imap to gmail, generally the client would strip out the CR's from the
> > mail.
> 
> Michael, how are you applying the "email"?  Are you using 'git am'? or
> possibly are you trying to use 'git apply'?  You need to use 'git am'.

In another reply that crossed paths with yours, he wrote:

  5. Apply patch on a fresh branch with git apply.

So yeah, I think that is the problem.

-Peff