v4, 3 partssend-email: add oauth2 support and fix outlook breaking threads

> From: Julian Swagemakers <julian@swagemakers.org>
>
> Manually send SMTP AUTH command for auth type OAUTHBEARER and XOAUTH2.
> This is necessary since they are currently not supported by the Perls
> Authen::SASL module.
>
> The bearer token needs to be passed in as the password. This can be done
> with git-credential-oauth[0] after minor modifications[1]. Which will
> allow using git send-email with Gmail and oauth2 authentication:

> On 23 Apr 2025, at 11:34 PM, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Aditya Garg <gargaditya08@live.com> writes:
> 
>> From: Julian Swagemakers <julian@swagemakers.org>
>> 
>> Manually send SMTP AUTH command for auth type OAUTHBEARER and XOAUTH2.
>> This is necessary since they are currently not supported by the Perls
>> Authen::SASL module.
>> 
>> The bearer token needs to be passed in as the password. This can be done
>> with git-credential-oauth[0] after minor modifications[1]. Which will
>> allow using git send-email with Gmail and oauth2 authentication:
> 
> I am not familiar with this area, especially with Authen::SASL, so
> I'd appreciate a second eye from other reviewers.

> 
> Having said that, the documentation is very clearly written, so is
> the above log message.
> 
> Please fold overly long lines the patch adds.  We officially aim for
> 80-column soft limit, and we allow going over it when folding lines
> to stay under it makes the result less readable. But lines added to
> the credential callback to call smtp_bearer_auth() are a way too
> wide, for example [*].
> 
> Footnote [*] The text themselves are not overly wide, but the long
> lines there are primarily due to them deeply indented.  I have to
> wonder if it is a sign that the part of the code needs to be a bit
> better refactored, e.g., by defining the callback sub defined
> elsewhere and passed to Git::credential() call as a variable that
> holds a reference to it, instead of defining an anonymous sub in
> place there, for example.

> 
> 
> > On 23 Apr 2025, at 11:34 PM, Junio C Hamano <gitster@pobox.com> wrote:
> > 
> > Aditya Garg <gargaditya08@live.com> writes:
> > 
> >> From: Julian Swagemakers <julian@swagemakers.org>
> >> 
> >> Manually send SMTP AUTH command for auth type OAUTHBEARER and XOAUTH2.
> >> This is necessary since they are currently not supported by the Perls
> >> Authen::SASL module.
> >> 
> >> The bearer token needs to be passed in as the password. This can be done
> >> with git-credential-oauth[0] after minor modifications[1]. Which will
> >> allow using git send-email with Gmail and oauth2 authentication:
> > 
> > I am not familiar with this area, especially with Authen::SASL, so
> > I'd appreciate a second eye from other reviewers.
> 
> What I've noticed is that other reviewers didn't seem to have reviewed this
> patch for more than a year when the original author proposed this patch.
> Also, oauth2 is something that is significant in today's world and is definitely
> more secure as well.
> 
> Nevertheless, your concern is quite valid, but I would also appreciate atleast
> pinging the other reviewers who might have the knowledge. 
> 
> I'm Ccing Greg, who is credited for this script and the maintainer of the perl module
> as well, with a hope to get a positive response.

> The script generates a Message-ID alongwith the other headers when
> gen_header is called, and is sent alongwith the email. For most email
> providers, including gmail, the Message-ID goes unchanged to the
> recipient.
>
> But, this does not seem to be a case with Outlook. In Outlook, when we
> send our own Message-ID as a part of the headers, it discards it. Rather
> it generates a new random Message-ID and that is was the recipient gets.

> This is a problem because the Message-ID is crucial when we are sending
> multiple emails in a thread. The current implementation for threads in
> the script replies to the Message-ID it generated, but due to Outlook's
> behavior, it is not the same as the one that the recipient got, thus
> breaking threads. So a need arises to retrieve the Message-ID from the
> server response and set it in the In-Reply-To and References email
> headers instead of using the self generated one for the purpose of
> replies.
>
> The $smtp->message variable in this script for outlook is something like
> this:
>
> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
>
> The Message-ID here is the one the receipient gets, rather than the one
> the script generated.
>
> This patch uses the fact above and retrieves the Message-ID from the
> server response. It then changes the value of the $message_id variable
> to the one received from the server. This value will be used when next
> and subsequent messages are sent as replies to the message, thus
> preserving the threading of the messages.
>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---

>  git-send-email.perl | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
>
> diff --git a/git-send-email.perl b/git-send-email.perl
> index a6cafda29c..a18e978e22 100755
> --- a/git-send-email.perl
> +++ b/git-send-email.perl
> @@ -1636,6 +1636,11 @@ sub gen_header {
>  	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
>  }
>  
> +sub is_outlook {
> +	my ($host) = @_;
> +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +}
> +
>  # Prepares the email, then asks the user what to do.
>  #
>  # If the user chooses to send the email, it's sent and 1 is returned.
> @@ -1799,6 +1804,21 @@ sub send_message {
>  			$smtp->datasend("$line") or die $smtp->message;
>  		}
>  		$smtp->dataend() or die $smtp->message;
> +
> +		# Outlook discards the Message-ID header we set while sending the email.
> +		# It instead saves it in its proprietary X-Microsoft-Original-Message-ID
> +		# header and assigns a new random Message-ID to the email. So in order to

> +		if (is_outlook($smtp_server)) {
> +			if ($smtp->message =~ /<([^>]+)>/) {
> +				$message_id = "<$1>";
> +				printf __("Outlook reassigned Message-ID to: %s\n"), $message_id;
> +			} else {
> +				warn __("Warning: Could not retrieve Message-ID from server response.\n");
> +			}
> +		}
> +
>  		$smtp->code =~ /250|200/ or die sprintf(__("Failed to send %s\n"), $subject).$smtp->message;
>  	}
>  	if ($quiet) {

> This is a problem because the Message-ID is crucial when we are sending
> multiple emails in a thread. The current implementation for threads in
> the script replies to the Message-ID it generated, but due to Outlook's
> behavior, it is not the same as the one that the recipient got, thus
> breaking threads. So a need arises to retrieve the Message-ID from the
> server response and set it in the In-Reply-To and References email
> headers instead of using the self generated one for the purpose of
> replies.

> The $smtp->message variable in this script for outlook is something like
> this:
> 
> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
> 
> The Message-ID here is the one the receipient gets, rather than the one
> the script generated.

> diff --git a/git-send-email.perl b/git-send-email.perl
> index a6cafda29c..a18e978e22 100755
> --- a/git-send-email.perl
> +++ b/git-send-email.perl
> @@ -1636,6 +1636,11 @@ sub gen_header {
>  	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
>  }
>  
> +sub is_outlook {
> +	my ($host) = @_;
> +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +}

> On 24 Apr 2025, at 4:23 AM, brian m. carlson <sandals@crustytoothpaste.net> wrote:
> 
> On 2025-04-23 at 12:19:46, Aditya Garg wrote:
>> This is a problem because the Message-ID is crucial when we are sending
>> multiple emails in a thread. The current implementation for threads in
>> the script replies to the Message-ID it generated, but due to Outlook's
>> behavior, it is not the same as the one that the recipient got, thus
>> breaking threads. So a need arises to retrieve the Message-ID from the
>> server response and set it in the In-Reply-To and References email
>> headers instead of using the self generated one for the purpose of
>> replies.
> 
> This behaviour is allowed by the standard.  It's not uncommon for
> smarthosts to replace the Message-ID header because they are responsible
> for making it unique.
> 
> I certainly don't love it and it has the possibility to break lots of
> things, as this patch demonstrates, but it is technically allowed.
> 
>> The $smtp->message variable in this script for outlook is something like
>> this:
>> 
>> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
>> 
>> The Message-ID here is the one the receipient gets, rather than the one
>> the script generated.
> 
> "recipient"
> 
>> diff --git a/git-send-email.perl b/git-send-email.perl
>> index a6cafda29c..a18e978e22 100755
>> --- a/git-send-email.perl
>> +++ b/git-send-email.perl
>> @@ -1636,6 +1636,11 @@ sub gen_header {
>>    return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
>> }
>> 
>> +sub is_outlook {
>> +    my ($host) = @_;
>> +    return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
>> +}
> 
> Are we certain that these are the only two possible values for this?  My
> worry is that we'll have some other host (or the same host with some
> other hostname) that does the same thing and then they'll have the same
> problem.  For instance, if I set my domain `smtp-outlook.example.com` to
> be a CNAME for `smtp.office365.com`, then this would fail and I'm
> concerned that we'll have corporate environments with that
> configuration.
> 
> What I would recommend here is that instead we set an option that
> controls the message ID generation.  We might have "as-is" for the
> default behaviour, "auto" to use the `is_outlook` function above to
> guess, and something like "data-response" to always use the approach
> you've written below.

> @@ -230,6 +230,14 @@ or on the command line. If a username has been specified (with
>  specified (with `--smtp-pass` or `sendemail.smtpPass`), then
>  a password is obtained using 'git-credential'.
>  
> +--smtp-passeval[=<command>]::

> diff --git a/git-send-email.perl b/git-send-email.perl
> index a18e978e22..cafb9aa43b 100755
> --- a/git-send-email.perl
> +++ b/git-send-email.perl
> @@ -59,6 +59,8 @@ sub usage {
>      --smtp-server-port      <int>  * Outgoing SMTP server port.
>      --smtp-user             <str>  * Username for SMTP-AUTH.
>      --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
> +    --smtp-passeval         <str>  * Path to script or a command to generate
> +                                     password like OAuth2 token for SMTP-AUTH.
>      --smtp-encryption       <str>  * tls or ssl; anything else disables.
>      --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
>      --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).

> On 24 Apr 2025, at 12:33 AM, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Aditya Garg <gargaditya08@live.com> writes:
> 
>> @@ -230,6 +230,14 @@ or on the command line. If a username has been specified (with
>> specified (with `--smtp-pass` or `sendemail.smtpPass`), then
>> a password is obtained using 'git-credential'.
>> 
>> +--smtp-passeval[=<command>]::
> 
> Lose the pair of [] that marks the value optional.  Compare it with,
> say, --smtp-user that is described as:

> 
>    --smtp-user=<user>::
>            Username for SMTP-AUTH. Default is ...
> 
> because they are defined in %options (below) in a similar way, like
> so:
> 
>>            "smtp-user=s" => \$smtp_authuser,
>>            "smtp-pass:s" => \$smtp_authpass,
>> +            "smtp-passeval=s" => \$smtp_authpasseval,
>>            "smtp-ssl" => sub { $smtp_encryption = 'ssl' },
> 
> taking a string value =s that is not optional.
> 
>> +    Generate password like OAuth2 token for SMTP AUTH. If specified,
>> +    it will use the output of the command specified as a password for
>> +    authentication.
>> ++
> 
>> diff --git a/git-send-email.perl b/git-send-email.perl
>> index a18e978e22..cafb9aa43b 100755
>> --- a/git-send-email.perl
>> +++ b/git-send-email.perl
>> @@ -59,6 +59,8 @@ sub usage {
>>     --smtp-server-port      <int>  * Outgoing SMTP server port.
>>     --smtp-user             <str>  * Username for SMTP-AUTH.
>>     --smtp-pass             <str>  * Password for SMTP-AUTH; not necessary.
>> +    --smtp-passeval         <str>  * Path to script or a command to generate
>> +                                     password like OAuth2 token for SMTP-AUTH.
>>     --smtp-encryption       <str>  * tls or ssl; anything else disables.
>>     --smtp-ssl                     * Deprecated. Use '--smtp-encryption ssl'.
>>     --smtp-ssl-cert-path    <str>  * Path to ca-certificates (either directory or file).
> 
> Looking good.
> 
>> +    # If smtpPassEval is set, run the user specified command to get the password
>> +    if (defined $smtp_authpasseval) {
>> +        printf __("Executing token generating script: %s\n"), $smtp_authpasseval;
>> +        chomp(my $generated_password = `$smtp_authpasseval 2>&1`);
> 
> How careful do we need to protect ourselves against a bad value in
> this variable (like "rm -rf $HOME; password-command") ?  Are we OK
> with trusting that the command line and the configuration file are
> not under control of an attacker?

>  I am assuming it is OK, but your
> folks have thought about this code path much longer than I have, so
> I thought I should ask just to make sure.
> 
> Thanks.

>> On 24 Apr 2025, at 12:33 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> 
>> Aditya Garg <gargaditya08@live.com> writes:
>> 
>>> @@ -230,6 +230,14 @@ or on the command line. If a username has been specified (with
>>> specified (with `--smtp-pass` or `sendemail.smtpPass`), then
>>> a password is obtained using 'git-credential'.
>>> 
>>> +--smtp-passeval[=<command>]::
>> 
>> Lose the pair of [] that marks the value optional.  Compare it with,
>> say, --smtp-user that is described as:
>
> The value is not optional. It doesn't make sense to leave this empty right?

> This patch series includes three changes:
>
> 1. It adds support for Oauth2 authentication, which is now compulsory by Microsoft.
>    This patch has been rebased to the latest version from the original version
>    at https://lore.kernel.org/git/20250125190131.48717-1-julian@swagemakers.org/

> On 24 Apr 2025, at 2:20 AM, M Hickford <mirth.hickford@gmail.com> wrote:
> 
> On Wed, 23 Apr 2025 at 13:29, Aditya Garg <gargaditya08@live.com> wrote:
>> This patch series includes three changes:
>> 
>> 1. It adds support for Oauth2 authentication, which is now compulsory by Microsoft.
>>   This patch has been rebased to the latest version from the original version
>>   at https://lore.kernel.org/git/20250125190131.48717-1-julian@swagemakers.org/
> 
> Fantastic!
> 
>> BTW, I am sending this series using the patched send-email by these patches from
>> Outlook!
> 
> Nice. I notice that patch v4 is a separate thread from patch v3. Any idea why?

> The script generates a Message-ID alongwith the other headers when
> gen_header is called, and is sent alongwith the email. For most email
> providers, including gmail, the Message-ID goes unchanged to the
> recipient.
> 
> But, this does not seem to be a case with Outlook. In Outlook, when we
> send our own Message-ID as a part of the headers, it discards it. Then
> it generates a new random Message-ID and that is what the recipient
> gets.
> 
> This is a problem because the Message-ID is crucial when we are sending
> multiple emails in a thread. The current implementation for threads in
> the script replies to the Message-ID it generated, but due to Outlook's
> behavior, it is not the same as the one that the recipient got, thus
> breaking threads. So a need arises to retrieve the Message-ID from the
> server response and set it in the In-Reply-To and References email
> headers instead of using the self generated one for the purpose of
> replies.
> 
> The $smtp->message variable in this script for outlook is something like
> this:
> 
> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
> 
> The Message-ID here is the one the recipient gets, rather than the one
> the script generated.
> 
> This patch uses the fact above and retrieves the Message-ID from the
> server response. It then changes the value of the $message_id variable
> to the one received from the server. This value will be used when next
> and subsequent messages are sent as replies to the message, thus
> preserving the threading of the messages.
> 
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---
>  git-send-email.perl | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
> 
> diff --git a/git-send-email.perl b/git-send-email.perl
> index 9ba47a6f38..8c8544f120 100755
> --- a/git-send-email.perl
> +++ b/git-send-email.perl
> @@ -1643,6 +1643,11 @@ sub gen_header {
>  	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
>  }
>  
> +sub is_outlook {
> +	my ($host) = @_;
> +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +}

> On Thu, Apr 24, 2025 at 07:53:54AM +0000, Aditya Garg wrote:
> > The script generates a Message-ID alongwith the other headers when
> > gen_header is called, and is sent alongwith the email. For most email
> > providers, including gmail, the Message-ID goes unchanged to the
> > recipient.
> > 
> > But, this does not seem to be a case with Outlook. In Outlook, when we
> > send our own Message-ID as a part of the headers, it discards it. Then
> > it generates a new random Message-ID and that is what the recipient
> > gets.
> > 
> > This is a problem because the Message-ID is crucial when we are sending
> > multiple emails in a thread. The current implementation for threads in
> > the script replies to the Message-ID it generated, but due to Outlook's
> > behavior, it is not the same as the one that the recipient got, thus
> > breaking threads. So a need arises to retrieve the Message-ID from the
> > server response and set it in the In-Reply-To and References email
> > headers instead of using the self generated one for the purpose of
> > replies.
> > 
> > The $smtp->message variable in this script for outlook is something like
> > this:
> > 
> > 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
> > 
> > The Message-ID here is the one the recipient gets, rather than the one
> > the script generated.
> > 
> > This patch uses the fact above and retrieves the Message-ID from the
> > server response. It then changes the value of the $message_id variable
> > to the one received from the server. This value will be used when next
> > and subsequent messages are sent as replies to the message, thus
> > preserving the threading of the messages.
> > 
> > Signed-off-by: Aditya Garg <gargaditya08@live.com>
> > ---
> >  git-send-email.perl | 21 +++++++++++++++++++++
> >  1 file changed, 21 insertions(+)
> > 
> > diff --git a/git-send-email.perl b/git-send-email.perl
> > index 9ba47a6f38..8c8544f120 100755
> > --- a/git-send-email.perl
> > +++ b/git-send-email.perl
> > @@ -1643,6 +1643,11 @@ sub gen_header {
> >  	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
> >  }
> >  
> > +sub is_outlook {
> > +	my ($host) = @_;
> > +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> > +}
> 
> No real objection here, but what about all of the company-hosted outlook
> server systems out there?  Do they need this same type of "flag"?  And
> if so, why not make it a config variable?

> Not only Outlook comes with such quirk, AFAIK the mail service that
> Tencent provides for personal usage does as well. I don't think it's a
> good idea to hardcode the problematic providers.
> 
> Not sure whether similar ideas have been proposed earlier: since this
> quirk affects only following e-mails but not the coverletter which
> doesn't have a In-reply-to field, is it possible to detect the quirk
> with the response of sending the coverletter by comparing the desired
> Message-ID and the one in response? We could throw a warning and
> automatically fixes following mails if the bad case really happens.

> This could avoid a broken thread for newcomers and should play well with
> an option introduced together for specifying dedicated behaviour.
> 
>> thanks,
>>
>> greg k-h
> 
> Please Cc me on future updates of the series, thank you Aditya.
> 
> Best regards,
> Yao Zi

> > Not only Outlook comes with such quirk, AFAIK the mail service that
> > Tencent provides for personal usage does as well. I don't think it's a
> > good idea to hardcode the problematic providers.
> > 
> > Not sure whether similar ideas have been proposed earlier: since this
> > quirk affects only following e-mails but not the coverletter which
> > doesn't have a In-reply-to field, is it possible to detect the quirk
> > with the response of sending the coverletter by comparing the desired
> > Message-ID and the one in response? We could throw a warning and
> > automatically fixes following mails if the bad case really happens.
> 
> From what I understand, what you want here is, that
> 
> 1. Irrespective of the email provider, I use the logic I am using for each mail.
> 2. I extract the message-id, and compare it with the intended one.
> 3. Give a warning and fix it.

> But, no. Outlook luckily shows the message ID in its response, and is there in
> $smtp->message. In fact you can see the whole server response here:
> 
> https://github.com/marlam/msmtp/issues/190#issuecomment-2794784869
> 
> Now say I use this logic with gmail. The $smtp->message in gmail is similar to
> outlook, but has 2 main differences:
> 
> 1. The angular brackets <> are missing.
> 2. There is no message id! Rather it has a random string of numbers and letters
>    that I also am not aware of what they mean.
> 
> So, different providers have different ways, to respond.

> In any case I don't find any way to automatically determine this. And I certainly
> don't think we can poke into receiver's email to see what message id they got.
> 
> I also don't know how corporates work. Is the SMTP server for them the same ultimately?
> Or are they using some Azure server?
> 
> Tbh Microsoft does not like following standards with Outlook. It doesn't even support
> OAUTHBEARER which is supposed to be the standard, rather uses Google's XOAUTH2. So why
> not hardcode? Do you have any other solution?> 
> > This could avoid a broken thread for newcomers and should play well with
> > an option introduced together for specifying dedicated behaviour.
> > 
> >> thanks,
> >>
> >> greg k-h
> > 
> > Please Cc me on future updates of the series, thank you Aditya.
> > 
> > Best regards,
> > Yao Zi
> 

> Some email providers like outlook allow only OAuth2 tokens to be used
> for authentication. This commit adds an option to generate OAuth2 tokens
> using scripts like M365-IMAP[1]. This option is similar to passwordeval
> in msmtp.
>
> Example usage:
>
> [sendemail]
>     smtpEncryption = tls
>     smtpServer = smtp.office365.com
>     smtpUser = someone@outlook.com
>     smtpServerPort = 587
>     smtpauth = XOAUTH2
>     smtpPassEval = cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py
>

> Something like:
> 
>     [credential "smtp://smtp.office365.com:587"]
>         username = someone@outlook.com
>         helper = "!f() { test \"$1\" = get && echo \"password=$(cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py)\"; }; f"

>> Something like:
>> 
>>     [credential "smtp://smtp.office365.com:587"]
>>         username = someone@outlook.com
>>         helper = "!f() { test \"$1\" = get && echo \"password=$(cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py)\"; }; f"
>
> Interesting, and this works too!. I wasn't aware of this.
>
> Junio, I can drop the third patch if you want.

> Aditya Garg <gargaditya08@live.com> writes:
> 
>>> Something like:
>>>
>>>     [credential "smtp://smtp.office365.com:587"]
>>>         username = someone@outlook.com
>>>         helper = "!f() { test \"$1\" = get && echo \"password=$(cd /workspaces/codespaces-blank/M365-IMAP && python3 ./refresh_token.py)\"; }; f"
>>
>> Interesting, and this works too!. I wasn't aware of this.
>>
>> Junio, I can drop the third patch if you want.
> 
> What I want does not matter in this case, no? ;-)
> 
> It does look like an additional mechanism only for this use case is
> unneeded, but it would be nice to address a related but different
> problem, which is
> 
>     What made you write the extra mechanism in the first place?
> 
> In other words, was the current documentation insufficient for you
> to realize that credential helper is an existing good solution for
> your problem and there was no need to add a new mechanism?  If so,
> there will be numerous next "you" who will also want to use OAuth2
> token from a program output and get frustrated because they cannot
> find how to do so in our documentation, no?  Would a new paragraph
> with an example like Julian gave above be something we want to add,
> or do we already have enough information there and the only thing
> users need to do is to look a bit more carefully?

> The script generates a Message-ID alongwith the other headers when
> gen_header is called, and is sent alongwith the email. For most email
> providers, including gmail, the Message-ID goes unchanged to the
> recipient.
> 
> But, this does not seem to be a case with Outlook. In Outlook, when we
> send our own Message-ID as a part of the headers, it discards it. Then
> it generates a new random Message-ID and that is what the recipient
> gets.
> 
> This is a problem because the Message-ID is crucial when we are sending
> multiple emails in a thread. The current implementation for threads in
> the script replies to the Message-ID it generated, but due to Outlook's
> behavior, it is not the same as the one that the recipient got, thus
> breaking threads. So a need arises to retrieve the Message-ID from the
> server response and set it in the In-Reply-To and References email
> headers instead of using the self generated one for the purpose of
> replies.
> 
> The $smtp->message variable in this script for outlook is something like
> this:
> 
> 2.0.0 OK <Message-ID> [Hostname=Some-hostname]
> 
> The Message-ID here is the one the recipient gets, rather than the one
> the script generated.
> 
> This patch uses the fact above and retrieves the Message-ID from the
> server response. It then changes the value of the $message_id variable
> to the one received from the server. This value will be used when next
> and subsequent messages are sent as replies to the message, thus
> preserving the threading of the messages.
> 
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---

> Hi Junio, Aditya,
>
>>
>> Authen::SASL now finally supports XOAUTH2 and OAUTHBEARER thanks to Erik
>> and Julian! (Link: https://github.com/gbarr/perl-authen-sasl/commit/958a3aa165d30cf4e3cbb36dc45306de627aa13f)
>
> And it's official: https://metacpan.org/release/EHUELS/Authen-SASL-2.1800

> >> Authen::SASL now finally supports XOAUTH2 and OAUTHBEARER thanks to Erik
> >> and Julian! (Link: https://github.com/gbarr/perl-authen-sasl/commit/958a3aa165d30cf4e3cbb36dc45306de627aa13f)
> >
> > And it's official: https://metacpan.org/release/EHUELS/Authen-SASL-2.1800
>
> Wonderful.
>
> We tend to, however, try to cater to those whose distros are slow to
> adjust to upstream changes.  What's the ETA for the updated module
> to major distros?

> On Fri, Apr 25, 2025 at 7:08 PM Junio C Hamano <gitster@pobox.com> wrote:
> 
>>>> Authen::SASL now finally supports XOAUTH2 and OAUTHBEARER thanks to Erik
>>>> and Julian! (Link: https://github.com/gbarr/perl-authen-sasl/commit/958a3aa165d30cf4e3cbb36dc45306de627aa13f)
>>>
>>> And it's official: https://metacpan.org/release/EHUELS/Authen-SASL-2.1800
>>
>> Wonderful.
>>
>> We tend to, however, try to cater to those whose distros are slow to
>> adjust to upstream changes.  What's the ETA for the updated module
>> to major distros?
> 
> To be honest, I have *no* idea. I think Debian is stabilizing Trixie
> now, so maybe it's in the one that will be after that (in 2 years?).
> 
> You could however decide to support XOAUTH2 and OAUTHBEARER only when
> they are available? Then you don't need to increase the minimum
> library requirement: there's no API difference between 2.1700 (the
> current version until today) and 2.1800 (the newly released version).
> So if you were to probe existence of Authen::SASL::Perl::XOAUTH2
> and/or Authen::SASL::Perl::OAUTHBEARER, you could conditionally
> disable the feature if the probe fails. (Using "eval { require
> Authen::SASL::Perl::XOAUTH2; 1 }" should do what you need: return
> false if the probe fails; true if it succeeds.)

> Aditya Garg <gargaditya08@live.com> writes:
> 
>> +sub is_outlook {
>> +	my ($host) = @_;
>> +	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
>> +}
> 
> There were a few messages that raised concerns with respect to
> on-prem installations of Outlook based servers, where the hostname
> cannot be used to tell if we need this message-id tweaking.
> 
> The following is a completely untested patch, but it should be
> sufficient to illustrate how simple it would be to support an
> option to do so, if we cared about the issue enough.
> 
> Will queue your patch _without_ this tweak, at least for now.

> There were a few messages that raised concerns with respect to
> on-prem installations of Outlook based servers, where the hostname
> cannot be used to tell if we need this message-id tweaking.
> 
> The following is a completely untested patch, but it should be
> sufficient to illustrate how simple it would be to support an
> option to do so, if we cared about the issue enough.

> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
>
> Co-authored-by: Aditya Garg <gargaditya08@live.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---
> diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
> @@ -421,6 +421,21 @@ recipient's MUA.
> +--[no-]smtp-outlook-id-tweak::
> +       Outlook servers discard the Message-ID sent via email and assign a
> +       new random Message-ID, thus breaking threads.
> ++
> +--
> +- '--smtp-outlook-id-tweak' will attempt to retrieve the ID from the server
> +  irrespective of the SMTP server being used. Use only if Microsoft is your
> +  email provider.
> +- '--no-smtp-outlook-id-tweak' will disable this tweak irrespective of the
> +  SMTP server being used.
> +--
> ++
> +If not sepcified, the default behaviour will be to enable the tweak only if the
> +SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.

> diff --git a/git-send-email.perl b/git-send-email.perl
> @@ -60,6 +60,8 @@ sub usage {
>      --smtp-encryption       <str>  * tls or ssl; anything else disables.
> +    --smtp-outlook-id-tweak <0|1>  * This server munges Message-ID. Retrive it from
> +                                     the server and assign to \$message_id.

>> Add an option to allow users to specifically enable or disable
>> retrieving the Message-ID from the Outlook SMTP server. This can be
>> for other hosts mimicking the behaviour of Outlook, or for users who set
>> a custom domain to be a CNAME for the Outlook SMTP server.
>>
>> Co-authored-by: Aditya Garg <gargaditya08@live.com>
>> Signed-off-by: Aditya Garg <gargaditya08@live.com>
>> ---
>> diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
>> @@ -421,6 +421,21 @@ recipient's MUA.
>> +--[no-]smtp-outlook-id-tweak::
>> +       Outlook servers discard the Message-ID sent via email and assign a
>> +       new random Message-ID, thus breaking threads.
>> ++
>> +--
>> +- '--smtp-outlook-id-tweak' will attempt to retrieve the ID from the server
>> +  irrespective of the SMTP server being used. Use only if Microsoft is your
>> +  email provider.
>> +- '--no-smtp-outlook-id-tweak' will disable this tweak irrespective of the
>> +  SMTP server being used.
>> +--
>> ++
>> +If not sepcified, the default behaviour will be to enable the tweak only if the
>> +SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
> 
> s/sepcified/specified/
> 
>> diff --git a/git-send-email.perl b/git-send-email.perl
>> @@ -60,6 +60,8 @@ sub usage {
>>      --smtp-encryption       <str>  * tls or ssl; anything else disables.
>> +    --smtp-outlook-id-tweak <0|1>  * This server munges Message-ID. Retrive it from
>> +                                     the server and assign to \$message_id.
> 
> s/Retrive/Retrieve/
> 
> As this is a user-facing help message, it seems unusual and unhelpful
> for it to be talking about a variable ($message_id) which is internal
> to the script. I realize that there is a slight precedent in which the
> help for the --foo-cmd options talk about $patch_path, but those cases
> are semantically different. Thus, it probably would be better to drop
> mention of $message_id and just present a short and sweet explanation,
> such as:
> 
>     Retrieve Message-ID from the server.

> I tested this patch and it works well. I've added some docs as
> well and it now looks like the one below.
>
> --->8---
> From e06ccabb5a0cef100e50e2b9d6d3c0a1769bda59 Mon Sep 17 00:00:00 2001
> From: Aditya Garg <gargaditya08@live.com>
> Date: Sat, 26 Apr 2025 08:25:25 +0000
> Subject: [PATCH] send-email: add --[no-]smtp-outlook-id-tweak option
>
> From: Junio C Hamano <gitster@pobox.com>
>
> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
>
> Co-authored-by: Aditya Garg <gargaditya08@live.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---

> +		$outlook_id_tweak =
> +			($host eq 'smtp.office365.com' ||
> +			 $host eq 'smtp-mail.outlook.com') ? 1 : 0;
> +	}	return $outlook_id_tweak;
>  }

> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be used
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
>
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---
> v2: Replace tab with spaces in "outlookidtweak" => \$outlook_id_tweak,

> +--smtp-outlook-id-tweak=(always|never|auto)::
> +	Outlook servers discard the Message-ID sent via email and assign a
> +	new random Message-ID, thus breaking threads.
> ++
> +--
> +- 'auto' will attempt to retrieve the ID from the server only if the SMTP
> +  server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
> +- 'always' will attempt to retrieve the ID from the server irrespective of
> +  the SMTP server being used. Use only if Microsoft is your email provider.
> +- 'never' will disable this tweak irrespective of theSMTP server being used.

> On 29 Apr 2025, at 2:17 AM, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Aditya Garg <gargaditya08@live.com> writes:
> 
>> Add an option to allow users to specifically enable or disable
>> retrieving the Message-ID from the Outlook SMTP server. This can be used
>> for other hosts mimicking the behaviour of Outlook, or for users who set
>> a custom domain to be a CNAME for the Outlook SMTP server.
>> 
>> Helped-by: Junio C Hamano <gitster@pobox.com>
>> Signed-off-by: Aditya Garg <gargaditya08@live.com>
>> ---
>> v2: Replace tab with spaces in "outlookidtweak" => \$outlook_id_tweak,
> 
> Good eyes ;-)
> 
>> +--smtp-outlook-id-tweak=(always|never|auto)::
>> +    Outlook servers discard the Message-ID sent via email and assign a
>> +    new random Message-ID, thus breaking threads.
>> ++
>> +--
>> +- 'auto' will attempt to retrieve the ID from the server only if the SMTP
>> +  server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
>> +- 'always' will attempt to retrieve the ID from the server irrespective of
>> +  the SMTP server being used. Use only if Microsoft is your email provider.
>> +- 'never' will disable this tweak irrespective of theSMTP server being used.
> 
> It is a shame that this is not just a simple boolean.  Those who
> expect 'true' to kick in would be disappointed to find that their
> 'true' means the same thing as 'auto'.
> 
> If I were designing this feature, I would rather make it say:
> 
>    --[no-]outlook-message-id-fix::

> 
>    Outlook servers [DO THIS].  Giving this option reads the
>    message-id assigned by the Outlook server and use it as the
>    In-Reply-To message ID for subsequent messages.  Without the
>    option, connections to only ... and ... automatically gets
>    this tweak.  Pass `--no-outlook-message-id-fix` to disable
>    the fix even for these hosts.
> 
> and make the implementation behave that way.
> 
> Note that this is a command line option that is specific to
> git-send-email, so I omitted "smtp" from the name and instead
> replaced 'id' with 'message-id' to make it more explicit what
> gets munged.
> 
> But I do not care too deeply either way.  Let me queue the patch
> as-is for now.
> 
> Thanks.

> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be used
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
> 
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---

>  sub is_outlook {
>  	my ($host) = @_;
> -	return ($host eq 'smtp.office365.com' || $host eq 'smtp-mail.outlook.com');
> +	if ($outlook_id_fix eq 'auto') {
> +		$outlook_id_fix =
> +			($host eq 'smtp.office365.com' ||
> +			 $host eq 'smtp-mail.outlook.com') ? 1 : 0;
> +	}	return $outlook_id_fix;

> +--[no-]outlook-id-fix::
> +	Outlook servers discard the Message-ID sent via email and assign a
> +	new random Message-ID, thus breaking threads.
> ++
> +--
> +- '--outlook-id-fix' will attempt to retrieve the ID from the server
> +  irrespective of the SMTP server being used. Use only if Microsoft is your
> +  email provider.
> +- '--no-outlook-id-fix' will disable this tweak irrespective of the SMTP
> +  server being used.
> +--
> ++
> +If not specified, the default behaviour will be to enable the tweak only if the
> +SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.

> Aditya Garg <gargaditya08@live.com> writes:
> 
>> +--[no-]outlook-id-fix::
>> +	Outlook servers discard the Message-ID sent via email and assign a
>> +	new random Message-ID, thus breaking threads.
>> ++
>> +--
>> +- '--outlook-id-fix' will attempt to retrieve the ID from the server
>> +  irrespective of the SMTP server being used. Use only if Microsoft is your
>> +  email provider.
>> +- '--no-outlook-id-fix' will disable this tweak irrespective of the SMTP
>> +  server being used.
>> +--
>> ++
>> +If not specified, the default behaviour will be to enable the tweak only if the
>> +SMTP server is 'smtp.office365.com' or 'smtp-mail.outlook.com'.
> 
> I found "irrespective of the SMTP server being used" part a bit
> confusing, which seemed to contradict with the next sentence to tell
> you to use it only against Outlook based servers.
> 
> After reading the "If not specified" part, I can sort of guess that
> you wanted to "irrespective" mean "even when we are (or are not)
> talking to outlook.com", but it is still a confusing description.
> 
> Would the following work better, I wonder...?
> 
> 	Microsoft Outlook SMTP servers ..., thus breaking threads.
> 
> 	With `--outlook-id-fix`, "git send-email" uses a mechanism
> 	specific to Outlook servers to learn the Message-ID the
> 	server assigned to fix the threading.  Use it only when you
> 	know that the server reports the rewritten Message-ID the
> 	same way as Outlook servers do.
> 
> 	Without this option specified, the fix is done by default
> 	when talking to smtp.office365.com or smtp-mail.outlook.com.
> 	Use `--no-outlook-id-fix` to disable even when talking to
> 	these two servers.

> Add an option to allow users to specifically enable or disable
> retrieving the Message-ID from the Outlook SMTP server. This can be used
> for other hosts mimicking the behaviour of Outlook, or for users who set
> a custom domain to be a CNAME for the Outlook SMTP server.
>
> While at it, lets also add missing * in description of --no-smtp-auth.
>
> Helped-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Aditya Garg <gargaditya08@live.com>
> ---
> v2: Replace tab with spaces in "outlookidtweak" => \$outlook_id_tweak,
> v3: Rename to --[no-]outlook-id-fix and make it bool. Also add missing
>     * in description of --no-smtp-auth.
> v4: Improve documentation and the log message. Also, have the final
>     "return $outlook_id_fix;" on its own line.
>
>  Documentation/git-send-email.adoc | 13 +++++++++++++
>  git-send-email.perl               | 14 ++++++++++++--
>  2 files changed, 25 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/git-send-email.adoc b/Documentation/git-send-email.adoc
> index 7f223db42d..7ccca16296 100644
> --- a/Documentation/git-send-email.adoc
> +++ b/Documentation/git-send-email.adoc
> @@ -115,6 +115,19 @@ illustration below where `[PATCH v2 0/3]` is in reply to `[PATCH 0/2]`:
>  Only necessary if --compose is also set.  If --compose
>  is not set, this will be prompted for.
>  
> +--[no-]outlook-id-fix::
> +	Microsoft Outlook SMTP servers discard the Message-ID sent via email and
> +	assign a new random Message-ID, thus breaking threads.
> +
> +	With `--outlook-id-fix`, 'git send-email' uses a mechanism specific to
> +	Outlook servers to learn the Message-ID the server assigned to fix the
> +	threading. Use it only when you know that the server reports the
> +	rewritten Message-ID the same way as Outlook servers do.
> +
> +	Without this option specified, the fix is done by default when talking
> +	to 'smtp.office365.com' or 'smtp-mail.outlook.com'. Use
> +	`--no-outlook-id-fix` to disable even when talking to these two servers.

> 
> Please let me know who to talk to to have it added!
> 
> 
> -- 
> Bye,
> 
> Erik.
> 
> http://efficito.com <http://efficito.com/> -- Hosted accounting and ERP.
> Robust and Flexible. No vendor lock-in.

>
> Hi Eric
>
> > As I said in https://github.com/gbarr/perl-authen-sasl/issues/18#issuecomment-2453040190 <https://github.com/gbarr/perl-authen-sasl/issues/18#issuecomment-2453040190> : I'd love to implement XOAUTH2 and/or OAUTHBEARER, but I don't have a setup available to test with, so I need someone to provide a patch *and* a description of how to test, preferably against a publicly available service so I don't need to go through complex setup myself.
>
> I saw the code of perl-authen-sasl and I miserably failed in porting the logic used here to the repo.
>
> Perl is not something I am very strong at, so not really feasible for me to work ahead. I do was able to return the formatted base64 encoded string as per the logic, but still I got authentication issues.

> Julian, you might be interested here?
>
> If you are interested, you can checkout my XOAUTH2.pm file, which doesn't authenticate for some reason.
>
> -->8--
>
> package Authen::SASL::Perl::XOAUTH2;
>
> use strict;
> use warnings;
> use MIME::Base64;
> use vars qw($VERSION @ISA);
>
> $VERSION = "1.00";
> @ISA     = qw(Authen::SASL::Perl);
>
> my %secflags = (
>     noanonymous => 1,
> );
>
> sub _order { 1 }
>
> sub _secflags {
>     shift;
>     scalar grep { $secflags{$_} } @_;
> }
>
> sub mechanism { 'XOAUTH2' }
>
> sub client_start {
>     my $self = shift;
>     $self->{stage} = 0;
>     '';
> }
>
> sub client_step {
>     my ($self, $challenge) = @_;
>
>     my $stage = ++$self->{stage};
>     if ($stage == 1) {

>         # Generate the XOAUTH2 authentication string
>         my $username = $self->_call('user');
>         my $token    = $self->_call('pass'); # OAuth 2.0 access token
>         my $auth_string = "user=$username\001auth=Bearer $token\001\001";
>         my $encoded_auth_string = encode_base64($auth_string, '');
>
>         # Send the encoded authentication string
>         return $encoded_auth_string;

>     } else {
>         # Handle authentication failure by sending a dummy request
>         my $dummy_request = encode_base64("\001", '');
>         return $dummy_request;
>     }
> }
> }
>
> 1;
>
> __END__
>
> =head1 NAME
>
> Authen::SASL::Perl::XOAUTH2 - XOAUTH2 Authentication class
>
> =head1 SYNOPSIS
>
>   use Authen::SASL qw(Perl);
>
>   $sasl = Authen::SASL->new(
>     mechanism => 'XOAUTH2',
>     callback  => {
>       user => $user,
>       pass => $access_token,
>     },
>   );
>
> =head1 DESCRIPTION
>
> This module implements the client side of the XOAUTH2 SASL mechanism, which is used for OAuth 2.0-based authentication.
>
> =head2 CALLBACK
>
> The callbacks used are:
>
> =head3 Client
>
> =over 4
>
> =item user
>
> The username to be used for authentication.
>
> =item pass
>
> The OAuth 2.0 access token to be used for authentication.
>
> =back
>
> =head1 SEE ALSO
>
> L<Authen::SASL>,
> L<MIME::Base64>
>
> =head1 AUTHORS
>
> Written by [Your Name].
>
> =head1 COPYRIGHT
>
> This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
>
> =cut
>
> ----
> >
> > Please let me know who to talk to to have it added!
> >
> >
> > --
> > Bye,
> >
> > Erik.
> >
> > http://efficito.com <http://efficito.com/> -- Hosted accounting and ERP.
> > Robust and Flexible. No vendor lock-in.
>

> I saw the code of perl-authen-sasl and I miserably failed in porting
> the logic used here to the repo.
>
> Perl is not something I am very strong at, so not really feasible for
> me to work ahead. I do was able to return the formatted base64 encoded
> string as per the logic, but still I got authentication issues.
> Currently I aim to have git-send-email working, and a review from an
> experienced person is needed.
>
> I would be happy to be able to test any proposed patch though.
>
> Julian, you might be interested here?

> On 25 Apr 2025, at 11:49 AM, Julian Swagemakers <julian@swagemakers.org> wrote:
> 
> On Thu Apr 24, 2025 at 8:22 PM CEST, Aditya Garg wrote:
>> I saw the code of perl-authen-sasl and I miserably failed in porting
>> the logic used here to the repo.
>> 
>> Perl is not something I am very strong at, so not really feasible for
>> me to work ahead. I do was able to return the formatted base64 encoded
>> string as per the logic, but still I got authentication issues.
>> Currently I aim to have git-send-email working, and a review from an
>> experienced person is needed.
>> 
>> I would be happy to be able to test any proposed patch though.
>> 
>> Julian, you might be interested here?
> 
> I fear I'm also a rookie when it comes to perl, but with your code and
> the comment from Erik I managed to get it working. client_start and
> client_step have to look like this:
> 
> sub client_start {
>    my $self = shift;
>    $self->{stage} = 0;
>    # Generate the XOAUTH2 authentication string
>    my $username = $self->_call('user');
>    my $token    = $self->_call('pass'); # OAuth 2.0 access token
>    my $auth_string = "user=$username\001auth=Bearer $token\001\001";
>    return $auth_string
> }
> 
> sub client_step {
>    my ($self, $challenge) = @_;
> 
>    my $stage = ++$self->{stage};
> 
>    if ($stage == 1) {
>        # Handle authentication failure by sending a dummy request
>        return "\001"
>    } else {
>        return $self->set_error("Invalid sequence");
>    }
> }
> 
> Aditya, do you want to create a pull request on perl-authen-sasl, or
> should I?

> 
> Please let me know who to talk to to have it added!
> 
> 
> -- 
> Bye,
> 
> Erik.
> 
> http://efficito.com <http://efficito.com/> -- Hosted accounting and ERP.
> Robust and Flexible. No vendor lock-in.