git-send-email: Send with mutt(1)

> I think there's a lot of overlap between what git-send-email does and
> what mutt does, to the point that you probably don't need to use
> send-email at all.
> 
> I assume what you want out of send-email here is the actual generation
> of patch emails. But under the hood that is all done by git-format-patch
> anyway. So for example if you do:

> 
>   git format-patch --stdout origin..HEAD >patches
>   mutt -f patches
> 
> And then you can use mutt's "resend-message" function to send each one.
> I use config like this:
> 
>   macro index,pager b ":set edit_headers=yes<enter><resend-message>:set edit_headers=no<enter>"
> 
> If you're sending a long series, it's helpful to pre-populate various
> headers in the format-patch command with "--to", etc. I usually do so by
> sending the cover letter directly via mutt, and then using some perl
> hackery to convert those headers into format-patch args. The script I

> use is below (it will also, when run without a terminal, generate the
> patch summary for the cover letter; I use it with "r!my-script" while
> writing the cover letter in vim).
> 
> (This script is what I use every day, so it should be fairly robust. But
> it is also over 15 years old, so I don't promise there isn't a simpler
> way to do some of what it does ;) ).
> 
> -- >8 --
> #!/bin/sh
> upstream_branch() {
>   current=`git symbolic-ref HEAD`
>   upstream=`git for-each-ref --format='%(upstream)' "$current"`
>   if test -n "$upstream"; then
>     echo $upstream
>   else
>     echo origin
>   fi
> }
> 
> get_reply_headers() {
>   perl -ne '
>     if (defined $opt) {
>       if (/^\s+(.*)/) {
>         $val .= " $1";
>         next;
>       }
>       print "--$opt=", quotemeta($val), " ";
>       $opt = $val = undef;
>     }
>     if (/^(cc|to):\s*(.*)/i) {
>       $opt = lc($1);
>       $val = $2;
>     }
>     elsif (/^message-id:\s*(.*)/i) {
>       $opt = "in-reply-to";
>       $val = $1;
>     }
>     elsif (/^subject:\s*\[PATCH v(\d+)/i) {
>       print "-v$1 ";
>     }
>     elsif (/^$/) {
>       last;
>     }
>   '
> }
> 
> format_patch() {
>   git format-patch -s --stdout --from "$@"
> }
> 
> has_nonoption=
> for i in "$@"; do
>   case "$i" in
>     -[0-9]*) has_nonoption=yes ;;
>     -*) ;;
>      *) has_nonoption=yes
>   esac
> done
> 
> : ${REPLY:=$HOME/patch}
> test -e "$REPLY" && eval "set -- `get_reply_headers <\"$REPLY\"` \"\$@\""
> test "$has_nonoption" = "yes" || set -- "$@" `upstream_branch`
> 
> if test -t 1; then
>   format_patch "$@" >.mbox
>   mutt -e 'set sort=mailbox-order' -f .mbox
>   rm -f .mbox
> else
>   format_patch "$@" |
>   perl -lne '
>     if (/^Subject: (.*)/) {
>       $subject = $1;
>     }
>     elsif ($subject && /^\s+(.*)/) {
>       $subject .= " $1";
>     }
>     elsif ($subject) {
>       print $subject;
>       $subject = undef;
>     }
>   ' |
>   sed -e 's/\[PATCH /[/' \
>       -e 's/]/]:/' \
>       -e 's/^/  /'
>   echo
>   format_patch --cover-letter "$@" |
>   sed -ne '/|/,/^$/p; /^-- /q'
> fi

> > I assume what you want out of send-email here is the actual generation
> > of patch emails. But under the hood that is all done by git-format-patch
> > anyway. So for example if you do:
> 
> Yeah, most of it is done by format-patch.  There are few things I
> actually need from send-email.  One of them is generating the Cc from
> the sign-offs and other tags found in the patch.
> 
> I had been thinking these days that it would be useful to have
> format-patch generate those.  How about adding a --signed-off-by-cc to
> format-patch?

> > If you're sending a long series, it's helpful to pre-populate various
> > headers in the format-patch command with "--to", etc. I usually do so by
> > sending the cover letter directly via mutt, and then using some perl
> > hackery to convert those headers into format-patch args. The script I
> 
> Indeed, that hackery is what send-email already does, so how about
> moving those features a bit upstream so that format-patch can do them
> too?

> > I had been thinking these days that it would be useful to have
> > format-patch generate those.  How about adding a --signed-off-by-cc to
> > format-patch?
> 
> That seems like a reasonable feature. Probably it should be
> --cc-from-trailer=signed-off-by, and then you could do the same with
> other trailers.

> 
> It feels like you could _almost_ do it with the existing
> --format='%(trailers)' functionality, but there's no way to say "do the
> regular --format=email output, but also stick this extra format in the
> headers section". Plus there are probably some niceties you'd get from
> Git knowing that you're adding headers (like de-duping addresses).
> 
> That feature might end up somewhat hairy, though, as then you get into
> questions of parsing address lists, etc. We do all that now in perl with
> send-email, where we can lean on some parsing libraries. So I dunno.

> 
> > > If you're sending a long series, it's helpful to pre-populate various
> > > headers in the format-patch command with "--to", etc. I usually do so by
> > > sending the cover letter directly via mutt, and then using some perl
> > > hackery to convert those headers into format-patch args. The script I
> > 
> > Indeed, that hackery is what send-email already does, so how about
> > moving those features a bit upstream so that format-patch can do them
> > too?
> 
> Yeah, if they existed in format-patch I might be able to reuse them. I
> am hesitant, though, just because handling all the corner cases on
> parsing is going to be a bit of new C code.
> 
> > Although then, maybe it's simpler to teach send-email to learn to use
> > mutt(1) under the hood for the actual send.
> 
> I think you will find some corner cases in trying to make mutt act just
> like an mta accepting delivery. Two I can think of:
> 
>   1. It will take a body on stdin, but not a whole message. We can hack
>      around that with some postponed-folder magic, though.
> 
>   2. Bcc headers are stripped before sendmail sees the message (but
>      those addresses appear on the command-line). Converting that back
>      to bcc so that mutt can then re-strip them would be annoying but
>      possible. If you don't use bcc, it probably makes sense to just
>      punt on this.

> 
> So maybe a script like this:
> 
> -- >8 --
> #!/bin/sh
> 
> # ignore arguments; mutt will parse them itself
> # from to/cc headers. Note that we'll miss bcc this
> # way, but handling that would probably be kind of
> # tricky; we'd need to re-add those recipients as actual
> # bcc headers so that mutt knows how to handle them.
> 
> # spool the message to a fake mbox; we need to add
> # a "From" line to make it look legit
> trap 'rm -f to-send' 0 &&
> {
>   echo "From whatever Mon Sep 17 00:00:00 2001" &&
>   cat
> } >to-send &&

> 
> # and then have mutt "resume" it. We have to redirect
> # stdin back from the terminal, since ours is a pipe
> # with the message contents.
> mutt -p \
>   -e 'set postponed=to-send' \
>   -e 'set edit_headers=yes' \
>   </dev/tty
> -- 8< --

> 
> and then in your git config:
> 
>   [sendemail]
>   sendmailcmd = /path/to/mutt-as-mta.sh
> 
> There are mutt-specific bits there that I don't think send-email should
> have to know about. Perhaps there are generic options that send-email
> could learn, but it really feels like you'd do better teaching mutt to
> be more ready to handle this (like taking a whole message on stdin,
> headers and all, rather than just a body).
> 
> -Peff

> > That seems like a reasonable feature. Probably it should be
> > --cc-from-trailer=signed-off-by, and then you could do the same with
> > other trailers.
> 
> That would work for me.  I only suggested the other one because it's
> aleady in send-email in that form.  But yeah, it might be useful to have
> finer control.  In my case, something that CCs every mail in the trailer
> would work (Reviewed-by, Suggested-by, ...  I want them all, always).

> > # spool the message to a fake mbox; we need to add
> > # a "From" line to make it look legit
> > trap 'rm -f to-send' 0 &&
> > {
> >   echo "From whatever Mon Sep 17 00:00:00 2001" &&
> >   cat
> > } >to-send &&
> 
> Would a named pipe work?  Or maybe we could use $(mktemp)?

> Huh, this is magic sauce!  Works perfect for what I need.  This would
> need to be packaged to the masses.  :-)
> 
> I found a minor problem: If I ctrl+C within mutt(1), I expect it to
> cancel the last action, but this script intercepts the signal and exits.
> We would probably need to ignore SIGINT from mutt-as-mta.

> > > # spool the message to a fake mbox; we need to add
> > > # a "From" line to make it look legit
> > > trap 'rm -f to-send' 0 &&
> > > {
> > >   echo "From whatever Mon Sep 17 00:00:00 2001" &&
> > >   cat
> > > } >to-send &&
> > 
> > Would a named pipe work?  Or maybe we could use $(mktemp)?
> 
> I suspect mutt wants it to be a real file. But yeah, mktemp would
> definitely work. I actually started to write it that way but switched to
> a static name for simplicity in demonstrating the idea. :)
> 
> One note, though. Later we need to pass this filename to mutt config:
> 
> > > mutt -p \
> > >   -e 'set postponed=to-send' \
> 
> so it's a potential worry if "mktemp" might use a path with spaces or
> funny characters (e.g., from $TMPDIR). Probably not much of a problem in
> practice, though.
> 
> > Huh, this is magic sauce!  Works perfect for what I need.  This would
> > need to be packaged to the masses.  :-)
> > 
> > I found a minor problem: If I ctrl+C within mutt(1), I expect it to
> > cancel the last action, but this script intercepts the signal and exits.
> > We would probably need to ignore SIGINT from mutt-as-mta.
> 
> Yeah, that might make sense, and can be done with trap.

> 
> > Would you mind adding this as part of git?  Or should we suggest the
> > mutt project adding this script?
> 
> IMHO it is a little too weird and user-specific to really make sense in
> either project. It's really glue-ing together two systems. And as it's
> not something I use myself, I don't plan it moving it further along. But
> you are welcome to take what I wrote and do what you will with it,
> including submitting it to mutt.

> > > Would you mind adding this as part of git?  Or should we suggest the
> > > mutt project adding this script?
> > 
> > IMHO it is a little too weird and user-specific to really make sense in
> > either project. It's really glue-ing together two systems. And as it's
> > not something I use myself, I don't plan it moving it further along. But
> > you are welcome to take what I wrote and do what you will with it,
> > including submitting it to mutt.
> 
> I'll start by creating a git repository in my own server, and will write
> something about it to let the public know about it.  I'll also start
> requiring contributors to linux-man@ to sign their patches, and
> recommend them using this if they use mutt(1).

> On Thu, Nov 09, 2023 at 04:26:23PM +0100, Alejandro Colomar wrote:
> > I used it for sending a couple of patches to linux-man@, and it seems to
> > work.  I don't have much experience with mutt, so maybe I'm missing some
> > corner cases.  Do you expect it to not work for some case?  Otherwise,
> > we might have a winner.  :)
> 
> Since it's a Linux project, I suggest also checking out b4, which will do the
> mail sending for you as part of the contributor-oriented features:
> 
> https://b4.docs.kernel.org/en/latest/contributor/overview.html
> 
> We also provide a web relay for people who can't configure or use SMTP due to
> their company policies.
> 
> > > > Would you mind adding this as part of git?  Or should we suggest the
> > > > mutt project adding this script?
> > > 
> > > IMHO it is a little too weird and user-specific to really make sense in
> > > either project. It's really glue-ing together two systems. And as it's
> > > not something I use myself, I don't plan it moving it further along. But
> > > you are welcome to take what I wrote and do what you will with it,
> > > including submitting it to mutt.
> > 
> > I'll start by creating a git repository in my own server, and will write
> > something about it to let the public know about it.  I'll also start
> > requiring contributors to linux-man@ to sign their patches, and
> > recommend them using this if they use mutt(1).
> 
> B4 will also sign your patches for you. ;)

> There's not much
> difference, if any, in authenticability terms, but there's a big
> difference in usability terms:
> 
> To authenticate a given patch submitted to a mailing list, the receiver
> needs to also have patatt(1) configured.  Otherwise, it looks like a
> random message.

> MUAs normally don't show random headers (patatt(1)
> signs by adding the signature header), so unless one is searching for
> that header, it will be ignored.  This means, if I contribute to other
> projects, I need to tell them my patch is signed via patatt(1) in
> order for them to verify.  If instead, I sign the email as usual with my
> MUA, every MUA will recognize the signature by default and show it to
> the reader.

> Also, and I don't know if b4(1) handles this automagically, but AFAIR,
> patatt(1) didn't: fo signing a patch, I had to configure per-directory
> with `patatt install-hook`.  I have more than a hundred git worktrees
> (think of dozens of git repositories, and half a dozen worktrees --see
> git-worktree(1)-- per repository).  If I need to configure every one of
> those worktrees to sign all of my patches, that's going to be
> cumbersome.  Also, I scrape and re-create worktrees for new branches
> all the time, so I'd need to be installing hooks for patatt(1) all the
> time.  Compare that to having mutt(1) configured once.  It doesn't
> scale that well.

> On Thu, Nov 09, 2023 at 06:42:19PM +0100, Alejandro Colomar wrote:
> > I haven't yet tried b4(1), and considered trying it some time ago, but
> > really didn't find git-send-email(1) and mutt(1) so difficult to use
> > that b4(1) would simplify much.
> 
> Well, sending is only a small part of what b4 will do for you -- the core
> benefits are really cover letter management, automatic versioning and
> simplified list trailer collection. It's all tailored to kernel needs, but it
> will work for any project that depends on mailed patches.
> 
> > But I have tried patatt(1) before, which is what I think b4(1) uses for
> > signing.  Here are some of my concerns about patatt(4):
> > 
> > It doesn't sign the mail, but just the patch.
> 
> Well, no, it signs the entire thing, not just the patch, but it's true that
> it's specifically targeted at patches (hence the name).
> 
> > There's not much
> > difference, if any, in authenticability terms, but there's a big
> > difference in usability terms:
> > 
> > To authenticate a given patch submitted to a mailing list, the receiver
> > needs to also have patatt(1) configured.  Otherwise, it looks like a
> > random message.
> 
> Yes, but that's a feature.
> 
> > MUAs normally don't show random headers (patatt(1)
> > signs by adding the signature header), so unless one is searching for
> > that header, it will be ignored.  This means, if I contribute to other
> > projects, I need to tell them my patch is signed via patatt(1) in
> > order for them to verify.  If instead, I sign the email as usual with my
> > MUA, every MUA will recognize the signature by default and show it to
> > the reader.
> 
> I go into this in the FAQ for patatt:
> https://github.com/mricon/patatt#why-not-simply-pgp-sign-all-patches
> 
> Basically, developers really hated getting patches signed with PGP, either
> inline or MIME, which is why it never took off. Putting it into the header
> where it's not seen except by specialized tooling was a design choice.

> 
> > It also doesn't allow encrypting mail, so let's say I send some patch
> > fixing a security vulnerability, I'll need a custom tool to send it.  If
> > instead, I use mutt(1) to send it signed+encrypted to a mailing list
> > that provides a PGP public key, I can reuse my usual tools.
> 
> Right, the goal was really *just* attestation. For encrypted patch exchange we
> have remail (https://korg.docs.kernel.org/remail.html), which worked
> significantly better than any other alternative we've considered.
> 
> > Also, and I don't know if b4(1) handles this automagically, but AFAIR,
> > patatt(1) didn't: fo signing a patch, I had to configure per-directory
> > with `patatt install-hook`.  I have more than a hundred git worktrees
> > (think of dozens of git repositories, and half a dozen worktrees --see
> > git-worktree(1)-- per repository).  If I need to configure every one of
> > those worktrees to sign all of my patches, that's going to be
> > cumbersome.  Also, I scrape and re-create worktrees for new branches
> > all the time, so I'd need to be installing hooks for patatt(1) all the
> > time.  Compare that to having mutt(1) configured once.  It doesn't
> > scale that well.
> 
> Also true -- patatt was really envisioned as a library for b4, where you can
> configure patch signing in your ~/.gitconfig for all projects.

> I've tried something even simpler:
> 
> ---8<---
> #!/bin/sh
> 
> mutt -H -;
> --->8---
> 
> I used it for sending a couple of patches to linux-man@, and it seems to
> work.  I don't have much experience with mutt, so maybe I'm missing some
> corner cases.  Do you expect it to not work for some case?  Otherwise,
> we might have a winner.  :)

> On Thu, Nov 09, 2023 at 04:26:23PM +0100, Alejandro Colomar wrote:
> 
> > I've tried something even simpler:
> > 
> > ---8<---
> > #!/bin/sh
> > 
> > mutt -H -;
> > --->8---
> > 
> > I used it for sending a couple of patches to linux-man@, and it seems to
> > work.  I don't have much experience with mutt, so maybe I'm missing some
> > corner cases.  Do you expect it to not work for some case?  Otherwise,
> > we might have a winner.  :)
> 
> Wow, I don't know how I missed that when I read the manual. That was
> exactly the feature I was thinking that mutt would need. ;)
> 
> So yeah, that is obviously better than the "postponed" hackery I showed.
> I notice that "-H" even causes mutt to ignore "-i" (a sendmail flag that
> Git adds to sendemail.sendmailcmd). So you can just invoke it directly
> from your config like:
> 
>   git config sendemail.sendmailcmd "mutt -H -"

> 
> Annoyingly, "-E" doesn't work when reading over stdin (I guess mutt
> isn't willing to re-open the tty itself). But if you're happy with not
> editing as they go through, then "-H" is then that's enough (in my
> workflow, I do the final proofread via mutt).

> On Thu, Nov 09, 2023 at 04:26:23PM +0100, Alejandro Colomar wrote:
> 
> > I've tried something even simpler:
> > 
> > ---8<---
> > #!/bin/sh
> > 
> > mutt -H -;
> > --->8---
> > 
> > I used it for sending a couple of patches to linux-man@, and it seems to
> > work.  I don't have much experience with mutt, so maybe I'm missing some
> > corner cases.  Do you expect it to not work for some case?  Otherwise,
> > we might have a winner.  :)
> 
> Wow, I don't know how I missed that when I read the manual. That was
> exactly the feature I was thinking that mutt would need. ;)
> 
> So yeah, that is obviously better than the "postponed" hackery I showed.
> I notice that "-H" even causes mutt to ignore "-i" (a sendmail flag that
> Git adds to sendemail.sendmailcmd). So you can just invoke it directly
> from your config like:
> 
>   git config sendemail.sendmailcmd "mutt -H -"

> 
> Annoyingly, "-E" doesn't work when reading over stdin (I guess mutt
> isn't willing to re-open the tty itself). But if you're happy with not
> editing as they go through, then "-H" is then that's enough (in my
> workflow, I do the final proofread via mutt).
> 
> -Peff

> Hi Jeff,
> 
> On Thu, Nov 09, 2023 at 01:03:08PM -0500, Jeff King wrote:
> > On Thu, Nov 09, 2023 at 04:26:23PM +0100, Alejandro Colomar wrote:
> > 
> > > I've tried something even simpler:
> > > 
> > > ---8<---
> > > #!/bin/sh
> > > 
> > > mutt -H -;
> > > --->8---
> > > 
> > > I used it for sending a couple of patches to linux-man@, and it seems to
> > > work.  I don't have much experience with mutt, so maybe I'm missing some
> > > corner cases.  Do you expect it to not work for some case?  Otherwise,
> > > we might have a winner.  :)
> > 
> > Wow, I don't know how I missed that when I read the manual. That was
> > exactly the feature I was thinking that mutt would need. ;)
> > 
> > So yeah, that is obviously better than the "postponed" hackery I showed.
> > I notice that "-H" even causes mutt to ignore "-i" (a sendmail flag that
> > Git adds to sendemail.sendmailcmd). So you can just invoke it directly
> > from your config like:
> > 
> >   git config sendemail.sendmailcmd "mutt -H -"
> 
> Having it directly in sendmailcmd causes some glitch: It repeats all CCs
> in TO.  See a log:
> 
> 	Send this email? ([y]es|[n]o|[e]dit|[q]uit|[a]ll): y
> 	OK. Log says:
> 	Sendmail: mutt -H - -i kevin@8t8.us mutt-dev@mutt.org alx@kernel.org e.sovetkin@gmail.com neomutt-devel@neomutt.org
> 	From: Alejandro Colomar <alx@kernel.org>
> 	To: Kevin McCarthy <kevin@8t8.us>,
> 		mutt-dev@mutt.org
> 	Cc: Alejandro Colomar <alx@kernel.org>,
> 		Jenya Sovetkin <e.sovetkin@gmail.com>,
> 		neomutt-devel@neomutt.org
> 	Subject: [PATCH] send.c: Allow crypto operations in batch and mailx modes.
> 	Date: Fri, 10 Nov 2023 01:41:24 +0100
> 	Message-ID: <20231110004128.5972-2-alx@kernel.org>
> 	X-Mailer: git-send-email 2.42.0
> 	MIME-Version: 1.0
> 	Content-Transfer-Encoding: 8bit
> 
> 	Result: OK
> 
> The sent mail ended up being
> 
> 	From: Alejandro Colomar <alx@kernel.org>
> 	To: Kevin McCarthy <kevin@8t8.us>, mutt-dev@mutt.org, alx@kernel.org,
> 		e.sovetkin@gmail.com, neomutt-devel@neomutt.org
> 	Cc: Alejandro Colomar <alx@kernel.org>,
> 		Jenya Sovetkin <e.sovetkin@gmail.com>, neomutt-devel@neomutt.org
> 
> So maybe we need the wrapper script to ignore the arguments.

> 
> Cheers,
> Alex
> 
> > 
> > Annoyingly, "-E" doesn't work when reading over stdin (I guess mutt
> > isn't willing to re-open the tty itself). But if you're happy with not
> > editing as they go through, then "-H" is then that's enough (in my
> > workflow, I do the final proofread via mutt).
> > 
> > -Peff
> 
> -- 
> <https://www.alejandro-colomar.es/>

> > So maybe we need the wrapper script to ignore the arguments.
> 
> Heh!  The following trick works as well, without needing a script:
> 
> [sendemail]
> 	sendmailcmd = mutt -H - && true
> 
> It probably relies too much on git-send-email(1)'s current
> implementation, but it works.  :)