{"thread":{"id":"52015","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","startedAt":"2019-10-10T19:41:38Z","lastAt":"2019-10-13T11:06:00Z","messageCount":15,"participants":["Jonathan Nieder","Andrew Donnellan","Jeff King","Daniel Axtens","Stephen Rothwell","Junio C Hamano","Christian Schoenebeck","Ian Kelling"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"383833","messageId":"20191010194132.GA191800@google.com","threadId":"52015","inReplyTo":"20191010062047.21549-1-ajd@linux.ibm.com","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2019-10-10T19:41:32Z","receivedAt":"2019-10-10T19:41:38Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nAndrew Donnellan wrote:\n\n> To avoid triggering spam filters due to failed signature validation, many\n> mailing lists mangle the From header to change the From address to be the\n> address of the list, typically where the sender's domain has a strict DMARC\n> policy enabled.\n>\n> In this case, we should try to unmangle the From header.\n>\n> Add support for using the X-Original-Sender or Reply-To headers, as used by\n> Google Groups and Mailman respectively, to unmangle the From header when\n> necessary.\n>\n> Closes: #64 (\"Incorrect submitter when using googlegroups\")\n> Reported-by: Alexandre Belloni \n> Reported-by: Stephen Rothwell \n> Signed-off-by: Andrew Donnellan \n> ---\n> patchwork/parser.py | 75 ++++++++++++++++++++++++++++++----\n> patchwork/tests/test_parser.py | 68 ++++++++++++++++++++++++++++--\n> 2 files changed, 130 insertions(+), 13 deletions(-)\n\nInteresting! I'm cc-ing the Git mailing list in case \"git am\" might\nwnat to learn the same support.\n\nThanks,\nJonathan\n\n(patch left unsnipped for reference)\n> diff --git a/patchwork/parser.py b/patchwork/parser.py\n> index 7dc66bc05a5b..79beac5617b1 100644\n> --- a/patchwork/parser.py\n> +++ b/patchwork/parser.py\n> @@ -321,12 +321,7 @@ def find_series(project, mail, author):\n> return _find_series_by_markers(project, mail, author)\n> \n> \n> -def get_or_create_author(mail):\n> - from_header = clean_header(mail.get('From'))\n> -\n> - if not from_header:\n> - raise ValueError(\"Invalid 'From' header\")\n> -\n> +def split_from_header(from_header):\n> name, email = (None, None)\n> \n> # tuple of (regex, fn)\n> @@ -355,6 +350,65 @@ def get_or_create_author(mail):\n> (name, email) = fn(match.groups())\n> break\n> \n> + return (name, email)\n> +\n> +\n> +# Unmangle From addresses that have been mangled for DMARC purposes.\n> +#\n> +# To avoid triggering spam filters due to failed signature validation, many\n> +# mailing lists mangle the From header to change the From address to be the\n> +# address of the list, typically where the sender's domain has a strict\n> +# DMARC policy enabled.\n> +#\n> +# Unfortunately, there's no standardised way of preserving the original\n> +# From address.\n> +#\n> +# Google Groups adds an X-Original-Sender header. If present, we use that.\n> +#\n> +# Mailman preserves the original address by adding a Reply-To, except in the\n> +# case where the list is set to either reply to list, or reply to a specific\n> +# address, in which case the original From is added to Cc instead. These corner\n> +# cases are dumb, but we try and handle things as sensibly as possible by\n> +# looking for a name in Reply-To/Cc that matches From. It's inexact but should\n> +# be good enough for our purposes.\n> +def get_original_sender(mail, name, email):\n> + if name and ' via ' in name:\n> + # Mailman uses the format \" via \"\n> + # Google Groups uses \"'' via \"\n> + stripped_name = name[:name.rfind(' via ')].strip().strip(\"'\")\n> +\n> + original_sender = clean_header(mail.get('X-Original-Sender', ''))\n> + if original_sender:\n> + new_email = split_from_header(original_sender)[1].strip()[:255]\n> + return (stripped_name, new_email)\n> +\n> + addrs = []\n> + reply_to_headers = mail.get_all('Reply-To') or []\n> + cc_headers = mail.get_all('Cc') or []\n> + for header in reply_to_headers + cc_headers:\n> + header = clean_header(header)\n> + addrs = header.split(\",\")\n> + for addr in addrs:\n> + new_name, new_email = split_from_header(addr)\n> + if new_name:\n> + new_name = new_name.strip()[:255]\n> + if new_email:\n> + new_email = new_email.strip()[:255]\n> + if new_name == stripped_name:\n> + return (stripped_name, new_email)\n> +\n> + # If we can't figure out the original sender, just keep it as is\n> + return (name, email)\n> +\n> +\n> +def get_or_create_author(mail, project=None):\n> + from_header = clean_header(mail.get('From'))\n> +\n> + if not from_header:\n> + raise ValueError(\"Invalid 'From' header\")\n> +\n> + name, email = split_from_header(from_header)\n> +\n> if not email:\n> raise ValueError(\"Invalid 'From' header\")\n> \n> @@ -362,6 +416,9 @@ def get_or_create_author(mail):\n> if name is not None:\n> name = name.strip()[:255]\n> \n> + if project and email.lower() == project.listemail.lower():\n> + name, email = get_original_sender(mail, name, email)\n> +\n> # this correctly handles the case where we lose the race to create\n> # the person and another process beats us to it. (If the record\n> # does not exist, g_o_c invokes _create_object_from_params which\n> @@ -1004,7 +1061,7 @@ def parse_mail(mail, list_id=None):\n> \n> if not is_comment and (diff or pull_url): # patches or pull requests\n> # we delay the saving until we know we have a patch.\n> - author = get_or_create_author(mail)\n> + author = get_or_create_author(mail, project)\n> \n> delegate = find_delegate_by_header(mail)\n> if not delegate and diff:\n> @@ -1099,7 +1156,7 @@ def parse_mail(mail, list_id=None):\n> is_cover_letter = True\n> \n> if is_cover_letter:\n> - author = get_or_create_author(mail)\n> + author = get_or_create_author(mail, project)\n> \n> # we don't use 'find_series' here as a cover letter will\n> # always be the first item in a thread, thus the references\n> @@ -1159,7 +1216,7 @@ def parse_mail(mail, list_id=None):\n> if not submission:\n> return\n> \n> - author = get_or_create_author(mail)\n> + author = get_or_create_author(mail, project)\n> \n> try:\n> comment = Comment.objects.create(\n> diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py\n> index e5a2fca3044e..85c6e7550f93 100644\n> --- a/patchwork/tests/test_parser.py\n> +++ b/patchwork/tests/test_parser.py\n> @@ -265,11 +265,23 @@ class SenderCorrelationTest(TestCase):\n> \"\"\"\n> \n> @staticmethod\n> - def _create_email(from_header):\n> + def _create_email(from_header, reply_tos=None, ccs=None,\n> + x_original_sender=None):\n> mail = 'Message-Id: %s\\n' % make_msgid() + \\\n> - 'From: %s\\n' % from_header + \\\n> - 'Subject: Tests\\n\\n'\\\n> - 'test\\n'\n> + 'From: %s\\n' % from_header\n> +\n> + if reply_tos:\n> + mail += 'Reply-To: %s\\n' % ', '.join(reply_tos)\n> +\n> + if ccs:\n> + mail += 'Cc: %s\\n' % ', '.join(ccs)\n> +\n> + if x_original_sender:\n> + mail += 'X-Original-Sender: %s\\n' % x_original_sender\n> +\n> + mail += 'Subject: Tests\\n\\n'\\\n> + 'test\\n'\n> +\n> return message_from_string(mail)\n> \n> def test_existing_sender(self):\n> @@ -311,6 +323,54 @@ class SenderCorrelationTest(TestCase):\n> self.assertEqual(person_b._state.adding, False)\n> self.assertEqual(person_b.id, person_a.id)\n> \n> + def test_mailman_dmarc_munging(self):\n> + project = create_project()\n> + real_sender = 'Existing Sender '\n> + munged_sender = 'Existing Sender via List <{}>'.format(\n> + project.listemail)\n> + other_email = 'Other Person '\n> +\n> + # Unmunged author\n> + mail = self._create_email(real_sender)\n> + person_a = get_or_create_author(mail, project)\n> + person_a.save()\n> +\n> + # Single Reply-To\n> + mail = self._create_email(munged_sender, [real_sender])\n> + person_b = get_or_create_author(mail, project)\n> + self.assertEqual(person_b._state.adding, False)\n> + self.assertEqual(person_b.id, person_a.id)\n> +\n> + # Single Cc\n> + mail = self._create_email(munged_sender, [], [real_sender])\n> + person_b = get_or_create_author(mail, project)\n> + self.assertEqual(person_b._state.adding, False)\n> + self.assertEqual(person_b.id, person_a.id)\n> +\n> + # Multiple Reply-Tos and Ccs\n> + mail = self._create_email(munged_sender, [other_email, real_sender],\n> + [other_email, other_email])\n> + person_b = get_or_create_author(mail, project)\n> + self.assertEqual(person_b._state.adding, False)\n> + self.assertEqual(person_b.id, person_a.id)\n> +\n> + def test_google_dmarc_munging(self):\n> + project = create_project()\n> + real_sender = 'Existing Sender '\n> + munged_sender = \"'Existing Sender' via List <{}>\".format(\n> + project.listemail)\n> +\n> + # Unmunged author\n> + mail = self._create_email(real_sender)\n> + person_a = get_or_create_author(mail, project)\n> + person_a.save()\n> +\n> + # X-Original-Sender header\n> + mail = self._create_email(munged_sender, None, None, real_sender)\n> + person_b = get_or_create_author(mail, project)\n> + self.assertEqual(person_b._state.adding, False)\n> + self.assertEqual(person_b.id, person_a.id)\n> +\n> \n> class SeriesCorrelationTest(TestCase):\n> \"\"\"Validate correct behavior of find_series.\"\"\"\n"},{"id":"383840","messageId":"236b9c32-b501-79a7-8366-26d64ceac24f@linux.ibm.com","threadId":"52015","inReplyTo":"20191010194132.GA191800@google.com","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Andrew Donnellan","fromEmail":"ajd@linux.ibm.com","sentAt":"2019-10-10T21:13:01Z","receivedAt":"2019-10-10T21:13:59Z","isPatch":true,"sender":{"key":"ajd@linux.ibm.com","avatar":null},"body":"On 11/10/19 6:41 am, Jonathan Nieder wrote:\n> Interesting! I'm cc-ing the Git mailing list in case \"git am\" might\n> wnat to learn the same support.\nArgh, that reminds me... this patch only rewrites the name and email \nthat is recorded as the Patchwork submitter, it doesn't actually rewrite \nthe From: header when you fetch the mbox off Patchwork.\n\nPart of me would really like to keep Patchwork mboxes as close as \npossible to the mbox we ingested, but on the other hand it means the \nmangled address is still going to land in the git repo at the end... so \nI should probably just change it?\n\n-- \nAndrew Donnellan OzLabs, ADL Canberra\najd@linux.ibm.com IBM Australia Limited\n\n"},{"id":"383843","messageId":"20191010225405.GA19475@sigill.intra.peff.net","threadId":"52015","inReplyTo":"20191010194132.GA191800@google.com","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-10-10T22:54:06Z","receivedAt":"2019-10-10T22:54:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 10, 2019 at 12:41:32PM -0700, Jonathan Nieder wrote:\n\n> > Add support for using the X-Original-Sender or Reply-To headers, as used by\n> > Google Groups and Mailman respectively, to unmangle the From header when\n> > necessary.\n> [...]\n> Interesting! I'm cc-ing the Git mailing list in case \"git am\" might\n> wnat to learn the same support.\n\nNeat. There was discussion on a similar issue recently in:\n\n https://public-inbox.org/git/305577c2-709a-b632-4056-6582771176ac@redhat.com/\n\nwhere a possible solution was to get senders to use in-body From\nheaders even when sending their own patches.\n\nThis might provide an alternate solution (or vice versa). I kind of like\nthis one better in that it doesn't require the sender to do anything\ndifferently (but it may be less robust, as it assumes the receiver\nreliably de-mangling).\n\n-Peff\n"},{"id":"383844","messageId":"06541640-7eca-bc40-5c4b-9aa682d774a8@linux.ibm.com","threadId":"52015","inReplyTo":"20191010225405.GA19475@sigill.intra.peff.net","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Andrew Donnellan","fromEmail":"ajd@linux.ibm.com","sentAt":"2019-10-10T23:01:23Z","receivedAt":"2019-10-10T23:01:35Z","isPatch":true,"sender":{"key":"ajd@linux.ibm.com","avatar":null},"body":"On 11/10/19 9:54 am, Jeff King wrote:\n> Neat. There was discussion on a similar issue recently in:\n> \n> https://public-inbox.org/git/305577c2-709a-b632-4056-6582771176ac@redhat.com/\n> \n> where a possible solution was to get senders to use in-body From\n> headers even when sending their own patches.\n\nI think that's a good idea.\n\n> \n> This might provide an alternate solution (or vice versa). I kind of like\n> this one better in that it doesn't require the sender to do anything\n> differently (but it may be less robust, as it assumes the receiver\n> reliably de-mangling).\n\nYep, it's less robust - but OTOH there's always a long tail of users \nstuck on old versions of git for whatever reason and having some logic \nto detect DMARC munging may thus still be useful.\n\n-- \nAndrew Donnellan OzLabs, ADL Canberra\najd@linux.ibm.com IBM Australia Limited\n\n"},{"id":"383846","messageId":"20191010230631.GB19475@sigill.intra.peff.net","threadId":"52015","inReplyTo":"06541640-7eca-bc40-5c4b-9aa682d774a8@linux.ibm.com","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-10-10T23:06:32Z","receivedAt":"2019-10-10T23:06:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 11, 2019 at 10:01:23AM +1100, Andrew Donnellan wrote:\n\n> > This might provide an alternate solution (or vice versa). I kind of like\n> > this one better in that it doesn't require the sender to do anything\n> > differently (but it may be less robust, as it assumes the receiver\n> > reliably de-mangling).\n> \n> Yep, it's less robust - but OTOH there's always a long tail of users stuck\n> on old versions of git for whatever reason and having some logic to detect\n> DMARC munging may thus still be useful.\n\nI think the two features would work together nicely out of the box: if\nsomebody has an in-body from, we'd respect that before looking at email\nheaders anyway. So senders who do the extra work will be covered, and\nchecking other email headers would just improve the fallback case.\n\n-Peff\n"},{"id":"383848","messageId":"87y2xstcmf.fsf@dja-thinkpad.axtens.net","threadId":"52015","inReplyTo":"236b9c32-b501-79a7-8366-26d64ceac24f@linux.ibm.com","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Daniel Axtens","fromEmail":"dja@axtens.net","sentAt":"2019-10-10T23:16:40Z","receivedAt":"2019-10-10T23:16:49Z","isPatch":true,"sender":{"key":"dja@axtens.net","avatar":null},"body":"Andrew Donnellan writes:\n\n> On 11/10/19 6:41 am, Jonathan Nieder wrote:\n>> Interesting! I'm cc-ing the Git mailing list in case \"git am\" might\n>> wnat to learn the same support.\n> Argh, that reminds me... this patch only rewrites the name and email \n> that is recorded as the Patchwork submitter, it doesn't actually rewrite \n> the From: header when you fetch the mbox off Patchwork.\n>\n> Part of me would really like to keep Patchwork mboxes as close as \n> possible to the mbox we ingested, but on the other hand it means the \n> mangled address is still going to land in the git repo at the end... so \n> I should probably just change it?\n\nYes, I think change it. If you're worried, stash the original one in the\nheaders.\n\n>\n> -- \n> Andrew Donnellan OzLabs, ADL Canberra\n> ajd@linux.ibm.com IBM Australia Limited\n>\n> _______________________________________________\n> Patchwork mailing list\n> Patchwork@lists.ozlabs.org\n> https://lists.ozlabs.org/listinfo/patchwork\n"},{"id":"383851","messageId":"20191011104040.0dd8db07@canb.auug.org.au","threadId":"52015","inReplyTo":"87y2xstcmf.fsf@dja-thinkpad.axtens.net","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Stephen Rothwell","fromEmail":"sfr@canb.auug.org.au","sentAt":"2019-10-10T23:40:40Z","receivedAt":"2019-10-10T23:40:50Z","isPatch":true,"sender":{"key":"sfr@canb.auug.org.au","avatar":null},"body":"On Fri, 11 Oct 2019 10:16:40 +1100 Daniel Axtens wrote:\n>\n> Andrew Donnellan writes:\n> \n> > On 11/10/19 6:41 am, Jonathan Nieder wrote: \n> >> Interesting! I'm cc-ing the Git mailing list in case \"git am\" might\n> >> wnat to learn the same support. \n> > Argh, that reminds me... this patch only rewrites the name and email \n> > that is recorded as the Patchwork submitter, it doesn't actually rewrite \n> > the From: header when you fetch the mbox off Patchwork.\n> >\n> > Part of me would really like to keep Patchwork mboxes as close as \n> > possible to the mbox we ingested, but on the other hand it means the \n> > mangled address is still going to land in the git repo at the end... so \n> > I should probably just change it? \n> \n> Yes, I think change it. If you're worried, stash the original one in the\n> headers.\n\nOr add a From: line to the start of the body (if one is not already there).\n\n-- \nCheers,\nStephen Rothwell\n"},{"id":"383863","messageId":"xmqqblunj461.fsf@gitster-ct.c.googlers.com","threadId":"52015","inReplyTo":"20191010225405.GA19475@sigill.intra.peff.net","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-10-11T04:29:26Z","receivedAt":"2019-10-11T04:29:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> This might provide an alternate solution (or vice versa). I kind of like\n> this one better in that it doesn't require the sender to do anything\n> differently (but it may be less robust, as it assumes the receiver\n> reliably de-mangling).\n\nI share the assessment. I also feel that relying on Reply-To: would\nmake the result a lot less reliable (I do not have much problem with\nthe use of X-Original-Sender, though).\n"},{"id":"383864","messageId":"c942d9ce-d8fe-32ca-bedd-1cdb3837823d@linux.ibm.com","threadId":"52015","inReplyTo":"xmqqblunj461.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Andrew Donnellan","fromEmail":"ajd@linux.ibm.com","sentAt":"2019-10-11T04:36:02Z","receivedAt":"2019-10-11T04:36:14Z","isPatch":true,"sender":{"key":"ajd@linux.ibm.com","avatar":null},"body":"On 11/10/19 3:29 pm, Junio C Hamano wrote:\n> Jeff King writes:\n> \n>> This might provide an alternate solution (or vice versa). I kind of like\n>> this one better in that it doesn't require the sender to do anything\n>> differently (but it may be less robust, as it assumes the receiver\n>> reliably de-mangling).\n> \n> I share the assessment. I also feel that relying on Reply-To: would\n> make the result a lot less reliable (I do not have much problem with\n> the use of X-Original-Sender, though).\n> \n\nIt would be nice if Mailman could adopt X-Original-Sender too. As it is, \nit adds the original sender to Reply-To, but in some cases (where the \nlist is set as reply-to-list, or has a custom reply-to setting) it adds \nto Cc instead. (In the patch that started this thread, I match the name \nfrom the munged From field against the name in Reply-To/Cc for the case \nwhere there's multiple Reply-Tos/Ccs.)\n\nFor the Patchwork use case, I'm quite okay with accepting the risk of \nusing Reply-To, as the alternative is worse, the corner cases are rare, \nand ultimately a maintainer can still fix the odd stuff-up before \napplying the patch.\n\n-- \nAndrew Donnellan OzLabs, ADL Canberra\najd@linux.ibm.com IBM Australia Limited\n\n"},{"id":"383866","messageId":"7c2f16e3-1397-9ced-e334-a52e99b27e9b@linux.ibm.com","threadId":"52015","inReplyTo":"c942d9ce-d8fe-32ca-bedd-1cdb3837823d@linux.ibm.com","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Andrew Donnellan","fromEmail":"ajd@linux.ibm.com","sentAt":"2019-10-11T04:50:14Z","receivedAt":"2019-10-11T04:50:28Z","isPatch":true,"sender":{"key":"ajd@linux.ibm.com","avatar":null},"body":"On 11/10/19 3:36 pm, Andrew Donnellan wrote:\n> It would be nice if Mailman could adopt X-Original-Sender too. As it is, \n\n(which I have gone ahead and reported as \nhttps://gitlab.com/mailman/mailman/issues/641)\n\n-- \nAndrew Donnellan OzLabs, ADL Canberra\najd@linux.ibm.com IBM Australia Limited\n\n"},{"id":"383888","messageId":"6574162.ouEm0onZRE@silver","threadId":"52015","inReplyTo":"7c2f16e3-1397-9ced-e334-a52e99b27e9b@linux.ibm.com","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Christian Schoenebeck","fromEmail":"qemu_oss@crudebyte.com","sentAt":"2019-10-11T13:13:50Z","receivedAt":"2019-10-11T13:14:20Z","isPatch":true,"sender":{"key":"qemu_oss@crudebyte.com","avatar":null},"body":"On Freitag, 11. Oktober 2019 06:50:14 CEST Andrew Donnellan wrote:\n> On 11/10/19 3:36 pm, Andrew Donnellan wrote:\n> > It would be nice if Mailman could adopt X-Original-Sender too. As it is,\n> \n> (which I have gone ahead and reported as\n> https://gitlab.com/mailman/mailman/issues/641)\n\nNot stopping you from doing that, since I still think that it'd be helpful if \nmailman added some kind X-Original-Sender header in case the email has to be \nmunged for some reason. Just some notes about status & consensus we had:\n\n1. On GNU lists the default mailman settings are now to prevent munging in \nfirst place (if possible):\nhttps://lists.gnu.org/archive/html/qemu-devel/2019-09/msg00416.html\n\n2. If any list member has the \"nodup\" mailman option turned on, mailman would \nstill munge emails due to that. Ian (on CC) worked on a patch to override that \nindividual user setting automatically if necessary:\nhttps://bugs.launchpad.net/mailman/+bug/1845751\n\n3. On git side it was suggested to add some kind of \"always_use_in_body_from\" \noption:\nhttps://public-inbox.org/git/20190923222415.GA22495@sigill.intra.peff.net/\n\nUnless that git option exists, this little trick proofed as usable workaround \nfor git patch submitters suffering from munging:\nhttps://lists.gnu.org/archive/html/qemu-devel/2019-09/msg00932.html\n\n4. MTA's should also address this DKIM issue more accurately. For instance \nExim is currently by default filling the \"dkim h=...\" header with \"all header \nnames listed in RFC4871 will be used, whether or not each header is present in \nthe message\":\nhttps://www.exim.org/exim-html-current/doc/html/spec_html/ch-dkim_and_spf.html\nThat \"h=\" tag in email's dkim header lists all email headers which were \nincluded by MTA for signing the message. However IMO MTA's should not list any \n\"List-*\" header name in \"dkim h=...\" (at least not if not present in message), \notherwise mailman is forced to munge any of such messages when adding its \nrequired List-* headers.\n\nBTW section 5.5. (page 38) of that RFC4871 actually sais these headers \"SHOULD \nbe included in the signature, if they are present in the message being \nsigned\".\n\nFor now you can override this setting, e.g. by using Exim's \n\"dkim_sign_headers\" setting and providing your own list of header names, but \nfrom security point of view that's suboptimal, since admins probably leave \nthat untouched for years and new security relevant headers might not be \nincluded for signing at some point in future. So IMO it would make sense to \nadd more fine graded MTA DKIM config options like:\n\"include these headers for dkim signing only if present in message\"\nand/or\n\"use default header names except of these\".\n\nBy taking these things into account, emails of domains with strict DMARC \npolicies are no longer munged on gnu lists.\n\nBest regards,\nChristian Schoenebeck\n\n\n"},{"id":"383893","messageId":"87pnj3thja.fsf@dja-thinkpad.axtens.net","threadId":"52015","inReplyTo":"06541640-7eca-bc40-5c4b-9aa682d774a8@linux.ibm.com","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Daniel Axtens","fromEmail":"dja@axtens.net","sentAt":"2019-10-11T15:42:49Z","receivedAt":"2019-10-11T15:42:56Z","isPatch":true,"sender":{"key":"dja@axtens.net","avatar":null},"body":"Hi,\n\n>> Neat. There was discussion on a similar issue recently in:\n>> \n>> https://public-inbox.org/git/305577c2-709a-b632-4056-6582771176ac@redhat.com/\n>> \n>> where a possible solution was to get senders to use in-body From\n>> headers even when sending their own patches.\n>\n> I think that's a good idea.\n>\n>> \n>> This might provide an alternate solution (or vice versa). I kind of like\n>> this one better in that it doesn't require the sender to do anything\n>> differently (but it may be less robust, as it assumes the receiver\n>> reliably de-mangling).\n>\n> Yep, it's less robust - but OTOH there's always a long tail of users \n> stuck on old versions of git for whatever reason and having some logic \n> to detect DMARC munging may thus still be useful.\n\nI'm not sure this solution is correct.\n\nIf I take a patch from Andrew, backport it, and send to the list, Andrew\nwill be listed in the in-body From. However, he shouldn't be the sender\nfrom the Patchwork point of view: he shouldn't get the patch status\nnotification emails - I should. We don't want to spam an original author\nif their patch is backported to several different releases, or picked up\nand resent in someone else's series, etc etc. So unless I've\nmisunderstood something, we can't rely on the in-body from matching\nPatchwork's understanding of the sender.\n\nRegards,\nDaniel\n\n>\n> -- \n> Andrew Donnellan OzLabs, ADL Canberra\n> ajd@linux.ibm.com IBM Australia Limited\n>\n> _______________________________________________\n> Patchwork mailing list\n> Patchwork@lists.ozlabs.org\n> https://lists.ozlabs.org/listinfo/patchwork\n"},{"id":"383897","messageId":"20191011155151.GA19395@sigill.intra.peff.net","threadId":"52015","inReplyTo":"87pnj3thja.fsf@dja-thinkpad.axtens.net","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-10-11T15:51:51Z","receivedAt":"2019-10-11T15:51:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Oct 12, 2019 at 02:42:49AM +1100, Daniel Axtens wrote:\n\n> >> where a possible solution was to get senders to use in-body From\n> >> headers even when sending their own patches.\n> [...]\n> I'm not sure this solution is correct.\n> \n> If I take a patch from Andrew, backport it, and send to the list, Andrew\n> will be listed in the in-body From. However, he shouldn't be the sender\n> from the Patchwork point of view: he shouldn't get the patch status\n> notification emails - I should. We don't want to spam an original author\n> if their patch is backported to several different releases, or picked up\n> and resent in someone else's series, etc etc. So unless I've\n> misunderstood something, we can't rely on the in-body from matching\n> Patchwork's understanding of the sender.\n\nYeah, it may be that patchwork and git have two different priorities\nhere. From my perspective, the problem is getting the patch into a git\nrepo with the right author name. But patchwork may want to make the\ndistinction between author and sender.\n\n-Peff\n"},{"id":"383914","messageId":"871rvjdw1b.fsf@fsf.org","threadId":"52015","inReplyTo":"6574162.ouEm0onZRE@silver","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Ian Kelling","fromEmail":"iank@fsf.org","sentAt":"2019-10-11T17:36:16Z","receivedAt":"2019-10-11T18:17:18Z","isPatch":true,"sender":{"key":"iank@fsf.org","avatar":null},"body":"\nChristian Schoenebeck writes:\n\n> 4. MTA's should also address this DKIM issue more accurately.\n\nI agree that Exim should be changed as you suggest.\n\n>\n> By taking these things into account, emails of domains with strict DMARC \n> policies are no longer munged on gnu lists.\n\nAdditional info: Migration of many gnu/nongnu.gnu.org lists is still in\nprogress for another week or so, then that will be true for most of\nthem. For a minority of lists, the list administrators have set weird\nsettings like making all messages have from: rewritten as from this\nlist, and we are leaving them as is since the list administrators opted\nin to that at some point. But if the list deals with patches and not\nmodifying the headers is useful to the people on the list, I think a\nrequest to change the list settings is likely to be accepted by the list\nadmin.\n\n-- \nIan Kelling | Senior Systems Administrator, Free Software Foundation\nGPG Key: B125 F60B 7B28 7FF6 A2B7 DF8F 170A F0E2 9542 95DF\nhttps://fsf.org | https://gnu.org\n"},{"id":"383990","messageId":"a59c318f-476b-c3ab-fa6b-5067503820a3@linux.ibm.com","threadId":"52015","inReplyTo":"20191011155151.GA19395@sigill.intra.peff.net","subject":"Re: [PATCH] parser: Unmangle From: headers that have been mangled for DMARC purposes","fromName":"Andrew Donnellan","fromEmail":"ajd@linux.ibm.com","sentAt":"2019-10-13T11:05:49Z","receivedAt":"2019-10-13T11:06:00Z","isPatch":true,"sender":{"key":"ajd@linux.ibm.com","avatar":null},"body":"On 12/10/19 2:51 am, Jeff King wrote:\n> On Sat, Oct 12, 2019 at 02:42:49AM +1100, Daniel Axtens wrote:\n> \n>>>> where a possible solution was to get senders to use in-body From\n>>>> headers even when sending their own patches.\n>> [...]\n>> I'm not sure this solution is correct.\n>>\n>> If I take a patch from Andrew, backport it, and send to the list, Andrew\n>> will be listed in the in-body From. However, he shouldn't be the sender\n>> from the Patchwork point of view: he shouldn't get the patch status\n>> notification emails - I should. We don't want to spam an original author\n>> if their patch is backported to several different releases, or picked up\n>> and resent in someone else's series, etc etc. So unless I've\n>> misunderstood something, we can't rely on the in-body from matching\n>> Patchwork's understanding of the sender.\n> \n> Yeah, it may be that patchwork and git have two different priorities\n> here. From my perspective, the problem is getting the patch into a git\n> repo with the right author name. But patchwork may want to make the\n> distinction between author and sender.\n> \n\nYes, I was referring to the git am case, not the Patchwork case.\n\n-- \nAndrew Donnellan OzLabs, ADL Canberra\najd@linux.ibm.com IBM Australia Limited\n\n"}]}