{"thread":{"id":"51824","subject":"RFC: Cryptographic attestation for email-based patch workflows","startedAt":"2019-09-10T12:13:31Z","lastAt":"2019-09-30T15:37:46Z","messageCount":3,"participants":["Konstantin Ryabitsev","dwh@linuxprogrammer.org"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"382125","messageId":"20190910121324.GA6867@pure.paranoia.local","threadId":"51824","inReplyTo":null,"subject":"RFC: Cryptographic attestation for email-based patch workflows","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2019-09-10T12:13:24Z","receivedAt":"2019-09-10T12:13:31Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"Hello, all:\n\nThis is a very \"raw\" idea that stems from a handful of conversations\nthat took place at the Kernel Summit. I wanted to pass it along to this\nlist in hopes that it can generate some workable ideas (or shot down and\nallowed to die early).\n\n# Problem\n\nOne of the recurring concerns raised by kernel developers is the fact\nthat email-based patch workflow offers no git-native mechanism of\ncryptographic integrity attestation. In other words, the only mechanism\nfor someone to verify that patch contents have not been altered is\nvia PGP-signed email. For a slew of reasons, this is not a sufficiently\ngood solution:\n\n- PGP support in mail clients continues to be sub-par\n- Patch archival and management tools (like patchwork) remove easy\n  ability to verify PGP signatures because they need to modify email\n  bodies (but not patch content), e.g. to add Reviewed-By: or similar\n  taglines\n- Tools like git-am have no native support for verifying PGP signatures\n\n# Proposed approach\n\nI recommend that we provide a way to include cryptographic signature\ninformation natively using git-format-patch, using roughly the following\nprocess:\n\n- generate a signify-compatible cryptographic signature of the verbatim\n  patch content, perhaps slightly normalized for things like LF vs. CRLF\n  line endings (see minisign/libsodium for crypto details)\n- include both the signature and the public key in the area below '---',\n  using \"Minisig:\" and \"Minikey:\" taglines\n\nFor example:\n\n---8<---\nFrom b41a2a0f817caddc9a76f43c3c9ed7d8edd6b2de Mon Sep 17 00:00:00 2001\nFrom: Konstantin Ryabitsev <konstantin@linuxfoundation.org>\nDate: Tue, 10 Sep 2019 06:15:36 -0400\nSubject: [PATCH] Second commit\n\nChange the greeting.\n\nSigned-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>\n---\n foo.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\n Minisig: RWT9fcUvSnHPLiqWgXEnn98sgk8nl4FteDRkD+9lVK+He//eLOxNZ5QjCROoKJgPGpL4uzoHicN+f6gB54qmtO1cQtfvjS+++QU=\n Minikey: RWT9fcUvSnHPLqqyfLbkGBMEscBWciFFp2iBj2XnZPzW69OVIoYwZ25q\n\ndiff --git a/foo.c b/foo.c\nindex d40a2b9..dcfad55 100644\n--- a/foo.c\n+++ b/foo.c\n@@ -1,5 +1,5 @@\n #include <stdio.h>\n int main()\n {\n-    printf(\"Hello, World!\");\n+    printf(\"Hello, Signed World!\");\n }\n--ᐞ\n2.21.0\n---8<---\n\nWhen git-am encounters a signed patch, it should:\n\n1. check if the email in From: matches existing entries in git's TOFU\n   (trust on first use) database, which is a simple key-value store\n   like:\n\n   konstantin@linuxfoundation.org: RWT9fcUvSnHPLqqyfLbkGBMEscBWciFFp2iBj2XnZPzW69OVIoYwZ25q\n\n2. if no matches, add a new entry to the TOFU tracking database and\n   consider the key automatically trusted (perhaps configurable)\n3. if there are existing matches:\n\n   a. compare the keys to make sure they haven't changed\n   b. if keys changed, emit a warning and let developer decide if they\n      trust the key change\n   c. if keys did not change, validate the signature\n   d. if validation failed, alert the developer and error out\n\n4. if the TOFU db exists at all, git-am should check if the email\n   address in From: matches any existing records and alert if the patch\n   carries no signature (in case it's been removed by a malicious\n   attacker).\n\nAll of these operations should be sufficiently fast, since both ECC\ncrypto and key-value lookups are fast operations that don't require a\nlot of resources.\n\n# Why minisigs?\n\nIn my experience, the kinds of developers who submit patches to mailing\nlists would consider PGP/GnuPG too cumbersome to bootstrap, which is why I\nlean towards managing keys natively by git. In my mind, the process\nwould go like this:\n\n- developer sends patches to the mailing list\n- maintainer responds with \"looks good, but please sign and resubmit\n  by passing --minisign to git-format-patch\"\n- developer runs `git format-patch --minisign`, which walks them through\n  generating the key and storing it in a dedicated file\n- git can take care of passphrase handling by hooking into\n  credential-helper and credential-cache routines\n\n# Coupling with PGP\n\nCommunities relying on the PGP web of trust can tie minikeys with their\nPGP identity by creating a UID entry containing their minisign public key,\ne.g.:\n\npub   rsa4096/E63EDCA9329DD07E 2011-11-07 [SC]\n      DE0E66E32F1FDD0902666B96E63EDCA9329DD07E\nuid                 [ultimate] Konstantin Ryabitsev <konstantin@linuxfoundation.org>\nuid                 [ultimate] Konstantin Ryabitsev <RWT9fcUvSnHPLqqyfLbkGBMEscBWciFFp2iBj2XnZPzW69OVIoYwZ25q>\n\nSuch UIDs can be revoked as necessary and new ones can be created --\nplus they are searchable using standard gnupg/keyserver tools.\n\n# Comments?\n\nI'd love to hear your feedback on the idea. Even if this scheme is not\nused by maintainers directly, it offers ways of verifying if patches\nstored in public archives (such as public-inbox) have been modified and\nprovides some developer attestation of email-based workflows.\n\n-K\n"},{"id":"383068","messageId":"20190927152437.be6d7jnowuvvuyra@dev","threadId":"51824","inReplyTo":"20190910121324.GA6867@pure.paranoia.local","subject":"Re: RFC: Cryptographic attestation for email-based patch workflows","fromName":"","fromEmail":"dwh@linuxprogrammer.org","sentAt":"2019-09-27T15:24:37Z","receivedAt":"2019-09-27T15:24:42Z","isPatch":false,"sender":{"key":"dwh@linuxprogrammer.org","avatar":null},"body":"On 10.09.2019 08:13, Konstantin Ryabitsev wrote:\n># Proposed approach\n>\n>I recommend that we provide a way to include cryptographic signature\n>information natively using git-format-patch, using roughly the following\n>process:\n>\n>- generate a signify-compatible cryptographic signature of the verbatim\n>  patch content, perhaps slightly normalized for things like LF vs. CRLF\n>  line endings (see minisign/libsodium for crypto details)\n>- include both the signature and the public key in the area below '---',\n>  using \"Minisig:\" and \"Minikey:\" taglines\n\nI like where you're heading with this suggestion however there are some\nissues. It is not clear what bytes the signature was calculated over.\nDoes it include the \"From:\" line of the email? How about the\n\"Signed-off-by\"? If there is no binding of the identity of the submitter\nto the key pair then you'll have problems with the TOFU policy you\ndescribe further down (explained later). Also, since we're trying to\nmove to a Git that supports signatures from multiple different signing\ntools and to also support multi-sig sign-offs (e.g. first the author,\nthen the reviewer, then the merger) these taglines need to be more\ncompex. At the very least either there needs to be a signature type\ntagline or the type of the signature needs to be baked into the key and\nsignture values (see Secure Scuttlebutt encoding of keys/sigs). Also, if\nwe want to do chained multisign there should be some framing of what is\nsigned by each signature. If you're looking for something email like, we\ncould borrow from mime email attachment encoding to provide framing.\n\n>For example:\n>\n>---8<---\n>From b41a2a0f817caddc9a76f43c3c9ed7d8edd6b2de Mon Sep 17 00:00:00 2001\n>From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>\n>Date: Tue, 10 Sep 2019 06:15:36 -0400\n>Subject: [PATCH] Second commit\n>\n>Change the greeting.\n>\n>Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>\n>---\n> foo.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> Minisig: RWT9fcUvSnHPLiqWgXEnn98sgk8nl4FteDRkD+9lVK+He//eLOxNZ5QjCROoKJgPGpL4uzoHicN+f6gB54qmtO1cQtfvjS+++QU=\n> Minikey: RWT9fcUvSnHPLqqyfLbkGBMEscBWciFFp2iBj2XnZPzW69OVIoYwZ25q\n>\n>diff --git a/foo.c b/foo.c\n>index d40a2b9..dcfad55 100644\n>--- a/foo.c\n>+++ b/foo.c\n>@@ -1,5 +1,5 @@\n> #include <stdio.h>\n> int main()\n> {\n>-    printf(\"Hello, World!\");\n>+    printf(\"Hello, Signed World!\");\n> }\n>--ᐞ\n>2.21.0\n>---8<---\n\nI would instead have at the very least the signature tool in the value:\n\n\tKey: minisign|RWT9fcUvSnHPLqqyfLbkGBMEscBWciFFp2iBj2XnZPzW69OVIoYwZ25q\n  Sig: minisign|RWT9fcUvSnHPLiqWgXEnn98sgk8nl4FteDRkD+9lVK+He//eLOxNZ5QjCROoKJgPGpL4uzoHicN+f6gB54qmtO1cQtfvjS+++QU=\n\nBut this doesn't solve multi-sig. \n\n>When git-am encounters a signed patch, it should:\n>\n>1. check if the email in From: matches existing entries in git's TOFU\n>   (trust on first use) database, which is a simple key-value store\n>   like:\n>\n>   konstantin@linuxfoundation.org: RWT9fcUvSnHPLqqyfLbkGBMEscBWciFFp2iBj2XnZPzW69OVIoYwZ25q\n>\n>2. if no matches, add a new entry to the TOFU tracking database and\n>   consider the key automatically trusted (perhaps configurable)\n>3. if there are existing matches:\n>\n>   a. compare the keys to make sure they haven't changed\n>   b. if keys changed, emit a warning and let developer decide if they\n>      trust the key change\n>   c. if keys did not change, validate the signature\n>   d. if validation failed, alert the developer and error out\n>\n>4. if the TOFU db exists at all, git-am should check if the email\n>   address in From: matches any existing records and alert if the patch\n>   carries no signature (in case it's been removed by a malicious\n>   attacker).\n>\n>All of these operations should be sufficiently fast, since both ECC\n>crypto and key-value lookups are fast operations that don't require a\n>lot of resources.\n\nTOFU has the problem of not providing cryptographic provenance over keys\nwhile maintaining provenance on the binding between other identity\nattributes and those keys (e.g. author string). In step 3 above there's\nno way to know for sure that the submitter is actually who they claim to\nbe. Reviewers have no way of knowing that the new key used with the\npatch is a legitimate key update. The only option with this design is to\ndo some out-of-band key verification (i.e. call the submitter and have\nthem read the key to you over the phone). Out-of-band key validation\nhasn't scaled for GPG and it won't scale here either.\n\nInstead of TOFU, a more secure design would require key enrollment, key\nrotation, key recovery, and key revocation to all be separate,\ncryptographically verified updates to the attribute-to-key database in\nthe repo. Your instinct for storing the attribute+key data in the repo\nitself (i.e. in-band) is correct because it makes Git repos\nself-verifiable in that cloning is all you need to do to get all of the\ndata necessary for verifying all of the digital signatures.\n\nKey enrollment should be a separate patch submission that adds the\nauthor and public key to the database file. The patch must be signed\nusing the key that is being added to the database. This provides the\nprovenance anchor for the key and also binds the attribute to the key.\nThe record in the data should also contain some data that enables secure\nkey recovery/rotation. A simple hashed secret passphrase and nonce\nworks. In the future, key recovery/rotation can be done by the owner by\nsubmitting a new patch that updates the database record with a new key\nand recovery data, signed with the new key and also including the nonce\nand secret passphrase used to generate the previous key recovery hash.\nKey revocation is a patch that removes the record from the database that\nis signed by the key that is being removed or a new key plus the\nrecovery secret passphrase and nonce.\n\nSo back to the original proposal, I like the simplicity but with a few\ntweaks, it could be an air tight digital signature scheme for emailed\npatches. If air tight provenance is not what you're aiming for, then why\nare you even using cryptography?\n\nThat's my 2p on this. I like where you're head is at Kontantin.\n\nCheers!\nDave\n"},{"id":"383159","messageId":"20190930153741.GA6124@chatter.i7.local","threadId":"51824","inReplyTo":"20190927152437.be6d7jnowuvvuyra@dev","subject":"Re: RFC: Cryptographic attestation for email-based patch workflows","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2019-09-30T15:37:41Z","receivedAt":"2019-09-30T15:37:46Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Sep 27, 2019 at 08:24:37AM -0700, dwh@linuxprogrammer.org wrote:\n>>- generate a signify-compatible cryptographic signature of the \n>>verbatim\n>> patch content, perhaps slightly normalized for things like LF vs. CRLF\n>> line endings (see minisign/libsodium for crypto details)\n>>- include both the signature and the public key in the area below '---',\n>> using \"Minisig:\" and \"Minikey:\" taglines\n>\n>I like where you're heading with this suggestion however there are some\n>issues. It is not clear what bytes the signature was calculated over.\n\nJust the actual patch.\n\n>Does it include the \"From:\" line of the email? How about the\n>\"Signed-off-by\"? If there is no binding of the identity of the submitter\n>to the key pair then you'll have problems with the TOFU policy you\n>describe further down (explained later). \n\nOk, I'll argue my point on this later. :)\n\n>Also, since we're trying to\n>move to a Git that supports signatures from multiple different signing\n>tools and to also support multi-sig sign-offs (e.g. first the author,\n>then the reviewer, then the merger) these taglines need to be more\n>compex. At the very least either there needs to be a signature type\n>tagline or the type of the signature needs to be baked into the key and\n>signture values (see Secure Scuttlebutt encoding of keys/sigs). Also, if\n>we want to do chained multisign there should be some framing of what is\n>signed by each signature. If you're looking for something email like, we\n>could borrow from mime email attachment encoding to provide framing.\n\nNo, we definitely don't want to go down the MIME path (it's routinely \nmangled by archivers, so we're much more likely to lose anything that \ncomes in via MIME attachments).\n\n>I would instead have at the very least the signature tool in the value:\n>\n>\tKey: minisign|RWT9fcUvSnHPLqqyfLbkGBMEscBWciFFp2iBj2XnZPzW69OVIoYwZ25q\n> Sig: minisign|RWT9fcUvSnHPLiqWgXEnn98sgk8nl4FteDRkD+9lVK+He//eLOxNZ5QjCROoKJgPGpL4uzoHicN+f6gB54qmtO1cQtfvjS+++QU=\n\nI don't think it's worth it to abstract this out. The main benefits of\nminisigs are:\n\n- it's an emerging standard (www.kernel.org should soon offer minisig \n  signatures on tarball downloads)\n- it's short enough to include both the key and the signature into a few \n  bytes of information -- attempting to do the same with PGP would \n  balloon the message into kilobytes, even if ECC keys/subkeys are used\n\n>But this doesn't solve multi-sig.\n\nIt doesn't attempt to, but it can be achieved in a number of clever \nways. For example, the reviewer can sign the minisig signature on the \noriginal patch. E.g.:\n\nReviewed-by: Alter Ego <mricon@kernel.org>\nReviewed-minisig: {minisig signature of RWT9fcUvS...fvjS+++QU=}\nReviewed-minikey: {reviewer pubkey}\n\nThis would give you a chain of attestation to the original patch.\n\nFor series, this would be more complicated, since Reviewed-by: is \nusually posted for the cover letter. Perhaps series cover letters could \ninclude a signature of all individual patch signatures.\n\nThat said, this is not something I'm trying to solve -- my goal is to \nprovide tamper-evident attestation of patches sent to mailing lists, and \nI expound on that further down.\n\n>TOFU has the problem of not providing cryptographic provenance over \n>keys\n>while maintaining provenance on the binding between other identity\n>attributes and those keys (e.g. author string). In step 3 above there's\n>no way to know for sure that the submitter is actually who they claim to\n>be. \n\nCorrect, but the same is true for any other key distribution mechanism.  \nEven with PGP, most people I work with use the TOFU approach -- if a key \nis in their keyring, it's considered automatically trusted. My goal is \nnot really to come up with a tamper-proof solution, but to offer a chain \nof cryptographic attestation. Developers can then *choose* to \nincorporate tamper-proof features of it into their workflows via tool \nsupport.\n\n>Reviewers have no way of knowing that the new key used with the\n>patch is a legitimate key update. The only option with this design is to\n>do some out-of-band key verification (i.e. call the submitter and have\n>them read the key to you over the phone). Out-of-band key validation\n>hasn't scaled for GPG and it won't scale here either.\n\nYes, because delegated trust is *hard*. :) We either must rely on \ndelegated trust via certification authorities -- with all the potential \nfor abuse there -- or we must make trust decisions on our own, which \ndoesn't scale well. As far as I can tell, there is no easy solution to \nthis problem. TOFU just formalizes everyone's current coping mechanism.\n\n>Instead of TOFU, a more secure design would require key enrollment, key\n>rotation, key recovery, and key revocation to all be separate,\n>cryptographically verified updates to the attribute-to-key database in\n>the repo. \n\nRight, as long as it's understood that this delegates trust to people \nwith write access to this repository (and infrastructure admins). This \nalso has important drawbacks in the case of the Linux kernel, for \nexample:\n\n- there are thousands of people committing patches to the Linux kernel, \n  and it's not the same thousands of people with each mainline release, \n  so keeping key information for all of them in the repository would be \n  next to impossible\n- while torvalds/linux.git is considered \"canonical\" for Linux, most \n  developers would be working off of other trees (netdev, arm, etc), so \n  it would make little sense for them to put their key information into \n  the mainline repository\n\n>Your instinct for storing the attribute+key data in the repo\n>itself (i.e. in-band) is correct because it makes Git repos\n>self-verifiable in that cloning is all you need to do to get all of the\n>data necessary for verifying all of the digital signatures.\n\nRight, this is an important problem to solve, but it's not the one I'm \ntrying to address. :) I'm specifically interested in cryptographic \nattestation of patches sent to mailing lists -- *before* code even makes \nit into git. Since there is no way to translate signatures on patches \ninto git commit signatures, I'm not even attempting to solve that \nproblem.\n\n>Key enrollment should be a separate patch submission that adds the\n>author and public key to the database file. The patch must be signed\n>using the key that is being added to the database. This provides the\n>provenance anchor for the key and also binds the attribute to the key.\n\nI like it, but this won't scale for Linux kernel due to the reasons I've \ndescribed above -- thousands of developers who come and go, plus \nmultiple \"canonical\" linux.git trees, depending on the component you're \nworking on.\n\n>So back to the original proposal, I like the simplicity but with a few\n>tweaks, it could be an air tight digital signature scheme for emailed\n>patches. If air tight provenance is not what you're aiming for, then why\n>are you even using cryptography?\n\nMy primary goal is to remove one of the last bits where we explicitly \ntrust infrastructure in the Linux Kernel development -- vger.kernel.org \nand lore.kernel.org. If either of these systems are compromised, an \nattacker would be able to modify patches on the fly in order to insert \nmalicious code without leaving a trace (e.g. intercept a patch as it is \nsent to the actual maintainer, but not when sent to others or to the \narchiver). Adding basic cryptographic signatures to the process will \nhopefully make this attack vector less likely and will at least offer a \nmechanism to perform post-mortem forensic examinations -- without \nintroducing a central certification authority.\n\n-K\n"}]}