{"thread":{"id":"45341","subject":"Stable GnuPG interface, git should use GPGME","startedAt":"2017-03-10T10:07:38Z","lastAt":"2017-03-23T11:03:16Z","messageCount":19,"participants":["Bernhard E. Reiter","Ævar Arnfjörð Bjarmason","Linus Torvalds","Theodore Ts'o","brian m. carlson","Michael J Gruber","Bernhard Reiter","Jeff King","Christian Neukirchen","Werner Koch","Peter Lebbing"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"313741","messageId":"201703101100.15214.bernhard.reiter@intevation.de","threadId":"45341","inReplyTo":null,"subject":"Stable GnuPG interface, git should use GPGME","fromName":"Bernhard E. Reiter","fromEmail":"bernhard.reiter@intevation.de","sentAt":"2017-03-10T10:00:07Z","receivedAt":"2017-03-10T10:07:38Z","isPatch":false,"sender":{"key":"bernhard.reiter@intevation.de","avatar":null},"body":"Dear Git-Devs,\n\ngit uses an pipe-and-exec approach to running a GnuPG binary \nas writen in the documentation [1]:\n\n gpg.program\n Use this custom program instead of \"gpg\" found on $PATH when making\n or verifying a PGP signature. The program must support the same\n command-line interface as GPG\n\nplease consider using libgpgme interfacing to GnuPG, because the gpg \ncommand-line interface is not considered an official API to GnuPG by the \nGnuPG-devs and thus potentially unstable. \n\n== Details\n\nI'm involved in GnuPG development. For most applications using libgpgme is the \nway what GnuPG-devs would recommend, also see \n\n https://wiki.gnupg.org/APIs .\n\nGnuPG devs are making a good effort of trying to keep the command-line \ninterface stable, though it is not for sure. Git is only using a small part \nof the interface, so the risk when keeping the current way is small. \nStill I believe git's stability and usability would profit when moving to \nlibgpgme, especially with the coming move to GnuPG 2.2, better diagnosing \nmessages and for cross-plattform usage.\n\n== Usability problem with `gpg2` vs `gpg`\n\nMy use case today was signing and git by default found the `gpg` binary by \ndefault and the command failed. The reason is that I have `gpg2` installed \nand most applications use it right away. So git failed signing because \nthe .gnupg configuration of the user was not ready for the old `gpg` which is \nstill installed on Debian GNU/Linux for purposes of the operating system. If \ngit would have used libgpgme, gpgme would have choosen the most uptodate \nversion of `gpg` available (or configured) without me intervening via \ngpg.program. Now because of this problem you could adding a check for `gpg2` \nand fallback to `gpg`, but even better would be to move to libgpgme. >:)\n\nBest Regards and thanks for maintaining Git as Free Software,\nBernhard\n\n== how to respond\n\nps: Please copy me on replies as I am not on git@vger.kernel.org. \npps: I've copied gnupg-devel@ so they can see I've send this report, you don't \nhave to.\n\n\n[1] \nhttps://github.com/git/git/blob/3bc53220cb2dcf709f7a027a3f526befd021d858/Documentation/config.txt\nsearch for 'gpg.program'.\n\n-- \nwww.intevation.de/~bernhard (CEO) +49 541 33 508 3-3\nIntevation GmbH, Osnabrück, Germany; Amtsgericht Osnabrück, HRB 18998\nOwned and run by Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner\n"},{"id":"313750","messageId":"CACBZZX4Av-D6hxE9ceDFPuG-_qUQbH_6KW5JKsJf0SuH62jkuQ@mail.gmail.com","threadId":"45341","inReplyTo":"201703101100.15214.bernhard.reiter@intevation.de","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-03-10T14:23:27Z","receivedAt":"2017-03-10T14:25:18Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Fri, Mar 10, 2017 at 11:00 AM, Bernhard E. Reiter\n wrote:\n> Dear Git-Devs,\n\nI haven't contributed to Git's GPG code, but I'm taking the liberty of\nCC-ing some people who have.\n\n> git uses an pipe-and-exec approach to running a GnuPG binary\n> as writen in the documentation [1]:\n>\n> gpg.program\n> Use this custom program instead of \"gpg\" found on $PATH when making\n> or verifying a PGP signature. The program must support the same\n> command-line interface as GPG\n>\n> please consider using libgpgme interfacing to GnuPG, because the gpg\n> command-line interface is not considered an official API to GnuPG by the\n> GnuPG-devs and thus potentially unstable.\n>\n> == Details\n>\n> I'm involved in GnuPG development. For most applications using libgpgme is the\n> way what GnuPG-devs would recommend, also see\n>\n> https://wiki.gnupg.org/APIs .\n>\n> GnuPG devs are making a good effort of trying to keep the command-line\n> interface stable, though it is not for sure. Git is only using a small part\n> of the interface, so the risk when keeping the current way is small.\n> Still I believe git's stability and usability would profit when moving to\n> libgpgme, especially with the coming move to GnuPG 2.2, better diagnosing\n> messages and for cross-plattform usage.\n>\n> == Usability problem with `gpg2` vs `gpg`\n>\n> My use case today was signing and git by default found the `gpg` binary by\n> default and the command failed. The reason is that I have `gpg2` installed\n> and most applications use it right away. So git failed signing because\n> the .gnupg configuration of the user was not ready for the old `gpg` which is\n> still installed on Debian GNU/Linux for purposes of the operating system. If\n> git would have used libgpgme, gpgme would have choosen the most uptodate\n> version of `gpg` available (or configured) without me intervening via\n> gpg.program. Now because of this problem you could adding a check for `gpg2`\n> and fallback to `gpg`, but even better would be to move to libgpgme. >:)\n\nI'm on Debian but haven't had these issues. What's your gpg & gpg2\n--version & Debian release? And what in particular failed?\n\nAnd what git version was this? I see we've had a couple of workarounds\nfor gpg2, in particular Linus's v2.8.4-1-gb624a3e67f, but if you have\nv2.10.0 or later that won't fix whatever issue you had.\n\nUsing the library sounds good, but a shorter-term immediate fix would\nbe to figure out what bug you encountered in our use of the\ncommand-line version, and see if we've fixed that already or not.\nRegardless of what we do with a gpg library in the future some distros\nmight want to backport such a small patch if we can come up with it.\n\n> Best Regards and thanks for maintaining Git as Free Software,\n> Bernhard\n>\n> == how to respond\n>\n> ps: Please copy me on replies as I am not on git@vger.kernel.org.\n> pps: I've copied gnupg-devel@ so they can see I've send this report, you don't\n> have to.\n>\n>\n> [1]\n> https://github.com/git/git/blob/3bc53220cb2dcf709f7a027a3f526befd021d858/Documentation/config.txt\n> search for 'gpg.program'.\n>\n> --\n> www.intevation.de/~bernhard (CEO) +49 541 33 508 3-3\n> Intevation GmbH, Osnabrück, Germany; Amtsgericht Osnabrück, HRB 18998\n> Owned and run by Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner\n"},{"id":"313769","messageId":"CA+55aFxk7F103LADnmwc8wFySYQNiK6TcCQ0WSj+UTP-GihgcQ@mail.gmail.com","threadId":"45341","inReplyTo":"201703101100.15214.bernhard.reiter@intevation.de","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2017-03-10T18:54:19Z","receivedAt":"2017-03-10T18:54:27Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Fri, Mar 10, 2017 at 2:00 AM, Bernhard E. Reiter\n wrote:\n>\n> git uses an pipe-and-exec approach to running a GnuPG binary\n> as writen in the documentation [1]:\n>\n> gpg.program\n> Use this custom program instead of \"gpg\" found on $PATH when making\n> or verifying a PGP signature. The program must support the same\n> command-line interface as GPG\n>\n> please consider using libgpgme interfacing to GnuPG, because the gpg\n> command-line interface is not considered an official API to GnuPG by the\n> GnuPG-devs and thus potentially unstable.\n\nQuite frankly, I will NAK this just based on previous bad experiences\nwith using \"helpful\" libraries.\n\nMaybe you can lay my worries to rest, but the problems with libraries\nin this context tend to be\n\n - hard to personalize.\n\n At least right now, we just allow people to set their gpg binary.\nI'm betting that the library would pick whatever random preferred\nversion, and in the process possibly screwed us.\n\n Example: what if somebody is actually using another pgp\nimplementation entirely for some reason, and is just scripting around\nit?\n\n Or maybe he's using the regular gnupg, but using different keys for\ndifferent projects (using \"--homedir\"). That's trivial with the\ncurrent model. How trivial is that with a library?\n\n - existing configuration\n\n This is the main problem I've seen in the past. Using the \"ssh\"\n_program_ is easy. You add your keys, your config files, whatever, and\nit \"just works\" (or rather, you fight it once and it definitely\ndoesn't \"just\" work, but then you copy your .ssh directory around for\nthe rest of your and forget how it ever worked, but it does).\n\n Using \"libssh2\" is an exercise in futility, and you have to do a\ncrazy amount of stupid \"look up keys\" and simple configuration in your\n.ssh/config (like per-host keys, hostname swizzling etc) just don't\npick up the configurations you already did for the program.\n\n - UI\n\n For things like gpg, the UI is traditionally horrible. But there\ntends to be various things like password agents that help with caching\npasswords and/or add a graphical UI to get the password when\nnecessary.\n\n - library versioning.\n\n I don't know why, but I've never *ever* met a library developer who\nrealized that libraries were all about stable API's, and the library\nusers don't want to fight different versions.\n\n And to make matters worse, the different versions (particularly if\nyou end up having to use a development version due to bugs or required\nfeatures etc) are always made horribly bad to even detect at\nbuilt-time automatically with simple #ifdef etc, so now you have to do\nautoconf crap etc.\n\nNow, it may be that the pgpme library \"just works\" across\narchitectures and handles all of the above situations as gracefully as\nthe external program does. In that case - but _ONLY_ in that case -\nwould a switch-over to the library possibly be a good thing.\n\nI'd be pleasantly surprised. But I *would* be surprised, because every\ntime I've seen that \"library vs program\" model, I've seen the above\nissues.\n\nIn fact, we have those exact issues very much in git itself too. Yes,\nI've used libgit2 (for subsurface). It's a pain in the arse to do\n*exactly* the above kinds of things, and the thing is, that isn't\ngit-specific.\n\nSo I'm very down on using external libraries unless they are stable\nand have no need for configuration etc. Things like zlib is fine -\nthere just isn't much to configure outside of the \"just how hard do\nyou want me to try to compress\". Nobody has a \".zlib/config\" file that\nyou need to worry about accessing etc.\n\nOf course, maybe pgpme is a world first, and actually does read your\n.gnupg/config file trivially, and has all the gpg agent integration\nthat it picks up automatically, and allows various per-user\nconfigurations, and all my worries are bogus.\n\nBut that would literally be the first time I've ever seen that.\n\n Linus\n"},{"id":"313793","messageId":"20170310202609.cegd6jak6cklq6my@thunk.org","threadId":"45341","inReplyTo":"CA+55aFxk7F103LADnmwc8wFySYQNiK6TcCQ0WSj+UTP-GihgcQ@mail.gmail.com","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2017-03-10T20:26:09Z","receivedAt":"2017-03-10T20:26:25Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Fri, Mar 10, 2017 at 10:54:19AM -0800, Linus Torvalds wrote:\n> - library versioning.\n> \n> I don't know why, but I've never *ever* met a library developer who\n> realized that libraries were all about stable API's, and the library\n> users don't want to fight different versions.\n\nActually, you have. (Raises hand :-)\n\nlibext2fs has a stable API *and* ABI. We add new functions instead of\nchanging function parameters (so ext2fs_block_iterate2() is\nimplemented in terms of ext2fs_block_iterate3(), and so on). And\nstructures have magic numbers that have served as versioning signal.\nThis is actually not rocket science. If you've met anyone who's\nprogrammed for Multics, they did something similar. And of course,\nthat's why we have the wait3(2) and wait(4) system calls.\n\nI do have to agree with your general point, that most developers tend\nto be *incredibly* sloppy with their interfaces. That being said, not\nall library developers are as bad as GNOME. :-)\n\n \t \t \t \t - Ted\n"},{"id":"313811","messageId":"20170311001031.f5534omsrzkrzfzb@genre.crustytoothpaste.net","threadId":"45341","inReplyTo":"201703101100.15214.bernhard.reiter@intevation.de","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2017-03-11T00:10:31Z","receivedAt":"2017-03-11T00:10:48Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Fri, Mar 10, 2017 at 11:00:07AM +0100, Bernhard E. Reiter wrote:\n> My use case today was signing and git by default found the `gpg` binary by\n> default and the command failed. The reason is that I have `gpg2` installed\n> and most applications use it right away. So git failed signing because\n> the .gnupg configuration of the user was not ready for the old `gpg` which is\n> still installed on Debian GNU/Linux for purposes of the operating system. If\n> git would have used libgpgme, gpgme would have choosen the most uptodate\n> version of `gpg` available (or configured) without me intervening via\n> gpg.program. Now because of this problem you could adding a check for `gpg2`\n> and fallback to `gpg`, but even better would be to move to libgpgme. >:)\n\nThere are a couple potential problems I see with this approach. First,\nI'd want to know whether gpgme supports gpgsm, which I know some people\nuse to sign commits and tags.\n\nAnother issue is what happens to the git verify-* --raw output. Some\npeople want the ability to script signature verification. This can be\nreally important when you have automated systems verifying tags and\ncommits.\n\nFor example, running the following commands, we can determine that Junio\nsigns his tags with SHA-1 (algorithm 2), while I sign my commits with\nSHA-512 (algorithm 10).\n\ngenre ok % git verify-tag --raw v2.12.0 2>&1 | grep VALIDSIG\n[GNUPG:] VALIDSIG E1F036B1FEE7221FC778ECEFB0B5E88696AFE6CB 2017-02-24 1487962205 0 4 0 1 2 00 96E07AF25771955980DAD10020D04E5A713660A7\ngenre ok % git verify-commit --raw object-id-part10 2>&1 | grep VALIDSIG\n[GNUPG:] VALIDSIG 5FC3A781776B26DF87F70C37BF535D811F52F68B 2017-03-06 1488760639 0 4 0 1 10 00 88ACE9B29196305BA9947552F1BA225C0223B187\n\nThere's literally no other way to get this information at the moment\n(which is why I added the --raw option). A gpgme implementation would\nneed to expose this same information, at which point, we might as well\nhave used gpg directly.\n\nThis is not an idle consideration; we have automated systems at work\nthat update software automatically and submit it for human review,\nincluding verifying signatures and hashes. This saves hundreds of hours\nof staff time and results in better security.\n\nBecause the amount of the gpg API we actually use is very small, a user\nwho wants to use a custom signature program (say, OpenBSD's signify),\ncan actually write a simple wrapper that mimics it and use that instead.\n\nFinally, I work on a development system where work is done both as an\nunprivileged user and as root. Because I use the same socket for both,\nGnuPG screams bloody murder that the permissions are wrong. I know this\nis secure in my scenario, but without a custom wrapper, I have to deal\nwith GnuPG polluting my terminal every time I sign a commit or a tag. A\ngpgme implementation would need to honor the same wrapper script or\notherwise not scream to the terminal.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"313880","messageId":"a5161dd2-977b-8195-e558-01787fb3f01b@drmicha.warpmail.net","threadId":"45341","inReplyTo":"CACBZZX4Av-D6hxE9ceDFPuG-_qUQbH_6KW5JKsJf0SuH62jkuQ@mail.gmail.com","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2017-03-13T10:14:57Z","receivedAt":"2017-03-13T10:15:05Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Ævar Arnfjörð Bjarmason venit, vidit, dixit 10.03.2017 15:23:\n> On Fri, Mar 10, 2017 at 11:00 AM, Bernhard E. Reiter\n> wrote:\n>> Dear Git-Devs,\n> \n> I haven't contributed to Git's GPG code, but I'm taking the liberty of\n> CC-ing some people who have.\n> \n>> git uses an pipe-and-exec approach to running a GnuPG binary\n>> as writen in the documentation [1]:\n>>\n>> gpg.program\n>> Use this custom program instead of \"gpg\" found on $PATH when making\n>> or verifying a PGP signature. The program must support the same\n>> command-line interface as GPG\n>>\n>> please consider using libgpgme interfacing to GnuPG, because the gpg\n>> command-line interface is not considered an official API to GnuPG by the\n>> GnuPG-devs and thus potentially unstable.\n>>\n>> == Details\n>>\n>> I'm involved in GnuPG development. For most applications using libgpgme is the\n>> way what GnuPG-devs would recommend, also see\n>>\n>> https://wiki.gnupg.org/APIs .\n>>\n>> GnuPG devs are making a good effort of trying to keep the command-line\n>> interface stable, though it is not for sure. Git is only using a small part\n>> of the interface, so the risk when keeping the current way is small.\n>> Still I believe git's stability and usability would profit when moving to\n>> libgpgme, especially with the coming move to GnuPG 2.2, better diagnosing\n>> messages and for cross-plattform usage.\n>>\n>> == Usability problem with `gpg2` vs `gpg`\n>>\n>> My use case today was signing and git by default found the `gpg` binary by\n>> default and the command failed. The reason is that I have `gpg2` installed\n>> and most applications use it right away. So git failed signing because\n>> the .gnupg configuration of the user was not ready for the old `gpg` which is\n>> still installed on Debian GNU/Linux for purposes of the operating system. If\n>> git would have used libgpgme, gpgme would have choosen the most uptodate\n>> version of `gpg` available (or configured) without me intervening via\n>> gpg.program. Now because of this problem you could adding a check for `gpg2`\n>> and fallback to `gpg`, but even better would be to move to libgpgme. >:)\n> \n> I'm on Debian but haven't had these issues. What's your gpg & gpg2\n> --version & Debian release? And what in particular failed?\n> \n> And what git version was this? I see we've had a couple of workarounds\n> for gpg2, in particular Linus's v2.8.4-1-gb624a3e67f, but if you have\n> v2.10.0 or later that won't fix whatever issue you had.\n> \n> Using the library sounds good, but a shorter-term immediate fix would\n> be to figure out what bug you encountered in our use of the\n> command-line version, and see if we've fixed that already or not.\n> Regardless of what we do with a gpg library in the future some distros\n> might want to backport such a small patch if we can come up with it.\n\nAs far as I know, Git handles different GPG versions just fine.\n\nThe problem is the \"difficult\" upgrade path and mixed installations with\ngpg and gpg2.1+ that some distributions force upon you:\n\nAs soon as you start gpg2.1, your (secret) key store is migrated to a\nnew format without technically invalidating it. Similarly, users may\nenter gpg2.1+-only comand in the config that is actually shared with\ngpg, throwing off any use of gpg - not just by git, but also by anything\nthat your distro requires gpg for (such as packaging tools and the like).\n\nIn short: Users will run into problems anyway; git provides the quick\nway out (git config gpg.program gpg2), users won't be as lucky with\nother things that require gpg.\n\nAs for the library: While - technically speaking - the command line is\nnot a stable API for gpg, it does work across versions of gpg, and gpg\n2.2 will be the first real stable branch that uses the new key store\nlayout. So I'd rather wait for that to stabilize before going away from\nwhat turned out to be most stable so far.\n\nNote that we (git) refrain from parsing ordinary output/return codes of\ngpg and use status-fd as we should (and as documented).\n\nMichael\n"},{"id":"313881","messageId":"201703131130.31623.bernhard@intevation.de","threadId":"45341","inReplyTo":"CACBZZX4Av-D6hxE9ceDFPuG-_qUQbH_6KW5JKsJf0SuH62jkuQ@mail.gmail.com","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Bernhard Reiter","fromEmail":"bernhard@intevation.de","sentAt":"2017-03-13T10:30:27Z","receivedAt":"2017-03-13T10:37:27Z","isPatch":false,"sender":{"key":"bernhard@intevation.de","avatar":null},"body":"Am Freitag 10 März 2017 15:23:27 schrieb Ævar Arnfjörð Bjarmason:\n> On Fri, Mar 10, 2017 at 11:00 AM, Bernhard E. Reiter wrote\n> > please consider using libgpgme interfacing to GnuPG, because the gpg\n> > command-line interface is not considered an official API to GnuPG by the\n> > GnuPG-devs and thus potentially unstable.\n\n> > == Usability problem with `gpg2` vs `gpg`\n> >\n> > My use case today was signing and git by default found the `gpg` binary\n> > by default and the command failed.\n\nI've mentioned this as one example for a possible advantage using libgpgme\nwhen interfacing with GnuPG.\n\n> > The reason is that I have `gpg2` \n> > installed and most applications use it right away. So git failed signing\n> > because the .gnupg configuration of the user was not ready for the old\n> > `gpg` which is still installed on Debian GNU/Linux for purposes of the\n> > operating system. If git would have used libgpgme, gpgme would have\n> > choosen the most uptodate version of `gpg` available (or configured)\n> > without me intervening via gpg.program. Now because of this problem you\n> > could adding a check for `gpg2` and fallback to `gpg`, but even better\n> > would be to move to libgpgme. >:)\n>\n> I'm on Debian but haven't had these issues. What's your gpg & gpg2\n> --version & Debian release? And what in particular failed?\n\nIf you use options in your configuration that only gpg2 understands, gpg(1)\nwill barf. For example the following lines in ~/.gnupg/gpg.conf\n\n debug-level basic\n log-file socket:///home/bern/.gnupg/log-socket\n\nwill lead to\nLANG=C gpg -K\n gpg: /powerhome/bern/.gnupg/gpg.conf:102: argument not expected\n gpg: /powerhome/bern/.gnupg/gpg.conf:103: invalid option\nwhere gpg2 works as expected.\n\nAs a number of application already uses gpg2 (via libgpgme or not), this may \ngo unnoticed for a while. So when I've started to sign with git on this \nmachine I ran into the problem (current Jessie default versions):\n\n dpkg -s gnupg | grep ^Version\n #Version: 1.4.18-7+deb8u3\n dpkg -s gnupg2 | grep ^Version\n #Version: 2.0.26-6+deb8u1\n\nWorkarounds are:\n * Use a different config for gpg2 and gpg, e.g. ~/.gnupg/gpg.conf-2\n (https://www.gnupg.org/documentation/manuals/gnupg-2.0/Invoking-GPG.html )\n * or set gpg.program for git to gpg2.\n\n> And what git version was this? I see we've had a couple of workarounds\n> for gpg2, in particular Linus's v2.8.4-1-gb624a3e67f, but if you have\n> v2.10.0 or later that won't fix whatever issue you had.\n\ndpkg -s git | grep ^Version\nVersion: 1:2.1.4-2.1+deb8u2\n(I've checked the most current master source to see that git still calls \ngpg.program, otherwise followed advise on https://git-scm.com/community\nto send reports and questions to the list.)\n\n> Using the library sounds good, but a shorter-term immediate fix would\n> be to figure out what bug you encountered in our use of the\n> command-line version, and see if we've fixed that already or not.\n> Regardless of what we do with a gpg library in the future some distros\n> might want to backport such a small patch if we can come up with it.\n\nI guess a good simple approach would be to try \"gpg2\" first and then fall back \nto \"gpg\" or \"gpgv\" in case only these version are available.\n(Here is a report that puts forward using gpgv in some situations \nhttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852684 )\n\nAs there are other subtle potential issues with directly calling a gpg binary,\nusing libgpgme by default probably has other advantages as well. And if there \nare important functions missing the GnuPG-devs would like to hear about them.\n\nRegards,\nBernhard\n\n-- \nwww.intevation.de/~bernhard   +49 541 33 508 3-3\nIntevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998\nGeschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner\n"},{"id":"313884","messageId":"201703131214.31588.bernhard.reiter@intevation.de","threadId":"45341","inReplyTo":"CA+55aFxk7F103LADnmwc8wFySYQNiK6TcCQ0WSj+UTP-GihgcQ@mail.gmail.com","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Bernhard E. Reiter","fromEmail":"bernhard.reiter@intevation.de","sentAt":"2017-03-13T11:14:17Z","receivedAt":"2017-03-13T11:14:43Z","isPatch":false,"sender":{"key":"bernhard.reiter@intevation.de","avatar":null},"body":"Am Freitag 10 März 2017 19:54:19 schrieb Linus Torvalds:\n> On Fri, Mar 10, 2017 at 2:00 AM, Bernhard E. Reiter\n\n> > please consider using libgpgme interfacing to GnuPG, because the gpg\n> > command-line interface is not considered an official API to GnuPG by the\n> > GnuPG-devs and thus potentially unstable.\n>\n> Quite frankly, I will NAK this just based on previous bad experiences\n> with using \"helpful\" libraries.\n>\n> Maybe you can lay my worries to rest, but the problems with libraries\n> in this context tend to be\n\nAs gpgme is not just a helpful library, but the official API to GnuPG, it is \nwell supported by the GnuPG-Initiative itself and stable. Still there could \nbe problems and of course in some situations the disadvantages outweigh the \nadvantages. On the other hand we have seen a number of systematic problems \nwith \"just calling gpg\" that libgpgme tries to provide a solution to.\n\nSo it is too early to say that libgpgme would be right choice for git to me, \nbut it should be seriously considered. Grateful that you have written down \nsome of your concern, let me try give you some pointers.\n\n> - hard to personalize.\n>\n> At least right now, we just allow people to set their gpg binary.\n> I'm betting that the library would pick whatever random preferred\n> version, and in the process possibly screwed us.\n>\n> Example: what if somebody is actually using another pgp\n> implementation entirely for some reason, and is just scripting around\n> it?\n\n> Or maybe he's using the regular gnupg, but using different keys for\n> different projects (using \"--homedir\"). That's trivial with the\n> current model. How trivial is that with a library?\n\nhttps://www.gnupg.org/documentation/manuals/gpgme/Engine-Configuration.html\n\"\nYou can change the configuration of a backend engine, and thus change the \nexecutable program and configuration directory to be used. You can make these \nchanges the default or set them for some contexts individually. \n\"\n\nUsing a completely different OpenPGP implementation maybe a potential use case \nfor keeping a configuration option around. I did not deeply examine what git \nreally needs. Usually a different implementation will have quite a different \ncommand line interface, so it may require substaintial work to come up with a \nwrapper about that other OpenPGP implementation to provide the same command \nline interface as GnuPG.\n\n> - existing configuration\n>\n> This is the main problem I've seen in the past. Using the \"ssh\"\n> _program_ is easy. You add your keys, your config files, whatever, and\n> it \"just works\" (or rather, you fight it once and it definitely\n> doesn't \"just\" work, but then you copy your .ssh directory around for\n> the rest of your and forget how it ever worked, but it does).\n\ngpgme via gpg uses the existing configuration files (which you can also read \nand modify with gpgconf for implementiong GUIs).\n\n> - UI\n>\n> For things like gpg, the UI is traditionally horrible. But there\n> tends to be various things like password agents that help with caching\n> passwords and/or add a graphical UI to get the password when\n> necessary.\n\nAs the gpg binary itself speaks to gpg-agent, this is fully integrated when \nused via gpgme. Our GPA and Kleopatra GUIs work fine with gpgme and others as \nwell, because they call come together in the deeper engine functions of \nGnuPG.\n\n> - library versioning.\n>\n> I don't know why, but I've never *ever* met a library developer who\n> realized that libraries were all about stable API's, and the library\n> users don't want to fight different versions.\n\nIn my experience Werner (the lead GnuPG developers) is quite reasonable about \nkeeping APIs stable (he often goes out of his way to keep even the command \nline version stable, maybe he shouldn't do that to the command line options \nso you are more motivated to go to this official API gpgme. >:) )\n\n> And to make matters worse, the different versions (particularly if\n> you end up having to use a development version due to bugs or required\n> features etc) are always made horribly bad to even detect at\n> built-time automatically with simple #ifdef etc, so now you have to do\n> autoconf crap etc.\n\nhttps://www.gnupg.org/documentation/manuals/gpgme/Library-Version-Check.html\n\"\nThe function gpgme_check_version has four purposes. It can be used to retrieve \nthe version number of the library. In addition it can verify that the version \nnumber is higher than a certain required version number. In either case, the \nfunction initializes some sub-systems, and for this reason alone it must be \ninvoked early in your program, before you make use of the other functions in \nGPGME. The last purpose is to run selftests.\n\nAs a side effect for W32 based systems, the socket layer will get initialized. \n\"\n\n> Now, it may be that the pgpme library \"just works\" across\n> architectures and handles all of the above situations as gracefully as\n> the external program does. In that case - but _ONLY_ in that case -\n> would a switch-over to the library possibly be a good thing.\n\nAt least gpgme aims to fulfill these goals (and is used on many \narchitectures).\n\n> I'd be pleasantly surprised. But I *would* be surprised, because every\n> time I've seen that \"library vs program\" model, I've seen the above\n> issues.\n\nYour concerns are understandable, I've seen similiar problems with \"library vs \nprogram\" and the unix tools box approach gives a number of lessons on how to \nlosely couple components. Thanks again for taking the time and writing them \ndown. I've given you some pointers why gpgme indeed could be different and \nmay be an improvement for git (or other applications). I guess one of the \nnext steps would be for someone to look for specific points or try gpgme for \ngit purposes. Me and gnupg-devel@ are happy to take your questions or get \nfeedback.\n\nRegards,\nBernhard\n\n\n-- \nwww.intevation.de/~bernhard (CEO) +49 541 33 508 3-3\nIntevation GmbH, Osnabrück, Germany; Amtsgericht Osnabrück, HRB 18998\nOwned and run by Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner\n"},{"id":"313886","messageId":"201703131329.50322.bernhard.reiter@intevation.de","threadId":"45341","inReplyTo":"20170311001031.f5534omsrzkrzfzb@genre.crustytoothpaste.net","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Bernhard E. Reiter","fromEmail":"bernhard.reiter@intevation.de","sentAt":"2017-03-13T12:29:40Z","receivedAt":"2017-03-13T12:30:05Z","isPatch":false,"sender":{"key":"bernhard.reiter@intevation.de","avatar":null},"body":"Am Samstag 11 März 2017 01:10:31 schrieb brian m. carlson:\n> On Fri, Mar 10, 2017 at 11:00:07AM +0100, Bernhard E. Reiter wrote:\n> > but even better would be to move to libgpgme. >:)\n>\n> There are a couple potential problems I see with this approach. First,\n> I'd want to know whether gpgme supports gpgsm, which I know some people\n> use to sign commits and tags.\n\nYes, gpgme supports gpgsm.\nhttps://www.gnupg.org/documentation/manuals/gpgme/Cryptographic-Message-Syntax.html\n\"CMS is implemented by GpgSM, the S/MIME implementation for GnuPG.\"\n\n> Another issue is what happens to the git verify-* --raw output. Some\n> people want the ability to script signature verification. This can be\n> really important when you have automated systems verifying tags and\n> commits.\n>\n> For example, running the following commands, we can determine that Junio\n> signs his tags with SHA-1 (algorithm 2), while I sign my commits with\n> SHA-512 (algorithm 10).\n>\n> genre ok % git verify-tag --raw v2.12.0 2>&1 | grep VALIDSIG\n> [GNUPG:] VALIDSIG E1F036B1FEE7221FC778ECEFB0B5E88696AFE6CB 2017-02-24\n> 1487962205 0 4 0 1 2 00 96E07AF25771955980DAD10020D04E5A713660A7 genre ok %\n> git verify-commit --raw object-id-part10 2>&1 | grep VALIDSIG [GNUPG:]\n> VALIDSIG 5FC3A781776B26DF87F70C37BF535D811F52F68B 2017-03-06 1488760639 0 4\n> 0 1 10 00 88ACE9B29196305BA9947552F1BA225C0223B187\n>\n> There's literally no other way to get this information at the moment\n> (which is why I added the --raw option). A gpgme implementation would\n> need to expose this same information, at which point, we might as well\n> have used gpg directly.\n\nI'm quite optimistic that the use case can be covered implementing the \ngit --raw option using gpgme, but I don't know the best way right away.\n(I also wonder what would happen if someone manages to put in two or more \nsignatures in the object you are verifying.)\n\n> Because the amount of the gpg API we actually use is very small, a user\n> who wants to use a custom signature program (say, OpenBSD's signify),\n> can actually write a simple wrapper that mimics it and use that instead.\n\nI agree that this can be a use case against using libgpgme.\n\n> Finally, I work on a development system where work is done both as an\n> unprivileged user and as root. Because I use the same socket for both,\n> GnuPG screams bloody murder that the permissions are wrong. I know this\n> is secure in my scenario, but without a custom wrapper, I have to deal\n> with GnuPG polluting my terminal every time I sign a commit or a tag. A\n> gpgme implementation would need to honor the same wrapper script or\n> otherwise not scream to the terminal.\n\nI don't understand the scenario well enought to advise. Using the same socket \nsounds strange at the onset. In general libgpgme allows someone to use other \nbinaries (see my answer to Linus) and offers quite a number of possibilities. \nHowever there may be special cases that are not covered as good as using \neverything raw and manually. Libgpgme has to be this way and offering some \nstandard default otherwise it would use some of its merits. On the other hand\nthe code doing special things or calling gpg directly can have defects as well\nwhich is a significant drawback.\n\nRegards,\nBernhard\n-- \nwww.intevation.de/~bernhard (CEO) +49 541 33 508 3-3\nIntevation GmbH, Osnabrück, Germany; Amtsgericht Osnabrück, HRB 18998\nOwned and run by Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner\n"},{"id":"313887","messageId":"201703131350.00139.bernhard.reiter@intevation.de","threadId":"45341","inReplyTo":"a5161dd2-977b-8195-e558-01787fb3f01b@drmicha.warpmail.net","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Bernhard E. Reiter","fromEmail":"bernhard.reiter@intevation.de","sentAt":"2017-03-13T12:49:47Z","receivedAt":"2017-03-13T12:50:13Z","isPatch":false,"sender":{"key":"bernhard.reiter@intevation.de","avatar":null},"body":"Am Montag 13 März 2017 11:14:57 schrieb Michael J Gruber:\n> Ævar Arnfjörð Bjarmason venit, vidit, dixit 10.03.2017 15:23:\n> > On Fri, Mar 10, 2017 at 11:00 AM, Bernhard E. Reiter\n\n> >> please consider using libgpgme interfacing to GnuPG, because the gpg\n> >> command-line interface is not considered an official API to GnuPG by the\n> >> GnuPG-devs and thus potentially unstable.\n\n[example of gpg2 vs gpg option incompatibility cut]\n\n> > Using the library sounds good, but a shorter-term immediate fix would\n> > be to figure out what bug you encountered in our use of the\n> > command-line version, and see if we've fixed that already or not.\n\n> As far as I know, Git handles different GPG versions just fine.\n\nAs mentioned before: explicitely setting gpg.program to gpg2 helps if gpg\nchokes on the new config. Trying the `gpg2` binary first can be a simple fix. \nUsing libgpgme potentially solves this and other compatility options.\n\n> The problem is the \"difficult\" upgrade path and mixed installations with\n> gpg and gpg2.1+ that some distributions force upon you:\n>\n> As soon as you start gpg2.1, your (secret) key store is migrated to a\n> new format without technically invalidating it. Similarly, users may\n> enter gpg2.1+-only comand in the config that is actually shared with\n> gpg, throwing off any use of gpg - not just by git, but also by anything\n> that your distro requires gpg for (such as packaging tools and the like).\n\nYes, this is another example why trying `gpg2? first by default or using \nlibgpgme keeps trouble away from users.\n\n> In short: Users will run into problems anyway; git provides the quick\n> way out (git config gpg.program gpg2), users won't be as lucky with\n> other things that require gpg.\n\nApplication using libgpgme will behave fine and many user facing components \nuse it already. \n\n> As for the library: While - technically speaking - the command line is\n> not a stable API for gpg, it does work across versions of gpg, and gpg\n\n... to some extend.\n\n> 2.2 will be the first real stable branch that uses the new key store\n> layout. So I'd rather wait for that to stabilize before going away from\n> what turned out to be most stable so far.\n\nIt is not just about the key-store change as mentioned before. However\nI agree that a potential switch should be done with a current version of gpgme \nthat already has support for GnuPG 2.1/2, e.g. gpgme v1.8.0.\n\n> Note that we (git) refrain from parsing ordinary output/return codes of\n> gpg and use status-fd as we should (and as documented).\n\nIt is good to use --status-fd and --with-colons when calling gpg, you still \nhave to parse the results of status-fd as described in doc/DETAILs. \nhttps://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob;f=doc/DETAILS;hb=HEAD\n\nRegards,\nBernhard\n\n-- \nwww.intevation.de/~bernhard (CEO) +49 541 33 508 3-3\nIntevation GmbH, Osnabrück, Germany; Amtsgericht Osnabrück, HRB 18998\nOwned and run by Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner\n"},{"id":"313888","messageId":"20170313125324.ahznmrwflqvdrnnn@sigill.intra.peff.net","threadId":"45341","inReplyTo":"201703131214.31588.bernhard.reiter@intevation.de","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-13T12:53:24Z","receivedAt":"2017-03-13T12:53:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 13, 2017 at 12:14:17PM +0100, Bernhard E. Reiter wrote:\n\n> Using a completely different OpenPGP implementation maybe a potential use case \n> for keeping a configuration option around. I did not deeply examine what git \n> really needs. Usually a different implementation will have quite a different \n> command line interface, so it may require substaintial work to come up with a \n> wrapper about that other OpenPGP implementation to provide the same command \n> line interface as GnuPG.\n\nThe existing config option will have to stay, period. If we take it\naway, then we'd be breaking existing setups that do anything more exotic\nthan point it at a particular gpg binary.\n\nSo I think the possible patches in this area are:\n\n - adding gpgme support (probably as an optional build-time dependency)\n along with a config option to use it instead of gpg.program (or\n possibly just \"gpg.program = gpgme\" to trigger it).\n\n - defaulting to gpgme (when it's compiled in) instead of gpg.program\n\nThe first one by itself shouldn't hurt anybody, aside from carrying some\nextra code around.\n\nWe'd have to see how the first one goes before considering the second.\nIf we hit a point where people have problems and the advice is \"just set\ngpg.program to gpgme\", then that's a good sign that the default should\nbe flipped.\n\n-Peff\n"},{"id":"313939","messageId":"871su1arfj.fsf@gmail.com","threadId":"45341","inReplyTo":"20170311001031.f5534omsrzkrzfzb@genre.crustytoothpaste.net","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Christian Neukirchen","fromEmail":"chneukirchen@gmail.com","sentAt":"2017-03-13T19:48:48Z","receivedAt":"2017-03-13T19:49:23Z","isPatch":false,"sender":{"key":"chneukirchen@gmail.com","avatar":"https://avatars.githubusercontent.com/u/139?v=4"},"body":"\"brian m. carlson\" writes:\n\n> Because the amount of the gpg API we actually use is very small, a user\n> who wants to use a custom signature program (say, OpenBSD's signify),\n> can actually write a simple wrapper that mimics it and use that instead.\n\nFWIW, I did this, and it was quite some effort to figure out the\nactual API that is needed:\n\nhttp://chneukirchen.org/dotfiles/bin/git-signify\n\n-- \nChristian Neukirchen http://chneukirchen.org\n\n"},{"id":"314031","messageId":"13c66211-9671-5bd3-f3eb-96ffd5c39975@drmicha.warpmail.net","threadId":"45341","inReplyTo":"201703131350.00139.bernhard.reiter@intevation.de","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2017-03-14T10:39:12Z","receivedAt":"2017-03-14T10:39:20Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Bernhard E. Reiter venit, vidit, dixit 13.03.2017 13:49:\n> Am Montag 13 März 2017 11:14:57 schrieb Michael J Gruber:\n>> Ævar Arnfjörð Bjarmason venit, vidit, dixit 10.03.2017 15:23:\n>>> On Fri, Mar 10, 2017 at 11:00 AM, Bernhard E. Reiter\n> \n>>>> please consider using libgpgme interfacing to GnuPG, because the gpg\n>>>> command-line interface is not considered an official API to GnuPG by the\n>>>> GnuPG-devs and thus potentially unstable.\n> \n> [example of gpg2 vs gpg option incompatibility cut]\n> \n>>> Using the library sounds good, but a shorter-term immediate fix would\n>>> be to figure out what bug you encountered in our use of the\n>>> command-line version, and see if we've fixed that already or not.\n> \n>> As far as I know, Git handles different GPG versions just fine.\n> \n> As mentioned before: explicitely setting gpg.program to gpg2 helps if gpg\n> chokes on the new config. Trying the `gpg2` binary first can be a simple fix. \n> Using libgpgme potentially solves this and other compatility options.\n> \n>> The problem is the \"difficult\" upgrade path and mixed installations with\n>> gpg and gpg2.1+ that some distributions force upon you:\n>>\n>> As soon as you start gpg2.1, your (secret) key store is migrated to a\n>> new format without technically invalidating it. Similarly, users may\n>> enter gpg2.1+-only comand in the config that is actually shared with\n>> gpg, throwing off any use of gpg - not just by git, but also by anything\n>> that your distro requires gpg for (such as packaging tools and the like).\n> \n> Yes, this is another example why trying `gpg2? first by default or using \n> libgpgme keeps trouble away from users.\n\nNo, not at all. On the contrary: Using gpg2.1 without being asked to by\nthe user will migrate his key store! This can lead to tremendous\nproblems when a user manages his secret key store using gpg and git uses\nthe other secret key store (via gpg2.1)!\n\n>> In short: Users will run into problems anyway; git provides the quick\n>> way out (git config gpg.program gpg2), users won't be as lucky with\n>> other things that require gpg.\n> \n> Application using libgpgme will behave fine and many user facing components \n> use it already. \n> \n>> As for the library: While - technically speaking - the command line is\n>> not a stable API for gpg, it does work across versions of gpg, and gpg\n> \n> ... to some extend.\n\nI have no idea what you're implying here - but I have a pretty good idea\nof what works in current git and what not, including gpg usage (which is\nthe qualifier that you cut out).\n\n>> 2.2 will be the first real stable branch that uses the new key store\n>> layout. So I'd rather wait for that to stabilize before going away from\n>> what turned out to be most stable so far.\n> \n> It is not just about the key-store change as mentioned before. However\n> I agree that a potential switch should be done with a current version of gpgme \n> that already has support for GnuPG 2.1/2, e.g. gpgme v1.8.0.\n\nUnfortunately, gpgme does not solve the interoperability problems\nbetween gpg (1, classic, stable maint mode) or gpg2.0 (stable) and\ngpg2.1 (modern) key stores, and gpg2.2 (modern, stable) is not released yet.\n\n>> Note that we (git) refrain from parsing ordinary output/return codes of\n>> gpg and use status-fd as we should (and as documented).\n> \n> It is good to use --status-fd and --with-colons when calling gpg, you still \n> have to parse the results of status-fd as described in doc/DETAILs. \n> https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob;f=doc/DETAILS;hb=HEAD\n\nAgain, have you checked what we are doing in git land?\n\nYour agenda is pretty clear. It's also pretty clear from the above that\ngpgme is not the solution to the problem which is introduced by the\nmigration path to versions of gpg which are not declared stable by the\ngpg project, away from a gpg version which is not obsoleted by the gpg\nproject (2.0, maybe 1).\n\nAnd, really, key store migration was the only \"major\" thing we had to\nthink about to cope with gpg 2.1+ - it affected the test suite setup.\n\nSo, once 2.2 is out and stable and mainstream and we don't risk\nmigrating users' secret key store sneakily I'm all for defaulting to\ngpg2, and then is a good time for us to look into gpgme.\n\nMichael\n"},{"id":"314316","messageId":"201703171056.10468.bernhard.reiter@intevation.de","threadId":"45341","inReplyTo":"13c66211-9671-5bd3-f3eb-96ffd5c39975@drmicha.warpmail.net","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Bernhard E. Reiter","fromEmail":"bernhard.reiter@intevation.de","sentAt":"2017-03-17T09:56:02Z","receivedAt":"2017-03-17T09:58:11Z","isPatch":false,"sender":{"key":"bernhard.reiter@intevation.de","avatar":null},"body":"Am Dienstag 14 März 2017 11:39:12 schrieb Michael J Gruber:\n> >> The problem is the \"difficult\" upgrade path and mixed installations with\n> >> gpg and gpg2.1+ that some distributions force upon you:\n> >>\n> >> As soon as you start gpg2.1, your (secret) key store is migrated to a\n> >> new format without technically invalidating it. Similarly, users may\n> >> enter gpg2.1+-only comand in the config that is actually shared with\n> >> gpg, throwing off any use of gpg - not just by git, but also by anything\n> >> that your distro requires gpg for (such as packaging tools and the\n> >> like).\n> >\n> > Yes, this is another example why trying `gpg2? first by default or using\n> > libgpgme keeps trouble away from users.\n>\n> No, not at all. On the contrary: Using gpg2.1 without being asked to by\n> the user will migrate his key store! This can lead to tremendous\n> problems when a user manages his secret key store using gpg and git uses\n> the other secret key store (via gpg2.1)!\n\nIn the following scenario I agree with you:\n* The user only uses gpg (1) and only applications that use gpg (1)\n and does not have gpg2 installed.\n* Now a GnuPG package for 2.1/2.2 is installed and only the \"gpg2\"\n binary now points to it.\n* If then git will just switches using `gpg2` which is v2.1 because it \n came available, it will force the migration to the new store.\n\nMy statement was assuming a scenario where\n* the user has gpg v1 and gpg2 v2.0 installed and many (or all) applications \n use gpg2 via gpgme or by other means. I consider this typical for desktops\n or application servers using for desktop or development work.\n* Now git also uses gpg2 (via gpgme or by other means, a possible improvement \n we are just talking about)\n* If GnuPG 2.1/2.2 becomes available and of the already used application will\n trigger the keystore migration. git users will have a smooth experience\n because they were using gpg2 all along and then to use the new keystore.\n\n(It is still possible for some to configure libgpgme to use `gpg` v1 and all \napplications using gpgme will use it and you do not have to configure each \none separately. Because you should either use gpg v2.1 or gpg v1 \nexclusively, it makes sense to configure it close to gpg in its official api \nand not within each application. Again you can deviate from this.)\n\n> >> As for the library: While - technically speaking - the command line is\n> >> not a stable API for gpg, it does work across versions of gpg, and gpg\n> >\n> > ... to some extend.\n>\n> I have no idea what you're implying here - but I have a pretty good idea\n> of what works in current git and what not, including gpg usage (which is\n> the qualifier that you cut out).\n\nAs the command line is not a stable API to GnuPG, there were changes and there \nwill be changes to the command line over several versions. It maybe in the \npast that git's usage of the command line was stable and works well, because \nof the subset of functionality of GnuPG that git is using and good work done \nby you and other developers of the git GnuPG support. It may or may not be a \ngood choice for the future. As I've written before I'm not familiar enough \nwith git to make a final call here, but there are some signs and examples \nwhere using gpgme may be an improvements in the areas:\n\n* better configuration which gpg and which options to use for all applications\n* consistent behaviour from git with other applications\n* more robust programming interface over several gpg versions\n* (better performance when reusing the same gpgme \"context\" for many \nconsecutive crypto operations)\n\n> Unfortunately, gpgme does not solve the interoperability problems\n> between gpg (1, classic, stable maint mode) or gpg2.0 (stable) and\n> gpg2.1 (modern) key stores, and gpg2.2 (modern, stable) is not released\n> yet.\n\nThat is right, I also believe gpgme does not solve all interoperability \nproblems. I guess it solves some, but I would do more research to come up \nwith some specific examples.\n\n> >> Note that we (git) refrain from parsing ordinary output/return codes of\n> >> gpg and use status-fd as we should (and as documented).\n> >\n> > It is good to use --status-fd and --with-colons when calling gpg, you\n> > still have to parse the results of status-fd as described in doc/DETAILs.\n> > https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob;f=doc/DETAILS\n> >;hb=HEAD\n>\n> Again, have you checked what we are doing in git land?\n\nAs mentioned before: Only on the surface. I'm not deeply familiar with all git \nuse case of GnuPG. \n\n> Your agenda is pretty clear.\n\nYes, I'd like to improve the user experience of git users regarding crypto\noperations done with GnuPG. There are some signs that indicate that using \ngpgme may potentially improve this user experience (as outlined).\n\n> It's also pretty clear from the above that \n> gpgme is not the solution to the problem which is introduced by the\n> migration path to versions of gpg which are not declared stable by the\n> gpg project, away from a gpg version which is not obsoleted by the gpg\n> project (2.0, maybe 1).\n\nWhile I agree that gpgme does not solve all problems with this migration,\nit solves some problems with it (see above and my other email) \nand offers other advantages as the official API of GnuPG. Still it is the call \nof people more familiar with git usage and use cases to see if they have a \ngood reason to deviate from the common case and use a non-official way of \nautomating crypto operations with GnuPG.\n\n> And, really, key store migration was the only \"major\" thing we had to\n> think about to cope with gpg 2.1+ - it affected the test suite setup.\n>\n> So, once 2.2 is out and stable and mainstream and we don't risk\n> migrating users' secret key store sneakily I'm all for defaulting to\n> gpg2, and then is a good time for us to look into gpgme.\n\nThe key store migration is mainly an orthogonal issue and the problems will \nhappen with or without using gpgme. As GnuPG 2.2 is under way, it makes sense \nto look into an improved support now, to be ready if GnuPG 2.2 becomes more \nwidely used. It probably will get a lot more usage this year because of the \nECC support (and some other reasons).\n\nThanks all for your thoughts, using GnuPG, considering gpgme \nand in general for maintaining git as Free Software! :)\n\nBest Regards,\nBernhard\n\n-- \nwww.intevation.de/~bernhard (CEO) +49 541 33 508 3-3\nIntevation GmbH, Osnabrück, Germany; Amtsgericht Osnabrück, HRB 18998\nOwned and run by Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner\n"},{"id":"314932","messageId":"87poh9p70n.fsf@wheatstone.g10code.de","threadId":"45341","inReplyTo":"201703171056.10468.bernhard.reiter@intevation.de","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Werner Koch","fromEmail":"wk@gnupg.org","sentAt":"2017-03-22T17:15:36Z","receivedAt":"2017-03-22T17:56:13Z","isPatch":false,"sender":{"key":"wk@gnupg.org","avatar":null},"body":"On Fri, 17 Mar 2017 10:56, bernhard.reiter@intevation.de said:\n\n> As the command line is not a stable API to GnuPG, there were changes and there \n> will be changes to the command line over several versions. It maybe in the \n\nThat is not true. The command line interface has been stable for the\nlast 19 years. We only removed a left over PGP-2 backward compatibility\nin 2.1 (-kvv). I doubt that this has even been noticed.\n\n>> Unfortunately, gpgme does not solve the interoperability problems\n>> between gpg (1, classic, stable maint mode) or gpg2.0 (stable) and\n>> gpg2.1 (modern) key stores, and gpg2.2 (modern, stable) is not released\n>> yet.\n\nIt actually does. For the tasks git uses gpg you should not notice a\ndifference in gpgme between any of these versions. BTW, 2.1 was\nrealeased more than 2 years ago and 2.0 will reach EOL in 9 months.\n\n> That is right, I also believe gpgme does not solve all interoperability \n> problems. I guess it solves some, but I would do more research to come up \n\nThe main arguments pro GPGME are\n\n - There is generic stable API and ABI which did not changes since 0.4.1\n (2003-06-06) when we decided to redesign the API.\n\n - Iff verification of signatures needs a speedup we can do this inside\n GPGME and GnuPG without the GPGME user noticing that. Technically we\n will then run gpg as co-process controlled by GPGME; for gpgsm\n (S/MIME) this is already done but there has not yet been a need to do\n this also for gpg. Git could be the trigger to implement that.\n\n - The GPGME API would allow to provide a rich and stable output with\n the pretty print format. AFAICS the current %G* format characters\n are a bit limited and require that scripts need to look at the key\n for further information.\n\n> The key store migration is mainly an orthogonal issue and the problems will \n> happen with or without using gpgme. As GnuPG 2.2 is under way, it makes sense \n\nPlease note that 2.2 is more of a marketing term than a change. 2.1 is\nalready in use and will be the standard gpg version in several distros,\nincluding Debian.\n\nInteroperability with 1.4 is a bit cumbersome if you often add new keys.\nHowever, \"gpg --export | gpg1 --import\" is not too complicated. \n\nBe aware that ECC keys will be used more and more and they are not\nsupported by gpg1. Further we are currently defining a v5 key format to\nbe prepared for weaknesses in the SHA-1 based fingerprint. It is very\nunlikely that a v6 key format will be added to gpg1.\n\n\nSalam-Shalom,\n\n Werner\n\n-- \nDie Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz.\n"},{"id":"314942","messageId":"bec6098f-3016-0a41-02b0-a4e541a66bc4@digitalbrains.com","threadId":"45341","inReplyTo":"87poh9p70n.fsf@wheatstone.g10code.de","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Peter Lebbing","fromEmail":"peter@digitalbrains.com","sentAt":"2017-03-22T18:46:28Z","receivedAt":"2017-03-22T18:46:52Z","isPatch":false,"sender":{"key":"peter@digitalbrains.com","avatar":null},"body":"On 22/03/17 18:15, Werner Koch wrote:\n> It actually does. For the tasks git uses gpg you should not notice a\n> difference in gpgme between any of these versions.\n\nBernhard wrote \"interoperability problems between [...] key stores\". I'm\nunder the impression you are actually answering the question \"can GPGME\nbe used in the same way regardless of the GnuPG version\" instead?\n\n> Interoperability with 1.4 is a bit cumbersome if you often add new keys.\n> However, \"gpg --export | gpg1 --import\" is not too complicated. \n\nThis presumes that\n\n1) Keys are only updated on the 2.1 side\n2) Keys are not deleted\n3) Secret keys are never changed\n\nright?\n\n1) is trivially solvable. 2) is trickier, but can be done.\n\n3) is because GnuPG 1.4 cannot update a secret key at all. Adding a new\nsubkey fails with:\n\ngpg: key DCDFDFA4: already in secret keyring\ngpg: Total number processed: 1\ngpg: secret keys read: 1\ngpg: secret keys unchanged: 1\n\nYou could delete before re-importing, but what if the key on the 1.4\nside is actually the newer one? You'd lose all changes. Worst case:\nupdates of a single key on both sides. But that is perhaps pushing the\nlimit of sane use. Perhaps not, perhaps the user likes to use two\ndifferent frontends which use different GnuPG versions as their backend.\n\nLuckily, expiration extensions are picked up by just transferring the\npublic key part of a secret key, so that does work.\n\nPeter.\n\n-- \nI use the GNU Privacy Guard (GnuPG) in combination with Enigmail.\nYou can send me encrypted mail if you want some privacy.\nMy key is available at \n\n"},{"id":"315040","messageId":"201703230829.40741.bernhard.reiter@intevation.de","threadId":"45341","inReplyTo":"87poh9p70n.fsf@wheatstone.g10code.de","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Bernhard E. Reiter","fromEmail":"bernhard.reiter@intevation.de","sentAt":"2017-03-23T07:29:31Z","receivedAt":"2017-03-23T07:38:32Z","isPatch":false,"sender":{"key":"bernhard.reiter@intevation.de","avatar":null},"body":"Am Mittwoch 22 März 2017 18:15:36 schrieb Werner Koch:\n> On Fri, 17 Mar 2017 10:56, bernhard.reiter@intevation.de said:\n> > As the command line is not a stable API to GnuPG, there were changes and\n> > there will be changes to the command line over several versions. It maybe\n> > in the\n>\n> That is not true. The command line interface has been stable for the\n> last 19 years. We only removed a left over PGP-2 backward compatibility\n> in 2.1 (-kvv). I doubt that this has even been noticed.\n\nI was under the impression (and I do remember you telling me a few times)\nthat the output of the command line interaction did change a lot over times\nand using applications had issues. I've experienced a few of those over the \nlast 17 years. :) Still I would have to dig up the examples.\nYou seem to refer more to the existence and principal meaning of command line \noptions, which is very stable because of your efforts, which I've mentioned \nbefore. Contents of the resulting input and output strings has changed though \nand gpgme does this parsing for a using application in a more stable way.\n\nBest Regards,\nBernhard\n\n-- \nwww.intevation.de/~bernhard (CEO) +49 541 33 508 3-3\nIntevation GmbH, Osnabrück, Germany; Amtsgericht Osnabrück, HRB 18998\nOwned and run by Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner\n"},{"id":"315042","messageId":"87mvcco57j.fsf@wheatstone.g10code.de","threadId":"45341","inReplyTo":"bec6098f-3016-0a41-02b0-a4e541a66bc4@digitalbrains.com","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Werner Koch","fromEmail":"wk@gnupg.org","sentAt":"2017-03-23T06:52:16Z","receivedAt":"2017-03-23T07:56:14Z","isPatch":false,"sender":{"key":"wk@gnupg.org","avatar":null},"body":"On Wed, 22 Mar 2017 19:46, peter@digitalbrains.com said:\n> under the impression you are actually answering the question \"can GPGME\n> be used in the same way regardless of the GnuPG version\" instead?\n\nRight.\n\n> 3) is because GnuPG 1.4 cannot update a secret key at all. Adding a new\n> subkey fails with:\n\nIndeed, I was not anymore aware of this limitation in versions < 2.1.\nThere are even more conflicts when PGP-2 keys (to me the only valid\nreason to use gpg1 along with gpg2) or ECC keys need to be considered.\n\n\nShalom-Salam,\n\n Werner\n\n-- \nDie Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz.\n"},{"id":"315045","messageId":"87shm4mfch.fsf@wheatstone.g10code.de","threadId":"45341","inReplyTo":"201703230829.40741.bernhard.reiter@intevation.de","subject":"Re: Stable GnuPG interface, git should use GPGME","fromName":"Werner Koch","fromEmail":"wk@gnupg.org","sentAt":"2017-03-23T10:56:14Z","receivedAt":"2017-03-23T11:03:16Z","isPatch":false,"sender":{"key":"wk@gnupg.org","avatar":null},"body":"On Thu, 23 Mar 2017 08:29, bernhard.reiter@intevation.de said:\n\n> I was under the impression (and I do remember you telling me a few times)\n> that the output of the command line interaction did change a lot over times\n> and using applications had issues. I've experienced a few of those over the \n\nSure, but that is only for human consumption and not for scripts.\n\ngpg introduced its --status-fd machine interface 19 years ago.\n\n\nShalom-Salam,\n\n Werner\n\n\n-- \nDie Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz.\n"}]}