{"thread":{"id":"42997","subject":"Re: [OT] USENIX paper on Git","startedAt":"2016-08-03T17:04:49Z","lastAt":"2016-08-03T20:03:10Z","messageCount":15,"participants":["Santiago Torres","Johannes Schindelin","Jeff King","Stefan Beller","Junio C Hamano","David Lang"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"292904","messageId":"20160803152510.pgcd2edareqdxnmh@LykOS.localdomain","threadId":"42997","inReplyTo":"alpine.DEB.2.20.1608031721430.107993@virtualbox","subject":"Re: [OT] USENIX paper on Git","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-08-03T15:25:11Z","receivedAt":"2016-08-03T17:04:49Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":" > share things before they are published. Thankfully, this is OK in\n> > USENIX's book. Here's the link:\n> > http://i2.cdn.turner.com/cnnnext/dam/assets/160730192650-14new-week-in-politics-super-169.jpg\n> \n> While I had a good laugh, I am wondering whether this is the correct link?\n\nOh my god, sorry, I meant to p, not to ctrl + v. My head is all over the\nplace as of late.\n\nHere's the correct link:\n\nhttp://isis.poly.edu/~jcappos/papers/torres_toto_usenixsec-2016.pdf\n\nThanks!\n-Santiago.\n"},{"id":"292939","messageId":"20160803145830.uwssj4uhqfemhr4o@LykOS.localdomain","threadId":"42997","inReplyTo":"20160801224043.4qmf56pmv27riq4i@LykOS.localdomain","subject":"[OT] USENIX paper on Git","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-08-03T14:58:31Z","receivedAt":"2016-08-03T17:05:32Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"Hello everyone,\n\nI will be presenting a paper regarding the Git metadata issues that we\ndiscussed at the beginning on the year on USENIX '16. I'm writing To\nmake everyone in this ML aware that this work exists and to bring\neveryone into the loop.\n\nI'm open for feedback and corrections. If anything seems odd imprecise\nto the community, I can make an errata in the presentation (at least).\nI'll also try to work towards making corrections anywhere if possible;\nthis is my first publication, so I wasn't sure if it was possible to\nshare things before they are published. Thankfully, this is OK in\nUSENIX's book. Here's the link:\nhttp://i2.cdn.turner.com/cnnnext/dam/assets/160730192650-14new-week-in-politics-super-169.jpg\n\nI do mention of work towards fixing these issues in upcoming versions of\nGit. This is in reference to the issue with Git tags, although I hope to\ncontinue working on Git in general once I have more time for it. Thanks\nagain for all the patience reviewing patches and discussing everything.\n\nThanks!\n-Santiago.\n\nP.S. Let me know if anyone is going to USENIX. I'm looking forward to\nmeeting!\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/287649 *\nI believe it to be this, but gmane seems to be down. \n\n\n"},{"id":"292943","messageId":"alpine.DEB.2.20.1608031721430.107993@virtualbox","threadId":"42997","inReplyTo":"20160803145830.uwssj4uhqfemhr4o@LykOS.localdomain","subject":"Re: [OT] USENIX paper on Git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-03T15:22:21Z","receivedAt":"2016-08-03T17:05:36Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Santiago,\n\nOn Wed, 3 Aug 2016, Santiago Torres wrote:\n\n> I'm open for feedback and corrections. If anything seems odd imprecise\n> to the community, I can make an errata in the presentation (at least).\n> I'll also try to work towards making corrections anywhere if possible;\n> this is my first publication, so I wasn't sure if it was possible to\n> share things before they are published. Thankfully, this is OK in\n> USENIX's book. Here's the link:\n> http://i2.cdn.turner.com/cnnnext/dam/assets/160730192650-14new-week-in-politics-super-169.jpg\n\nWhile I had a good laugh, I am wondering whether this is the correct link?\n\nCiao,\nJohannes\n"},{"id":"292944","messageId":"20160803171124.yzm5xs67empuka7o@sigill.intra.peff.net","threadId":"42997","inReplyTo":"20160803145830.uwssj4uhqfemhr4o@LykOS.localdomain","subject":"Re: [OT] USENIX paper on Git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-03T17:11:24Z","receivedAt":"2016-08-03T17:13:06Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 03, 2016 at 10:58:31AM -0400, Santiago Torres wrote:\n\n> I will be presenting a paper regarding the Git metadata issues that we\n> discussed at the beginning on the year on USENIX '16. I'm writing To\n> make everyone in this ML aware that this work exists and to bring\n> everyone into the loop.\n> \n> I'm open for feedback and corrections. If anything seems odd imprecise\n> to the community, I can make an errata in the presentation (at least).\n> I'll also try to work towards making corrections anywhere if possible;\n> this is my first publication, so I wasn't sure if it was possible to\n> share things before they are published. Thankfully, this is OK in\n> USENIX's book. Here's the link:\n\nI read it over. As far as technical descriptions of Git, it looked OK.\nI found a few minor nits, but nothing worth caring about (e.g., ref\nstorage is not quite so simple these days as 40 bytes in a file, but\nthere is no point describing the whole packed-refs scheme in your paper,\nas it does not change anything with respect to your work).\n\nHere are my comments on the work itself. They're critical, but meant in\na friendly way. :)\n\nAs far as the attack goes, I'm still not convinced this is all that\n_interesting_ an attack in the real world. What it boils down to is: the\nref state is not signed or authenticated in any way, so somebody who can\ncompromise your server repo or do a MiTM can lie about where the refs\nare (even if individual commits are signed).\n\nSo if you want to treat Git as a cryptographic end-to-end distribution\nmechanism, then no, it fails horribly at that. But the state of the art\nin source code distribution, no matter which system you use, is much\nless advanced than that. People download tarballs, even ones with GPG\nsignatures, all the time without verifying their contents. Most packages\ndistribute a sha1sum or similar (sometimes even gpg-signed), but quite\noften the source of authority is questionable.\n\nFor example, consider somebody downloading a new package for the first\ntime. They don't know the author in any out-of-band way, so any\nsignatures are likely meaningless. They _might_ be depending on the\nsource domain for some security (and using some hierarchical PKI like\nTLS+x.509 to be sure they're talking to that domain), but in your threat\nmodel, even well-known hosts like FSF could be compromised internally.\n\nSo yes, I think the current state of affairs (especially open-source) is\nthat people download and run possibly-compromised code all the time. But\nI'm not sure that lack of tool support is really the limiting factor. Or\nthat it has turned out to be all that big a problem in practice.\n\nAnyway. As far as your solution goes, I'll admit I skimmed over the\ndetails, but it looks like basically a sequence of signatures producing\na chain of state (so the tip state is signed, but you can also make sure\nthe chain connects from your current state to what the server claims is\nthe new tip state, and not a replay of some old state). Please correct\nme if that's not accurate. :)\n\nWithout having thought too hard about it, it seems like you could do the\nsame thing with push certs, as they have both a \"before\" and \"after\" for\neach ref. So if in addition to fetching the refs from a server, I fetch\nall of the push certs, I should be able to walk the chain of push certs\nfrom the one at my current state, to the one at the tip state, making\nsure that each one builds on the last.\n\nThere are two cases that I don't think that handles, but that I also\ndon't see are handled in your solution:\n\n  - if I am cloning for the first time, I have no \"current\" state to\n    base the chain from. An attacker could serve me any old signed ref\n    state, and I have no way to know that it's old (except perhaps by\n    seeing the wall-clock timestamp and comparing it to my clock; this\n    isn't a proof but may be cause for suspicion if it's too old)\n\n  - if there is a chain of signatures, the attacker must follow the\n    chain, but they can always withhold links from the end. So imagine a\n    repository has held a sequence of signed states (A, B, C), that B\n    has a bug, C has the fix, and I am at A. An attacker can serve me B\n    and I cannot know without out-of-band information that it is not the\n    correct tip (because until C was created, it _was_ the correct tip).\n\n    I think this is actually a generalization of the cloning issue\n    (where state \"A\" is simply \"I have no existing state yet\").\n\nSo it seems like there is room for better tooling around push-certs\n(e.g., to fetch and verify the chaining automatically). I think git in\ngeneral is quite weak in automatic tooling for verifications. There are\nroom for signatures in the data format and tools for checking that the\nbytes haven't been touched, but there's almost nothing to tell you that\nsignatures make any sense, tools for handling trust, etc.\n\nI think your solution also had some mechanisms for adding trusted keys\nas part of the hash chain. I'm not convinced that's something that\nshould be part of git's solution in particular, and not an out-of-band\nthing handled as part of the PKI. Because it's really a group key\nmanagement problem, and applies to anything you might sign.\n\n-Peff\n"},{"id":"292945","messageId":"CAGZ79kZPPvi2jpFgu1MZ97gZm=QqBBiZ-XcXUJphm3UzNRP-FQ@mail.gmail.com","threadId":"42997","inReplyTo":"20160803152510.pgcd2edareqdxnmh@LykOS.localdomain","subject":"Re: [OT] USENIX paper on Git","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-08-03T17:14:21Z","receivedAt":"2016-08-03T17:14:27Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Aug 3, 2016 at 8:25 AM, Santiago Torres <santiago@nyu.edu> wrote:\n>  > share things before they are published. Thankfully, this is OK in\n>> > USENIX's book. Here's the link:\n>> > http://i2.cdn.turner.com/cnnnext/dam/assets/160730192650-14new-week-in-politics-super-169.jpg\n>>\n>> While I had a good laugh, I am wondering whether this is the correct link?\n>\n> Oh my god, sorry, I meant to p, not to ctrl + v. My head is all over the\n> place as of late.\n>\n> Here's the correct link:\n>\n> http://isis.poly.edu/~jcappos/papers/torres_toto_usenixsec-2016.pdf\n\nIn 4.1 you write:\n> Finally, Git submodules are also vulnerable, as they automatically track\n> a tag (or branch). If a build dependency is included in a project as a part\n> of the submodule, a package might be vulnerable via an underlying library.\n\nSubmodules actually track commits, not tags or branches.\n\nThis is confusing for some users, e.g. the user intended to track\na library at version 1.1, but it tracks 1234abcd instead (which is what\n1.1 points at).\n\nThanks,\nStefan\n\n>\n> Thanks!\n> -Santiago.\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"292946","messageId":"20160803172242.5jrd4saq7sga6iql@LykOS.localdomain","threadId":"42997","inReplyTo":"CAGZ79kZPPvi2jpFgu1MZ97gZm=QqBBiZ-XcXUJphm3UzNRP-FQ@mail.gmail.com","subject":"Re: [OT] USENIX paper on Git","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-08-03T17:22:43Z","receivedAt":"2016-08-03T17:29:07Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"On Wed, Aug 03, 2016 at 10:14:21AM -0700, Stefan Beller wrote:\n> On Wed, Aug 3, 2016 at 8:25 AM, Santiago Torres <santiago@nyu.edu> wrote:\n> >  > share things before they are published. Thankfully, this is OK in\n> >> > USENIX's book. Here's the link:\n> >> > http://i2.cdn.turner.com/cnnnext/dam/assets/160730192650-14new-week-in-politics-super-169.jpg\n> >>\n> >> While I had a good laugh, I am wondering whether this is the correct link?\n> >\n> > Oh my god, sorry, I meant to p, not to ctrl + v. My head is all over the\n> > place as of late.\n> >\n> > Here's the correct link:\n> >\n> > http://isis.poly.edu/~jcappos/papers/torres_toto_usenixsec-2016.pdf\n> \n> In 4.1 you write:\n> > Finally, Git submodules are also vulnerable, as they automatically track\n> > a tag (or branch). If a build dependency is included in a project as a part\n> > of the submodule, a package might be vulnerable via an underlying library.\n> \n> Submodules actually track commits, not tags or branches.\n> \n> This is confusing for some users, e.g. the user intended to track\n> a library at version 1.1, but it tracks 1234abcd instead (which is what\n> 1.1 points at).\n\nI'm assuming that git submodule update does update where the ref points\nto, does it not?\n\nlet me dig into this and try to take the necessary measures to correct\nthis\n\nThanks for the feedback!\n\n-Santiago.\n"},{"id":"292947","messageId":"xmqq37mlahzi.fsf@gitster.mtv.corp.google.com","threadId":"42997","inReplyTo":"20160803171124.yzm5xs67empuka7o@sigill.intra.peff.net","subject":"Re: [OT] USENIX paper on Git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-03T17:18:41Z","receivedAt":"2016-08-03T17:34:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> Here are my comments on the work itself. They're critical, but meant in\n> a friendly way. :)\n\nA tl;dr version of your analysis seems to me that \"you solve it the\nsame way as the push certificate solves it (including the limitation\nthe latter has)\".\n\nIf that is the case, I think the solution presented in the paper is\na good one ;-).\n"},{"id":"292948","messageId":"20160803174459.knlmyx6txmggixzi@LykOS.localdomain","threadId":"42997","inReplyTo":"20160803171124.yzm5xs67empuka7o@sigill.intra.peff.net","subject":"Re: [OT] USENIX paper on Git","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-08-03T17:45:00Z","receivedAt":"2016-08-03T17:45:45Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"Hello,\n\n> Here are my comments on the work itself. They're critical, but meant in\n> a friendly way. :)\n> \n\nThanks! If anything, the community here has been incredibly helpful in\nhelping me understand everything.\n\n> As far as the attack goes, I'm still not convinced this is all that\n> _interesting_ an attack in the real world. What it boils down to is: the\n> ref state is not signed or authenticated in any way, so somebody who can\n> compromise your server repo or do a MiTM can lie about where the refs\n> are (even if individual commits are signed).\n\nYup, that's pretty much it. I do agree that some of these attacks feel\ntangential in the way it should be used. I also agree that, if git is\nused how the Linux kernel or git.git does, then these attacks are rather\nhard to pull off. \n\n> \n> So if you want to treat Git as a cryptographic end-to-end distribution\n> mechanism, then no, it fails horribly at that. But the state of the art\n> in source code distribution, no matter which system you use, is much\n> less advanced than that. People download tarballs, even ones with GPG\n> signatures, all the time without verifying their contents. Most packages\n> distribute a sha1sum or similar (sometimes even gpg-signed), but quite\n> often the source of authority is questionable.\n\nYes, this happens an awful lot of times. We did some work with python's\npypi last year, and we found out that less than 1% of people actually\ndownloaded the gpg signature for the package they are retrieving[1].\n\n> \n> For example, consider somebody downloading a new package for the first\n> time. They don't know the author in any out-of-band way, so any\n> signatures are likely meaningless. They _might_ be depending on the\n> source domain for some security (and using some hierarchical PKI like\n> TLS+x.509 to be sure they're talking to that domain), but in your threat\n> model, even well-known hosts like FSF could be compromised internally.\n> \n> So yes, I think the current state of affairs (especially open-source) is\n> that people download and run possibly-compromised code all the time. But\n> I'm not sure that lack of tool support is really the limiting factor. Or\n> that it has turned out to be all that big a problem in practice.\n\nI couldn't agree more. I feel that OSS is slowly moving towards a more\ncryptographically robust, trust-based way of doing things, which I find\npleasing.\n\n> \n> Anyway. As far as your solution goes, I'll admit I skimmed over the\n> details, but it looks like basically a sequence of signatures producing\n> a chain of state (so the tip state is signed, but you can also make sure\n> the chain connects from your current state to what the server claims is\n> the new tip state, and not a replay of some old state). Please correct\n> me if that's not accurate. :)\n\nYeah, this sounds about right.\n\n> \n> Without having thought too hard about it, it seems like you could do the\n> same thing with push certs, as they have both a \"before\" and \"after\" for\n> each ref. So if in addition to fetching the refs from a server, I fetch\n> all of the push certs, I should be able to walk the chain of push certs\n> from the one at my current state, to the one at the tip state, making\n> sure that each one builds on the last.\n\nYeah, when we looked at push certs I actually thought that \"chaining\" the\ncerts could achieve a similar effect to the solution I described.\n\n> \n> There are two cases that I don't think that handles, but that I also\n> don't see are handled in your solution:\n> \n>   - if I am cloning for the first time, I have no \"current\" state to\n>     base the chain from. An attacker could serve me any old signed ref\n>     state, and I have no way to know that it's old (except perhaps by\n>     seeing the wall-clock timestamp and comparing it to my clock; this\n>     isn't a proof but may be cause for suspicion if it's too old)\n\nHmm, I didn't think about it. Let me jot it down. This sounds\ninteresting. I recall you mentioning metadata expiration back then.\n\n> \n>   - if there is a chain of signatures, the attacker must follow the\n>     chain, but they can always withhold links from the end. So imagine a\n>     repository has held a sequence of signed states (A, B, C), that B\n>     has a bug, C has the fix, and I am at A. An attacker can serve me B\n>     and I cannot know without out-of-band information that it is not the\n>     correct tip (because until C was created, it _was_ the correct tip).\n\nI think we address this by using the \"nonce bag\". We basically force the\nserver to fork the user's history if it withholds changes from on group\nto the other. By doing so, the user's nonce can't be added to any other\nhistory. I don't think this is noticeable from start though.\n\n> \n>     I think this is actually a generalization of the cloning issue\n>     (where state \"A\" is simply \"I have no existing state yet\").\n> \n> So it seems like there is room for better tooling around push-certs\n> (e.g., to fetch and verify the chaining automatically). \n\nYeah, I think that in-band cert distribution and automatic validation\nmay be a desirable feature, but there may be reasons as to why this is\nnot appealing. (I already have some in mind).\n\n> I think git in general is quite weak in automatic tooling for\n> verifications. There are room for signatures in the data format and\n> tools for checking that the bytes haven't been touched, but there's\n> almost nothing to tell you that signatures make any sense, tools for\n> handling trust, etc.\n\nYes, from our previous interactions, it seems that git's philosophy\nfocuses on providing the right information to users/tools and let those\ntools make the call of whether something is fishy. I don't think this is\nnecessarily bad.\n\n> \n> I think your solution also had some mechanisms for adding trusted keys\n> as part of the hash chain. I'm not convinced that's something that\n> should be part of git's solution in particular, and not an out-of-band\n> thing handled as part of the PKI. Because it's really a group key\n> management problem, and applies to anything you might sign.\n\nI see. What about, for example, having an official \"overlay\" on git for\nsigning and verification of a repository? (e.g., similar to what\nmonotone does). I see that other VCS's have a plugin mechanism, and they\nhost official plugins.\n\nThanks a lot!\n-Santiago.\n\n[1] https://isis.poly.edu/~jcappos/papers/kuppusamy_nsdi_16.pdf\n"},{"id":"292952","messageId":"xmqqy44d92md.fsf@gitster.mtv.corp.google.com","threadId":"42997","inReplyTo":"20160803172242.5jrd4saq7sga6iql@LykOS.localdomain","subject":"Re: [OT] USENIX paper on Git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-03T17:35:54Z","receivedAt":"2016-08-03T17:49:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Santiago Torres <santiago@nyu.edu> writes:\n\n>> Submodules actually track commits, not tags or branches.\n>> \n>> This is confusing for some users, e.g. the user intended to track\n>> a library at version 1.1, but it tracks 1234abcd instead (which is what\n>> 1.1 points at).\n>\n> I'm assuming that git submodule update does update where the ref points\n> to, does it not?\n\nI think you may configure the command to do so, instead of the\ndefault \"detach at the commit recorded in the superproject\".\n\nBut then your tree immediately will be marked by \"git status\" as\n\"modified\" at such a submodule, meaning \"what you have in the\nworking tree is different from what the commit in the superproject\nwants you to have\", I would think.\n\n"},{"id":"292953","messageId":"20160803175836.yrsnusbuhl2hvp3h@LykOS.localdomain","threadId":"42997","inReplyTo":"xmqqy44d92md.fsf@gitster.mtv.corp.google.com","subject":"Re: [OT] USENIX paper on Git","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-08-03T17:58:36Z","receivedAt":"2016-08-03T17:59:09Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"On Wed, Aug 03, 2016 at 10:35:54AM -0700, Junio C Hamano wrote:\n> Santiago Torres <santiago@nyu.edu> writes:\n> \n> >> Submodules actually track commits, not tags or branches.\n> >> \n> >> This is confusing for some users, e.g. the user intended to track a\n> >> library at version 1.1, but it tracks 1234abcd instead (which is\n> >> what 1.1 points at).\n> >\n> > I'm assuming that git submodule update does update where the ref\n> > points to, does it not?\n> \n> I think you may configure the command to do so, instead of the default\n> \"detach at the commit recorded in the superproject\".\n> \n> But then your tree immediately will be marked by \"git status\" as\n> \"modified\" at such a submodule, meaning \"what you have in the working\n> tree is different from what the commit in the superproject wants you\n> to have\", I would think.\n> \n\nAh, I see where is my confusion. Thanks for the correction :)\n\n-Santiago.\n"},{"id":"292954","messageId":"20160803175854.vys3f7tpfhzeqzsu@sigill.intra.peff.net","threadId":"42997","inReplyTo":"20160803174459.knlmyx6txmggixzi@LykOS.localdomain","subject":"Re: [OT] USENIX paper on Git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-03T17:58:54Z","receivedAt":"2016-08-03T17:59:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 03, 2016 at 01:45:00PM -0400, Santiago Torres wrote:\n\n> >   - if there is a chain of signatures, the attacker must follow the\n> >     chain, but they can always withhold links from the end. So imagine a\n> >     repository has held a sequence of signed states (A, B, C), that B\n> >     has a bug, C has the fix, and I am at A. An attacker can serve me B\n> >     and I cannot know without out-of-band information that it is not the\n> >     correct tip (because until C was created, it _was_ the correct tip).\n> \n> I think we address this by using the \"nonce bag\". We basically force the\n> server to fork the user's history if it withholds changes from on group\n> to the other. By doing so, the user's nonce can't be added to any other\n> history. I don't think this is noticeable from start though.\n\nOK, I think that is in the details I glossed over. ;)\n\nIf you are effectively preventing the server from showing different\nstates to different people, then at least that lets the \"main\"\ndevelopers notice problems (because at least one of them already saw \"C\"\nbecause they wrote it).\n\n> > I think git in general is quite weak in automatic tooling for\n> > verifications. There are room for signatures in the data format and\n> > tools for checking that the bytes haven't been touched, but there's\n> > almost nothing to tell you that signatures make any sense, tools for\n> > handling trust, etc.\n> \n> Yes, from our previous interactions, it seems that git's philosophy\n> focuses on providing the right information to users/tools and let those\n> tools make the call of whether something is fishy. I don't think this is\n> necessarily bad.\n\nI think it's half philosophy (git strives for flexibility, and so aims\nto provide low-level tooling that you can build on), and half that\nnobody has bothered to implement a sane set of automatic checks.\n\nThere's definitely some low-hanging fruit there. I think we've discussed\nthings like checking that verifying refs/tags/v1.0.0 actually gets you a\ntag that says \"v1.0.0\" in it. But I'd love to see a framework either\nbuilt into or on top of git that would implement sensible policies, and\nmake out-of-the-box verification easy to do. Then people might actually\nuse it. :)\n\n> > I think your solution also had some mechanisms for adding trusted keys\n> > as part of the hash chain. I'm not convinced that's something that\n> > should be part of git's solution in particular, and not an out-of-band\n> > thing handled as part of the PKI. Because it's really a group key\n> > management problem, and applies to anything you might sign.\n> \n> I see. What about, for example, having an official \"overlay\" on git for\n> signing and verification of a repository? (e.g., similar to what\n> monotone does). I see that other VCS's have a plugin mechanism, and they\n> host official plugins.\n\nIn general, if something is more general than git, I'd like to see a\ngeneral tool address it, and then add support to git to make use of the\ntool.\n\nFor group key management, I specifically was wondering if you could do\nsomething like:\n\n  - start with some seed GPG keys for the project\n\n  - existing keys can sign or revoke certificates to add or remove other\n    keys to/from the project; you could even require a threshold of\n    signatures, etc.\n\n  - those keys could be used for signing git pushes, but also for other\n    things, like signing tarballs, used as encryption keys for sending\n    for-developers-eyes-only security reports, etc\n\n    You'd want a tool that asks not just \"is this signed\" but \"is this\n    signed _by a key that is valid for this project_\".\n\nAnd then git support would just consist of feeding signatures to\n\"gpg-group --project=...\" instead of \"gpg\". Management of the group\nwould be out-of-band from git, which is in some ways good and in some\nways bad.\n\nI think a naive implementation would be pretty easy, but I've glossed\nover all of the chaining properties we've discussed. So whatever\nmechanism you use to receive updates to the key-group would have all the\nsame problems (e.g., withholding revocations of compromised keys). It's\nstill orders of magnitude ahead of what's currently happening\nday-to-day. :)\n\n-Peff\n"},{"id":"292955","messageId":"CAGZ79kY8knsXEVqfbdoHGdEzOfygVoaZi43jy_yERnx=enuRcA@mail.gmail.com","threadId":"42997","inReplyTo":"20160803172242.5jrd4saq7sga6iql@LykOS.localdomain","subject":"Re: [OT] USENIX paper on Git","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-08-03T17:35:39Z","receivedAt":"2016-08-03T18:02:31Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Aug 3, 2016 at 10:22 AM, Santiago Torres <santiago@nyu.edu> wrote:\n> On Wed, Aug 03, 2016 at 10:14:21AM -0700, Stefan Beller wrote:\n>> On Wed, Aug 3, 2016 at 8:25 AM, Santiago Torres <santiago@nyu.edu> wrote:\n>> >  > share things before they are published. Thankfully, this is OK in\n>> >> > USENIX's book. Here's the link:\n>> >> > http://i2.cdn.turner.com/cnnnext/dam/assets/160730192650-14new-week-in-politics-super-169.jpg\n>> >>\n>> >> While I had a good laugh, I am wondering whether this is the correct link?\n>> >\n>> > Oh my god, sorry, I meant to p, not to ctrl + v. My head is all over the\n>> > place as of late.\n>> >\n>> > Here's the correct link:\n>> >\n>> > http://isis.poly.edu/~jcappos/papers/torres_toto_usenixsec-2016.pdf\n>>\n>> In 4.1 you write:\n>> > Finally, Git submodules are also vulnerable, as they automatically track\n>> > a tag (or branch). If a build dependency is included in a project as a part\n>> > of the submodule, a package might be vulnerable via an underlying library.\n>>\n>> Submodules actually track commits, not tags or branches.\n>>\n>> This is confusing for some users, e.g. the user intended to track\n>> a library at version 1.1, but it tracks 1234abcd instead (which is what\n>> 1.1 points at).\n>\n> I'm assuming that git submodule update does update where the ref points\n> to, does it not?\n>\n> let me dig into this and try to take the necessary measures to correct\n> this\n>\n\n\"git submodule update\" updates to the recorded sha1, which I assume is used\nby downstream users. If you are a maintainer and  you want to update the\nlibrary used, you'd be interested in have \"git submodule update\n--remote\" to update\nthe sha1 to the tracking branch, which then exposes the attacks presented.\n\nThanks,\nStefan\n"},{"id":"292957","messageId":"20160803180214.rxz455nlcm5hqn37@LykOS.localdomain","threadId":"42997","inReplyTo":"CAGZ79kY8knsXEVqfbdoHGdEzOfygVoaZi43jy_yERnx=enuRcA@mail.gmail.com","subject":"Re: [OT] USENIX paper on Git","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-08-03T18:02:15Z","receivedAt":"2016-08-03T18:02:35Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"On Wed, Aug 03, 2016 at 10:35:39AM -0700, Stefan Beller wrote:\n> On Wed, Aug 3, 2016 at 10:22 AM, Santiago Torres <santiago@nyu.edu> wrote:\n> > On Wed, Aug 03, 2016 at 10:14:21AM -0700, Stefan Beller wrote:\n> >> On Wed, Aug 3, 2016 at 8:25 AM, Santiago Torres <santiago@nyu.edu> wrote:\n> >> >  > share things before they are published. Thankfully, this is OK in\n> >> >> > USENIX's book. Here's the link:\n> >> >> > http://i2.cdn.turner.com/cnnnext/dam/assets/160730192650-14new-week-in-politics-super-169.jpg\n> >> >>\n> >> >> While I had a good laugh, I am wondering whether this is the correct link?\n> >> >\n> >> > Oh my god, sorry, I meant to p, not to ctrl + v. My head is all over the\n> >> > place as of late.\n> >> >\n> >> > Here's the correct link:\n> >> >\n> >> > http://isis.poly.edu/~jcappos/papers/torres_toto_usenixsec-2016.pdf\n> >>\n> >> In 4.1 you write:\n> >> > Finally, Git submodules are also vulnerable, as they automatically track\n> >> > a tag (or branch). If a build dependency is included in a project as a part\n> >> > of the submodule, a package might be vulnerable via an underlying library.\n> >>\n> >> Submodules actually track commits, not tags or branches.\n> >>\n> >> This is confusing for some users, e.g. the user intended to track\n> >> a library at version 1.1, but it tracks 1234abcd instead (which is what\n> >> 1.1 points at).\n> >\n> > I'm assuming that git submodule update does update where the ref points\n> > to, does it not?\n> >\n> > let me dig into this and try to take the necessary measures to correct\n> > this\n> >\n> \n> \"git submodule update\" updates to the recorded sha1, which I assume is used\n> by downstream users. If you are a maintainer and  you want to update the\n> library used, you'd be interested in have \"git submodule update\n> --remote\" to update\n> the sha1 to the tracking branch, which then exposes the attacks presented.\n\nI see, I just tried to reproduce this, and it seems that, with a simple\ngit clone --recursive [path], the submodule fetched does not update to\nthe \"malicious ref.\" You're right.\n\nSo, in the end, git submodule update --remote also requires you to\ncreate a new tree, right? Then this attack wouldn't be possible by just\nfiddling with the refs if signing is in place, right?\n\nThanks for clarifying!\n-Santiago.\n"},{"id":"292963","messageId":"20160803183136.mck5fqo4jm7hj6i2@LykOS.localdomain","threadId":"42997","inReplyTo":"20160803175854.vys3f7tpfhzeqzsu@sigill.intra.peff.net","subject":"Re: [OT] USENIX paper on Git","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-08-03T18:31:37Z","receivedAt":"2016-08-03T18:31:58Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"On Wed, Aug 03, 2016 at 01:58:54PM -0400, Jeff King wrote:\n> On Wed, Aug 03, 2016 at 01:45:00PM -0400, Santiago Torres wrote:\n> \n> > >   - if there is a chain of signatures, the attacker must follow the\n> > >     chain, but they can always withhold links from the end. So imagine a\n> > >     repository has held a sequence of signed states (A, B, C), that B\n> > >     has a bug, C has the fix, and I am at A. An attacker can serve me B\n> > >     and I cannot know without out-of-band information that it is not the\n> > >     correct tip (because until C was created, it _was_ the correct tip).\n> > \n> > I think we address this by using the \"nonce bag\". We basically force the\n> > server to fork the user's history if it withholds changes from on group\n> > to the other. By doing so, the user's nonce can't be added to any other\n> > history. I don't think this is noticeable from start though.\n> \n> OK, I think that is in the details I glossed over. ;)\n> \n> If you are effectively preventing the server from showing different\n> states to different people, then at least that lets the \"main\"\n> developers notice problems (because at least one of them already saw \"C\"\n> because they wrote it).\n\nyeah, that was one of our assumptions. I think it's unrealistic to think\nthat people do not coordinate over mailing lists or other means.\n\n> \n> > > I think git in general is quite weak in automatic tooling for\n> > > verifications. There are room for signatures in the data format and\n> > > tools for checking that the bytes haven't been touched, but there's\n> > > almost nothing to tell you that signatures make any sense, tools for\n> > > handling trust, etc.\n> > \n> > Yes, from our previous interactions, it seems that git's philosophy\n> > focuses on providing the right information to users/tools and let those\n> > tools make the call of whether something is fishy. I don't think this is\n> > necessarily bad.\n> \n> I think it's half philosophy (git strives for flexibility, and so aims\n> to provide low-level tooling that you can build on), and half that\n> nobody has bothered to implement a sane set of automatic checks.\n> \n> There's definitely some low-hanging fruit there. I think we've discussed\n> things like checking that verifying refs/tags/v1.0.0 actually gets you a\n> tag that says \"v1.0.0\" in it. But I'd love to see a framework either\n> built into or on top of git that would implement sensible policies, and\n> make out-of-the-box verification easy to do. Then people might actually\n> use it. :)\n\nYeah, that's one of the long-term goals with my PhD, but it's still on\nthe early stages, and I don't have much done yet in that field. I can of\ncourse share this around once it's more mature if that's ok with people\nin here :)\n\n> \n> > > I think your solution also had some mechanisms for adding trusted keys\n> > > as part of the hash chain. I'm not convinced that's something that\n> > > should be part of git's solution in particular, and not an out-of-band\n> > > thing handled as part of the PKI. Because it's really a group key\n> > > management problem, and applies to anything you might sign.\n> > \n> > I see. What about, for example, having an official \"overlay\" on git for\n> > signing and verification of a repository? (e.g., similar to what\n> > monotone does). I see that other VCS's have a plugin mechanism, and they\n> > host official plugins.\n> \n> In general, if something is more general than git, I'd like to see a\n> general tool address it, and then add support to git to make use of the\n> tool.\n> \n> For group key management, I specifically was wondering if you could do\n> something like:\n> \n>   - start with some seed GPG keys for the project\n> \n>   - existing keys can sign or revoke certificates to add or remove other\n>     keys to/from the project; you could even require a threshold of\n>     signatures, etc.\n> \n>   - those keys could be used for signing git pushes, but also for other\n>     things, like signing tarballs, used as encryption keys for sending\n>     for-developers-eyes-only security reports, etc\n> \n>     You'd want a tool that asks not just \"is this signed\" but \"is this\n>     signed _by a key that is valid for this project_\".\n\nYep and also \"is this signed thing the thing I should be looking at?\"\n\n> \n> And then git support would just consist of feeding signatures to\n> \"gpg-group --project=...\" instead of \"gpg\". Management of the group\n> would be out-of-band from git, which is in some ways good and in some\n> ways bad.\n\nyep, what I like from in-band solution is that it is easy to piggyback\non existing git mechanisms (e.g., git ref backend). \n\n> \n> I think a naive implementation would be pretty easy, but I've glossed\n> over all of the chaining properties we've discussed. So whatever\n> mechanism you use to receive updates to the key-group would have all the\n> same problems (e.g., withholding revocations of compromised keys). It's\n> still orders of magnitude ahead of what's currently happening\n> day-to-day. :)\n\nyeah, a naive implementation may be easy, but there are little details\nto consider when trying to get something more robust. I don't see why\nsketching something would be a bad idea though.\n\nThanks all for all your feedback! I'll keep this mail archived around to\nrevisit it in the future; all of this seems really interesting/helpful. \n\n-Santiago.\n"},{"id":"292974","messageId":"alpine.DEB.2.02.1608031259500.16896@nftneq.ynat.uz","threadId":"42997","inReplyTo":"20160803174459.knlmyx6txmggixzi@LykOS.localdomain","subject":"Re: [OT] USENIX paper on Git","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2016-08-03T20:03:00Z","receivedAt":"2016-08-03T20:03:10Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Wed, 3 Aug 2016, Santiago Torres wrote:\n\n>> So if you want to treat Git as a cryptographic end-to-end distribution\n>> mechanism, then no, it fails horribly at that. But the state of the art\n>> in source code distribution, no matter which system you use, is much\n>> less advanced than that. People download tarballs, even ones with GPG\n>> signatures, all the time without verifying their contents. Most packages\n>> distribute a sha1sum or similar (sometimes even gpg-signed), but quite\n>> often the source of authority is questionable.\n>\n> Yes, this happens an awful lot of times. We did some work with python's\n> pypi last year, and we found out that less than 1% of people actually\n> downloaded the gpg signature for the package they are retrieving[1].\n>\n>>\n>> For example, consider somebody downloading a new package for the first\n>> time. They don't know the author in any out-of-band way, so any\n>> signatures are likely meaningless. They _might_ be depending on the\n>> source domain for some security (and using some hierarchical PKI like\n>> TLS+x.509 to be sure they're talking to that domain), but in your threat\n>> model, even well-known hosts like FSF could be compromised internally.\n>>\n>> So yes, I think the current state of affairs (especially open-source) is\n>> that people download and run possibly-compromised code all the time. But\n>> I'm not sure that lack of tool support is really the limiting factor. Or\n>> that it has turned out to be all that big a problem in practice.\n>\n> I couldn't agree more. I feel that OSS is slowly moving towards a more\n> cryptographically robust, trust-based way of doing things, which I find\n> pleasing.\n\nIt's too easy to look at this from purely a technical, cryptographic point of \nview and miss a very important point.\n\nIt may be very easy to see that this was signed by \"cool-internet-name\" but how \ncan I tell if this is really Joe Blow the developer? and if it is, I still have \nno way of knowing if he's working for the NSA or not.\n\nThe lack of meaningful termination of the signatures to the real world is why so \nfew people bother to check package signatures, etc.\n\nDavid Lang\n"}]}