{"thread":{"id":"41014","subject":"[RFC] Malicously tampering git metadata?","startedAt":"2015-12-16T03:26:39Z","lastAt":"2016-01-22T18:51:09Z","messageCount":14,"participants":["Santiago Torres","Stefan Beller","Jeff King","Theodore Ts'o"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"274547","messageId":"20151216032639.GA1901@LykOS","threadId":"41014","inReplyTo":null,"subject":"[RFC] Malicously tampering git metadata?","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2015-12-16T03:26:39Z","receivedAt":"2015-12-16T03:26:39Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"Hello everyone,\n\nI'm Santiago, a PhD student at NYU doing research about secure software\ndevelopment pipelines. We've been studying different aspects of Git\nlately, (as it is an integral part of many projects) and we believe\nwe've found a vulnerabilty in the way Git structures/signs metadata. \n\nAn attacker capable of performing as a Man in the Middle between a\nGitHub server and a developer is able to trick such developer into\nmerging vulnerable commit objects, or omit security patches --- even if\nall users sign all commit objects. Given that Git metadata is unsigned,\nit can be modified to provide incorrect views of a repository to\ndownstream developers.\n\nAn example of a malicious commit merge follows:\n\n1) The attacker controlling or acting as the upstream server identifies\ntwo branches: one in which the unsuspecting developer is working on, and\nanother in which a vulnerable piece of code is located.\n\n2) Branch pointers are modified: the packed-refs file (or ref/heads/*)\nis edited so that the master branch points to the vulnerable commit\nobject. Having performed the change, no additional configuration must be\nmade by the attacker, who now waits for an unsuspecting developer to\npull.\n\n3) Once a developer pulls, he or she will be prompted to merge his code\nwith the new change-set (the vulnerable commit). This operation will\nonly resemble developer negligence. If no conflicts arise, the attack\nwill pass unsuspected.\n\n4) The developer pushes to upstream. All the traffic can be re-routed\nback to the original repository. The target branch now contains a\nvulnerable piece of code.\n\nWe have identified additional attack scenarios for modifying the\nmetadata that result in a incorrect state of the target repository, and\nwe are ready to disclose information about other variants of this attack\nas well.\n\nWe also designed a backwards-compatible defense mechanism to prevent\nattacks based on Git metadata tampering. Also we implemented a proof of\nconcept of the scheme, and performed timing, stress and concurrency\ntests; our results show that the overhead should be minimal, even in\nlarge software repositories such as the Linux Kernel.\n\nWe already approached people from CERT and GitHub regarding this attack\nscenario, and we'd also like to hear your comments regarding this.\n\nThanks!\n-Santiago.\n\nP.S. We also elaborate more about this attack vector in this document: \nhttps://drive.google.com/a/nyu.edu/file/d/0B2KBm0fULlS1RDR5UHVESjVua3M/view?usp=sharing\n"},{"id":"274553","messageId":"CAGZ79kZNim0wp2fYKv2+6t+CaqqzjTThsm_KoAv1D_8OsD0qTQ@mail.gmail.com","threadId":"41014","inReplyTo":"20151216032639.GA1901@LykOS","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2015-12-16T07:20:04Z","receivedAt":"2015-12-16T07:20:04Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Dec 15, 2015 at 7:26 PM, Santiago Torres <santiago@nyu.edu> wrote:\n> Hello everyone,\n>\n> I'm Santiago, a PhD student at NYU doing research about secure software\n> development pipelines. We've been studying different aspects of Git\n> lately, (as it is an integral part of many projects) and we believe\n> we've found a vulnerabilty in the way Git structures/signs metadata.\n>\n> An attacker capable of performing as a Man in the Middle between a\n> GitHub server and a developer is able to trick such developer into\n> merging vulnerable commit objects, or omit security patches --- even if\n> all users sign all commit objects. Given that Git metadata is unsigned,\n> it can be modified to provide incorrect views of a repository to\n> downstream developers.\n>\n> An example of a malicious commit merge follows:\n>\n> 1) The attacker controlling or acting as the upstream server identifies\n> two branches: one in which the unsuspecting developer is working on, and\n> another in which a vulnerable piece of code is located.\n>\n> 2) Branch pointers are modified: the packed-refs file (or ref/heads/*)\n> is edited so that the master branch points to the vulnerable commit\n> object. Having performed the change, no additional configuration must be\n> made by the attacker, who now waits for an unsuspecting developer to\n> pull.\n>\n> 3) Once a developer pulls, he or she will be prompted to merge his code\n> with the new change-set (the vulnerable commit). This operation will\n> only resemble developer negligence. If no conflicts arise, the attack\n> will pass unsuspected.\n>\n> 4) The developer pushes to upstream. All the traffic can be re-routed\n> back to the original repository. The target branch now contains a\n> vulnerable piece of code.\n>\n> We have identified additional attack scenarios for modifying the\n> metadata that result in a incorrect state of the target repository, and\n> we are ready to disclose information about other variants of this attack\n> as well.\n>\n> We also designed a backwards-compatible defense mechanism to prevent\n> attacks based on Git metadata tampering. Also we implemented a proof of\n> concept of the scheme, and performed timing, stress and concurrency\n> tests; our results show that the overhead should be minimal, even in\n> large software repositories such as the Linux Kernel.\n>\n> We already approached people from CERT and GitHub regarding this attack\n> scenario, and we'd also like to hear your comments regarding this.\n\nThis is what push certificates ought to solve.\nThe server records all pushes and its signed certificates of pushes\nand by the difference of the\nrefs (either in packed refs or as a loose ref) to the push certificate\nthis tampering of\nthe server can be detected.\n\nThe push certs can however not be obtained via Git itself (they are\njust stored on the\nserver for now for later inspection or similar), because to be really\nsure the client would\nneed to learn about these push certificates out of band.\n\nThe model there would be:\n* A vulnerable piece of software exists.\n* It get's fixed (and the fix is pushed with a signed push)\n* the MITM server doesn't show the fix (show the code from before fix) nor\n  the push certificate thereof\n* client still pulls vulnerable code\n\nThis model shows the distribution of push certs via the server itself may not be\noptimal.\n\nThanks for researching on Git,\nStefan\n\n>\n> Thanks!\n> -Santiago.\n>\n> P.S. We also elaborate more about this attack vector in this document:\n> https://drive.google.com/a/nyu.edu/file/d/0B2KBm0fULlS1RDR5UHVESjVua3M/view?usp=sharing\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"274705","messageId":"20151218010635.GA16508@LykOS","threadId":"41014","inReplyTo":"CAGZ79kZNim0wp2fYKv2+6t+CaqqzjTThsm_KoAv1D_8OsD0qTQ@mail.gmail.com","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2015-12-18T01:06:36Z","receivedAt":"2015-12-18T01:06:36Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"Hi Stefan, thanks for the insight. \n\n> This is what push certificates ought to solve.\n> The server records all pushes and its signed certificates of pushes\n> and by the difference of the\n> refs (either in packed refs or as a loose ref) to the push certificate\n> this tampering of\n> the server can be detected.\n\nIs there any specification about push certificates? I would like to read\nabout them, but I don't seem to find documentation anywhere. Are they a\npart of git's specification?\n\n> \n> The push certs can however not be obtained via Git itself (they are\n> just stored on the\n> server for now for later inspection or similar), because to be really\n> sure the client would\n> need to learn about these push certificates out of band.\n\nI was thinking that an in-band solution could be integrated as long as\nwe assume a compromise would result in an complete (unreconcilable)\nfork attack; fork attacks aren't subtle and could be detected easily.\n\n> \n> The model there would be:\n> * A vulnerable piece of software exists.\n> * It get's fixed (and the fix is pushed with a signed push)\n> * the MITM server doesn't show the fix (show the code from before fix) nor\n>   the push certificate thereof\n> * client still pulls vulnerable code\n\nYes, this is a possible attack vector. However, a server could also\npresent a branch pointer as different (e.g., point an experimental\nbranch to an unsigned v1.1 tag). This has other implications, as the\ncode is pushed/pulled from upstream, it just goes somewhere different.\n\n> \n> This model shows the distribution of push certs via the server itself may not be\n> optimal.\n\nYes, it might not be optimal, but it could provide protection against\nthe attack I just described, for more complex attacks might not be so\nsubtle. Adding to this, developers likely coordinate their efforts\nthrough other means (sic), so the lack of a push certificate (withheld\nby a server) could raise some yellow flags.\n\nWe've made a proof of concept of such tool (in-bandh push certificates),\nand would like to share the basic design of it here. However, it follows\nour threat model: a compromised server that can't introduce malicious\ncode (thanks to commit signing), but can modify branch pointers and\nother unsigned metadata to alter the repository's state.\n\n> \n> Thanks for researching on Git,\n\nThanks for working in such a great tool :)\n-Santiago\n\n> Stefan\n"},{"id":"274712","messageId":"20151218035501.GA9273@sigill.intra.peff.net","threadId":"41014","inReplyTo":"20151218010635.GA16508@LykOS","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-12-18T03:55:01Z","receivedAt":"2015-12-18T03:55:01Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 17, 2015 at 08:06:36PM -0500, Santiago Torres wrote:\n\n> > This is what push certificates ought to solve.\n> > The server records all pushes and its signed certificates of pushes\n> > and by the difference of the\n> > refs (either in packed refs or as a loose ref) to the push certificate\n> > this tampering of\n> > the server can be detected.\n> \n> Is there any specification about push certificates? I would like to read\n> about them, but I don't seem to find documentation anywhere. Are they a\n> part of git's specification?\n\nTry pack-protocol.txt and protocol-capabilities.txt in the\nDocumentation/technical directory of the git.git repo.\n\nE.g.:\n\n  https://github.com/git/git/blob/bdfc6b364a51b19efbacbd46fcef5be41a5db50e/Documentation/technical/pack-protocol.txt#L489\n\n-Peff\n"},{"id":"274713","messageId":"20151218040210.GA9585@sigill.intra.peff.net","threadId":"41014","inReplyTo":"20151216032639.GA1901@LykOS","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-12-18T04:02:10Z","receivedAt":"2015-12-18T04:02:10Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 15, 2015 at 10:26:39PM -0500, Santiago Torres wrote:\n\n> An example of a malicious commit merge follows:\n> \n> 1) The attacker controlling or acting as the upstream server identifies\n> two branches: one in which the unsuspecting developer is working on, and\n> another in which a vulnerable piece of code is located.\n\nOne thing to make clear here: the side branch with the vulnerable code\nmust be a _new_ vulnerability that was not already part of the \"main\"\nbranch the developer is working on. That is, I do not immediately see a\nway to resurrect an old vulnerability, because a merge of the old,\nbroken commit would not result in reintroducing it.\n\nThis is more about \"there was experimental junk on branch X, and you\ntricked some developer into pulling X onto Y, and now Y unexpectedly has\nthe junk on it\". And I agree with Stefan that push-certs are the\nintended defense against this.\n\nOf course, in the real world things are much easier. Most projects do\nnot sign commits at all, let alone use push certs. If developers are\npulling from a compromised server, then you can simply make up whatever\nbroken commits you want, and there's no way to tell the difference\nbetween them and the real commits.\n\n-Peff\n"},{"id":"274731","messageId":"20151218231032.GA16904@thunk.org","threadId":"41014","inReplyTo":"20151216032639.GA1901@LykOS","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2015-12-18T23:10:32Z","receivedAt":"2015-12-18T23:10:32Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Tue, Dec 15, 2015 at 10:26:39PM -0500, Santiago Torres wrote:\n> 4) The developer pushes to upstream. All the traffic can be re-routed\n> back to the original repository. The target branch now contains a\n> vulnerable piece of code.\n\nI assume you are assuming here that the \"push to upstream\" doesn't\ninvolve some kind of human verification?  If someone tried pushing\nsomething like this to Linus, he would be checking the git diff stats\nand git commit structure for things that might look like \"developer\nnegligence\".  He's been known to complain to subsystem developers in\nhis own inimitable when the git commit structure looks suspicious, so\nI'm pretty sure he would notice this.\n\nBut normally that developnment process we don't talk about \"pushing to\nupstream\" as much as \"requesting a pull\".  So it would be useful when\nyou describe the attack to explicit describe the development workflow\nthat is vulnerable to your attack.\n\nFor example, in my use case, I'm authorative over changes in fs/ext4.\nSo when I pull from Linus's repo, I examine (using \"gitk fs/ext4\") all\ncommits coming from upstream that modify fs/ext4.  So if someone tries\nintroducing a change in fs/ext4 coming from \"upstream\", I will notice.\nThen when I request a pull request from Linus, the git pull request\ndescribes what commits are new in my tree that are not in his, and\nshows the diffstats from upstream.  When Linus verifies my pull, there\nare multiple opportunities where he will notice funny business:\n\na) He would notice that my origin commit is something that is not in\nhis upstream tree.\n\nb) His diffstat is different from my diffstat (since thanks to the\nman-in-the middle, the conception of what commits are new in the git\npull request will be different from his).\n\nc) His diffstat will show that files outside of fs/ext4 have been\nmodified, which is a red flag that will merit more close examination.\n(And if the attacker had tried to introduce a change in fs/ext4, I\nwould have noticed when I pulled from the man-in-the-middle git\nrepo.)\n\nNow, if there is zero checking when the user pushes to upstream, then\nyes, there are all sorts of potential problems.  But that's one of the\nreasons why it's generally considered a good thing for Linux\ndevelopers to use as the origin commit for their work official\nreleases (which can be demarked using GPG-signed git tags).\n\nSo for example, the changes for ext4 that were sent to Linus for v4.4\nwas based off of v4.3-rc2:\n\ngit tag  --verify v4.3-rc2\nobject 1f93e4a96c9109378204c147b3eec0d0e8100fde\ntype commit\ntag v4.3-rc2\ntagger Linus Torvalds <torvalds@linux-foundation.org> 1442784761 -0700\n\nLinux 4.3-rc2\ngpg: Signature made Sun 20 Sep 2015 05:32:41 PM EDT using RSA key ID 00411886\ngpg: Good signature from \"Linus Torvalds <torvalds@linux-foundation.org>\" [full]\n\n\nAnd the changes which I sent to Linus were also signed by a tag, and\nbetter yet, someone can indepedently verify this after the fact:\n\n% git show --oneline --show-signature f41683a204ea61568f0fd0804d47c19561f2ee39\nf41683a merged tag 'ext4_for_linus_stable'\ngpg: Signature made Sun 06 Dec 2015 10:35:27 PM EST using RSA key ID 950D81A3\ngpg: Good signature from \"Theodore Ts'o <tytso@mit.edu>\" [ultimate]\ngpg:                 aka \"Theodore Ts'o <tytso@debian.org>\" [ultimate]\ngpg:                 aka \"Theodore Ts'o <tytso@google.com>\" [ultimate]\n\nThey can also verify that the chain of commits that I sent to Linus\nwere rooted in Linus's signed v4.3-rc2 tag, so this kind of\nafter-the-fact auditing means that if there *were* funny business, it\ncould be caught even if Linus slipped up in his checking.\n\n\nNow, the crazy behavior where github users randomly and promiscuously\ndo pushes and pull without doing any kind of verification may very\nwell be dangerous.  But so is someone who saves a 80 patch series from\ntheir inbox, and without reading or verifying all of the patches\napplies them blindly to their tree using \"git am\" --- or if they were\nusing cvs or svn, bulk applied the patches without doing any\nverification....\n\nCheers,\n\n\t\t\t\t\t- Ted\n"},{"id":"274760","messageId":"20151219173018.GA1178@LykOS","threadId":"41014","inReplyTo":"20151218231032.GA16904@thunk.org","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2015-12-19T17:30:18Z","receivedAt":"2015-12-19T17:30:18Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"> I assume you are assuming here that the \"push to upstream\" doesn't\n> involve some kind of human verification?  If someone tried pushing\n> something like this to Linus, he would be checking the git diff stats\n> and git commit structure for things that might look like \"developer\n> negligence\".  He's been known to complain to subsystem developers in\n> his own inimitable when the git commit structure looks suspicious, so\n> I'm pretty sure he would notice this.\n> \n> For example, in my use case, I'm authorative over changes in fs/ext4.\n> So when I pull from Linus's repo, I examine (using \"gitk fs/ext4\") all\n> commits coming from upstream that modify fs/ext4.  So if someone tries\n> introducing a change in fs/ext4 coming from \"upstream\", I will notice.\n> Then when I request a pull request from Linus, the git pull request\n> describes what commits are new in my tree that are not in his, and\n> shows the diffstats from upstream.  When Linus verifies my pull, there\n> are multiple opportunities where he will notice funny business:\n> \n> a) He would notice that my origin commit is something that is not in\n> his upstream tree.\n> \n> b) His diffstat is different from my diffstat (since thanks to the\n> man-in-the middle, the conception of what commits are new in the git\n> pull request will be different from his).\n> \n> c) His diffstat will show that files outside of fs/ext4 have been\n> modified, which is a red flag that will merit more close examination.\n> (And if the attacker had tried to introduce a change in fs/ext4, I\n> would have noticed when I pulled from the man-in-the-middle git\n> repo.)\n\nYes, we've been considering that these kind of attacks wouldn't be too\neffective against, let's say, dictator-lieutenant workflows. \n\n> \n> Now, the crazy behavior where github users randomly and promiscuously\n> do pushes and pull without doing any kind of verification may very\n> well be dangerous. \n\nYes, we were mostly familiar with this workflow before starting this\nresearch. I can see how the \"github generation\" is open to many attacks\nof this nature. Would git be interested in integrating a defense that\ncovers users of this nature (which seems to be a growing userbase)?\n\n> But so is someone who saves a 80 patch series from\n> their inbox, and without reading or verifying all of the patches\n> applies them blindly to their tree using \"git am\" --- or if they were\n> using cvs or svn, bulk applied the patches without doing any\n> verification....\n\nJust out of curiosity, are there known cases of projects in which this\nhas happened (I've noticed that both Git and Linux are quite stringent\nin their review/merge process so this wouldn't be the case).\n\n> \n> Cheers,\n\nThanks for the insight!\n-Santiago.\n"},{"id":"274764","messageId":"20151220012835.GA3013@thunk.org","threadId":"41014","inReplyTo":"20151219173018.GA1178@LykOS","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2015-12-20T01:28:35Z","receivedAt":"2015-12-20T01:28:35Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Sat, Dec 19, 2015 at 12:30:18PM -0500, Santiago Torres wrote:\n> > Now, the crazy behavior where github users randomly and promiscuously\n> > do pushes and pull without doing any kind of verification may very\n> > well be dangerous. \n> \n> Yes, we were mostly familiar with this workflow before starting this\n> research. I can see how the \"github generation\" is open to many attacks\n> of this nature. Would git be interested in integrating a defense that\n> covers users of this nature (which seems to be a growing userbase)?\n\nOne of the interesting challenges is that git is a pretty low-level\ntool, and so people have built all sorts of different workflows on top\nof it.\n\nFor example, at $WORK, we use gerrit, which is a code review tool, so\nall git commits that are to be merged into the \"upstream\" repository\n0gets pushed to a gerrit server, where it goes through a code review\nprocess where a second engineer can review the code, request changes,\nmake comments, or ask questions, and where the git commits can go\nthrough multiple rounds of review / revision before they are finally\naccepted (at least one reviewer must give a +2 review, and there must\nbe no -2 reviews; and there can be automated tools that do build or\nregression tests that can give automated -1 or -2 reviews) --- and\nwhere all of the information collected during the code review process\nis saved as part of the audit trail for a Sarbanes-Oxley (SOX)\ncompliance regime.\n\nOther people use github-style workflows, and others use signed tags\nwith e-mail code reviews, etc.  And I'm sure there must be many others.\n\nSo the challenge is that in order to accomodate many possible\nworkflows, some of which use third-party tools, changes to make git\nmore secure for one workflow must not get in the way of these other\nworkflows --- which means that enforcement of new controls for the\n\"github generation\" probably will have to be optional.  But then\npeople belonging to the \"github generation\" can also easily turn off\nthese features.  And as the NSA learned the hard way in Vietnam, if\nthe tools cause any inconenience, or has the perception of\nconstraining legitmate users, security features can have a way of\ngetting turned off.[1]\n\n[1] A History of US Communications Security, The David G. Boak\nlectures, Volume II, \"Nestor in Vietname\".  pg 43-44.  (A declassified\nversion can be found at:\nhttp://www.governmentattic.org/18docs/Hist_US_COMSEC_Boak_NSA_1973u.pdf)\n\n> > But so is someone who saves a 80 patch series from\n> > their inbox, and without reading or verifying all of the patches\n> > applies them blindly to their tree using \"git am\" --- or if they were\n> > using cvs or svn, bulk applied the patches without doing any\n> > verification....\n> \n> Just out of curiosity, are there known cases of projects in which this\n> has happened (I've noticed that both Git and Linux are quite stringent\n> in their review/merge process so this wouldn't be the case).\n\nI can't point at specific instances, but given that in the \"github\ngeneration\", people are fine with blindly pulling someone else's\nDocker images and running it on their production servers or\nworkstations, and where software installation gets done with \"wget\nhttp://example.org | bash\" or the equivalent, that it's probably more\noften than we might be comfortable.\n\nI also suspect that a bad guy would probably find inserting a\nman-in-the-middle server into one of these installation flows is\nprobably a much more practical attack in terms of real world\nconsiderations.  :-)\n\nCheers,\n\n\t\t\t\t\t\t- Ted\n"},{"id":"275823","messageId":"20160112182137.GE27334@LykOS","threadId":"41014","inReplyTo":"20151220012835.GA3013@thunk.org","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-01-12T18:21:38Z","receivedAt":"2016-01-12T18:21:38Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"Hello Everyone,\n\nThanks for the feedback regarding our attack scenario; it certainly shed some\nlight in what is the current state of git's metadata protection. We were\npleasantly surprised that attacks of this nature were considered, yet we think\nwe can improve on the current mechanisms.\n\nWe have been designing an extension that addresses this attack scenario (and\nother similar attacks). Although originally it is not based on push\ncertificates, we feel that it works similar to them. The principal advantages\nover push certificates are:\n\n1) It doesn't require (although it could support it) a side channel. We store similar\n    information about branch status (push status) on the repository itself. \n\n2) It is backwards compatible, as it doesn't modify the existing metadata\n    format.\n\n3) Following Ted's email, it could be easily integrated in any git workflow.\n    Although some workflows might be benefitted more than others, it doesn't\n    get in the way of any existing workflow that we know of.\n\n4) It covers a broader attack suurface (e.g., our malicious-merge scenario).\n\nTo keep things simple (we can elaborate in further emails), our solution\nbasically works by keeping track of pushes by developers in an append only\nfile, so that, everytime a branch is pushed, the deloper signs his version of\nthe log and his \"push entry\" (similar to a push certificate). Right now, we\npush this log to a separate branch called BSL (for Branch State Log), but\nideallly this could be part of the git metadata. \n\nUpon pulling/fetching, this push certificate chain (BSL) is also fetched\nand used to verify whether all branches are pointing to a sensible\nlocation (i.e., the location reported by the last user who\npushed/merged).This ensures that a malicious server can't change the\nlocation to which branches point to.\n\nFurthermore, upon fetching, users with write access also push an entry\nindicating that they fetched the BSL. This avoids cases in which a malicious\nattacker keeps older versions of the BSL and withhold changes to other users.\nThis mainly addresses an attack we refer to as \"effort duplication attack\".\n\nFor key distribution, we consider it could be done via the BSL itself, where a\nuser can sign the public key of a newly-added developer to the system as an\nentry. To produce a local trust keychain, our tool simply requires an initial\nroot of trust (say, the main developer's trusted key), and it builds up the\nkeychain by traversing the BSL. Revocation could be done similarly too.\n\nWe already have a proof of concept of this scheme, and we have performed timing\nand storage measurements, as well as stress tests. We believe integration\nshould not be hard, and that it is not particularly invasive with any type of\nworkflow. We wanted to note that our current implementation was designed in\nwith the aim of not changing the Git source code (although we would like to do\nthat in the future), which makes it a little more network-storage\nexpensive than a native solution.\n\nHere is the overhead of our proof of concept:\n\n    1) Storage overhead only increases for about 1kb per push entry. Right now,\n    since we are storing each entry as a separate file in a separate branch, the\n    storage overhead is non-optimal. If this file were to be stored as git\n    metadata, it would only be around 345 bytes, depending on the signature\n    scheme). \n        \n    2) Network overhead is linear, and it adds less than 23kb per\n    pull/fetch/push, mainly because we are opening three different tcp sessions\n    with git. If we were able to integrate this in the git protocol,\n    network overhead should be really close to the storage overhead x\n    unseen push entries.\n\n    3) For \"fetch entries\", storage and network overhead is minimal, as users\n    only need to upload a 4-byte random number per fetch. We consider this cost to\n    be negligible, (although we'd like to hear the insight from people\n    knowledgeable to git).\n\n    4) Timing shouldn't be a concern. Verifying an entry (using 1024 dsa with libgcrypt)\n    is done in 30 microseconds per entry with an average laptop. A BSL with 46633\n    entries --- imagine git.git had a push entry for each commit --- is less than\n    1.28 seconds (assuming the whole BSL is verified upon cloning, for example).\n\nIn general, this solution seems to be achieveable, nonintrusive and, overall, a\nnice thing to have. We were considering working towards a patch if it sounds\nreasonable to the community.\n\nThanks!\n-Santiago.\n\nOn Sat, Dec 19, 2015 at 08:28:35PM -0500, Theodore Ts'o wrote:\n> On Sat, Dec 19, 2015 at 12:30:18PM -0500, Santiago Torres wrote:\n> > > Now, the crazy behavior where github users randomly and promiscuously\n> > > do pushes and pull without doing any kind of verification may very\n> > > well be dangerous. \n> > \n> > Yes, we were mostly familiar with this workflow before starting this\n> > research. I can see how the \"github generation\" is open to many attacks\n> > of this nature. Would git be interested in integrating a defense that\n> > covers users of this nature (which seems to be a growing userbase)?\n> \n> One of the interesting challenges is that git is a pretty low-level\n> tool, and so people have built all sorts of different workflows on top\n> of it.\n> \n> For example, at $WORK, we use gerrit, which is a code review tool, so\n> all git commits that are to be merged into the \"upstream\" repository\n> 0gets pushed to a gerrit server, where it goes through a code review\n> process where a second engineer can review the code, request changes,\n> make comments, or ask questions, and where the git commits can go\n> through multiple rounds of review / revision before they are finally\n> accepted (at least one reviewer must give a +2 review, and there must\n> be no -2 reviews; and there can be automated tools that do build or\n> regression tests that can give automated -1 or -2 reviews) --- and\n> where all of the information collected during the code review process\n> is saved as part of the audit trail for a Sarbanes-Oxley (SOX)\n> compliance regime.\n> \n> Other people use github-style workflows, and others use signed tags\n> with e-mail code reviews, etc.  And I'm sure there must be many others.\n> \n> So the challenge is that in order to accomodate many possible\n> workflows, some of which use third-party tools, changes to make git\n> more secure for one workflow must not get in the way of these other\n> workflows --- which means that enforcement of new controls for the\n> \"github generation\" probably will have to be optional.  But then\n> people belonging to the \"github generation\" can also easily turn off\n> these features.  And as the NSA learned the hard way in Vietnam, if\n> the tools cause any inconenience, or has the perception of\n> constraining legitmate users, security features can have a way of\n> getting turned off.[1]\n> \n> [1] A History of US Communications Security, The David G. Boak\n> lectures, Volume II, \"Nestor in Vietname\".  pg 43-44.  (A declassified\n> version can be found at:\n> http://www.governmentattic.org/18docs/Hist_US_COMSEC_Boak_NSA_1973u.pdf)\n> \n> > > But so is someone who saves a 80 patch series from\n> > > their inbox, and without reading or verifying all of the patches\n> > > applies them blindly to their tree using \"git am\" --- or if they were\n> > > using cvs or svn, bulk applied the patches without doing any\n> > > verification....\n> > \n> > Just out of curiosity, are there known cases of projects in which this\n> > has happened (I've noticed that both Git and Linux are quite stringent\n> > in their review/merge process so this wouldn't be the case).\n> \n> I can't point at specific instances, but given that in the \"github\n> generation\", people are fine with blindly pulling someone else's\n> Docker images and running it on their production servers or\n> workstations, and where software installation gets done with \"wget\n> http://example.org | bash\" or the equivalent, that it's probably more\n> often than we might be comfortable.\n> \n> I also suspect that a bad guy would probably find inserting a\n> man-in-the-middle server into one of these installation flows is\n> probably a much more practical attack in terms of real world\n> considerations.  :-)\n> \n> Cheers,\n> \n> \t\t\t\t\t\t- Ted\n"},{"id":"275825","messageId":"CAGZ79kadpy9N0qEpxK-USVxCmNfYJm1g5xr8ZiFxf7sOVKZnEw@mail.gmail.com","threadId":"41014","inReplyTo":"20160112182137.GE27334@LykOS","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-01-12T18:39:07Z","receivedAt":"2016-01-12T18:39:07Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 12, 2016 at 10:21 AM, Santiago Torres <santiago@nyu.edu> wrote:\n> Hello Everyone,\n>\n> Thanks for the feedback regarding our attack scenario; it certainly shed some\n> light in what is the current state of git's metadata protection. We were\n> pleasantly surprised that attacks of this nature were considered, yet we think\n> we can improve on the current mechanisms.\n>\n> We have been designing an extension that addresses this attack scenario (and\n> other similar attacks). Although originally it is not based on push\n> certificates, we feel that it works similar to them. The principal advantages\n> over push certificates are:\n>\n> 1) It doesn't require (although it could support it) a side channel. We store similar\n>     information about branch status (push status) on the repository itself.\n>\n> 2) It is backwards compatible, as it doesn't modify the existing metadata\n>     format.\n>\n> 3) Following Ted's email, it could be easily integrated in any git workflow.\n>     Although some workflows might be benefitted more than others, it doesn't\n>     get in the way of any existing workflow that we know of.\n>\n> 4) It covers a broader attack suurface (e.g., our malicious-merge scenario).\n>\n> To keep things simple (we can elaborate in further emails), our solution\n> basically works by keeping track of pushes by developers in an append only\n> file, so that, everytime a branch is pushed, the deloper signs his version of\n> the log and his \"push entry\" (similar to a push certificate). Right now, we\n> push this log to a separate branch called BSL (for Branch State Log), but\n> ideallly this could be part of the git metadata.\n\nRecently in another context (an alternative refs backend) there was a proposal\nby Shawn to keep the .git directory versioned by git itself, i.e.\nhaving only loose refs in\n.git/refs and then there is a repository tracking .git/refs as a\ndirectory structure.\n\nUsing that idea of a refs back end, combined with signed tags in the\nrefs repository\nwould give you signed version of the log of possible push entries.\n\n>\n> Upon pulling/fetching, this push certificate chain (BSL) is also fetched\n> and used to verify whether all branches are pointing to a sensible\n> location (i.e., the location reported by the last user who\n> pushed/merged).This ensures that a malicious server can't change the\n> location to which branches point to.\n\nThis is what push certs ought to solve already?\nAFAIU the main issue with untrustworthy servers is holding back the latest push.\nAs Ted said, usually there is problem in the code and then the fix is pushed,\nbut the malicious server would not advertise the update, but deliver the old\nunfixed version.\n\nThis attack cannot be mitigated by having either a side channel (email\nannouncements)\nor time outs (state is only good if push cert is newer than <amount of\ntime>, but this may\nrequire empty pushes)\n\n\n>\n> Furthermore, upon fetching, users with write access also push an entry\n> indicating that they fetched the BSL. This avoids cases in which a malicious\n> attacker keeps older versions of the BSL and withhold changes to other users.\n\nThis would make it a \"be malicious to all or none\" thing? So the\nattacker cannot attack\na single target IIUC.\n\nI have a bad feeling about repository modifications upon fetching as\nsoftware distribution\nis a highly asymmetric workflow (number of fetches is many orders of\nmagnitudes larger than\npushes), which may not scale well? (How would you serialize parallel\nfetches into the BSL?)\n\nThanks,\nStefan\n"},{"id":"276057","messageId":"20160114171639.GB25541@LykOS","threadId":"41014","inReplyTo":"CAGZ79kadpy9N0qEpxK-USVxCmNfYJm1g5xr8ZiFxf7sOVKZnEw@mail.gmail.com","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-01-14T17:16:39Z","receivedAt":"2016-01-14T17:16:39Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"Hello Stefan, thanks for your feedback again. \n\n> This is what push certs ought to solve already?\n\nYes, they aim to solve the same issue. Unfortunately, push certificates\ndon't solve all posible scenarios of metadata manipulation (e.g., a\nmalicious server changing branch pointers to trick a user into merging\nunwanted changes).\n\n> AFAIU the main issue with untrustworthy servers is holding back the latest push.\n> As Ted said, usually there is problem in the code and then the fix is pushed,\n> but the malicious server would not advertise the update, but deliver the old\n> unfixed version.\n> \n> This attack cannot be mitigated by having either a side channel (email\n> announcements)\n> or time outs (state is only good if push cert is newer than <amount of\n> time>, but this may\n> require empty pushes)\n> \n\nI'm sorry, did you mean to say \"can\"? \n\nYes, this is a possible solution to this issue. The solution I'm\nproposing here wouldn't require a side channel though. As long as users'\nkeys are not compromised, the BSL could either expire (raise an alarm),\nor force an irreconcileable fork attack --- the user who submited the\nchanges won't be able to push until this change makes it through.\n\n> \n> >\n> > Furthermore, upon fetching, users with write access also push an entry\n> > indicating that they fetched the BSL. This avoids cases in which a malicious\n> > attacker keeps older versions of the BSL and withhold changes to other users.\n> \n> This would make it a \"be malicious to all or none\" thing? So the\n> attacker cannot attack\n> a single target IIUC.\n\nYes, this is true. The ides is that any attack to a single target is\neasy to identify.\n\n> \n> I have a bad feeling about repository modifications upon fetching as\n> software distribution is a highly asymmetric workflow (number of\n> fetches is many orders of magnitudes larger than pushes), which may\n> not scale well? \n\nYes, we were worried about the same ourselves. The upside is that, push\nand fetch entries in our scheme are also orders of magnitude smaller;\nfetch entries do not need to be signed and they can be as little as four\nbytes (they might be gc-able also). \n\n\n> (How would you serialize parallel fetches into the BSL?)\n\nYes, this would imply that locking needs to be performed on the server\nside. It is important to note that fetch entries are only relevant for\nusers to have write access (as only they are beneffited by them). For\nread-only fetches, like an automated build, this feature could be\ndisabled.\n\n> \n> Thanks,\n> Stefan\n\nThanks again!\n-Santiago.\n"},{"id":"276058","messageId":"CAGZ79ka51e+-24RyMgUGAOUkBYXxnWZb8Pg7vrgjGHvvWU770Q@mail.gmail.com","threadId":"41014","inReplyTo":"20160114171639.GB25541@LykOS","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-01-14T17:21:28Z","receivedAt":"2016-01-14T17:21:28Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Jan 14, 2016 at 9:16 AM, Santiago Torres <santiago@nyu.edu> wrote:\n> Hello Stefan, thanks for your feedback again.\n>\n>> This is what push certs ought to solve already?\n>\n> Yes, they aim to solve the same issue. Unfortunately, push certificates\n> don't solve all posible scenarios of metadata manipulation (e.g., a\n> malicious server changing branch pointers to trick a user into merging\n> unwanted changes).\n>\n>> AFAIU the main issue with untrustworthy servers is holding back the latest push.\n>> As Ted said, usually there is problem in the code and then the fix is pushed,\n>> but the malicious server would not advertise the update, but deliver the old\n>> unfixed version.\n>>\n>> This attack cannot be mitigated by having either a side channel (email\n>> announcements)\n>> or time outs (state is only good if push cert is newer than <amount of\n>> time>, but this may\n>> require empty pushes)\n>>\n>\n> I'm sorry, did you mean to say \"can\"?\n\nYes, formulating that sentence took a while and I did not proofread it.\n"},{"id":"276558","messageId":"20160122180007.GB28871@LykOS","threadId":"41014","inReplyTo":"CAGZ79ka51e+-24RyMgUGAOUkBYXxnWZb8Pg7vrgjGHvvWU770Q@mail.gmail.com","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Santiago Torres","fromEmail":"santiago@nyu.edu","sentAt":"2016-01-22T18:00:08Z","receivedAt":"2016-01-22T18:00:08Z","isPatch":false,"sender":{"key":"santiago@nyu.edu","avatar":"https://avatars.githubusercontent.com/u/3579933?v=4"},"body":"On Thu, Jan 14, 2016 at 09:21:28AM -0800, Stefan Beller wrote:\n> On Thu, Jan 14, 2016 at 9:16 AM, Santiago Torres <santiago@nyu.edu> wrote:\n> > Hello Stefan, thanks for your feedback again.\n> >\n> >> This is what push certs ought to solve already?\n> >\n> > Yes, they aim to solve the same issue. Unfortunately, push certificates\n> > don't solve all posible scenarios of metadata manipulation (e.g., a\n> > malicious server changing branch pointers to trick a user into merging\n> > unwanted changes).\n> >\n> >> AFAIU the main issue with untrustworthy servers is holding back the latest push.\n> >> As Ted said, usually there is problem in the code and then the fix is pushed,\n> >> but the malicious server would not advertise the update, but deliver the old\n> >> unfixed version.\n> >>\n> >> This attack cannot be mitigated by having either a side channel (email\n> >> announcements)\n> >> or time outs (state is only good if push cert is newer than <amount of\n> >> time>, but this may\n> >> require empty pushes)\n> >>\n> >\n> > I'm sorry, did you mean to say \"can\"?\n> \n> Yes, formulating that sentence took a while and I did not proofread it.\n\nSorry, Stefan. I didn't mean to come off as rude; I just wanted to make\nsure I understood correctly what you were proposing.\n\nDo you have any further insight? I think that, besides the supporting\nmultiple workflows, maybe synchronizing concurrent fetches might be an\nissue to our solution.\n\nThanks a lot!\n-Santiago.\n"},{"id":"276564","messageId":"CAGZ79kYOQ1sphdozTXGf+Q2n=kNpmGx1pvzLD5SHBqfhWDBA3Q@mail.gmail.com","threadId":"41014","inReplyTo":"20160122180007.GB28871@LykOS","subject":"Re: [RFC] Malicously tampering git metadata?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-01-22T18:51:09Z","receivedAt":"2016-01-22T18:51:09Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Jan 22, 2016 at 10:00 AM, Santiago Torres <santiago@nyu.edu> wrote:\n> On Thu, Jan 14, 2016 at 09:21:28AM -0800, Stefan Beller wrote:\n>> On Thu, Jan 14, 2016 at 9:16 AM, Santiago Torres <santiago@nyu.edu> wrote:\n>> > Hello Stefan, thanks for your feedback again.\n>> >\n>> >> This is what push certs ought to solve already?\n>> >\n>> > Yes, they aim to solve the same issue. Unfortunately, push certificates\n>> > don't solve all posible scenarios of metadata manipulation (e.g., a\n>> > malicious server changing branch pointers to trick a user into merging\n>> > unwanted changes).\n>> >\n>> >> AFAIU the main issue with untrustworthy servers is holding back the latest push.\n>> >> As Ted said, usually there is problem in the code and then the fix is pushed,\n>> >> but the malicious server would not advertise the update, but deliver the old\n>> >> unfixed version.\n>> >>\n>> >> This attack cannot be mitigated by having either a side channel (email\n>> >> announcements)\n>> >> or time outs (state is only good if push cert is newer than <amount of\n>> >> time>, but this may\n>> >> require empty pushes)\n>> >>\n>> >\n>> > I'm sorry, did you mean to say \"can\"?\n>>\n>> Yes, formulating that sentence took a while and I did not proofread it.\n>\n> Sorry, Stefan. I didn't mean to come off as rude; I just wanted to make\n> sure I understood correctly what you were proposing.\n\nNot at all, I just made a typo. :)\n\n>\n> Do you have any further insight? I think that, besides the supporting\n> multiple workflows, maybe synchronizing concurrent fetches might be an\n> issue to our solution.\n\nI did not think further about any issues there.\n\nThanks,\nStefan\n\n>\n> Thanks a lot!\n> -Santiago.\n"}]}