Question about scm security holes

I just saw this article about the "google hackers" exploiting weaknesses in scms, Perforce in particular:

http://www.wired.com/threatlevel/2010/03/source-code-hacks/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29

I guess google didn't take Linus's advice to dump Perforce :)

I can't tell from the article if Perforce is any worse than any other scm for security holes, in fact it seems to imply that others haven't been tested in the same way.

Just curious if anyone here has any thoughts about how the article may or may not have any relevance for git (git being the scm I use most, by far, which is the reason I'm interested).

Thanks

On Thu, Mar 4, 2010 at 3:09 PM, walt <w41ter@gmail.com> wrote:

The attack was uninteresting. The paper seems to go on and on about different ways that an attacker can steal source code by accessing a poorly-secured SCM server. This discussion is kind of moot for git, where every single developer workstation has a complete copy of the entire project history anyway.

An attack in which someone untraceably modified the repo to contain modified code would be a little more interesting. git makes this sort of thing pretty much impossible to do without it being *noticeable* at least. Traceable, not so much, because you can create a commit with whatever committer/author names you want and then push them in. Commits aren't GPG-signed, only tags are, so there are lots of ways to forge a commit from someone else and mess up the audit log. At least you can't edit old commits without people noticing, though.

Have fun,

Avery

On 5 March 2010 02:03, Avery Pennarun <apenwarr@gmail.com> wrote:

Which is why you simply record the username of whoever pushed them in.
 This is what gitorious.org does etc.

John

On Thu, Mar 4, 2010 at 10:00 PM, John Tapsell <johnflux@gmail.com> wrote:

Not bad, but it's still very hard to trace properly. Imagine I pull from a peer, then push my combined branch into the central repo. It'll say I'm pushing in patches from me *and* my friend. Did I forge them or are they real?

Avery

On 5 March 2010 03:19, Avery Pennarun <apenwarr@gmail.com> wrote:

While true, it's still traceable back to you. You did the push, so you are responsible for that code. It wouldn't be any different to just pushing a bad commit yourself.

On 03/04/2010 06:03 PM, Avery Pennarun wrote:

Thanks, that's the kind of reply I was hoping for. Do you think there should be a way to sign the commits themselves, at least as an option?

I certainly wouldn't bother, but OTOH nobody wants to steal my code :-/

Do you suppose the devs at Adobe carry the complete source repository home on their laptops every night? (Not if they use Perforce, of course, but they might if they adopted git as their scm.)

On Thu, Mar 4, 2010 at 10:20 PM, walt <w41ter@gmail.com> wrote:

The whole thing is a bit overblown. One of my friends once took me on a tour of Microsoft on a weekend. The place was mostly deserted, but tons of developers left their workstations unlocked overnight, and everyone had a private office. And with tens of thousands of developers on the campus, nobody would know if you're supposed to be there or not.

It would have been easy to walk off with the source code to Windows from one of those workstations. The fact is, nobody really *wants* the source code to Windows, except probably to look at it and be horrified.

What would you do if you stole the source code to Adobe's flash player? It's illegal (in the U.S. anyway) to reverse engineer it and it's illegal to steal it, so you're on the wrong side of the law no matter how you pretend you managed to figure out a way around their DRM or whatever.

People describe source code as a company's "crown jewels," but that's a bit of a joke. I can barely get our interns to figure out how to compile and understand our code. Expecting a thief to do it, with nothing but a raw repo and hundreds of gigabytes of crap, is pure paranoia.

Sneaking in patches? Yeah, watch out for that. But you should be reviewing patch changelogs anyway. At least git prevents people from *retroactively* changing stuff; they can only add patches on top, so it's easy to review after a break-in.

Have fun,

Avery

On Thu, 04 Mar 2010 21:03:08 +0000, Avery Pennarun wrote: ...

It's the point of a dev workstation to have access to the code, so McAfees whining about SCMs letting that happen is moot.

What would be helping here is a separation between internet-facing and local work into separate machines.

You can still log who pushed what into your blessed repo, and hold that person accountable.

Andreas

Hi,

On Thu, 4 Mar 2010, Avery Pennarun wrote:

I disagree that stealing the code in this particular case is uninteresting. You know that there are billions in the business how to manipulate Google's search results. If you can see how Google rates websites, you can prepare your xxx sites for it, and nobody would be able to know, let alone prove, that you "reverse-engineered" the system.

That is not true in all cases.

If you're talking about a workflow as git.git has it, you're right, there is a maintainer, and a refused push would ring all kinds of alarm bells there.

Except, of course, when the maintainer happens to work on different machines, and is likely to pull from her main repository quite often. Think "get something compiling on an obscure platform while developing something different on your main computer, then do a criss-cross merge at the end".

It gets even much, much worse in the common setup of companies: a central repository. (The two main reasons why a central repository is used are: tradition (we did it with Subversion, too), and bottleneck problems: a single maintainer reviewing all changes is often deemed too expensive and slow.)

So in the regular case, it is _very_ easy to sneak in a code-change unnoticedly.

The trick now is to craft the commit in such a manner that it will not be noticed retro-actively. This is a simple case of social engineering: you have to imitate the style of the committer/author you are impersonating. The commit message must look like the usual ones (typos, preferred words, grammar, length of paragraphs, comprehensibility, etc)

Likewise, the code has to be analyzed for style, and obviously for most likely targets of a backdoor (both in terms of "it is a perfect spot for a backdoor" and "it is not uncommon for the author to touch that part of the code").

Crafting the commit message and the backdoor needs some time, and it needs to be done _after_ succeeding with the break-in, as you can only then start analyzing style (and most likely workflow -- whether there is a single maintainer or whether everybody pushes to a single repository).

The most likely route, therefore is to have _two_ break-ins. One for reconaissance, the second for the actual change.

Conclusion: there are no technical reasons why Git should be better than 
Perforce when it comes to a break-in.

Short version: it's a social problem, so it needs a social solution.

Ciao, Dscho

P.S.: Sorry for the overly long mail. I did not have time to make it short.

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

[...]

About "bottleneck problem". Frederick Brooks wrote in his seminal book "The Mythical Man-Month" that recommended way of organizing teams is *with a maintainer*. But this is less known that his most famous statement: "Adding manpower to a late software project makes it later." (The Brooks's Law)... and I guess companies do not know about this one either :-)

On Fri, Mar 5, 2010 at 4:25 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

There is still one major advantage to preventing modification of past commits: once you find out there's been a breach, you can just go back through the commits *since* the breach and double-check them. Without that guarantee, you have to recheck *every* commit, which is much more work.

Not to say that a sneaky commit would be easy to detect, though. I often add bugs to my own code without even trying to hide them, and they're still pretty hard to find afterward.

Have fun,

Avery

Hi,

On Fri, 5 Mar 2010, Avery Pennarun wrote:

If you find out which commit it was in the past, you can always revert it. It does not take Git to do it.

I am all in favor of Git, yes, but let's be honest: Git does not prevent an intelligent break-in.

To repeat, as I seem to not have made the point before: a break-in is a social problem, so it requires a social solution.

Ciao, Dscho

On Thu, 4 Mar 2010, walt wrote:

I took a look at the white paper the article links to. I had to ignore a lot of the introductory sections (yes, the most secure system would be to prevent people from doing any work that might be stolen or released after it was corrupted), but I assume that the "findings" are the actually relevant part. Comparing git and Perforce here:

 - The Perforce server software for Windows installs to run as root. I'm 
   not sure what the norm is for git central repositories on Windows, but 
   it's probably better. I don't know if people actually run Perforce 
   servers on Windows in practice, either.

 - Perforce has built-in authorization and authentication. By default, it 
   allows unauthenticated people to create users without any specific 
   authorization. It transmits passwords in cleartext in some cases. It 
   discloses a lot of information about the authorization and 
   authentication in force to arbitrary people, including users of the 
   internal web site who do not have protocol access at all. It issues 
   login tickets that last a long time. The authorization controls are not 
   applied reliably to operations that modify the authorization and 
   authentication information in some of the server software. The initial 
   configuration with respect to access control is completely 
   unrestrictive. Git does not have built-in authorization or 
   authentication, so avoiding or making these mistakes is outside git's 
   scope.

 - Perforce sends all of content over the network in cleartext. This is 
   essentially true of git as well, but in order to get any sort of access 
   control with git, you need to use some wrapping method, which will 
   generally provide encryption as well.

 - Perforce stores, on the server, the location of the working directory 
   on the client, and this is used by the client to place files. Git does 
   not store this information at all.

In general, they seem to have found numerous flaws due to the fact that Perforce includes security-related code while not being designed by security specialists. Git is designed not to include security-related code, and to have properly developed security code control access to it. It is possible to run Perforce in a configuration where access control is external to Perforce, but it's not easy or standard.

On the other hand, I don't see any indication that the attack they were investigating used any of the problems they found, or any problems of a similar class. The actual attack seemed to involve a successful attack on the workstation of someone with legitimate priviledges, which the attackers then used. It's hard to say if any security measures on the part of the SCM could have any effect other than limiting the choice of the user to target.

	-Daniel
*This .sig left intentionally blank*