encrypted repositories?

Greetings,

I have a rather special usage scenario.

Assume you have a repository where you want to work on embargoed information, so that not even system administrators of the server you're pushing to can get a hold of the cleartext data.

"Server" would be a central reference repository that I can push to. "Client" would by my working computer that has a clone of the crypted repo, and an unencrypted checkout of it. Perhaps the client would also need an unencrypted copy of the repo (for performance reasons, I'm not sure about that) that gets encrypted on the fly when pushing and decrypted when fetching.

Examples of use might be press releases of upcoming products, written exams for students, whatever.

Requirements:
- "client" that is about to push must encrypt the data before pushing it  
to the server.
- all data (including file names, log messages,

Allowed restrictions:
- "server" limited to bare repositories
- initial version limited to symmetric encryption with pre-shared secret

In a later step, some key management and asymmetric crypto would be useful, but that's not crucial now. In my current scenario, those who are working on the embargoed material would trust one another.

How would one go about this from the user side? I sincerely doubt I have the resources (time!) to actually implement this in Git.

TIA

Matthias Andree venit, vidit, dixit 17.07.2009 17:14:

If the server can not decrypt anything then it can not serve anything, at least not as a git server. Note that if you're really fussy about security then you should not allow the server to see even the DAG (which would be the case if you encrypt blobs only), which makes it impossible to do any smart serving.

So, why not share some form of remote storage on which you have an encrypted luks partition? That way you can even set up multiple access keys and revoke them when necessary.

Michael

Michael J Gruber <git@drmicha.warpmail.net> writes:

There was shown here on git mailing list script which was meant to help in situation where you have repository with sensitive deta, discovered repository corruption or bug in git, and cannot be reproduced otherwise. But I think it didn't encrypt repositry, but just emulate it's structure.

As to encrypting repository: you can encrypt blobs (content of files), you can encrypt filenames (but the structure remains) or you can put files in a flat encrypted structure, and you can encrypt commit messages and comitter and author info, and encrypt / rename branch names. Still some DAG structure will be visible, and need be visible for "smart" git server (access via ssh and git protocols) to work.

On Fri, Jul 17, 2009 at 3:14 PM, Matthias Andree<matthias.andree@gmx.de> wrote:

Maybe you could send around packages created by git-bundle as pgp-encrypted emails, and keep the original on your computer on an encrypted filesystem? You could also encrypt the bundles and put them onto a public server if that suits you better than email.

The biggest advantage of this approach would be that it can be done now, and no changes to git itself are required.

Matthias

On Fri, 17 Jul 2009, Matthias Andree wrote:

If the server can't ever read it, you're basically limited to just one story:

 - use rsync-like "stupid" transports to upload and download things.

 - a "smart" git server (eg the native git:// style protocol is not going 
   to be possible)

and you strictly speaking need no real git changes, because you might as well just do it by uploading an encrypted tar-file of the .git directory. And there is literally no upside in doing anything else - any native git support is almost entirely pointless.

You could make it a _bit_ more useful perhaps by adding some helper wrappers, probably by just implementing a new transport name (ie instead of using "rsync://", you'd just use "crypt-tgz://" or something).

Now, that said, there are probably situations where maybe you'd allow the server to decrypt things _temporarily_, but you don't want to be encrypted on disk, and no persistent keys on the server, then that would open up a lot more possibilities.

Of course, that still does require that you trust the server admin to _some_ degree - anybody who has root would be able to get the keys by running a debugger on the git upload/download sequence when you do a upload or download.

Maybe that kind of security is still acceptable to you, though? 

IF that is the case, then at least in theory we could add support for "encryption key exchange" to the native git protocol, and then you could have encryption over the network access (ssh obviously already does that, but I'm including things like the anonymous git:// protocol too), and you'd have encrypted data on disk, but git-upload-pack would be able to decrypt things in order to do deltas etc.

But see above: in order for that to work, you do have to allow the pack upload and download processes on the server to decrypt things (in memory). So it would not be "absolutely secure".

			Linus

2009/7/17 Linus Torvalds <torvalds@linux-foundation.org>:

Why couldn't you have the actual code encrypted, but have the server still know about the SHAs etc? You would expose the actual commit structure, but that might be acceptable?

John

On Fri, 17 Jul 2009, John Tapsell wrote:

Even that wouldn't really work, because you'd never be able to generate any deltas.

So there would be no real advantage. In fact, there would be only disadvantages, because without any delta generation, you'd now have to actually transfer _more_ data.

			Linus

On Fri, 17 Jul 2009, Linus Torvalds wrote:

Oh, if you let the server know all the SHA's at _all_ levels (ie down to the blob itself), and then just make the blobs be encrypted, we'd be able to do some trivial optimizations, like only sending the actual blobs that changed. HOWEVER. That would reveal absolutely tons of data about the repository, and about the history. You'd have lost a _lot_ of security.

			Linus

Wouldn't this be a use case for git-torrent? http://code.google.com/p/gittorrent/ http://repo.or.cz/w/VCS-Git-Torrent.git

As I understand it, all data would be stored decentraliced and the (optional?) central server only saves, who has which objects.

(Just hearing a podcast on bittorrent while reading GIT mailinglist :-)

Thomas Koch, http://www.koch.ro

Am 18.07.2009, 21:09 Uhr, schrieb Thomas Koch <thomas@koch.ro>:

I wonder about latency and accessibility here if clients are disconnected. Seems this is more for transferring repositories to large numbers of customers as sort of content distribution network for high load, rather than low connectivity - and the latter is my prime concern and also a detail of my scenario.

Am 17.07.2009, 21:38 Uhr, schrieb Linus Torvalds <torvalds@linux-foundation.org>:

I don't know all its features, apparently it's online recompression - this is no longer going to be available.

No, the server can't be allowed access to the keys or decrypted data.

I'm not sure about the graph, and if I should be concerned. Exposing the DAG might be in order.

It would be ok if the disk storage and the over-the-wire format cannot use delta compression then. It would suffice to just send a set of objects efficiently - and perhaps smaller revisions can be delta-compressed by the clients when pushing.

I admit haven't checked how the current git:// over-the-wire protocol[s] work[s]. I think client-side delta compression may require limiting the graph depths or delta size (when exceeded, the client must send the standalone self-contained object rather than a delta), so that the server can refuse patches when the delta nesting or size gets too deep/big.

I think this would generate the git server to something like a storage device for objects, perhaps with the DAG if exposed.

On a more general note, is someone looking into improving the http:// efficiency? Perhaps there are synergies between my plan of (a) encryption and (b) more efficient "dumb" (http/rsync/...) protocol use.

"Matthias Andree" <matthias.andree@gmx.de> writes:

There was idea about improving http:// efficiency, but it was via crating git-over-HTTP aka. "smart" HTTP server, i.e. you would have to have DAG exposed, like for git:// and ssh://

On the other hand for http:// server need only "dumb" web server, and additional metadata generated by git-update-server-info. It is client who does "walking" the DAG, so all data including server metadata can be encrypted, and decrypted on-the-fly by client.

I don't know though what information leakage you would get from existence of loose objects and packfiles, and their sizes. Probably negligible...

Am 20.07.2009, 15:48 Uhr, schrieb Jakub Narebski <jnareb@gmail.com>:

Fine by me, and seems to be some "minimal disclosure to server".

Dunno. Given that it's just a collection of object sizes, you can't tell  
 from the SHA1 if the object in question is tree, tag, blob, or commit.

I'm really not after on-the-fly delta re-compression on the server-side for crypto stuff. I'm more thinking along the lines of zsync/bsdiff/xdelta (http://zsync.moria.org.uk/ for the least-known) - but zsync can't work on encrypted data. Perhaps encrypting the diffs could work, but then what's the difference to using the http:// and update-server-info related material and combining that with client-side on-the-fly (de/en)cryption?

On Mon, Jul 20, 2009 at 02:09:28PM +0200, Matthias Andree wrote:

The problem is that you need to expose not just the DAG, but also the hashes of trees and blobs. Because if I know you have master^, and I want to send you master, then I need to know which objects are referenced by master that are not referenced by master^.

So now you have security implications, because I can do an offline guessing attack against your files (i.e., calculate git blob hashes for likely candidates and see if you have them). Whether that is a problem really depends on your data.

Not to mention that it makes the protocol a lot more complex, as you would be encrypting _parts_ of objects, like the filenames of a tree, and the commit message of a commit object.

I suppose in theory you could obfuscate the sha1's in a way that preserved the object relationships but revealed no information. That is, the server would have one "fake" set of sha1's, and the client would map its real sha1's to the fake ones when talking with the server. But that is again potentially getting complex.

-Peff

Am 20.07.2009, 17:30 Uhr, schrieb Jeff King <peff@peff.net>:

Yes, you need to know that. Not all of the push logic needs to be implemented on the server though.

In my scenario, the server degenerates into sort of a general object store  
- I really don't expect much smartness there. What is easily available  
(clients providing deltas rather than full objects) could be exploited,  
and that's it.

We can always have two local repositories, one reference and one checkout. The reference is a decrypted (unencrypted) copy of the set of objects on the server, and I could use that for tracking the server-side view (for instance, what are master^ and master pointing at so I can derive git rev-list master^...master, what do I need to send to the server).

I'm well aware that crypto requires more efforts on the client side if we don't trust the server, that's just natural.

The question is: which VCS can serve my scenario?

Or look at commit frequency and push sources. There's always a leak of information even if I just upload a series of blah-2009MMDD-NNN.tar.lzma.gpg files... The data is going to be obsolete, say, 3 months; students then write the exam and then it's sort of public anyways. Even if your model does not entail not publishing exams (as opposed to embargoed press releases under development), but you can't prevent someone from writing their recollection of the problems from memory afterwards and sharing it with other students.

Is your concern that the object name (SHA1) is derived from the unencrypted version?

On Tue, Jul 21, 2009 at 10:25:50AM +0200, Matthias Andree wrote:

Yes, though fetching is much harder, since the server is the one holding the information about how the transfer can be optimized. Still, you should be able to achieve roughly the same performance as http fetching from a dumb server.

Yes. You are potentially leaking considerable information about the unencrypted contents which an attacker could use to guess those contents (especially if the file is mostly composed of low-entropy parts, like text formatting).

-Peff

To Matthias.

My name is John Brumbelow, and I know how to solve your issue. Its requires client-driven-encryption, such that the server never gets the data un-encrypted, ever, and thus can never decrypt it, but yet, on behalf of the client, and other clients, users can still search, partially search, wild-char-search, document-reference-search, and most important, sort data stored on the server, that the server (and its administrators) can never "see" cause it never left the client making/editing it, un-encrypted.

Note the key phrase is client-driven-encryption, not, client-encryption. That is, the client drives the encryption process in conjunction with the server(s).

This makes it so the data can never be stolen from the server, even if a villan showed up at the server and held the administrators hostage, or at any point beyond the client.

There is more...

I also know how to further obfuscate the data, as it is being made, to protect the end user, so if the villan showed up at the client, and held them hostage and/or hacked their computer(s), their data would still be protected. This is not just a simple, extra-hash over the data, but something that gives the villan fake/false data, which could be traced/tracked, by the server/authorities.

Please email me at John-S-Brumbelow@hotmail.com

-- View this message in context: http://git.661346.n2.nabble.com/encrypted-repositories-tp3275970p7564308.html Sent from the git mailing list archive at Nabble.com.