[JGIT PATCH 00/21] Push support over SFTP and (encrypted) Amazon S3

People have often asked on the mailing list if Git can push to a remote server directly over SFTP, without needing to have Git installed on the remote system. This mode of operation can be useful if the remote server is an inexpensive hosting account and the user wants to publish over HTTP.

With this series jgit can now push directly over sftp:// style URI without needing Git to be installed on the remote system. Both the real refs and the dumb transport support files (such as info/refs) are updated during the push.

A transport for Amazon S3 (http://aws.amazon.com/s3) is also included. S3 is an inexpensive network based storage system provided as a commerical service by Amazon. Public data stored in S3 is available over standard HTTP, making it an inexpensive hosting provider.

Git repositories pushed to S3 may optionally be transparently encrypted with an encryption key of the user's choosing, hiding the repository content from Amazon. An encrypted repository may only be accessed by jgit, or by downloading it through jets3t, and accessing the local copy. (This is because all encryption/decryption occurs on the client.)

I wanted the S3 support just so I could backup my repositories as easily as I can backup through native Git. Its as simple as:

	touch ~/.s3_ident
	chmod 600 ~/.s3_ident
	cat >~/.s3_ident
	accesskey: <AWSAccessKeyId>
	secretkey: <AWSSecretAccessKey>
	password: <secretpassphrasetoseedencryption>
	^D
	jgit push amazon-s3://.s3_ident@bucket/repo.git refs/heads/master

The bucket must have already been created with another S3 client. I consider it outside of the scope of jgit to register buckets. However the repository name can be any string of your choosing and the repository will be created on S3 during the first push.

You can also clone this branch off S3 using anonymous HTTP:

	git clone http://gitney.s3.amazonaws.com/projects/egit.git

I pushed the above repository with:

	touch ~/.s3_pub
	chmod 600 ~/.s3_pub
	cat >~/.s3_pub
	accesskey: <AWSAccessKeyId>
	secretkey: <AWSSecretAccessKey>
	acl: public
	^D
	git remote add s3 amazon-s3://.s3_pub@gitney/projects/egit.git
	jgit push s3 refs/heads/dumb-push

After writing the full S3 client from scratch and implementing an encryption scheme that is compatible with jets3t (a popular Java based S3 client) I've realized that jets3t's encryption scheme is not as strong as it could be, especially if you can recognize a pattern in the plain text (such as the format of info/refs, or even of a pack and pack index). As such the encryption used by jgit is "eh, ok". This may be an area of improvement in future versions, but at present should at least stop any sort of casual snooping.

This series is based on `pu` as it requires both Marek's push topic and my index-v2 topic.

----
Robert Harder (1):
  Add Robert Harder's public domain Base64 encoding utility

Shawn O. Pearce (20):
  Remove unused index files when WalkFetchConnection closes
  Do not show URIish passwords in TransportExceptions
  Use PackedObjectInfo as a base class for PackWriter's ObjectToPack
  Refactor PackWriter to hold onto the sorted object list
  Save the pack checksum after computing it in PackWriter
  Allow PackIndexWriter to use any subclass of PackedObjectInfo
  Allow PackWriter to create a corresponding index file
  Allow PackWriter to prepare object list and compute name before
    writing
  Remember how a Ref was read in from disk and created
  Simplify walker transport ref advertisement setup
  Indicate the protocol jgit doesn't support push over
  WalkTransport must allow subclasses to implement openPush
  Support push over the sftp:// dumb transport
  Extract readPackedRefs from TransportSftp for reuse
  Specialized byte array output stream for large files
  Misc. documentation fixes to Base64 utility
  Extract the basic HTTP proxy support to its own class
  Create a really simple Amazon S3 REST client
  Add client side encryption to Amazon S3 client library
  Bidirectional protocol support for Amazon S3

 .../tst/org/spearce/jgit/lib/PackWriterTest.java   |    8 +-
 .../spearce/jgit/transport/PushProcessTest.java    |   94 +-
 .../spearce/jgit/transport/RefSpecTestCase.java    |   26 +-
 .../spearce/jgit/errors/TransportException.java    |    4 +-
 .../src/org/spearce/jgit/lib/PackIndexWriter.java  |    6 +-
 .../src/org/spearce/jgit/lib/PackWriter.java       |  216 ++--
 org.spearce.jgit/src/org/spearce/jgit/lib/Ref.java |   91 ++-
 .../src/org/spearce/jgit/lib/RefDatabase.java      |   23 +-
 .../src/org/spearce/jgit/pgm/Main.java             |   36 +-
 .../src/org/spearce/jgit/transport/AmazonS3.java   |  770 ++++++++++
 .../spearce/jgit/transport/BasePackConnection.java |    6 +-
 .../jgit/transport/BasePackPushConnection.java     |    5 +-
 .../src/org/spearce/jgit/transport/Transport.java  |    3 +
 .../spearce/jgit/transport/TransportAmazonS3.java  |  319 +++++
 .../spearce/jgit/transport/TransportBundle.java    |    3 +-
 .../org/spearce/jgit/transport/TransportHttp.java  |   64 +-
 .../org/spearce/jgit/transport/TransportSftp.java  |  162 ++-
 .../src/org/spearce/jgit/transport/URIish.java     |   24 +-
 .../org/spearce/jgit/transport/WalkEncryption.java |  188 +++
 .../jgit/transport/WalkFetchConnection.java        |    2 +
 .../spearce/jgit/transport/WalkPushConnection.java |  296 ++++
 .../jgit/transport/WalkRemoteObjectDatabase.java   |  301 ++++
 .../org/spearce/jgit/transport/WalkTransport.java  |    8 +-
 .../src/org/spearce/jgit/util/Base64.java          | 1465 ++++++++++++++++++++
 .../src/org/spearce/jgit/util/HttpSupport.java     |  165 +++
 .../src/org/spearce/jgit/util/TemporaryBuffer.java |  260 ++++
 26 files changed, 4250 insertions(+), 295 deletions(-)
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/TransportAmazonS3.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/WalkEncryption.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/WalkPushConnection.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/util/Base64.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/util/HttpSupport.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/util/TemporaryBuffer.java