{"thread":{"id":"14207","subject":"[JGIT PATCH 03/21] Use PackedObjectInfo as a base class for PackWriter's ObjectToPack","startedAt":"2008-06-29T07:59:10Z","lastAt":"2008-06-29T18:06:55Z","messageCount":27,"participants":["Shawn O. Pearce","Robin Rosenberg","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":21},"messages":[{"id":"81601","messageId":"1214726371-93520-1-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":null,"subject":"[JGIT PATCH 00/21] Push support over SFTP and (encrypted) Amazon S3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:10Z","receivedAt":"2008-06-29T07:59:10Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"People have often asked on the mailing list if Git can push to\na remote server directly over SFTP, without needing to have Git\ninstalled on the remote system. This mode of operation can be\nuseful if the remote server is an inexpensive hosting account\nand the user wants to publish over HTTP.\n\nWith this series jgit can now push directly over sftp:// style\nURI without needing Git to be installed on the remote system.\nBoth the real refs and the dumb transport support files (such\nas info/refs) are updated during the push.\n\n\nA transport for Amazon S3 (http://aws.amazon.com/s3) is also\nincluded. S3 is an inexpensive network based storage system\nprovided as a commerical service by Amazon. Public data stored\nin S3 is available over standard HTTP, making it an inexpensive\nhosting provider.\n\nGit repositories pushed to S3 may optionally be transparently\nencrypted with an encryption key of the user's choosing, hiding the\nrepository content from Amazon. An encrypted repository may only be\naccessed by jgit, or by downloading it through jets3t, and accessing\nthe local copy. (This is because all encryption/decryption occurs\non the client.)\n\nI wanted the S3 support just so I could backup my repositories as\neasily as I can backup through native Git. Its as simple as:\n\n\ttouch ~/.s3_ident\n\tchmod 600 ~/.s3_ident\n\tcat >~/.s3_ident\n\taccesskey: \n\tsecretkey: \n\tpassword: \n\t^D\n\tjgit push amazon-s3://.s3_ident@bucket/repo.git refs/heads/master\n\nThe bucket must have already been created with another S3 client.\nI consider it outside of the scope of jgit to register buckets.\nHowever the repository name can be any string of your choosing and\nthe repository will be created on S3 during the first push.\n\nYou can also clone this branch off S3 using anonymous HTTP:\n\n\tgit clone http://gitney.s3.amazonaws.com/projects/egit.git\n\nI pushed the above repository with:\n\n\ttouch ~/.s3_pub\n\tchmod 600 ~/.s3_pub\n\tcat >~/.s3_pub\n\taccesskey: \n\tsecretkey: \n\tacl: public\n\t^D\n\tgit remote add s3 amazon-s3://.s3_pub@gitney/projects/egit.git\n\tjgit push s3 refs/heads/dumb-push\n\n\nAfter writing the full S3 client from scratch and implementing an\nencryption scheme that is compatible with jets3t (a popular Java\nbased S3 client) I've realized that jets3t's encryption scheme is\nnot as strong as it could be, especially if you can recognize a\npattern in the plain text (such as the format of info/refs, or even\nof a pack and pack index). As such the encryption used by jgit is\n\"eh, ok\". This may be an area of improvement in future versions,\nbut at present should at least stop any sort of casual snooping.\n\n\nThis series is based on `pu` as it requires both Marek's push topic\nand my index-v2 topic.\n\n----\nRobert Harder (1):\n Add Robert Harder's public domain Base64 encoding utility\n\nShawn O. Pearce (20):\n Remove unused index files when WalkFetchConnection closes\n Do not show URIish passwords in TransportExceptions\n Use PackedObjectInfo as a base class for PackWriter's ObjectToPack\n Refactor PackWriter to hold onto the sorted object list\n Save the pack checksum after computing it in PackWriter\n Allow PackIndexWriter to use any subclass of PackedObjectInfo\n Allow PackWriter to create a corresponding index file\n Allow PackWriter to prepare object list and compute name before\n writing\n Remember how a Ref was read in from disk and created\n Simplify walker transport ref advertisement setup\n Indicate the protocol jgit doesn't support push over\n WalkTransport must allow subclasses to implement openPush\n Support push over the sftp:// dumb transport\n Extract readPackedRefs from TransportSftp for reuse\n Specialized byte array output stream for large files\n Misc. documentation fixes to Base64 utility\n Extract the basic HTTP proxy support to its own class\n Create a really simple Amazon S3 REST client\n Add client side encryption to Amazon S3 client library\n Bidirectional protocol support for Amazon S3\n\n .../tst/org/spearce/jgit/lib/PackWriterTest.java | 8 +-\n .../spearce/jgit/transport/PushProcessTest.java | 94 +-\n .../spearce/jgit/transport/RefSpecTestCase.java | 26 +-\n .../spearce/jgit/errors/TransportException.java | 4 +-\n .../src/org/spearce/jgit/lib/PackIndexWriter.java | 6 +-\n .../src/org/spearce/jgit/lib/PackWriter.java | 216 ++--\n org.spearce.jgit/src/org/spearce/jgit/lib/Ref.java | 91 ++-\n .../src/org/spearce/jgit/lib/RefDatabase.java | 23 +-\n .../src/org/spearce/jgit/pgm/Main.java | 36 +-\n .../src/org/spearce/jgit/transport/AmazonS3.java | 770 ++++++++++\n .../spearce/jgit/transport/BasePackConnection.java | 6 +-\n .../jgit/transport/BasePackPushConnection.java | 5 +-\n .../src/org/spearce/jgit/transport/Transport.java | 3 +\n .../spearce/jgit/transport/TransportAmazonS3.java | 319 +++++\n .../spearce/jgit/transport/TransportBundle.java | 3 +-\n .../org/spearce/jgit/transport/TransportHttp.java | 64 +-\n .../org/spearce/jgit/transport/TransportSftp.java | 162 ++-\n .../src/org/spearce/jgit/transport/URIish.java | 24 +-\n .../org/spearce/jgit/transport/WalkEncryption.java | 188 +++\n .../jgit/transport/WalkFetchConnection.java | 2 +\n .../spearce/jgit/transport/WalkPushConnection.java | 296 ++++\n .../jgit/transport/WalkRemoteObjectDatabase.java | 301 ++++\n .../org/spearce/jgit/transport/WalkTransport.java | 8 +-\n .../src/org/spearce/jgit/util/Base64.java | 1465 ++++++++++++++++++++\n .../src/org/spearce/jgit/util/HttpSupport.java | 165 +++\n .../src/org/spearce/jgit/util/TemporaryBuffer.java | 260 ++++\n 26 files changed, 4250 insertions(+), 295 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/TransportAmazonS3.java\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/WalkEncryption.java\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/WalkPushConnection.java\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/util/Base64.java\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/util/HttpSupport.java\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/util/TemporaryBuffer.java\n"},{"id":"81602","messageId":"1214726371-93520-2-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-1-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 01/21] Remove unused index files when WalkFetchConnection closes","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:11Z","receivedAt":"2008-06-29T07:59:11Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"If we downloaded an index but then didn't download the corresponding\npack file we never deleted the index from disk. We should clear any\nunused indexes that are left when we terminate the connection.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../jgit/transport/WalkFetchConnection.java | 2 ++\n 1 files changed, 2 insertions(+), 0 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java\nindex 78116b2..5a21d24 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java\n@@ -201,6 +201,8 @@ class WalkFetchConnection extends BaseFetchConnection {\n \n \t@Override\n \tpublic void close() {\n+\t\tfor (final RemotePack p : unfetchedPacks)\n+\t\t\tp.tmpIdx.delete();\n \t\tfor (final WalkRemoteObjectDatabase r : remotes)\n \t\t\tr.close();\n \t}\n-- \n1.5.6.74.g8a5e\n"},{"id":"81603","messageId":"1214726371-93520-3-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-2-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 02/21] Do not show URIish passwords in TransportExceptions","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:12Z","receivedAt":"2008-06-29T07:59:12Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When construction a transport exception the message may be shown\non screen. If a password was in the URIish then we may wind up\nshowing the user's password, perhaps while there is someone else\nlooking over the user's shoulder and reading their monitor. By\nsetting the field to null we avoid displaying it.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../spearce/jgit/errors/TransportException.java | 4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/errors/TransportException.java b/org.spearce.jgit/src/org/spearce/jgit/errors/TransportException.java\nindex 4a8e37c..7fbbc5a 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/errors/TransportException.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/errors/TransportException.java\n@@ -58,7 +58,7 @@ public class TransportException extends IOException {\n \t * message\n \t */\n \tpublic TransportException(final URIish uri, final String s) {\n-\t\tsuper(uri + \": \" + s);\n+\t\tsuper(uri.setPass(null) + \": \" + s);\n \t}\n \n \t/**\n@@ -74,7 +74,7 @@ public class TransportException extends IOException {\n \t */\n \tpublic TransportException(final URIish uri, final String s,\n \t\t\tfinal Throwable cause) {\n-\t\tthis(uri + \": \" + s, cause);\n+\t\tthis(uri.setPass(null) + \": \" + s, cause);\n \t}\n \n \t/**\n-- \n1.5.6.74.g8a5e\n"},{"id":"81600","messageId":"1214726371-93520-4-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-3-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 03/21] Use PackedObjectInfo as a base class for PackWriter's ObjectToPack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:13Z","receivedAt":"2008-06-29T07:59:13Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The ObjectId and offset portions of PackedObjectInfo are also\nneeded by ObjectToPack. By sharing the same base class with\nIndexPack we can later abstract the index writing function out\nand use it inside of both IndexPack and PackWriter.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/lib/PackWriter.java | 28 +++-----------------\n 1 files changed, 4 insertions(+), 24 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\nindex cec2ab0..ccc6cfe 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n@@ -56,6 +56,7 @@ import org.spearce.jgit.revwalk.ObjectWalk;\n import org.spearce.jgit.revwalk.RevFlag;\n import org.spearce.jgit.revwalk.RevObject;\n import org.spearce.jgit.revwalk.RevSort;\n+import org.spearce.jgit.transport.PackedObjectInfo;\n import org.spearce.jgit.util.CountingOutputStream;\n import org.spearce.jgit.util.NB;\n \n@@ -617,7 +618,7 @@ public class PackWriter {\n \n \t\tassert !otp.isWritten();\n \n-\t\totp.markWritten(countingOut.getCount());\n+\t\totp.setOffset(countingOut.getCount());\n \t\tif (otp.isDeltaRepresentation())\n \t\t\twriteDeltaObject(otp);\n \t\telse\n@@ -762,13 +763,11 @@ public class PackWriter {\n \t * pack-file and object status.\n \t *\n \t */\n-\tstatic class ObjectToPack extends ObjectId {\n+\tstatic class ObjectToPack extends PackedObjectInfo {\n \t\tprivate ObjectId deltaBase;\n \n \t\tprivate PackedObjectLoader reuseLoader;\n \n-\t\tprivate long offset = -1;\n-\n \t\tprivate int deltaDepth;\n \n \t\tprivate boolean wantWrite;\n@@ -838,26 +837,7 @@ public class PackWriter {\n \t\t * @return true if object is already written; false otherwise.\n \t\t */\n \t\tboolean isWritten() {\n-\t\t\treturn offset != -1;\n-\t\t}\n-\n-\t\t/**\n-\t\t * @return offset in pack when object has been already written, or -1 if\n-\t\t * it has not been written yet\n-\t\t */\n-\t\tlong getOffset() {\n-\t\t\treturn offset;\n-\t\t}\n-\n-\t\t/**\n-\t\t * Mark object as written. This information is used to achieve\n-\t\t * delta-base precedence in a pack file.\n-\t\t *\n-\t\t * @param offset\n-\t\t * offset where written object starts\n-\t\t */\n-\t\tvoid markWritten(long offset) {\n-\t\t\tthis.offset = offset;\n+\t\t\treturn getOffset() != 0;\n \t\t}\n \n \t\tPackedObjectLoader getReuseLoader() {\n-- \n1.5.6.74.g8a5e\n"},{"id":"81604","messageId":"1214726371-93520-5-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-4-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 04/21] Refactor PackWriter to hold onto the sorted object list","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:14Z","receivedAt":"2008-06-29T07:59:14Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When creating pack files we sometimes need the sorted object list\nof all contents for two reasons. The first is to get the name of\nthe pack, which computeName() returns today. The other reason is\nto generate a corresponding .idx file for this pack, to support\nrandom access into the data.\n\nSince not all uses of PackWriter require the sorted object name list\n(for example streaming the pack to a network socket) the sorting is\ndone on demand, and cached to avoid needing to do it a second time.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/lib/PackWriter.java | 24 ++++++++++++-------\n 1 files changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\nindex ccc6cfe..0f4cbb4 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n@@ -180,6 +180,8 @@ public class PackWriter {\n \n \tprivate final WindowCursor windowCursor = new WindowCursor();\n \n+\tprivate List sortedByName;\n+\n \tprivate boolean reuseDeltas = DEFAULT_REUSE_DELTAS;\n \n \tprivate boolean reuseObjects = DEFAULT_REUSE_OBJECTS;\n@@ -470,22 +472,26 @@ public class PackWriter {\n \t * @return ObjectId representing SHA-1 name of a pack that was created.\n \t */\n \tpublic ObjectId computeName() {\n-\t\tfinal ArrayList sorted = new ArrayList(\n-\t\t\t\tobjectsMap.size());\n-\t\tfor (List list : objectsLists) {\n-\t\t\tfor (ObjectToPack otp : list)\n-\t\t\t\tsorted.add(otp);\n-\t\t}\n-\n \t\tfinal MessageDigest md = Constants.newMessageDigest();\n-\t\tCollections.sort(sorted);\n-\t\tfor (ObjectToPack otp : sorted) {\n+\t\tfor (ObjectToPack otp : sortByName()) {\n \t\t\totp.copyRawTo(buf, 0);\n \t\t\tmd.update(buf, 0, Constants.OBJECT_ID_LENGTH);\n \t\t}\n \t\treturn ObjectId.fromRaw(md.digest());\n \t}\n \n+\tprivate List sortByName() {\n+\t\tif (sortedByName == null) {\n+\t\t\tsortedByName = new ArrayList(objectsMap.size());\n+\t\t\tfor (List list : objectsLists) {\n+\t\t\t\tfor (ObjectToPack otp : list)\n+\t\t\t\t\tsortedByName.add(otp);\n+\t\t\t}\n+\t\t\tCollections.sort(sortedByName);\n+\t\t}\n+\t\treturn sortedByName;\n+\t}\n+\n \tprivate void writePackInternal() throws IOException {\n \t\tif (reuseDeltas || reuseObjects)\n \t\t\tsearchForReuse();\n-- \n1.5.6.74.g8a5e\n"},{"id":"81611","messageId":"1214726371-93520-6-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-5-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 05/21] Save the pack checksum after computing it in PackWriter","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:15Z","receivedAt":"2008-06-29T07:59:15Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"In order to create a matching .idx file for the pack we have\nwritten out we must retain the last 20 bytes of the pack file\nso we can include it at the trailing end of the index file.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/lib/PackWriter.java | 6 ++++--\n 1 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\nindex 0f4cbb4..6adb629 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n@@ -182,6 +182,8 @@ public class PackWriter {\n \n \tprivate List sortedByName;\n \n+\tprivate byte packcsum[];\n+\n \tprivate boolean reuseDeltas = DEFAULT_REUSE_DELTAS;\n \n \tprivate boolean reuseObjects = DEFAULT_REUSE_OBJECTS;\n@@ -690,8 +692,8 @@ public class PackWriter {\n \n \tprivate void writeChecksum() throws IOException {\n \t\tout.on(false);\n-\t\tfinal byte checksum[] = out.getMessageDigest().digest();\n-\t\tout.write(checksum);\n+\t\tpackcsum = out.getMessageDigest().digest();\n+\t\tout.write(packcsum);\n \t}\n \n \tprivate ObjectWalk setUpWalker(\n-- \n1.5.6.74.g8a5e\n"},{"id":"81608","messageId":"1214726371-93520-7-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-6-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 06/21] Allow PackIndexWriter to use any subclass of PackedObjectInfo","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:16Z","receivedAt":"2008-06-29T07:59:16Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Some users of PackIndexWriter may have extended PackedObjectInfo\nto store additional implementation specific details, but wish to\npass off their own subclass instances to avoid allocating even\nmore memory.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/lib/PackIndexWriter.java | 6 +++---\n 1 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java\nindex 3d0050d..567f099 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexWriter.java\n@@ -86,7 +86,7 @@ public abstract class PackIndexWriter {\n \t */\n \t@SuppressWarnings(\"fallthrough\")\n \tpublic static PackIndexWriter createOldestPossible(final OutputStream dst,\n-\t\t\tfinal List objs) {\n+\t\t\tfinal List objs) {\n \t\tint version = 1;\n \t\tLOOP: for (final PackedObjectInfo oe : objs) {\n \t\t\tswitch (version) {\n@@ -137,7 +137,7 @@ public abstract class PackIndexWriter {\n \tprotected final byte[] tmp;\n \n \t/** The entries this writer must pack. */\n-\tprotected List entries;\n+\tprotected List entries;\n \n \t/** SHA-1 checksum for the entire pack data. */\n \tprotected byte[] packChecksum;\n@@ -172,7 +172,7 @@ public abstract class PackIndexWriter {\n \t * an error occurred while writing to the output stream, or this\n \t * index format cannot store the object data supplied.\n \t */\n-\tpublic void write(final List toStore,\n+\tpublic void write(final List toStore,\n \t\t\tfinal byte[] packDataChecksum) throws IOException {\n \t\tentries = toStore;\n \t\tpackChecksum = packDataChecksum;\n-- \n1.5.6.74.g8a5e\n"},{"id":"81616","messageId":"1214726371-93520-8-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-7-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 07/21] Allow PackWriter to create a corresponding index file","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:17Z","receivedAt":"2008-06-29T07:59:17Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"If we are packing for local use, or are sending the pack file to\na dumb server we must also genrate the matching .idx file so Git\ncan use random access requests to read object data. Since all\nof the necessary information is available in our ObjectToPack we\ncan just pass off the sorted list to PackIndexWriter.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/lib/PackWriter.java | 38 ++++++++++++++++++++\n 1 files changed, 38 insertions(+), 0 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\nindex 6adb629..e346668 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n@@ -192,6 +192,8 @@ public class PackWriter {\n \n \tprivate int maxDeltaDepth = DEFAULT_MAX_DELTA_DEPTH;\n \n+\tprivate int outputVersion;\n+\n \tprivate boolean thin;\n \n \t/**\n@@ -348,6 +350,18 @@ public class PackWriter {\n \t}\n \n \t/**\n+\t * Set the pack index file format version this instance will create.\n+\t * \n+\t * @param version\n+\t * the version to write. The special version 0 designates the\n+\t * oldest (most compatible) format available for the objects.\n+\t * @see PackIndexWriter\n+\t */\n+\tpublic void setIndexVersion(final int version) {\n+\t\toutputVersion = version;\n+\t}\n+\n+\t/**\n \t * Returns objects number in a pack file that was created by this writer.\n \t *\n \t * @return number of objects in pack.\n@@ -482,6 +496,30 @@ public class PackWriter {\n \t\treturn ObjectId.fromRaw(md.digest());\n \t}\n \n+\t/**\n+\t * Create an index file to match the pack file just written.\n+\t *

\n+\t * This method can only be invoked after {@link #writePack(Iterator)} or\n+\t * {@link #writePack(Collection, Collection, boolean, boolean)} has been\n+\t * invoked and completed successfully. Writing a corresponding index is an\n+\t * optional feature that not all pack users may require.\n+\t * \n+\t * @param indexStream\n+\t * output for the index data. Caller is responsible for closing\n+\t * this stream.\n+\t * @throws IOException\n+\t * the index data could not be written to the supplied stream.\n+\t */\n+\tpublic void writeIndex(final OutputStream indexStream) throws IOException {\n+\t\tfinal List list = sortByName();\n+\t\tfinal PackIndexWriter iw;\n+\t\tif (outputVersion <= 0)\n+\t\t\tiw = PackIndexWriter.createOldestPossible(indexStream, list);\n+\t\telse\n+\t\t\tiw = PackIndexWriter.createVersion(indexStream, outputVersion);\n+\t\tiw.write(list, packcsum);\n+\t}\n+\n \tprivate List sortByName() {\n \t\tif (sortedByName == null) {\n \t\t\tsortedByName = new ArrayList(objectsMap.size());\n-- \n1.5.6.74.g8a5e\n"},{"id":"81618","messageId":"1214726371-93520-9-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-8-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 08/21] Allow PackWriter to prepare object list and compute name before writing","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:18Z","receivedAt":"2008-06-29T07:59:18Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When we are writing a pack for a dumb protocol transport we want to\nget the name of the pack prior to generating its output stream. This\npermits us to open the pack stream directly under its final name and\nwrite into it without needing to issue a rename in the middle of the\nprocess. By splitting the pack preparing phase from the writing phase\nwe are able to request computeName() between the two stages and create\nthe OutputStream based upon the result.\n\nTo improve performance we also now buffer what we write to the pack\nstream, if the pack stream was not already a buffered stream.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../tst/org/spearce/jgit/lib/PackWriterTest.java | 8 +-\n .../src/org/spearce/jgit/lib/PackWriter.java | 124 +++++++++-----------\n .../jgit/transport/BasePackPushConnection.java | 5 +-\n 3 files changed, 66 insertions(+), 71 deletions(-)\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/PackWriterTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/PackWriterTest.java\nindex 3f07a57..4dd4b2a 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/PackWriterTest.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/PackWriterTest.java\n@@ -87,7 +87,7 @@ public class PackWriterTest extends RepositoryTestCase {\n \t\tpackBase = new File(trash, \"tmp_pack\");\n \t\tpackFile = new File(trash, \"tmp_pack.pack\");\n \t\tindexFile = new File(trash, \"tmp_pack.idx\");\n-\t\twriter = new PackWriter(db, cos, new TextProgressMonitor());\n+\t\twriter = new PackWriter(db, new TextProgressMonitor());\n \t}\n \n \t/**\n@@ -438,14 +438,16 @@ public class PackWriterTest extends RepositoryTestCase {\n \t\t\tfinal Collection uninterestings, final boolean thin,\n \t\t\tfinal boolean ignoreMissingUninteresting)\n \t\t\tthrows MissingObjectException, IOException {\n-\t\twriter.writePack(interestings, uninterestings, thin,\n+\t\twriter.preparePack(interestings, uninterestings, thin,\n \t\t\t\tignoreMissingUninteresting);\n+\t\twriter.writePack(cos);\n \t\tverifyOpenPack(thin);\n \t}\n \n \tprivate void createVerifyOpenPack(final Iterator objectSource)\n \t\t\tthrows MissingObjectException, IOException {\n-\t\twriter.writePack(objectSource);\n+\t\twriter.preparePack(objectSource);\n+\t\twriter.writePack(cos);\n \t\tverifyOpenPack(false);\n \t}\n \ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\nindex e346668..c7aa061 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackWriter.java\n@@ -37,6 +37,7 @@\n \n package org.spearce.jgit.lib;\n \n+import java.io.BufferedOutputStream;\n import java.io.IOException;\n import java.io.OutputStream;\n import java.security.DigestOutputStream;\n@@ -76,10 +77,10 @@ import org.spearce.jgit.util.NB;\n * order of objects in pack\n * \n * Typical usage consists of creating instance intended for some pack,\n- * configuring options through accessors methods and finally call\n- * {@link #writePack(Iterator)} or\n- * {@link #writePack(Collection, Collection, boolean, boolean)} with objects\n- * specification, to generate a pack stream.\n+ * configuring options, preparing the list of objects by calling\n+ * {@link #preparePack(Iterator)} or\n+ * {@link #preparePack(Collection, Collection, boolean, boolean)}, and finally\n+ * producing the stream with {@link #writePack(OutputStream)}.\n *

\n *

\n * Class provide set of configurable options and {@link ProgressMonitor}\n@@ -99,7 +100,7 @@ public class PackWriter {\n \t * Title of {@link ProgressMonitor} task used during counting objects to\n \t * pack.\n \t *\n-\t * @see #writePack(Collection, Collection, boolean, boolean)\n+\t * @see #preparePack(Collection, Collection, boolean, boolean)\n \t */\n \tpublic static final String COUNTING_OBJECTS_PROGRESS = \"Counting objects to pack\";\n \n@@ -107,8 +108,7 @@ public class PackWriter {\n \t * Title of {@link ProgressMonitor} task used during searching for objects\n \t * reuse or delta reuse.\n \t *\n-\t * @see #writePack(Iterator)\n-\t * @see #writePack(Collection, Collection, boolean, boolean)\n+\t * @see #writePack(OutputStream)\n \t */\n \tpublic static final String SEARCHING_REUSE_PROGRESS = \"Searching for delta and object reuse\";\n \n@@ -116,8 +116,7 @@ public class PackWriter {\n \t * Title of {@link ProgressMonitor} task used during writing out pack\n \t * (objects)\n \t *\n-\t * @see #writePack(Iterator)\n-\t * @see #writePack(Collection, Collection, boolean, boolean)\n+\t * @see #writePack(OutputStream)\n \t */\n \tpublic static final String WRITING_OBJECTS_PROGRESS = \"Writing objects\";\n \n@@ -168,9 +167,9 @@ public class PackWriter {\n \n \tprivate final Repository db;\n \n-\tprivate final DigestOutputStream out;\n+\tprivate DigestOutputStream out;\n \n-\tprivate final CountingOutputStream countingOut;\n+\tprivate CountingOutputStream countingOut;\n \n \tprivate final Deflater deflater;\n \n@@ -197,28 +196,22 @@ public class PackWriter {\n \tprivate boolean thin;\n \n \t/**\n-\t * Create writer for specified repository, that will write a pack to\n-\t * provided output stream. Objects for packing are specified in\n-\t * {@link #writePack(Iterator)} or\n-\t * {@link #writePack(Collection, Collection, boolean, boolean)}.\n+\t * Create writer for specified repository.\n+\t *

\n+\t * Objects for packing are specified in {@link #preparePack(Iterator)} or\n+\t * {@link #preparePack(Collection, Collection, boolean, boolean)}.\n \t *\n \t * @param repo\n \t * repository where objects are stored.\n-\t * @param out\n-\t * output stream of pack data; no buffering is guaranteed by\n-\t * writer.\n \t * @param monitor\n \t * operations progress monitor, used within\n-\t * {@link #writePack(Iterator)} or\n-\t * {@link #writePack(Collection, Collection, boolean, boolean)}.\n+\t * {@link #preparePack(Iterator)},\n+\t * {@link #preparePack(Collection, Collection, boolean, boolean)},\n+\t * or {@link #writePack(OutputStream)}.\n \t */\n-\tpublic PackWriter(final Repository repo, final OutputStream out,\n-\t\t\tfinal ProgressMonitor monitor) {\n+\tpublic PackWriter(final Repository repo, final ProgressMonitor monitor) {\n \t\tthis.db = repo;\n \t\tthis.monitor = monitor;\n-\t\tthis.countingOut = new CountingOutputStream(out);\n-\t\tthis.out = new DigestOutputStream(countingOut, Constants\n-\t\t\t\t.newMessageDigest());\n \t\tthis.deflater = new Deflater(db.getConfig().getCore().getCompression());\n \t}\n \n@@ -241,9 +234,9 @@ public class PackWriter {\n \t * use it if possible. Normally, only deltas with base to another object\n \t * existing in set of objects to pack will be used. Exception is however\n \t * thin-pack (see\n-\t * {@link #writePack(Collection, Collection, boolean, boolean)} and\n-\t * {@link #writePack(Iterator)}) where base object must exist on other side\n-\t * machine.\n+\t * {@link #preparePack(Collection, Collection, boolean, boolean)} and\n+\t * {@link #preparePack(Iterator)}) where base object must exist on other\n+\t * side machine.\n \t *

\n \t * When raw delta data is directly copied from a pack file, checksum is\n \t * computed to verify data.\n@@ -371,8 +364,7 @@ public class PackWriter {\n \t}\n \n \t/**\n-\t * Write pack to output stream according to current writer configuration for\n-\t * provided source iterator of objects.\n+\t * Prepare the list of objects to be written to the pack stream.\n \t *

\n \t * Iterator exactly determines which objects are included in a pack\n \t * and order they appear in pack (except that objects order by type is not\n@@ -391,17 +383,6 @@ public class PackWriter {\n \t * {@link RevFlag#UNINTERESTING} flag. This type of pack is used only for\n \t * transport.\n \t *

\n-\t *

\n-\t * At first, this method collects and sorts objects to pack, then deltas\n-\t * search is performed if set up accordingly, finally pack stream is\n-\t * written. {@link ProgressMonitor} tasks {@value #SEARCHING_REUSE_PROGRESS}\n-\t * (only if resueDeltas or reuseObjects is enabled) and\n-\t * {@value #WRITING_OBJECTS_PROGRESS} are updated during packing.\n-\t *

\n-\t *

\n-\t * All reused objects data checksum (Adler32/CRC32) is computed and\n-\t * validated against existing checksum.\n-\t *

\n \t *\n \t * @param objectsSource\n \t * iterator of object to store in a pack; order of objects within\n@@ -414,20 +395,17 @@ public class PackWriter {\n \t * {@link RevFlag#UNINTERESTING} flag set, it won't be included\n \t * in a pack, but is considered as edge-object for thin-pack.\n \t * @throws IOException\n-\t * when some I/O problem occur during reading objects for pack\n-\t * or writing pack stream.\n+\t * when some I/O problem occur during reading objects.\n \t */\n-\tpublic void writePack(final Iterator objectsSource)\n+\tpublic void preparePack(final Iterator objectsSource)\n \t\t\tthrows IOException {\n \t\twhile (objectsSource.hasNext()) {\n \t\t\taddObject(objectsSource.next());\n \t\t}\n-\t\twritePackInternal();\n \t}\n \n \t/**\n-\t * Write pack to output stream according to current writer configuration for\n-\t * provided sets of interesting and uninteresting objects.\n+\t * Prepare the list of objects to be written to the pack stream.\n \t *

\n \t * Basing on these 2 sets, another set of objects to put in a pack file is\n \t * created: this set consists of all objects reachable (ancestors) from\n@@ -437,18 +415,6 @@ public class PackWriter {\n \t * Order is consistent with general git in-pack rules: sort by object type,\n \t * recency, path and delta-base first.\n \t *

\n-\t *

\n-\t * At first, this method collects and sorts objects to pack, then deltas\n-\t * search is performed if set up accordingly, finally pack stream is\n-\t * written. {@link ProgressMonitor} tasks\n-\t * {@value #COUNTING_OBJECTS_PROGRESS}, {@value #SEARCHING_REUSE_PROGRESS}\n-\t * (only if resueDeltas or reuseObjects is enabled) and\n-\t * {@value #WRITING_OBJECTS_PROGRESS} are updated during packing.\n-\t *

\n-\t *

\n-\t * All reused objects data checksum (Adler32/CRC32) is computed and\n-\t * validated against existing checksum.\n-\t *

\n \t *\n \t * @param interestingObjects\n \t * collection of objects to be marked as interesting (start\n@@ -468,17 +434,15 @@ public class PackWriter {\n \t * otherwise - non existing uninteresting objects may cause\n \t * {@link MissingObjectException}\n \t * @throws IOException\n-\t * when some I/O problem occur during reading objects for pack\n-\t * or writing pack stream.\n+\t * when some I/O problem occur during reading objects.\n \t */\n-\tpublic void writePack(final Collection interestingObjects,\n+\tpublic void preparePack(final Collection interestingObjects,\n \t\t\tfinal Collection uninterestingObjects,\n \t\t\tfinal boolean thin, final boolean ignoreMissingUninteresting)\n \t\t\tthrows IOException {\n \t\tObjectWalk walker = setUpWalker(interestingObjects,\n \t\t\t\tuninterestingObjects, thin, ignoreMissingUninteresting);\n \t\tfindObjectsToPack(walker);\n-\t\twritePackInternal();\n \t}\n \n \t/**\n@@ -499,8 +463,8 @@ public class PackWriter {\n \t/**\n \t * Create an index file to match the pack file just written.\n \t *

\n-\t * This method can only be invoked after {@link #writePack(Iterator)} or\n-\t * {@link #writePack(Collection, Collection, boolean, boolean)} has been\n+\t * This method can only be invoked after {@link #preparePack(Iterator)} or\n+\t * {@link #preparePack(Collection, Collection, boolean, boolean)} has been\n \t * invoked and completed successfully. Writing a corresponding index is an\n \t * optional feature that not all pack users may require.\n \t * \n@@ -532,10 +496,38 @@ public class PackWriter {\n \t\treturn sortedByName;\n \t}\n \n-\tprivate void writePackInternal() throws IOException {\n+\t/**\n+\t * Write the prepared pack to the supplied stream.\n+\t *

\n+\t * At first, this method collects and sorts objects to pack, then deltas\n+\t * search is performed if set up accordingly, finally pack stream is\n+\t * written. {@link ProgressMonitor} tasks {@value #SEARCHING_REUSE_PROGRESS}\n+\t * (only if resueDeltas or reuseObjects is enabled) and\n+\t * {@value #WRITING_OBJECTS_PROGRESS} are updated during packing.\n+\t *

\n+\t *

\n+\t * All reused objects data checksum (Adler32/CRC32) is computed and\n+\t * validated against existing checksum.\n+\t *

\n+\t *\n+\t * @param packStream\n+\t * output stream of pack data. If the stream is not buffered it\n+\t * will be buffered by the writer. Caller is responsible for\n+\t * closing the stream.\n+\t * @throws IOException\n+\t * an error occurred reading a local object's data to include in\n+\t * the pack, or writing compressed object data to the output\n+\t * stream.\n+\t */\n+\tpublic void writePack(OutputStream packStream) throws IOException {\n \t\tif (reuseDeltas || reuseObjects)\n \t\t\tsearchForReuse();\n \n+\t\tif (!(packStream instanceof BufferedOutputStream))\n+\t\t\tpackStream = new BufferedOutputStream(packStream);\n+\t\tcountingOut = new CountingOutputStream(packStream);\n+\t\tout = new DigestOutputStream(countingOut, Constants.newMessageDigest());\n+\n \t\tmonitor.beginTask(WRITING_OBJECTS_PROGRESS, getObjectsNumber());\n \t\twriteHeader();\n \t\twriteObjects();\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackPushConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackPushConnection.java\nindex 217486a..7ae3aa7 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackPushConnection.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackPushConnection.java\n@@ -161,7 +161,7 @@ class BasePackPushConnection extends BasePackConnection implements\n \n \tprivate void writePack(final Map refUpdates,\n \t\t\tfinal ProgressMonitor monitor) throws IOException {\n-\t\tfinal PackWriter writer = new PackWriter(local, out, monitor);\n+\t\tfinal PackWriter writer = new PackWriter(local, monitor);\n \t\tfinal ArrayList remoteObjects = new ArrayList(\n \t\t\t\tgetRefs().size());\n \t\tfinal ArrayList newObjects = new ArrayList(\n@@ -172,7 +172,8 @@ class BasePackPushConnection extends BasePackConnection implements\n \t\tfor (final RemoteRefUpdate r : refUpdates.values())\n \t\t\tnewObjects.add(r.getNewObjectId());\n \n-\t\twriter.writePack(newObjects, remoteObjects, thinPack, true);\n+\t\twriter.preparePack(newObjects, remoteObjects, thinPack, true);\n+\t\twriter.writePack(out);\n \t}\n \n \tprivate void readStatusReport(final Map refUpdates)\n-- \n1.5.6.74.g8a5e\n"},{"id":"81605","messageId":"1214726371-93520-10-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-9-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 09/21] Remember how a Ref was read in from disk and created","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:19Z","receivedAt":"2008-06-29T07:59:19Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"To efficiently deleted or update a ref we need to know where\nit came from when it was read into the process. If the ref\nis being updated we can usually just write the loose file,\nbut if it is being deleted we may need to remove not just a\nloose file but also delete it from the packed-refs.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../spearce/jgit/transport/PushProcessTest.java | 94 ++++++++++----------\n .../spearce/jgit/transport/RefSpecTestCase.java | 26 +++---\n org.spearce.jgit/src/org/spearce/jgit/lib/Ref.java | 91 ++++++++++++++++++-\n .../src/org/spearce/jgit/lib/RefDatabase.java | 23 +++--\n .../spearce/jgit/transport/BasePackConnection.java | 6 +-\n .../spearce/jgit/transport/TransportBundle.java | 3 +-\n .../org/spearce/jgit/transport/TransportHttp.java | 6 +-\n .../org/spearce/jgit/transport/TransportSftp.java | 18 +++-\n 8 files changed, 186 insertions(+), 81 deletions(-)\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/transport/PushProcessTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/transport/PushProcessTest.java\nindex cfea4d5..bb912e6 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/transport/PushProcessTest.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/transport/PushProcessTest.java\n@@ -75,65 +75,65 @@ public class PushProcessTest extends RepositoryTestCase {\n \n \t/**\n \t * Test for fast-forward remote update.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateFastForward() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", false, null, null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.OK, true);\n \t}\n \n \t/**\n \t * Test for non fast-forward remote update, when remote object is not known\n \t * to local repository.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateNonFastForwardUnknownObject() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", false, null, null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"0000000000000000000000000000000000000001\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"0000000000000000000000000000000000000001\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.REJECTED_NONFASTFORWARD, null);\n \t}\n \n \t/**\n \t * Test for non fast-forward remote update, when remote object is known to\n \t * local repository, but it is not an ancestor of new object.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateNonFastForward() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\",\n \t\t\t\t\"refs/heads/master\", false, null, null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.REJECTED_NONFASTFORWARD, null);\n \t}\n \n \t/**\n \t * Test for non fast-forward remote update, when force update flag is set.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateNonFastForwardForced() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\",\n \t\t\t\t\"refs/heads/master\", true, null, null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.OK, false);\n \t}\n \n \t/**\n \t * Test for remote ref creation.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateCreateRef() throws IOException {\n@@ -145,21 +145,21 @@ public class PushProcessTest extends RepositoryTestCase {\n \n \t/**\n \t * Test for remote ref deletion.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateDelete() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db, null,\n \t\t\t\t\"refs/heads/master\", false, null, null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.OK, true);\n \t}\n \n \t/**\n \t * Test for remote ref deletion (try), when that ref doesn't exist on remote\n \t * repo.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateDeleteNonExisting() throws IOException {\n@@ -170,21 +170,21 @@ public class PushProcessTest extends RepositoryTestCase {\n \n \t/**\n \t * Test for remote ref update, when it is already up to date.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateUpToDate() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", false, null, null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.UP_TO_DATE, null);\n \t}\n \n \t/**\n \t * Test for remote ref update with expected remote object.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateExpectedRemote() throws IOException {\n@@ -192,15 +192,15 @@ public class PushProcessTest extends RepositoryTestCase {\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", false, null, ObjectId\n \t\t\t\t\t\t.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.OK, true);\n \t}\n \n \t/**\n \t * Test for remote ref update with expected old object set, when old object\n \t * is not that expected one.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateUnexpectedRemote() throws IOException {\n@@ -208,8 +208,8 @@ public class PushProcessTest extends RepositoryTestCase {\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", false, null, ObjectId\n \t\t\t\t\t\t.fromString(\"0000000000000000000000000000000000000001\"));\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.REJECTED_REMOTE_CHANGED, null);\n \t}\n \n@@ -217,7 +217,7 @@ public class PushProcessTest extends RepositoryTestCase {\n \t * Test for remote ref update with expected old object set, when old object\n \t * is not that expected one and force update flag is set (which should have\n \t * lower priority) - shouldn't change behavior.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateUnexpectedRemoteVsForce() throws IOException {\n@@ -225,14 +225,14 @@ public class PushProcessTest extends RepositoryTestCase {\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", true, null, ObjectId\n \t\t\t\t\t\t.fromString(\"0000000000000000000000000000000000000001\"));\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.REJECTED_REMOTE_CHANGED, null);\n \t}\n \n \t/**\n \t * Test for remote ref udpate, when connection rejects update.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateRejectedByConnection() throws IOException {\n@@ -240,22 +240,22 @@ public class PushProcessTest extends RepositoryTestCase {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", false, null, null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n \t\ttestOneUpdateStatus(rru, ref, Status.REJECTED_OTHER_REASON, null);\n \t}\n \n \t/**\n \t * Test for remote refs updates with mixed cases that shouldn't depend on\n \t * each other.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testUpdateMixedCases() throws IOException {\n \t\tfinal RemoteRefUpdate rruOk = new RemoteRefUpdate(db, null,\n \t\t\t\t\"refs/heads/master\", false, null, null);\n-\t\tfinal Ref refToChange = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n+\t\tfinal Ref refToChange = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n \t\tfinal RemoteRefUpdate rruReject = new RemoteRefUpdate(db, null,\n \t\t\t\t\"refs/heads/nonexisting\", false, null, null);\n \t\trefUpdates.add(rruOk);\n@@ -269,15 +269,15 @@ public class PushProcessTest extends RepositoryTestCase {\n \n \t/**\n \t * Test for local tracking ref update.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testTrackingRefUpdateEnabled() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", false, \"refs/remotes/test/master\", null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n \t\trefUpdates.add(rru);\n \t\tadvertisedRefs.add(ref);\n \t\tfinal PushResult result = executePush();\n@@ -290,15 +290,15 @@ public class PushProcessTest extends RepositoryTestCase {\n \n \t/**\n \t * Test for local tracking ref update disabled.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testTrackingRefUpdateDisabled() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", false, null, null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n \t\trefUpdates.add(rru);\n \t\tadvertisedRefs.add(ref);\n \t\tfinal PushResult result = executePush();\n@@ -307,15 +307,15 @@ public class PushProcessTest extends RepositoryTestCase {\n \n \t/**\n \t * Test for local tracking ref update when remote update has failed.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testTrackingRefUpdateOnReject() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\",\n \t\t\t\t\"refs/heads/master\", false, null, null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\"));\n \t\tfinal PushResult result = testOneUpdateStatus(rru, ref,\n \t\t\t\tStatus.REJECTED_NONFASTFORWARD, null);\n \t\tassertTrue(result.getTrackingRefUpdates().isEmpty());\n@@ -323,15 +323,15 @@ public class PushProcessTest extends RepositoryTestCase {\n \n \t/**\n \t * Test for push operation result - that contains expected elements.\n-\t *\n+\t * \n \t * @throws IOException\n \t */\n \tpublic void testPushResult() throws IOException {\n \t\tfinal RemoteRefUpdate rru = new RemoteRefUpdate(db,\n \t\t\t\t\"2c349335b7f797072cf729c4f3bb0914ecb6dec9\",\n \t\t\t\t\"refs/heads/master\", false, \"refs/remotes/test/master\", null);\n-\t\tfinal Ref ref = new Ref(\"refs/heads/master\", ObjectId\n-\t\t\t\t.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n+\t\tfinal Ref ref = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\",\n+\t\t\t\tObjectId.fromString(\"ac7e7e44c1885efb472ad54a78327d66bfc4ecef\"));\n \t\trefUpdates.add(rru);\n \t\tadvertisedRefs.add(ref);\n \t\tfinal PushResult result = executePush();\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/transport/RefSpecTestCase.java b/org.spearce.jgit.test/tst/org/spearce/jgit/transport/RefSpecTestCase.java\nindex 33b3fba..341b4a4 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/transport/RefSpecTestCase.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/transport/RefSpecTestCase.java\n@@ -53,12 +53,12 @@ public class RefSpecTestCase extends TestCase {\n \t\tassertEquals(sn + \":\" + sn, rs.toString());\n \t\tassertEquals(rs, new RefSpec(rs.toString()));\n \n-\t\tRef r = new Ref(sn, null);\n+\t\tRef r = new Ref(Ref.Storage.LOOSE, sn, null);\n \t\tassertTrue(rs.matchSource(r));\n \t\tassertTrue(rs.matchDestination(r));\n \t\tassertSame(rs, rs.expandFromSource(r));\n \n-\t\tr = new Ref(sn + \"-and-more\", null);\n+\t\tr = new Ref(Ref.Storage.LOOSE, sn + \"-and-more\", null);\n \t\tassertFalse(rs.matchSource(r));\n \t\tassertFalse(rs.matchDestination(r));\n \t}\n@@ -73,12 +73,12 @@ public class RefSpecTestCase extends TestCase {\n \t\tassertEquals(\"+\" + sn + \":\" + sn, rs.toString());\n \t\tassertEquals(rs, new RefSpec(rs.toString()));\n \n-\t\tRef r = new Ref(sn, null);\n+\t\tRef r = new Ref(Ref.Storage.LOOSE, sn, null);\n \t\tassertTrue(rs.matchSource(r));\n \t\tassertTrue(rs.matchDestination(r));\n \t\tassertSame(rs, rs.expandFromSource(r));\n \n-\t\tr = new Ref(sn + \"-and-more\", null);\n+\t\tr = new Ref(Ref.Storage.LOOSE, sn + \"-and-more\", null);\n \t\tassertFalse(rs.matchSource(r));\n \t\tassertFalse(rs.matchDestination(r));\n \t}\n@@ -93,12 +93,12 @@ public class RefSpecTestCase extends TestCase {\n \t\tassertEquals(sn, rs.toString());\n \t\tassertEquals(rs, new RefSpec(rs.toString()));\n \n-\t\tRef r = new Ref(sn, null);\n+\t\tRef r = new Ref(Ref.Storage.LOOSE, sn, null);\n \t\tassertTrue(rs.matchSource(r));\n \t\tassertFalse(rs.matchDestination(r));\n \t\tassertSame(rs, rs.expandFromSource(r));\n \n-\t\tr = new Ref(sn + \"-and-more\", null);\n+\t\tr = new Ref(Ref.Storage.LOOSE, sn + \"-and-more\", null);\n \t\tassertFalse(rs.matchSource(r));\n \t\tassertFalse(rs.matchDestination(r));\n \t}\n@@ -113,12 +113,12 @@ public class RefSpecTestCase extends TestCase {\n \t\tassertEquals(\"+\" + sn, rs.toString());\n \t\tassertEquals(rs, new RefSpec(rs.toString()));\n \n-\t\tRef r = new Ref(sn, null);\n+\t\tRef r = new Ref(Ref.Storage.LOOSE, sn, null);\n \t\tassertTrue(rs.matchSource(r));\n \t\tassertFalse(rs.matchDestination(r));\n \t\tassertSame(rs, rs.expandFromSource(r));\n \n-\t\tr = new Ref(sn + \"-and-more\", null);\n+\t\tr = new Ref(Ref.Storage.LOOSE, sn + \"-and-more\", null);\n \t\tassertFalse(rs.matchSource(r));\n \t\tassertFalse(rs.matchDestination(r));\n \t}\n@@ -133,12 +133,12 @@ public class RefSpecTestCase extends TestCase {\n \t\tassertEquals(\":\" + sn, rs.toString());\n \t\tassertEquals(rs, new RefSpec(rs.toString()));\n \n-\t\tRef r = new Ref(sn, null);\n+\t\tRef r = new Ref(Ref.Storage.LOOSE, sn, null);\n \t\tassertFalse(rs.matchSource(r));\n \t\tassertTrue(rs.matchDestination(r));\n \t\tassertSame(rs, rs.expandFromSource(r));\n \n-\t\tr = new Ref(sn + \"-and-more\", null);\n+\t\tr = new Ref(Ref.Storage.LOOSE, sn + \"-and-more\", null);\n \t\tassertFalse(rs.matchSource(r));\n \t\tassertFalse(rs.matchDestination(r));\n \t}\n@@ -157,7 +157,7 @@ public class RefSpecTestCase extends TestCase {\n \t\tRef r;\n \t\tRefSpec expanded;\n \n-\t\tr = new Ref(\"refs/heads/master\", null);\n+\t\tr = new Ref(Ref.Storage.LOOSE, \"refs/heads/master\", null);\n \t\tassertTrue(rs.matchSource(r));\n \t\tassertFalse(rs.matchDestination(r));\n \t\texpanded = rs.expandFromSource(r);\n@@ -167,11 +167,11 @@ public class RefSpecTestCase extends TestCase {\n \t\tassertEquals(r.getName(), expanded.getSource());\n \t\tassertEquals(\"refs/remotes/origin/master\", expanded.getDestination());\n \n-\t\tr = new Ref(\"refs/remotes/origin/next\", null);\n+\t\tr = new Ref(Ref.Storage.LOOSE, \"refs/remotes/origin/next\", null);\n \t\tassertFalse(rs.matchSource(r));\n \t\tassertTrue(rs.matchDestination(r));\n \n-\t\tr = new Ref(\"refs/tags/v1.0\", null);\n+\t\tr = new Ref(Ref.Storage.LOOSE, \"refs/tags/v1.0\", null);\n \t\tassertFalse(rs.matchSource(r));\n \t\tassertFalse(rs.matchDestination(r));\n \t}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/Ref.java b/org.spearce.jgit/src/org/spearce/jgit/lib/Ref.java\nindex b7e361f..5b0e13f 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/Ref.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/Ref.java\n@@ -45,6 +45,74 @@ package org.spearce.jgit.lib;\n * commit, annotated tag, ...).\n */\n public class Ref {\n+\t/** Location where a {@link Ref} is stored. */\n+\tpublic static enum Storage {\n+\t\t/**\n+\t\t * The ref does not exist yet, updating it may create it.\n+\t\t *

\n+\t\t * Creation is likely to choose {@link #LOOSE} storage.\n+\t\t */\n+\t\tNEW(true, false),\n+\n+\t\t/**\n+\t\t * The ref is stored in a file by itself.\n+\t\t *

\n+\t\t * Updating this ref affects only this ref.\n+\t\t */\n+\t\tLOOSE(true, false),\n+\n+\t\t/**\n+\t\t * The ref is stored in the packed-refs file, with\n+\t\t * others.\n+\t\t *

\n+\t\t * Updating this ref requires rewriting the file, with perhaps many\n+\t\t * other refs being included at the same time.\n+\t\t */\n+\t\tPACKED(false, true),\n+\n+\t\t/**\n+\t\t * The ref is both {@link #LOOSE} and {@link #PACKED}.\n+\t\t *

\n+\t\t * Updating this ref requires only updating the loose file, but deletion\n+\t\t * requires updating both the loose file and the packed refs file.\n+\t\t */\n+\t\tLOOSE_PACKED(true, true),\n+\n+\t\t/**\n+\t\t * The ref came from a network advertisement and storage is unknown.\n+\t\t *

\n+\t\t * This ref cannot be updated without Git-aware support on the remote\n+\t\t * side, as Git-aware code consolidate the remote refs and reported them\n+\t\t * to this process.\n+\t\t */\n+\t\tNETWORK(false, false);\n+\n+\t\tprivate final boolean loose;\n+\n+\t\tprivate final boolean packed;\n+\n+\t\tprivate Storage(final boolean l, final boolean p) {\n+\t\t\tloose = l;\n+\t\t\tpacked = p;\n+\t\t}\n+\n+\t\t/**\n+\t\t * @return true if this storage has a loose file.\n+\t\t */\n+\t\tpublic boolean isLoose() {\n+\t\t\treturn loose;\n+\t\t}\n+\n+\t\t/**\n+\t\t * @return true if this storage is inside the packed file.\n+\t\t */\n+\t\tpublic boolean isPacked() {\n+\t\t\treturn packed;\n+\t\t}\n+\t}\n+\n+\tprivate final Storage storage;\n+\n \tprivate final String name;\n \n \tprivate ObjectId objectId;\n@@ -54,13 +122,16 @@ public class Ref {\n \t/**\n \t * Create a new ref pairing.\n \t * \n+\t * @param st\n+\t * method used to store this ref.\n \t * @param refName\n \t * name of this ref.\n \t * @param id\n \t * current value of the ref. May be null to indicate a ref that\n \t * does not exist yet.\n \t */\n-\tpublic Ref(final String refName, final ObjectId id) {\n+\tpublic Ref(final Storage st, final String refName, final ObjectId id) {\n+\t\tstorage = st;\n \t\tname = refName;\n \t\tobjectId = id;\n \t}\n@@ -68,6 +139,8 @@ public class Ref {\n \t/**\n \t * Create a new ref pairing.\n \t * \n+\t * @param st\n+\t * method used to store this ref.\n \t * @param refName\n \t * name of this ref.\n \t * @param id\n@@ -77,7 +150,9 @@ public class Ref {\n \t * peeled value of the ref's tag. May be null if this is not a\n \t * tag or the peeled value is not known.\n \t */\n-\tpublic Ref(final String refName, final ObjectId id, final ObjectId peel) {\n+\tpublic Ref(final Storage st, final String refName, final ObjectId id,\n+\t\t\tfinal ObjectId peel) {\n+\t\tstorage = st;\n \t\tname = refName;\n \t\tobjectId = id;\n \t\tpeeledObjectId = peel;\n@@ -112,6 +187,18 @@ public class Ref {\n \t\treturn peeledObjectId;\n \t}\n \n+\t/**\n+\t * How was this ref obtained?\n+\t *

\n+\t * The current storage model of a Ref may influence how the ref must be\n+\t * updated or deleted from the repository.\n+\t * \n+\t * @return type of ref.\n+\t */\n+\tpublic Storage getStorage() {\n+\t\treturn storage;\n+\t}\n+\n \tpublic String toString() {\n \t\treturn \"Ref[\" + name + \"=\" + getObjectId() + \"]\";\n \t}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java\nindex 1857982..9e3e020 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RefDatabase.java\n@@ -126,12 +126,12 @@ class RefDatabase {\n \tRefUpdate newUpdate(final String name) throws IOException {\n \t\tRef r = readRefBasic(name, 0);\n \t\tif (r == null)\n-\t\t\tr = new Ref(name, null);\n+\t\t\tr = new Ref(Ref.Storage.NEW, name, null);\n \t\treturn new RefUpdate(this, r, fileForRef(r.getName()));\n \t}\n \n \tvoid stored(final String name, final ObjectId id, final long time) {\n-\t\tlooseRefs.put(name, new CachedRef(name, id, time));\n+\t\tlooseRefs.put(name, new CachedRef(Ref.Storage.LOOSE, name, id, time));\n \t}\n \n \t/**\n@@ -256,7 +256,8 @@ class RefDatabase {\n \t\t\t\t\treturn;\n \t\t\t\t}\n \n-\t\t\t\tref = new CachedRef(refName, id, ent.lastModified());\n+\t\t\t\tref = new CachedRef(Ref.Storage.LOOSE, refName, id, ent\n+\t\t\t\t\t\t.lastModified());\n \t\t\t\tlooseRefs.put(ref.getName(), ref);\n \t\t\t\tavail.put(ref.getName(), ref);\n \t\t\t} finally {\n@@ -307,7 +308,7 @@ class RefDatabase {\n \t\t}\n \n \t\tif (line == null || line.length() == 0)\n-\t\t\treturn new Ref(name, null);\n+\t\t\treturn new Ref(Ref.Storage.LOOSE, name, null);\n \n \t\tif (line.startsWith(\"ref: \")) {\n \t\t\tif (depth >= 5) {\n@@ -317,7 +318,7 @@ class RefDatabase {\n \n \t\t\tfinal String target = line.substring(\"ref: \".length());\n \t\t\tfinal Ref r = readRefBasic(target, depth + 1);\n-\t\t\treturn r != null ? r : new Ref(target, null);\n+\t\t\treturn r != null ? r : new Ref(Ref.Storage.LOOSE, target, null);\n \t\t}\n \n \t\tfinal ObjectId id;\n@@ -327,7 +328,7 @@ class RefDatabase {\n \t\t\tthrow new IOException(\"Not a ref: \" + name + \": \" + line);\n \t\t}\n \n-\t\tref = new CachedRef(name, id, mtime);\n+\t\tref = new CachedRef(Ref.Storage.LOOSE, name, id, mtime);\n \t\tlooseRefs.put(name, ref);\n \t\treturn ref;\n \t}\n@@ -359,7 +360,8 @@ class RefDatabase {\n \t\t\t\t\t\t\tthrow new IOException(\"Peeled line before ref.\");\n \n \t\t\t\t\t\tfinal ObjectId id = ObjectId.fromString(p.substring(1));\n-\t\t\t\t\t\tlast = new Ref(last.getName(), last.getObjectId(), id);\n+\t\t\t\t\t\tlast = new Ref(Ref.Storage.PACKED, last.getName(), last\n+\t\t\t\t\t\t\t\t.getObjectId(), id);\n \t\t\t\t\t\tnewPackedRefs.put(last.getName(), last);\n \t\t\t\t\t\tcontinue;\n \t\t\t\t\t}\n@@ -367,7 +369,7 @@ class RefDatabase {\n \t\t\t\t\tfinal int sp = p.indexOf(' ');\n \t\t\t\t\tfinal ObjectId id = ObjectId.fromString(p.substring(0, sp));\n \t\t\t\t\tfinal String name = new String(p.substring(sp + 1));\n-\t\t\t\t\tlast = new Ref(name, id);\n+\t\t\t\t\tlast = new Ref(Ref.Storage.PACKED, name, id);\n \t\t\t\t\tnewPackedRefs.put(last.getName(), last);\n \t\t\t\t}\n \t\t\t} finally {\n@@ -406,8 +408,9 @@ class RefDatabase {\n \tprivate static class CachedRef extends Ref {\n \t\tfinal long lastModified;\n \n-\t\tCachedRef(final String refName, final ObjectId id, final long mtime) {\n-\t\t\tsuper(refName, id);\n+\t\tCachedRef(final Storage st, final String refName, final ObjectId id,\n+\t\t\t\tfinal long mtime) {\n+\t\t\tsuper(st, refName, id);\n \t\t\tlastModified = mtime;\n \t\t}\n \t}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackConnection.java\nindex 2ccd422..a878f01 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackConnection.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackConnection.java\n@@ -161,9 +161,11 @@ abstract class BasePackConnection extends BaseConnection {\n \t\t\t\tif (prior.getPeeledObjectId() != null)\n \t\t\t\t\tthrow duplicateAdvertisement(name + \"^{}\");\n \n-\t\t\t\tavail.put(name, new Ref(name, prior.getObjectId(), id));\n+\t\t\t\tavail.put(name, new Ref(Ref.Storage.NETWORK, name, prior\n+\t\t\t\t\t\t.getObjectId(), id));\n \t\t\t} else {\n-\t\t\t\tfinal Ref prior = avail.put(name, new Ref(name, id));\n+\t\t\t\tfinal Ref prior;\n+\t\t\t\tprior = avail.put(name, new Ref(Ref.Storage.NETWORK, name, id));\n \t\t\t\tif (prior != null)\n \t\t\t\t\tthrow duplicateAdvertisement(name);\n \t\t\t}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java\nindex 1bf081a..6169179 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java\n@@ -165,7 +165,8 @@ class TransportBundle extends PackTransport {\n \n \t\t\t\tfinal String name = line.substring(41, line.length());\n \t\t\t\tfinal ObjectId id = ObjectId.fromString(line.substring(0, 40));\n-\t\t\t\tfinal Ref prior = avail.put(name, new Ref(name, id));\n+\t\t\t\tfinal Ref prior = avail.put(name, new Ref(Ref.Storage.NETWORK,\n+\t\t\t\t\t\tname, id));\n \t\t\t\tif (prior != null)\n \t\t\t\t\tthrow duplicateAdvertisement(name);\n \t\t\t}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\nindex 33f9f90..b18b8e3 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n@@ -235,9 +235,11 @@ class TransportHttp extends WalkTransport {\n \t\t\t\t\tif (prior.getPeeledObjectId() != null)\n \t\t\t\t\t\tthrow duplicateAdvertisement(name + \"^{}\");\n \n-\t\t\t\t\tavail.put(name, new Ref(name, prior.getObjectId(), id));\n+\t\t\t\t\tavail.put(name, new Ref(Ref.Storage.NETWORK, name, prior\n+\t\t\t\t\t\t\t.getObjectId(), id));\n \t\t\t\t} else {\n-\t\t\t\t\tfinal Ref prior = avail.put(name, new Ref(name, id));\n+\t\t\t\t\tfinal Ref prior = avail.put(name, new Ref(\n+\t\t\t\t\t\t\tRef.Storage.NETWORK, name, id));\n \t\t\t\t\tif (prior != null)\n \t\t\t\t\t\tthrow duplicateAdvertisement(name);\n \t\t\t\t}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\nindex 21657ef..092c5d3 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\n@@ -54,6 +54,7 @@ import org.spearce.jgit.errors.TransportException;\n import org.spearce.jgit.lib.ObjectId;\n import org.spearce.jgit.lib.Ref;\n import org.spearce.jgit.lib.Repository;\n+import org.spearce.jgit.lib.Ref.Storage;\n \n import com.jcraft.jsch.Channel;\n import com.jcraft.jsch.ChannelSftp;\n@@ -277,7 +278,8 @@ class TransportSftp extends WalkTransport {\n \t\t\t\t\tif (last == null)\n \t\t\t\t\t\tthrow new TransportException(\"Peeled line before ref.\");\n \t\t\t\t\tfinal ObjectId id = ObjectId.fromString(line + 1);\n-\t\t\t\t\tlast = new Ref(last.getName(), last.getObjectId(), id);\n+\t\t\t\t\tlast = new Ref(Ref.Storage.PACKED, last.getName(), last\n+\t\t\t\t\t\t\t.getObjectId(), id);\n \t\t\t\t\tavail.put(last.getName(), last);\n \t\t\t\t\tcontinue;\n \t\t\t\t}\n@@ -287,7 +289,7 @@ class TransportSftp extends WalkTransport {\n \t\t\t\t\tthrow new TransportException(\"Unrecognized ref: \" + line);\n \t\t\t\tfinal ObjectId id = ObjectId.fromString(line.substring(0, sp));\n \t\t\t\tfinal String name = line.substring(sp + 1);\n-\t\t\t\tlast = new Ref(name, id);\n+\t\t\t\tlast = new Ref(Ref.Storage.PACKED, name, id);\n \t\t\t\tavail.put(last.getName(), last);\n \t\t\t}\n \t\t}\n@@ -342,14 +344,16 @@ class TransportSftp extends WalkTransport {\n \t\t\t\tif (r == null)\n \t\t\t\t\tr = avail.get(p);\n \t\t\t\tif (r != null) {\n-\t\t\t\t\tr = new Ref(name, r.getObjectId(), r.getPeeledObjectId());\n+\t\t\t\t\tr = new Ref(loose(r), name, r.getObjectId(), r\n+\t\t\t\t\t\t\t.getPeeledObjectId());\n \t\t\t\t\tavail.put(name, r);\n \t\t\t\t}\n \t\t\t\treturn r;\n \t\t\t}\n \n \t\t\tif (ObjectId.isId(line)) {\n-\t\t\t\tfinal Ref r = new Ref(name, ObjectId.fromString(line));\n+\t\t\t\tfinal Ref r = new Ref(loose(avail.get(name)), name, ObjectId\n+\t\t\t\t\t\t.fromString(line));\n \t\t\t\tavail.put(r.getName(), r);\n \t\t\t\treturn r;\n \t\t\t}\n@@ -357,6 +361,12 @@ class TransportSftp extends WalkTransport {\n \t\t\tthrow new TransportException(\"Bad ref: \" + name + \": \" + line);\n \t\t}\n \n+\t\tprivate Storage loose(final Ref r) {\n+\t\t\tif (r != null && r.getStorage() == Storage.PACKED)\n+\t\t\t\treturn Storage.LOOSE_PACKED;\n+\t\t\treturn Storage.LOOSE;\n+\t\t}\n+\n \t\t@Override\n \t\tvoid close() {\n \t\t\tif (ftp != null) {\n-- \n1.5.6.74.g8a5e\n"},{"id":"81619","messageId":"1214726371-93520-11-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-10-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 10/21] Simplify walker transport ref advertisement setup","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:20Z","receivedAt":"2008-06-29T07:59:20Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"We need to perform the same advertisement setup during push for\nany of these protocols but we won't have a WalkFetchConnection.\nReturning the map simplifies the code and allows it to be used\nfor the push variant.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../org/spearce/jgit/transport/TransportHttp.java | 15 +++++++--------\n .../org/spearce/jgit/transport/TransportSftp.java | 8 ++++----\n 2 files changed, 11 insertions(+), 12 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\nindex b18b8e3..231dbfe 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n@@ -50,6 +50,7 @@ import java.net.URL;\n import java.net.URLConnection;\n import java.util.ArrayList;\n import java.util.Collection;\n+import java.util.Map;\n import java.util.TreeMap;\n \n import org.spearce.jgit.errors.NotSupportedException;\n@@ -102,7 +103,7 @@ class TransportHttp extends WalkTransport {\n \tpublic FetchConnection openFetch() throws TransportException {\n \t\tfinal HttpObjectDB c = new HttpObjectDB(objectsUrl);\n \t\tfinal WalkFetchConnection r = new WalkFetchConnection(this, c);\n-\t\tc.readAdvertisedRefs(r);\n+\t\tr.available(c.readAdvertisedRefs());\n \t\treturn r;\n \t}\n \n@@ -188,12 +189,11 @@ class TransportHttp extends WalkTransport {\n \t\t\t}\n \t\t}\n \n-\t\tvoid readAdvertisedRefs(final WalkFetchConnection c)\n-\t\t\t\tthrows TransportException {\n+\t\tMap readAdvertisedRefs() throws TransportException {\n \t\t\ttry {\n \t\t\t\tfinal BufferedReader br = openReader(INFO_REFS);\n \t\t\t\ttry {\n-\t\t\t\t\treadAdvertisedImpl(br, c);\n+\t\t\t\t\treturn readAdvertisedImpl(br);\n \t\t\t\t} finally {\n \t\t\t\t\tbr.close();\n \t\t\t\t}\n@@ -208,9 +208,8 @@ class TransportHttp extends WalkTransport {\n \t\t\t}\n \t\t}\n \n-\t\tprivate void readAdvertisedImpl(final BufferedReader br,\n-\t\t\t\tfinal WalkFetchConnection connection) throws IOException,\n-\t\t\t\tPackProtocolException {\n+\t\tprivate Map readAdvertisedImpl(final BufferedReader br)\n+\t\t\t\tthrows IOException, PackProtocolException {\n \t\t\tfinal TreeMap avail = new TreeMap();\n \t\t\tfor (;;) {\n \t\t\t\tString line = br.readLine();\n@@ -244,7 +243,7 @@ class TransportHttp extends WalkTransport {\n \t\t\t\t\t\tthrow duplicateAdvertisement(name);\n \t\t\t\t}\n \t\t\t}\n-\t\t\tconnection.available(avail);\n+\t\t\treturn avail;\n \t\t}\n \n \t\tprivate PackProtocolException outOfOrderAdvertisement(final String n) {\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\nindex 092c5d3..c2f34f7 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\n@@ -48,6 +48,7 @@ import java.util.Collections;\n import java.util.Comparator;\n import java.util.HashMap;\n import java.util.List;\n+import java.util.Map;\n import java.util.TreeMap;\n \n import org.spearce.jgit.errors.TransportException;\n@@ -96,7 +97,7 @@ class TransportSftp extends WalkTransport {\n \tpublic FetchConnection openFetch() throws TransportException {\n \t\tfinal SftpObjectDB c = new SftpObjectDB(uri.getPath());\n \t\tfinal WalkFetchConnection r = new WalkFetchConnection(this, c);\n-\t\tc.readAdvertisedRefs(r);\n+\t\tr.available(c.readAdvertisedRefs());\n \t\treturn r;\n \t}\n \n@@ -245,8 +246,7 @@ class TransportSftp extends WalkTransport {\n \t\t\t}\n \t\t}\n \n-\t\tvoid readAdvertisedRefs(final WalkFetchConnection connection)\n-\t\t\t\tthrows TransportException {\n+\t\tMap readAdvertisedRefs() throws TransportException {\n \t\t\tfinal TreeMap avail = new TreeMap();\n \t\t\ttry {\n \t\t\t\tfinal BufferedReader br = openReader(\"../packed-refs\");\n@@ -262,7 +262,7 @@ class TransportSftp extends WalkTransport {\n \t\t\t}\n \t\t\treadRef(avail, \"../HEAD\", \"HEAD\");\n \t\t\treadLooseRefs(avail, \"../refs\", \"refs/\");\n-\t\t\tconnection.available(avail);\n+\t\t\treturn avail;\n \t\t}\n \n \t\tprivate void readPackedRefs(final TreeMap avail,\n-- \n1.5.6.74.g8a5e\n"},{"id":"81612","messageId":"1214726371-93520-12-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-11-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 11/21] Indicate the protocol jgit doesn't support push over","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:21Z","receivedAt":"2008-06-29T07:59:21Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Not all of jgit's protocols will support push, as it may take time\nto build their implementations or simply prove impractical. Some\nusers may not understand what a walking transport is, but they can\nunderstand that http isn't supported for push.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../org/spearce/jgit/transport/WalkTransport.java | 4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkTransport.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkTransport.java\nindex 29dd661..e208b12 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkTransport.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkTransport.java\n@@ -59,7 +59,7 @@ abstract class WalkTransport extends Transport {\n \n \t@Override\n \tpublic PushConnection openPush() throws NotSupportedException {\n-\t\tthrow new NotSupportedException(\n-\t\t\t\t\"Push is not supported by object walking transports\");\n+\t\tfinal String s = getURI().getScheme();\n+\t\tthrow new NotSupportedException(\"Push not supported over \" + s + \".\");\n \t}\n }\n-- \n1.5.6.74.g8a5e\n"},{"id":"81606","messageId":"1214726371-93520-13-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-12-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 12/21] WalkTransport must allow subclasses to implement openPush","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:22Z","receivedAt":"2008-06-29T07:59:22Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"If a walk implementation actually supports push (such as HTTP\npush via WebDAV) we need to override openPush to return back\na valid PushConnection, however construction could fail with\na TransportException.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../org/spearce/jgit/transport/WalkTransport.java | 4 +++-\n 1 files changed, 3 insertions(+), 1 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkTransport.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkTransport.java\nindex e208b12..b3ea4aa 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkTransport.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkTransport.java\n@@ -38,6 +38,7 @@\n package org.spearce.jgit.transport;\n \n import org.spearce.jgit.errors.NotSupportedException;\n+import org.spearce.jgit.errors.TransportException;\n import org.spearce.jgit.lib.Repository;\n \n /**\n@@ -58,7 +59,8 @@ abstract class WalkTransport extends Transport {\n \t}\n \n \t@Override\n-\tpublic PushConnection openPush() throws NotSupportedException {\n+\tpublic PushConnection openPush() throws NotSupportedException,\n+\t\t\tTransportException {\n \t\tfinal String s = getURI().getScheme();\n \t\tthrow new NotSupportedException(\"Push not supported over \" + s + \".\");\n \t}\n-- \n1.5.6.74.g8a5e\n"},{"id":"81614","messageId":"1214726371-93520-14-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-13-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 13/21] Support push over the sftp:// dumb transport","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:23Z","receivedAt":"2008-06-29T07:59:23Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"jgit now allows users to push changes over the sftp:// protocol,\ntaking advantage of the SFTP client available as part of JSch to\n(more or less) safely update the remote repository.\n\nSince locking is not available over SFTP this is not suitable for\nuse with concurrent pushes, but is safe with concurrent fetches\nduring a single push. This is sufficient support to safely update\na Git repository published over HTTP where the only means of making\nchanges is through SSH/SFTP and the remote side does not make Git\navailable through the shell, or does not offer direct shell access.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../org/spearce/jgit/transport/TransportHttp.java | 2 -\n .../org/spearce/jgit/transport/TransportSftp.java | 109 +++++++-\n .../spearce/jgit/transport/WalkPushConnection.java | 296 ++++++++++++++++++++\n .../jgit/transport/WalkRemoteObjectDatabase.java | 243 ++++++++++++++++\n 4 files changed, 647 insertions(+), 3 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/WalkPushConnection.java\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\nindex 231dbfe..4655950 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n@@ -112,8 +112,6 @@ class TransportHttp extends WalkTransport {\n \t}\n \n \tclass HttpObjectDB extends WalkRemoteObjectDatabase {\n-\t\tprivate static final String INFO_REFS = \"../info/refs\";\n-\n \t\tprivate final URL objectsUrl;\n \n \t\tHttpObjectDB(final URL b) {\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\nindex c2f34f7..e5db6cc 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\n@@ -40,6 +40,7 @@ package org.spearce.jgit.transport;\n import java.io.BufferedReader;\n import java.io.FileNotFoundException;\n import java.io.IOException;\n+import java.io.OutputStream;\n import java.net.ConnectException;\n import java.net.UnknownHostException;\n import java.util.ArrayList;\n@@ -78,6 +79,9 @@ import com.jcraft.jsch.SftpException;\n * listing files through SFTP we can avoid needing to have current\n * objects/info/packs or info/refs files on the\n * remote repository and access the data directly, much as Git itself would.\n+ *

\n+ * Concurrent pushing over this transport is not supported. Multiple concurrent\n+ * push operations may cause confusion in the repository state.\n * \n * @see WalkFetchConnection\n */\n@@ -101,6 +105,14 @@ class TransportSftp extends WalkTransport {\n \t\treturn r;\n \t}\n \n+\t@Override\n+\tpublic PushConnection openPush() throws TransportException {\n+\t\tfinal SftpObjectDB c = new SftpObjectDB(uri.getPath());\n+\t\tfinal WalkPushConnection r = new WalkPushConnection(this, c);\n+\t\tr.available(c.readAdvertisedRefs());\n+\t\treturn r;\n+\t}\n+\n \tSession openSession() throws TransportException {\n \t\tfinal String user = uri.getUser();\n \t\tfinal String pass = uri.getPass();\n@@ -246,10 +258,105 @@ class TransportSftp extends WalkTransport {\n \t\t\t}\n \t\t}\n \n+\t\t@Override\n+\t\tvoid deleteFile(final String path) throws IOException {\n+\t\t\ttry {\n+\t\t\t\tftp.rm(path);\n+\t\t\t} catch (SftpException je) {\n+\t\t\t\tif (je.id == ChannelSftp.SSH_FX_NO_SUCH_FILE)\n+\t\t\t\t\treturn;\n+\t\t\t\tthrow new TransportException(\"Can't delete \" + objectsPath\n+\t\t\t\t\t\t+ \"/\" + path + \": \" + je.getMessage(), je);\n+\t\t\t}\n+\n+\t\t\t// Prune any now empty directories.\n+\t\t\t//\n+\t\t\tString dir = path;\n+\t\t\tint s = dir.lastIndexOf('/');\n+\t\t\twhile (s > 0) {\n+\t\t\t\ttry {\n+\t\t\t\t\tdir = dir.substring(0, s);\n+\t\t\t\t\tftp.rmdir(dir);\n+\t\t\t\t\ts = dir.lastIndexOf('/');\n+\t\t\t\t} catch (SftpException je) {\n+\t\t\t\t\t// If we cannot delete it, leave it alone. It may have\n+\t\t\t\t\t// entries still in it, or maybe we lack write access on\n+\t\t\t\t\t// the parent. Either way it isn't a fatal error.\n+\t\t\t\t\t//\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\n+\t\t@Override\n+\t\tOutputStream writeFile(final String path) throws IOException {\n+\t\t\ttry {\n+\t\t\t\treturn ftp.put(path);\n+\t\t\t} catch (SftpException je) {\n+\t\t\t\tif (je.id == ChannelSftp.SSH_FX_NO_SUCH_FILE) {\n+\t\t\t\t\tmkdir_p(path);\n+\t\t\t\t\ttry {\n+\t\t\t\t\t\treturn ftp.put(path);\n+\t\t\t\t\t} catch (SftpException je2) {\n+\t\t\t\t\t\tje = je2;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\n+\t\t\t\tthrow new TransportException(\"Can't write \" + objectsPath + \"/\"\n+\t\t\t\t\t\t+ path + \": \" + je.getMessage(), je);\n+\t\t\t}\n+\t\t}\n+\n+\t\t@Override\n+\t\tvoid writeFile(final String path, final byte[] data) throws IOException {\n+\t\t\tfinal String lock = path + \".lock\";\n+\t\t\ttry {\n+\t\t\t\tsuper.writeFile(lock, data);\n+\t\t\t\ttry {\n+\t\t\t\t\tftp.rename(lock, path);\n+\t\t\t\t} catch (SftpException je) {\n+\t\t\t\t\tthrow new TransportException(\"Can't write \" + objectsPath\n+\t\t\t\t\t\t\t+ \"/\" + path + \": \" + je.getMessage(), je);\n+\t\t\t\t}\n+\t\t\t} catch (IOException err) {\n+\t\t\t\ttry {\n+\t\t\t\t\tftp.rm(lock);\n+\t\t\t\t} catch (SftpException e) {\n+\t\t\t\t\t// Ignore deletion failure, we are already\n+\t\t\t\t\t// failing anyway.\n+\t\t\t\t}\n+\t\t\t\tthrow err;\n+\t\t\t}\n+\t\t}\n+\n+\t\tprivate void mkdir_p(String path) throws IOException {\n+\t\t\tfinal int s = path.lastIndexOf('/');\n+\t\t\tif (s <= 0)\n+\t\t\t\treturn;\n+\n+\t\t\tpath = path.substring(0, s);\n+\t\t\ttry {\n+\t\t\t\tftp.mkdir(path);\n+\t\t\t} catch (SftpException je) {\n+\t\t\t\tif (je.id == ChannelSftp.SSH_FX_NO_SUCH_FILE) {\n+\t\t\t\t\tmkdir_p(path);\n+\t\t\t\t\ttry {\n+\t\t\t\t\t\tftp.mkdir(path);\n+\t\t\t\t\t\treturn;\n+\t\t\t\t\t} catch (SftpException je2) {\n+\t\t\t\t\t\tje = je2;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\n+\t\t\t\tthrow new TransportException(\"Can't mkdir \" + objectsPath + \"/\"\n+\t\t\t\t\t\t+ path + \": \" + je.getMessage(), je);\n+\t\t\t}\n+\t\t}\n+\n \t\tMap readAdvertisedRefs() throws TransportException {\n \t\t\tfinal TreeMap avail = new TreeMap();\n \t\t\ttry {\n-\t\t\t\tfinal BufferedReader br = openReader(\"../packed-refs\");\n+\t\t\t\tfinal BufferedReader br = openReader(PACKED_REFS);\n \t\t\t\ttry {\n \t\t\t\t\treadPackedRefs(avail, br);\n \t\t\t\t} finally {\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkPushConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkPushConnection.java\nnew file mode 100644\nindex 0000000..ab16f65\n--- /dev/null\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkPushConnection.java\n@@ -0,0 +1,296 @@\n+/*\n+ * Copyright (C) 2008, Shawn O. Pearce \n+ *\n+ * All rights reserved.\n+ *\n+ * Redistribution and use in source and binary forms, with or\n+ * without modification, are permitted provided that the following\n+ * conditions are met:\n+ *\n+ * - Redistributions of source code must retain the above copyright\n+ * notice, this list of conditions and the following disclaimer.\n+ *\n+ * - Redistributions in binary form must reproduce the above\n+ * copyright notice, this list of conditions and the following\n+ * disclaimer in the documentation and/or other materials provided\n+ * with the distribution.\n+ *\n+ * - Neither the name of the Git Development Community nor the\n+ * names of its contributors may be used to endorse or promote\n+ * products derived from this software without specific prior\n+ * written permission.\n+ *\n+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND\n+ * CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES,\n+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT\n+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\n+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+ */\n+\n+package org.spearce.jgit.transport;\n+\n+import java.io.IOException;\n+import java.io.OutputStream;\n+import java.util.ArrayList;\n+import java.util.Collection;\n+import java.util.LinkedHashMap;\n+import java.util.List;\n+import java.util.Map;\n+import java.util.TreeMap;\n+\n+import org.spearce.jgit.errors.TransportException;\n+import org.spearce.jgit.lib.AnyObjectId;\n+import org.spearce.jgit.lib.ObjectId;\n+import org.spearce.jgit.lib.PackWriter;\n+import org.spearce.jgit.lib.ProgressMonitor;\n+import org.spearce.jgit.lib.Ref;\n+import org.spearce.jgit.lib.Repository;\n+import org.spearce.jgit.lib.Ref.Storage;\n+import org.spearce.jgit.transport.RemoteRefUpdate.Status;\n+\n+/**\n+ * Generic push support for dumb transport protocols.\n+ *

\n+ * Since there are no Git-specific smarts on the remote side of the connection\n+ * the client side must handle everything on its own. The generic push support\n+ * requires being able to delete, create and overwrite files on the remote side,\n+ * as well as create any missing directories (if necessary). Typically this can\n+ * be handled through an FTP style protocol.\n+ *

\n+ * Objects not on the remote side are uploaded as pack files, using one pack\n+ * file per invocation. This simplifies the implementation as only two data\n+ * files need to be written to the remote repository.\n+ *

\n+ * Push support supplied by this class is not multiuser safe. Concurrent pushes\n+ * to the same repository may yield an inconsistent reference database which may\n+ * confuse fetch clients.\n+ *

\n+ * A single push is concurrently safe with multiple fetch requests, due to the\n+ * careful order of operations used to update the repository. Clients fetching\n+ * may receive transient failures due to short reads on certain files if the\n+ * protocol does not support atomic file replacement.\n+ * \n+ * @see WalkRemoteObjectDatabase\n+ */\n+class WalkPushConnection extends BaseConnection implements PushConnection {\n+\t/** The repository this transport pushes out of. */\n+\tprivate final Repository local;\n+\n+\t/** Location of the remote repository we are writing to. */\n+\tprivate final URIish uri;\n+\n+\t/** Database connection to the remote repository. */\n+\tprivate final WalkRemoteObjectDatabase dest;\n+\n+\t/**\n+\t * Packs already known to reside in the remote repository.\n+\t *

\n+\t * This is a LinkedHashMap to maintain the original order.\n+\t */\n+\tprivate LinkedHashMap packNames;\n+\n+\t/** Complete listing of refs the remote will have after our push. */\n+\tprivate Map newRefs;\n+\n+\t/**\n+\t * Updates which require altering the packed-refs file to complete.\n+\t *

\n+\t * If this collection is non-empty then any refs listed in {@link #newRefs}\n+\t * with a storage class of {@link Storage#PACKED} will be written.\n+\t */\n+\tprivate Collection packedRefUpdates;\n+\n+\tWalkPushConnection(final WalkTransport walkTransport,\n+\t\t\tfinal WalkRemoteObjectDatabase w) {\n+\t\tlocal = walkTransport.local;\n+\t\turi = walkTransport.getURI();\n+\t\tdest = w;\n+\t}\n+\n+\tpublic void push(final ProgressMonitor monitor,\n+\t\t\tfinal Map refUpdates)\n+\t\t\tthrows TransportException {\n+\t\tmarkStartedOperation();\n+\t\tpackNames = null;\n+\t\tnewRefs = new TreeMap(getRefsMap());\n+\t\tpackedRefUpdates = new ArrayList(refUpdates.size());\n+\n+\t\t// Filter the commands and issue all deletes first. This way we\n+\t\t// can correctly handle a directory being cleared out and a new\n+\t\t// ref using the directory name being created.\n+\t\t//\n+\t\tfinal List updates = new ArrayList();\n+\t\tfor (final RemoteRefUpdate u : refUpdates.values()) {\n+\t\t\tif (AnyObjectId.equals(ObjectId.zeroId(), u.getNewObjectId()))\n+\t\t\t\tdeleteCommand(u);\n+\t\t\telse\n+\t\t\t\tupdates.add(u);\n+\t\t}\n+\n+\t\t// If we have any updates we need to upload the objects first, to\n+\t\t// prevent creating refs pointing at non-existant data. Then we\n+\t\t// can update the refs, and the info-refs file for dumb transports.\n+\t\t//\n+\t\tif (!updates.isEmpty())\n+\t\t\tsendpack(updates, monitor);\n+\t\tfor (final RemoteRefUpdate u : updates)\n+\t\t\tupdateCommand(u);\n+\n+\t\tif (!packedRefUpdates.isEmpty()) {\n+\t\t\ttry {\n+\t\t\t\tdest.writePackedRefs(newRefs.values());\n+\t\t\t\tfor (final RemoteRefUpdate u : packedRefUpdates)\n+\t\t\t\t\tu.setStatus(Status.OK);\n+\t\t\t} catch (IOException err) {\n+\t\t\t\tfor (final RemoteRefUpdate u : packedRefUpdates) {\n+\t\t\t\t\tu.setStatus(Status.REJECTED_OTHER_REASON);\n+\t\t\t\t\tu.setMessage(err.getMessage());\n+\t\t\t\t}\n+\t\t\t\tthrow new TransportException(uri, \"failed updating refs\", err);\n+\t\t\t}\n+\t\t}\n+\n+\t\ttry {\n+\t\t\tdest.writeInfoRefs(newRefs.values());\n+\t\t} catch (IOException err) {\n+\t\t\tthrow new TransportException(uri, \"failed updating refs\", err);\n+\t\t}\n+\t}\n+\n+\t@Override\n+\tpublic void close() {\n+\t\tdest.close();\n+\t}\n+\n+\tprivate void sendpack(final List updates,\n+\t\t\tfinal ProgressMonitor monitor) throws TransportException {\n+\t\tString pathPack = null;\n+\t\tString pathIdx = null;\n+\n+\t\ttry {\n+\t\t\tfinal PackWriter pw = new PackWriter(local, monitor);\n+\t\t\tfinal List need = new ArrayList();\n+\t\t\tfinal List have = new ArrayList();\n+\t\t\tfor (final RemoteRefUpdate r : updates)\n+\t\t\t\tneed.add(r.getNewObjectId());\n+\t\t\tfor (final Ref r : getRefs()) {\n+\t\t\t\thave.add(r.getObjectId());\n+\t\t\t\tif (r.getPeeledObjectId() != null)\n+\t\t\t\t\thave.add(r.getPeeledObjectId());\n+\t\t\t}\n+\t\t\tpw.preparePack(need, have, false, true);\n+\n+\t\t\t// We don't have to continue further if the pack will\n+\t\t\t// be an empty pack, as the remote has all objects it\n+\t\t\t// needs to complete this change.\n+\t\t\t//\n+\t\t\tif (pw.getObjectsNumber() == 0)\n+\t\t\t\treturn;\n+\n+\t\t\tpackNames = new LinkedHashMap();\n+\t\t\tfor (final String n : dest.getPackNames())\n+\t\t\t\tpackNames.put(n, n);\n+\n+\t\t\tfinal String base = \"pack-\" + pw.computeName();\n+\t\t\tfinal String packName = base + \".pack\";\n+\t\t\tpathPack = \"pack/\" + packName;\n+\t\t\tpathIdx = \"pack/\" + base + \".idx\";\n+\n+\t\t\tif (packNames.remove(packName) != null) {\n+\t\t\t\t// The remote already contains this pack. We should\n+\t\t\t\t// remove the index before overwriting to prevent bad\n+\t\t\t\t// offsets from appearing to clients.\n+\t\t\t\t//\n+\t\t\t\tdest.writeInfoPacks(packNames.keySet());\n+\t\t\t\tdest.deleteFile(pathIdx);\n+\t\t\t}\n+\n+\t\t\t// Write the pack file, then the index, as readers look the\n+\t\t\t// other direction (index, then pack file).\n+\t\t\t//\n+\t\t\tOutputStream os = dest.writeFile(pathPack);\n+\t\t\ttry {\n+\t\t\t\tpw.writePack(os);\n+\t\t\t} finally {\n+\t\t\t\tos.close();\n+\t\t\t}\n+\n+\t\t\tos = dest.writeFile(pathIdx);\n+\t\t\ttry {\n+\t\t\t\tpw.writeIndex(os);\n+\t\t\t} finally {\n+\t\t\t\tos.close();\n+\t\t\t}\n+\n+\t\t\t// Record the pack at the start of the pack info list. This\n+\t\t\t// way clients are likely to consult the newest pack first,\n+\t\t\t// and discover the most recent objects there.\n+\t\t\t//\n+\t\t\tfinal ArrayList infoPacks = new ArrayList();\n+\t\t\tinfoPacks.add(packName);\n+\t\t\tinfoPacks.addAll(packNames.keySet());\n+\t\t\tdest.writeInfoPacks(infoPacks);\n+\n+\t\t} catch (IOException err) {\n+\t\t\tsafeDelete(pathIdx);\n+\t\t\tsafeDelete(pathPack);\n+\n+\t\t\tthrow new TransportException(uri, \"cannot store objects\", err);\n+\t\t}\n+\t}\n+\n+\tprivate void safeDelete(final String path) {\n+\t\tif (path != null) {\n+\t\t\ttry {\n+\t\t\t\tdest.deleteFile(path);\n+\t\t\t} catch (IOException cleanupFailure) {\n+\t\t\t\t// Ignore the deletion failure. We probably are\n+\t\t\t\t// already failing and were just trying to pick\n+\t\t\t\t// up after ourselves.\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tprivate void deleteCommand(final RemoteRefUpdate u) {\n+\t\tfinal Ref r = newRefs.remove(u.getRemoteName());\n+\t\tif (r == null) {\n+\t\t\t// Already gone.\n+\t\t\t//\n+\t\t\tu.setStatus(Status.OK);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tif (r.getStorage().isPacked())\n+\t\t\tpackedRefUpdates.add(u);\n+\n+\t\tif (r.getStorage().isLoose()) {\n+\t\t\ttry {\n+\t\t\t\tdest.deleteRef(u.getRemoteName());\n+\t\t\t\tu.setStatus(Status.OK);\n+\t\t\t} catch (IOException e) {\n+\t\t\t\tu.setStatus(Status.REJECTED_OTHER_REASON);\n+\t\t\t\tu.setMessage(e.getMessage());\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tprivate void updateCommand(final RemoteRefUpdate u) {\n+\t\ttry {\n+\t\t\tdest.writeRef(u.getRemoteName(), u.getNewObjectId());\n+\t\t\tnewRefs.put(u.getRemoteName(), new Ref(Storage.LOOSE, u\n+\t\t\t\t\t.getRemoteName(), u.getNewObjectId()));\n+\t\t\tu.setStatus(Status.OK);\n+\t\t} catch (IOException e) {\n+\t\t\tu.setStatus(Status.REJECTED_OTHER_REASON);\n+\t\t\tu.setMessage(e.getMessage());\n+\t\t}\n+\t}\n+}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkRemoteObjectDatabase.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkRemoteObjectDatabase.java\nindex 2196fc9..57d525f 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkRemoteObjectDatabase.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkRemoteObjectDatabase.java\n@@ -43,10 +43,14 @@ import java.io.FileNotFoundException;\n import java.io.IOException;\n import java.io.InputStream;\n import java.io.InputStreamReader;\n+import java.io.OutputStream;\n+import java.io.StringWriter;\n import java.util.ArrayList;\n import java.util.Collection;\n \n import org.spearce.jgit.lib.Constants;\n+import org.spearce.jgit.lib.ObjectId;\n+import org.spearce.jgit.lib.Ref;\n import org.spearce.jgit.util.NB;\n \n /**\n@@ -68,6 +72,10 @@ abstract class WalkRemoteObjectDatabase {\n \n \tstatic final String INFO_HTTP_ALTERNATES = \"info/http-alternates\";\n \n+\tstatic final String INFO_REFS = \"../info/refs\";\n+\n+\tstatic final String PACKED_REFS = \"../packed-refs\";\n+\n \t/**\n \t * Obtain the list of available packs (if any).\n \t *

\n@@ -163,6 +171,241 @@ abstract class WalkRemoteObjectDatabase {\n \tabstract void close();\n \n \t/**\n+\t * Delete a file from the object database.\n+\t *

\n+\t * Path may start with ../ to request deletion of a file that\n+\t * resides in the repository itself.\n+\t *

\n+\t * When possible empty directories must be removed, up to but not including\n+\t * the current object database directory itself.\n+\t *

\n+\t * This method does not support deletion of directories.\n+\t * \n+\t * @param path\n+\t * name of the item to be removed, relative to the current object\n+\t * database.\n+\t * @throws IOException\n+\t * deletion is not supported, or deletion failed.\n+\t */\n+\tvoid deleteFile(final String path) throws IOException {\n+\t\tthrow new IOException(\"Deleting '\" + path + \"' not supported.\");\n+\t}\n+\n+\t/**\n+\t * Open a remote file for writing.\n+\t *

\n+\t * Path may start with ../ to request writing of a file that\n+\t * resides in the repository itself.\n+\t *

\n+\t * The requested path may or may not exist. If the path already exists as a\n+\t * file the file should be truncated and completely replaced.\n+\t *

\n+\t * This method creates any missing parent directories, if necessary.\n+\t * \n+\t * @param path\n+\t * name of the file to write, relative to the current object\n+\t * database.\n+\t * @return stream to write into this file. Caller must close the stream to\n+\t * complete the write request. The stream is not buffered and each\n+\t * write may cause a network request/response so callers should\n+\t * buffer to smooth out small writes.\n+\t * @throws IOException\n+\t * writing is not supported, or attempting to write the file\n+\t * failed, possibly due to permissions or remote disk full, etc.\n+\t */\n+\tOutputStream writeFile(final String path) throws IOException {\n+\t\tthrow new IOException(\"Writing of '\" + path + \"' not supported.\");\n+\t}\n+\n+\t/**\n+\t * Atomically write a remote file.\n+\t *

\n+\t * This method attempts to perform as atomic of an update as it can,\n+\t * reducing (or eliminating) the time that clients might be able to see\n+\t * partial file content. This method is not suitable for very large\n+\t * transfers as the complete content must be passed as an argument.\n+\t *

\n+\t * Path may start with ../ to request writing of a file that\n+\t * resides in the repository itself.\n+\t *

\n+\t * The requested path may or may not exist. If the path already exists as a\n+\t * file the file should be truncated and completely replaced.\n+\t *

\n+\t * This method creates any missing parent directories, if necessary.\n+\t * \n+\t * @param path\n+\t * name of the file to write, relative to the current object\n+\t * database.\n+\t * @param data\n+\t * complete new content of the file.\n+\t * @throws IOException\n+\t * writing is not supported, or attempting to write the file\n+\t * failed, possibly due to permissions or remote disk full, etc.\n+\t */\n+\tvoid writeFile(final String path, final byte[] data) throws IOException {\n+\t\tfinal OutputStream os = writeFile(path);\n+\t\ttry {\n+\t\t\tos.write(data);\n+\t\t} finally {\n+\t\t\tos.close();\n+\t\t}\n+\t}\n+\n+\t/**\n+\t * Delete a loose ref from the remote repository.\n+\t * \n+\t * @param name\n+\t * name of the ref within the ref space, for example\n+\t * refs/heads/pu.\n+\t * @throws IOException\n+\t * deletion is not supported, or deletion failed.\n+\t */\n+\tvoid deleteRef(final String name) throws IOException {\n+\t\tdeleteFile(\"../\" + name);\n+\t}\n+\n+\t/**\n+\t * Overwrite (or create) a loose ref in the remote repository.\n+\t *

\n+\t * This method creates any missing parent directories, if necessary.\n+\t * \n+\t * @param name\n+\t * name of the ref within the ref space, for example\n+\t * refs/heads/pu.\n+\t * @param value\n+\t * new value to store in this ref. Must not be null.\n+\t * @throws IOException\n+\t * writing is not supported, or attempting to write the file\n+\t * failed, possibly due to permissions or remote disk full, etc.\n+\t */\n+\tvoid writeRef(final String name, final ObjectId value) throws IOException {\n+\t\tfinal ByteArrayOutputStream b;\n+\n+\t\tb = new ByteArrayOutputStream(Constants.OBJECT_ID_LENGTH * 2 + 1);\n+\t\tvalue.copyTo(b);\n+\t\tb.write('\\n');\n+\n+\t\twriteFile(\"../\" + name, b.toByteArray());\n+\t}\n+\n+\t/**\n+\t * Rebuild the {@link #INFO_PACKS} for dumb transport clients.\n+\t *

\n+\t * This method rebuilds the contents of the {@link #INFO_PACKS} file to\n+\t * match the passed list of pack names.\n+\t * \n+\t * @param packNames\n+\t * names of available pack files, in the order they should appear\n+\t * in the file. Valid pack name strings are of the form\n+\t * pack-035760ab452d6eebd123add421f253ce7682355a.pack.\n+\t * @throws IOException\n+\t * writing is not supported, or attempting to write the file\n+\t * failed, possibly due to permissions or remote disk full, etc.\n+\t */\n+\tvoid writeInfoPacks(final Collection packNames) throws IOException {\n+\t\tfinal StringBuilder w = new StringBuilder();\n+\t\tfor (final String n : packNames) {\n+\t\t\tw.append(\"P \");\n+\t\t\tw.append(n);\n+\t\t\tw.append('\\n');\n+\t\t}\n+\t\twriteFile(INFO_PACKS, Constants.encodeASCII(w.toString()));\n+\t}\n+\n+\t/**\n+\t * Rebuild the {@link #INFO_REFS} for dumb transport clients.\n+\t *

\n+\t * This method rebuilds the contents of the {@link #INFO_REFS} file to match\n+\t * the passed list of references.\n+\t * \n+\t * @param refs\n+\t * the complete set of references the remote side now has. This\n+\t * should have been computed by applying updates to the\n+\t * advertised refs already discovered.\n+\t * @throws IOException\n+\t * writing is not supported, or attempting to write the file\n+\t * failed, possibly due to permissions or remote disk full, etc.\n+\t */\n+\tvoid writeInfoRefs(final Collection refs) throws IOException {\n+\t\tfinal StringWriter w = new StringWriter();\n+\t\tfinal char[] tmp = new char[Constants.OBJECT_ID_LENGTH * 2];\n+\t\tfor (final Ref r : refs) {\n+\t\t\tif (Constants.HEAD.equals(r.getName())) {\n+\t\t\t\t// Historically HEAD has never been published through\n+\t\t\t\t// the INFO_REFS file. This is a mistake, but its the\n+\t\t\t\t// way things are.\n+\t\t\t\t//\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tr.getObjectId().copyTo(tmp, w);\n+\t\t\tw.write('\\t');\n+\t\t\tw.write(r.getName());\n+\t\t\tw.write('\\n');\n+\n+\t\t\tif (r.getPeeledObjectId() != null) {\n+\t\t\t\tr.getPeeledObjectId().copyTo(tmp, w);\n+\t\t\t\tw.write('\\t');\n+\t\t\t\tw.write(r.getName());\n+\t\t\t\tw.write(\"^{}\\n\");\n+\t\t\t}\n+\t\t}\n+\t\twriteFile(INFO_REFS, Constants.encodeASCII(w.toString()));\n+\t}\n+\n+\t/**\n+\t * Rebuild the {@link #PACKED_REFS} file.\n+\t *

\n+\t * This method rebuilds the contents of the {@link #PACKED_REFS} file to\n+\t * match the passed list of references, including only those refs that have\n+\t * a storage type of {@link Ref.Storage#PACKED}.\n+\t * \n+\t * @param refs\n+\t * the complete set of references the remote side now has. This\n+\t * should have been computed by applying updates to the\n+\t * advertised refs already discovered.\n+\t * @throws IOException\n+\t * writing is not supported, or attempting to write the file\n+\t * failed, possibly due to permissions or remote disk full, etc.\n+\t */\n+\tvoid writePackedRefs(final Collection refs) throws IOException {\n+\t\tboolean peeled = false;\n+\n+\t\tfor (final Ref r : refs) {\n+\t\t\tif (r.getStorage() != Ref.Storage.PACKED)\n+\t\t\t\tcontinue;\n+\t\t\tif (r.getPeeledObjectId() != null)\n+\t\t\t\tpeeled = true;\n+\t\t}\n+\n+\t\tfinal StringWriter w = new StringWriter();\n+\t\tif (peeled) {\n+\t\t\tw.write(\"# pack-refs with:\");\n+\t\t\tif (peeled)\n+\t\t\t\tw.write(\" peeled\");\n+\t\t\tw.write('\\n');\n+\t\t}\n+\n+\t\tfinal char[] tmp = new char[Constants.OBJECT_ID_LENGTH * 2];\n+\t\tfor (final Ref r : refs) {\n+\t\t\tif (r.getStorage() != Ref.Storage.PACKED)\n+\t\t\t\tcontinue;\n+\n+\t\t\tr.getObjectId().copyTo(tmp, w);\n+\t\t\tw.write(' ');\n+\t\t\tw.write(r.getName());\n+\t\t\tw.write('\\n');\n+\n+\t\t\tif (r.getPeeledObjectId() != null) {\n+\t\t\t\tw.write('^');\n+\t\t\t\tr.getPeeledObjectId().copyTo(tmp, w);\n+\t\t\t\tw.write('\\n');\n+\t\t\t}\n+\t\t}\n+\t\twriteFile(PACKED_REFS, Constants.encodeASCII(w.toString()));\n+\t}\n+\n+\t/**\n \t * Open a buffered reader around a file.\n \t *

\n \t * This is shorthand for calling {@link #open(String)} and then wrapping it\n-- \n1.5.6.74.g8a5e\n"},{"id":"81607","messageId":"1214726371-93520-15-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-14-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 14/21] Extract readPackedRefs from TransportSftp for reuse","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:24Z","receivedAt":"2008-06-29T07:59:24Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Other dumb transports may need this functionality available to them.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../org/spearce/jgit/transport/TransportHttp.java | 5 ++\n .../org/spearce/jgit/transport/TransportSftp.java | 47 ++--------------\n .../src/org/spearce/jgit/transport/URIish.java | 22 ++++++++\n .../jgit/transport/WalkRemoteObjectDatabase.java | 58 ++++++++++++++++++++\n 4 files changed, 91 insertions(+), 41 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\nindex 4655950..2f28f2c 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n@@ -119,6 +119,11 @@ class TransportHttp extends WalkTransport {\n \t\t}\n \n \t\t@Override\n+\t\tURIish getURI() {\n+\t\t\treturn new URIish(objectsUrl);\n+\t\t}\n+\n+\t\t@Override\n \t\tCollection getAlternates() throws IOException {\n \t\t\ttry {\n \t\t\t\treturn readAlternates(INFO_HTTP_ALTERNATES);\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\nindex e5db6cc..a33406b 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportSftp.java\n@@ -193,6 +193,11 @@ class TransportSftp extends WalkTransport {\n \t\t}\n \n \t\t@Override\n+\t\tURIish getURI() {\n+\t\t\treturn uri.setPath(objectsPath);\n+\t\t}\n+\n+\t\t@Override\n \t\tCollection getAlternates() throws IOException {\n \t\t\ttry {\n \t\t\t\treturn readAlternates(INFO_ALTERNATES);\n@@ -355,52 +360,12 @@ class TransportSftp extends WalkTransport {\n \n \t\tMap readAdvertisedRefs() throws TransportException {\n \t\t\tfinal TreeMap avail = new TreeMap();\n-\t\t\ttry {\n-\t\t\t\tfinal BufferedReader br = openReader(PACKED_REFS);\n-\t\t\t\ttry {\n-\t\t\t\t\treadPackedRefs(avail, br);\n-\t\t\t\t} finally {\n-\t\t\t\t\tbr.close();\n-\t\t\t\t}\n-\t\t\t} catch (FileNotFoundException notPacked) {\n-\t\t\t\t// Perhaps it wasn't worthwhile, or is just an older repository.\n-\t\t\t} catch (IOException e) {\n-\t\t\t\tthrow new TransportException(uri, \"error in packed-refs\", e);\n-\t\t\t}\n+\t\t\treadPackedRefs(avail);\n \t\t\treadRef(avail, \"../HEAD\", \"HEAD\");\n \t\t\treadLooseRefs(avail, \"../refs\", \"refs/\");\n \t\t\treturn avail;\n \t\t}\n \n-\t\tprivate void readPackedRefs(final TreeMap avail,\n-\t\t\t\tfinal BufferedReader br) throws IOException {\n-\t\t\tRef last = null;\n-\t\t\tfor (;;) {\n-\t\t\t\tString line = br.readLine();\n-\t\t\t\tif (line == null)\n-\t\t\t\t\tbreak;\n-\t\t\t\tif (line.charAt(0) == '#')\n-\t\t\t\t\tcontinue;\n-\t\t\t\tif (line.charAt(0) == '^') {\n-\t\t\t\t\tif (last == null)\n-\t\t\t\t\t\tthrow new TransportException(\"Peeled line before ref.\");\n-\t\t\t\t\tfinal ObjectId id = ObjectId.fromString(line + 1);\n-\t\t\t\t\tlast = new Ref(Ref.Storage.PACKED, last.getName(), last\n-\t\t\t\t\t\t\t.getObjectId(), id);\n-\t\t\t\t\tavail.put(last.getName(), last);\n-\t\t\t\t\tcontinue;\n-\t\t\t\t}\n-\n-\t\t\t\tfinal int sp = line.indexOf(' ');\n-\t\t\t\tif (sp < 0)\n-\t\t\t\t\tthrow new TransportException(\"Unrecognized ref: \" + line);\n-\t\t\t\tfinal ObjectId id = ObjectId.fromString(line.substring(0, sp));\n-\t\t\t\tfinal String name = line.substring(sp + 1);\n-\t\t\t\tlast = new Ref(Ref.Storage.PACKED, name, id);\n-\t\t\t\tavail.put(last.getName(), last);\n-\t\t\t}\n-\t\t}\n-\n \t\tprivate void readLooseRefs(final TreeMap avail,\n \t\t\t\tfinal String dir, final String prefix)\n \t\t\t\tthrows TransportException {\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java b/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java\nindex 307b591..9e7ca83 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java\n@@ -39,6 +39,7 @@\n package org.spearce.jgit.transport;\n \n import java.net.URISyntaxException;\n+import java.net.URL;\n import java.util.regex.Matcher;\n import java.util.regex.Pattern;\n \n@@ -100,6 +101,27 @@ public class URIish {\n \t\t}\n \t}\n \n+\t/**\n+\t * Construct a URIish from a standard URL.\n+\t * \n+\t * @param u\n+\t * the source URL to convert from.\n+\t */\n+\tpublic URIish(final URL u) {\n+\t\tscheme = u.getProtocol();\n+\t\tpath = u.getPath();\n+\n+\t\tfinal String ui = u.getUserInfo();\n+\t\tif (ui != null) {\n+\t\t\tfinal int d = ui.indexOf(':');\n+\t\t\tuser = d < 0 ? ui : ui.substring(0, d);\n+\t\t\tpass = d < 0 ? null : ui.substring(d + 1);\n+\t\t}\n+\n+\t\tport = u.getPort();\n+\t\thost = u.getHost();\n+\t}\n+\n \t/** Create an empty, non-configured URI. */\n \tpublic URIish() {\n \t\t// Configure nothing.\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkRemoteObjectDatabase.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkRemoteObjectDatabase.java\nindex 57d525f..4f5a1cb 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkRemoteObjectDatabase.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkRemoteObjectDatabase.java\n@@ -47,7 +47,9 @@ import java.io.OutputStream;\n import java.io.StringWriter;\n import java.util.ArrayList;\n import java.util.Collection;\n+import java.util.Map;\n \n+import org.spearce.jgit.errors.TransportException;\n import org.spearce.jgit.lib.Constants;\n import org.spearce.jgit.lib.ObjectId;\n import org.spearce.jgit.lib.Ref;\n@@ -76,6 +78,8 @@ abstract class WalkRemoteObjectDatabase {\n \n \tstatic final String PACKED_REFS = \"../packed-refs\";\n \n+\tabstract URIish getURI();\n+\n \t/**\n \t * Obtain the list of available packs (if any).\n \t *

\n@@ -469,6 +473,60 @@ abstract class WalkRemoteObjectDatabase {\n \t\t}\n \t}\n \n+\t/**\n+\t * Read a standard Git packed-refs file to discover known references.\n+\t * \n+\t * @param avail\n+\t * return collection of references. Any existing entries will be\n+\t * replaced if they are found in the packed-refs file.\n+\t * @throws TransportException\n+\t * an error occurred reading from the packed refs file.\n+\t */\n+\tprotected void readPackedRefs(final Map avail)\n+\t\t\tthrows TransportException {\n+\t\ttry {\n+\t\t\tfinal BufferedReader br = openReader(PACKED_REFS);\n+\t\t\ttry {\n+\t\t\t\treadPackedRefsImpl(avail, br);\n+\t\t\t} finally {\n+\t\t\t\tbr.close();\n+\t\t\t}\n+\t\t} catch (FileNotFoundException notPacked) {\n+\t\t\t// Perhaps it wasn't worthwhile, or is just an older repository.\n+\t\t} catch (IOException e) {\n+\t\t\tthrow new TransportException(getURI(), \"error in packed-refs\", e);\n+\t\t}\n+\t}\n+\n+\tprivate void readPackedRefsImpl(final Map avail,\n+\t\t\tfinal BufferedReader br) throws IOException {\n+\t\tRef last = null;\n+\t\tfor (;;) {\n+\t\t\tString line = br.readLine();\n+\t\t\tif (line == null)\n+\t\t\t\tbreak;\n+\t\t\tif (line.charAt(0) == '#')\n+\t\t\t\tcontinue;\n+\t\t\tif (line.charAt(0) == '^') {\n+\t\t\t\tif (last == null)\n+\t\t\t\t\tthrow new TransportException(\"Peeled line before ref.\");\n+\t\t\t\tfinal ObjectId id = ObjectId.fromString(line + 1);\n+\t\t\t\tlast = new Ref(Ref.Storage.PACKED, last.getName(), last\n+\t\t\t\t\t\t.getObjectId(), id);\n+\t\t\t\tavail.put(last.getName(), last);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tfinal int sp = line.indexOf(' ');\n+\t\t\tif (sp < 0)\n+\t\t\t\tthrow new TransportException(\"Unrecognized ref: \" + line);\n+\t\t\tfinal ObjectId id = ObjectId.fromString(line.substring(0, sp));\n+\t\t\tfinal String name = line.substring(sp + 1);\n+\t\t\tlast = new Ref(Ref.Storage.PACKED, name, id);\n+\t\t\tavail.put(last.getName(), last);\n+\t\t}\n+\t}\n+\n \tstatic final class FileStream {\n \t\tfinal InputStream in;\n \n-- \n1.5.6.74.g8a5e\n"},{"id":"81615","messageId":"1214726371-93520-16-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-15-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 15/21] Specialized byte array output stream for large files","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:25Z","receivedAt":"2008-06-29T07:59:25Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Some transports may require that we know the total byte count (and\nperhaps MD5 checksum) of a pack file before we can send it to the\ntransport during a push operation. Materializing the pack locally\nprior to transfer can be somewhat costly, but may be able to be in\ncore for very small packs.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/util/TemporaryBuffer.java | 260 ++++++++++++++++++++\n 1 files changed, 260 insertions(+), 0 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/util/TemporaryBuffer.java\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/util/TemporaryBuffer.java b/org.spearce.jgit/src/org/spearce/jgit/util/TemporaryBuffer.java\nnew file mode 100644\nindex 0000000..72bdbb1\n--- /dev/null\n+++ b/org.spearce.jgit/src/org/spearce/jgit/util/TemporaryBuffer.java\n@@ -0,0 +1,260 @@\n+/*\n+ * Copyright (C) 2008, Shawn O. Pearce \n+ *\n+ * All rights reserved.\n+ *\n+ * Redistribution and use in source and binary forms, with or\n+ * without modification, are permitted provided that the following\n+ * conditions are met:\n+ *\n+ * - Redistributions of source code must retain the above copyright\n+ * notice, this list of conditions and the following disclaimer.\n+ *\n+ * - Redistributions in binary form must reproduce the above\n+ * copyright notice, this list of conditions and the following\n+ * disclaimer in the documentation and/or other materials provided\n+ * with the distribution.\n+ *\n+ * - Neither the name of the Git Development Community nor the\n+ * names of its contributors may be used to endorse or promote\n+ * products derived from this software without specific prior\n+ * written permission.\n+ *\n+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND\n+ * CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES,\n+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT\n+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\n+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+ */\n+\n+package org.spearce.jgit.util;\n+\n+import java.io.BufferedOutputStream;\n+import java.io.File;\n+import java.io.FileInputStream;\n+import java.io.FileOutputStream;\n+import java.io.IOException;\n+import java.io.OutputStream;\n+import java.util.ArrayList;\n+\n+import org.spearce.jgit.lib.NullProgressMonitor;\n+import org.spearce.jgit.lib.ProgressMonitor;\n+\n+/**\n+ * A fully buffered output stream using local disk storage for large data.\n+ *

\n+ * Initially this output stream buffers to memory, like ByteArrayOutputStream\n+ * might do, but it shifts to using an on disk temporary file if the output gets\n+ * too large.\n+ *

\n+ * The content of this buffered stream may be sent to another OutputStream only\n+ * after this stream has been properly closed by {@link #close()}.\n+ */\n+public class TemporaryBuffer extends OutputStream {\n+\tprivate static final int DEFAULT_IN_CORE_LIMIT = 1024 * 1024;\n+\n+\t/** Chain of data, if we are still completely in-core; otherwise null. */\n+\tprivate ArrayList blocks;\n+\n+\t/**\n+\t * Maximum number of bytes we will permit storing in memory.\n+\t *

\n+\t * When this limit is reached the data will be shifted to a file on disk,\n+\t * preventing the JVM heap from growing out of control.\n+\t */\n+\tprivate int inCoreLimit;\n+\n+\t/**\n+\t * Location of our temporary file if we are on disk; otherwise null.\n+\t *

\n+\t * If we exceeded the {@link #inCoreLimit} we nulled out {@link #blocks} and\n+\t * created this file instead. All output goes here through {@link #diskOut}.\n+\t */\n+\tprivate File onDiskFile;\n+\n+\t/** If writing to {@link #onDiskFile} this is a buffered stream to it. */\n+\tprivate OutputStream diskOut;\n+\n+\t/** Create a new empty temporary buffer. */\n+\tpublic TemporaryBuffer() {\n+\t\tinCoreLimit = DEFAULT_IN_CORE_LIMIT;\n+\t\tblocks = new ArrayList(inCoreLimit / Block.SZ);\n+\t\tblocks.add(new Block());\n+\t}\n+\n+\t@Override\n+\tpublic void write(final int b) throws IOException {\n+\t\tif (blocks == null) {\n+\t\t\tdiskOut.write(b);\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tBlock s = last();\n+\t\tif (s.isFull()) {\n+\t\t\tif (reachedInCoreLimit()) {\n+\t\t\t\tdiskOut.write(b);\n+\t\t\t\treturn;\n+\t\t\t}\n+\n+\t\t\ts = new Block();\n+\t\t\tblocks.add(s);\n+\t\t}\n+\t\ts.buffer[s.count++] = (byte) b;\n+\t}\n+\n+\t@Override\n+\tpublic void write(final byte[] b, int off, int len) throws IOException {\n+\t\tif (blocks != null) {\n+\t\t\twhile (len > 0) {\n+\t\t\t\tBlock s = last();\n+\t\t\t\tif (s.isFull()) {\n+\t\t\t\t\tif (reachedInCoreLimit())\n+\t\t\t\t\t\tbreak;\n+\n+\t\t\t\t\ts = new Block();\n+\t\t\t\t\tblocks.add(s);\n+\t\t\t\t}\n+\n+\t\t\t\tfinal int n = Math.min(Block.SZ - s.count, len);\n+\t\t\t\tSystem.arraycopy(b, off, s.buffer, s.count, n);\n+\t\t\t\ts.count += n;\n+\t\t\t\tlen -= n;\n+\t\t\t\toff += n;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (len > 0)\n+\t\t\tdiskOut.write(b, off, len);\n+\t}\n+\n+\tprivate Block last() {\n+\t\treturn blocks.get(blocks.size() - 1);\n+\t}\n+\n+\tprivate boolean reachedInCoreLimit() throws IOException {\n+\t\tif (blocks.size() * Block.SZ < inCoreLimit)\n+\t\t\treturn false;\n+\n+\t\tonDiskFile = File.createTempFile(\"jgit_\", \".buffer\");\n+\t\tdiskOut = new FileOutputStream(onDiskFile);\n+\n+\t\tfinal Block last = blocks.remove(blocks.size() - 1);\n+\t\tfor (final Block b : blocks)\n+\t\t\tdiskOut.write(b.buffer, 0, b.count);\n+\t\tblocks = null;\n+\n+\t\tdiskOut = new BufferedOutputStream(diskOut, Block.SZ);\n+\t\tdiskOut.write(last.buffer, 0, last.count);\n+\t\treturn true;\n+\t}\n+\n+\tpublic void close() throws IOException {\n+\t\tif (diskOut != null) {\n+\t\t\ttry {\n+\t\t\t\tdiskOut.close();\n+\t\t\t} finally {\n+\t\t\t\tdiskOut = null;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\t/**\n+\t * Obtain the length (in bytes) of the buffer.\n+\t *

\n+\t * The length is only accurate after {@link #close()} has been invoked.\n+\t * \n+\t * @return total length of the buffer, in bytes.\n+\t */\n+\tpublic long length() {\n+\t\tif (onDiskFile != null)\n+\t\t\treturn onDiskFile.length();\n+\n+\t\tfinal Block last = last();\n+\t\treturn ((long) blocks.size()) * Block.SZ - (Block.SZ - last.count);\n+\t}\n+\n+\t/**\n+\t * Send this buffer to an output stream.\n+\t *

\n+\t * This method may only be invoked after {@link #close()} has completed\n+\t * normally, to ensure all data is completely transferred.\n+\t * \n+\t * @param os\n+\t * stream to send this buffer's complete content to.\n+\t * @param pm\n+\t * if not null progress updates are sent here. Caller should\n+\t * initialize the task and the number of work units to\n+\t * {@link #length()}/1024.\n+\t * @throws IOException\n+\t * an error occurred reading from a temporary file on the local\n+\t * system, or writing to the output stream.\n+\t */\n+\tpublic void writeTo(final OutputStream os, ProgressMonitor pm)\n+\t\t\tthrows IOException {\n+\t\tif (pm == null)\n+\t\t\tpm = new NullProgressMonitor();\n+\t\tif (blocks != null) {\n+\t\t\t// Everything is in core so we can stream directly to the output.\n+\t\t\t//\n+\t\t\tfor (final Block b : blocks) {\n+\t\t\t\tos.write(b.buffer, 0, b.count);\n+\t\t\t\tpm.update(b.count / 1024);\n+\t\t\t}\n+\t\t} else {\n+\t\t\t// Reopen the temporary file and copy the contents.\n+\t\t\t//\n+\t\t\tfinal FileInputStream in = new FileInputStream(onDiskFile);\n+\t\t\ttry {\n+\t\t\t\tint cnt;\n+\t\t\t\tfinal byte[] buf = new byte[Block.SZ];\n+\t\t\t\twhile ((cnt = in.read(buf)) >= 0) {\n+\t\t\t\t\tos.write(buf, 0, cnt);\n+\t\t\t\t\tpm.update(cnt / 1024);\n+\t\t\t\t}\n+\t\t\t} finally {\n+\t\t\t\tin.close();\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\t/** Clear this buffer so it has no data, and cannot be used again. */\n+\tpublic void destroy() {\n+\t\tblocks = null;\n+\n+\t\tif (diskOut != null) {\n+\t\t\ttry {\n+\t\t\t\tdiskOut.close();\n+\t\t\t} catch (IOException err) {\n+\t\t\t\t// We shouldn't encounter an error closing the file.\n+\t\t\t} finally {\n+\t\t\t\tdiskOut = null;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (onDiskFile != null) {\n+\t\t\tif (!onDiskFile.delete())\n+\t\t\t\tonDiskFile.deleteOnExit();\n+\t\t\tonDiskFile = null;\n+\t\t}\n+\t}\n+\n+\tprivate static class Block {\n+\t\tstatic final int SZ = 8 * 1024;\n+\n+\t\tfinal byte[] buffer = new byte[SZ];\n+\n+\t\tint count;\n+\n+\t\tboolean isFull() {\n+\t\t\treturn count == SZ;\n+\t\t}\n+\t}\n+}\n-- \n1.5.6.74.g8a5e\n"},{"id":"81617","messageId":"1214726371-93520-17-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-16-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 16/21] Add Robert Harder's public domain Base64 encoding utility","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:26Z","receivedAt":"2008-06-29T07:59:26Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"From: Robert Harder \n\nSome transports require Base64 encoding of certain data fields, such\nas the \"encryption\" used in HTTP basic authentication header fields.\n\nThe wise people at Sun have never included the incredibly widely used\nBase64 encoding/decoding algorithms as part of the base JRE, forcing\neveryone to include their own library. Fortunately for us Robert\nHarder distributes one in the public domain.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/util/Base64.java | 1459 ++++++++++++++++++++\n 1 files changed, 1459 insertions(+), 0 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/util/Base64.java\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/util/Base64.java b/org.spearce.jgit/src/org/spearce/jgit/util/Base64.java\nnew file mode 100644\nindex 0000000..b0c19b6\n--- /dev/null\n+++ b/org.spearce.jgit/src/org/spearce/jgit/util/Base64.java\n@@ -0,0 +1,1459 @@\n+//\n+// NOTE: The following source code is the iHarder.net public domain\n+// Base64 library and is provided here as a convenience. For updates,\n+// problems, questions, etc. regarding this code, please visit:\n+// http://iharder.sourceforge.net/current/java/base64/\n+//\n+\n+package org.spearce.jgit.util;\n+\n+\n+/**\n+ * Encodes and decodes to and from Base64 notation.\n+ *\n+ *

\n+ * Change Log:\n+ *

\n+ *
    \n+ *
  • v2.1 - Cleaned up javadoc comments and unused variables and methods. Added\n+ * some convenience methods for reading and writing to and from files.
    • \n+ *
  • v2.0.2 - Now specifies UTF-8 encoding in places where the code fails on systems\n+ * with other encodings (like EBCDIC).
    • \n+ *
  • v2.0.1 - Fixed an error when decoding a single byte, that is, when the\n+ * encoded data was a single byte.
    • \n+ *
  • v2.0 - I got rid of methods that used booleans to set options. \n+ * Now everything is more consolidated and cleaner. The code now detects\n+ * when data that's being decoded is gzip-compressed and will decompress it\n+ * automatically. Generally things are cleaner. You'll probably have to\n+ * change some method calls that you were making to support the new\n+ * options format (ints that you \"OR\" together).
    • \n+ *
  • v1.5.1 - Fixed bug when decompressing and decoding to a \n+ * byte[] using decode( String s, boolean gzipCompressed ). \n+ * Added the ability to \"suspend\" encoding in the Output Stream so \n+ * you can turn on and off the encoding if you need to embed base64 \n+ * data in an otherwise \"normal\" stream (like an XML file).
    • \n+ *
  • v1.5 - Output stream pases on flush() command but doesn't do anything itself.\n+ * This helps when using GZIP streams.\n+ * Added the ability to GZip-compress objects before encoding them.
    • \n+ *
  • v1.4 - Added helper methods to read/write files.
    • \n+ *
  • v1.3.6 - Fixed OutputStream.flush() so that 'position' is reset.
    • \n+ *
  • v1.3.5 - Added flag to turn on and off line breaks. Fixed bug in input stream\n+ * where last buffer being read, if not completely full, was not returned.
    • \n+ *
  • v1.3.4 - Fixed when \"improperly padded stream\" error was thrown at the wrong time.
    • \n+ *
  • v1.3.3 - Fixed I/O streams which were totally messed up.
    • \n+ *
\n+ *\n+ *

\n+ * I am placing this code in the Public Domain. Do with it as you will.\n+ * This software comes with no guarantees or warranties but with\n+ * plenty of well-wishing instead!\n+ * Please visit http://iharder.net/base64\n+ * periodically to check for updates or to contribute improvements.\n+ *

\n+ *\n+ * @author Robert Harder\n+ * @author rob@iharder.net\n+ * @version 2.1\n+ */\n+public class Base64\n+{\n+ \n+/* ******** P U B L I C F I E L D S ******** */ \n+ \n+ \n+ /** No options specified. Value is zero. */\n+ public final static int NO_OPTIONS = 0;\n+ \n+ /** Specify encoding. */\n+ public final static int ENCODE = 1;\n+ \n+ \n+ /** Specify decoding. */\n+ public final static int DECODE = 0;\n+ \n+ \n+ /** Specify that data should be gzip-compressed. */\n+ public final static int GZIP = 2;\n+ \n+ \n+ /** Don't break lines when encoding (violates strict Base64 specification) */\n+ public final static int DONT_BREAK_LINES = 8;\n+ \n+ \n+/* ******** P R I V A T E F I E L D S ******** */ \n+ \n+ \n+ /** Maximum line length (76) of Base64 output. */\n+ private final static int MAX_LINE_LENGTH = 76;\n+ \n+ \n+ /** The equals sign (=) as a byte. */\n+ private final static byte EQUALS_SIGN = (byte)'=';\n+ \n+ \n+ /** The new line character (\\n) as a byte. */\n+ private final static byte NEW_LINE = (byte)'\\n';\n+ \n+ \n+ /** Preferred encoding. */\n+ private final static String PREFERRED_ENCODING = \"UTF-8\";\n+ \n+ \n+ /** The 64 valid Base64 values. */\n+ private final static byte[] ALPHABET;\n+ private final static byte[] _NATIVE_ALPHABET = /* May be something funny like EBCDIC */\n+ {\n+ (byte)'A', (byte)'B', (byte)'C', (byte)'D', (byte)'E', (byte)'F', (byte)'G',\n+ (byte)'H', (byte)'I', (byte)'J', (byte)'K', (byte)'L', (byte)'M', (byte)'N',\n+ (byte)'O', (byte)'P', (byte)'Q', (byte)'R', (byte)'S', (byte)'T', (byte)'U', \n+ (byte)'V', (byte)'W', (byte)'X', (byte)'Y', (byte)'Z',\n+ (byte)'a', (byte)'b', (byte)'c', (byte)'d', (byte)'e', (byte)'f', (byte)'g',\n+ (byte)'h', (byte)'i', (byte)'j', (byte)'k', (byte)'l', (byte)'m', (byte)'n',\n+ (byte)'o', (byte)'p', (byte)'q', (byte)'r', (byte)'s', (byte)'t', (byte)'u', \n+ (byte)'v', (byte)'w', (byte)'x', (byte)'y', (byte)'z',\n+ (byte)'0', (byte)'1', (byte)'2', (byte)'3', (byte)'4', (byte)'5', \n+ (byte)'6', (byte)'7', (byte)'8', (byte)'9', (byte)'+', (byte)'/'\n+ };\n+ \n+ /** Determine which ALPHABET to use. */\n+ static\n+ {\n+ byte[] __bytes;\n+ try\n+ {\n+ __bytes = \"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/\".getBytes( PREFERRED_ENCODING );\n+ } // end try\n+ catch (java.io.UnsupportedEncodingException use)\n+ {\n+ __bytes = _NATIVE_ALPHABET; // Fall back to native encoding\n+ } // end catch\n+ ALPHABET = __bytes;\n+ } // end static\n+ \n+ \n+ /** \n+ * Translates a Base64 value to either its 6-bit reconstruction value\n+ * or a negative number indicating some other meaning.\n+ **/\n+ private final static byte[] DECODABET =\n+ { \n+ -9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 0 - 8\n+ -5,-5, // Whitespace: Tab and Linefeed\n+ -9,-9, // Decimal 11 - 12\n+ -5, // Whitespace: Carriage Return\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 14 - 26\n+ -9,-9,-9,-9,-9, // Decimal 27 - 31\n+ -5, // Whitespace: Space\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 33 - 42\n+ 62, // Plus sign at decimal 43\n+ -9,-9,-9, // Decimal 44 - 46\n+ 63, // Slash at decimal 47\n+ 52,53,54,55,56,57,58,59,60,61, // Numbers zero through nine\n+ -9,-9,-9, // Decimal 58 - 60\n+ -1, // Equals sign at decimal 61\n+ -9,-9,-9, // Decimal 62 - 64\n+ 0,1,2,3,4,5,6,7,8,9,10,11,12,13, // Letters 'A' through 'N'\n+ 14,15,16,17,18,19,20,21,22,23,24,25, // Letters 'O' through 'Z'\n+ -9,-9,-9,-9,-9,-9, // Decimal 91 - 96\n+ 26,27,28,29,30,31,32,33,34,35,36,37,38, // Letters 'a' through 'm'\n+ 39,40,41,42,43,44,45,46,47,48,49,50,51, // Letters 'n' through 'z'\n+ -9,-9,-9,-9 // Decimal 123 - 126\n+ /*,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 127 - 139\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 140 - 152\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 153 - 165\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 166 - 178\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 179 - 191\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 192 - 204\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 205 - 217\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 218 - 230\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9, // Decimal 231 - 243\n+ -9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9,-9 // Decimal 244 - 255 */\n+ };\n+ \n+ // I think I end up not using the BAD_ENCODING indicator.\n+ //private final static byte BAD_ENCODING = -9; // Indicates error in encoding\n+ private final static byte WHITE_SPACE_ENC = -5; // Indicates white space in encoding\n+ private final static byte EQUALS_SIGN_ENC = -1; // Indicates equals sign in encoding\n+\n+ \n+ /** Defeats instantiation. */\n+ private Base64(){}\n+ \n+ \n+ \n+/* ******** E N C O D I N G M E T H O D S ******** */ \n+ \n+ \n+ /**\n+ * Encodes up to the first three bytes of array threeBytes\n+ * and returns a four-byte array in Base64 notation.\n+ * The actual number of significant bytes in your array is\n+ * given by numSigBytes.\n+ * The array threeBytes needs only be as big as\n+ * numSigBytes.\n+ * Code can reuse a byte array by passing a four-byte array as b4.\n+ *\n+ * @param b4 A reusable byte array to reduce array instantiation\n+ * @param threeBytes the array to convert\n+ * @param numSigBytes the number of significant bytes in your array\n+ * @return four byte array in Base64 notation.\n+ * @since 1.5.1\n+ */\n+ private static byte[] encode3to4( byte[] b4, byte[] threeBytes, int numSigBytes )\n+ {\n+ encode3to4( threeBytes, 0, numSigBytes, b4, 0 );\n+ return b4;\n+ } // end encode3to4\n+\n+ \n+ /**\n+ * Encodes up to three bytes of the array source\n+ * and writes the resulting four Base64 bytes to destination.\n+ * The source and destination arrays can be manipulated\n+ * anywhere along their length by specifying \n+ * srcOffset and destOffset.\n+ * This method does not check to make sure your arrays\n+ * are large enough to accomodate srcOffset + 3 for\n+ * the source array or destOffset + 4 for\n+ * the destination array.\n+ * The actual number of significant bytes in your array is\n+ * given by numSigBytes.\n+ *\n+ * @param source the array to convert\n+ * @param srcOffset the index where conversion begins\n+ * @param numSigBytes the number of significant bytes in your array\n+ * @param destination the array to hold the conversion\n+ * @param destOffset the index where output will be put\n+ * @return the destination array\n+ * @since 1.3\n+ */\n+ private static byte[] encode3to4( \n+ byte[] source, int srcOffset, int numSigBytes,\n+ byte[] destination, int destOffset )\n+ {\n+ // 1 2 3 \n+ // 01234567890123456789012345678901 Bit position\n+ // --------000000001111111122222222 Array position from threeBytes\n+ // --------| || || || | Six bit groups to index ALPHABET\n+ // >>18 >>12 >> 6 >> 0 Right shift necessary\n+ // 0x3f 0x3f 0x3f Additional AND\n+ \n+ // Create buffer with zero-padding if there are only one or two\n+ // significant bytes passed in the array.\n+ // We have to shift left 24 in order to flush out the 1's that appear\n+ // when Java treats a value as negative that is cast from a byte to an int.\n+ int inBuff = ( numSigBytes > 0 ? ((source[ srcOffset ] << 24) >>> 8) : 0 )\n+ | ( numSigBytes > 1 ? ((source[ srcOffset + 1 ] << 24) >>> 16) : 0 )\n+ | ( numSigBytes > 2 ? ((source[ srcOffset + 2 ] << 24) >>> 24) : 0 );\n+\n+ switch( numSigBytes )\n+ {\n+ case 3:\n+ destination[ destOffset ] = ALPHABET[ (inBuff >>> 18) ];\n+ destination[ destOffset + 1 ] = ALPHABET[ (inBuff >>> 12) & 0x3f ];\n+ destination[ destOffset + 2 ] = ALPHABET[ (inBuff >>> 6) & 0x3f ];\n+ destination[ destOffset + 3 ] = ALPHABET[ (inBuff ) & 0x3f ];\n+ return destination;\n+ \n+ case 2:\n+ destination[ destOffset ] = ALPHABET[ (inBuff >>> 18) ];\n+ destination[ destOffset + 1 ] = ALPHABET[ (inBuff >>> 12) & 0x3f ];\n+ destination[ destOffset + 2 ] = ALPHABET[ (inBuff >>> 6) & 0x3f ];\n+ destination[ destOffset + 3 ] = EQUALS_SIGN;\n+ return destination;\n+ \n+ case 1:\n+ destination[ destOffset ] = ALPHABET[ (inBuff >>> 18) ];\n+ destination[ destOffset + 1 ] = ALPHABET[ (inBuff >>> 12) & 0x3f ];\n+ destination[ destOffset + 2 ] = EQUALS_SIGN;\n+ destination[ destOffset + 3 ] = EQUALS_SIGN;\n+ return destination;\n+ \n+ default:\n+ return destination;\n+ } // end switch\n+ } // end encode3to4\n+ \n+ \n+ \n+ /**\n+ * Serializes an object and returns the Base64-encoded\n+ * version of that serialized object. If the object\n+ * cannot be serialized or there is another error,\n+ * the method will return null.\n+ * The object is not GZip-compressed before being encoded.\n+ *\n+ * @param serializableObject The object to encode\n+ * @return The Base64-encoded object\n+ * @since 1.4\n+ */\n+ public static String encodeObject( java.io.Serializable serializableObject )\n+ {\n+ return encodeObject( serializableObject, NO_OPTIONS );\n+ } // end encodeObject\n+ \n+\n+\n+ /**\n+ * Serializes an object and returns the Base64-encoded\n+ * version of that serialized object. If the object\n+ * cannot be serialized or there is another error,\n+ * the method will return null.\n+ *

\n+ * Valid options:

\n+     *   GZIP: gzip-compresses object before encoding it.\n+     *   DONT_BREAK_LINES: don't break lines at 76 characters\n+     *     Note: Technically, this makes your encoding non-compliant.\n+     *
\n+ *

\n+ * Example: encodeObject( myObj, Base64.GZIP ) or\n+ *

\n+ * Example: encodeObject( myObj, Base64.GZIP | Base64.DONT_BREAK_LINES )\n+ *\n+ * @param serializableObject The object to encode\n+ * @param options Specified options\n+ * @return The Base64-encoded object\n+ * @see Base64#GZIP\n+ * @see Base64#DONT_BREAK_LINES\n+ * @since 2.0\n+ */\n+ public static String encodeObject( java.io.Serializable serializableObject, int options )\n+ {\n+ // Streams\n+ java.io.ByteArrayOutputStream baos = null; \n+ java.io.OutputStream b64os = null; \n+ java.io.ObjectOutputStream oos = null; \n+ java.util.zip.GZIPOutputStream gzos = null;\n+ \n+ // Isolate options\n+ int gzip = (options & GZIP);\n+ int dontBreakLines = (options & DONT_BREAK_LINES);\n+ \n+ try\n+ {\n+ // ObjectOutputStream -> (GZIP) -> Base64 -> ByteArrayOutputStream\n+ baos = new java.io.ByteArrayOutputStream();\n+ b64os = new Base64.OutputStream( baos, ENCODE | dontBreakLines );\n+ \n+ // GZip?\n+ if( gzip == GZIP )\n+ {\n+ gzos = new java.util.zip.GZIPOutputStream( b64os );\n+ oos = new java.io.ObjectOutputStream( gzos );\n+ } // end if: gzip\n+ else\n+ oos = new java.io.ObjectOutputStream( b64os );\n+ \n+ oos.writeObject( serializableObject );\n+ } // end try\n+ catch( java.io.IOException e )\n+ {\n+ e.printStackTrace();\n+ return null;\n+ } // end catch\n+ finally\n+ {\n+ try{ oos.close(); } catch( Exception e ){}\n+ try{ gzos.close(); } catch( Exception e ){}\n+ try{ b64os.close(); } catch( Exception e ){}\n+ try{ baos.close(); } catch( Exception e ){}\n+ } // end finally\n+ \n+ // Return value according to relevant encoding.\n+ try \n+ {\n+ return new String( baos.toByteArray(), PREFERRED_ENCODING );\n+ } // end try\n+ catch (java.io.UnsupportedEncodingException uue)\n+ {\n+ return new String( baos.toByteArray() );\n+ } // end catch\n+ \n+ } // end encode\n+ \n+ \n+\n+ /**\n+ * Encodes a byte array into Base64 notation.\n+ * Does not GZip-compress data.\n+ *\n+ * @param source The data to convert\n+ * @since 1.4\n+ */\n+ public static String encodeBytes( byte[] source )\n+ {\n+ return encodeBytes( source, 0, source.length, NO_OPTIONS );\n+ } // end encodeBytes\n+ \n+\n+\n+ /**\n+ * Encodes a byte array into Base64 notation.\n+ *

\n+ * Valid options:

\n+     *   GZIP: gzip-compresses object before encoding it.\n+     *   DONT_BREAK_LINES: don't break lines at 76 characters\n+     *     Note: Technically, this makes your encoding non-compliant.\n+     *
\n+ *

\n+ * Example: encodeBytes( myData, Base64.GZIP ) or\n+ *

\n+ * Example: encodeBytes( myData, Base64.GZIP | Base64.DONT_BREAK_LINES )\n+ *\n+ *\n+ * @param source The data to convert\n+ * @param options Specified options\n+ * @see Base64#GZIP\n+ * @see Base64#DONT_BREAK_LINES\n+ * @since 2.0\n+ */\n+ public static String encodeBytes( byte[] source, int options )\n+ { \n+ return encodeBytes( source, 0, source.length, options );\n+ } // end encodeBytes\n+ \n+ \n+ /**\n+ * Encodes a byte array into Base64 notation.\n+ * Does not GZip-compress data.\n+ *\n+ * @param source The data to convert\n+ * @param off Offset in array where conversion should begin\n+ * @param len Length of data to convert\n+ * @since 1.4\n+ */\n+ public static String encodeBytes( byte[] source, int off, int len )\n+ {\n+ return encodeBytes( source, off, len, NO_OPTIONS );\n+ } // end encodeBytes\n+ \n+ \n+\n+ /**\n+ * Encodes a byte array into Base64 notation.\n+ *

\n+ * Valid options:

\n+     *   GZIP: gzip-compresses object before encoding it.\n+     *   DONT_BREAK_LINES: don't break lines at 76 characters\n+     *     Note: Technically, this makes your encoding non-compliant.\n+     *
\n+ *

\n+ * Example: encodeBytes( myData, Base64.GZIP ) or\n+ *

\n+ * Example: encodeBytes( myData, Base64.GZIP | Base64.DONT_BREAK_LINES )\n+ *\n+ *\n+ * @param source The data to convert\n+ * @param off Offset in array where conversion should begin\n+ * @param len Length of data to convert\n+ * @param options Specified options\n+ * @see Base64#GZIP\n+ * @see Base64#DONT_BREAK_LINES\n+ * @since 2.0\n+ */\n+ public static String encodeBytes( byte[] source, int off, int len, int options )\n+ {\n+ // Isolate options\n+ int dontBreakLines = ( options & DONT_BREAK_LINES );\n+ int gzip = ( options & GZIP );\n+ \n+ // Compress?\n+ if( gzip == GZIP )\n+ {\n+ java.io.ByteArrayOutputStream baos = null;\n+ java.util.zip.GZIPOutputStream gzos = null;\n+ Base64.OutputStream b64os = null;\n+ \n+ \n+ try\n+ {\n+ // GZip -> Base64 -> ByteArray\n+ baos = new java.io.ByteArrayOutputStream();\n+ b64os = new Base64.OutputStream( baos, ENCODE | dontBreakLines );\n+ gzos = new java.util.zip.GZIPOutputStream( b64os ); \n+ \n+ gzos.write( source, off, len );\n+ gzos.close();\n+ } // end try\n+ catch( java.io.IOException e )\n+ {\n+ e.printStackTrace();\n+ return null;\n+ } // end catch\n+ finally\n+ {\n+ try{ gzos.close(); } catch( Exception e ){}\n+ try{ b64os.close(); } catch( Exception e ){}\n+ try{ baos.close(); } catch( Exception e ){}\n+ } // end finally\n+\n+ // Return value according to relevant encoding.\n+ try\n+ {\n+ return new String( baos.toByteArray(), PREFERRED_ENCODING );\n+ } // end try\n+ catch (java.io.UnsupportedEncodingException uue)\n+ {\n+ return new String( baos.toByteArray() );\n+ } // end catch\n+ } // end if: compress\n+ \n+ // Else, don't compress. Better not to use streams at all then.\n+ else\n+ {\n+ // Convert option to boolean in way that code likes it.\n+ boolean breakLines = dontBreakLines == 0;\n+ \n+ int len43 = len * 4 / 3;\n+ byte[] outBuff = new byte[ ( len43 ) // Main 4:3\n+ + ( (len % 3) > 0 ? 4 : 0 ) // Account for padding\n+ + (breakLines ? ( len43 / MAX_LINE_LENGTH ) : 0) ]; // New lines \n+ int d = 0;\n+ int e = 0;\n+ int len2 = len - 2;\n+ int lineLength = 0;\n+ for( ; d < len2; d+=3, e+=4 )\n+ {\n+ encode3to4( source, d+off, 3, outBuff, e );\n+\n+ lineLength += 4;\n+ if( breakLines && lineLength == MAX_LINE_LENGTH )\n+ { \n+ outBuff[e+4] = NEW_LINE;\n+ e++;\n+ lineLength = 0;\n+ } // end if: end of line\n+ } // en dfor: each piece of array\n+\n+ if( d < len )\n+ {\n+ encode3to4( source, d+off, len - d, outBuff, e );\n+ e += 4;\n+ } // end if: some padding needed\n+\n+ \n+ // Return value according to relevant encoding.\n+ try\n+ {\n+ return new String( outBuff, 0, e, PREFERRED_ENCODING );\n+ } // end try\n+ catch (java.io.UnsupportedEncodingException uue)\n+ {\n+ return new String( outBuff, 0, e );\n+ } // end catch\n+ \n+ } // end else: don't compress\n+ \n+ } // end encodeBytes\n+ \n+\n+ \n+ \n+ \n+/* ******** D E C O D I N G M E T H O D S ******** */\n+ \n+ \n+ /**\n+ * Decodes four bytes from array source\n+ * and writes the resulting bytes (up to three of them)\n+ * to destination.\n+ * The source and destination arrays can be manipulated\n+ * anywhere along their length by specifying \n+ * srcOffset and destOffset.\n+ * This method does not check to make sure your arrays\n+ * are large enough to accomodate srcOffset + 4 for\n+ * the source array or destOffset + 3 for\n+ * the destination array.\n+ * This method returns the actual number of bytes that \n+ * were converted from the Base64 encoding.\n+ * \n+ *\n+ * @param source the array to convert\n+ * @param srcOffset the index where conversion begins\n+ * @param destination the array to hold the conversion\n+ * @param destOffset the index where output will be put\n+ * @return the number of decoded bytes converted\n+ * @since 1.3\n+ */\n+ private static int decode4to3( byte[] source, int srcOffset, byte[] destination, int destOffset )\n+ {\n+ // Example: Dk==\n+ if( source[ srcOffset + 2] == EQUALS_SIGN )\n+ {\n+ // Two ways to do the same thing. Don't know which way I like best.\n+ //int outBuff = ( ( DECODABET[ source[ srcOffset ] ] << 24 ) >>> 6 )\n+ // | ( ( DECODABET[ source[ srcOffset + 1] ] << 24 ) >>> 12 );\n+ int outBuff = ( ( DECODABET[ source[ srcOffset ] ] & 0xFF ) << 18 )\n+ | ( ( DECODABET[ source[ srcOffset + 1] ] & 0xFF ) << 12 );\n+ \n+ destination[ destOffset ] = (byte)( outBuff >>> 16 );\n+ return 1;\n+ }\n+ \n+ // Example: DkL=\n+ else if( source[ srcOffset + 3 ] == EQUALS_SIGN )\n+ {\n+ // Two ways to do the same thing. Don't know which way I like best.\n+ //int outBuff = ( ( DECODABET[ source[ srcOffset ] ] << 24 ) >>> 6 )\n+ // | ( ( DECODABET[ source[ srcOffset + 1 ] ] << 24 ) >>> 12 )\n+ // | ( ( DECODABET[ source[ srcOffset + 2 ] ] << 24 ) >>> 18 );\n+ int outBuff = ( ( DECODABET[ source[ srcOffset ] ] & 0xFF ) << 18 )\n+ | ( ( DECODABET[ source[ srcOffset + 1 ] ] & 0xFF ) << 12 )\n+ | ( ( DECODABET[ source[ srcOffset + 2 ] ] & 0xFF ) << 6 );\n+ \n+ destination[ destOffset ] = (byte)( outBuff >>> 16 );\n+ destination[ destOffset + 1 ] = (byte)( outBuff >>> 8 );\n+ return 2;\n+ }\n+ \n+ // Example: DkLE\n+ else\n+ {\n+ try{\n+ // Two ways to do the same thing. Don't know which way I like best.\n+ //int outBuff = ( ( DECODABET[ source[ srcOffset ] ] << 24 ) >>> 6 )\n+ // | ( ( DECODABET[ source[ srcOffset + 1 ] ] << 24 ) >>> 12 )\n+ // | ( ( DECODABET[ source[ srcOffset + 2 ] ] << 24 ) >>> 18 )\n+ // | ( ( DECODABET[ source[ srcOffset + 3 ] ] << 24 ) >>> 24 );\n+ int outBuff = ( ( DECODABET[ source[ srcOffset ] ] & 0xFF ) << 18 )\n+ | ( ( DECODABET[ source[ srcOffset + 1 ] ] & 0xFF ) << 12 )\n+ | ( ( DECODABET[ source[ srcOffset + 2 ] ] & 0xFF ) << 6)\n+ | ( ( DECODABET[ source[ srcOffset + 3 ] ] & 0xFF ) );\n+\n+ \n+ destination[ destOffset ] = (byte)( outBuff >> 16 );\n+ destination[ destOffset + 1 ] = (byte)( outBuff >> 8 );\n+ destination[ destOffset + 2 ] = (byte)( outBuff );\n+\n+ return 3;\n+ }catch( Exception e){\n+ System.out.println(\"\"+source[srcOffset]+ \": \" + ( DECODABET[ source[ srcOffset ] ] ) );\n+ System.out.println(\"\"+source[srcOffset+1]+ \": \" + ( DECODABET[ source[ srcOffset + 1 ] ] ) );\n+ System.out.println(\"\"+source[srcOffset+2]+ \": \" + ( DECODABET[ source[ srcOffset + 2 ] ] ) );\n+ System.out.println(\"\"+source[srcOffset+3]+ \": \" + ( DECODABET[ source[ srcOffset + 3 ] ] ) );\n+ return -1;\n+ } //e nd catch\n+ }\n+ } // end decodeToBytes\n+ \n+ \n+ \n+ \n+ /**\n+ * Very low-level access to decoding ASCII characters in\n+ * the form of a byte array. Does not support automatically\n+ * gunzipping or any other \"fancy\" features.\n+ *\n+ * @param source The Base64 encoded data\n+ * @param off The offset of where to begin decoding\n+ * @param len The length of characters to decode\n+ * @return decoded data\n+ * @since 1.3\n+ */\n+ public static byte[] decode( byte[] source, int off, int len )\n+ {\n+ int len34 = len * 3 / 4;\n+ byte[] outBuff = new byte[ len34 ]; // Upper limit on size of output\n+ int outBuffPosn = 0;\n+ \n+ byte[] b4 = new byte[4];\n+ int b4Posn = 0;\n+ int i = 0;\n+ byte sbiCrop = 0;\n+ byte sbiDecode = 0;\n+ for( i = off; i < off+len; i++ )\n+ {\n+ sbiCrop = (byte)(source[i] & 0x7f); // Only the low seven bits\n+ sbiDecode = DECODABET[ sbiCrop ];\n+ \n+ if( sbiDecode >= WHITE_SPACE_ENC ) // White space, Equals sign or better\n+ {\n+ if( sbiDecode >= EQUALS_SIGN_ENC )\n+ {\n+ b4[ b4Posn++ ] = sbiCrop;\n+ if( b4Posn > 3 )\n+ {\n+ outBuffPosn += decode4to3( b4, 0, outBuff, outBuffPosn );\n+ b4Posn = 0;\n+ \n+ // If that was the equals sign, break out of 'for' loop\n+ if( sbiCrop == EQUALS_SIGN )\n+ break;\n+ } // end if: quartet built\n+ \n+ } // end if: equals sign or better\n+ \n+ } // end if: white space, equals sign or better\n+ else\n+ {\n+ System.err.println( \"Bad Base64 input character at \" + i + \": \" + source[i] + \"(decimal)\" );\n+ return null;\n+ } // end else: \n+ } // each input character\n+ \n+ byte[] out = new byte[ outBuffPosn ];\n+ System.arraycopy( outBuff, 0, out, 0, outBuffPosn ); \n+ return out;\n+ } // end decode\n+ \n+ \n+ \n+ \n+ /**\n+ * Decodes data from Base64 notation, automatically\n+ * detecting gzip-compressed data and decompressing it.\n+ *\n+ * @param s the string to decode\n+ * @return the decoded data\n+ * @since 1.4\n+ */\n+ public static byte[] decode( String s )\n+ { \n+ byte[] bytes;\n+ try\n+ {\n+ bytes = s.getBytes( PREFERRED_ENCODING );\n+ } // end try\n+ catch( java.io.UnsupportedEncodingException uee )\n+ {\n+ bytes = s.getBytes();\n+ } // end catch\n+\t\t//\n+ \n+ // Decode\n+ bytes = decode( bytes, 0, bytes.length );\n+ \n+ \n+ // Check to see if it's gzip-compressed\n+ // GZIP Magic Two-Byte Number: 0x8b1f (35615)\n+ if( bytes != null && bytes.length >= 4 )\n+ {\n+ \n+ int head = ((int)bytes[0] & 0xff) | ((bytes[1] << 8) & 0xff00); \n+ if( java.util.zip.GZIPInputStream.GZIP_MAGIC == head ) \n+ {\n+ java.io.ByteArrayInputStream bais = null;\n+ java.util.zip.GZIPInputStream gzis = null;\n+ java.io.ByteArrayOutputStream baos = null;\n+ byte[] buffer = new byte[2048];\n+ int length = 0;\n+\n+ try\n+ {\n+ baos = new java.io.ByteArrayOutputStream();\n+ bais = new java.io.ByteArrayInputStream( bytes );\n+ gzis = new java.util.zip.GZIPInputStream( bais );\n+\n+ while( ( length = gzis.read( buffer ) ) >= 0 )\n+ {\n+ baos.write(buffer,0,length);\n+ } // end while: reading input\n+\n+ // No error? Get new bytes.\n+ bytes = baos.toByteArray();\n+\n+ } // end try\n+ catch( java.io.IOException e )\n+ {\n+ // Just return originally-decoded bytes\n+ } // end catch\n+ finally\n+ {\n+ try{ baos.close(); } catch( Exception e ){}\n+ try{ gzis.close(); } catch( Exception e ){}\n+ try{ bais.close(); } catch( Exception e ){}\n+ } // end finally\n+\n+ } // end if: gzipped\n+ } // end if: bytes.length >= 2\n+ \n+ return bytes;\n+ } // end decode\n+\n+\n+ \n+\n+ /**\n+ * Attempts to decode Base64 data and deserialize a Java\n+ * Object within. Returns null if there was an error.\n+ *\n+ * @param encodedObject The Base64 data to decode\n+ * @return The decoded and deserialized object\n+ * @since 1.5\n+ */\n+ public static Object decodeToObject( String encodedObject )\n+ {\n+ // Decode and gunzip if necessary\n+ byte[] objBytes = decode( encodedObject );\n+ \n+ java.io.ByteArrayInputStream bais = null;\n+ java.io.ObjectInputStream ois = null;\n+ Object obj = null;\n+ \n+ try\n+ {\n+ bais = new java.io.ByteArrayInputStream( objBytes );\n+ ois = new java.io.ObjectInputStream( bais );\n+ \n+ obj = ois.readObject();\n+ } // end try\n+ catch( java.io.IOException e )\n+ {\n+ e.printStackTrace();\n+ obj = null;\n+ } // end catch\n+ catch( java.lang.ClassNotFoundException e )\n+ {\n+ e.printStackTrace();\n+ obj = null;\n+ } // end catch\n+ finally\n+ {\n+ try{ bais.close(); } catch( Exception e ){}\n+ try{ ois.close(); } catch( Exception e ){}\n+ } // end finally\n+ \n+ return obj;\n+ } // end decodeObject\n+ \n+ \n+ \n+ /**\n+ * Convenience method for encoding data to a file.\n+ *\n+ * @param dataToEncode byte array of data to encode in base64 form\n+ * @param filename Filename for saving encoded data\n+ * @return true if successful, false otherwise\n+ *\n+ * @since 2.1\n+ */\n+ public static boolean encodeToFile( byte[] dataToEncode, String filename )\n+ {\n+ boolean success = false;\n+ Base64.OutputStream bos = null;\n+ try\n+ {\n+ bos = new Base64.OutputStream( \n+ new java.io.FileOutputStream( filename ), Base64.ENCODE );\n+ bos.write( dataToEncode );\n+ success = true;\n+ } // end try\n+ catch( java.io.IOException e )\n+ {\n+ \n+ success = false;\n+ } // end catch: IOException\n+ finally\n+ {\n+ try{ bos.close(); } catch( Exception e ){}\n+ } // end finally\n+ \n+ return success;\n+ } // end encodeToFile\n+ \n+ \n+ /**\n+ * Convenience method for decoding data to a file.\n+ *\n+ * @param dataToDecode Base64-encoded data as a string\n+ * @param filename Filename for saving decoded data\n+ * @return true if successful, false otherwise\n+ *\n+ * @since 2.1\n+ */\n+ public static boolean decodeToFile( String dataToDecode, String filename )\n+ {\n+ boolean success = false;\n+ Base64.OutputStream bos = null;\n+ try\n+ {\n+ bos = new Base64.OutputStream( \n+ new java.io.FileOutputStream( filename ), Base64.DECODE );\n+ bos.write( dataToDecode.getBytes( PREFERRED_ENCODING ) );\n+ success = true;\n+ } // end try\n+ catch( java.io.IOException e )\n+ {\n+ success = false;\n+ } // end catch: IOException\n+ finally\n+ {\n+ try{ bos.close(); } catch( Exception e ){}\n+ } // end finally\n+ \n+ return success;\n+ } // end decodeToFile\n+ \n+ \n+ \n+ \n+ /**\n+ * Convenience method for reading a base64-encoded\n+ * file and decoding it.\n+ *\n+ * @param filename Filename for reading encoded data\n+ * @return decoded byte array or null if unsuccessful\n+ *\n+ * @since 2.1\n+ */\n+ public static byte[] decodeFromFile( String filename )\n+ {\n+ byte[] decodedData = null;\n+ Base64.InputStream bis = null;\n+ try\n+ {\n+ // Set up some useful variables\n+ java.io.File file = new java.io.File( filename );\n+ byte[] buffer = null;\n+ int length = 0;\n+ int numBytes = 0;\n+ \n+ // Check for size of file\n+ if( file.length() > Integer.MAX_VALUE )\n+ {\n+ System.err.println( \"File is too big for this convenience method (\" + file.length() + \" bytes).\" );\n+ return null;\n+ } // end if: file too big for int index\n+ buffer = new byte[ (int)file.length() ];\n+ \n+ // Open a stream\n+ bis = new Base64.InputStream( \n+ new java.io.BufferedInputStream( \n+ new java.io.FileInputStream( file ) ), Base64.DECODE );\n+ \n+ // Read until done\n+ while( ( numBytes = bis.read( buffer, length, 4096 ) ) >= 0 )\n+ length += numBytes;\n+ \n+ // Save in a variable to return\n+ decodedData = new byte[ length ];\n+ System.arraycopy( buffer, 0, decodedData, 0, length );\n+ \n+ } // end try\n+ catch( java.io.IOException e )\n+ {\n+ System.err.println( \"Error decoding from file \" + filename );\n+ } // end catch: IOException\n+ finally\n+ {\n+ try{ bis.close(); } catch( Exception e) {}\n+ } // end finally\n+ \n+ return decodedData;\n+ } // end decodeFromFile\n+ \n+ \n+ \n+ /**\n+ * Convenience method for reading a binary file\n+ * and base64-encoding it.\n+ *\n+ * @param filename Filename for reading binary data\n+ * @return base64-encoded string or null if unsuccessful\n+ *\n+ * @since 2.1\n+ */\n+ public static String encodeFromFile( String filename )\n+ {\n+ String encodedData = null;\n+ Base64.InputStream bis = null;\n+ try\n+ {\n+ // Set up some useful variables\n+ java.io.File file = new java.io.File( filename );\n+ byte[] buffer = new byte[ (int)(file.length() * 1.4) ];\n+ int length = 0;\n+ int numBytes = 0;\n+ \n+ // Open a stream\n+ bis = new Base64.InputStream( \n+ new java.io.BufferedInputStream( \n+ new java.io.FileInputStream( file ) ), Base64.ENCODE );\n+ \n+ // Read until done\n+ while( ( numBytes = bis.read( buffer, length, 4096 ) ) >= 0 )\n+ length += numBytes;\n+ \n+ // Save in a variable to return\n+ encodedData = new String( buffer, 0, length, Base64.PREFERRED_ENCODING );\n+ \n+ } // end try\n+ catch( java.io.IOException e )\n+ {\n+ System.err.println( \"Error encoding from file \" + filename );\n+ } // end catch: IOException\n+ finally\n+ {\n+ try{ bis.close(); } catch( Exception e) {}\n+ } // end finally\n+ \n+ return encodedData;\n+ } // end encodeFromFile\n+ \n+ \n+ \n+ \n+ /* ******** I N N E R C L A S S I N P U T S T R E A M ******** */\n+ \n+ \n+ \n+ /**\n+ * A {@link Base64.InputStream} will read data from another\n+ * java.io.InputStream, given in the constructor,\n+ * and encode/decode to/from Base64 notation on the fly.\n+ *\n+ * @see Base64\n+ * @since 1.3\n+ */\n+ public static class InputStream extends java.io.FilterInputStream\n+ {\n+ private boolean encode; // Encoding or decoding\n+ private int position; // Current position in the buffer\n+ private byte[] buffer; // Small buffer holding converted data\n+ private int bufferLength; // Length of buffer (3 or 4)\n+ private int numSigBytes; // Number of meaningful bytes in the buffer\n+ private int lineLength;\n+ private boolean breakLines; // Break lines at less than 80 characters\n+ \n+ \n+ /**\n+ * Constructs a {@link Base64.InputStream} in DECODE mode.\n+ *\n+ * @param in the java.io.InputStream from which to read data.\n+ * @since 1.3\n+ */\n+ public InputStream( java.io.InputStream in )\n+ { \n+ this( in, DECODE );\n+ } // end constructor\n+ \n+ \n+ /**\n+ * Constructs a {@link Base64.InputStream} in\n+ * either ENCODE or DECODE mode.\n+ *

\n+ * Valid options:

\n+         *   ENCODE or DECODE: Encode or Decode as data is read.\n+         *   DONT_BREAK_LINES: don't break lines at 76 characters\n+         *     (only meaningful when encoding)\n+         *     Note: Technically, this makes your encoding non-compliant.\n+         *
\n+ *

\n+ * Example: new Base64.InputStream( in, Base64.DECODE )\n+ *\n+ *\n+ * @param in the java.io.InputStream from which to read data.\n+ * @param options Specified options\n+ * @see Base64#ENCODE\n+ * @see Base64#DECODE\n+ * @see Base64#DONT_BREAK_LINES\n+ * @since 2.0\n+ */\n+ public InputStream( java.io.InputStream in, int options )\n+ { \n+ super( in );\n+ this.breakLines = (options & DONT_BREAK_LINES) != DONT_BREAK_LINES;\n+ this.encode = (options & ENCODE) == ENCODE;\n+ this.bufferLength = encode ? 4 : 3;\n+ this.buffer = new byte[ bufferLength ];\n+ this.position = -1;\n+ this.lineLength = 0;\n+ } // end constructor\n+ \n+ /**\n+ * Reads enough of the input stream to convert\n+ * to/from Base64 and returns the next byte.\n+ *\n+ * @return next byte\n+ * @since 1.3\n+ */\n+ public int read() throws java.io.IOException \n+ { \n+ // Do we need to get data?\n+ if( position < 0 )\n+ {\n+ if( encode )\n+ {\n+ byte[] b3 = new byte[3];\n+ int numBinaryBytes = 0;\n+ for( int i = 0; i < 3; i++ )\n+ {\n+ try\n+ { \n+ int b = in.read();\n+ \n+ // If end of stream, b is -1.\n+ if( b >= 0 )\n+ {\n+ b3[i] = (byte)b;\n+ numBinaryBytes++;\n+ } // end if: not end of stream\n+ \n+ } // end try: read\n+ catch( java.io.IOException e )\n+ { \n+ // Only a problem if we got no data at all.\n+ if( i == 0 )\n+ throw e;\n+ \n+ } // end catch\n+ } // end for: each needed input byte\n+ \n+ if( numBinaryBytes > 0 )\n+ {\n+ encode3to4( b3, 0, numBinaryBytes, buffer, 0 );\n+ position = 0;\n+ numSigBytes = 4;\n+ } // end if: got data\n+ else\n+ {\n+ return -1;\n+ } // end else\n+ } // end if: encoding\n+ \n+ // Else decoding\n+ else\n+ {\n+ byte[] b4 = new byte[4];\n+ int i = 0;\n+ for( i = 0; i < 4; i++ )\n+ {\n+ // Read four \"meaningful\" bytes:\n+ int b = 0;\n+ do{ b = in.read(); }\n+ while( b >= 0 && DECODABET[ b & 0x7f ] <= WHITE_SPACE_ENC );\n+ \n+ if( b < 0 )\n+ break; // Reads a -1 if end of stream\n+ \n+ b4[i] = (byte)b;\n+ } // end for: each needed input byte\n+ \n+ if( i == 4 )\n+ {\n+ numSigBytes = decode4to3( b4, 0, buffer, 0 );\n+ position = 0;\n+ } // end if: got four characters\n+ else if( i == 0 ){\n+ return -1;\n+ } // end else if: also padded correctly\n+ else\n+ {\n+ // Must have broken out from above.\n+ throw new java.io.IOException( \"Improperly padded Base64 input.\" );\n+ } // end \n+ \n+ } // end else: decode\n+ } // end else: get data\n+ \n+ // Got data?\n+ if( position >= 0 )\n+ {\n+ // End of relevant data?\n+ if( /*!encode &&*/ position >= numSigBytes )\n+ return -1;\n+ \n+ if( encode && breakLines && lineLength >= MAX_LINE_LENGTH )\n+ {\n+ lineLength = 0;\n+ return '\\n';\n+ } // end if\n+ else\n+ {\n+ lineLength++; // This isn't important when decoding\n+ // but throwing an extra \"if\" seems\n+ // just as wasteful.\n+ \n+ int b = buffer[ position++ ];\n+\n+ if( position >= bufferLength )\n+ position = -1;\n+\n+ return b & 0xFF; // This is how you \"cast\" a byte that's\n+ // intended to be unsigned.\n+ } // end else\n+ } // end if: position >= 0\n+ \n+ // Else error\n+ else\n+ { \n+ // When JDK1.4 is more accepted, use an assertion here.\n+ throw new java.io.IOException( \"Error in Base64 code reading stream.\" );\n+ } // end else\n+ } // end read\n+ \n+ \n+ /**\n+ * Calls {@link #read()} repeatedly until the end of stream\n+ * is reached or len bytes are read.\n+ * Returns number of bytes read into array or -1 if\n+ * end of stream is encountered.\n+ *\n+ * @param dest array to hold values\n+ * @param off offset for array\n+ * @param len max number of bytes to read into array\n+ * @return bytes read into array or -1 if end of stream is encountered.\n+ * @since 1.3\n+ */\n+ public int read( byte[] dest, int off, int len ) throws java.io.IOException\n+ {\n+ int i;\n+ int b;\n+ for( i = 0; i < len; i++ )\n+ {\n+ b = read();\n+ \n+ //if( b < 0 && i == 0 )\n+ // return -1;\n+ \n+ if( b >= 0 )\n+ dest[off + i] = (byte)b;\n+ else if( i == 0 )\n+ return -1;\n+ else\n+ break; // Out of 'for' loop\n+ } // end for: each byte read\n+ return i;\n+ } // end read\n+ \n+ } // end inner class InputStream\n+ \n+ \n+ \n+ \n+ \n+ \n+ /* ******** I N N E R C L A S S O U T P U T S T R E A M ******** */\n+ \n+ \n+ \n+ /**\n+ * A {@link Base64.OutputStream} will write data to another\n+ * java.io.OutputStream, given in the constructor,\n+ * and encode/decode to/from Base64 notation on the fly.\n+ *\n+ * @see Base64\n+ * @since 1.3\n+ */\n+ public static class OutputStream extends java.io.FilterOutputStream\n+ {\n+ private boolean encode;\n+ private int position;\n+ private byte[] buffer;\n+ private int bufferLength;\n+ private int lineLength;\n+ private boolean breakLines;\n+ private byte[] b4; // Scratch used in a few places\n+ private boolean suspendEncoding;\n+ \n+ /**\n+ * Constructs a {@link Base64.OutputStream} in ENCODE mode.\n+ *\n+ * @param out the java.io.OutputStream to which data will be written.\n+ * @since 1.3\n+ */\n+ public OutputStream( java.io.OutputStream out )\n+ { \n+ this( out, ENCODE );\n+ } // end constructor\n+ \n+ \n+ /**\n+ * Constructs a {@link Base64.OutputStream} in\n+ * either ENCODE or DECODE mode.\n+ *

\n+ * Valid options:

\n+         *   ENCODE or DECODE: Encode or Decode as data is read.\n+         *   DONT_BREAK_LINES: don't break lines at 76 characters\n+         *     (only meaningful when encoding)\n+         *     Note: Technically, this makes your encoding non-compliant.\n+         *
\n+ *

\n+ * Example: new Base64.OutputStream( out, Base64.ENCODE )\n+ *\n+ * @param out the java.io.OutputStream to which data will be written.\n+ * @param options Specified options.\n+ * @see Base64#ENCODE\n+ * @see Base64#DECODE\n+ * @see Base64#DONT_BREAK_LINES\n+ * @since 1.3\n+ */\n+ public OutputStream( java.io.OutputStream out, int options )\n+ { \n+ super( out );\n+ this.breakLines = (options & DONT_BREAK_LINES) != DONT_BREAK_LINES;\n+ this.encode = (options & ENCODE) == ENCODE;\n+ this.bufferLength = encode ? 3 : 4;\n+ this.buffer = new byte[ bufferLength ];\n+ this.position = 0;\n+ this.lineLength = 0;\n+ this.suspendEncoding = false;\n+ this.b4 = new byte[4];\n+ } // end constructor\n+ \n+ \n+ /**\n+ * Writes the byte to the output stream after\n+ * converting to/from Base64 notation.\n+ * When encoding, bytes are buffered three\n+ * at a time before the output stream actually\n+ * gets a write() call.\n+ * When decoding, bytes are buffered four\n+ * at a time.\n+ *\n+ * @param theByte the byte to write\n+ * @since 1.3\n+ */\n+ public void write(int theByte) throws java.io.IOException\n+ {\n+ // Encoding suspended?\n+ if( suspendEncoding )\n+ {\n+ super.out.write( theByte );\n+ return;\n+ } // end if: supsended\n+ \n+ // Encode?\n+ if( encode )\n+ {\n+ buffer[ position++ ] = (byte)theByte;\n+ if( position >= bufferLength ) // Enough to encode.\n+ {\n+ out.write( encode3to4( b4, buffer, bufferLength ) );\n+\n+ lineLength += 4;\n+ if( breakLines && lineLength >= MAX_LINE_LENGTH )\n+ {\n+ out.write( NEW_LINE );\n+ lineLength = 0;\n+ } // end if: end of line\n+\n+ position = 0;\n+ } // end if: enough to output\n+ } // end if: encoding\n+\n+ // Else, Decoding\n+ else\n+ {\n+ // Meaningful Base64 character?\n+ if( DECODABET[ theByte & 0x7f ] > WHITE_SPACE_ENC )\n+ {\n+ buffer[ position++ ] = (byte)theByte;\n+ if( position >= bufferLength ) // Enough to output.\n+ {\n+ int len = Base64.decode4to3( buffer, 0, b4, 0 );\n+ out.write( b4, 0, len );\n+ //out.write( Base64.decode4to3( buffer ) );\n+ position = 0;\n+ } // end if: enough to output\n+ } // end if: meaningful base64 character\n+ else if( DECODABET[ theByte & 0x7f ] != WHITE_SPACE_ENC )\n+ {\n+ throw new java.io.IOException( \"Invalid character in Base64 data.\" );\n+ } // end else: not white space either\n+ } // end else: decoding\n+ } // end write\n+ \n+ \n+ \n+ /**\n+ * Calls {@link #write(int)} repeatedly until len \n+ * bytes are written.\n+ *\n+ * @param theBytes array from which to read bytes\n+ * @param off offset for array\n+ * @param len max number of bytes to read into array\n+ * @since 1.3\n+ */\n+ public void write( byte[] theBytes, int off, int len ) throws java.io.IOException\n+ {\n+ // Encoding suspended?\n+ if( suspendEncoding )\n+ {\n+ super.out.write( theBytes, off, len );\n+ return;\n+ } // end if: supsended\n+ \n+ for( int i = 0; i < len; i++ )\n+ {\n+ write( theBytes[ off + i ] );\n+ } // end for: each byte written\n+ \n+ } // end write\n+ \n+ \n+ \n+ /**\n+ * Method added by PHIL. [Thanks, PHIL. -Rob]\n+ * This pads the buffer without closing the stream.\n+ */\n+ public void flushBase64() throws java.io.IOException \n+ {\n+ if( position > 0 )\n+ {\n+ if( encode )\n+ {\n+ out.write( encode3to4( b4, buffer, position ) );\n+ position = 0;\n+ } // end if: encoding\n+ else\n+ {\n+ throw new java.io.IOException( \"Base64 input not properly padded.\" );\n+ } // end else: decoding\n+ } // end if: buffer partially full\n+\n+ } // end flush\n+\n+ \n+ /** \n+ * Flushes and closes (I think, in the superclass) the stream. \n+ *\n+ * @since 1.3\n+ */\n+ public void close() throws java.io.IOException\n+ {\n+ // 1. Ensure that pending characters are written\n+ flushBase64();\n+\n+ // 2. Actually close the stream\n+ // Base class both flushes and closes.\n+ super.close();\n+ \n+ buffer = null;\n+ out = null;\n+ } // end close\n+ \n+ \n+ \n+ /**\n+ * Suspends encoding of the stream.\n+ * May be helpful if you need to embed a piece of\n+ * base640-encoded data in a stream.\n+ *\n+ * @since 1.5.1\n+ */\n+ public void suspendEncoding() throws java.io.IOException \n+ {\n+ flushBase64();\n+ this.suspendEncoding = true;\n+ } // end suspendEncoding\n+ \n+ \n+ /**\n+ * Resumes encoding of the stream.\n+ * May be helpful if you need to embed a piece of\n+ * base640-encoded data in a stream.\n+ *\n+ * @since 1.5.1\n+ */\n+ public void resumeEncoding()\n+ {\n+ this.suspendEncoding = false;\n+ } // end resumeEncoding\n+ \n+ \n+ \n+ } // end inner class OutputStream\n+ \n+ \n+} // end class Base64\n-- \n1.5.6.74.g8a5e\n"},{"id":"81610","messageId":"1214726371-93520-18-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-17-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 17/21] Misc. documentation fixes to Base64 utility","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:27Z","receivedAt":"2008-06-29T07:59:27Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"These fixes silence warnings in the utility which are triggered\nby our (rather pedantic) Eclipse compiler settings.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/util/Base64.java | 8 +++++++-\n 1 files changed, 7 insertions(+), 1 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/util/Base64.java b/org.spearce.jgit/src/org/spearce/jgit/util/Base64.java\nindex b0c19b6..9254bd0 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/util/Base64.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/util/Base64.java\n@@ -378,6 +378,7 @@ public class Base64\n * Does not GZip-compress data.\n *\n * @param source The data to convert\n+ * @return encoded base64 representation of source.\n * @since 1.4\n */\n public static String encodeBytes( byte[] source )\n@@ -403,6 +404,7 @@ public class Base64\n *\n * @param source The data to convert\n * @param options Specified options\n+ * @return encoded base64 representation of source.\n * @see Base64#GZIP\n * @see Base64#DONT_BREAK_LINES\n * @since 2.0\n@@ -420,6 +422,7 @@ public class Base64\n * @param source The data to convert\n * @param off Offset in array where conversion should begin\n * @param len Length of data to convert\n+ * @return encoded base64 representation of source.\n * @since 1.4\n */\n public static String encodeBytes( byte[] source, int off, int len )\n@@ -447,6 +450,7 @@ public class Base64\n * @param off Offset in array where conversion should begin\n * @param len Length of data to convert\n * @param options Specified options\n+ * @return encoded base64 representation of source.\n * @see Base64#GZIP\n * @see Base64#DONT_BREAK_LINES\n * @since 2.0\n@@ -729,7 +733,7 @@ public class Base64\n if( bytes != null && bytes.length >= 4 )\n {\n \n- int head = ((int)bytes[0] & 0xff) | ((bytes[1] << 8) & 0xff00); \n+ int head = (bytes[0] & 0xff) | ((bytes[1] << 8) & 0xff00); \n if( java.util.zip.GZIPInputStream.GZIP_MAGIC == head ) \n {\n java.io.ByteArrayInputStream bais = null;\n@@ -1386,6 +1390,7 @@ public class Base64\n /**\n * Method added by PHIL. [Thanks, PHIL. -Rob]\n * This pads the buffer without closing the stream.\n+ * @throws java.io.IOException input was not properly padded.\n */\n public void flushBase64() throws java.io.IOException \n {\n@@ -1430,6 +1435,7 @@ public class Base64\n * May be helpful if you need to embed a piece of\n * base640-encoded data in a stream.\n *\n+ * @throws java.io.IOException input was not properly padded.\n * @since 1.5.1\n */\n public void suspendEncoding() throws java.io.IOException \n-- \n1.5.6.74.g8a5e\n"},{"id":"81609","messageId":"1214726371-93520-19-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-18-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 18/21] Extract the basic HTTP proxy support to its own class","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:28Z","receivedAt":"2008-06-29T07:59:28Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This way the proxy can be initialized from locations other than\njgit's Main, such as from Eclipse plugin initialization or other\ncommand line tools that wrap jgit.\n\nWe also moved the proxy lookup code to the utility class as the\nerror handling is several lines of code and may be shared.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/pgm/Main.java | 36 +----\n .../org/spearce/jgit/transport/TransportHttp.java | 36 ++---\n .../src/org/spearce/jgit/util/HttpSupport.java | 165 ++++++++++++++++++++\n 3 files changed, 182 insertions(+), 55 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/util/HttpSupport.java\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/pgm/Main.java b/org.spearce.jgit/src/org/spearce/jgit/pgm/Main.java\nindex 8afd0d7..3d507c6 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/pgm/Main.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/pgm/Main.java\n@@ -39,13 +39,12 @@\n package org.spearce.jgit.pgm;\n \n import java.io.File;\n-import java.net.MalformedURLException;\n-import java.net.URL;\n import java.util.Arrays;\n \n import org.spearce.jgit.awtui.AwtAuthenticator;\n import org.spearce.jgit.errors.TransportException;\n import org.spearce.jgit.lib.Repository;\n+import org.spearce.jgit.util.HttpSupport;\n \n /** Command line entry point. */\n public class Main {\n@@ -60,7 +59,7 @@ public class Main {\n \tpublic static void main(final String[] argv) {\n \t\ttry {\n \t\t\tAwtAuthenticator.install();\n-\t\t\tconfigureHttpProxy();\n+\t\t\tHttpSupport.configureHttpProxy();\n \t\t\texecute(argv);\n \t\t} catch (Die err) {\n \t\t\tSystem.err.println(\"fatal: \" + err.getMessage());\n@@ -170,35 +169,4 @@ public class Main {\n \t\tSystem.err.println(\"jgit [--git-dir=path] cmd ...\");\n \t\tSystem.exit(1);\n \t}\n-\n-\tprivate static void configureHttpProxy() {\n-\t\tfinal String s = System.getenv(\"http_proxy\");\n-\t\tif (s == null || s.equals(\"\"))\n-\t\t\treturn;\n-\n-\t\tfinal URL u;\n-\t\ttry {\n-\t\t\tu = new URL(s);\n-\t\t} catch (MalformedURLException e) {\n-\t\t\tthrow new Die(\"Invalid http_proxy: \" + s + \": \" + e.getMessage());\n-\t\t}\n-\t\tif (!\"http\".equals(u.getProtocol()))\n-\t\t\tthrow new Die(\"Invalid http_proxy: \" + s + \": Only http supported.\");\n-\n-\t\tfinal String proxyHost = u.getHost();\n-\t\tfinal int proxyPort = u.getPort();\n-\n-\t\tSystem.setProperty(\"http.proxyHost\", proxyHost);\n-\t\tif (proxyPort > 0)\n-\t\t\tSystem.setProperty(\"http.proxyPort\", String.valueOf(proxyPort));\n-\n-\t\tfinal String userpass = u.getUserInfo();\n-\t\tif (userpass != null && userpass.contains(\":\")) {\n-\t\t\tfinal int c = userpass.indexOf(':');\n-\t\t\tfinal String user = userpass.substring(0, c);\n-\t\t\tfinal String pass = userpass.substring(c + 1);\n-\t\t\tAwtAuthenticator.add(new AwtAuthenticator.CachedAuthentication(\n-\t\t\t\t\tproxyHost, proxyPort, user, pass));\n-\t\t}\n-\t}\n }\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\nindex 2f28f2c..9351a12 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportHttp.java\n@@ -41,13 +41,11 @@ import java.io.BufferedReader;\n import java.io.FileNotFoundException;\n import java.io.IOException;\n import java.io.InputStream;\n-import java.net.ConnectException;\n+import java.net.HttpURLConnection;\n import java.net.MalformedURLException;\n import java.net.Proxy;\n import java.net.ProxySelector;\n-import java.net.URISyntaxException;\n import java.net.URL;\n-import java.net.URLConnection;\n import java.util.ArrayList;\n import java.util.Collection;\n import java.util.Map;\n@@ -59,6 +57,7 @@ import org.spearce.jgit.errors.TransportException;\n import org.spearce.jgit.lib.ObjectId;\n import org.spearce.jgit.lib.Ref;\n import org.spearce.jgit.lib.Repository;\n+import org.spearce.jgit.util.HttpSupport;\n \n /**\n * Transport over the non-Git aware HTTP and FTP protocol.\n@@ -107,10 +106,6 @@ class TransportHttp extends WalkTransport {\n \t\treturn r;\n \t}\n \n-\tProxy proxyFor(final URL u) throws URISyntaxException {\n-\t\treturn proxySelector.select(u.toURI()).get(0);\n-\t}\n-\n \tclass HttpObjectDB extends WalkRemoteObjectDatabase {\n \t\tprivate final URL objectsUrl;\n \n@@ -172,23 +167,22 @@ class TransportHttp extends WalkTransport {\n \t\t@Override\n \t\tFileStream open(final String path) throws IOException {\n \t\t\tfinal URL base = objectsUrl;\n-\t\t\ttry {\n-\t\t\t\tfinal URL u = new URL(base, path);\n-\t\t\t\tfinal URLConnection c = u.openConnection(proxyFor(u));\n+\t\t\tfinal URL u = new URL(base, path);\n+\t\t\tfinal Proxy proxy = HttpSupport.proxyFor(proxySelector, u);\n+\t\t\tfinal HttpURLConnection c;\n+\n+\t\t\tc = (HttpURLConnection) u.openConnection(proxy);\n+\t\t\tswitch (HttpSupport.response(c)) {\n+\t\t\tcase HttpURLConnection.HTTP_OK:\n \t\t\t\tfinal InputStream in = c.getInputStream();\n \t\t\t\tfinal int len = c.getContentLength();\n \t\t\t\treturn new FileStream(in, len);\n-\t\t\t} catch (ConnectException ce) {\n-\t\t\t\t// The standard J2SE error message is not very useful.\n-\t\t\t\t//\n-\t\t\t\tif (\"Connection timed out: connect\".equals(ce.getMessage()))\n-\t\t\t\t\tthrow new ConnectException(\"Connection timed out: \" + base);\n-\t\t\t\tthrow new ConnectException(ce.getMessage() + \" \" + base);\n-\t\t\t} catch (URISyntaxException e) {\n-\t\t\t\tfinal ConnectException err;\n-\t\t\t\terr = new ConnectException(\"Cannot determine proxy for \" + base);\n-\t\t\t\terr.initCause(e);\n-\t\t\t\tthrow err;\n+\t\t\tcase HttpURLConnection.HTTP_NOT_FOUND:\n+\t\t\t\tthrow new FileNotFoundException(u.toString());\n+\t\t\tdefault:\n+\t\t\t\tthrow new IOException(u.toString() + \": \"\n+\t\t\t\t\t\t+ HttpSupport.response(c) + \" \"\n+\t\t\t\t\t\t+ c.getResponseMessage());\n \t\t\t}\n \t\t}\n \ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/util/HttpSupport.java b/org.spearce.jgit/src/org/spearce/jgit/util/HttpSupport.java\nnew file mode 100644\nindex 0000000..29b4d8e\n--- /dev/null\n+++ b/org.spearce.jgit/src/org/spearce/jgit/util/HttpSupport.java\n@@ -0,0 +1,165 @@\n+/*\n+ * Copyright (C) 2008, Shawn O. Pearce \n+ *\n+ * All rights reserved.\n+ *\n+ * Redistribution and use in source and binary forms, with or\n+ * without modification, are permitted provided that the following\n+ * conditions are met:\n+ *\n+ * - Redistributions of source code must retain the above copyright\n+ * notice, this list of conditions and the following disclaimer.\n+ *\n+ * - Redistributions in binary form must reproduce the above\n+ * copyright notice, this list of conditions and the following\n+ * disclaimer in the documentation and/or other materials provided\n+ * with the distribution.\n+ *\n+ * - Neither the name of the Git Development Community nor the\n+ * names of its contributors may be used to endorse or promote\n+ * products derived from this software without specific prior\n+ * written permission.\n+ *\n+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND\n+ * CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES,\n+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT\n+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\n+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+ */\n+\n+package org.spearce.jgit.util;\n+\n+import java.io.IOException;\n+import java.io.UnsupportedEncodingException;\n+import java.net.ConnectException;\n+import java.net.HttpURLConnection;\n+import java.net.MalformedURLException;\n+import java.net.Proxy;\n+import java.net.ProxySelector;\n+import java.net.URISyntaxException;\n+import java.net.URL;\n+import java.net.URLEncoder;\n+\n+import org.spearce.jgit.awtui.AwtAuthenticator;\n+\n+/** Extra utilities to support usage of HTTP. */\n+public class HttpSupport {\n+\t/**\n+\t * Configure the JRE's standard HTTP based on http_proxy.\n+\t *

\n+\t * The popular libcurl library honors the http_proxy\n+\t * environment variable as a means of specifying an HTTP proxy for requests\n+\t * made behind a firewall. This is not natively recognized by the JRE, so\n+\t * this method can be used by command line utilities to configure the JRE\n+\t * before the first request is sent.\n+\t * \n+\t * @throws MalformedURLException\n+\t * the value in http_proxy is unsupportable.\n+\t */\n+\tpublic static void configureHttpProxy() throws MalformedURLException {\n+\t\tfinal String s = System.getenv(\"http_proxy\");\n+\t\tif (s == null || s.equals(\"\"))\n+\t\t\treturn;\n+\n+\t\tfinal URL u = new URL(s);\n+\t\tif (!\"http\".equals(u.getProtocol()))\n+\t\t\tthrow new MalformedURLException(\"Invalid http_proxy: \" + s\n+\t\t\t\t\t+ \": Only http supported.\");\n+\n+\t\tfinal String proxyHost = u.getHost();\n+\t\tfinal int proxyPort = u.getPort();\n+\n+\t\tSystem.setProperty(\"http.proxyHost\", proxyHost);\n+\t\tif (proxyPort > 0)\n+\t\t\tSystem.setProperty(\"http.proxyPort\", String.valueOf(proxyPort));\n+\n+\t\tfinal String userpass = u.getUserInfo();\n+\t\tif (userpass != null && userpass.contains(\":\")) {\n+\t\t\tfinal int c = userpass.indexOf(':');\n+\t\t\tfinal String user = userpass.substring(0, c);\n+\t\t\tfinal String pass = userpass.substring(c + 1);\n+\t\t\tAwtAuthenticator.add(new AwtAuthenticator.CachedAuthentication(\n+\t\t\t\t\tproxyHost, proxyPort, user, pass));\n+\t\t}\n+\t}\n+\n+\t/**\n+\t * URL encode a value string into an output buffer.\n+\t * \n+\t * @param urlstr\n+\t * the output buffer.\n+\t * @param key\n+\t * value which must be encoded to protected special characters.\n+\t */\n+\tpublic static void encode(final StringBuilder urlstr, final String key) {\n+\t\tif (key == null || key.length() == 0)\n+\t\t\treturn;\n+\t\ttry {\n+\t\t\turlstr.append(URLEncoder.encode(key, \"UTF-8\"));\n+\t\t} catch (UnsupportedEncodingException e) {\n+\t\t\tthrow new RuntimeException(\"Could not URL encode to UTF-8\", e);\n+\t\t}\n+\t}\n+\n+\t/**\n+\t * Get the HTTP response code from the request.\n+\t *

\n+\t * Roughly the same as c.getResponseCode() but the\n+\t * ConnectException is translated to be more understandable.\n+\t * \n+\t * @param c\n+\t * connection the code should be obtained from.\n+\t * @return r HTTP status code, usually 200 to indicate success. See\n+\t * {@link HttpURLConnection} for other defined constants.\n+\t * @throws IOException\n+\t * communications error prevented obtaining the response code.\n+\t */\n+\tpublic static int response(final HttpURLConnection c) throws IOException {\n+\t\ttry {\n+\t\t\treturn c.getResponseCode();\n+\t\t} catch (ConnectException ce) {\n+\t\t\tfinal String host = c.getURL().getHost();\n+\t\t\t// The standard J2SE error message is not very useful.\n+\t\t\t//\n+\t\t\tif (\"Connection timed out: connect\".equals(ce.getMessage()))\n+\t\t\t\tthrow new ConnectException(\"Connection time out: \" + host);\n+\t\t\tthrow new ConnectException(ce.getMessage() + \" \" + host);\n+\t\t}\n+\t}\n+\n+\t/**\n+\t * Determine the proxy server (if any) needed to obtain a URL.\n+\t * \n+\t * @param proxySelector\n+\t * proxy support for the caller.\n+\t * @param u\n+\t * location of the server caller wants to talk to.\n+\t * @return proxy to communicate with the supplied URL.\n+\t * @throws ConnectException\n+\t * the proxy could not be computed as the supplied URL could not\n+\t * be read. This failure should never occur.\n+\t */\n+\tpublic static Proxy proxyFor(final ProxySelector proxySelector, final URL u)\n+\t\t\tthrows ConnectException {\n+\t\ttry {\n+\t\t\treturn proxySelector.select(u.toURI()).get(0);\n+\t\t} catch (URISyntaxException e) {\n+\t\t\tfinal ConnectException err;\n+\t\t\terr = new ConnectException(\"Cannot determine proxy for \" + u);\n+\t\t\terr.initCause(e);\n+\t\t\tthrow err;\n+\t\t}\n+\t}\n+\n+\tprivate HttpSupport() {\n+\t\t// Utility class only.\n+\t}\n+}\n-- \n1.5.6.74.g8a5e\n"},{"id":"81613","messageId":"1214726371-93520-20-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-19-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 19/21] Create a really simple Amazon S3 REST client","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:29Z","receivedAt":"2008-06-29T07:59:29Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This is a very basic client for Amazon's Simple Storage Service (S3).\nThe client is able to perform the basic FTP like operations necessary\nto support storing a Git repository on the S3 servers, assuming the\ntransport is implemented as a dumb protocol style transport similar\nto the existing sftp:// transport.\n\nA tiny command line client is included to facilitate manual testing\nagainst the S3 servers, as well as emergency operations such as\ngetting content or deleting content.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/transport/AmazonS3.java | 696 ++++++++++++++++++++\n 1 files changed, 696 insertions(+), 0 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java b/org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java\nnew file mode 100644\nindex 0000000..466d9e9\n--- /dev/null\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java\n@@ -0,0 +1,696 @@\n+/*\n+ * Copyright (C) 2008, Shawn O. Pearce \n+ *\n+ * All rights reserved.\n+ *\n+ * Redistribution and use in source and binary forms, with or\n+ * without modification, are permitted provided that the following\n+ * conditions are met:\n+ *\n+ * - Redistributions of source code must retain the above copyright\n+ * notice, this list of conditions and the following disclaimer.\n+ *\n+ * - Redistributions in binary form must reproduce the above\n+ * copyright notice, this list of conditions and the following\n+ * disclaimer in the documentation and/or other materials provided\n+ * with the distribution.\n+ *\n+ * - Neither the name of the Git Development Community nor the\n+ * names of its contributors may be used to endorse or promote\n+ * products derived from this software without specific prior\n+ * written permission.\n+ *\n+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND\n+ * CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES,\n+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT\n+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\n+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+ */\n+\n+package org.spearce.jgit.transport;\n+\n+import java.io.EOFException;\n+import java.io.File;\n+import java.io.FileInputStream;\n+import java.io.FileNotFoundException;\n+import java.io.IOException;\n+import java.io.InputStream;\n+import java.io.OutputStream;\n+import java.net.HttpURLConnection;\n+import java.net.Proxy;\n+import java.net.ProxySelector;\n+import java.net.URL;\n+import java.net.URLConnection;\n+import java.security.DigestOutputStream;\n+import java.security.InvalidKeyException;\n+import java.security.MessageDigest;\n+import java.security.NoSuchAlgorithmException;\n+import java.text.SimpleDateFormat;\n+import java.util.ArrayList;\n+import java.util.Collections;\n+import java.util.Date;\n+import java.util.HashSet;\n+import java.util.Iterator;\n+import java.util.List;\n+import java.util.Locale;\n+import java.util.Map;\n+import java.util.Properties;\n+import java.util.Set;\n+import java.util.SortedMap;\n+import java.util.TimeZone;\n+import java.util.TreeMap;\n+\n+import javax.crypto.Mac;\n+import javax.crypto.spec.SecretKeySpec;\n+\n+import org.spearce.jgit.awtui.AwtAuthenticator;\n+import org.spearce.jgit.lib.Constants;\n+import org.spearce.jgit.util.Base64;\n+import org.spearce.jgit.util.HttpSupport;\n+import org.spearce.jgit.util.TemporaryBuffer;\n+import org.xml.sax.Attributes;\n+import org.xml.sax.InputSource;\n+import org.xml.sax.SAXException;\n+import org.xml.sax.XMLReader;\n+import org.xml.sax.helpers.DefaultHandler;\n+import org.xml.sax.helpers.XMLReaderFactory;\n+\n+/**\n+ * A simple HTTP REST client for the Amazon S3 service.\n+ *

\n+ * This client uses the REST API to communicate with the Amazon S3 servers and\n+ * read or write content through a bucket that the user has access to. It is a\n+ * very lightweight implementation of the S3 API and therefore does not have all\n+ * of the bells and whistles of popular client implementations.\n+ *

\n+ * Authentication is always performed using the user's AWSAccessKeyId and their\n+ * private AWSSecretAccessKey.\n+ */\n+public class AmazonS3 {\n+\tprivate static final Set SIGNED_HEADERS;\n+\n+\tprivate static final String HMAC = \"HmacSHA1\";\n+\n+\tprivate static final String DOMAIN = \"s3.amazonaws.com\";\n+\n+\tprivate static final String X_AMZ_ACL = \"x-amz-acl\";\n+\n+\tstatic {\n+\t\tSIGNED_HEADERS = new HashSet();\n+\t\tSIGNED_HEADERS.add(\"content-type\");\n+\t\tSIGNED_HEADERS.add(\"content-md5\");\n+\t\tSIGNED_HEADERS.add(\"date\");\n+\t}\n+\n+\tprivate static boolean isSignedHeader(final String name) {\n+\t\tfinal String nameLC = name.toLowerCase();\n+\t\treturn SIGNED_HEADERS.contains(nameLC) || nameLC.startsWith(\"x-amz-\");\n+\t}\n+\n+\tprivate static String toCleanString(final List list) {\n+\t\tfinal StringBuilder s = new StringBuilder();\n+\t\tfor (final String v : list) {\n+\t\t\tif (s.length() > 0)\n+\t\t\t\ts.append(',');\n+\t\t\ts.append(v.replaceAll(\"\\n\", \"\").trim());\n+\t\t}\n+\t\treturn s.toString();\n+\t}\n+\n+\tprivate static String remove(final Map m, final String k) {\n+\t\tfinal String r = m.remove(k);\n+\t\treturn r != null ? r : \"\";\n+\t}\n+\n+\tprivate static String httpNow() {\n+\t\tfinal String tz = \"GMT\";\n+\t\tfinal SimpleDateFormat fmt;\n+\t\tfmt = new SimpleDateFormat(\"EEE, dd MMM yyyy HH:mm:ss\", Locale.US);\n+\t\tfmt.setTimeZone(TimeZone.getTimeZone(tz));\n+\t\treturn fmt.format(new Date()) + \" \" + tz;\n+\t}\n+\n+\tprivate static MessageDigest newMD5() {\n+\t\ttry {\n+\t\t\treturn MessageDigest.getInstance(\"MD5\");\n+\t\t} catch (NoSuchAlgorithmException e) {\n+\t\t\tthrow new RuntimeException(\"JRE lacks MD5 implementation\", e);\n+\t\t}\n+\t}\n+\n+\t/** AWSAccessKeyId, public string that identifies the user's account. */\n+\tprivate final String publicKey;\n+\n+\t/** Decoded form of the private AWSSecretAccessKey, to sign requests. */\n+\tprivate final SecretKeySpec privateKey;\n+\n+\t/** Our HTTP proxy support, in case we are behind a firewall. */\n+\tprivate final ProxySelector proxySelector;\n+\n+\t/** ACL to apply to created objects. */\n+\tprivate final String acl;\n+\n+\t/** Maximum number of times to try an operation. */\n+\tprivate final int maxAttempts;\n+\n+\t/**\n+\t * Create a new S3 client for the supplied user information.\n+\t *

\n+\t * The connection properties are a subset of those supported by the popular\n+\t * jets3t library.\n+\t * For example:\n+\t * \n+\t * 

\n+\t * # AWS Access and Secret Keys (required)\n+\t * accesskey: <YourAWSAccessKey>\n+\t * secretkey: <YourAWSSecretKey>\n+\t * \n+\t * # Access Control List setting to apply to uploads, must be one of:\n+\t * # PRIVATE, PUBLIC_READ (defaults to PRIVATE).\n+\t * acl: PRIVATE\n+\t * \n+\t * # Number of times to retry after internal error from S3.\n+\t * httpclient.retry-max: 3\n+\t *
\n+\t * \n+\t * @param props\n+\t * connection properties.\n+\t * \n+\t */\n+\tpublic AmazonS3(final Properties props) {\n+\t\tpublicKey = props.getProperty(\"accesskey\");\n+\t\tif (publicKey == null)\n+\t\t\tthrow new IllegalArgumentException(\"Missing accesskey.\");\n+\n+\t\tfinal String secret = props.getProperty(\"secretkey\");\n+\t\tif (secret == null)\n+\t\t\tthrow new IllegalArgumentException(\"Missing secretkey.\");\n+\t\tprivateKey = new SecretKeySpec(Constants.encodeASCII(secret), HMAC);\n+\n+\t\tfinal String pacl = props.getProperty(\"acl\", \"PRIVATE\");\n+\t\tif (\"PRIVATE\".equalsIgnoreCase(pacl))\n+\t\t\tacl = \"private\";\n+\t\telse if (\"PUBLIC\".equalsIgnoreCase(pacl))\n+\t\t\tacl = \"public-read\";\n+\t\telse if (\"PUBLIC-READ\".equalsIgnoreCase(pacl))\n+\t\t\tacl = \"public-read\";\n+\t\telse if (\"PUBLIC_READ\".equalsIgnoreCase(pacl))\n+\t\t\tacl = \"public-read\";\n+\t\telse\n+\t\t\tthrow new IllegalArgumentException(\"Invalid acl: \" + pacl);\n+\n+\t\tmaxAttempts = Integer.parseInt(props.getProperty(\n+\t\t\t\t\"httpclient.retry-max\", \"3\"));\n+\t\tproxySelector = ProxySelector.getDefault();\n+\t}\n+\n+\t/**\n+\t * Get the content of a bucket object.\n+\t * \n+\t * @param bucket\n+\t * name of the bucket storing the object.\n+\t * @param key\n+\t * key of the object within its bucket.\n+\t * @return connection to stream the content of the object. The request\n+\t * properties of the connection may not be modified by the caller as\n+\t * the request parameters have already been signed.\n+\t * @throws IOException\n+\t * sending the request was not possible.\n+\t */\n+\tpublic URLConnection get(final String bucket, final String key)\n+\t\t\tthrows IOException {\n+\t\tfor (int curAttempt = 0; curAttempt < maxAttempts; curAttempt++) {\n+\t\t\tfinal HttpURLConnection c = open(\"GET\", bucket, key);\n+\t\t\tauthorize(c);\n+\t\t\tswitch (HttpSupport.response(c)) {\n+\t\t\tcase HttpURLConnection.HTTP_OK:\n+\t\t\t\treturn c;\n+\t\t\tcase HttpURLConnection.HTTP_NOT_FOUND:\n+\t\t\t\tthrow new FileNotFoundException(key);\n+\t\t\tcase HttpURLConnection.HTTP_INTERNAL_ERROR:\n+\t\t\t\tcontinue;\n+\t\t\tdefault:\n+\t\t\t\tthrow error(\"Reading\", key, c);\n+\t\t\t}\n+\t\t}\n+\t\tthrow maxAttempts(\"Reading\", key);\n+\t}\n+\n+\t/**\n+\t * List the names of keys available within a bucket.\n+\t *

\n+\t * This method is primarily meant for obtaining a \"recursive directory\n+\t * listing\" rooted under the specified bucket and prefix location.\n+\t * \n+\t * @param bucket\n+\t * name of the bucket whose objects should be listed.\n+\t * @param prefix\n+\t * common prefix to filter the results by. Must not be null.\n+\t * Supplying the empty string will list all keys in the bucket.\n+\t * Supplying a non-empty string will act as though a trailing '/'\n+\t * appears in prefix, even if it does not.\n+\t * @return list of keys starting with prefix, after removing\n+\t * prefix (or prefix + \"/\")from all\n+\t * of them.\n+\t * @throws IOException\n+\t * sending the request was not possible, or the response XML\n+\t * document could not be parsed properly.\n+\t */\n+\tpublic List list(final String bucket, String prefix)\n+\t\t\tthrows IOException {\n+\t\tif (prefix.length() > 0 && !prefix.endsWith(\"/\"))\n+\t\t\tprefix += \"/\";\n+\t\tfinal ListParser lp = new ListParser(bucket, prefix);\n+\t\tdo {\n+\t\t\tlp.list();\n+\t\t} while (lp.truncated);\n+\t\treturn lp.entries;\n+\t}\n+\n+\t/**\n+\t * Delete a single object.\n+\t *

\n+\t * Deletion always succeeds, even if the object does not exist.\n+\t * \n+\t * @param bucket\n+\t * name of the bucket storing the object.\n+\t * @param key\n+\t * key of the object within its bucket.\n+\t * @throws IOException\n+\t * deletion failed due to communications error.\n+\t */\n+\tpublic void delete(final String bucket, final String key)\n+\t\t\tthrows IOException {\n+\t\tfor (int curAttempt = 0; curAttempt < maxAttempts; curAttempt++) {\n+\t\t\tfinal HttpURLConnection c = open(\"DELETE\", bucket, key);\n+\t\t\tauthorize(c);\n+\t\t\tswitch (HttpSupport.response(c)) {\n+\t\t\tcase HttpURLConnection.HTTP_NO_CONTENT:\n+\t\t\t\treturn;\n+\t\t\tcase HttpURLConnection.HTTP_INTERNAL_ERROR:\n+\t\t\t\tcontinue;\n+\t\t\tdefault:\n+\t\t\t\tthrow error(\"Deletion\", key, c);\n+\t\t\t}\n+\t\t}\n+\t\tthrow maxAttempts(\"Deletion\", key);\n+\t}\n+\n+\t/**\n+\t * Atomically create or replace a single small object.\n+\t *

\n+\t * This form is only suitable for smaller contents, where the caller can\n+\t * reasonable fit the entire thing into memory.\n+\t *

\n+\t * End-to-end data integrity is assured by internally computing the MD5\n+\t * checksum of the supplied data and transmitting the checksum along with\n+\t * the data itself.\n+\t * \n+\t * @param bucket\n+\t * name of the bucket storing the object.\n+\t * @param key\n+\t * key of the object within its bucket.\n+\t * @param data\n+\t * new data content for the object. Must not be null. Zero length\n+\t * array will create a zero length object.\n+\t * @throws IOException\n+\t * creation/updating failed due to communications error.\n+\t */\n+\tpublic void put(final String bucket, final String key, final byte[] data)\n+\t\t\tthrows IOException {\n+\t\tfinal String md5str = Base64.encodeBytes(newMD5().digest(data));\n+\t\tfinal String lenstr = String.valueOf(data.length);\n+\t\tfor (int curAttempt = 0; curAttempt < maxAttempts; curAttempt++) {\n+\t\t\tfinal HttpURLConnection c = open(\"PUT\", bucket, key);\n+\t\t\tc.setRequestProperty(\"Content-Length\", lenstr);\n+\t\t\tc.setRequestProperty(\"Content-MD5\", md5str);\n+\t\t\tc.setRequestProperty(X_AMZ_ACL, acl);\n+\t\t\tauthorize(c);\n+\t\t\tc.setDoOutput(true);\n+\t\t\tc.setFixedLengthStreamingMode(data.length);\n+\t\t\tfinal OutputStream os = c.getOutputStream();\n+\t\t\ttry {\n+\t\t\t\tos.write(data);\n+\t\t\t} finally {\n+\t\t\t\tos.close();\n+\t\t\t}\n+\n+\t\t\tswitch (HttpSupport.response(c)) {\n+\t\t\tcase HttpURLConnection.HTTP_OK:\n+\t\t\t\treturn;\n+\t\t\tcase HttpURLConnection.HTTP_INTERNAL_ERROR:\n+\t\t\t\tcontinue;\n+\t\t\tdefault:\n+\t\t\t\tthrow error(\"Writing\", key, c);\n+\t\t\t}\n+\t\t}\n+\t\tthrow maxAttempts(\"Writing\", key);\n+\t}\n+\n+\t/**\n+\t * Atomically create or replace a single large object.\n+\t *

\n+\t * Initially the returned output stream buffers data into memory, but if the\n+\t * total number of written bytes starts to exceed an internal limit the data\n+\t * is spooled to a temporary file on the local drive.\n+\t *

\n+\t * Network transmission is attempted only when close() gets\n+\t * called at the end of output. Closing the returned stream can therefore\n+\t * take significant time, especially if the written content is very large.\n+\t *

\n+\t * End-to-end data integrity is assured by internally computing the MD5\n+\t * checksum of the supplied data and transmitting the checksum along with\n+\t * the data itself.\n+\t * \n+\t * @param bucket\n+\t * name of the bucket storing the object.\n+\t * @param key\n+\t * key of the object within its bucket.\n+\t * @return a stream which accepts the new data, and transmits once closed.\n+\t */\n+\tpublic OutputStream beginPut(final String bucket, final String key) {\n+\t\tfinal MessageDigest md5 = newMD5();\n+\t\tfinal TemporaryBuffer buffer = new TemporaryBuffer() {\n+\t\t\t@Override\n+\t\t\tpublic void close() throws IOException {\n+\t\t\t\tsuper.close();\n+\t\t\t\ttry {\n+\t\t\t\t\tputImpl(bucket, key, md5.digest(), this);\n+\t\t\t\t} finally {\n+\t\t\t\t\tdestroy();\n+\t\t\t\t}\n+\t\t\t}\n+\t\t};\n+\t\treturn new DigestOutputStream(buffer, md5);\n+\t}\n+\n+\tprivate void putImpl(final String bucket, final String key,\n+\t\t\tfinal byte[] csum, final TemporaryBuffer buf) throws IOException {\n+\t\tfinal String md5str = Base64.encodeBytes(csum);\n+\t\tfinal long len = buf.length();\n+\t\tfinal String lenstr = String.valueOf(len);\n+\t\tfor (int curAttempt = 0; curAttempt < maxAttempts; curAttempt++) {\n+\t\t\tfinal HttpURLConnection c = open(\"PUT\", bucket, key);\n+\t\t\tc.setRequestProperty(\"Content-Length\", lenstr);\n+\t\t\tc.setRequestProperty(\"Content-MD5\", md5str);\n+\t\t\tc.setRequestProperty(X_AMZ_ACL, acl);\n+\t\t\tauthorize(c);\n+\t\t\tc.setDoOutput(true);\n+\t\t\tc.setFixedLengthStreamingMode((int) len);\n+\t\t\tfinal OutputStream os = c.getOutputStream();\n+\t\t\ttry {\n+\t\t\t\tbuf.writeTo(os, null);\n+\t\t\t} finally {\n+\t\t\t\tos.close();\n+\t\t\t}\n+\n+\t\t\tswitch (HttpSupport.response(c)) {\n+\t\t\tcase HttpURLConnection.HTTP_OK:\n+\t\t\t\treturn;\n+\t\t\tcase HttpURLConnection.HTTP_INTERNAL_ERROR:\n+\t\t\t\tcontinue;\n+\t\t\tdefault:\n+\t\t\t\tthrow error(\"Writing\", key, c);\n+\t\t\t}\n+\t\t}\n+\t\tthrow maxAttempts(\"Writing\", key);\n+\t}\n+\n+\tprivate IOException error(final String action, final String key,\n+\t\t\tfinal HttpURLConnection c) throws IOException {\n+\t\treturn new IOException(action + \" of '\" + key + \"' failed: \"\n+\t\t\t\t+ HttpSupport.response(c) + \" \" + c.getResponseMessage());\n+\t}\n+\n+\tprivate IOException maxAttempts(final String action, final String key) {\n+\t\treturn new IOException(action + \" of '\" + key + \"' failed:\"\n+\t\t\t\t+ \" Giving up after \" + maxAttempts + \" attempts.\");\n+\t}\n+\n+\tprivate HttpURLConnection open(final String method, final String bucket,\n+\t\t\tfinal String key) throws IOException {\n+\t\tfinal Map noArgs = Collections.emptyMap();\n+\t\treturn open(method, bucket, key, noArgs);\n+\t}\n+\n+\tprivate HttpURLConnection open(final String method, final String bucket,\n+\t\t\tfinal String key, final Map args)\n+\t\t\tthrows IOException {\n+\t\tfinal StringBuilder urlstr = new StringBuilder();\n+\t\turlstr.append(\"http://\");\n+\t\turlstr.append(bucket);\n+\t\turlstr.append('.');\n+\t\turlstr.append(DOMAIN);\n+\t\turlstr.append('/');\n+\t\tif (key.length() > 0)\n+\t\t\tHttpSupport.encode(urlstr, key);\n+\t\tif (!args.isEmpty()) {\n+\t\t\tfinal Iterator> i;\n+\n+\t\t\turlstr.append('?');\n+\t\t\ti = args.entrySet().iterator();\n+\t\t\twhile (i.hasNext()) {\n+\t\t\t\tfinal Map.Entry e = i.next();\n+\t\t\t\turlstr.append(e.getKey());\n+\t\t\t\turlstr.append('=');\n+\t\t\t\tHttpSupport.encode(urlstr, e.getValue());\n+\t\t\t\tif (i.hasNext())\n+\t\t\t\t\turlstr.append('&');\n+\t\t\t}\n+\t\t}\n+\n+\t\tfinal URL url = new URL(urlstr.toString());\n+\t\tfinal Proxy proxy = HttpSupport.proxyFor(proxySelector, url);\n+\t\tfinal HttpURLConnection c;\n+\n+\t\tc = (HttpURLConnection) url.openConnection(proxy);\n+\t\tc.setRequestMethod(method);\n+\t\tc.setRequestProperty(\"User-Agent\", \"jgit/1.0\");\n+\t\tc.setRequestProperty(\"Date\", httpNow());\n+\t\treturn c;\n+\t}\n+\n+\tprivate void authorize(final HttpURLConnection c) throws IOException {\n+\t\tfinal Map> reqHdr = c.getRequestProperties();\n+\t\tfinal SortedMap sigHdr = new TreeMap();\n+\t\tfor (final String hdr : reqHdr.keySet()) {\n+\t\t\tif (isSignedHeader(hdr))\n+\t\t\t\tsigHdr.put(hdr.toLowerCase(), toCleanString(reqHdr.get(hdr)));\n+\t\t}\n+\n+\t\tfinal StringBuilder s = new StringBuilder();\n+\t\ts.append(c.getRequestMethod());\n+\t\ts.append('\\n');\n+\n+\t\ts.append(remove(sigHdr, \"content-md5\"));\n+\t\ts.append('\\n');\n+\n+\t\ts.append(remove(sigHdr, \"content-type\"));\n+\t\ts.append('\\n');\n+\n+\t\ts.append(remove(sigHdr, \"date\"));\n+\t\ts.append('\\n');\n+\n+\t\tfor (final Map.Entry e : sigHdr.entrySet()) {\n+\t\t\ts.append(e.getKey());\n+\t\t\ts.append(':');\n+\t\t\ts.append(e.getValue());\n+\t\t\ts.append('\\n');\n+\t\t}\n+\n+\t\tfinal String host = c.getURL().getHost();\n+\t\ts.append('/');\n+\t\ts.append(host.substring(0, host.length() - DOMAIN.length() - 1));\n+\t\ts.append(c.getURL().getPath());\n+\n+\t\tfinal String sec;\n+\t\ttry {\n+\t\t\tfinal Mac m = Mac.getInstance(HMAC);\n+\t\t\tm.init(privateKey);\n+\t\t\tsec = Base64.encodeBytes(m.doFinal(s.toString().getBytes(\"UTF-8\")));\n+\t\t} catch (NoSuchAlgorithmException e) {\n+\t\t\tthrow new IOException(\"No \" + HMAC + \" support:\" + e.getMessage());\n+\t\t} catch (InvalidKeyException e) {\n+\t\t\tthrow new IOException(\"Invalid key: \" + e.getMessage());\n+\t\t}\n+\t\tc.setRequestProperty(\"Authorization\", \"AWS \" + publicKey + \":\" + sec);\n+\t}\n+\n+\t/**\n+\t * Simple command line interface to {@link AmazonS3}.\n+\t * \n+\t * @param argv\n+\t * command line arguments. See usage for details.\n+\t * @throws IOException\n+\t * an error occurred.\n+\t */\n+\tpublic static void main(final String[] argv) throws IOException {\n+\t\tif (argv.length != 4) {\n+\t\t\tcommandLineUsage();\n+\t\t\treturn;\n+\t\t}\n+\n+\t\tAwtAuthenticator.install();\n+\t\tHttpSupport.configureHttpProxy();\n+\n+\t\tfinal AmazonS3 s3 = new AmazonS3(properties(new File(argv[0])));\n+\t\tfinal String op = argv[1];\n+\t\tfinal String bucket = argv[2];\n+\t\tfinal String key = argv[3];\n+\t\tif (\"get\".equals(op)) {\n+\t\t\tfinal URLConnection c = s3.get(bucket, key);\n+\t\t\tint len = c.getContentLength();\n+\t\t\tfinal InputStream in = c.getInputStream();\n+\t\t\ttry {\n+\t\t\t\tfinal byte[] tmp = new byte[2048];\n+\t\t\t\twhile (len > 0) {\n+\t\t\t\t\tfinal int n = in.read(tmp);\n+\t\t\t\t\tif (n < 0)\n+\t\t\t\t\t\tthrow new EOFException(\"Expected \" + len + \" bytes.\");\n+\t\t\t\t\tSystem.out.write(tmp, 0, n);\n+\t\t\t\t\tlen -= n;\n+\t\t\t\t}\n+\t\t\t} finally {\n+\t\t\t\tin.close();\n+\t\t\t}\n+\t\t} else if (\"ls\".equals(op) || \"list\".equals(op)) {\n+\t\t\tfor (final String k : s3.list(bucket, key))\n+\t\t\t\tSystem.out.println(k);\n+\t\t} else if (\"rm\".equals(op) || \"delete\".equals(op)) {\n+\t\t\ts3.delete(bucket, key);\n+\t\t} else if (\"put\".equals(op)) {\n+\t\t\tfinal OutputStream os = s3.beginPut(bucket, key);\n+\t\t\tfinal byte[] tmp = new byte[2048];\n+\t\t\tint n;\n+\t\t\twhile ((n = System.in.read(tmp)) > 0)\n+\t\t\t\tos.write(tmp, 0, n);\n+\t\t\tos.close();\n+\t\t} else {\n+\t\t\tcommandLineUsage();\n+\t\t}\n+\t}\n+\n+\tprivate static void commandLineUsage() {\n+\t\tSystem.err.println(\"usage: conn.prop op bucket key\");\n+\t\tSystem.err.println();\n+\t\tSystem.err.println(\" where conn.prop is a jets3t properties file.\");\n+\t\tSystem.err.println(\" op is one of: get ls rm put\");\n+\t\tSystem.err.println(\" bucket is the name of the S3 bucket\");\n+\t\tSystem.err.println(\" key is the name of the object.\");\n+\t\tSystem.exit(1);\n+\t}\n+\n+\tstatic Properties properties(final File authFile)\n+\t\t\tthrows FileNotFoundException, IOException {\n+\t\tfinal Properties p = new Properties();\n+\t\tfinal FileInputStream in = new FileInputStream(authFile);\n+\t\ttry {\n+\t\t\tp.load(in);\n+\t\t} finally {\n+\t\t\tin.close();\n+\t\t}\n+\t\treturn p;\n+\t}\n+\n+\tprivate final class ListParser extends DefaultHandler {\n+\t\tfinal List entries = new ArrayList();\n+\n+\t\tprivate final String bucket;\n+\n+\t\tprivate final String prefix;\n+\n+\t\tboolean truncated;\n+\n+\t\tprivate StringBuilder data;\n+\n+\t\tListParser(final String bn, final String p) {\n+\t\t\tbucket = bn;\n+\t\t\tprefix = p;\n+\t\t}\n+\n+\t\tvoid list() throws IOException {\n+\t\t\tfinal Map args = new TreeMap();\n+\t\t\tif (prefix.length() > 0)\n+\t\t\t\targs.put(\"prefix\", prefix);\n+\t\t\tif (!entries.isEmpty())\n+\t\t\t\targs.put(\"marker\", prefix + entries.get(entries.size() - 1));\n+\n+\t\t\tfor (int curAttempt = 0; curAttempt < maxAttempts; curAttempt++) {\n+\t\t\t\tfinal HttpURLConnection c = open(\"GET\", bucket, \"\", args);\n+\t\t\t\tauthorize(c);\n+\t\t\t\tswitch (HttpSupport.response(c)) {\n+\t\t\t\tcase HttpURLConnection.HTTP_OK:\n+\t\t\t\t\ttruncated = false;\n+\t\t\t\t\tdata = null;\n+\n+\t\t\t\t\tfinal XMLReader xr;\n+\t\t\t\t\ttry {\n+\t\t\t\t\t\txr = XMLReaderFactory.createXMLReader();\n+\t\t\t\t\t} catch (SAXException e) {\n+\t\t\t\t\t\tthrow new IOException(\"No XML parser available.\");\n+\t\t\t\t\t}\n+\t\t\t\t\txr.setContentHandler(this);\n+\t\t\t\t\tfinal InputStream in = c.getInputStream();\n+\t\t\t\t\ttry {\n+\t\t\t\t\t\txr.parse(new InputSource(in));\n+\t\t\t\t\t} catch (SAXException parsingError) {\n+\t\t\t\t\t\tfinal IOException p;\n+\t\t\t\t\t\tp = new IOException(\"Error listing \" + prefix);\n+\t\t\t\t\t\tp.initCause(parsingError);\n+\t\t\t\t\t\tthrow p;\n+\t\t\t\t\t} finally {\n+\t\t\t\t\t\tin.close();\n+\t\t\t\t\t}\n+\t\t\t\t\treturn;\n+\n+\t\t\t\tcase HttpURLConnection.HTTP_INTERNAL_ERROR:\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\tdefault:\n+\t\t\t\t\tthrow AmazonS3.this.error(\"Listing\", prefix, c);\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tthrow maxAttempts(\"Listing\", prefix);\n+\t\t}\n+\n+\t\t@Override\n+\t\tpublic void startElement(final String uri, final String name,\n+\t\t\t\tfinal String qName, final Attributes attributes)\n+\t\t\t\tthrows SAXException {\n+\t\t\tif (\"Key\".equals(name) || \"IsTruncated\".equals(name))\n+\t\t\t\tdata = new StringBuilder();\n+\t\t}\n+\n+\t\t@Override\n+\t\tpublic void ignorableWhitespace(final char[] ch, final int s,\n+\t\t\t\tfinal int n) throws SAXException {\n+\t\t\tif (data != null)\n+\t\t\t\tdata.append(ch, s, n);\n+\t\t}\n+\n+\t\t@Override\n+\t\tpublic void characters(final char[] ch, final int s, final int n)\n+\t\t\t\tthrows SAXException {\n+\t\t\tif (data != null)\n+\t\t\t\tdata.append(ch, s, n);\n+\t\t}\n+\n+\t\t@Override\n+\t\tpublic void endElement(final String uri, final String name,\n+\t\t\t\tfinal String qName) throws SAXException {\n+\t\t\tif (\"Key\".equals(name))\n+\t\t\t\tentries.add(data.toString().substring(prefix.length()));\n+\t\t\telse if (\"IsTruncated\".equals(name))\n+\t\t\t\ttruncated = \"true\".equalsIgnoreCase(data.toString());\n+\t\t\tdata = null;\n+\t\t}\n+\t}\n+}\n-- \n1.5.6.74.g8a5e\n"},{"id":"81621","messageId":"1214726371-93520-21-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-20-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 20/21] Add client side encryption to Amazon S3 client library","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:30Z","receivedAt":"2008-06-29T07:59:30Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"By encrypting (and decrypting) all data on the client side we\nare able to safely hide the content of our repository from the\nowners/operators the Amazon S3 service, making it a secure backup\nsolution for Git repositories.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/transport/AmazonS3.java | 82 ++++++++-\n .../org/spearce/jgit/transport/WalkEncryption.java | 188 ++++++++++++++++++++\n 2 files changed, 266 insertions(+), 4 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/WalkEncryption.java\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java b/org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java\nindex 466d9e9..4c82967 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/AmazonS3.java\n@@ -37,6 +37,7 @@\n \n package org.spearce.jgit.transport;\n \n+import java.io.ByteArrayOutputStream;\n import java.io.EOFException;\n import java.io.File;\n import java.io.FileInputStream;\n@@ -53,6 +54,7 @@ import java.security.DigestOutputStream;\n import java.security.InvalidKeyException;\n import java.security.MessageDigest;\n import java.security.NoSuchAlgorithmException;\n+import java.security.spec.InvalidKeySpecException;\n import java.text.SimpleDateFormat;\n import java.util.ArrayList;\n import java.util.Collections;\n@@ -93,6 +95,11 @@ import org.xml.sax.helpers.XMLReaderFactory;\n *

\n * Authentication is always performed using the user's AWSAccessKeyId and their\n * private AWSSecretAccessKey.\n+ *

\n+ * Optional client-side encryption may be enabled if requested. The format is\n+ * compatible with jets3t,\n+ * a popular Java based Amazon S3 client library. Enabling encryption can hide\n+ * sensitive data from the operators of the S3 service.\n */\n public class AmazonS3 {\n \tprivate static final Set SIGNED_HEADERS;\n@@ -103,6 +110,8 @@ public class AmazonS3 {\n \n \tprivate static final String X_AMZ_ACL = \"x-amz-acl\";\n \n+\tprivate static final String X_AMZ_META = \"x-amz-meta-\";\n+\n \tstatic {\n \t\tSIGNED_HEADERS = new HashSet();\n \t\tSIGNED_HEADERS.add(\"content-type\");\n@@ -161,6 +170,9 @@ public class AmazonS3 {\n \t/** Maximum number of times to try an operation. */\n \tprivate final int maxAttempts;\n \n+\t/** Encryption algorithm, may be a null instance that provides pass-through. */\n+\tprivate final WalkEncryption encryption;\n+\n \t/**\n \t * Create a new S3 client for the supplied user information.\n \t *

\n@@ -179,6 +191,10 @@ public class AmazonS3 {\n \t * \n \t * # Number of times to retry after internal error from S3.\n \t * httpclient.retry-max: 3\n+\t * \n+\t * # End-to-end encryption (hides content from S3 owners)\n+\t * password: <encryption pass-phrase>\n+\t * crypto.algorithm: PBEWithMD5AndDES\n \t * \n \t * \n \t * @param props\n@@ -207,6 +223,22 @@ public class AmazonS3 {\n \t\telse\n \t\t\tthrow new IllegalArgumentException(\"Invalid acl: \" + pacl);\n \n+\t\ttry {\n+\t\t\tfinal String cPas = props.getProperty(\"password\");\n+\t\t\tif (cPas != null) {\n+\t\t\t\tString cAlg = props.getProperty(\"crypto.algorithm\");\n+\t\t\t\tif (cAlg == null)\n+\t\t\t\t\tcAlg = \"PBEWithMD5AndDES\";\n+\t\t\t\tencryption = new WalkEncryption.ObjectEncryptionV2(cAlg, cPas);\n+\t\t\t} else {\n+\t\t\t\tencryption = WalkEncryption.NONE;\n+\t\t\t}\n+\t\t} catch (InvalidKeySpecException e) {\n+\t\t\tthrow new IllegalArgumentException(\"Invalid encryption\", e);\n+\t\t} catch (NoSuchAlgorithmException e) {\n+\t\t\tthrow new IllegalArgumentException(\"Invalid encryption\", e);\n+\t\t}\n+\n \t\tmaxAttempts = Integer.parseInt(props.getProperty(\n \t\t\t\t\"httpclient.retry-max\", \"3\"));\n \t\tproxySelector = ProxySelector.getDefault();\n@@ -232,6 +264,7 @@ public class AmazonS3 {\n \t\t\tauthorize(c);\n \t\t\tswitch (HttpSupport.response(c)) {\n \t\t\tcase HttpURLConnection.HTTP_OK:\n+\t\t\t\tencryption.validate(c, X_AMZ_META);\n \t\t\t\treturn c;\n \t\t\tcase HttpURLConnection.HTTP_NOT_FOUND:\n \t\t\t\tthrow new FileNotFoundException(key);\n@@ -245,6 +278,19 @@ public class AmazonS3 {\n \t}\n \n \t/**\n+\t * Decrypt an input stream from {@link #get(String, String)}.\n+\t * \n+\t * @param u\n+\t * connection previously created by {@link #get(String, String)}}.\n+\t * @return stream to read plain text from.\n+\t * @throws IOException\n+\t * decryption could not be configured.\n+\t */\n+\tpublic InputStream decrypt(final URLConnection u) throws IOException {\n+\t\treturn encryption.decrypt(u.getInputStream());\n+\t}\n+\n+\t/**\n \t * List the names of keys available within a bucket.\n \t *

\n \t * This method is primarily meant for obtaining a \"recursive directory\n@@ -326,6 +372,16 @@ public class AmazonS3 {\n \t */\n \tpublic void put(final String bucket, final String key, final byte[] data)\n \t\t\tthrows IOException {\n+\t\tif (encryption != WalkEncryption.NONE) {\n+\t\t\t// We have to copy to produce the cipher text anyway so use\n+\t\t\t// the large object code path as it supports that behavior.\n+\t\t\t//\n+\t\t\tfinal OutputStream os = beginPut(bucket, key);\n+\t\t\tos.write(data);\n+\t\t\tos.close();\n+\t\t\treturn;\n+\t\t}\n+\n \t\tfinal String md5str = Base64.encodeBytes(newMD5().digest(data));\n \t\tfinal String lenstr = String.valueOf(data.length);\n \t\tfor (int curAttempt = 0; curAttempt < maxAttempts; curAttempt++) {\n@@ -375,8 +431,11 @@ public class AmazonS3 {\n \t * @param key\n \t * key of the object within its bucket.\n \t * @return a stream which accepts the new data, and transmits once closed.\n+\t * @throws IOException\n+\t * if encryption was enabled it could not be configured.\n \t */\n-\tpublic OutputStream beginPut(final String bucket, final String key) {\n+\tpublic OutputStream beginPut(final String bucket, final String key)\n+\t\t\tthrows IOException {\n \t\tfinal MessageDigest md5 = newMD5();\n \t\tfinal TemporaryBuffer buffer = new TemporaryBuffer() {\n \t\t\t@Override\n@@ -389,7 +448,7 @@ public class AmazonS3 {\n \t\t\t\t}\n \t\t\t}\n \t\t};\n-\t\treturn new DigestOutputStream(buffer, md5);\n+\t\treturn encryption.encrypt(new DigestOutputStream(buffer, md5));\n \t}\n \n \tprivate void putImpl(final String bucket, final String key,\n@@ -402,6 +461,7 @@ public class AmazonS3 {\n \t\t\tc.setRequestProperty(\"Content-Length\", lenstr);\n \t\t\tc.setRequestProperty(\"Content-MD5\", md5str);\n \t\t\tc.setRequestProperty(X_AMZ_ACL, acl);\n+\t\t\tencryption.request(c, X_AMZ_META);\n \t\t\tauthorize(c);\n \t\t\tc.setDoOutput(true);\n \t\t\tc.setFixedLengthStreamingMode((int) len);\n@@ -426,8 +486,22 @@ public class AmazonS3 {\n \n \tprivate IOException error(final String action, final String key,\n \t\t\tfinal HttpURLConnection c) throws IOException {\n-\t\treturn new IOException(action + \" of '\" + key + \"' failed: \"\n-\t\t\t\t+ HttpSupport.response(c) + \" \" + c.getResponseMessage());\n+\t\tfinal IOException err = new IOException(action + \" of '\" + key\n+\t\t\t\t+ \"' failed: \" + HttpSupport.response(c) + \" \"\n+\t\t\t\t+ c.getResponseMessage());\n+\t\tfinal ByteArrayOutputStream b = new ByteArrayOutputStream();\n+\t\tbyte[] buf = new byte[2048];\n+\t\tfor (;;) {\n+\t\t\tfinal int n = c.getErrorStream().read(buf);\n+\t\t\tif (n < 0)\n+\t\t\t\tbreak;\n+\t\t\tif (n > 0)\n+\t\t\t\tb.write(buf, 0, n);\n+\t\t}\n+\t\tbuf = b.toByteArray();\n+\t\tif (buf.length > 0)\n+\t\t\terr.initCause(new IOException(\"\\n\" + new String(buf)));\n+\t\treturn err;\n \t}\n \n \tprivate IOException maxAttempts(final String action, final String key) {\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkEncryption.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkEncryption.java\nnew file mode 100644\nindex 0000000..cec6d75\n--- /dev/null\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkEncryption.java\n@@ -0,0 +1,188 @@\n+/*\n+ * Copyright (C) 2008, Shawn O. Pearce \n+ *\n+ * All rights reserved.\n+ *\n+ * Redistribution and use in source and binary forms, with or\n+ * without modification, are permitted provided that the following\n+ * conditions are met:\n+ *\n+ * - Redistributions of source code must retain the above copyright\n+ * notice, this list of conditions and the following disclaimer.\n+ *\n+ * - Redistributions in binary form must reproduce the above\n+ * copyright notice, this list of conditions and the following\n+ * disclaimer in the documentation and/or other materials provided\n+ * with the distribution.\n+ *\n+ * - Neither the name of the Git Development Community nor the\n+ * names of its contributors may be used to endorse or promote\n+ * products derived from this software without specific prior\n+ * written permission.\n+ *\n+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND\n+ * CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES,\n+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT\n+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\n+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+ */\n+\n+package org.spearce.jgit.transport;\n+\n+import java.io.IOException;\n+import java.io.InputStream;\n+import java.io.OutputStream;\n+import java.net.HttpURLConnection;\n+import java.security.InvalidAlgorithmParameterException;\n+import java.security.InvalidKeyException;\n+import java.security.NoSuchAlgorithmException;\n+import java.security.spec.InvalidKeySpecException;\n+\n+import javax.crypto.Cipher;\n+import javax.crypto.CipherInputStream;\n+import javax.crypto.CipherOutputStream;\n+import javax.crypto.NoSuchPaddingException;\n+import javax.crypto.SecretKey;\n+import javax.crypto.SecretKeyFactory;\n+import javax.crypto.spec.PBEKeySpec;\n+import javax.crypto.spec.PBEParameterSpec;\n+\n+abstract class WalkEncryption {\n+\tstatic final WalkEncryption NONE = new NoEncryption();\n+\n+\tstatic final String JETS3T_CRYPTO_VER = \"jets3t-crypto-ver\";\n+\n+\tstatic final String JETS3T_CRYPTO_ALG = \"jets3t-crypto-alg\";\n+\n+\tabstract OutputStream encrypt(OutputStream os) throws IOException;\n+\n+\tabstract InputStream decrypt(InputStream in) throws IOException;\n+\n+\tabstract void request(HttpURLConnection u, String prefix);\n+\n+\tabstract void validate(HttpURLConnection u, String p) throws IOException;\n+\n+\tprotected void validateImpl(final HttpURLConnection u, final String p,\n+\t\t\tfinal String version, final String name) throws IOException {\n+\t\tString v;\n+\n+\t\tv = u.getHeaderField(p + JETS3T_CRYPTO_VER);\n+\t\tif (v == null)\n+\t\t\tv = \"\";\n+\t\tif (!version.equals(v))\n+\t\t\tthrow new IOException(\"Unsupported encryption version: \" + v);\n+\n+\t\tv = u.getHeaderField(p + JETS3T_CRYPTO_ALG);\n+\t\tif (v == null)\n+\t\t\tv = \"\";\n+\t\tif (!name.equals(v))\n+\t\t\tthrow new IOException(\"Unsupported encryption algorithm: \" + v);\n+\t}\n+\n+\tIOException error(final Throwable why) {\n+\t\tfinal IOException e;\n+\t\te = new IOException(\"Encryption error: \" + why.getMessage());\n+\t\te.initCause(why);\n+\t\treturn e;\n+\t}\n+\n+\tprivate static class NoEncryption extends WalkEncryption {\n+\t\t@Override\n+\t\tvoid request(HttpURLConnection u, String prefix) {\n+\t\t\t// Don't store any request properties.\n+\t\t}\n+\n+\t\t@Override\n+\t\tvoid validate(final HttpURLConnection u, final String p)\n+\t\t\t\tthrows IOException {\n+\t\t\tvalidateImpl(u, p, \"\", \"\");\n+\t\t}\n+\n+\t\t@Override\n+\t\tInputStream decrypt(InputStream in) {\n+\t\t\treturn in;\n+\t\t}\n+\n+\t\t@Override\n+\t\tOutputStream encrypt(OutputStream os) {\n+\t\t\treturn os;\n+\t\t}\n+\t}\n+\n+\tstatic class ObjectEncryptionV2 extends WalkEncryption {\n+\t\tprivate static int ITERATION_COUNT = 5000;\n+\n+\t\tprivate static byte[] salt = { (byte) 0xA4, (byte) 0x0B, (byte) 0xC8,\n+\t\t\t\t(byte) 0x34, (byte) 0xD6, (byte) 0x95, (byte) 0xF3, (byte) 0x13 };\n+\n+\t\tprivate final String algorithmName;\n+\n+\t\tprivate final SecretKey skey;\n+\n+\t\tprivate final PBEParameterSpec aspec;\n+\n+\t\tObjectEncryptionV2(final String algo, final String key)\n+\t\t\t\tthrows InvalidKeySpecException, NoSuchAlgorithmException {\n+\t\t\talgorithmName = algo;\n+\n+\t\t\tfinal PBEKeySpec s;\n+\t\t\ts = new PBEKeySpec(key.toCharArray(), salt, ITERATION_COUNT, 32);\n+\t\t\tskey = SecretKeyFactory.getInstance(algo).generateSecret(s);\n+\t\t\taspec = new PBEParameterSpec(salt, ITERATION_COUNT);\n+\t\t}\n+\n+\t\t@Override\n+\t\tvoid request(final HttpURLConnection u, final String prefix) {\n+\t\t\tu.setRequestProperty(prefix + JETS3T_CRYPTO_VER, \"2\");\n+\t\t\tu.setRequestProperty(prefix + JETS3T_CRYPTO_ALG, algorithmName);\n+\t\t}\n+\n+\t\t@Override\n+\t\tvoid validate(final HttpURLConnection u, final String p)\n+\t\t\t\tthrows IOException {\n+\t\t\tvalidateImpl(u, p, \"2\", algorithmName);\n+\t\t}\n+\n+\t\t@Override\n+\t\tOutputStream encrypt(final OutputStream os) throws IOException {\n+\t\t\ttry {\n+\t\t\t\tfinal Cipher c = Cipher.getInstance(algorithmName);\n+\t\t\t\tc.init(Cipher.ENCRYPT_MODE, skey, aspec);\n+\t\t\t\treturn new CipherOutputStream(os, c);\n+\t\t\t} catch (NoSuchAlgorithmException e) {\n+\t\t\t\tthrow error(e);\n+\t\t\t} catch (NoSuchPaddingException e) {\n+\t\t\t\tthrow error(e);\n+\t\t\t} catch (InvalidKeyException e) {\n+\t\t\t\tthrow error(e);\n+\t\t\t} catch (InvalidAlgorithmParameterException e) {\n+\t\t\t\tthrow error(e);\n+\t\t\t}\n+\t\t}\n+\n+\t\t@Override\n+\t\tInputStream decrypt(final InputStream in) throws IOException {\n+\t\t\ttry {\n+\t\t\t\tfinal Cipher c = Cipher.getInstance(algorithmName);\n+\t\t\t\tc.init(Cipher.DECRYPT_MODE, skey, aspec);\n+\t\t\t\treturn new CipherInputStream(in, c);\n+\t\t\t} catch (NoSuchAlgorithmException e) {\n+\t\t\t\tthrow error(e);\n+\t\t\t} catch (NoSuchPaddingException e) {\n+\t\t\t\tthrow error(e);\n+\t\t\t} catch (InvalidKeyException e) {\n+\t\t\t\tthrow error(e);\n+\t\t\t} catch (InvalidAlgorithmParameterException e) {\n+\t\t\t\tthrow error(e);\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n-- \n1.5.6.74.g8a5e\n"},{"id":"81620","messageId":"1214726371-93520-22-git-send-email-spearce@spearce.org","threadId":"14207","inReplyTo":"1214726371-93520-21-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 21/21] Bidirectional protocol support for Amazon S3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T07:59:31Z","receivedAt":"2008-06-29T07:59:31Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The new \"amazon-s3://\" transport provides bi-directional communication\nfor Git repositories to the S3 service. This may be useful for backup\nof private data which users do not want published to the world.\n\nSigned-off-by: Shawn O. Pearce \n---\n .../src/org/spearce/jgit/transport/Transport.java | 3 +\n .../spearce/jgit/transport/TransportAmazonS3.java | 319 ++++++++++++++++++++\n .../src/org/spearce/jgit/transport/URIish.java | 2 +-\n 3 files changed, 323 insertions(+), 1 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/TransportAmazonS3.java\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/Transport.java b/org.spearce.jgit/src/org/spearce/jgit/transport/Transport.java\nindex 5376a9e..b962162 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/Transport.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/Transport.java\n@@ -140,6 +140,9 @@ public abstract class Transport {\n \t\telse if (TransportGitAnon.canHandle(remote))\n \t\t\treturn new TransportGitAnon(local, remote);\n \n+\t\telse if (TransportAmazonS3.canHandle(remote))\n+\t\t\treturn new TransportAmazonS3(local, remote);\n+\n \t\telse if (TransportBundle.canHandle(remote))\n \t\t\treturn new TransportBundle(local, remote);\n \ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportAmazonS3.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportAmazonS3.java\nnew file mode 100644\nindex 0000000..ceb6848\n--- /dev/null\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportAmazonS3.java\n@@ -0,0 +1,319 @@\n+/*\n+ * Copyright (C) 2008, Shawn O. Pearce \n+ *\n+ * All rights reserved.\n+ *\n+ * Redistribution and use in source and binary forms, with or\n+ * without modification, are permitted provided that the following\n+ * conditions are met:\n+ *\n+ * - Redistributions of source code must retain the above copyright\n+ * notice, this list of conditions and the following disclaimer.\n+ *\n+ * - Redistributions in binary form must reproduce the above\n+ * copyright notice, this list of conditions and the following\n+ * disclaimer in the documentation and/or other materials provided\n+ * with the distribution.\n+ *\n+ * - Neither the name of the Git Development Community nor the\n+ * names of its contributors may be used to endorse or promote\n+ * products derived from this software without specific prior\n+ * written permission.\n+ *\n+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND\n+ * CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES,\n+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT\n+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\n+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+ */\n+\n+package org.spearce.jgit.transport;\n+\n+import java.io.BufferedReader;\n+import java.io.File;\n+import java.io.FileNotFoundException;\n+import java.io.IOException;\n+import java.io.InputStream;\n+import java.io.OutputStream;\n+import java.net.URLConnection;\n+import java.util.ArrayList;\n+import java.util.Collection;\n+import java.util.HashSet;\n+import java.util.Map;\n+import java.util.Properties;\n+import java.util.TreeMap;\n+\n+import org.spearce.jgit.errors.NotSupportedException;\n+import org.spearce.jgit.errors.TransportException;\n+import org.spearce.jgit.lib.ObjectId;\n+import org.spearce.jgit.lib.Ref;\n+import org.spearce.jgit.lib.Repository;\n+import org.spearce.jgit.lib.Ref.Storage;\n+import org.spearce.jgit.util.FS;\n+\n+/**\n+ * Transport over the non-Git aware Amazon S3 protocol.\n+ *

\n+ * This transport communicates with the Amazon S3 servers (a non-free commercial\n+ * hosting service that users must subscribe to). Some users may find transport\n+ * to and from S3 to be a useful backup service.\n+ *

\n+ * The transport does not require any specialized Git support on the remote\n+ * (server side) repository, as Amazon does not provide any such support.\n+ * Repository files are retrieved directly through the S3 API, which uses\n+ * extended HTTP/1.1 semantics. This make it possible to read or write Git data\n+ * from a remote repository that is stored on S3.\n+ *

\n+ * Unlike the HTTP variant (see {@link TransportHttp}) we rely upon being able\n+ * to list objects in a bucket, as the S3 API supports this function. By listing\n+ * the bucket contents we can avoid relying on objects/info/packs\n+ * or info/refs in the remote repository.\n+ *

\n+ * Concurrent pushing over this transport is not supported. Multiple concurrent\n+ * push operations may cause confusion in the repository state.\n+ * \n+ * @see WalkFetchConnection\n+ * @see WalkPushConnection\n+ */\n+class TransportAmazonS3 extends WalkTransport {\n+\tstatic final String S3_SCHEME = \"amazon-s3\";\n+\n+\tstatic boolean canHandle(final URIish uri) {\n+\t\tif (!uri.isRemote())\n+\t\t\treturn false;\n+\t\treturn S3_SCHEME.equals(uri.getScheme());\n+\t}\n+\n+\t/** User information necessary to connect to S3. */\n+\tprivate final AmazonS3 s3;\n+\n+\t/** Bucket the remote repository is stored in. */\n+\tprivate final String bucket;\n+\n+\t/**\n+\t * Key prefix which all objects related to the repository start with.\n+\t *

\n+\t * The prefix does not start with \"/\".\n+\t *

\n+\t * The prefix does not end with \"/\". The trailing slash is stripped during\n+\t * the constructor if a trailing slash was supplied in the URIish.\n+\t *

\n+\t * All files within the remote repository start with\n+\t * keyPrefix + \"/\".\n+\t */\n+\tprivate final String keyPrefix;\n+\n+\tTransportAmazonS3(final Repository local, final URIish uri)\n+\t\t\tthrows NotSupportedException {\n+\t\tsuper(local, uri);\n+\n+\t\tProperties props = null;\n+\t\tFile propsFile = new File(local.getDirectory(), uri.getUser());\n+\t\tif (!propsFile.isFile())\n+\t\t\tpropsFile = new File(FS.userHome(), uri.getUser());\n+\t\tif (propsFile.isFile()) {\n+\t\t\ttry {\n+\t\t\t\tprops = AmazonS3.properties(propsFile);\n+\t\t\t} catch (IOException e) {\n+\t\t\t\tthrow new NotSupportedException(\"cannot read \" + propsFile, e);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tprops = new Properties();\n+\t\t\tprops.setProperty(\"accesskey\", uri.getUser());\n+\t\t\tprops.setProperty(\"secretkey\", uri.getPass());\n+\t\t}\n+\n+\t\ts3 = new AmazonS3(props);\n+\t\tbucket = uri.getHost();\n+\n+\t\tString p = uri.getPath();\n+\t\tif (p.startsWith(\"/\"))\n+\t\t\tp = p.substring(1);\n+\t\tif (p.endsWith(\"/\"))\n+\t\t\tp = p.substring(0, p.length() - 1);\n+\t\tkeyPrefix = p;\n+\t}\n+\n+\t@Override\n+\tpublic FetchConnection openFetch() throws TransportException {\n+\t\tfinal DatabaseS3 c = new DatabaseS3(bucket, keyPrefix + \"/objects\");\n+\t\tfinal WalkFetchConnection r = new WalkFetchConnection(this, c);\n+\t\tr.available(c.readAdvertisedRefs());\n+\t\treturn r;\n+\t}\n+\n+\t@Override\n+\tpublic PushConnection openPush() throws TransportException {\n+\t\tfinal DatabaseS3 c = new DatabaseS3(bucket, keyPrefix + \"/objects\");\n+\t\tfinal WalkPushConnection r = new WalkPushConnection(this, c);\n+\t\tr.available(c.readAdvertisedRefs());\n+\t\treturn r;\n+\t}\n+\n+\tclass DatabaseS3 extends WalkRemoteObjectDatabase {\n+\t\tprivate final String bucketName;\n+\n+\t\tprivate final String objectsKey;\n+\n+\t\tDatabaseS3(final String b, final String o) {\n+\t\t\tbucketName = b;\n+\t\t\tobjectsKey = o;\n+\t\t}\n+\n+\t\tprivate String resolveKey(String subpath) {\n+\t\t\tif (subpath.endsWith(\"/\"))\n+\t\t\t\tsubpath = subpath.substring(0, subpath.length() - 1);\n+\t\t\tString k = objectsKey;\n+\t\t\twhile (subpath.startsWith(\"../\")) {\n+\t\t\t\tk = k.substring(0, k.lastIndexOf('/'));\n+\t\t\t\tsubpath = subpath.substring(3);\n+\t\t\t}\n+\t\t\treturn k + \"/\" + subpath;\n+\t\t}\n+\n+\t\t@Override\n+\t\tURIish getURI() {\n+\t\t\tURIish u = new URIish();\n+\t\t\tu = u.setScheme(S3_SCHEME);\n+\t\t\tu = u.setHost(bucketName);\n+\t\t\tu = u.setPath(\"/\" + objectsKey);\n+\t\t\treturn u;\n+\t\t}\n+\n+\t\t@Override\n+\t\tCollection getAlternates() throws IOException {\n+\t\t\ttry {\n+\t\t\t\treturn readAlternates(INFO_ALTERNATES);\n+\t\t\t} catch (FileNotFoundException err) {\n+\t\t\t\t// Fall through.\n+\t\t\t}\n+\t\t\treturn null;\n+\t\t}\n+\n+\t\t@Override\n+\t\tWalkRemoteObjectDatabase openAlternate(final String location)\n+\t\t\t\tthrows IOException {\n+\t\t\treturn new DatabaseS3(bucketName, resolveKey(location));\n+\t\t}\n+\n+\t\t@Override\n+\t\tCollection getPackNames() throws IOException {\n+\t\t\tfinal HashSet have = new HashSet();\n+\t\t\thave.addAll(s3.list(bucket, resolveKey(\"pack\")));\n+\n+\t\t\tfinal Collection packs = new ArrayList();\n+\t\t\tfor (final String n : have) {\n+\t\t\t\tif (!n.startsWith(\"pack-\") || !n.endsWith(\".pack\"))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\tfinal String in = n.substring(0, n.length() - 5) + \".idx\";\n+\t\t\t\tif (have.contains(in))\n+\t\t\t\t\tpacks.add(n);\n+\t\t\t}\n+\t\t\treturn packs;\n+\t\t}\n+\n+\t\t@Override\n+\t\tFileStream open(final String path) throws IOException {\n+\t\t\tfinal URLConnection c = s3.get(bucket, resolveKey(path));\n+\t\t\tfinal InputStream raw = c.getInputStream();\n+\t\t\tfinal InputStream in = s3.decrypt(c);\n+\t\t\tfinal int len = c.getContentLength();\n+\t\t\treturn new FileStream(in, raw == in ? len : -1);\n+\t\t}\n+\n+\t\t@Override\n+\t\tvoid deleteFile(final String path) throws IOException {\n+\t\t\ts3.delete(bucket, resolveKey(path));\n+\t\t}\n+\n+\t\t@Override\n+\t\tOutputStream writeFile(final String path) throws IOException {\n+\t\t\treturn s3.beginPut(bucket, resolveKey(path));\n+\t\t}\n+\n+\t\t@Override\n+\t\tvoid writeFile(final String path, final byte[] data) throws IOException {\n+\t\t\ts3.put(bucket, resolveKey(path), data);\n+\t\t}\n+\n+\t\tMap readAdvertisedRefs() throws TransportException {\n+\t\t\tfinal TreeMap avail = new TreeMap();\n+\t\t\treadPackedRefs(avail);\n+\t\t\treadLooseRefs(avail);\n+\t\t\treadRef(avail, \"HEAD\");\n+\t\t\treturn avail;\n+\t\t}\n+\n+\t\tprivate void readLooseRefs(final TreeMap avail)\n+\t\t\t\tthrows TransportException {\n+\t\t\ttry {\n+\t\t\t\tfor (final String n : s3.list(bucket, resolveKey(\"../refs\")))\n+\t\t\t\t\treadRef(avail, \"refs/\" + n);\n+\t\t\t} catch (IOException e) {\n+\t\t\t\tthrow new TransportException(getURI(), \"cannot list refs\", e);\n+\t\t\t}\n+\t\t}\n+\n+\t\tprivate Ref readRef(final TreeMap avail, final String rn)\n+\t\t\t\tthrows TransportException {\n+\t\t\tfinal String s;\n+\t\t\ttry {\n+\t\t\t\tfinal BufferedReader br = openReader(\"../\" + rn);\n+\t\t\t\ttry {\n+\t\t\t\t\ts = br.readLine();\n+\t\t\t\t} finally {\n+\t\t\t\t\tbr.close();\n+\t\t\t\t}\n+\t\t\t} catch (FileNotFoundException noRef) {\n+\t\t\t\treturn null;\n+\t\t\t} catch (IOException err) {\n+\t\t\t\tthrow new TransportException(getURI(), \"read ../\" + rn, err);\n+\t\t\t}\n+\n+\t\t\tif (s == null)\n+\t\t\t\tthrow new TransportException(getURI(), \"Empty ref: \" + rn);\n+\n+\t\t\tif (s.startsWith(\"ref: \")) {\n+\t\t\t\tfinal String target = s.substring(\"ref: \".length());\n+\t\t\t\tRef r = avail.get(target);\n+\t\t\t\tif (r == null)\n+\t\t\t\t\tr = readRef(avail, target);\n+\t\t\t\tif (r == null)\n+\t\t\t\t\treturn null;\n+\t\t\t\tr = new Ref(r.getStorage(), rn, r.getObjectId(), r\n+\t\t\t\t\t\t.getPeeledObjectId());\n+\t\t\t\tavail.put(r.getName(), r);\n+\t\t\t\treturn r;\n+\t\t\t}\n+\n+\t\t\tif (ObjectId.isId(s)) {\n+\t\t\t\tfinal Ref r = new Ref(loose(avail.get(rn)), rn, ObjectId\n+\t\t\t\t\t\t.fromString(s));\n+\t\t\t\tavail.put(r.getName(), r);\n+\t\t\t\treturn r;\n+\t\t\t}\n+\n+\t\t\tthrow new TransportException(getURI(), \"Bad ref: \" + rn + \": \" + s);\n+\t\t}\n+\n+\t\tprivate Storage loose(final Ref r) {\n+\t\t\tif (r != null && r.getStorage() == Storage.PACKED)\n+\t\t\t\treturn Storage.LOOSE_PACKED;\n+\t\t\treturn Storage.LOOSE;\n+\t\t}\n+\n+\t\t@Override\n+\t\tvoid close() {\n+\t\t\t// We do not maintain persistent connections.\n+\t\t}\n+\t}\n+}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java b/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java\nindex 9e7ca83..8aa5d35 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java\n@@ -51,7 +51,7 @@ import java.util.regex.Pattern;\n */\n public class URIish {\n \tprivate static final Pattern FULL_URI = Pattern\n-\t\t\t.compile(\"^(?:([a-z+]+)://(?:([^/]+?)(?::([^/]+?))?@)?(?:([^/]+?))?(?::(\\\\d+))?)?((?:[A-Za-z]:)?/.+)$\");\n+\t\t\t.compile(\"^(?:([a-z0-9+-]+)://(?:([^/]+?)(?::([^/]+?))?@)?(?:([^/]+?))?(?::(\\\\d+))?)?((?:[A-Za-z]:)?/.+)$\");\n \n \tprivate static final Pattern SCP_URI = Pattern\n \t\t\t.compile(\"^(?:([^@]+?)@)?([^:]+?):(.+)$\");\n-- \n1.5.6.74.g8a5e\n"},{"id":"81640","messageId":"200806291551.06201.robin.rosenberg@dewire.com","threadId":"14207","inReplyTo":"1214726371-93520-10-git-send-email-spearce@spearce.org","subject":"Re: [JGIT PATCH 09/21] Remember how a Ref was read in from disk and created","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2008-06-29T13:51:05Z","receivedAt":"2008-06-29T13:51:05Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"söndagen den 29 juni 2008 09.59.19 skrev Shawn O. Pearce:\n> To efficiently deleted or update a ref we need to know where\n> it came from when it was read into the process. If the ref\n> is being updated we can usually just write the loose file,\n> but if it is being deleted we may need to remove not just a\n> loose file but also delete it from the packed-refs.\n\nOne could argue that we should not normally just delete a ref, but\nmark it as deleted and let git gc delete it when it expires, just like\nany old ref, but then we should try to get C Git to do the same. There\nwas a thread relating to this recently.\n\n-- robi\n"},{"id":"81641","messageId":"200806291551.08166.robin.rosenberg@dewire.com","threadId":"14207","inReplyTo":"1214726371-93520-17-git-send-email-spearce@spearce.org","subject":"Re: [JGIT PATCH 16/21] Add Robert Harder's public domain Base64 encoding utility","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2008-06-29T13:51:07Z","receivedAt":"2008-06-29T13:51:07Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"\nDragging in apache commons libraries seems to be too much at this point, one class, so this choice is\nunderstandable at this point.\n\nOther that that Apache Commons are almost ubiquotous these days and so be considered at every\npoint when we need external code.\n\n-- robin\n"},{"id":"81644","messageId":"alpine.DEB.1.00.0806291516540.9925@racer","threadId":"14207","inReplyTo":"200806291551.06201.robin.rosenberg@dewire.com","subject":"Re: [JGIT PATCH 09/21] Remember how a Ref was read in from disk and created","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-06-29T14:17:53Z","receivedAt":"2008-06-29T14:17:53Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 29 Jun 2008, Robin Rosenberg wrote:\n\n> söndagen den 29 juni 2008 09.59.19 skrev Shawn O. Pearce:\n> > To efficiently deleted or update a ref we need to know where it came \n> > from when it was read into the process. If the ref is being updated \n> > we can usually just write the loose file, but if it is being deleted \n> > we may need to remove not just a loose file but also delete it from \n> > the packed-refs.\n> \n> One could argue that we should not normally just delete a ref, but mark \n> it as deleted and let git gc delete it when it expires, just like any \n> old ref, but then we should try to get C Git to do the same. There was a \n> thread relating to this recently.\n\n... but it petered out, so you should consider any ideas in that thread \nrejected.\n\nCiao,\nDscho"},{"id":"81652","messageId":"20080629180029.GG11793@spearce.org","threadId":"14207","inReplyTo":"alpine.DEB.1.00.0806291516540.9925@racer","subject":"Re: [JGIT PATCH 09/21] Remember how a Ref was read in from disk and created","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T18:00:29Z","receivedAt":"2008-06-29T18:00:29Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin wrote:\n> On Sun, 29 Jun 2008, Robin Rosenberg wrote:\n> \n> > söndagen den 29 juni 2008 09.59.19 skrev Shawn O. Pearce:\n> > > To efficiently deleted or update a ref we need to know where it came \n> > > from when it was read into the process. If the ref is being updated \n> > > we can usually just write the loose file, but if it is being deleted \n> > > we may need to remove not just a loose file but also delete it from \n> > > the packed-refs.\n> > \n> > One could argue that we should not normally just delete a ref, but mark \n> > it as deleted and let git gc delete it when it expires, just like any \n> > old ref, but then we should try to get C Git to do the same. There was a \n> > thread relating to this recently.\n> \n> ... but it petered out, so you should consider any ideas in that thread \n> rejected.\n\nRight. Its a nice idea, but until there is a really solid agreement in\nthe community about how this should be stored on disk, I don't want to\ntry and implement it in jgit, or in C Git for that matter. And I don't\nreally care enough to come up with something and reopen the thread myself.\n\nI just realized that the dumb transport push support doesn't delete\nthe reflog when it deletes the ref. Whoops. That's a problem\nif you later try to create a ref where a directory used to be.\nEither git will run into errors trying to create the reflog.\n\n-- \nShawn.\n"},{"id":"81653","messageId":"20080629180655.GH11793@spearce.org","threadId":"14207","inReplyTo":"200806291551.08166.robin.rosenberg@dewire.com","subject":"Re: [JGIT PATCH 16/21] Add Robert Harder's public domain Base64 encoding utility","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-29T18:06:55Z","receivedAt":"2008-06-29T18:06:55Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Robin Rosenberg wrote:\n> \n> Dragging in apache commons libraries seems to be too much at this\n> point, one class, so this choice is understandable at this point.\n> \n> Other that that Apache Commons are almost ubiquotous these days\n> and so be considered at every point when we need external code.\n\nTrue. But I have a nearly allergic reaction to Apache code;\nfor some reason its always sort of not quite there. Which also\ndescribes jgit so I shouldn't say anything. ;-)\n\nWe can always rip this implementation out if we do wind up taking\nin the Apache Commons libraries for other support. There is only\n1 or two calls sites and it should be easy enough to change over\nto the Apache Commons implementation.\n\n-- \nShawn.\n"}]}