{"thread":{"id":"18282","subject":"[PATCH JGIT] Allow writeObject() write to OutputStream","startedAt":"2009-03-12T08:02:10Z","lastAt":"2009-03-13T15:47:48Z","messageCount":3,"participants":["Daniel Cheng (aka SDiZ)","Shawn O. Pearce","Daniel Cheng"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"107779","messageId":"1236844931-12807-1-git-send-email-j16sdiz+freenet@gmail.com","threadId":"18282","inReplyTo":null,"subject":"[PATCH JGIT] Allow writeObject() write to OutputStream","fromName":"Daniel Cheng (aka SDiZ)","fromEmail":"j16sdiz+freenet@gmail.com","sentAt":"2009-03-12T08:02:10Z","receivedAt":"2009-03-12T08:02:10Z","isPatch":true,"sender":{"key":"j16sdiz+freenet@gmail.com","avatar":"https://gravatar.com/avatar/3e796e8a156ee86e305bfc1fbe01608302554ee3b5fa7eb8d1213b8877bfcabd?d=mp&s=160"},"body":"\nSigned-off-by: Daniel Cheng (aka SDiZ) <j16sdiz+freenet@gmail.com>\n---\n\nThis patch make factor out the object writing code in ObjectWriter,\nallow it to write to any OutputStream. \nSubclass class may then override\n  writeObject(final int type, long len, InputStream is, boolean store)\nto make it write to alternative locations.\n\nThere are some discussion on devl@freenetproject.org to use raw \n(uncompressed) object to freenet. This patch allow the testing.\n\n .../src/org/spearce/jgit/lib/ObjectWriter.java     |   93 +++++++++++---------\n 1 files changed, 50 insertions(+), 43 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java\nindex 546cc68..97acae4 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java\n@@ -45,6 +45,7 @@\n import java.io.FileOutputStream;\n import java.io.IOException;\n import java.io.InputStream;\n+import java.io.OutputStream;\n import java.io.OutputStreamWriter;\n import java.security.MessageDigest;\n import java.util.zip.Deflater;\n@@ -297,8 +298,52 @@ public ObjectId computeBlobSha1(final long len, final InputStream is)\n \t\treturn writeObject(Constants.OBJ_BLOB, len, is, false);\n \t}\n \n-\tObjectId writeObject(final int type, long len, final InputStream is,\n-\t\t\tboolean store) throws IOException {\n+\tprotected ObjectId writeObject(final int type, long len,\n+\t\t\tfinal InputStream is, final OutputStream deflateStream)\n+\t\t\tthrows IOException {\n+\t\tmd.reset();\n+\n+\t\tbyte[] header;\n+\t\tint n;\n+\n+\t\theader = Constants.encodedTypeString(type);\n+\t\tmd.update(header);\n+\t\tif (deflateStream != null)\n+\t\t\tdeflateStream.write(header);\n+\n+\t\tmd.update((byte) ' ');\n+\t\tif (deflateStream != null)\n+\t\t\tdeflateStream.write((byte) ' ');\n+\n+\t\theader = Constants.encodeASCII(len);\n+\t\tmd.update(header);\n+\t\tif (deflateStream != null)\n+\t\t\tdeflateStream.write(header);\n+\n+\t\tmd.update((byte) 0);\n+\t\tif (deflateStream != null)\n+\t\t\tdeflateStream.write((byte) 0);\n+\n+\t\twhile (len > 0\n+\t\t\t\t&& (n = is.read(buf, 0, (int) Math.min(len, buf.length))) > 0) {\n+\t\t\tmd.update(buf, 0, n);\n+\t\t\tif (deflateStream != null)\n+\t\t\t\tdeflateStream.write(buf, 0, n);\n+\t\t\tlen -= n;\n+\t\t}\n+\n+\t\tif (len != 0)\n+\t\t\tthrow new IOException(\"Input did not match supplied length. \" + len\n+\t\t\t\t\t+ \" bytes are missing.\");\n+\n+\t\tif (deflateStream != null)\n+\t\t\tdeflateStream.close();\n+\n+\t\treturn ObjectId.fromRaw(md.digest());\n+\t}\n+\n+\tprotected ObjectId writeObject(final int type, long len,\n+\t\t\tfinal InputStream is, boolean store) throws IOException {\n \t\tfinal File t;\n \t\tfinal DeflaterOutputStream deflateStream;\n \t\tfinal FileOutputStream fileStream;\n@@ -312,7 +357,6 @@ ObjectId writeObject(final int type, long len, final InputStream is,\n \t\t\tfileStream = null;\n \t\t}\n \n-\t\tmd.reset();\n \t\tif (store) {\n \t\t\tdef.reset();\n \t\t\tdeflateStream = new DeflaterOutputStream(fileStream, def);\n@@ -320,46 +364,9 @@ ObjectId writeObject(final int type, long len, final InputStream is,\n \t\t\tdeflateStream = null;\n \n \t\ttry {\n-\t\t\tbyte[] header;\n-\t\t\tint n;\n-\n-\t\t\theader = Constants.encodedTypeString(type);\n-\t\t\tmd.update(header);\n-\t\t\tif (deflateStream != null)\n-\t\t\t\tdeflateStream.write(header);\n-\n-\t\t\tmd.update((byte) ' ');\n-\t\t\tif (deflateStream != null)\n-\t\t\t\tdeflateStream.write((byte) ' ');\n-\n-\t\t\theader = Constants.encodeASCII(len);\n-\t\t\tmd.update(header);\n-\t\t\tif (deflateStream != null)\n-\t\t\t\tdeflateStream.write(header);\n-\n-\t\t\tmd.update((byte) 0);\n-\t\t\tif (deflateStream != null)\n-\t\t\t\tdeflateStream.write((byte) 0);\n-\n-\t\t\twhile (len > 0\n-\t\t\t\t\t&& (n = is.read(buf, 0, (int) Math.min(len, buf.length))) > 0) {\n-\t\t\t\tmd.update(buf, 0, n);\n-\t\t\t\tif (deflateStream != null)\n-\t\t\t\t\tdeflateStream.write(buf, 0, n);\n-\t\t\t\tlen -= n;\n-\t\t\t}\n-\n-\t\t\tif (len != 0)\n-\t\t\t\tthrow new IOException(\"Input did not match supplied length. \"\n-\t\t\t\t\t\t+ len + \" bytes are missing.\");\n-\n-\t\t\tif (deflateStream != null ) {\n-\t\t\t\tdeflateStream.close();\n-\t\t\t\tif (t != null)\n-\t\t\t\t\tt.setReadOnly();\n-\t\t\t}\n-\n-\t\t\tid = ObjectId.fromRaw(md.digest());\n+\t\t\tid = writeObject(type, len, is, deflateStream);\n+\t\t\tif (t != null)\n+\t\t\t\tt.setReadOnly();\n \t\t} finally {\n \t\t\tif (id == null && deflateStream != null) {\n \t\t\t\ttry {\n-- \n1.6.2\n"},{"id":"107955","messageId":"20090313150806.GE22920@spearce.org","threadId":"18282","inReplyTo":"1236844931-12807-1-git-send-email-j16sdiz+freenet@gmail.com","subject":"Re: [PATCH JGIT] Allow writeObject() write to OutputStream","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-13T15:08:06Z","receivedAt":"2009-03-13T15:08:06Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Daniel Cheng (aka SDiZ)\" <j16sdiz+freenet@gmail.com> wrote:\n> \n> This patch make factor out the object writing code in ObjectWriter,\n> allow it to write to any OutputStream. \n> Subclass class may then override\n>   writeObject(final int type, long len, InputStream is, boolean store)\n> to make it write to alternative locations.\n> \n> There are some discussion on devl@freenetproject.org to use raw \n> (uncompressed) object to freenet. This patch allow the testing.\n\nOk, I understand the code as-is, but I'm not sure I understand the\nreasoning for the change, or where you are trying to go with it.\n\nAre you guys talking about making every object a loose object on\nfreenet, and avoiding pack files?  Or making a form of JGit that\naccess a Repository directly stored on freenode?\n\nI ask because there's some folks starting to talk about putting JGit\nonto a distributed hash table sort of system like Hadoop HBase,\nto allow the underlying storage to scale efficiently for really\nbig hosting sites.  I would rather see a pooling of effort here\nthan folks going in different directions.\n \n-- \nShawn.\n"},{"id":"107961","messageId":"ff6a9c820903130847s5ade3df6p31089cbd4b688966@mail.gmail.com","threadId":"18282","inReplyTo":"20090313150806.GE22920@spearce.org","subject":"Re: [PATCH JGIT] Allow writeObject() write to OutputStream","fromName":"Daniel Cheng","fromEmail":"j16sdiz+freenet@gmail.com","sentAt":"2009-03-13T15:47:48Z","receivedAt":"2009-03-13T15:47:48Z","isPatch":true,"sender":{"key":"j16sdiz+freenet@gmail.com","avatar":"https://gravatar.com/avatar/3e796e8a156ee86e305bfc1fbe01608302554ee3b5fa7eb8d1213b8877bfcabd?d=mp&s=160"},"body":"On Fri, Mar 13, 2009 at 11:08 PM, Shawn O. Pearce <spearce@spearce.org> wrote:\n> \"Daniel Cheng (aka SDiZ)\" <j16sdiz+freenet@gmail.com> wrote:\n>>\n>> This patch make factor out the object writing code in ObjectWriter,\n>> allow it to write to any OutputStream.\n>> Subclass class may then override\n>>   writeObject(final int type, long len, InputStream is, boolean store)\n>> to make it write to alternative locations.\n>>\n>> There are some discussion on devl@freenetproject.org to use raw\n>> (uncompressed) object to freenet. This patch allow the testing.\n>\n> Ok, I understand the code as-is, but I'm not sure I understand the\n> reasoning for the change, or where you are trying to go with it.\n\nIn freenet (and most content-addressable network), file can be dropped\nout when unused.\nWe need some method to \"heal\" the lost data.\nThe easiest way is to do this is re-insert the very same file we used\noriginally.\n\nPack files may change on different compression parameter, object order, etc.\nIt need some tricks to get the original file.\nLoose object are immutable, so it is easier to use loose objects.\n\nNothing have finalized yet, we are just evaluating different approaches.\n\nI know loose object are always larger, and have very large number of them.\nSo it may backfire ....\n\n> Are you guys talking about making every object a loose object on\n> freenet, and avoiding pack files?  Or making a form of JGit that\n> access a Repository directly stored on freenode?\n>\n> I ask because there's some folks starting to talk about putting JGit\n> onto a distributed hash table sort of system like Hadoop HBase,\n> to allow the underlying storage to scale efficiently for really\n> big hosting sites.  I would rather see a pooling of effort here\n> than folks going in different directions.\n>\n> --\n> Shawn.\n>\n"}]}