threads / patch / 18282

patchAllow writeObject() write to OutputStream

Subject: [PATCH JGIT] Allow writeObject() write to OutputStream

## tl;dr

3 messages between Mar 12, 2009 and Mar 13, 2009. Diffs are folded; open one to read it.

replies: 2people: 2as markdown or json

Daniel Cheng (aka SDiZ)· Mar 12, 2009, 08:02 UTC · lore
Signed-off-by: Daniel Cheng (aka SDiZ) <j16sdiz+freenet@gmail.com>
---
This patch make factor out the object writing code in ObjectWriter,
allow it to write to any OutputStream. 
Subclass class may then override
  writeObject(final int type, long len, InputStream is, boolean store)
to make it write to alternative locations.

There are some discussion on devl@freenetproject.org to use raw (uncompressed) object to freenet. This patch allow the testing.

 .../src/org/spearce/jgit/lib/ObjectWriter.java     |   93 +++++++++++---------
 1 files changed, 50 insertions(+), 43 deletions(-)
Show changes to org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java +50 −43
diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java b/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java
index 546cc68..97acae4 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectWriter.java
@@ -45,6 +45,7 @@
 import java.io.FileOutputStream;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.OutputStream;
 import java.io.OutputStreamWriter;
 import java.security.MessageDigest;
 import java.util.zip.Deflater;
@@ -297,8 +298,52 @@ public ObjectId computeBlobSha1(final long len, final InputStream is)
 		return writeObject(Constants.OBJ_BLOB, len, is, false);
 	}
 
-	ObjectId writeObject(final int type, long len, final InputStream is,
-			boolean store) throws IOException {
+	protected ObjectId writeObject(final int type, long len,
+			final InputStream is, final OutputStream deflateStream)
+			throws IOException {
+		md.reset();
+
+		byte[] header;
+		int n;
+
+		header = Constants.encodedTypeString(type);
+		md.update(header);
+		if (deflateStream != null)
+			deflateStream.write(header);
+
+		md.update((byte) ' ');
+		if (deflateStream != null)
+			deflateStream.write((byte) ' ');
+
+		header = Constants.encodeASCII(len);
+		md.update(header);
+		if (deflateStream != null)
+			deflateStream.write(header);
+
+		md.update((byte) 0);
+		if (deflateStream != null)
+			deflateStream.write((byte) 0);
+
+		while (len > 0
+				&& (n = is.read(buf, 0, (int) Math.min(len, buf.length))) > 0) {
+			md.update(buf, 0, n);
+			if (deflateStream != null)
+				deflateStream.write(buf, 0, n);
+			len -= n;
+		}
+
+		if (len != 0)
+			throw new IOException("Input did not match supplied length. " + len
+					+ " bytes are missing.");
+
+		if (deflateStream != null)
+			deflateStream.close();
+
+		return ObjectId.fromRaw(md.digest());
+	}
+
+	protected ObjectId writeObject(final int type, long len,
+			final InputStream is, boolean store) throws IOException {
 		final File t;
 		final DeflaterOutputStream deflateStream;
 		final FileOutputStream fileStream;
@@ -312,7 +357,6 @@ ObjectId writeObject(final int type, long len, final InputStream is,
 			fileStream = null;
 		}
 
-		md.reset();
 		if (store) {
 			def.reset();
 			deflateStream = new DeflaterOutputStream(fileStream, def);
@@ -320,46 +364,9 @@ ObjectId writeObject(final int type, long len, final InputStream is,
 			deflateStream = null;
 
 		try {
-			byte[] header;
-			int n;
-
-			header = Constants.encodedTypeString(type);
-			md.update(header);
-			if (deflateStream != null)
-				deflateStream.write(header);
-
-			md.update((byte) ' ');
-			if (deflateStream != null)
-				deflateStream.write((byte) ' ');
-
-			header = Constants.encodeASCII(len);
-			md.update(header);
-			if (deflateStream != null)
-				deflateStream.write(header);
-
-			md.update((byte) 0);
-			if (deflateStream != null)
-				deflateStream.write((byte) 0);
-
-			while (len > 0
-					&& (n = is.read(buf, 0, (int) Math.min(len, buf.length))) > 0) {
-				md.update(buf, 0, n);
-				if (deflateStream != null)
-					deflateStream.write(buf, 0, n);
-				len -= n;
-			}
-
-			if (len != 0)
-				throw new IOException("Input did not match supplied length. "
-						+ len + " bytes are missing.");
-
-			if (deflateStream != null ) {
-				deflateStream.close();
-				if (t != null)
-					t.setReadOnly();
-			}
-
-			id = ObjectId.fromRaw(md.digest());
+			id = writeObject(type, len, is, deflateStream);
+			if (t != null)
+				t.setReadOnly();
 		} finally {
 			if (id == null && deflateStream != null) {
 				try {

-- 
1.6.2
Shawn O. Pearce· Mar 13, 2009, 15:08 UTC · re: Daniel Cheng (aka SDiZ) · lore

Re: [PATCH JGIT] Allow writeObject() write to OutputStream

"Daniel Cheng (aka SDiZ)" <j16sdiz+freenet@gmail.com> wrote:
Show 9 quoted lines
> 
> This patch make factor out the object writing code in ObjectWriter,
> allow it to write to any OutputStream. 
> Subclass class may then override
>   writeObject(final int type, long len, InputStream is, boolean store)
> to make it write to alternative locations.
> 
> There are some discussion on devl@freenetproject.org to use raw 
> (uncompressed) object to freenet. This patch allow the testing.

Ok, I understand the code as-is, but I'm not sure I understand the reasoning for the change, or where you are trying to go with it.

Are you guys talking about making every object a loose object on freenet, and avoiding pack files? Or making a form of JGit that access a Repository directly stored on freenode?

I ask because there's some folks starting to talk about putting JGit
onto a distributed hash table sort of system like Hadoop HBase,
to allow the underlying storage to scale efficiently for really
big hosting sites.  I would rather see a pooling of effort here
than folks going in different directions.
 
-- 
Shawn.
Daniel Cheng· Mar 13, 2009, 15:47 UTC · re: Shawn O. Pearce · lore

Re: [PATCH JGIT] Allow writeObject() write to OutputStream

On Fri, Mar 13, 2009 at 11:08 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
Show 13 quoted lines
> "Daniel Cheng (aka SDiZ)" <j16sdiz+freenet@gmail.com> wrote:
>>
>> This patch make factor out the object writing code in ObjectWriter,
>> allow it to write to any OutputStream.
>> Subclass class may then override
>>   writeObject(final int type, long len, InputStream is, boolean store)
>> to make it write to alternative locations.
>>
>> There are some discussion on devl@freenetproject.org to use raw
>> (uncompressed) object to freenet. This patch allow the testing.
>
> Ok, I understand the code as-is, but I'm not sure I understand the
> reasoning for the change, or where you are trying to go with it.

In freenet (and most content-addressable network), file can be dropped out when unused. We need some method to "heal" the lost data. The easiest way is to do this is re-insert the very same file we used originally.

Pack files may change on different compression parameter, object order, etc. It need some tricks to get the original file. Loose object are immutable, so it is easier to use loose objects.

Nothing have finalized yet, we are just evaluating different approaches.

I know loose object are always larger, and have very large number of them. So it may backfire ....

Show 13 quoted lines
> Are you guys talking about making every object a loose object on
> freenet, and avoiding pack files?  Or making a form of JGit that
> access a Repository directly stored on freenode?
>
> I ask because there's some folks starting to talk about putting JGit
> onto a distributed hash table sort of system like Hadoop HBase,
> to allow the underlying storage to scale efficiently for really
> big hosting sites.  I would rather see a pooling of effort here
> than folks going in different directions.
>
> --
> Shawn.
>

← back to recent threads