threads / patch / 17831

patch1/2 : (reworked) Externalizable/Serializable Items

Subject: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

## tl;dr

10 messages between Feb 16, 2009 and Feb 20, 2009. Diffs are folded; open one to read it.

replies: 9people: 5as markdown or json

Nigel Magnay· Feb 16, 2009, 20:12 UTC · lore

JGit is used as a library in external projects such as build tools. Some of the representations of git data structures are useful in these external tools - but - it is often desirable to be able to either persist these objects, or serialize them across the wire.

Apply Externalizable to URIish and RefSpec, and Serializable to ObjectId and RemoteConfig (in order to avoid the undesirable requirement of a public, no-args constructor needed for Externalizable).

 Signed-off-by: Nigel Magnay <nigel.magnay@gmail.com>
---
 .../src/org/spearce/jgit/lib/ObjectId.java         |   22 +++++-
 .../src/org/spearce/jgit/transport/RefSpec.java    |   77 +++++++++++++------
 .../org/spearce/jgit/transport/RemoteConfig.java   |   79 +++++++++++++++++++-
 .../src/org/spearce/jgit/transport/URIish.java     |   29 +++++++-
 4 files changed, 179 insertions(+), 28 deletions(-)
Show changes to 4 files +179 −28

org.spearce.jgit/src/org/spearce/jgit/lib/ObjectId.java, org.spearce.jgit/src/org/spearce/jgit/transport/RefSpec.java, org.spearce.jgit/src/org/spearce/jgit/transport/RemoteConfig.java, org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java

diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectId.java
b/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectId.java
index 52ce0d4..7c3b922 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectId.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/ObjectId.java
@@ -38,6 +38,10 @@

 package org.spearce.jgit.lib;

+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
 import java.io.UnsupportedEncodingException;

 import org.spearce.jgit.util.NB;
@@ -45,7 +49,7 @@
 /**
  * A SHA-1 abstraction.
  */
-public class ObjectId extends AnyObjectId {
+public class ObjectId extends AnyObjectId implements Serializable {
 	private static final ObjectId ZEROID;

 	private static final String ZEROID_STR;
@@ -269,4 +273,20 @@ protected ObjectId(final AnyObjectId src) {
 	public ObjectId toObjectId() {
 		return this;
 	}
+
+	private void writeObject(ObjectOutputStream os)  throws IOException {
+		os.writeInt(w1);
+		os.writeInt(w2);
+		os.writeInt(w3);
+		os.writeInt(w4);
+		os.writeInt(w5);
+	}
+	
+	private void readObject(ObjectInputStream ois)  throws IOException {
+		w1 = ois.readInt();
+		w2 = ois.readInt();
+		w3 = ois.readInt();
+		w4 = ois.readInt();
+		w5 = ois.readInt();
+	}
 }
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/RefSpec.java
b/org.spearce.jgit/src/org/spearce/jgit/transport/RefSpec.java
index 521110b..0ee89b0 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/RefSpec.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/RefSpec.java
@@ -37,6 +37,11 @@

 package org.spearce.jgit.transport;

+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+
 import org.spearce.jgit.lib.Constants;
 import org.spearce.jgit.lib.Ref;

@@ -46,7 +51,7 @@
  * A ref specification provides matching support and limited rules to rewrite a
  * reference in one repository to another reference in another repository.
  */
-public class RefSpec {
+public class RefSpec implements Externalizable {
 	/**
 	 * Suffix for wildcard ref spec component, that indicate matching all refs
 	 * with specified prefix.
@@ -109,30 +114,7 @@ public RefSpec() {
 	 *             the specification is invalid.
 	 */
 	public RefSpec(final String spec) {
-		String s = spec;
-		if (s.startsWith("+")) {
-			force = true;
-			s = s.substring(1);
-		}
-
-		final int c = s.indexOf(':');
-		if (c == 0) {
-			s = s.substring(1);
-			if (isWildcard(s))
-				throw new IllegalArgumentException("Invalid wildcards " + spec);
-			dstName = s;
-		} else if (c > 0) {
-			srcName = s.substring(0, c);
-			dstName = s.substring(c + 1);
-			if (isWildcard(srcName) && isWildcard(dstName))
-				wildcard = true;
-			else if (isWildcard(srcName) || isWildcard(dstName))
-				throw new IllegalArgumentException("Invalid wildcards " + spec);
-		} else {
-			if (isWildcard(s))
-				throw new IllegalArgumentException("Invalid wildcards " + spec);
-			srcName = s;
-		}
+		initializeFromString(spec);
 	}

 	/**
@@ -161,6 +143,42 @@ private RefSpec(final RefSpec p) {
 	}

 	/**
+	 * Initialize the ref specification from a string.
+	 *
+	 * @param spec
+	 *            string describing the specification.
+	 * @throws IllegalArgumentException
+	 *             the specification is invalid.
+	 */
+	private void initializeFromString(final String spec) {
+		srcName = null;
+		String s = spec;
+		if (s.startsWith("+")) {
+			force = true;
+			s = s.substring(1);
+		}
+
+		final int c = s.indexOf(':');
+		if (c == 0) {
+			s = s.substring(1);
+			if (isWildcard(s))
+				throw new IllegalArgumentException("Invalid wildcards " + spec);
+			dstName = s;
+		} else if (c > 0) {
+			srcName = s.substring(0, c);
+			dstName = s.substring(c + 1);
+			if (isWildcard(srcName) && isWildcard(dstName))
+				wildcard = true;
+			else if (isWildcard(srcName) || isWildcard(dstName))
+				throw new IllegalArgumentException("Invalid wildcards " + spec);
+		} else {
+			if (isWildcard(s))
+				throw new IllegalArgumentException("Invalid wildcards " + spec);
+			srcName = s;
+		}
+	}
+	
+	/**
 	 * Check if this specification wants to forcefully update the destination.
 	 *
 	 * @return true if this specification asks for updates without merge tests.
@@ -421,4 +439,13 @@ public String toString() {
 		}
 		return r.toString();
 	}
+
+	public void readExternal(ObjectInput in) throws IOException,
+			ClassNotFoundException {
+		initializeFromString(in.readUTF());	
+	}
+
+	public void writeExternal(ObjectOutput out) throws IOException {
+		out.writeUTF(toString());
+	}
 }
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/RemoteConfig.java
b/org.spearce.jgit/src/org/spearce/jgit/transport/RemoteConfig.java
index 5bbf664..899f73f 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/RemoteConfig.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/RemoteConfig.java
@@ -38,6 +38,10 @@

 package org.spearce.jgit.transport;

+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.Serializable;
 import java.net.URISyntaxException;
 import java.util.ArrayList;
 import java.util.Collections;
@@ -53,7 +57,7 @@
  * describing how refs should be transferred between this repository and the
  * remote repository.
  */
-public class RemoteConfig {
+public class RemoteConfig implements Serializable {
 	private static final String SECTION = "remote";

 	private static final String KEY_URL = "url";
@@ -382,4 +386,77 @@ public TagOpt getTagOpt() {
 	public void setTagOpt(final TagOpt option) {
 		tagopt = option != null ? option : TagOpt.AUTO_FOLLOW;
 	}
+	
+	private void writeObject(ObjectOutputStream os) throws IOException {
+
+	    // Name
+	    os.writeUTF(name);
+	
+	    // Key, Value pairs
+	    for (URIish uri : uris) {
+            os.writeUTF(KEY_URL);
+            os.writeUTF(uri.toPrivateString());
+        }
+
+        for (RefSpec refspec : fetch) {
+            os.writeUTF(KEY_FETCH);
+            os.writeUTF(refspec.toString());
+        }
+
+        for (RefSpec refspec : push) {
+        	os.writeUTF(KEY_PUSH);
+            os.writeUTF(refspec.toString());
+        }
+
+        os.writeUTF(KEY_UPLOADPACK);
+        os.writeUTF(uploadpack);
+
+        os.writeUTF(KEY_RECEIVEPACK);
+        os.writeUTF(receivepack);
+
+        os.writeUTF(KEY_TAGOPT);
+        os.writeUTF(tagopt.option());
+
+        // End marker
+        os.writeUTF("");
+    }
+
+    private void readObject(ObjectInputStream ois) throws IOException {
+		uris = new ArrayList<URIish>();
+        fetch = new ArrayList<RefSpec>();
+        push = new ArrayList<RefSpec>();
+        uploadpack = DEFAULT_UPLOAD_PACK;
+        receivepack = DEFAULT_RECEIVE_PACK;
+
+        name = ois.readUTF();
+
+		for (String key = ois.readUTF(); key.length() > 0; key = ois.readUTF()) {
+			String value = ois.readUTF();
+
+			if (key.equals(KEY_URL)) {
+				try {
+					uris.add(new URIish(value));
+				} catch (URISyntaxException e) {
+					throw new IOException("Invalid URI in RemoteConfig : "
+							+ value);
+				}
+			} else if (key.equals(KEY_FETCH)) {
+				fetch.add(new RefSpec(value));
+
+			} else if (key.equals(KEY_PUSH)) {
+				push.add(new RefSpec(value));
+
+			} else if (key.equals(KEY_UPLOADPACK)) {
+				uploadpack = value;
+
+			} else if (key.equals(KEY_RECEIVEPACK)) {
+				receivepack = value;
+
+			} else if (key.equals(KEY_TAGOPT)) {
+				tagopt = TagOpt.fromOption(value);
+			}
+
+		}
+
+    }
 }
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java
b/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java
index b86e00c..6b85f45 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/URIish.java
@@ -38,6 +38,10 @@

 package org.spearce.jgit.transport;

+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
 import java.net.URISyntaxException;
 import java.net.URL;
 import java.util.regex.Matcher;
@@ -49,7 +53,7 @@
  * RFC 2396 URI's is that no URI encoding/decoding ever takes place. A space or
  * any special character is written as-is.
  */
-public class URIish {
+public class URIish implements Externalizable {
 	private static final Pattern FULL_URI = Pattern
 			.compile("^(?:([a-z][a-z0-9+-]+)://(?:([^/]+?)(?::([^/]+?))?@)?(?:([^/]+?))?(?::(\\d+))?)?((?:[A-Za-z]:)?/.+)$");

@@ -75,6 +79,16 @@
 	 * @throws URISyntaxException
 	 */
 	public URIish(String s) throws URISyntaxException {
+		initializeFromString(s);
+	}
+	
+	/**
+	 * Set fields from string based URI.
+	 *
+	 * @param s
+	 * @throws URISyntaxException
+	 */
+	private void initializeFromString(String s)  throws URISyntaxException {
 		s = s.replace('\\', '/');
 		Matcher matcher = FULL_URI.matcher(s);
 		if (matcher.matches()) {
@@ -357,4 +371,17 @@ private String format(final boolean includePassword) {

 		return r.toString();
 	}
+
+	public void readExternal(ObjectInput in) throws IOException,
+			ClassNotFoundException {
+	    try {
+			initializeFromString(in.readUTF());
+		} catch (URISyntaxException e) {
+			throw new IOException("Incorrect format URI");
+		}
+	}
+
+	public void writeExternal(ObjectOutput out) throws IOException {
+		out.writeUTF(format(true));
+	}
 }
-- 
1.6.0.2
Johannes Schindelin· Feb 16, 2009, 20:16 UTC · re: Nigel Magnay · lore

Re: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

Hi,
On Mon, 16 Feb 2009, Nigel Magnay wrote:
> JGit is used as a library in external projects such as build tools.
> Some of the representations of git data structures are useful
> in these external tools - but - it is often desirable to be able to
> either persist these objects, or serialize them across the wire.
I am still missing the description of the format.
>  Signed-off-by: Nigel Magnay <nigel.magnay@gmail.com>
And I wonder what that leading space is all about.

Ciao, Dscho

Shawn O. Pearce· Feb 18, 2009, 16:30 UTC · re: Nigel Magnay · lore

Re: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

Nigel Magnay <nigel.magnay@gmail.com> wrote:
> +public class ObjectId extends AnyObjectId implements Serializable {
We should define our own serialVersionUID:
  private static final long serialVersionUID = 1L;
is good enough to make Java happy.
Show 7 quoted lines
> @@ -269,4 +273,20 @@ protected ObjectId(final AnyObjectId src) {
>  	public ObjectId toObjectId() {
>  		return this;
>  	}
> +
> +	private void writeObject(ObjectOutputStream os)  throws IOException {
> +	private void readObject(ObjectInputStream ois)  throws IOException {
Minor nit: Only 1 space between ) and throws, please.
Show 11 quoted lines
> diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/RemoteConfig.java
> b/org.spearce.jgit/src/org/spearce/jgit/transport/RemoteConfig.java
> index 5bbf664..899f73f 100644
> --- a/org.spearce.jgit/src/org/spearce/jgit/transport/RemoteConfig.java
> +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/RemoteConfig.java
> @@ -53,7 +57,7 @@
>   * describing how refs should be transferred between this repository and the
>   * remote repository.
>   */
> -public class RemoteConfig {
> +public class RemoteConfig implements Serializable {
Please set a serialVersionUID.
Show 17 quoted lines
> @@ -382,4 +386,77 @@ public TagOpt getTagOpt() {
>  	public void setTagOpt(final TagOpt option) {
>  		tagopt = option != null ? option : TagOpt.AUTO_FOLLOW;
>  	}
> +	
> +	private void writeObject(ObjectOutputStream os) throws IOException {
> +
> +	    // Name
> +	    os.writeUTF(name);
> +	
> +	    // Key, Value pairs
> +	    for (URIish uri : uris) {
> +            os.writeUTF(KEY_URL);
> +            os.writeUTF(uri.toPrivateString());
> +        }
> +
> +        for (RefSpec refspec : fetch) {

There is some sort of whitespace damage here, the second for loop is not lined up at the same starting column as the first for loop. My guess is, you have tabs in here. We only indent with spaces.

Show 8 quoted lines
> +            os.writeUTF(KEY_FETCH);
> +            os.writeUTF(refspec.toString());
> +        }
> +
> +        for (RefSpec refspec : push) {
> +        	os.writeUTF(KEY_PUSH);
> +            os.writeUTF(refspec.toString());
> +        }

Should we maybe allow RefSpec to serialize itself with os.writeObject() rather than using writeUTF() directly?

FWIW, I did find this new implementation to be much easier to read. Thanks.

-- 
Shawn.
Robin Rosenberg· Feb 18, 2009, 20:59 UTC · re: Shawn O. Pearce · lore

Re: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

Shawn wrote:
Show 11 quoted lines
> > +            os.writeUTF(KEY_FETCH);
> > +            os.writeUTF(refspec.toString());
> > +        }
> > +
> > +        for (RefSpec refspec : push) {
> > +        	os.writeUTF(KEY_PUSH);
> > +            os.writeUTF(refspec.toString());
> > +        }
> 
> Should we maybe allow RefSpec to serialize itself with
> os.writeObject() rather than using writeUTF() directly?

Doesn't the style above make it easy to define and document a format that is easy for non-java programs to write and read, while writeObject introduces java-centric stuff (depending on the full class name etc).

-- robin
Shawn O. Pearce· Feb 18, 2009, 21:48 UTC · re: Robin Rosenberg · lore

Re: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
Show 17 quoted lines
> Shawn wrote:
> > > +            os.writeUTF(KEY_FETCH);
> > > +            os.writeUTF(refspec.toString());
> > > +        }
> > > +
> > > +        for (RefSpec refspec : push) {
> > > +        	os.writeUTF(KEY_PUSH);
> > > +            os.writeUTF(refspec.toString());
> > > +        }
> > 
> > Should we maybe allow RefSpec to serialize itself with
> > os.writeObject() rather than using writeUTF() directly?
> 
> Doesn't the style above make it easy to define and document
> a format that is easy for non-java programs to write and read,
> while writeObject introduces java-centric stuff (depending on
> the full class name etc).
Non-Java reading a Java serialization stream?  Seriously?
-- 
Shawn.
Sverre Rabbelier· Feb 18, 2009, 22:19 UTC · re: Shawn O. Pearce · lore

Re: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

On Wed, Feb 18, 2009 at 22:48, Shawn O. Pearce <spearce@spearce.org> wrote:
> Non-Java reading a Java serialization stream?  Seriously?

That would be a contender for UJSFWIINI :P (with JS standing for Java Serialization).

-- 
Cheers,

Sverre Rabbelier
Robin Rosenberg· Feb 18, 2009, 23:21 UTC · re: Shawn O. Pearce · lore

Re: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

onsdag 18 februari 2009 22:48:59 skrev "Shawn O. Pearce" <spearce@spearce.org>:
Show 20 quoted lines
> Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
> > Shawn wrote:
> > > > +            os.writeUTF(KEY_FETCH);
> > > > +            os.writeUTF(refspec.toString());
> > > > +        }
> > > > +
> > > > +        for (RefSpec refspec : push) {
> > > > +        	os.writeUTF(KEY_PUSH);
> > > > +            os.writeUTF(refspec.toString());
> > > > +        }
> > > 
> > > Should we maybe allow RefSpec to serialize itself with
> > > os.writeObject() rather than using writeUTF() directly?
> > 
> > Doesn't the style above make it easy to define and document
> > a format that is easy for non-java programs to write and read,
> > while writeObject introduces java-centric stuff (depending on
> > the full class name etc).
> 
> Non-Java reading a Java serialization stream?  Seriously?

No, that was my objection to using writeObject, as that make it a Java-only stream, but then it might not be worth doing it via the serialization mechanism.

-- robin
Shawn O. Pearce· Feb 18, 2009, 23:27 UTC · re: Robin Rosenberg · lore

Re: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:
Show 7 quoted lines
> onsdag 18 februari 2009 22:48:59 skrev "Shawn O. Pearce" <spearce@spearce.org>:
> > 
> > Non-Java reading a Java serialization stream?  Seriously?
> 
> No, that was my objection to using writeObject, as that make
> it a Java-only stream, but then it might not be worth doing
> it via the serialization mechanism.

IMHO, if we are talking about either java.io.Serializable or java.io.Externalizable, there's no point in considering a non Java peer.

If you want a non-Java format, we'd need to consider a much more neutral encoding, like Google's protobuf, or *shudder* XML/JSON, or cooking up our own format.

That wasn't this thread started with. The original poster just wanted an easy way to serialize some basic data types from JGit, as part of some higher level stream being done in the container application. Since that higher level stream is a apparently a Java object serialization stream, we just need to match that.

-- 
Shawn.
Nigel Magnay· Feb 20, 2009, 09:28 UTC · lore

Re: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

On Wed, Feb 18, 2009 at 4:30 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
Show 9 quoted lines
>
> Nigel Magnay <nigel.magnay@gmail.com> wrote:
> > +public class ObjectId extends AnyObjectId implements Serializable {
>
> We should define our own serialVersionUID:
>
>  private static final long serialVersionUID = 1L;
>
> is good enough to make Java happy.

I'll re-roll. Actually both that and the whitespace damage I'm partially blaming on eclipse. Eclipse curiously removes the warning to declare serialVersionUID if you declare both writeObject and readObject - that's wrong!

Also the formatting came from .settings/org.eclipse.core.resources.prefs which has a tab policy of "Tabs only" which has overridden my environment setting of spaces-only.

I can modify it to spaces only - but the comment in 9268ced9d38 talks about spaces-per-tab..?

Shawn O. Pearce· Feb 20, 2009, 20:32 UTC · lore

Re: [JGIT PATCH] 1/2 : (reworked) Externalizable/Serializable Items

Nigel Magnay <nigel.magnay@gmail.com> wrote:
Show 16 quoted lines
> On Wed, Feb 18, 2009 at 4:30 PM, Shawn O. Pearce <spearce@spearce.org>wrote:
> 
> > Nigel Magnay <nigel.magnay@gmail.com> wrote:
> > > +public class ObjectId extends AnyObjectId implements Serializable {
> >
> > We should define our own serialVersionUID:
> >
> >  private static final long serialVersionUID = 1L;
> >
> > is good enough to make Java happy.
> >
> 
> I'll re-roll. Actually both that and the whitespace damage I'm partially
> blaming on eclipse. Eclipse curiously removes the warning to declare
> serialVersionUID if you declare both writeObject and readObject - that's
> wrong!
Yikes.  What a nice feature.
 
Show 6 quoted lines
> Also the formatting came from .settings/org.eclipse.core.resources.prefs
> which has a tab policy of "Tabs only" which has overridden my environment
> setting of spaces-only.
>
> I can modify it to spaces only - but the comment in 9268ced9d38 talks about
> spaces-per-tab..?
Oh, yea, my bad.

We prefer tabs, and only tabs, and we only indent the leading part of the line, we never try to "line up" columns of variables (for example).

I guess there was a mixture here. Not sure why. Sometimes I've seen Eclipse not immediately honor the per-project settings files. Closing or deleting and re-importing the project usually fixes it, but not always. :-|

-- 
Shawn.

← back to recent threads