{"thread":{"id":"15879","subject":"[JGIT PATCH 1/2] Check object connectivity during fetch if fsck is enabled","startedAt":"2008-10-13T17:36:40Z","lastAt":"2008-10-13T17:36:41Z","messageCount":2,"participants":["Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"92948","messageId":"1223919401-15873-1-git-send-email-spearce@spearce.org","threadId":"15879","inReplyTo":null,"subject":"[JGIT PATCH 1/2] Check object connectivity during fetch if fsck is enabled","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-10-13T17:36:40Z","receivedAt":"2008-10-13T17:36:40Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"If we are fetching over a pack oriented connection and we are doing\nobject-level fsck validation we need to also verify the graph is\nfully connected after the fetch is complete.  This additional check\nis necessary to ensure the peer didn't omit objects that we don't\nhave, but which are listed as needing to be present.\n\nOn the walk style fetch connection we can bypass this check, as the\nconnectivity was implicitly verified by the walker as it downloaded\nobjects and built its queue of things to fetch.  Native pack and\nbundle transports however do not have this check built into them,\nand require that we execute the work ourselves.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n\n This is a follow-on to my fsck during fetch series, which is already\n in master.  Making calls to ObjectChecker isn't enough as it does\n not check the DAG's connectivity.\n\n .../jgit/transport/BasePackFetchConnection.java    |    4 +++\n .../spearce/jgit/transport/FetchConnection.java    |   22 ++++++++++++++++++++\n .../org/spearce/jgit/transport/FetchProcess.java   |   13 ++++++++++-\n .../spearce/jgit/transport/TransportBundle.java    |    4 +++\n .../jgit/transport/WalkFetchConnection.java        |    4 +++\n 5 files changed, 45 insertions(+), 2 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java\nindex a542eb7..542a8a9 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java\n@@ -146,6 +146,10 @@ public boolean didFetchIncludeTags() {\n \t\treturn false;\n \t}\n \n+\tpublic boolean didFetchTestConnectivity() {\n+\t\treturn false;\n+\t}\n+\n \tprotected void doFetch(final ProgressMonitor monitor,\n \t\t\tfinal Collection<Ref> want) throws TransportException {\n \t\ttry {\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java\nindex 9d25b0d..d93972d 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java\n@@ -111,4 +111,26 @@ public void fetch(final ProgressMonitor monitor, final Collection<Ref> want)\n \t *         false if tags were not implicitly obtained.\n \t */\n \tpublic boolean didFetchIncludeTags();\n+\n+\t/**\n+\t * Did the last {@link #fetch(ProgressMonitor, Collection)} validate graph?\n+\t * <p>\n+\t * Some transports walk the object graph on the client side, with the client\n+\t * looking for what objects it is missing and requesting them individually\n+\t * from the remote peer. By virtue of completing the fetch call the client\n+\t * implicitly tested the object connectivity, as every object in the graph\n+\t * was either already local or was requested successfully from the peer. In\n+\t * such transports this method returns true.\n+\t * <p>\n+\t * Some transports assume the remote peer knows the Git object graph and is\n+\t * able to supply a fully connected graph to the client (although it may\n+\t * only be transferring the parts the client does not yet have). Its faster\n+\t * to assume such remote peers are well behaved and send the correct\n+\t * response to the client. In such tranports this method returns false.\n+\t * \n+\t * @return true if the last fetch had to perform a connectivity check on the\n+\t *         client side in order to succeed; false if the last fetch assumed\n+\t *         the remote peer supplied a complete graph.\n+\t */\n+\tpublic boolean didFetchTestConnectivity();\n }\n\\ No newline at end of file\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java\nindex 654572d..bb2d051 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java\n@@ -118,7 +118,7 @@ else if (tagopt == TagOpt.FETCH_TAGS)\n \n \t\t\tfinal boolean includedTags;\n \t\t\tif (!askFor.isEmpty() && !askForIsComplete()) {\n-\t\t\t\tconn.fetch(monitor, askFor.values());\n+\t\t\t\tfetchObjects(monitor);\n \t\t\t\tincludedTags = conn.didFetchIncludeTags();\n \n \t\t\t\t// Connection was used for object transfer. If we\n@@ -143,7 +143,7 @@ else if (tagopt == TagOpt.FETCH_TAGS)\n \t\t\t\tif (!askFor.isEmpty() && (!includedTags || !askForIsComplete())) {\n \t\t\t\t\treopenConnection();\n \t\t\t\t\tif (!askFor.isEmpty())\n-\t\t\t\t\t\tconn.fetch(monitor, askFor.values());\n+\t\t\t\t\t\tfetchObjects(monitor);\n \t\t\t\t}\n \t\t\t}\n \t\t} finally {\n@@ -171,6 +171,15 @@ else if (tagopt == TagOpt.FETCH_TAGS)\n \t\t}\n \t}\n \n+\tprivate void fetchObjects(final ProgressMonitor monitor)\n+\t\t\tthrows TransportException {\n+\t\tconn.fetch(monitor, askFor.values());\n+\t\tif (transport.isCheckFetchedObjects()\n+\t\t\t\t&& !conn.didFetchTestConnectivity() && !askForIsComplete())\n+\t\t\tthrow new TransportException(transport.getURI(),\n+\t\t\t\t\t\"peer did not supply a complete object graph\");\n+\t}\n+\n \tprivate void closeConnection() {\n \t\tif (conn != null) {\n \t\t\tconn.close();\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java\nindex 5b321a0..7d38b02 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java\n@@ -165,6 +165,10 @@ private String readLine(final byte[] hdrbuf) throws IOException {\n \t\t\treturn RawParseUtils.decode(Constants.CHARSET, hdrbuf, 0, lf);\n \t\t}\n \n+\t\tpublic boolean didFetchTestConnectivity() {\n+\t\t\treturn false;\n+\t\t}\n+\n \t\t@Override\n \t\tprotected void doFetch(final ProgressMonitor monitor,\n \t\t\t\tfinal Collection<Ref> want) throws TransportException {\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java\nindex 5638454..d089f7b 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java\n@@ -189,6 +189,10 @@ WalkFetchConnection(final WalkTransport wt, final WalkRemoteObjectDatabase w) {\n \t\tworkQueue = new LinkedList<ObjectId>();\n \t}\n \n+\tpublic boolean didFetchTestConnectivity() {\n+\t\treturn true;\n+\t}\n+\n \t@Override\n \tprotected void doFetch(final ProgressMonitor monitor,\n \t\t\tfinal Collection<Ref> want) throws TransportException {\n-- \n1.6.0.2.706.g340fc\n"},{"id":"92949","messageId":"1223919401-15873-2-git-send-email-spearce@spearce.org","threadId":"15879","inReplyTo":"1223919401-15873-1-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 2/2] Add --[no-]thin and --[no-]fsck optiosn to fetch command line tool","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-10-13T17:36:41Z","receivedAt":"2008-10-13T17:36:41Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This way users can force verification on the fly, such as when\nfetching from an untrusted URL pasted on the command line.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n .../src/org/spearce/jgit/pgm/Fetch.java            |   20 ++++++++++++++++++++\n 1 files changed, 20 insertions(+), 0 deletions(-)\n\ndiff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java\nindex e14e213..ad7e08f 100644\n--- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java\n+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java\n@@ -54,6 +54,22 @@\n \t@Option(name = \"--verbose\", aliases = { \"-v\" }, usage = \"be more verbose\")\n \tprivate boolean verbose;\n \n+\t@Option(name = \"--fsck\", usage = \"perform fsck style checks on receive\")\n+\tprivate Boolean fsck;\n+\n+\t@Option(name = \"--no-fsck\")\n+\tvoid nofsck(final boolean ignored) {\n+\t\tfsck = Boolean.FALSE;\n+\t}\n+\n+\t@Option(name = \"--thin\", usage = \"fetch thin pack\")\n+\tprivate Boolean thin;\n+\n+\t@Option(name = \"--no-thin\")\n+\tvoid nothin(final boolean ignored) {\n+\t\tthin = Boolean.FALSE;\n+\t}\n+\n \t@Argument(index = 0, metaVar = \"uri-ish\")\n \tprivate String remote = \"origin\";\n \n@@ -63,6 +79,10 @@\n \t@Override\n \tprotected void run() throws Exception {\n \t\tfinal Transport tn = Transport.open(db, remote);\n+\t\tif (fsck != null)\n+\t\t\ttn.setCheckFetchedObjects(fsck.booleanValue());\n+\t\tif (thin != null)\n+\t\t\ttn.setFetchThin(thin.booleanValue());\n \t\tfinal FetchResult r;\n \t\ttry {\n \t\t\tr = tn.fetch(new TextProgressMonitor(), toget);\n-- \n1.6.0.2.706.g340fc\n"}]}