git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[JGIT PATCH 13/15] Patch parse test comparing "git log -p" output to "git log --numstat"

From
Shawn O. Pearce <spearce@spearce.org>
Date
Dec 12, 2008, 02:46 UTC
Message-ID
<1229049981-14152-14-git-send-email-spearce@spearce.org>
In-Reply-To
<1229049981-14152-13-git-send-email-spearce@spearce.org>

By comparing the output of "git log -p", once parsed by our patch parser class, to the output of "git log --numstat" we can be quite certain we are reading the patches from Git with a high degree of accuracy, at least for typical add/remove sorts of changes (no rename detection).

Unfortunately two commits in our history produce an off-by-one bug in git log --numstat. The bug appears to be in log --numstat and not in JGit as git apply --numstat matches JGit's result, and is thus also differing from log --numstat. Since this occurs on only 2 commits out of 1,211 processed during the test I'm not worrying about the difference on these two items. Besides the numbers from JGit and git apply --numstat look to be more correct.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../spearce/jgit/patch/EGitPatchHistoryTest.java   |  221 ++++++++++++++++++++
 1 files changed, 221 insertions(+), 0 deletions(-)
 create mode 100644 org.spearce.jgit.test/exttst/org/spearce/jgit/patch/EGitPatchHistoryTest.java
diff --git a/org.spearce.jgit.test/exttst/org/spearce/jgit/patch/EGitPatchHistoryTest.java b/org.spearce.jgit.test/exttst/org/spearce/jgit/patch/EGitPatchHistoryTest.java
new file mode 100644
index 0000000..d0c2632
--- /dev/null
+++ b/org.spearce.jgit.test/exttst/org/spearce/jgit/patch/EGitPatchHistoryTest.java
@@ -0,0 +1,221 @@
+/*
+ * Copyright (C) 2008, Google Inc.
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.patch;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.UnsupportedEncodingException;
+import java.util.HashMap;
+import java.util.HashSet;
+
+import junit.framework.TestCase;
+
+import org.spearce.jgit.lib.Constants;
+import org.spearce.jgit.util.MutableInteger;
+import org.spearce.jgit.util.RawParseUtils;
+import org.spearce.jgit.util.TemporaryBuffer;
+
+public class EGitPatchHistoryTest extends TestCase {
+	public void testParseHistory() throws Exception {
+		final NumStatReader numstat = new NumStatReader();
+		numstat.read();
+
+		final HashMap<String, HashMap<String, StatInfo>> stats = numstat.stats;
+		assertEquals(1211, stats.size());
+
+		new PatchReader(stats).read();
+	}
+
+	static class StatInfo {
+		int added, deleted;
+	}
+
+	static class PatchReader extends CommitReader {
+		final HashSet<String> offBy1;
+
+		final HashMap<String, HashMap<String, StatInfo>> stats;
+
+		int errors;
+
+		PatchReader(final HashMap<String, HashMap<String, StatInfo>> s)
+				throws IOException {
+			super(new String[] { "-p" });
+			stats = s;
+
+			offBy1 = new HashSet<String>();
+			offBy1.add("9bda5ece6806cd797416eaa47c7b927cc6e9c3b2");
+		}
+
+		@Override
+		void onCommit(String cid, byte[] buf) {
+			final HashMap<String, StatInfo> files = stats.remove(cid);
+			assertNotNull("No files for " + cid, files);
+
+			final Patch p = new Patch();
+			p.parse(buf, 0, buf.length - 1);
+			assertEquals("File count " + cid, files.size(), p.getFiles().size());
+			if (!p.getErrors().isEmpty()) {
+				for (final FormatError e : p.getErrors()) {
+					System.out.println("error " + e.getMessage());
+					System.out.println("  at " + e.getLineText());
+				}
+				dump(buf);
+				fail("Unexpected error in " + cid);
+			}
+
+			for (final FileHeader fh : p.getFiles()) {
+				final String fileName;
+				if (fh.getChangeType() != FileHeader.ChangeType.DELETE)
+					fileName = fh.getNewName();
+				else
+					fileName = fh.getOldName();
+				final StatInfo s = files.remove(fileName);
+				final String nid = fileName + " in " + cid;
+				assertNotNull("No " + nid, s);
+				int added = 0, deleted = 0;
+				for (final HunkHeader h : fh.getHunks()) {
+					added += h.getLinesAdded();
+					deleted += h.getLinesDeleted();
+				}
+
+				if (s.added == added) {
+					//
+				} else if (s.added == added + 1 && offBy1.contains(cid)) {
+					//
+				} else {
+					dump(buf);
+					assertEquals("Added diff in " + nid, s.added, added);
+				}
+
+				if (s.deleted == deleted) {
+					//
+				} else if (s.deleted == deleted + 1 && offBy1.contains(cid)) {
+					//
+				} else {
+					dump(buf);
+					assertEquals("Deleted diff in " + nid, s.deleted, deleted);
+				}
+			}
+			assertTrue("Missed files in " + cid, files.isEmpty());
+		}
+
+		private static void dump(final byte[] buf) {
+			String str;
+			try {
+				str = new String(buf, 0, buf.length - 1, "ISO-8859-1");
+			} catch (UnsupportedEncodingException e) {
+				throw new RuntimeException(e);
+			}
+			System.out.println("<<" + str + ">>");
+		}
+	}
+
+	static class NumStatReader extends CommitReader {
+		final HashMap<String, HashMap<String, StatInfo>> stats = new HashMap<String, HashMap<String, StatInfo>>();
+
+		NumStatReader() throws IOException {
+			super(new String[] { "--numstat" });
+		}
+
+		@Override
+		void onCommit(String commitId, byte[] buf) {
+			final HashMap<String, StatInfo> files = new HashMap<String, StatInfo>();
+			final MutableInteger ptr = new MutableInteger();
+			while (ptr.value < buf.length) {
+				if (buf[ptr.value] == '\n')
+					break;
+				final StatInfo i = new StatInfo();
+				i.added = RawParseUtils.parseBase10(buf, ptr.value, ptr);
+				i.deleted = RawParseUtils.parseBase10(buf, ptr.value + 1, ptr);
+				final int eol = RawParseUtils.nextLF(buf, ptr.value);
+				final String name = RawParseUtils.decode(Constants.CHARSET,
+						buf, ptr.value + 1, eol - 1);
+				files.put(name, i);
+				ptr.value = eol;
+			}
+			stats.put(commitId, files);
+		}
+	}
+
+	static abstract class CommitReader {
+		private Process proc;
+
+		CommitReader(final String[] args) throws IOException {
+			final String[] realArgs = new String[3 + args.length + 1];
+			realArgs[0] = "git";
+			realArgs[1] = "log";
+			realArgs[2] = "--pretty=format:commit %H";
+			System.arraycopy(args, 0, realArgs, 3, args.length);
+			realArgs[3 + args.length] = "a4b98ed15ea5f165a7aa0f2fd2ea6fcce6710925";
+
+			proc = Runtime.getRuntime().exec(realArgs);
+			proc.getOutputStream().close();
+			proc.getErrorStream().close();
+		}
+
+		void read() throws IOException, InterruptedException {
+			final BufferedReader in = new BufferedReader(new InputStreamReader(
+					proc.getInputStream(), "ISO-8859-1"));
+			String commitId = null;
+			TemporaryBuffer buf = null;
+			for (;;) {
+				String line = in.readLine();
+				if (line == null)
+					break;
+				if (line.startsWith("commit ")) {
+					if (buf != null) {
+						buf.close();
+						onCommit(commitId, buf.toByteArray());
+						buf.destroy();
+					}
+					commitId = line.substring("commit ".length());
+					buf = new TemporaryBuffer();
+				} else if (buf != null) {
+					buf.write(line.getBytes("ISO-8859-1"));
+					buf.write('\n');
+				}
+			}
+			in.close();
+			assertEquals(0, proc.waitFor());
+			proc = null;
+		}
+
+		abstract void onCommit(String commitId, byte[] buf);
+	}
+}
-- 
1.6.1.rc2.306.ge5d5e
Previous: Shawn O. PearceNext: Shawn O. Pearce
Message 14 of 22 in “More patch parsing support”
  1. 00/15 More patch parsing supportShawn O. Pearce, Dec 12, 2008
  2. 01/15 Correct use of TemporaryBuffer in PatchShawn O. Pearce, Dec 12, 2008
  3. 02/15 Add tests for TemporaryBufferShawn O. Pearce, Dec 12, 2008
  4. 03/15 Add IntList as a more efficient representation of List<Integer>Shawn O. Pearce, Dec 12, 2008
  5. 04/15 Add lineMap computer to RawParseUtils to index locations of line startsShawn O. Pearce, Dec 12, 2008
  6. 05/15 Define FileHeader.PatchType to report the style of patch usedShawn O. Pearce, Dec 12, 2008
  7. 06/15 Test for non-git binary files and mark them as PatchType.BINARYShawn O. Pearce, Dec 12, 2008
  8. 07/15 Set empty patches with no Git metadata to PatchType.BINARYShawn O. Pearce, Dec 12, 2008
  9. 08/15 Always use the FileHeader buffer during Patch.parseHunksShawn O. Pearce, Dec 12, 2008
  10. 09/15 Parse "GIT binary patch" style patch metadataShawn O. Pearce, Dec 12, 2008
  11. 10/15 Record patch parsing errors for later inspection by applicationsShawn O. Pearce, Dec 12, 2008
  12. 11/15 Fix Patch.parse to honor the end point passed inShawn O. Pearce, Dec 12, 2008
  13. 12/15 Correctly handle hunk headers such as "@@ -0,0 +1 @@"Shawn O. Pearce, Dec 12, 2008
  14. 13/15 Patch parse test comparing "git log -p" output to "git log --numstat"Shawn O. Pearce, Dec 12, 2008
  15. 14/15 Abstract the hunk header testing into a methodShawn O. Pearce, Dec 12, 2008
  16. 15/15 Treat "diff --combined" the same as "diff --cc"Shawn O. Pearce, Dec 12, 2008
  17. Robin RosenbergDec 12, 2008
  18. 15/15 Treat "diff --combined" the same as "diff --cc"Shawn O. Pearce, Dec 12, 2008
  19. Shawn O. PearceDec 12, 2008
  20. Sverre RabbelierDec 12, 2008
  21. Shawn O. PearceDec 12, 2008
  22. Sverre RabbelierDec 12, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.