{"thread":{"id":"18940","subject":"JGit cache bug","startedAt":"2009-04-18T17:05:54Z","lastAt":"2009-04-20T14:32:56Z","messageCount":6,"participants":["Robin Rosenberg","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"111599","messageId":"200904181905.55119.robin.rosenberg.lists@dewire.com","threadId":"18940","inReplyTo":null,"subject":"JGit cache bug","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-04-18T17:05:54Z","receivedAt":"2009-04-18T17:05:54Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"This is fun... ~\n\nI cannot figure why things don't work with Patch #4? We are not\neven multithreaded.\n\nThis is related to trying to fix http://code.google.com/p/egit/issues/detail?id=79 which is related to all\nsorts of NPE problems that may occur in Eclipse when working with\nEGit and C/msysgit in parallell.\n\nPlatform info:\n\n# java version\njava version \"1.6.0_0\"\nIcedTea6 1.3.1 (6b12-0ubuntu6.4) Runtime Environment (build 1.6.0_0-b12)\nOpenJDK Server VM (build 1.6.0_0-b12, mixed mode)\n\n# uname -a\nLinux sleipner 2.6.27-14-generic #1 SMP Wed Apr 15 18:59:16 UTC 2009 i686 GNU/Linux\n\nI've tried to insert GC and sleep calls.\n\n-- robin\n\n\nFrom c557d6900d4952f5bee0d8dcab82a1730ba549eb Mon Sep 17 00:00:00 2001\nFrom: Robin Rosenberg <robin.rosenberg@dewire.com>\nDate: Fri, 17 Apr 2009 09:34:39 +0200\nSubject: [EGIT PATCH 1/4] Rescan for packs once when failing to lookup an id.\n\nSigned-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>\n---\n .../src/org/spearce/jgit/lib/Repository.java       |   31 ++++++++++++++-----\n 1 files changed, 23 insertions(+), 8 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java\nindex cfd92b8..8ff3b63 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java\n@@ -291,6 +291,21 @@ public ObjectLoader openObject(final AnyObjectId id)\n \t */\n \tpublic ObjectLoader openObject(final WindowCursor curs, final AnyObjectId id)\n \t\t\tthrows IOException {\n+\t\tObjectLoader loader = openObjectInPacks(curs, id);\n+\t\tif (loader != null)\n+\t\t\treturn loader;\n+\t\ttry {\n+\t\t\treturn new UnpackedObjectLoader(this, id);\n+\t\t} catch (FileNotFoundException fnfe) {\n+\t\t\tSystem.out.println(new java.util.Date().toString() + \": failed to load \" + id + \", rescanning for new packs\");\n+\t\t\tif (scanForPacks())\n+\t\t\t\treturn openObjectInPacks(curs, id);\n+\t\t\treturn null;\n+\t\t}\n+\t}\n+\n+\tprivate ObjectLoader openObjectInPacks(final WindowCursor curs, final AnyObjectId id)\n+\t\t\tthrows IOException {\n \t\tfinal PackFile[] packs = packs();\n \t\tint k = packs.length;\n \t\twhile (k > 0) {\n@@ -298,11 +313,7 @@ public ObjectLoader openObject(final WindowCursor curs, final AnyObjectId id)\n \t\t\tif (ol != null)\n \t\t\t\treturn ol;\n \t\t}\n-\t\ttry {\n-\t\t\treturn new UnpackedObjectLoader(this, id);\n-\t\t} catch (FileNotFoundException fnfe) {\n-\t\t\treturn null;\n-\t\t}\n+\t\treturn null;\n \t}\n \n \t/**\n@@ -813,10 +824,11 @@ synchronized (this) {\n \t}\n \n \t/**\n-\t * Scan the object dirs, including alternates for packs\n-\t * to use.\n+\t * Scan the object dirs, including alternates for packs to use.\n+\t * \n+\t * @return true if new packs were found\n \t */\n-\tpublic void scanForPacks() {\n+\tpublic boolean scanForPacks() {\n \t\tfinal ArrayList<PackFile> p = new ArrayList<PackFile>();\n \t\tp.addAll(Arrays.asList(packs()));\n \t\tfor (final File d : objectsDirs())\n@@ -824,9 +836,11 @@ public void scanForPacks() {\n \t\tfinal PackFile[] arr = new PackFile[p.size()];\n \t\tp.toArray(arr);\n \t\tArrays.sort(arr, PackFile.SORT);\n+\t\tboolean ret = packFileList.length != arr.length;\n \t\tsynchronized (this) {\n \t\t\tpackFileList = arr;\n \t\t}\n+\t\treturn ret;\n \t}\n \n \tprivate void scanForPacks(final File packDir, Collection<PackFile> packList) {\n@@ -1200,5 +1214,6 @@ synchronized (allListeners) {\n \tpublic void scanForRepoChanges() throws IOException {\n \t\tgetAllRefs(); // This will look for changes to refs\n \t\tgetIndex(); // This will detect changes in the index\n+\t\tscanForPacks(); // This will detect new packs\n \t}\n }\n-- \n1.6.2.2.446.gfbdc0\n\n\n\nFrom 92d37694970963ad6d0d5fbf4ec811630a58df90 Mon Sep 17 00:00:00 2001\nFrom: Robin Rosenberg <robin.rosenberg@dewire.com>\nDate: Sat, 18 Apr 2009 16:15:28 +0200\nSubject: [EGIT PATCH 2/4] Add test cases dedicated to the WindowCache\n\n---\n .../tst/org/spearce/jgit/lib/WindowCacheTest.java  |   72 ++++++++++++++++++++\n 1 files changed, 72 insertions(+), 0 deletions(-)\n create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\nnew file mode 100644\nindex 0000000..a438d46\n--- /dev/null\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n@@ -0,0 +1,72 @@\n+package org.spearce.jgit.lib;\n+\n+import java.io.ByteArrayInputStream;\n+import java.io.File;\n+import java.io.FileOutputStream;\n+import java.io.IOException;\n+\n+import org.spearce.jgit.errors.IncorrectObjectTypeException;\n+import org.spearce.jgit.errors.MissingObjectException;\n+import org.spearce.jgit.revwalk.RevObject;\n+import org.spearce.jgit.revwalk.RevWalk;\n+\n+public class WindowCacheTest extends RepositoryTestCase {\n+\n+\tpublic void setUp() throws Exception {\n+\t\tWindowCacheConfig windowCacheConfig = new WindowCacheConfig();\n+\t\twindowCacheConfig.setPackedGitOpenFiles(1);\n+\t\tWindowCache.reconfigure(windowCacheConfig);\n+\t\tsuper.setUp();\n+\t}\n+\n+\tprotected void tearDown() throws Exception {\n+\t\tsuper.tearDown();\n+\t\tWindowCacheConfig windowCacheConfig = new WindowCacheConfig();\n+\t\tWindowCache.reconfigure(windowCacheConfig);\n+\t}\n+\n+\t/*\n+\t * Add pack, find objects in new pack Replace packs, find objects in another\n+\t * pack than original, don't choke on lost pack\n+\t */\n+\tpublic void testObjectInNewPack() throws IncorrectObjectTypeException,\n+\t\t\tIOException {\n+\t\tRepository extra = createNewEmptyRepo();\n+\t\tRevObject o1 = writeBlob(extra, db,\n+\t\t\t\t\"nihuioijoisuoiudyosyudoiuyusyuoiyewData\");\n+\t\tPackWriter packWriter = new PackWriter(extra, new TextProgressMonitor());\n+\t\tpackWriter.addObject(o1);\n+\t\tObjectId name = packWriter.computeName();\n+\t\tFile packFileName = fullPackFileName(db, name, \".pack\");\n+\t\tFileOutputStream pos = new FileOutputStream(packFileName);\n+\t\tpackWriter.writePack(pos);\n+\t\tpos.close();\n+\t\tFile idxfname = fullPackFileName(db, name, \".idx\");\n+\t\tFileOutputStream ios = new FileOutputStream(idxfname);\n+\t\tpackWriter.writeIndex(ios);\n+\t\tios.close();\n+\t\tassertEquals(o1.name(), new RevWalk(db).parseAny(o1).name());\n+\t}\n+\n+\tprivate File fullPackFileName(Repository repository, ObjectId hash,\n+\t\t\tString suffix) {\n+\t\treturn new File(new File(repository.getObjectsDirectory(), \"pack\"),\n+\t\t\t\t\"pack-\" + hash.name() + suffix);\n+\t}\n+\n+\tprivate RevObject writeBlob(final Repository repo, final Repository notIn,\n+\t\t\tfinal String data) throws IOException {\n+\t\tRevWalk revWalk = new RevWalk(repo);\n+\t\tbyte[] bytes = data.getBytes(Constants.CHARACTER_ENCODING);\n+\t\tObjectId id = new ObjectWriter(repo).writeBlob(bytes.length,\n+\t\t\t\tnew ByteArrayInputStream(bytes));\n+\t\ttry {\n+\t\t\tassertNull(\n+\t\t\t\t\t\"Oops, We want new objects that we do not have yet, for this test!\",\n+\t\t\t\t\tnew RevWalk(notIn).parseAny(id));\n+\t\t} catch (MissingObjectException e) {\n+\t\t\t// Ok\n+\t\t}\n+\t\treturn revWalk.lookupBlob(id);\n+\t}\n+}\n-- \n1.6.2.2.446.gfbdc0\n\n\n\nFrom 81f9174ebf605030ea95d18bb81dc1ece98bf290 Mon Sep 17 00:00:00 2001\nFrom: Robin Rosenberg <robin.rosenberg@dewire.com>\nDate: Sat, 18 Apr 2009 18:29:23 +0200\nSubject: [EGIT PATCH 3/4] Fails like this in the debugger (expected), but runs ok outside the debugger.\n\nBreakpoints in the following places. Wait a second or two everytime it stops\n\nWindowCacheTest [line: 101] - testObjectMovedToNewPack()\nWindowedFile [line: 283] - cacheOpen()\nWindowedFile [line: 301] - cacheClose()\n\njava.io.FileNotFoundException: /home/me/SW/EGIT/org.spearce.jgit.test/trash/trash1240071609317.0/.git/objects/pack/pack-fc4dadd6673a2740f43468f868ba5dd39fdeba69.pack (Filen eller katalogen finns inte)\n\tat java.io.RandomAccessFile.open(Native Method)\n\tat java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)\n\tat org.spearce.jgit.lib.WindowedFile.cacheOpen(WindowedFile.java:283)\n\tat org.spearce.jgit.lib.WindowCache.getImpl(WindowCache.java:249)\n\tat org.spearce.jgit.lib.WindowCache.get(WindowCache.java:222)\n\tat org.spearce.jgit.lib.WindowCursor.pin(WindowCursor.java:148)\n\tat org.spearce.jgit.lib.WindowCursor.copy(WindowCursor.java:82)\n\tat org.spearce.jgit.lib.WindowedFile.read(WindowedFile.java:176)\n\tat org.spearce.jgit.lib.WindowedFile.readFully(WindowedFile.java:203)\n\tat org.spearce.jgit.lib.PackFile.reader(PackFile.java:311)\n\tat org.spearce.jgit.lib.PackFile.get(PackFile.java:147)\n\tat org.spearce.jgit.lib.Repository.openObjectInPacks(Repository.java:312)\n\tat org.spearce.jgit.lib.Repository.openObject(Repository.java:294)\n\tat org.spearce.jgit.revwalk.RevWalk.parseAny(RevWalk.java:702)\n\tat org.spearce.jgit.lib.WindowCacheTest.testObjectMovedToNewPack(WindowCacheTest.java:115)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat java.lang.reflect.Method.invoke(Method.java:597)\n\tat junit.framework.TestCase.runTest(TestCase.java:164)\n\tat junit.framework.TestCase.runBare(TestCase.java:130)\n\tat junit.framework.TestResult$1.protect(TestResult.java:106)\n\tat junit.framework.TestResult.runProtected(TestResult.java:124)\n\tat junit.framework.TestResult.run(TestResult.java:109)\n\tat junit.framework.TestCase.run(TestCase.java:120)\n\tat org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)\n\tat org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)\n---\n .../tst/org/spearce/jgit/lib/WindowCacheTest.java  |   53 ++++++++++++++++++++\n 1 files changed, 53 insertions(+), 0 deletions(-)\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\nindex a438d46..0cd55c9 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n@@ -48,6 +48,59 @@ public void testObjectInNewPack() throws IncorrectObjectTypeException,\n \t\tassertEquals(o1.name(), new RevWalk(db).parseAny(o1).name());\n \t}\n \n+\t/*\n+\t * Add pack, find objects in new pack Replace packs, find objects in another\n+\t * pack than original, don't choke on lost pack\n+\t */\n+\tpublic void testObjectMovedToNewPack() throws IncorrectObjectTypeException,\n+\t\t\tIOException {\n+\t\tRepository extra = createNewEmptyRepo();\n+\t\tRevObject o1 = writeBlob(extra, db,\n+\t\t\t\t\"nihuioijoisuoiudyosyudoiuyusyuoiyewData1\");\n+\t\tPackWriter packWriter1 = new PackWriter(extra,\n+\t\t\t\tnew TextProgressMonitor());\n+\t\tpackWriter1.addObject(o1);\n+\t\tObjectId name1 = packWriter1.computeName();\n+\t\tFile packFileName1 = fullPackFileName(db, name1, \".pack\");\n+\t\tFileOutputStream pos1 = new FileOutputStream(packFileName1);\n+\t\tpackWriter1.writePack(pos1);\n+\t\tpos1.close();\n+\t\tFile idxfname1 = fullPackFileName(db, name1, \".idx\");\n+\t\tFileOutputStream ios1 = new FileOutputStream(idxfname1);\n+\t\tpackWriter1.writeIndex(ios1);\n+\t\tios1.close();\n+\t\tassertEquals(o1.name(), new RevWalk(db).parseAny(o1).name());\n+\n+\t\t// Ok, lets repack this repo\n+\t\t// create an additional object\n+\t\tRevObject o2 = writeBlob(extra, db,\n+\t\t\t\t\"nihuioijoisuoiudyosyudoiuyusyuoiyewData2\");\n+\t\tPackWriter packWriter2 = new PackWriter(extra,\n+\t\t\t\tnew TextProgressMonitor());\n+\t\tpackWriter2.addObject(o1);\n+\t\tpackWriter2.addObject(o2);\n+\t\tObjectId name2 = packWriter2.computeName();\n+\t\tFile packFileName2 = fullPackFileName(db, name2, \".pack\");\n+\t\tFileOutputStream pos2 = new FileOutputStream(packFileName2);\n+\t\tpackWriter2.writePack(pos2);\n+\t\tpos2.close();\n+\t\tFile idxfname2 = fullPackFileName(db, name2, \".idx\");\n+\t\tFileOutputStream ios2 = new FileOutputStream(idxfname2);\n+\t\tpackWriter2.writeIndex(ios2);\n+\t\tios2.close();\n+\t\tassertEquals(o2.name(), new RevWalk(db).parseAny(o2).name());\n+\n+\t\tWindowCacheConfig windowCacheConfig = new WindowCacheConfig();\n+\t\twindowCacheConfig.setPackedGitOpenFiles(1);\n+\t\tWindowCache.reconfigure(windowCacheConfig);\n+\t\tpackFileName1.delete();\n+\t\tidxfname1.delete();\n+\n+\t\t// Now here is the interesting thing. Will git figure the new\n+\t\t// object exists in the new pack, and not the old one.\n+\t\tassertEquals(o1.name(), new RevWalk(db).parseAny(o1).name());\n+\t}\n+\n \tprivate File fullPackFileName(Repository repository, ObjectId hash,\n \t\t\tString suffix) {\n \t\treturn new File(new File(repository.getObjectsDirectory(), \"pack\"),\n-- \n1.6.2.2.446.gfbdc0\n\n\n\nFrom 44638f9060d845a208638ac75d392a168b87c0df Mon Sep 17 00:00:00 2001\nFrom: Robin Rosenberg <robin.rosenberg@dewire.com>\nDate: Sat, 18 Apr 2009 19:00:32 +0200\nSubject: [EGIT PATCH 4/4] Simulate breakpoints\n\n---\n .../tst/org/spearce/jgit/lib/WindowCacheTest.java  |    7 +++++++\n .../src/org/spearce/jgit/lib/WindowedFile.java     |    7 +++++++\n 2 files changed, 14 insertions(+), 0 deletions(-)\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\nindex 0cd55c9..eba1aed 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n@@ -98,6 +98,13 @@ public void testObjectMovedToNewPack() throws IncorrectObjectTypeException,\n \n \t\t// Now here is the interesting thing. Will git figure the new\n \t\t// object exists in the new pack, and not the old one.\n+\t\ttry {\n+\t\t\tThread.sleep(2000);\n+\t\t} catch (InterruptedException e) {\n+\t\t\t// TODO Auto-generated catch block\n+\t\t\te.printStackTrace();\n+\t\t}\n+\t\tSystem.gc();\n \t\tassertEquals(o1.name(), new RevWalk(db).parseAny(o1).name());\n \t}\n \ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java\nindex 938f44c..f7377d1 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java\n@@ -280,6 +280,13 @@ public void close() {\n \t}\n \n \tvoid cacheOpen() throws IOException {\n+\t\ttry {\n+\t\t\tThread.sleep(2000);\n+\t\t} catch (InterruptedException e) {\n+\t\t\t// TODO Auto-generated catch block\n+\t\t\te.printStackTrace();\n+\t\t}\n+\t\tSystem.gc();\n \t\tfd = new RandomAccessFile(fPath, \"r\");\n \t\tlength = fd.length();\n \t\ttry {\n-- \n1.6.2.2.446.gfbdc0\n\n"},{"id":"111657","messageId":"1240165809-16703-1-git-send-email-robin.rosenberg@dewire.com","threadId":"18940","inReplyTo":"200904181905.55119.robin.rosenberg.lists@dewire.com","subject":"[JGIT PATCH 0/3 RFC] http://code.google.com/p/egit/issues/detail?id=79","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-04-19T18:30:06Z","receivedAt":"2009-04-19T18:30:06Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Here is a possible solution, though I have a problem with the test case. I'd\nlike to invent something better. The problem with the \"Simulate breakpoints\"\npatch is that it patches core functions and not just the test case. Using AOP\nwould solve it, but that is too much for a single test case.\n\nRobin Rosenberg (3):\n  Add test cases dedicated to the WindowCache\n  Simulate breakpoints\n  Rescan for packs and retry once if object lookup scan fails\n\n .../tst/org/spearce/jgit/lib/WindowCacheTest.java  |  132 ++++++++++++++++++++\n .../src/org/spearce/jgit/lib/Repository.java       |   76 +++++++++--\n .../src/org/spearce/jgit/lib/WindowedFile.java     |    7 +\n 3 files changed, 202 insertions(+), 13 deletions(-)\n create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n"},{"id":"111656","messageId":"1240165809-16703-2-git-send-email-robin.rosenberg@dewire.com","threadId":"18940","inReplyTo":"1240165809-16703-1-git-send-email-robin.rosenberg@dewire.com","subject":"[JGIT PATCH 1/3] Add test cases dedicated to the WindowCache","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-04-19T18:30:07Z","receivedAt":"2009-04-19T18:30:07Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Fails like this in the debugger (expected), but runs ok outside the debugger.\n\nBreakpoints in the following places. Wait a second or two everytime it stops\n\nWindowCacheTest [line: 101] - testObjectMovedToNewPack()\nWindowedFile [line: 283] - cacheOpen()\n\njava.io.FileNotFoundException: /home/me/SW/EGIT/org.spearce.jgit.test/trash/trash1240071609317.0/.git/objects/pack/pack-fc4dadd6673a2740f43468f868ba5dd39fdeba69.pack (Filen eller katalogen finns inte)\n\tat java.io.RandomAccessFile.open(Native Method)\n\tat java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)\n\tat org.spearce.jgit.lib.WindowedFile.cacheOpen(WindowedFile.java:283)\n\tat org.spearce.jgit.lib.WindowCache.getImpl(WindowCache.java:249)\n\tat org.spearce.jgit.lib.WindowCache.get(WindowCache.java:222)\n\tat org.spearce.jgit.lib.WindowCursor.pin(WindowCursor.java:148)\n\tat org.spearce.jgit.lib.WindowCursor.copy(WindowCursor.java:82)\n\tat org.spearce.jgit.lib.WindowedFile.read(WindowedFile.java:176)\n\tat org.spearce.jgit.lib.WindowedFile.readFully(WindowedFile.java:203)\n\tat org.spearce.jgit.lib.PackFile.reader(PackFile.java:311)\n\tat org.spearce.jgit.lib.PackFile.get(PackFile.java:147)\n\tat org.spearce.jgit.lib.Repository.openObjectInPacks(Repository.java:312)\n\tat org.spearce.jgit.lib.Repository.openObject(Repository.java:294)\n\tat org.spearce.jgit.revwalk.RevWalk.parseAny(RevWalk.java:702)\n\tat org.spearce.jgit.lib.WindowCacheTest.testObjectMovedToNewPack(WindowCacheTest.java:115)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat java.lang.reflect.Method.invoke(Method.java:597)\n\tat junit.framework.TestCase.runTest(TestCase.java:164)\n\tat junit.framework.TestCase.runBare(TestCase.java:130)\n\tat junit.framework.TestResult$1.protect(TestResult.java:106)\n\tat junit.framework.TestResult.runProtected(TestResult.java:124)\n\tat junit.framework.TestResult.run(TestResult.java:109)\n\tat junit.framework.TestCase.run(TestCase.java:120)\n\tat org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130)\n\tat org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)\n---\n .../tst/org/spearce/jgit/lib/WindowCacheTest.java  |  125 ++++++++++++++++++++\n 1 files changed, 125 insertions(+), 0 deletions(-)\n create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\nnew file mode 100644\nindex 0000000..0cd55c9\n--- /dev/null\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n@@ -0,0 +1,125 @@\n+package org.spearce.jgit.lib;\n+\n+import java.io.ByteArrayInputStream;\n+import java.io.File;\n+import java.io.FileOutputStream;\n+import java.io.IOException;\n+\n+import org.spearce.jgit.errors.IncorrectObjectTypeException;\n+import org.spearce.jgit.errors.MissingObjectException;\n+import org.spearce.jgit.revwalk.RevObject;\n+import org.spearce.jgit.revwalk.RevWalk;\n+\n+public class WindowCacheTest extends RepositoryTestCase {\n+\n+\tpublic void setUp() throws Exception {\n+\t\tWindowCacheConfig windowCacheConfig = new WindowCacheConfig();\n+\t\twindowCacheConfig.setPackedGitOpenFiles(1);\n+\t\tWindowCache.reconfigure(windowCacheConfig);\n+\t\tsuper.setUp();\n+\t}\n+\n+\tprotected void tearDown() throws Exception {\n+\t\tsuper.tearDown();\n+\t\tWindowCacheConfig windowCacheConfig = new WindowCacheConfig();\n+\t\tWindowCache.reconfigure(windowCacheConfig);\n+\t}\n+\n+\t/*\n+\t * Add pack, find objects in new pack Replace packs, find objects in another\n+\t * pack than original, don't choke on lost pack\n+\t */\n+\tpublic void testObjectInNewPack() throws IncorrectObjectTypeException,\n+\t\t\tIOException {\n+\t\tRepository extra = createNewEmptyRepo();\n+\t\tRevObject o1 = writeBlob(extra, db,\n+\t\t\t\t\"nihuioijoisuoiudyosyudoiuyusyuoiyewData\");\n+\t\tPackWriter packWriter = new PackWriter(extra, new TextProgressMonitor());\n+\t\tpackWriter.addObject(o1);\n+\t\tObjectId name = packWriter.computeName();\n+\t\tFile packFileName = fullPackFileName(db, name, \".pack\");\n+\t\tFileOutputStream pos = new FileOutputStream(packFileName);\n+\t\tpackWriter.writePack(pos);\n+\t\tpos.close();\n+\t\tFile idxfname = fullPackFileName(db, name, \".idx\");\n+\t\tFileOutputStream ios = new FileOutputStream(idxfname);\n+\t\tpackWriter.writeIndex(ios);\n+\t\tios.close();\n+\t\tassertEquals(o1.name(), new RevWalk(db).parseAny(o1).name());\n+\t}\n+\n+\t/*\n+\t * Add pack, find objects in new pack Replace packs, find objects in another\n+\t * pack than original, don't choke on lost pack\n+\t */\n+\tpublic void testObjectMovedToNewPack() throws IncorrectObjectTypeException,\n+\t\t\tIOException {\n+\t\tRepository extra = createNewEmptyRepo();\n+\t\tRevObject o1 = writeBlob(extra, db,\n+\t\t\t\t\"nihuioijoisuoiudyosyudoiuyusyuoiyewData1\");\n+\t\tPackWriter packWriter1 = new PackWriter(extra,\n+\t\t\t\tnew TextProgressMonitor());\n+\t\tpackWriter1.addObject(o1);\n+\t\tObjectId name1 = packWriter1.computeName();\n+\t\tFile packFileName1 = fullPackFileName(db, name1, \".pack\");\n+\t\tFileOutputStream pos1 = new FileOutputStream(packFileName1);\n+\t\tpackWriter1.writePack(pos1);\n+\t\tpos1.close();\n+\t\tFile idxfname1 = fullPackFileName(db, name1, \".idx\");\n+\t\tFileOutputStream ios1 = new FileOutputStream(idxfname1);\n+\t\tpackWriter1.writeIndex(ios1);\n+\t\tios1.close();\n+\t\tassertEquals(o1.name(), new RevWalk(db).parseAny(o1).name());\n+\n+\t\t// Ok, lets repack this repo\n+\t\t// create an additional object\n+\t\tRevObject o2 = writeBlob(extra, db,\n+\t\t\t\t\"nihuioijoisuoiudyosyudoiuyusyuoiyewData2\");\n+\t\tPackWriter packWriter2 = new PackWriter(extra,\n+\t\t\t\tnew TextProgressMonitor());\n+\t\tpackWriter2.addObject(o1);\n+\t\tpackWriter2.addObject(o2);\n+\t\tObjectId name2 = packWriter2.computeName();\n+\t\tFile packFileName2 = fullPackFileName(db, name2, \".pack\");\n+\t\tFileOutputStream pos2 = new FileOutputStream(packFileName2);\n+\t\tpackWriter2.writePack(pos2);\n+\t\tpos2.close();\n+\t\tFile idxfname2 = fullPackFileName(db, name2, \".idx\");\n+\t\tFileOutputStream ios2 = new FileOutputStream(idxfname2);\n+\t\tpackWriter2.writeIndex(ios2);\n+\t\tios2.close();\n+\t\tassertEquals(o2.name(), new RevWalk(db).parseAny(o2).name());\n+\n+\t\tWindowCacheConfig windowCacheConfig = new WindowCacheConfig();\n+\t\twindowCacheConfig.setPackedGitOpenFiles(1);\n+\t\tWindowCache.reconfigure(windowCacheConfig);\n+\t\tpackFileName1.delete();\n+\t\tidxfname1.delete();\n+\n+\t\t// Now here is the interesting thing. Will git figure the new\n+\t\t// object exists in the new pack, and not the old one.\n+\t\tassertEquals(o1.name(), new RevWalk(db).parseAny(o1).name());\n+\t}\n+\n+\tprivate File fullPackFileName(Repository repository, ObjectId hash,\n+\t\t\tString suffix) {\n+\t\treturn new File(new File(repository.getObjectsDirectory(), \"pack\"),\n+\t\t\t\t\"pack-\" + hash.name() + suffix);\n+\t}\n+\n+\tprivate RevObject writeBlob(final Repository repo, final Repository notIn,\n+\t\t\tfinal String data) throws IOException {\n+\t\tRevWalk revWalk = new RevWalk(repo);\n+\t\tbyte[] bytes = data.getBytes(Constants.CHARACTER_ENCODING);\n+\t\tObjectId id = new ObjectWriter(repo).writeBlob(bytes.length,\n+\t\t\t\tnew ByteArrayInputStream(bytes));\n+\t\ttry {\n+\t\t\tassertNull(\n+\t\t\t\t\t\"Oops, We want new objects that we do not have yet, for this test!\",\n+\t\t\t\t\tnew RevWalk(notIn).parseAny(id));\n+\t\t} catch (MissingObjectException e) {\n+\t\t\t// Ok\n+\t\t}\n+\t\treturn revWalk.lookupBlob(id);\n+\t}\n+}\n-- \n1.6.2.2.446.gfbdc0\n"},{"id":"111658","messageId":"1240165809-16703-3-git-send-email-robin.rosenberg@dewire.com","threadId":"18940","inReplyTo":"1240165809-16703-2-git-send-email-robin.rosenberg@dewire.com","subject":"[JGIT PATCH 2/3] Simulate breakpoints","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-04-19T18:30:08Z","receivedAt":"2009-04-19T18:30:08Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"---\n .../tst/org/spearce/jgit/lib/WindowCacheTest.java  |    7 +++++++\n .../src/org/spearce/jgit/lib/WindowedFile.java     |    7 +++++++\n 2 files changed, 14 insertions(+), 0 deletions(-)\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\nindex 0cd55c9..eba1aed 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/WindowCacheTest.java\n@@ -98,6 +98,13 @@ public void testObjectMovedToNewPack() throws IncorrectObjectTypeException,\n \n \t\t// Now here is the interesting thing. Will git figure the new\n \t\t// object exists in the new pack, and not the old one.\n+\t\ttry {\n+\t\t\tThread.sleep(2000);\n+\t\t} catch (InterruptedException e) {\n+\t\t\t// TODO Auto-generated catch block\n+\t\t\te.printStackTrace();\n+\t\t}\n+\t\tSystem.gc();\n \t\tassertEquals(o1.name(), new RevWalk(db).parseAny(o1).name());\n \t}\n \ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java\nindex 938f44c..f7377d1 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowedFile.java\n@@ -280,6 +280,13 @@ public void close() {\n \t}\n \n \tvoid cacheOpen() throws IOException {\n+\t\ttry {\n+\t\t\tThread.sleep(2000);\n+\t\t} catch (InterruptedException e) {\n+\t\t\t// TODO Auto-generated catch block\n+\t\t\te.printStackTrace();\n+\t\t}\n+\t\tSystem.gc();\n \t\tfd = new RandomAccessFile(fPath, \"r\");\n \t\tlength = fd.length();\n \t\ttry {\n-- \n1.6.2.2.446.gfbdc0\n"},{"id":"111659","messageId":"1240165809-16703-4-git-send-email-robin.rosenberg@dewire.com","threadId":"18940","inReplyTo":"1240165809-16703-3-git-send-email-robin.rosenberg@dewire.com","subject":"[JGIT PATCH 3/3] Rescan for packs and retry once if object lookup scan fails","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-04-19T18:30:09Z","receivedAt":"2009-04-19T18:30:09Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"We do this by catching the FileNotFoundException thrown by the lower\nlayers an rescan for packs.\n\nA lookup failure may happen if no pack contains the requested object, so\nthe loose object lookup fails. The other case is that one of the packs\nwe use has been deleted by a repack. In either case there is a good chance\nthat there is a new pack or loose object that contains the object.\n\nTherefore we scan for new packs, invalidating an old ones that may have\nbeen deleted and retry the lookup.\n\nAt the same time we stop scanning for new packs regularly since we now\nhave another mechanism for finding them.\n---\n .../src/org/spearce/jgit/lib/Repository.java       |   76 ++++++++++++++++----\n 1 files changed, 63 insertions(+), 13 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java\nindex cfd92b8..ec4226c 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/Repository.java\n@@ -100,6 +100,8 @@\n \tprivate GitIndex index;\n \n \tprivate List<RepositoryListener> listeners = new Vector<RepositoryListener>(); // thread safe\n+\n+\tprivate int packFileListGeneration;\n \tstatic private List<RepositoryListener> allListeners = new Vector<RepositoryListener>(); // thread safe\n \n \t/**\n@@ -291,6 +293,40 @@ public ObjectLoader openObject(final AnyObjectId id)\n \t */\n \tpublic ObjectLoader openObject(final WindowCursor curs, final AnyObjectId id)\n \t\t\tthrows IOException {\n+\t\tboolean scanned = false;\n+\t\ttry {\n+\t\t\tObjectLoader loader = openObjectInPacks(curs, id);\n+\t\t\tif (loader != null)\n+\t\t\t\treturn loader;\n+\t\t} catch (FileNotFoundException fnfe) {\n+\t\t\tSystem.out.println(new java.util.Date().toString() + \": failed to load \" + id + \", rescanning for new packs\");\n+\t\t\tscanned = true;\n+\t\t\tif (scanForPacks()) {\n+\t\t\t\tObjectLoader loader = openObjectInPacks(curs, id);\n+\t\t\t\tif (loader != null)\n+\t\t\t\t\treturn loader;\n+\t\t\t}\n+\t\t}\n+\t\ttry {\n+\t\t\treturn new UnpackedObjectLoader(this, id);\n+\t\t} catch (FileNotFoundException fnfe) {\n+\t\t\tSystem.out.println(new java.util.Date().toString() + \": failed to load \" + id + \", rescanning for new packs\");\n+\t\t\tif (!scanned && scanForPacks()) {\n+\t\t\t\tObjectLoader loader = openObjectInPacks(curs, id);\n+\t\t\t\tif (loader != null)\n+\t\t\t\t\treturn loader;\n+\t\t\t\ttry {\n+\t\t\t\t\treturn new UnpackedObjectLoader(this, id);\n+\t\t\t\t} catch (FileNotFoundException fnfe2) {\n+\t\t\t\t\treturn null;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\treturn null;\n+\t\t}\n+\t}\n+\n+\tprivate ObjectLoader openObjectInPacks(final WindowCursor curs, final AnyObjectId id)\n+\t\t\tthrows IOException {\n \t\tfinal PackFile[] packs = packs();\n \t\tint k = packs.length;\n \t\twhile (k > 0) {\n@@ -298,11 +334,7 @@ public ObjectLoader openObject(final WindowCursor curs, final AnyObjectId id)\n \t\t\tif (ol != null)\n \t\t\t\treturn ol;\n \t\t}\n-\t\ttry {\n-\t\t\treturn new UnpackedObjectLoader(this, id);\n-\t\t} catch (FileNotFoundException fnfe) {\n-\t\t\treturn null;\n-\t\t}\n+\t\treturn null;\n \t}\n \n \t/**\n@@ -813,23 +845,39 @@ synchronized (this) {\n \t}\n \n \t/**\n-\t * Scan the object dirs, including alternates for packs\n-\t * to use.\n+\t * Scan the object dirs, including alternates for packs to use.\n+\t * \n+\t * @return true if new packs were found\n \t */\n-\tpublic void scanForPacks() {\n+\tpublic boolean scanForPacks() {\n+\t\tint oldPackFileListGeneration = packFileListGeneration;\n \t\tfinal ArrayList<PackFile> p = new ArrayList<PackFile>();\n-\t\tp.addAll(Arrays.asList(packs()));\n \t\tfor (final File d : objectsDirs())\n-\t\t\tscanForPacks(new File(d, \"pack\"), p);\n+\t\t\tscanForPacks(new File(d, \"pack\"), p, packFileList);\n \t\tfinal PackFile[] arr = new PackFile[p.size()];\n \t\tp.toArray(arr);\n \t\tArrays.sort(arr, PackFile.SORT);\n \t\tsynchronized (this) {\n+\t\t\tif (!samePackfileArray(packFileList, arr))\n+\t\t\t\tpackFileListGeneration++;\n \t\t\tpackFileList = arr;\n \t\t}\n+\t\treturn oldPackFileListGeneration != packFileListGeneration;\n \t}\n \n-\tprivate void scanForPacks(final File packDir, Collection<PackFile> packList) {\n+\tprivate static<T> boolean samePackfileArray(PackFile[] oldList, PackFile[] newList) {\n+\t\tif (oldList == null)\n+\t\t\treturn false;\n+\t\tif (oldList.length != newList.length)\n+\t\t\treturn false;\n+\t\tfor (int i = 0; i < oldList.length; ++i) {\n+\t\t\tif (!oldList[i].getPackFile().equals(newList[i].getPackFile()))\n+\t\t\t\treturn false;\n+\t\t}\n+\t\treturn true;\n+\t}\n+\t\n+\tprivate void scanForPacks(final File packDir, Collection<PackFile> packList, PackFile[] oldPackFileList) {\n \t\tfinal String[] idxList = packDir.list(new FilenameFilter() {\n \t\t\tpublic boolean accept(final File baseDir, final String n) {\n \t\t\t\t// Must match \"pack-[0-9a-f]{40}.idx\" to be an index.\n@@ -851,9 +899,11 @@ public boolean accept(final File baseDir, final String n) {\n \t\t\t\t\tcontinue;\n \t\t\t\t}\n \n-\t\t\t\tfor (final PackFile p : packList) {\n-\t\t\t\t\tif (packFile.equals(p.getPackFile()))\n+\t\t\t\tfor (final PackFile p : oldPackFileList) {\n+\t\t\t\t\tif (packFile.equals(p.getPackFile())) {\n+\t\t\t\t\t\tpackList.add(p);\n \t\t\t\t\t\tcontinue SCAN;\n+\t\t\t\t\t}\n \t\t\t\t}\n \n \t\t\t\tpackList.add(new PackFile(idxFile, packFile));\n-- \n1.6.2.2.446.gfbdc0\n"},{"id":"111743","messageId":"20090420143256.GX23604@spearce.org","threadId":"18940","inReplyTo":"1240165809-16703-4-git-send-email-robin.rosenberg@dewire.com","subject":"Re: [JGIT PATCH 3/3] Rescan for packs and retry once if object lookup scan fails","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-04-20T14:32:56Z","receivedAt":"2009-04-20T14:32:56Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Robin Rosenberg <robin.rosenberg@dewire.com> wrote:\n>  .../src/org/spearce/jgit/lib/Repository.java       |   76 ++++++++++++++++----\n>  1 files changed, 63 insertions(+), 13 deletions(-)\n\nNAK.\n\nI see a number of things I don't like about this implementation\nas it stands.  I'll try to work up a counter patch today.\n\nAt a glance, the new logic for openObject is pretty twisted\nto follow.  It mostly makes sense, but I had hoped for something\ncleaner to read.\n\nI'm really worried about the case where a pack file stays with\nthe same name, but its contents and index have been recreated,\nand thus our existing PackFile object is invalid for access to it.\nYour implementation as written would keep the same PackFile in\nmemory, and we'd lose access to those objects.\n\nYou also pointed out some concerns about windows stuck in the\nWindowCache.  I'm equally worried about that.\n \n-- \nShawn.\n"}]}