{"thread":{"id":"19099","subject":"[JGIT PATCH 1/2] Don't use ByteWindows when checking pack file headers/footers","startedAt":"2009-04-28T02:26:11Z","lastAt":"2009-05-06T14:15:26Z","messageCount":8,"participants":["Shawn O. Pearce","Robin Rosenberg","Ferry Huberts (Pelagic)"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"112488","messageId":"1240885572-1755-1-git-send-email-spearce@spearce.org","threadId":"19099","inReplyTo":null,"subject":"[JGIT PATCH 1/2] Don't use ByteWindows when checking pack file headers/footers","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-04-28T02:26:11Z","receivedAt":"2009-04-28T02:26:11Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Its highly unlikely we need the 8 KiB surrounding the pack file header\nor footer immediately after opening the pack file.  Reading those as\nfull blocks and registering them in the WindowCache is probably just\nchurning garbage through the cache.  Instead, read the header with a\nsingle 12 byte read, and the footer with a single 20 byte read, and\nbypass the cache altogether.\n\nThis nicely removes a deadlock condition we had previously where the\nWindowCache was recursively calling itself during the pack file open,\nand got stuck on its own locks.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n\n This I think can be applied as-is.\n\n We could quibble about whether or not caching the header and footer\n window is worthwhile during the pack open event.  But really I\n wrote this to remove a deadlock in the next patch.  Its just soooo\n much simpler to not make PackFile rely on WindowCache.\n\n .../src/org/spearce/jgit/lib/PackFile.java         |    5 +--\n org.spearce.jgit/src/org/spearce/jgit/util/NB.java |   32 ++++++++++++++++++++\n 2 files changed, 34 insertions(+), 3 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java\nindex 813ebc7..360442f 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java\n@@ -389,10 +389,9 @@ void allocWindow(final WindowCursor curs, final int windowId,\n \n \tprivate void onOpenPack() throws IOException {\n \t\tfinal PackIndex idx = idx();\n-\t\tfinal WindowCursor curs = new WindowCursor();\n \t\tfinal byte[] buf = new byte[20];\n \n-\t\treadFully(0, buf, 0, 12, curs);\n+\t\tNB.readFully(fd.getChannel(), 0, buf, 0, 12);\n \t\tif (RawParseUtils.match(buf, 0, Constants.PACK_SIGNATURE) != 4)\n \t\t\tthrow new IOException(\"Not a PACK file.\");\n \t\tfinal long vers = NB.decodeUInt32(buf, 4);\n@@ -406,7 +405,7 @@ private void onOpenPack() throws IOException {\n \t\t\t\t\t+ \" index \" + idx.getObjectCount()\n \t\t\t\t\t+ \": \" + getPackFile());\n \n-\t\treadFully(length - 20, buf, 0, 20, curs);\n+\t\tNB.readFully(fd.getChannel(), length - 20, buf, 0, 20);\n \t\tif (!Arrays.equals(buf, idx.packChecksum))\n \t\t\tthrow new PackMismatchException(\"Pack checksum mismatch:\"\n \t\t\t\t\t+ \" pack \" + ObjectId.fromRaw(buf).name()\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/util/NB.java b/org.spearce.jgit/src/org/spearce/jgit/util/NB.java\nindex c65c6fa..4a9c9b9 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/util/NB.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/util/NB.java\n@@ -40,6 +40,8 @@\n import java.io.EOFException;\n import java.io.IOException;\n import java.io.InputStream;\n+import java.nio.ByteBuffer;\n+import java.nio.channels.FileChannel;\n \n /** Conversion utilities for network byte order handling. */\n public final class NB {\n@@ -71,6 +73,36 @@ public static void readFully(final InputStream fd, final byte[] dst,\n \t}\n \n \t/**\n+\t * Read the entire byte array into memory, or throw an exception.\n+\t * \n+\t * @param fd\n+\t *            file to read the data from.\n+\t * @param pos\n+\t *            position to read from the file at.\n+\t * @param dst\n+\t *            buffer that must be fully populated, [off, off+len).\n+\t * @param off\n+\t *            position within the buffer to start writing to.\n+\t * @param len\n+\t *            number of bytes that must be read.\n+\t * @throws EOFException\n+\t *             the stream ended before dst was fully populated.\n+\t * @throws IOException\n+\t *             there was an error reading from the stream.\n+\t */\n+\tpublic static void readFully(final FileChannel fd, long pos,\n+\t\t\tfinal byte[] dst, int off, int len) throws IOException {\n+\t\twhile (len > 0) {\n+\t\t\tfinal int r = fd.read(ByteBuffer.wrap(dst, off, len), pos);\n+\t\t\tif (r <= 0)\n+\t\t\t\tthrow new EOFException(\"Short read of block.\");\n+\t\t\tpos += r;\n+\t\t\toff += r;\n+\t\t\tlen -= r;\n+\t\t}\n+\t}\n+\n+\t/**\n \t * Skip an entire region of an input stream.\n \t * <p>\n \t * The input stream's position is moved forward by the number of requested\n-- \n1.6.3.rc1.205.g37f8\n"},{"id":"112489","messageId":"1240885572-1755-2-git-send-email-spearce@spearce.org","threadId":"19099","inReplyTo":"1240885572-1755-1-git-send-email-spearce@spearce.org","subject":"[JGIT RFC PATCH 2/2] Rewrite WindowCache to be easier to follow and maintain","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-04-28T02:26:12Z","receivedAt":"2009-04-28T02:26:12Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The integration of WindowCache, ByteWindow, PackFile and WindowCursor\nwas a spaghetti of code that was impossible for even the original\nauthor (me) to follow.  Due to the way the responsibility for the\nPackFile's open RandomAccessFile \"fd\" was distributed between these\nfour classes I could no longer prove to myself that the fd wouldn't\nbe closed while it was being accessed by another thread.\n\nThis rewrite generalizes most of the cache logic into a new class,\nOffsetCache.  The hope is that we can later reuse this code to make a\nrewrite of UnpackedObjectCache, which uses similiar caching rules as\nthe WindowCache, but applies a different hash function.  That rewrite\nis deferred to another change, but is anticipated by this one.\n\nThe new OffsetCache class uses the Java 5 atomic APIs to create a\nmuch more concurrent hash table than we had before.  We can now\nperform no-miss reads without taking any locks.  Reads that do\nmiss acquire a lock in order to prevent concurrent threads from\nperforming duplicate work loading the same window from disk,\nhowever concurrent reads of different windows is still permitted.\n\nDue to the more concurrent nature of the OffsetCache, it is now\npossible for the cache to temporarily overshoot its resource limits.\nThis is a small temporary overshoot that is roughly bounded by the\nnumber of concurrent threads operating against the same cache.\n\nThe API of the ByteWindow subclasses is now simplified by removing\nthe base class of SoftReference.  It was a horrible idea to pass\nthe byte[] or MappedByteBuffer down through the call stack when the\nimplementation knew what type it should be operating on.  We now\ninstead use a more traditional OO pattern of allowing the subclass\nto directly specify its referent.\n\nResponsibility for the RandomAccessFile \"fd\" within PackFile is now\nstrictly within PackFile.  Two open reference counts track how the\ncallers are using the fd, ensuring that the fd remains open, so long\nas the caller has made the appropriate begin*() invocation prior\nto data access.  One counter, beginWindowCache() is exclusively\nfor the ByteWindows created by WindowCache.  Another counter,\nbeginCopyRawData(), is exclusively for PackWriter's need to lock\nthe PackFile open while it performs object reuse.\n\nTo keep the code simple a WindowCache.reconfigure() now discards the\nentire current cache, and creates a new one.  That invalidates every\nopen file, and every open ByteWindow, and forces them to load again.\n\nreconfigure is no longer a thread safe operation, as there is no easy\nway to lock out other threads while the cache change is taking place.\nI don't think cache reconfigurations occur frequently enough in\napplication code that we can justify the additional overhead required\nby a multi-reader/single-writer lock around every cache access.\nInstead, the Javadoc is updated to warn application authors against\nchanging this on the fly.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n\n I'm tossing this out there for tonight.  Please don't apply until\n I give a final yay or nay.\n\n I think the code is easier to follow once this patch is applied\n due to the scope of responsibilities being restored back to their\n proper OO-theory suggested classes.  The re-write of WindowCache\n is horrible to read as a diff, sorry.\n\n At present, it passes all of the unit tests.  It passes my minimal\n concurrent clone test.  At first glance, it runs within the same\n performance bounds as the prior code on a single thread clone test.\n\n I have not yet done any performance tests with more than one\n concurrent clone client.  I don't yet have hard performance test\n results to give you to back up my claims that its no worse in\n single thread usage, or that it should perform better in server\n environments.\n\n I'm going to burn this in tonight for about 12 hours by pounding\n a whole bunch of clients against it.  If that load test goes well\n tomorrow morning, I'll try to work up more concrete performance\n test data to at least show we're no worse off than before in that\n department, and ask you to apply it.\n\n .../org/spearce/jgit/lib/ConcurrentRepackTest.java |    4 -\n .../src/org/spearce/jgit/lib/ByteArrayWindow.java  |   57 +--\n .../src/org/spearce/jgit/lib/ByteBufferWindow.java |   41 +-\n .../src/org/spearce/jgit/lib/ByteWindow.java       |   91 +---\n .../src/org/spearce/jgit/lib/OffsetCache.java      |  522 ++++++++++++++++++++\n .../src/org/spearce/jgit/lib/PackFile.java         |  123 +++--\n .../org/spearce/jgit/lib/PackedObjectLoader.java   |    4 +-\n .../src/org/spearce/jgit/lib/WindowCache.java      |  408 ++++------------\n .../src/org/spearce/jgit/lib/WindowCursor.java     |   42 +-\n 9 files changed, 763 insertions(+), 529 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/ConcurrentRepackTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/ConcurrentRepackTest.java\nindex b56e0f4..fa6345e 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/ConcurrentRepackTest.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/ConcurrentRepackTest.java\n@@ -170,10 +170,6 @@ public void testObjectMovedToNewPack2()\n \n \tprivate static void whackCache() {\n \t\tfinal WindowCacheConfig config = new WindowCacheConfig();\n-\n-\t\tconfig.setPackedGitOpenFiles(0);\n-\t\tWindowCache.reconfigure(config);\n-\n \t\tconfig.setPackedGitOpenFiles(1);\n \t\tWindowCache.reconfigure(config);\n \t}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java\nindex 5dc3d28..6b96b10 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java\n@@ -38,41 +38,29 @@\n \n package org.spearce.jgit.lib;\n \n-import java.io.IOException;\n-import java.nio.ByteBuffer;\n import java.util.zip.DataFormatException;\n import java.util.zip.Inflater;\n \n /**\n  * A {@link ByteWindow} with an underlying byte array for storage.\n  */\n-final class ByteArrayWindow extends ByteWindow<byte[]> {\n-\tboolean loaded;\n+final class ByteArrayWindow extends ByteWindow {\n+\tprivate final byte[] array;\n \n-\t/**\n-\t * Constructor for ByteWindow.\n-\t * \n-\t * @param o\n-\t *            the PackFile providing data access\n-\t * @param p\n-\t *            the file offset.\n-\t * @param d\n-\t *            an id provided by the PackFile. See\n-\t *            {@link WindowCache#get(WindowCursor, PackFile, long)}.\n-\t * @param b\n-\t *            byte array for storage\n-\t */\n-\tByteArrayWindow(final PackFile o, final long p, final int d, final byte[] b) {\n-\t\tsuper(o, p, d, b, b.length);\n+\tByteArrayWindow(final PackFile pack, final long o, final byte[] b) {\n+\t\tsuper(pack, o, b.length);\n+\t\tarray = b;\n \t}\n \n-\tint copy(final byte[] array, final int p, final byte[] b, final int o, int n) {\n+\t@Override\n+\tprotected int copy(final int p, final byte[] b, final int o, int n) {\n \t\tn = Math.min(array.length - p, n);\n \t\tSystem.arraycopy(array, p, b, o, n);\n \t\treturn n;\n \t}\n \n-\tint inflate(final byte[] array, final int pos, final byte[] b, int o,\n+\t@Override\n+\tprotected int inflate(final int pos, final byte[] b, int o,\n \t\t\tfinal Inflater inf) throws DataFormatException {\n \t\twhile (!inf.finished()) {\n \t\t\tif (inf.needsInput()) {\n@@ -86,7 +74,8 @@ int inflate(final byte[] array, final int pos, final byte[] b, int o,\n \t\treturn o;\n \t}\n \n-\tvoid inflateVerify(final byte[] array, final int pos, final Inflater inf)\n+\t@Override\n+\tprotected void inflateVerify(final int pos, final Inflater inf)\n \t\t\tthrows DataFormatException {\n \t\twhile (!inf.finished()) {\n \t\t\tif (inf.needsInput()) {\n@@ -98,26 +87,4 @@ void inflateVerify(final byte[] array, final int pos, final Inflater inf)\n \t\twhile (!inf.finished() && !inf.needsInput())\n \t\t\tinf.inflate(verifyGarbageBuffer, 0, verifyGarbageBuffer.length);\n \t}\n-\n-\tvoid ensureLoaded(final byte[] array) {\n-\t\tboolean release = false;\n-\t\ttry {\n-\t\t\tsynchronized (this) {\n-\t\t\t\tif (!loaded) {\n-\t\t\t\t\trelease = true;\n-\t\t\t\t\ttry {\n-\t\t\t\t\t\tprovider.fd.getChannel().read(ByteBuffer.wrap(array),\n-\t\t\t\t\t\t\t\tstart);\n-\t\t\t\t\t} catch (IOException e) {\n-\t\t\t\t\t\tthrow new RuntimeException(\"Cannot fault in window\", e);\n-\t\t\t\t\t}\n-\t\t\t\t\tloaded = true;\n-\t\t\t\t}\n-\t\t\t}\n-\t\t} finally {\n-\t\t\tif (release) {\n-\t\t\t\tWindowCache.markLoaded(this);\n-\t\t\t}\n-\t\t}\n-\t}\n-}\n\\ No newline at end of file\n+}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteBufferWindow.java b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteBufferWindow.java\nindex 01956fd..f9de9b4 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteBufferWindow.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteBufferWindow.java\n@@ -47,24 +47,16 @@\n  *\n  * @see ByteWindow\n  */\n-final class ByteBufferWindow extends ByteWindow<ByteBuffer> {\n-\t/**\n-\t * Constructor.\n-\t *\n-\t * @See ByteWindow\n-\t *\n-\t * @param o The PackFile\n-\t * @param p the file offset.\n-\t * @param d Window id\n-\t * @param b ByteBuffer storage\n-\t */\n-\tByteBufferWindow(final PackFile o, final long p, final int d,\n-\t\t\tfinal ByteBuffer b) {\n-\t\tsuper(o, p, d, b, b.capacity());\n+final class ByteBufferWindow extends ByteWindow {\n+\tprivate final ByteBuffer buffer;\n+\n+\tByteBufferWindow(final PackFile pack, final long o, final ByteBuffer b) {\n+\t\tsuper(pack, o, b.capacity());\n+\t\tbuffer = b;\n \t}\n \n-\tfinal int copy(final ByteBuffer buffer, final int p, final byte[] b,\n-\t\t\tfinal int o, int n) {\n+\t@Override\n+\tprotected int copy(final int p, final byte[] b, final int o, int n) {\n \t\tfinal ByteBuffer s = buffer.slice();\n \t\ts.position(p);\n \t\tn = Math.min(s.remaining(), n);\n@@ -72,9 +64,9 @@ final int copy(final ByteBuffer buffer, final int p, final byte[] b,\n \t\treturn n;\n \t}\n \n-\tint inflate(final ByteBuffer buffer, final int pos, final byte[] b, int o,\n-\t\t\tfinal Inflater inf)\n-\t\t\tthrows DataFormatException {\n+\t@Override\n+\tprotected int inflate(final int pos, final byte[] b, int o,\n+\t\t\tfinal Inflater inf) throws DataFormatException {\n \t\tfinal byte[] tmp = new byte[512];\n \t\tfinal ByteBuffer s = buffer.slice();\n \t\ts.position(pos);\n@@ -91,8 +83,9 @@ int inflate(final ByteBuffer buffer, final int pos, final byte[] b, int o,\n \t\treturn o;\n \t}\n \n-\tvoid inflateVerify(final ByteBuffer buffer, final int pos,\n-\t\t\tfinal Inflater inf) throws DataFormatException {\n+\t@Override\n+\tprotected void inflateVerify(final int pos, final Inflater inf)\n+\t\t\tthrows DataFormatException {\n \t\tfinal byte[] tmp = new byte[512];\n \t\tfinal ByteBuffer s = buffer.slice();\n \t\ts.position(pos);\n@@ -107,8 +100,4 @@ void inflateVerify(final ByteBuffer buffer, final int pos,\n \t\twhile (!inf.finished() && !inf.needsInput())\n \t\t\tinf.inflate(verifyGarbageBuffer, 0, verifyGarbageBuffer.length);\n \t}\n-\n-\tvoid ensureLoaded(final ByteBuffer ref) {\n-\t\t// Do nothing.\n-\t}\n-}\n\\ No newline at end of file\n+}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteWindow.java b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteWindow.java\nindex 0d01fca..e0bda2d 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteWindow.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteWindow.java\n@@ -38,8 +38,6 @@\n \n package org.spearce.jgit.lib;\n \n-import java.lang.ref.ReferenceQueue;\n-import java.lang.ref.SoftReference;\n import java.util.zip.DataFormatException;\n import java.util.zip.Inflater;\n \n@@ -51,64 +49,31 @@\n  * is very low and has paged part of this process out to disk. Therefore copying\n  * bytes from a window is very inexpensive.\n  * </p>\n- * \n- * @param <T>\n- *            type of object reference used to manage the window data.\n  */\n-abstract class ByteWindow<T> extends SoftReference<T> {\n-\tboolean sizeActive = true;\n+abstract class ByteWindow {\n+\tprotected final PackFile pack;\n \n-\tByteWindow<?> chainNext;\n+\tprotected final long start;\n \n-\tByteWindow<?> lruPrev;\n+\tprotected final long end;\n \n-\tByteWindow<?> lruNext;\n-\n-\tfinal PackFile provider;\n-\n-\tfinal int id;\n-\n-\tfinal int size;\n-\n-\tfinal long start;\n-\n-\tfinal long end;\n+\tprotected ByteWindow(final PackFile p, final long s, final int n) {\n+\t\tpack = p;\n+\t\tstart = s;\n+\t\tend = start + n;\n+\t}\n \n-\t/**\n-\t * Constructor for ByteWindow.\n-\t * \n-\t * @param o\n-\t *            the PackFile providing data access\n-\t * @param pos\n-\t *            the position in the file the data comes from.\n-\t * @param d\n-\t *            an id provided by the PackFile. See\n-\t *            {@link WindowCache#get(WindowCursor, PackFile, long)}.\n-\t * @param ref\n-\t *            the object value required to perform data access.\n-\t * @param sz\n-\t *            the total number of bytes in this window.\n-\t */\n-\t@SuppressWarnings(\"unchecked\")\n-\tByteWindow(final PackFile o, final long pos, final int d, final T ref,\n-\t\t\tfinal int sz) {\n-\t\tsuper(ref, (ReferenceQueue<T>) WindowCache.clearedWindowQueue);\n-\t\tprovider = o;\n-\t\tsize = sz;\n-\t\tid = d;\n-\t\tstart = pos;\n-\t\tend = start + size;\n+\tfinal int size() {\n+\t\treturn (int) (end - start);\n \t}\n \n \tfinal boolean contains(final PackFile neededFile, final long neededPos) {\n-\t\treturn provider == neededFile && start <= neededPos && neededPos < end;\n+\t\treturn pack == neededFile && start <= neededPos && neededPos < end;\n \t}\n \n \t/**\n \t * Copy bytes from the window to a caller supplied buffer.\n \t * \n-\t * @param ref\n-\t *            the object value required to perform data access.\n \t * @param pos\n \t *            offset within the file to start copying from.\n \t * @param dstbuf\n@@ -123,15 +88,13 @@ final boolean contains(final PackFile neededFile, final long neededPos) {\n \t *         <code>cnt</code> if <code>cnt</code> exceeded the number of\n \t *         bytes available.\n \t */\n-\tfinal int copy(T ref, long pos, byte[] dstbuf, int dstoff, int cnt) {\n-\t\treturn copy(ref, (int) (pos - start), dstbuf, dstoff, cnt);\n+\tfinal int copy(long pos, byte[] dstbuf, int dstoff, int cnt) {\n+\t\treturn copy((int) (pos - start), dstbuf, dstoff, cnt);\n \t}\n \n \t/**\n \t * Copy bytes from the window to a caller supplied buffer.\n \t * \n-\t * @param ref\n-\t *            the object value required to perform data access.\n \t * @param pos\n \t *            offset within the window to start copying from.\n \t * @param dstbuf\n@@ -146,15 +109,13 @@ final int copy(T ref, long pos, byte[] dstbuf, int dstoff, int cnt) {\n \t *         <code>cnt</code> if <code>cnt</code> exceeded the number of\n \t *         bytes available.\n \t */\n-\tabstract int copy(T ref, int pos, byte[] dstbuf, int dstoff, int cnt);\n+\tprotected abstract int copy(int pos, byte[] dstbuf, int dstoff, int cnt);\n \n \t/**\n \t * Pump bytes into the supplied inflater as input.\n \t * \n-\t * @param ref\n-\t *            the object value required to perform data access.\n \t * @param pos\n-\t *            offset within the window to start supplying input from.\n+\t *            offset within the file to start supplying input from.\n \t * @param dstbuf\n \t *            destination buffer the inflater should output decompressed\n \t *            data to.\n@@ -174,16 +135,14 @@ final int copy(T ref, long pos, byte[] dstbuf, int dstoff, int cnt) {\n \t *             the inflater encountered an invalid chunk of data. Data\n \t *             stream corruption is likely.\n \t */\n-\tfinal int inflate(T ref, long pos, byte[] dstbuf, int dstoff, Inflater inf)\n+\tfinal int inflate(long pos, byte[] dstbuf, int dstoff, Inflater inf)\n \t\t\tthrows DataFormatException {\n-\t\treturn inflate(ref, (int) (pos - start), dstbuf, dstoff, inf);\n+\t\treturn inflate((int) (pos - start), dstbuf, dstoff, inf);\n \t}\n \n \t/**\n \t * Pump bytes into the supplied inflater as input.\n \t * \n-\t * @param ref\n-\t *            the object value required to perform data access.\n \t * @param pos\n \t *            offset within the window to start supplying input from.\n \t * @param dstbuf\n@@ -205,18 +164,16 @@ final int inflate(T ref, long pos, byte[] dstbuf, int dstoff, Inflater inf)\n \t *             the inflater encountered an invalid chunk of data. Data\n \t *             stream corruption is likely.\n \t */\n-\tabstract int inflate(T ref, int pos, byte[] dstbuf, int dstoff, Inflater inf)\n-\t\t\tthrows DataFormatException;\n+\tprotected abstract int inflate(int pos, byte[] dstbuf, int dstoff,\n+\t\t\tInflater inf) throws DataFormatException;\n \n \tprotected static final byte[] verifyGarbageBuffer = new byte[2048];\n \n-\tfinal void inflateVerify(T ref, long pos, Inflater inf)\n+\tfinal void inflateVerify(final long pos, final Inflater inf)\n \t\t\tthrows DataFormatException {\n-\t\tinflateVerify(ref, (int) (pos - start), inf);\n+\t\tinflateVerify((int) (pos - start), inf);\n \t}\n \n-\tabstract void inflateVerify(T ref, int pos, Inflater inf)\n+\tprotected abstract void inflateVerify(int pos, Inflater inf)\n \t\t\tthrows DataFormatException;\n-\n-\tabstract void ensureLoaded(T ref);\n-}\n\\ No newline at end of file\n+}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\nnew file mode 100644\nindex 0000000..170c5d2\n--- /dev/null\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\n@@ -0,0 +1,522 @@\n+/*\n+ * Copyright (C) 2009, Google Inc.\n+ *\n+ * All rights reserved.\n+ *\n+ * Redistribution and use in source and binary forms, with or\n+ * without modification, are permitted provided that the following\n+ * conditions are met:\n+ *\n+ * - Redistributions of source code must retain the above copyright\n+ *   notice, this list of conditions and the following disclaimer.\n+ *\n+ * - Redistributions in binary form must reproduce the above\n+ *   copyright notice, this list of conditions and the following\n+ *   disclaimer in the documentation and/or other materials provided\n+ *   with the distribution.\n+ *\n+ * - Neither the name of the Git Development Community nor the\n+ *   names of its contributors may be used to endorse or promote\n+ *   products derived from this software without specific prior\n+ *   written permission.\n+ *\n+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND\n+ * CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES,\n+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT\n+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\n+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+ */\n+\n+package org.spearce.jgit.lib;\n+\n+import java.io.IOException;\n+import java.lang.ref.ReferenceQueue;\n+import java.lang.ref.SoftReference;\n+import java.util.Random;\n+import java.util.concurrent.atomic.AtomicLong;\n+import java.util.concurrent.atomic.AtomicReferenceArray;\n+import java.util.concurrent.locks.ReentrantLock;\n+\n+/**\n+ * Least frequently used cache for objects specified by PackFile positions.\n+ * <p>\n+ * This cache maps a <code>({@link PackFile},position)</code> tuple to an Object.\n+ * <p>\n+ * This cache is suitable for objects that are \"relative expensive\" to compute\n+ * from the underlying PackFile, given some known position in that file.\n+ * <p>\n+ * Whenever a cache miss occurs, {@link #load(PackFile, long)} is invoked by\n+ * exactly one thread for the given <code>(PackFile,position)</code> key tuple.\n+ * This is ensured by an array of locks, with the tuple hashed to a lock\n+ * instance.\n+ * <p>\n+ * During a miss, older entries are evicted from the cache so long as\n+ * {@link #isFull()} returns true.\n+ * <p>\n+ * Its too expensive during object access to be 100% accurate with a least\n+ * recently used (LRU) algorithm. Strictly ordering every read is a lot of\n+ * overhead that typically doesn't yield a corresponding benefit to the\n+ * application.\n+ * <p>\n+ * This cache implements a loose LRU policy by randomly picking a window\n+ * comprised of roughly 10% of the cache, and evicting the oldest accessed entry\n+ * within that window.\n+ * <p>\n+ * Entities created by the cache are held under SoftReferences, permitting the\n+ * Java runtime's garbage collector to evict entries when heap memory gets low.\n+ * Most JREs implement a loose least recently used algorithm for this eviction.\n+ * <p>\n+ * The internal hash table does not expand at runtime, instead it is fixed in\n+ * size at cache creation time. The internal lock table used to gate load\n+ * invocations is also fixed in size.\n+ * <p>\n+ * The key tuple is passed through to methods as a pair of parameters rather\n+ * than as a single Object, thus reducing the transient memory allocations of\n+ * callers. It is more efficient to avoid the allocation, as we can't be 100%\n+ * sure that a JIT would be able to stack-allocate a key tuple.\n+ * <p>\n+ * This cache has an implementation rule such that:\n+ * <ul>\n+ * <li>{@link #load(PackFile, long)} is invoked by at most one thread at a time\n+ * for a given <code>(PackFile,position)</code> tuple.</li>\n+ * <li>For every <code>load()</code> invocation there is exactly one\n+ * {@link #createRef(PackFile, long, Object)} invocation to wrap a SoftReference\n+ * around the cached entity.</li>\n+ * <li>For every Reference created by <code>createRef()</code> there will be\n+ * exactly one call to {@link #clear(Ref)} to cleanup any resources associated\n+ * with the (now expired) cached entity.</li>\n+ * </ul>\n+ * <p>\n+ * Therefore, it is safe to perform resource accounting increments during the\n+ * {@link #load(PackFile, long)} or {@link #createRef(PackFile, long, Object)}\n+ * methods, and matching decrements during {@link #clear(Ref)}. Implementors may\n+ * need to override {@link #createRef(PackFile, long, Object)} in order to embed\n+ * additional accounting information into an implementation specific\n+ * {@link OffsetCache.Ref} subclass, as the cached entity may have already been\n+ * evicted by the JRE's garbage collector.\n+ * <p>\n+ * To maintain higher concurrency workloads, during eviction only one thread\n+ * performs the eviction work, while other threads can continue to insert new\n+ * objects in parallel. This means that the cache can be temporarily over limit,\n+ * especially if the nominated eviction thread is being starved relative to the\n+ * other threads.\n+ * \n+ * @param <V>\n+ *            type of value stored in the cache.\n+ * @param <R>\n+ *            type of {@link OffsetCache.Ref} subclass used by the cache.\n+ */\n+abstract class OffsetCache<V, R extends OffsetCache.Ref<V>> {\n+\tprivate static final Random rng = new Random();\n+\n+\t/** ReferenceQueue that {@link #createRef(PackFile, long, Object)} must use. */\n+\tprotected final ReferenceQueue<V> queue;\n+\n+\t/** Number of entries in {@link #table}. */\n+\tprivate final int tableSize;\n+\n+\t/** Access clock for loose LRU. */\n+\tprivate final AtomicLong clock;\n+\n+\t/** Hash bucket directory; entries are chained below. */\n+\tprivate final AtomicReferenceArray<Entry<V>> table;\n+\n+\t/** Locks to prevent concurrent loads for same (PackFile,position). */\n+\tprivate final Lock[] locks;\n+\n+\t/** Lock to elect the eviction thread after a load occurs. */\n+\tprivate final ReentrantLock evictLock;\n+\n+\t/** Number of {@link #table} buckets to scan for an eviction window. */\n+\tprivate final int evictBatch;\n+\n+\t/**\n+\t * Create a new cache with a fixed size entry table and lock table.\n+\t * \n+\t * @param tSize\n+\t *            number of entries in the entry hash table.\n+\t * @param lockCount\n+\t *            number of entries in the lock table. This is the maximum\n+\t *            concurrency rate for creation of new objects through\n+\t *            {@link #load(PackFile, long)} invocations.\n+\t */\n+\tOffsetCache(final int tSize, final int lockCount) {\n+\t\tif (tSize < 1)\n+\t\t\tthrow new IllegalArgumentException(\"tSize must be >= 1\");\n+\t\tif (lockCount < 1)\n+\t\t\tthrow new IllegalArgumentException(\"lockCount must be >= 1\");\n+\n+\t\tqueue = new ReferenceQueue<V>();\n+\t\ttableSize = tSize;\n+\t\tclock = new AtomicLong(1);\n+\t\ttable = new AtomicReferenceArray<Entry<V>>(tableSize);\n+\t\tlocks = new Lock[lockCount];\n+\t\tfor (int i = 0; i < locks.length; i++)\n+\t\t\tlocks[i] = new Lock();\n+\t\tevictLock = new ReentrantLock();\n+\n+\t\tint eb = (int) (tableSize * .1);\n+\t\tif (64 < eb)\n+\t\t\teb = 64;\n+\t\telse if (eb < 4)\n+\t\t\teb = 4;\n+\t\tif (tableSize < eb)\n+\t\t\teb = tableSize;\n+\t\tevictBatch = eb;\n+\t}\n+\n+\t/**\n+\t * Lookup a cached object, creating and loading it if it doesn't exist.\n+\t * \n+\t * @param pack\n+\t *            the pack that \"contains\" the cached object.\n+\t * @param position\n+\t *            offset within <code>pack</code> of the object.\n+\t * @return the object reference.\n+\t * @throws IOException\n+\t *             the object reference was not in the cache and could not be\n+\t *             obtained by {@link #load(PackFile, long)}.\n+\t */\n+\tV getOrLoad(final PackFile pack, final long position) throws IOException {\n+\t\tfinal int slot = slot(pack, position);\n+\t\tfinal Entry<V> e1 = table.get(slot);\n+\t\tV v = scan(e1, pack, position);\n+\t\tif (v != null)\n+\t\t\treturn v;\n+\n+\t\tsynchronized (lock(pack, position)) {\n+\t\t\tEntry<V> e2 = table.get(slot);\n+\t\t\tif (e2 != e1) {\n+\t\t\t\tv = scan(e2, pack, position);\n+\t\t\t\tif (v != null)\n+\t\t\t\t\treturn v;\n+\t\t\t}\n+\n+\t\t\tv = load(pack, position);\n+\t\t\tfinal Ref<V> ref = createRef(pack, position, v);\n+\t\t\thit(ref);\n+\t\t\tfor (;;) {\n+\t\t\t\tfinal Entry<V> n = new Entry<V>(clean(e2), ref);\n+\t\t\t\tif (table.compareAndSet(slot, e2, n))\n+\t\t\t\t\tbreak;\n+\t\t\t\te2 = table.get(slot);\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (evictLock.tryLock()) {\n+\t\t\ttry {\n+\t\t\t\tgc();\n+\t\t\t\tevict();\n+\t\t\t} finally {\n+\t\t\t\tevictLock.unlock();\n+\t\t\t}\n+\t\t}\n+\n+\t\treturn v;\n+\t}\n+\n+\tprivate V scan(Entry<V> n, final PackFile pack, final long position) {\n+\t\tfor (; n != null; n = n.next) {\n+\t\t\tfinal Ref<V> r = n.ref;\n+\t\t\tif (r.pack == pack && r.position == position) {\n+\t\t\t\tfinal V v = r.get();\n+\t\t\t\tif (v != null) {\n+\t\t\t\t\thit(r);\n+\t\t\t\t\treturn v;\n+\t\t\t\t}\n+\t\t\t\tn.dead = true;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t\treturn null;\n+\t}\n+\n+\tprivate void hit(final Ref<V> r) {\n+\t\t// We don't need to be 100% accurate here. Its sufficient that at least\n+\t\t// one thread performs the increment. Any other concurrent access at\n+\t\t// exactly the same time can simply use the same clock value.\n+\t\t//\n+\t\t// Consequently we attempt the set, but we don't try to recover should\n+\t\t// it fail. This is why we don't use getAndIncrement() here.\n+\t\t//\n+\t\tfinal long c = clock.get();\n+\t\tclock.compareAndSet(c, c + 1);\n+\t\tr.lastAccess = c;\n+\t}\n+\n+\tprivate void evict() {\n+\t\tfinal int start = rng.nextInt(tableSize);\n+\t\tint ptr = start;\n+\t\twhile (isFull()) {\n+\t\t\tEntry<V> old = null;\n+\t\t\tint slot = 0;\n+\t\t\tfor (int b = evictBatch - 1; b >= 0; b--) {\n+\t\t\t\tif (tableSize <= ptr)\n+\t\t\t\t\tptr = 0;\n+\t\t\t\tfor (Entry<V> e = table.get(ptr); e != null; e = e.next) {\n+\t\t\t\t\tif (e.dead)\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tif (old == null || e.ref.lastAccess < old.ref.lastAccess) {\n+\t\t\t\t\t\told = e;\n+\t\t\t\t\t\tslot = ptr;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t\tif (++ptr == start)\n+\t\t\t\t\treturn;\n+\t\t\t}\n+\t\t\tif (old != null) {\n+\t\t\t\told.kill();\n+\t\t\t\tgc();\n+\t\t\t\tfinal Entry<V> e1 = table.get(slot);\n+\t\t\t\ttable.compareAndSet(slot, e1, clean(e1));\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\t/**\n+\t * Clear every entry from the cache.\n+\t *<p>\n+\t * This is a last-ditch effort to clear out the cache, such as before it\n+\t * gets replaced by another cache that is configured differently. This\n+\t * method tries to force every cached entry through {@link #clear(Ref)} to\n+\t * ensure that resources are correctly accounted for and cleaned up by the\n+\t * subclass. A concurrent reader loading entries while this method is\n+\t * running may cause resource accounting failures.\n+\t */\n+\tvoid removeAll() {\n+\t\tfor (int s = 0; s < tableSize; s++) {\n+\t\t\tEntry<V> e1;\n+\t\t\tdo {\n+\t\t\t\te1 = table.get(s);\n+\t\t\t\tfor (Entry<V> e = e1; e != null; e = e.next)\n+\t\t\t\t\te.kill();\n+\t\t\t} while (!table.compareAndSet(s, e1, null));\n+\t\t}\n+\t\tgc();\n+\t}\n+\n+\t/**\n+\t * Clear all entries related to a single file.\n+\t * <p>\n+\t * Typically this method is invoked during {@link PackFile#close()}, when we\n+\t * know the pack is never going to be useful to us again (for example, it no\n+\t * longer exists on disk). A concurrent reader loading an entry from this\n+\t * same pack may cause the pack to become stuck in the cache anyway.\n+\t * \n+\t * @param pack\n+\t *            the file to purge all entries of.\n+\t */\n+\tvoid removeAll(final PackFile pack) {\n+\t\tfor (int s = 0; s < tableSize; s++) {\n+\t\t\tfinal Entry<V> e1 = table.get(s);\n+\t\t\tboolean hasDead = false;\n+\t\t\tfor (Entry<V> e = e1; e != null; e = e.next) {\n+\t\t\t\tif (e.ref.pack == pack) {\n+\t\t\t\t\te.kill();\n+\t\t\t\t\thasDead = true;\n+\t\t\t\t} else if (e.dead)\n+\t\t\t\t\thasDead = true;\n+\t\t\t}\n+\t\t\tif (hasDead)\n+\t\t\t\ttable.compareAndSet(s, e1, clean(e1));\n+\t\t}\n+\t\tgc();\n+\t}\n+\n+\t/**\n+\t * Materialize an object that doesn't yet exist in the cache.\n+\t * <p>\n+\t * This method is invoked by {@link #getOrLoad(PackFile, long)} when the\n+\t * specified entity does not yet exist in the cache. Internal locking\n+\t * ensures that at most one thread can call this method for each unique\n+\t * <code>(pack,position)</code>, but multiple threads can call this method\n+\t * concurrently for different <code>(pack,position)</code> tuples.\n+\t * \n+\t * @param pack\n+\t *            the file to materialize the entry from.\n+\t * @param position\n+\t *            offset within the file of the entry.\n+\t * @return the materialized object. Must never be null.\n+\t * @throws IOException\n+\t *             the method was unable to materialize the object for this\n+\t *             input pair. The usual reasons would be file corruption, file\n+\t *             not found, out of file descriptors, etc.\n+\t */\n+\tprotected abstract V load(PackFile pack, long position) throws IOException;\n+\n+\t/**\n+\t * Construct a Ref (SoftReference) around a cached entity.\n+\t * <p>\n+\t * Implementing this is only necessary if the subclass is performing\n+\t * resource accounting during {@link #load(PackFile, long)} and\n+\t * {@link #clear(Ref)} requires some information to update the accounting.\n+\t * <p>\n+\t * Implementors <b>MUST</b> ensure that the returned reference uses the\n+\t * {@link #queue} ReferenceQueue, otherwise {@link #clear(Ref)} will not be\n+\t * invoked at the proper time.\n+\t * \n+\t * @param pack\n+\t *            the file to materialize the entry from.\n+\t * @param position\n+\t *            offset within the file of the entry.\n+\t * @param v\n+\t *            the object returned by {@link #load(PackFile, long)}.\n+\t * @return a soft reference subclass wrapped around <code>v</code>.\n+\t */\n+\t@SuppressWarnings(\"unchecked\")\n+\tprotected R createRef(final PackFile pack, final long position, final V v) {\n+\t\treturn (R) new Ref<V>(pack, position, v, queue);\n+\t}\n+\n+\t/**\n+\t * Update accounting information now that an object has left the cache.\n+\t * <p>\n+\t * This method is invoked exactly once for the combined\n+\t * {@link #load(PackFile, long)} and\n+\t * {@link #createRef(PackFile, long, Object)} invocation pair that was used\n+\t * to construct and insert an object into the cache.\n+\t * \n+\t * @param ref\n+\t *            the reference wrapped around the object. Implementations must\n+\t *            be prepared for <code>ref.get()</code> to return null.\n+\t */\n+\tprotected void clear(final R ref) {\n+\t\t// Do nothing by default.\n+\t}\n+\n+\t/**\n+\t * Determine if the cache is full and requires eviction of entries.\n+\t * <p>\n+\t * By default this method returns false. Implementors may override to\n+\t * consult with the accounting updated by {@link #load(PackFile, long)},\n+\t * {@link #createRef(PackFile, long, Object)} and {@link #clear(Ref)}.\n+\t * \n+\t * @return true if the cache is still over-limit and requires eviction of\n+\t *         more entries.\n+\t */\n+\tprotected boolean isFull() {\n+\t\treturn false;\n+\t}\n+\n+\t@SuppressWarnings(\"unchecked\")\n+\tprivate void gc() {\n+\t\tR r;\n+\t\twhile ((r = (R) queue.poll()) != null) {\n+\t\t\t// Sun's Java 5 and 6 implementation have a bug where a Reference\n+\t\t\t// can be enqueued and dequeued twice on the same reference queue\n+\t\t\t// due to a race condition within ReferenceQueue.enqueue(Reference).\n+\t\t\t//\n+\t\t\t// We CANNOT permit a Reference to come through us twice, as it will\n+\t\t\t// skew the resource counters we maintain. Our canClear() check here\n+\t\t\t// provides a way to skip the redundant dequeues, if any.\n+\t\t\t//\n+\t\t\tif (r.canClear())\n+\t\t\t\tclear(r);\n+\t\t}\n+\t}\n+\n+\t/**\n+\t * Compute the hash code value for a <code>(PackFile,position)</code> tuple.\n+\t * <p>\n+\t * By default: <code>(packHash + (int) (position >>> 4)) >>> 1</code>.\n+\t * Implementors may override with a more suitable hash (for example, a\n+\t * larger right shift on the position).\n+\t * \n+\t * @param packHash\n+\t *            hash code for the file being accessed.\n+\t * @param position\n+\t *            position within the file being accessed.\n+\t * @return a reasonable hash code mixing the two values.\n+\t */\n+\tprotected int hash(final int packHash, final long position) {\n+\t\treturn (packHash + (int) (position >>> 4)) >>> 1;\n+\t}\n+\n+\tprivate int slot(final PackFile pack, final long position) {\n+\t\treturn hash(pack.hash, position) % tableSize;\n+\t}\n+\n+\tprivate Lock lock(final PackFile pack, final long position) {\n+\t\treturn locks[hash(pack.hash, position) % locks.length];\n+\t}\n+\n+\tprivate static <V> Entry<V> clean(Entry<V> top) {\n+\t\twhile (top != null && top.dead) {\n+\t\t\ttop.ref.enqueue();\n+\t\t\ttop = top.next;\n+\t\t}\n+\t\tif (top == null)\n+\t\t\treturn null;\n+\t\tfinal Entry<V> n = clean(top.next);\n+\t\treturn n == top.next ? top : new Entry<V>(n, top.ref);\n+\t}\n+\n+\tprivate static class Entry<V> {\n+\t\t/** Next entry in the hash table's chain list. */\n+\t\tfinal Entry<V> next;\n+\n+\t\t/** The referenced object. */\n+\t\tfinal Ref<V> ref;\n+\n+\t\t/**\n+\t\t * Marked true when ref.get() returns null and the ref is dead.\n+\t\t * <p>\n+\t\t * A true here indicates that the ref is no longer accessible, and that\n+\t\t * we therefore need to eventually purge this Entry object out of the\n+\t\t * bucket's chain.\n+\t\t */\n+\t\tvolatile boolean dead;\n+\n+\t\tEntry(final Entry<V> n, final Ref<V> r) {\n+\t\t\tnext = n;\n+\t\t\tref = r;\n+\t\t}\n+\n+\t\tfinal void kill() {\n+\t\t\tdead = true;\n+\t\t\tref.enqueue();\n+\t\t}\n+\t}\n+\n+\t/**\n+\t * A soft reference wrapped around a cached object.\n+\t * \n+\t * @param <V>\n+\t *            type of the cached object.\n+\t */\n+\tprotected static class Ref<V> extends SoftReference<V> {\n+\t\tfinal PackFile pack;\n+\n+\t\tfinal long position;\n+\n+\t\tlong lastAccess;\n+\n+\t\tprivate boolean cleared;\n+\n+\t\tprotected Ref(final PackFile pack, final long position, final V v,\n+\t\t\t\tfinal ReferenceQueue<V> queue) {\n+\t\t\tsuper(v, queue);\n+\t\t\tthis.pack = pack;\n+\t\t\tthis.position = position;\n+\t\t}\n+\n+\t\tfinal synchronized boolean canClear() {\n+\t\t\tif (cleared)\n+\t\t\t\treturn false;\n+\t\t\tcleared = true;\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\n+\tprivate static final class Lock {\n+\t\t// Used only for its implicit monitor.\n+\t}\n+}\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java\nindex 360442f..b107dfe 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackFile.java\n@@ -77,12 +77,13 @@ public int compare(final PackFile a, final PackFile b) {\n \n \tfinal int hash;\n \n-\tRandomAccessFile fd;\n+\tprivate RandomAccessFile fd;\n \n \tlong length;\n \n-\t/** Total number of windows actively in the associated cache. */\n-\tint openCount;\n+\tprivate int activeWindows;\n+\n+\tprivate int activeCopyRawData;\n \n \tprivate int packLastModified;\n \n@@ -310,27 +311,57 @@ private void copyToStream(long position, final byte[] buf, long cnt,\n \t\t}\n \t}\n \n-\tvoid cacheOpen() throws IOException {\n-\t\tfd = new RandomAccessFile(packFile, \"r\");\n-\t\tlength = fd.length();\n+\tsynchronized void beginCopyRawData() throws IOException {\n+\t\tif (++activeCopyRawData == 1 && activeWindows == 0)\n+\t\t\tdoOpen();\n+\t}\n+\n+\tsynchronized void endCopyRawData() {\n+\t\tif (--activeCopyRawData == 0 && activeWindows == 0)\n+\t\t\tdoClose();\n+\t}\n+\n+\tsynchronized boolean beginWindowCache() throws IOException {\n+\t\tif (++activeWindows == 1) {\n+\t\t\tif (activeCopyRawData == 0)\n+\t\t\t\tdoOpen();\n+\t\t\treturn true;\n+\t\t}\n+\t\treturn false;\n+\t}\n+\n+\tsynchronized boolean endWindowCache() {\n+\t\tfinal boolean r = --activeWindows == 0;\n+\t\tif (r && activeCopyRawData == 0)\n+\t\t\tdoClose();\n+\t\treturn r;\n+\t}\n+\n+\tprivate void doOpen() throws IOException {\n \t\ttry {\n+\t\t\tfd = new RandomAccessFile(packFile, \"r\");\n+\t\t\tlength = fd.length();\n \t\t\tonOpenPack();\n \t\t} catch (IOException ioe) {\n-\t\t\tinvalid = true;\n-\t\t\tcacheClose();\n+\t\t\topenFail();\n \t\t\tthrow ioe;\n \t\t} catch (RuntimeException re) {\n-\t\t\tinvalid = true;\n-\t\t\tcacheClose();\n+\t\t\topenFail();\n \t\t\tthrow re;\n \t\t} catch (Error re) {\n-\t\t\tinvalid = true;\n-\t\t\tcacheClose();\n+\t\t\topenFail();\n \t\t\tthrow re;\n \t\t}\n \t}\n \n-\tvoid cacheClose() {\n+\tprivate void openFail() {\n+\t\tactiveWindows = 0;\n+\t\tactiveCopyRawData = 0;\n+\t\tinvalid = true;\n+\t\tdoClose();\n+\t}\n+\n+\tprivate void doClose() {\n \t\tif (fd != null) {\n \t\t\ttry {\n \t\t\t\tfd.close();\n@@ -343,48 +374,34 @@ void cacheClose() {\n \t\t}\n \t}\n \n-\tvoid allocWindow(final WindowCursor curs, final int windowId,\n-\t\t\tfinal long pos, final int size) {\n-\t\tif (WindowCache.mmap) {\n-\t\t\tMappedByteBuffer map;\n-\t\t\ttry {\n-\t\t\t\tmap = fd.getChannel().map(MapMode.READ_ONLY, pos, size);\n-\t\t\t} catch (IOException e) {\n-\t\t\t\t// The most likely reason this failed is the JVM has run out\n-\t\t\t\t// of virtual memory. We need to discard quickly, and try to\n-\t\t\t\t// force the GC to finalize and release any existing mappings.\n-\t\t\t\ttry {\n-\t\t\t\t\tcurs.release();\n-\t\t\t\t\tSystem.gc();\n-\t\t\t\t\tSystem.runFinalization();\n-\t\t\t\t\tmap = fd.getChannel().map(MapMode.READ_ONLY, pos, size);\n-\t\t\t\t} catch (IOException ioe2) {\n-\t\t\t\t\t// Temporarily disable mmap and do buffered disk IO.\n-\t\t\t\t\t//\n-\t\t\t\t\tmap = null;\n-\t\t\t\t\tSystem.err.println(\"warning: mmap failure: \"+ioe2);\n-\t\t\t\t}\n-\t\t\t}\n-\t\t\tif (map != null) {\n-\t\t\t\tif (map.hasArray()) {\n-\t\t\t\t\tfinal byte[] b = map.array();\n-\t\t\t\t\tfinal ByteArrayWindow w;\n-\t\t\t\t\tw = new ByteArrayWindow(this, pos, windowId, b);\n-\t\t\t\t\tw.loaded = true;\n-\t\t\t\t\tcurs.window = w;\n-\t\t\t\t\tcurs.handle = b;\n-\t\t\t\t} else {\n-\t\t\t\t\tcurs.window = new ByteBufferWindow(this, pos, windowId, map);\n-\t\t\t\t\tcurs.handle = map;\n-\t\t\t\t}\n-\t\t\t\treturn;\n-\t\t\t}\n+\tByteArrayWindow read(final long pos, int size) throws IOException {\n+\t\tif (length < pos + size)\n+\t\t\tsize = (int) (length - pos);\n+\t\tfinal byte[] buf = new byte[size];\n+\t\tNB.readFully(fd.getChannel(), pos, buf, 0, size);\n+\t\treturn new ByteArrayWindow(this, pos, buf);\n+\t}\n+\n+\tByteWindow mmap(final long pos, int size) throws IOException {\n+\t\tif (length < pos + size)\n+\t\t\tsize = (int) (length - pos);\n+\n+\t\tMappedByteBuffer map;\n+\t\ttry {\n+\t\t\tmap = fd.getChannel().map(MapMode.READ_ONLY, pos, size);\n+\t\t} catch (IOException ioe1) {\n+\t\t\t// The most likely reason this failed is the JVM has run out\n+\t\t\t// of virtual memory. We need to discard quickly, and try to\n+\t\t\t// force the GC to finalize and release any existing mappings.\n+\t\t\t//\n+\t\t\tSystem.gc();\n+\t\t\tSystem.runFinalization();\n+\t\t\tmap = fd.getChannel().map(MapMode.READ_ONLY, pos, size);\n \t\t}\n \n-\t\tfinal byte[] b = new byte[size];\n-\t\tcurs.window = new ByteArrayWindow(this, pos, windowId, b);\n-\t\tcurs.handle = b;\n-\t\topenCount++; // Until the window loads, we must stay open.\n+\t\tif (map.hasArray())\n+\t\t\treturn new ByteArrayWindow(this, pos, map.array());\n+\t\treturn new ByteBufferWindow(this, pos, map);\n \t}\n \n \tprivate void onOpenPack() throws IOException {\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/PackedObjectLoader.java b/org.spearce.jgit/src/org/spearce/jgit/lib/PackedObjectLoader.java\nindex d49562a..0c3e783 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/PackedObjectLoader.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/PackedObjectLoader.java\n@@ -126,7 +126,7 @@ public final long getDataOffset() {\n \t *             deleted, and the object has moved to another pack file.\n \t */\n \tpublic void beginCopyRawData() throws IOException {\n-\t\tWindowCache.pin(pack);\n+\t\tpack.beginCopyRawData();\n \t}\n \n \t/**\n@@ -154,7 +154,7 @@ public void copyRawData(OutputStream out, byte buf[], WindowCursor curs)\n \n \t/** Release resources after {@link #beginCopyRawData()}. */\n \tpublic void endCopyRawData() {\n-\t\tWindowCache.unpin(pack);\n+\t\tpack.endCopyRawData();\n \t}\n \n \t/**\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\nindex 51d149c..3eb1204 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\n@@ -40,12 +40,17 @@\n \n import java.io.IOException;\n import java.lang.ref.ReferenceQueue;\n+import java.util.concurrent.atomic.AtomicInteger;\n \n /**\n- * The WindowCache manages reusable <code>Windows</code> and inflaters used by\n- * the other windowed file access classes.\n+ * Caches slices of a {@link PackFile} in memory for faster read access.\n+ * <p>\n+ * The WindowCache serves as a Java based \"buffer cache\", loading segments of a\n+ * PackFile into the JVM heap prior to use. As JGit often wants to do reads of\n+ * only tiny slices of a file, the WindowCache tries to smooth out these tiny\n+ * reads into larger block-sized IO operations.\n  */\n-public class WindowCache {\n+public class WindowCache extends OffsetCache<ByteWindow, WindowCache.WindowRef> {\n \tprivate static final int bits(int newSize) {\n \t\tif (newSize < 4096)\n \t\t\tthrow new IllegalArgumentException(\"Invalid window size\");\n@@ -54,41 +59,10 @@ private static final int bits(int newSize) {\n \t\treturn Integer.numberOfTrailingZeros(newSize);\n \t}\n \n-\tprivate static int maxFileCount;\n-\n-\tprivate static int maxByteCount;\n-\n-\tprivate static int windowSize;\n-\n-\tprivate static int windowSizeShift;\n-\n-\tstatic boolean mmap;\n-\n-\tstatic final ReferenceQueue<?> clearedWindowQueue;\n-\n-\tprivate static ByteWindow[] cache;\n-\n-\tprivate static ByteWindow lruHead;\n-\n-\tprivate static ByteWindow lruTail;\n-\n-\tprivate static int openFileCount;\n-\n-\tprivate static int openByteCount;\n+\tprivate static volatile WindowCache cache;\n \n \tstatic {\n-\t\tfinal WindowCacheConfig c = new WindowCacheConfig();\n-\t\tmaxFileCount = c.getPackedGitOpenFiles();\n-\t\tmaxByteCount = c.getPackedGitLimit();\n-\t\twindowSizeShift = bits(c.getPackedGitWindowSize());\n-\t\twindowSize = 1 << windowSizeShift;\n-\t\tmmap = c.isPackedGitMMAP();\n-\t\tcache = new ByteWindow[cacheTableSize()];\n-\t\tclearedWindowQueue = new ReferenceQueue<Object>();\n-\t}\n-\n-\tprivate static int cacheTableSize() {\n-\t\treturn 5 * (maxByteCount / windowSize) / 2;\n+\t\treconfigure(new WindowCacheConfig());\n \t}\n \n \t/**\n@@ -125,325 +99,133 @@ public static void reconfigure(final int packedGitLimit,\n \t * The new configuration is applied immediately. If the new limits are\n \t * smaller than what what is currently cached, older entries will be purged\n \t * as soon as possible to allow the cache to meet the new limit.\n+\t * <p>\n+\t * Applying a new configuration while repositories are being accessed may\n+\t * cause files to become stuck open until the Java garbage collector can\n+\t * eventually finalize their streams. Applications are encouraged to set the\n+\t * cache only when concurrent access is impossible, or highly improbable.\n \t *\n \t * @param cfg\n \t *            the new window cache configuration.\n \t */\n \tpublic static void reconfigure(final WindowCacheConfig cfg) {\n-\t\treconfigureImpl(cfg);\n+\t\tfinal WindowCache c = cache;\n+\t\tif (c != null)\n+\t\t\tc.removeAll();\n+\t\tcache = new WindowCache(cfg);\n \t\tUnpackedObjectCache.reconfigure(cfg);\n \t}\n \n-\tprivate static synchronized void reconfigureImpl(final WindowCacheConfig cfg) {\n-\t\tboolean prune = false;\n-\t\tboolean evictAll = false;\n-\n-\t\tif (maxFileCount < cfg.getPackedGitOpenFiles())\n-\t\t\tmaxFileCount = cfg.getPackedGitOpenFiles();\n-\t\telse if (maxFileCount > cfg.getPackedGitOpenFiles()) {\n-\t\t\tmaxFileCount = cfg.getPackedGitOpenFiles();\n-\t\t\tprune = true;\n-\t\t}\n-\n-\t\tif (maxByteCount < cfg.getPackedGitLimit()) {\n-\t\t\tmaxByteCount = cfg.getPackedGitLimit();\n-\t\t} else if (maxByteCount > cfg.getPackedGitLimit()) {\n-\t\t\tmaxByteCount = cfg.getPackedGitLimit();\n-\t\t\tprune = true;\n-\t\t}\n-\n-\t\tif (bits(cfg.getPackedGitWindowSize()) != windowSizeShift) {\n-\t\t\twindowSizeShift = bits(cfg.getPackedGitWindowSize());\n-\t\t\twindowSize = 1 << windowSizeShift;\n-\t\t\tevictAll = true;\n-\t\t}\n-\n-\t\tif (mmap != cfg.isPackedGitMMAP()) {\n-\t\t\tmmap = cfg.isPackedGitMMAP();\n-\t\t\tevictAll = true;\n-\t\t}\n-\n-\t\tif (evictAll) {\n-\t\t\t// We have to throw away every window we have. None\n-\t\t\t// of them are suitable for the new configuration.\n-\t\t\t//\n-\t\t\tfor (ByteWindow<?> e : cache) {\n-\t\t\t\tfor (; e != null; e = e.chainNext)\n-\t\t\t\t\tclear(e);\n-\t\t\t}\n-\t\t\trunClearedWindowQueue();\n-\t\t\tcache = new ByteWindow[cacheTableSize()];\n-\n-\t\t} else {\n-\t\t\tif (prune) {\n-\t\t\t\t// We should decrease our memory usage.\n-\t\t\t\t//\n-\t\t\t\treleaseMemory();\n-\t\t\t\trunClearedWindowQueue();\n-\t\t\t}\n-\n-\t\t\tif (cache.length != cacheTableSize()) {\n-\t\t\t\t// The cache table should be resized.\n-\t\t\t\t// Rehash every entry.\n-\t\t\t\t//\n-\t\t\t\tfinal ByteWindow[] priorTable = cache;\n-\n-\t\t\t\tcache = new ByteWindow[cacheTableSize()];\n-\t\t\t\tfor (ByteWindow<?> e : priorTable) {\n-\t\t\t\t\tfor (ByteWindow<?> n; e != null; e = n) {\n-\t\t\t\t\t\tn = e.chainNext;\n-\t\t\t\t\t\tfinal int idx = hash(e.provider, e.id);\n-\t\t\t\t\t\te.chainNext = cache[idx];\n-\t\t\t\t\t\tcache[idx] = e;\n-\t\t\t\t\t}\n-\t\t\t\t}\n-\t\t\t}\n-\t\t}\n-\t}\n-\n-\t/**\n-\t * Get a specific window.\n-\t * \n-\t * @param curs\n-\t *            an active cursor object to maintain the window reference while\n-\t *            the caller needs it.\n-\t * @param wp\n-\t *            the provider of the window. If the window is not currently in\n-\t *            the cache then the provider will be asked to load it.\n-\t * @param position\n-\t *            offset (in bytes) within the file that the caller needs access\n-\t *            to.\n-\t * @throws IOException\n-\t *             the window was not found in the cache and the given provider\n-\t *             was unable to load the window on demand.\n-\t */\n-\tpublic static final void get(final WindowCursor curs, final PackFile wp,\n-\t\t\tfinal long position) throws IOException {\n-\t\tgetImpl(curs, wp, position);\n-\t\tcurs.window.ensureLoaded(curs.handle);\n-\t}\n-\n-\tstatic synchronized final void pin(final PackFile wp) throws IOException {\n-\t\tif (++wp.openCount == 1) {\n-\t\t\topenFile(wp);\n-\t\t}\n+\tstatic final ByteWindow get(final PackFile pack, final long offset)\n+\t\t\tthrows IOException {\n+\t\tfinal WindowCache c = cache;\n+\t\treturn c.getOrLoad(pack, c.toStart(offset));\n \t}\n \n-\tstatic synchronized final void unpin(final PackFile wp) {\n-\t\tif (--wp.openCount == 0) {\n-\t\t\topenFileCount--;\n-\t\t\twp.cacheClose();\n-\t\t}\n+\tstatic final void purge(final PackFile pack) {\n+\t\tcache.removeAll(pack);\n \t}\n \n-\tprivate static synchronized final void getImpl(final WindowCursor curs,\n-\t\t\tfinal PackFile wp, final long position) throws IOException {\n-\t\tfinal int id = (int) (position >> windowSizeShift);\n-\t\tfinal int idx = hash(wp, id);\n-\t\tfor (ByteWindow<?> e = cache[idx]; e != null; e = e.chainNext) {\n-\t\t\tif (e.provider == wp && e.id == id) {\n-\t\t\t\tif ((curs.handle = e.get()) != null) {\n-\t\t\t\t\tcurs.window = e;\n-\t\t\t\t\tmakeMostRecent(e);\n-\t\t\t\t\treturn;\n-\t\t\t\t}\n+\tprivate final int maxFiles;\n \n-\t\t\t\tclear(e);\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n+\tprivate final int maxBytes;\n \n-\t\tif (wp.openCount == 0) {\n-\t\t\topenFile(wp);\n+\tprivate final boolean mmap;\n \n-\t\t\t// The cacheOpen may have mapped the window we are trying to\n-\t\t\t// map ourselves. Retrying the search ensures that does not\n-\t\t\t// happen to us.\n-\t\t\t//\n-\t\t\tfor (ByteWindow<?> e = cache[idx]; e != null; e = e.chainNext) {\n-\t\t\t\tif (e.provider == wp && e.id == id) {\n-\t\t\t\t\tif ((curs.handle = e.get()) != null) {\n-\t\t\t\t\t\tcurs.window = e;\n-\t\t\t\t\t\tmakeMostRecent(e);\n-\t\t\t\t\t\treturn;\n-\t\t\t\t\t}\n+\tprivate final int windowSizeShift;\n \n-\t\t\t\t\tclear(e);\n-\t\t\t\t\tbreak;\n-\t\t\t\t}\n-\t\t\t}\n-\t\t}\n+\tprivate final int windowSize;\n \n-\t\tfinal int wsz = windowSize(wp, id);\n-\t\twp.openCount++;\n-\t\topenByteCount += wsz;\n-\t\treleaseMemory();\n-\t\trunClearedWindowQueue();\n+\tprivate final AtomicInteger openFiles;\n \n-\t\twp.allocWindow(curs, id, (position >>> windowSizeShift) << windowSizeShift, wsz);\n-\t\tfinal ByteWindow<?> e = curs.window;\n-\t\te.chainNext = cache[idx];\n-\t\tcache[idx] = e;\n-\t\tinsertLRU(e);\n-\t}\n+\tprivate final AtomicInteger openBytes;\n \n-\tprivate static void openFile(final PackFile wp) throws IOException {\n-\t\ttry {\n-\t\t\topenFileCount++;\n-\t\t\treleaseMemory();\n-\t\t\trunClearedWindowQueue();\n-\t\t\twp.openCount = 1;\n-\t\t\twp.cacheOpen();\n-\t\t} catch (IOException ioe) {\n-\t\t\topenFileCount--;\n-\t\t\twp.openCount = 0;\n-\t\t\tthrow ioe;\n-\t\t} catch (RuntimeException ioe) {\n-\t\t\topenFileCount--;\n-\t\t\twp.openCount = 0;\n-\t\t\tthrow ioe;\n-\t\t} catch (Error ioe) {\n-\t\t\topenFileCount--;\n-\t\t\twp.openCount = 0;\n-\t\t\tthrow ioe;\n-\t\t} finally {\n-\t\t\twp.openCount--;\n-\t\t}\n-\t}\n+\tprivate WindowCache(final WindowCacheConfig cfg) {\n+\t\tsuper(tableSize(cfg), lockCount(cfg));\n+\t\tmaxFiles = cfg.getPackedGitOpenFiles();\n+\t\tmaxBytes = cfg.getPackedGitLimit();\n+\t\tmmap = cfg.isPackedGitMMAP();\n+\t\twindowSizeShift = bits(cfg.getPackedGitWindowSize());\n+\t\twindowSize = 1 << windowSizeShift;\n \n-\tstatic synchronized void markLoaded(final ByteWindow w) {\n-\t\tif (--w.provider.openCount == 0) {\n-\t\t\topenFileCount--;\n-\t\t\tw.provider.cacheClose();\n-\t\t}\n-\t}\n+\t\topenFiles = new AtomicInteger();\n+\t\topenBytes = new AtomicInteger();\n \n-\tprivate static void makeMostRecent(ByteWindow<?> e) {\n-\t\tif (lruHead != e) {\n-\t\t\tunlinkLRU(e);\n-\t\t\tinsertLRU(e);\n-\t\t}\n+\t\tif (maxFiles < 1)\n+\t\t\tthrow new IllegalArgumentException();\n+\t\tif (maxBytes < windowSize)\n+\t\t\tthrow new IllegalArgumentException();\n \t}\n \n-\tprivate static void releaseMemory() {\n-\t\tByteWindow<?> e = lruTail;\n-\t\twhile (isOverLimit() && e != null) {\n-\t\t\tfinal ByteWindow<?> p = e.lruPrev;\n-\t\t\tclear(e);\n-\t\t\te = p;\n-\t\t}\n-\t}\n-\n-\tprivate static boolean isOverLimit() {\n-\t\treturn openByteCount > maxByteCount || openFileCount > maxFileCount;\n+\t@Override\n+\tprotected int hash(final int packHash, final long off) {\n+\t\treturn (packHash + (int) (off >>> windowSizeShift)) >>> 1;\n \t}\n \n-\t/**\n-\t * Remove all windows associated with a specific provider.\n-\t * <p>\n-\t * Providers should invoke this method as part of their cleanup/close\n-\t * routines, ensuring that the window cache releases all windows that cannot\n-\t * ever be requested again.\n-\t * </p>\n-\t * \n-\t * @param wp\n-\t *            the window provider whose windows should be removed from the\n-\t *            cache.\n-\t */\n-\tpublic static synchronized final void purge(final PackFile wp) {\n-\t\tfor (ByteWindow e : cache) {\n-\t\t\tfor (; e != null; e = e.chainNext) {\n-\t\t\t\tif (e.provider == wp)\n-\t\t\t\t\tclear(e);\n-\t\t\t}\n+\t@Override\n+\tprotected ByteWindow load(final PackFile pack, final long offset)\n+\t\t\tthrows IOException {\n+\t\tif (pack.beginWindowCache())\n+\t\t\topenFiles.incrementAndGet();\n+\t\ttry {\n+\t\t\tif (mmap)\n+\t\t\t\treturn pack.mmap(offset, windowSize);\n+\t\t\treturn pack.read(offset, windowSize);\n+\t\t} catch (IOException e) {\n+\t\t\tclose(pack);\n+\t\t\tthrow e;\n+\t\t} catch (RuntimeException e) {\n+\t\t\tclose(pack);\n+\t\t\tthrow e;\n+\t\t} catch (Error e) {\n+\t\t\tclose(pack);\n+\t\t\tthrow e;\n \t\t}\n-\t\trunClearedWindowQueue();\n \t}\n \n-\tprivate static void runClearedWindowQueue() {\n-\t\tByteWindow<?> e;\n-\t\twhile ((e = (ByteWindow) clearedWindowQueue.poll()) != null) {\n-\t\t\tunlinkSize(e);\n-\t\t\tunlinkLRU(e);\n-\t\t\tunlinkCache(e);\n-\t\t\te.chainNext = null;\n-\t\t\te.lruNext = null;\n-\t\t\te.lruPrev = null;\n-\t\t}\n+\t@Override\n+\tprotected WindowRef createRef(final PackFile p, final long o,\n+\t\t\tfinal ByteWindow v) {\n+\t\tfinal WindowRef ref = new WindowRef(p, o, v, queue);\n+\t\topenBytes.addAndGet(ref.size);\n+\t\treturn ref;\n \t}\n \n-\tprivate static void clear(final ByteWindow<?> e) {\n-\t\tunlinkSize(e);\n-\t\te.clear();\n-\t\te.enqueue();\n+\t@Override\n+\tprotected void clear(final WindowRef ref) {\n+\t\topenBytes.addAndGet(-ref.size);\n+\t\tclose(ref.pack);\n \t}\n \n-\tprivate static void unlinkSize(final ByteWindow<?> e) {\n-\t\tif (e.sizeActive) {\n-\t\t\tif (--e.provider.openCount == 0) {\n-\t\t\t\topenFileCount--;\n-\t\t\t\te.provider.cacheClose();\n-\t\t\t}\n-\t\t\topenByteCount -= e.size;\n-\t\t\te.sizeActive = false;\n-\t\t}\n+\tprivate void close(final PackFile pack) {\n+\t\tif (pack.endWindowCache())\n+\t\t\topenFiles.decrementAndGet();\n \t}\n \n-\tprivate static void unlinkCache(final ByteWindow dead) {\n-\t\tfinal int idx = hash(dead.provider, dead.id);\n-\t\tByteWindow<?> e = cache[idx], p = null, n;\n-\t\tfor (; e != null; p = e, e = n) {\n-\t\t\tn = e.chainNext;\n-\t\t\tif (e == dead) {\n-\t\t\t\tif (p == null)\n-\t\t\t\t\tcache[idx] = n;\n-\t\t\t\telse\n-\t\t\t\t\tp.chainNext = n;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n+\t@Override\n+\tprotected boolean isFull() {\n+\t\treturn maxFiles < openFiles.get() || maxBytes < openBytes.get();\n \t}\n \n-\tprivate static void unlinkLRU(final ByteWindow e) {\n-\t\tfinal ByteWindow<?> prev = e.lruPrev;\n-\t\tfinal ByteWindow<?> next = e.lruNext;\n-\n-\t\tif (prev != null)\n-\t\t\tprev.lruNext = next;\n-\t\telse\n-\t\t\tlruHead = next;\n-\n-\t\tif (next != null)\n-\t\t\tnext.lruPrev = prev;\n-\t\telse\n-\t\t\tlruTail = prev;\n+\tprivate long toStart(final long offset) {\n+\t\treturn (offset >>> windowSizeShift) << windowSizeShift;\n \t}\n \n-\tprivate static void insertLRU(final ByteWindow<?> e) {\n-\t\tfinal ByteWindow h = lruHead;\n-\t\te.lruPrev = null;\n-\t\te.lruNext = h;\n-\t\tif (h != null)\n-\t\t\th.lruPrev = e;\n-\t\telse\n-\t\t\tlruTail = e;\n-\t\tlruHead = e;\n+\tprivate static int tableSize(final WindowCacheConfig cfg) {\n+\t\treturn 5 * (cfg.getPackedGitLimit() / cfg.getPackedGitWindowSize()) / 2;\n \t}\n \n-\tprivate static int hash(final PackFile wp, final int id) {\n-\t\t// wp.hash was already \"stirred up\" a bit by * 31 when\n-\t\t// it was created. Its reasonable to just add here.\n-\t\t//\n-\t\treturn ((wp.hash + id) >>> 1) % cache.length;\n+\tprivate static int lockCount(final WindowCacheConfig cfg) {\n+\t\treturn Math.max(cfg.getPackedGitOpenFiles(), 32);\n \t}\n \n-\tprivate static int windowSize(final PackFile file, final int id) {\n-\t\tfinal long len = file.length;\n-\t\tfinal long pos = id << windowSizeShift;\n-\t\treturn len < pos + windowSize ? (int) (len - pos) : windowSize;\n-\t}\n+\tstatic class WindowRef extends OffsetCache.Ref<ByteWindow> {\n+\t\tfinal int size;\n \n-\tprivate WindowCache() {\n-\t\tthrow new UnsupportedOperationException();\n+\t\tWindowRef(final PackFile pack, final long position, final ByteWindow v,\n+\t\t\t\tfinal ReferenceQueue<ByteWindow> queue) {\n+\t\t\tsuper(pack, position, v, queue);\n+\t\t\tsize = v.size();\n+\t\t}\n \t}\n }\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java\nindex fb9d348..0723a78 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java\n@@ -48,14 +48,12 @@\n \n \tprivate Inflater inf;\n \n-\tByteWindow window;\n-\n-\tObject handle;\n+\tprivate ByteWindow window;\n \n \t/**\n \t * Copy bytes from the window to a caller supplied buffer.\n \t * \n-\t * @param provider\n+\t * @param pack\n \t *            the file the desired window is stored within.\n \t * @param position\n \t *            position within the file to read from.\n@@ -74,13 +72,13 @@\n \t *             this cursor does not match the provider or id and the proper\n \t *             window could not be acquired through the provider's cache.\n \t */\n-\tint copy(final PackFile provider, long position, final byte[] dstbuf,\n+\tint copy(final PackFile pack, long position, final byte[] dstbuf,\n \t\t\tint dstoff, final int cnt) throws IOException {\n-\t\tfinal long length = provider.length;\n+\t\tfinal long length = pack.length;\n \t\tint need = cnt;\n \t\twhile (need > 0 && position < length) {\n-\t\t\tpin(provider, position);\n-\t\t\tfinal int r = window.copy(handle, position, dstbuf, dstoff, need);\n+\t\t\tpin(pack, position);\n+\t\t\tfinal int r = window.copy(position, dstbuf, dstoff, need);\n \t\t\tposition += r;\n \t\t\tdstoff += r;\n \t\t\tneed -= r;\n@@ -91,7 +89,7 @@ int copy(final PackFile provider, long position, final byte[] dstbuf,\n \t/**\n \t * Pump bytes into the supplied inflater as input.\n \t * \n-\t * @param provider\n+\t * @param pack\n \t *            the file the desired window is stored within.\n \t * @param position\n \t *            position within the file to read from.\n@@ -109,47 +107,53 @@ int copy(final PackFile provider, long position, final byte[] dstbuf,\n \t *             the inflater encountered an invalid chunk of data. Data\n \t *             stream corruption is likely.\n \t */\n-\tint inflate(final PackFile provider, long position, final byte[] dstbuf,\n+\tint inflate(final PackFile pack, long position, final byte[] dstbuf,\n \t\t\tint dstoff) throws IOException, DataFormatException {\n \t\tif (inf == null)\n \t\t\tinf = InflaterCache.get();\n \t\telse\n \t\t\tinf.reset();\n \t\tfor (;;) {\n-\t\t\tpin(provider, position);\n-\t\t\tdstoff = window.inflate(handle, position, dstbuf, dstoff, inf);\n+\t\t\tpin(pack, position);\n+\t\t\tdstoff = window.inflate(position, dstbuf, dstoff, inf);\n \t\t\tif (inf.finished())\n \t\t\t\treturn dstoff;\n \t\t\tposition = window.end;\n \t\t}\n \t}\n \n-\tvoid inflateVerify(final PackFile provider, long position)\n+\tvoid inflateVerify(final PackFile pack, long position)\n \t\t\tthrows IOException, DataFormatException {\n \t\tif (inf == null)\n \t\t\tinf = InflaterCache.get();\n \t\telse\n \t\t\tinf.reset();\n \t\tfor (;;) {\n-\t\t\tpin(provider, position);\n-\t\t\twindow.inflateVerify(handle, position, inf);\n+\t\t\tpin(pack, position);\n+\t\t\twindow.inflateVerify(position, inf);\n \t\t\tif (inf.finished())\n \t\t\t\treturn;\n \t\t\tposition = window.end;\n \t\t}\n \t}\n \n-\tprivate void pin(final PackFile provider, final long position)\n+\tprivate void pin(final PackFile pack, final long position)\n \t\t\tthrows IOException {\n \t\tfinal ByteWindow w = window;\n-\t\tif (w == null || !w.contains(provider, position))\n-\t\t\tWindowCache.get(this, provider, position);\n+\t\tif (w == null || !w.contains(pack, position)) {\n+\t\t\t// If memory is low, we may need what is in our window field to\n+\t\t\t// be cleaned up by the GC during the get for the next window.\n+\t\t\t// So we always clear it, even though we are just going to set\n+\t\t\t// it again.\n+\t\t\t//\n+\t\t\twindow = null;\n+\t\t\twindow = WindowCache.get(pack, position);\n+\t\t}\n \t}\n \n \t/** Release the current window cursor. */\n \tpublic void release() {\n \t\twindow = null;\n-\t\thandle = null;\n \t\ttry {\n \t\t\tInflaterCache.release(inf);\n \t\t} finally {\n-- \n1.6.3.rc1.205.g37f8\n"},{"id":"112536","messageId":"20090428152822.GP23604@spearce.org","threadId":"19099","inReplyTo":"1240885572-1755-2-git-send-email-spearce@spearce.org","subject":"Re: [JGIT RFC PATCH 2/2] Rewrite WindowCache to be easier to follow and maintain","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-04-28T15:28:22Z","receivedAt":"2009-04-28T15:28:22Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Shawn O. Pearce\" <spearce@spearce.org> wrote:\n> \n> This rewrite generalizes most of the cache logic into a new class,\n...\n>  I'm tossing this out there for tonight.  Please don't apply until\n>  I give a final yay or nay.\n...\n>  I'm going to burn this in tonight for about 12 hours by pounding\n>  a whole bunch of clients against it.\n\nYay.\n\nThis patch looks solid to me.  \n\nI ran it overnight for 12 hours on a test rig.  4-way Intel(R)\nXeon(R) CPU 5148 @ 2.33GHz with 32 GiB physical memory.  The git://\nserver process was launched with:\n\n  java -Xmx8192m -classpath jgit \\\n    org.spearce.jgit.pgm.Main daemon\n    --port 8853 --export-all base\n\nNote that is the default WindowCacheConfig.\n\nI ran 8 concurrent clone clients on two 2-way systems, 4 clients\nper host.  The clone clients randomly selected between three\nrepositories on each clone attempt:\n\n  linux-2.6 fork (322M on disk)\n  repo.git       (1.6M on disk)\n  gerrit.git     (9.3M on disk)\n\nI used C git `git clone --bare ...` for the clients to try and\nspeed up the client side of the test.\n\nIn 12 hours the clients successfully completed a total of 2,456\nclones, and appear to have been averaging 10,999 KiB/s at peak on\none test host and 4,969 KiB/s on the other.  So JGit was pushing\nsomewhere around 63,872 KiB/s.  I'm not sure what the network can\nreally do here; the test clients are in my office and the server\nis in a data center somewhere in the same state.\n\nAt peak (when all clients picked the linux-2.6 fork roughly around\nthe same time) I saw the server JVM approaching 380% CPU utilization.\nNot too bad giving that the WindowCacheConfig's default settings are\nwoefully inadequate for 8 concurrent PackWriters on linux-2.6.\n\nUnfortunately, this quad is largest SMP box I have available.\nI'm sure we're wasting CPU spinning through cache entries under load.\nI still need to do throughput testing, and that may lead to some\ntuning changes.  But the code is cleaner and didn't fall over,\nso I say we move ahead and apply this rewrite.\n\n-- \nShawn.\n"},{"id":"112580","messageId":"200904290120.00039.robin.rosenberg.lists@dewire.com","threadId":"19099","inReplyTo":"1240885572-1755-2-git-send-email-spearce@spearce.org","subject":"Re: [JGIT RFC PATCH 2/2] Rewrite WindowCache to be easier to follow and maintain","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-04-28T23:19:59Z","receivedAt":"2009-04-28T23:19:59Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"tisdag 28 april 2009 04:26:12 skrev \"Shawn O. Pearce\" <spearce@spearce.org>:\n> To keep the code simple a WindowCache.reconfigure() now discards the\n> entire current cache, and creates a new one.  That invalidates every\n> open file, and every open ByteWindow, and forces them to load again.\n> \n> reconfigure is no longer a thread safe operation, as there is no easy\n> way to lock out other threads while the cache change is taking place.\n> I don't think cache reconfigurations occur frequently enough in\n> application code that we can justify the additional overhead required\n> by a multi-reader/single-writer lock around every cache access.\n> Instead, the Javadoc is updated to warn application authors against\n> changing this on the fly.\n\nAs for non-thread-safe reconfigure, we have to solve it somehow since\nI'd expect to be able to reconfigure the cache in Eclipse. Forcing a restart might\nbe an ok workaround for that particular case. Could one somehow, thread safely, \nlet the old cache live on until no-one uses it and the GC takes care of it, and \nredirect new accesses to the new cache.\n\n> diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java\n> index 5dc3d28..6b96b10 100644\n> --- a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java\n> +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java\n...\n> @@ -86,7 +74,8 @@ int inflate(final byte[] array, final int pos, final byte[] b, int o,\n>  \t\treturn o;\n>  \t}\n>  \n> -\tvoid inflateVerify(final byte[] array, final int pos, final Inflater inf)\n> +\t@Override\n> +\tprotected void inflateVerify(final int pos, final Inflater inf)\n>  \t\t\tthrows DataFormatException {\n>  \t\twhile (!inf.finished()) {\n>  \t\t\tif (inf.needsInput()) {\n> @@ -98,26 +87,4 @@ void inflateVerify(final byte[] array, final int pos, final Inflater inf)\n>  \t\twhile (!inf.finished() && !inf.needsInput())\n>  \t\t\tinf.inflate(verifyGarbageBuffer, 0, verifyGarbageBuffer.length);\n>  \t}\n\nNot related to this patche really, but the static buffer makes a but nervous,\nI don't think your test massaged  that part since it did not enable memory mapping.\n\n> diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\n> +\tOffsetCache(final int tSize, final int lockCount) {\n...\n> +\t\tint eb = (int) (tableSize * .1);\n> +\t\tif (64 < eb)\n> +\t\t\teb = 64;\n> +\t\telse if (eb < 4)\n> +\t\t\teb = 4;\n\t\t\t^no coverage in unit testt\n> +\t\tif (tableSize < eb)\n> +\t\t\teb = tableSize;\n\t\t\t^no coverage in unit testt\n> +\t\tevictBatch = eb;\n> +\t}\n> +\n...\n> +\tV getOrLoad(final PackFile pack, final long position) throws IOException {\n> +\t\tfinal int slot = slot(pack, position);\n> +\t\tfinal Entry<V> e1 = table.get(slot);\n> +\t\tV v = scan(e1, pack, position);\n> +\t\tif (v != null)\n> +\t\t\treturn v;\n> +\n> +\t\tsynchronized (lock(pack, position)) {\n> +\t\t\tEntry<V> e2 = table.get(slot);\n> +\t\t\tif (e2 != e1) {\n-- this block not coverred\n> +\t\t\t\tv = scan(e2, pack, position);\n> +\t\t\t\tif (v != null)\n> +\t\t\t\t\treturn v;\n> +\t\t\t}\n> +\n> +\t\t\tv = load(pack, position);\n> +\t\t\tfinal Ref<V> ref = createRef(pack, position, v);\n> +\t\t\thit(ref);\n> +\t\t\tfor (;;) {\n> +\t\t\t\tfinal Entry<V> n = new Entry<V>(clean(e2), ref);\n> +\t\t\t\tif (table.compareAndSet(slot, e2, n))\n> +\t\t\t\t\tbreak;\n> +\t\t\t\te2 = table.get(slot);\n--       ^not covered\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tif (evictLock.tryLock()) {\n> +\t\t\ttry {\n> +\t\t\t\tgc();\n> +\t\t\t\tevict();\n> +\t\t\t} finally {\n> +\t\t\t\tevictLock.unlock();\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\treturn v;\n> +\t}\n> +\n> +\tprivate V scan(Entry<V> n, final PackFile pack, final long position) {\n> +\t\tfor (; n != null; n = n.next) {\n> +\t\t\tfinal Ref<V> r = n.ref;\n> +\t\t\tif (r.pack == pack && r.position == position) {\n> +\t\t\t\tfinal V v = r.get();\n> +\t\t\t\tif (v != null) {\n> +\t\t\t\t\thit(r);\n> +\t\t\t\t\treturn v;\n> +\t\t\t\t}\n> +\t\t\t\tn.dead = true;\n> +\t\t\t\tbreak;\n\tthese two lines not covered^\n...\n> +\tprivate void evict() {\n> +\t\tfinal int start = rng.nextInt(tableSize);\n> +\t\tint ptr = start;\n> +\t\twhile (isFull()) {\n\tThe whole body of the loop not covered. It could be that RepositoryTestCase should\n\tset a very low limit on some parameters, such as maximum number of opened files.\n...\n> +\tvoid removeAll(final PackFile pack) {\n> +\t\tfor (int s = 0; s < tableSize; s++) {\n> +\t\t\tfinal Entry<V> e1 = table.get(s);\n> +\t\t\tboolean hasDead = false;\n> +\t\t\tfor (Entry<V> e = e1; e != null; e = e.next) {\n> +\t\t\t\tif (e.ref.pack == pack) {\n> +\t\t\t\t\te.kill();\n> +\t\t\t\t\thasDead = true;\n> +\t\t\t\t} else if (e.dead)\n> +\t\t\t\t\thasDead = true;\n\tuncovered statement ^\n> +\t\t\t}\n> +\t\t\tif (hasDead)\n> +\t\t\t\ttable.compareAndSet(s, e1, clean(e1));\n> +\t\t}\n> +\t\tgc();\n> +\t}\n\n> +\t@SuppressWarnings(\"unchecked\")\n> +\tprivate void gc() {\n> +\t\tR r;\n> +\t\twhile ((r = (R) queue.poll()) != null) {\n> +\t\t\t// Sun's Java 5 and 6 implementation have a bug where a Reference\n> +\t\t\t// can be enqueued and dequeued twice on the same reference queue\n> +\t\t\t// due to a race condition within ReferenceQueue.enqueue(Reference).\n\nReference to the official Sun bug? Might help if someone wants to implement\na flag to avoid this (if necessary...)\n\n> +\tprotected int hash(final int packHash, final long position) {\n> +\t\treturn (packHash + (int) (position >>> 4)) >>> 1;\n> +\t}\nSince we never use the baselass this one isn't covered... ok anyway I think.\n\n> +\tprivate static <V> Entry<V> clean(Entry<V> top) {\n> +\t\twhile (top != null && top.dead) {\n> +\t\t\ttop.ref.enqueue();\n> +\t\t\ttop = top.next;\n> +\t\t}\n> +\t\tif (top == null)\n> +\t\t\treturn null;\n> +\t\tfinal Entry<V> n = clean(top.next);\n> +\t\treturn n == top.next ? top : new Entry<V>(n, top.ref);\n\t\ttwo last lines uncovered.\n\n> +\t\tfinal synchronized boolean canClear() {\n> +\t\t\tif (cleared)\n> +\t\t\t\treturn false;\n\tuncovered return ^\n\nThis is a huge patch... I might comment on more later. As you may see, I think we need to lessen \nour dependence on faith based testing. (The MoveDeleteHook doesn't work\nwell either and it depends on code not covererd in unit tests at all, though\nI'm not sure that will reveal the problem yet).\n\n-- robin\n"},{"id":"112581","messageId":"20090428233048.GY23604@spearce.org","threadId":"19099","inReplyTo":"200904290120.00039.robin.rosenberg.lists@dewire.com","subject":"Re: [JGIT RFC PATCH 2/2] Rewrite WindowCache to be easier to follow and maintain","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-04-28T23:30:48Z","receivedAt":"2009-04-28T23:30:48Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:\n> tisdag 28 april 2009 04:26:12 skrev \"Shawn O. Pearce\" <spearce@spearce.org>:\n> > To keep the code simple a WindowCache.reconfigure() now discards the\n> > entire current cache, and creates a new one.  That invalidates every\n> > open file, and every open ByteWindow, and forces them to load again.\n> > \n> > reconfigure is no longer a thread safe operation, as there is no easy\n> > way to lock out other threads while the cache change is taking place.\n> > I don't think cache reconfigurations occur frequently enough in\n> > application code that we can justify the additional overhead required\n> > by a multi-reader/single-writer lock around every cache access.\n> > Instead, the Javadoc is updated to warn application authors against\n> > changing this on the fly.\n> \n> As for non-thread-safe reconfigure, we have to solve it somehow since\n> I'd expect to be able to reconfigure the cache in Eclipse. Forcing a restart might\n> be an ok workaround for that particular case. Could one somehow, thread safely, \n> let the old cache live on until no-one uses it and the GC takes care of it, and \n> redirect new accesses to the new cache.\n\nI think I've fixed it with a subsequent patch, see \"Better handle\nconcurrent reads during a WindowCache reconfiguration\" sent out a\nfew hours ago.\n \n> > diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java\n> > index 5dc3d28..6b96b10 100644\n> > --- a/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java\n> > +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/ByteArrayWindow.java\n> ...\n> > @@ -98,26 +87,4 @@ void inflateVerify(final byte[] array, final int pos, final Inflater inf)\n> >  \t\twhile (!inf.finished() && !inf.needsInput())\n> >  \t\t\tinf.inflate(verifyGarbageBuffer, 0, verifyGarbageBuffer.length);\n> >  \t}\n> \n> Not related to this patche really, but the static buffer makes a but nervous,\n> I don't think your test massaged  that part since it did not enable memory mapping.\n\nThis is old code, and isn't changed either way.\n\nIt isn't just memory mapping vs. not memory mapping, this same sort\nof code is in ByteBufferWindow too.\n\nThough now that you mention it, I'm recalling something about how\nlibz might try to read the output buffer when producing more output,\nin which case this code is not thread safe.  I wish I could remember.\n\nWe may want to just do a follow-up patch that creates a temporary\nbyte[] within the WindowCursor for this verifyGarbageBuffer and\npass that down through here instead.\n\nI'll consider it.\n \n> > diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\n> > +\tOffsetCache(final int tSize, final int lockCount) {\n> ...\n> > +\t\tint eb = (int) (tableSize * .1);\n> > +\t\tif (64 < eb)\n> > +\t\t\teb = 64;\n> > +\t\telse if (eb < 4)\n> > +\t\t\teb = 4;\n> \t\t\t^no coverage in unit testt\n> > +\t\tif (tableSize < eb)\n> > +\t\t\teb = tableSize;\n> \t\t\t^no coverage in unit testt\n\nBlargh.  My EclEmma installation is busted so I couldn't run it\nthrough coverage.  I just beat the tar out of it for 12 hours.\n\nI'll try to fix EclEmma tomorrow and increase coverage around the\nnew code.\n\n-- \nShawn.\n"},{"id":"112652","messageId":"20090429171659.GF23604@spearce.org","threadId":"19099","inReplyTo":"200904290120.00039.robin.rosenberg.lists@dewire.com","subject":"Re: [JGIT RFC PATCH 2/2] Rewrite WindowCache to be easier to follow and maintain","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-04-29T17:16:59Z","receivedAt":"2009-04-29T17:16:59Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:\n> > diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\n> > +\tprivate void gc() {\n> > +\t\tR r;\n> > +\t\twhile ((r = (R) queue.poll()) != null) {\n> > +\t\t\t// Sun's Java 5 and 6 implementation have a bug where a Reference\n> > +\t\t\t// can be enqueued and dequeued twice on the same reference queue\n> > +\t\t\t// due to a race condition within ReferenceQueue.enqueue(Reference).\n> \n> Reference to the official Sun bug? Might help if someone wants to\n> implement a flag to avoid this (if necessary...)\n\nActually, this is a new bug.  I tried looking through BugParade\nbut nobody has mentioned or discovered this before.\n\nI submitted a bug report yesterday, but they have yet to publish it.\n\nSo, no link.\n \n> > +\tprotected int hash(final int packHash, final long position) {\n> > +\t\treturn (packHash + (int) (position >>> 4)) >>> 1;\n> > +\t}\n>\n> Since we never use the baselass this one isn't covered... ok anyway I think.\n\nThis particular implementation I planned on using in\nUnpackedObjectCache if I rewrote it using OffsetCache.\n\nBut I could also just make it abstract here.  Maybe that is the\nbetter approach.  I'll amend that into the series.\n \n-- \nShawn.\n"},{"id":"112740","messageId":"49F9549F.3000702@pelagic.nl","threadId":"19099","inReplyTo":"20090429171659.GF23604@spearce.org","subject":"Re: [JGIT RFC PATCH 2/2] Rewrite WindowCache to be easier to follow and maintain","fromName":"Ferry Huberts (Pelagic)","fromEmail":"ferry.huberts@pelagic.nl","sentAt":"2009-04-30T07:34:55Z","receivedAt":"2009-04-30T07:34:55Z","isPatch":true,"sender":{"key":"ferry.huberts@pelagic.nl","avatar":"https://gravatar.com/avatar/9f63c0289ad23cbdef0f7609a0af85ff0f4b3babfd066de9ff58f62d48cfd6f2?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n> Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:\n>>> diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\n>>> +\tprivate void gc() {\n>>> +\t\tR r;\n>>> +\t\twhile ((r = (R) queue.poll()) != null) {\n>>> +\t\t\t// Sun's Java 5 and 6 implementation have a bug where a Reference\n>>> +\t\t\t// can be enqueued and dequeued twice on the same reference queue\n>>> +\t\t\t// due to a race condition within ReferenceQueue.enqueue(Reference).\n>> Reference to the official Sun bug? Might help if someone wants to\n>> implement a flag to avoid this (if necessary...)\n> \n> Actually, this is a new bug.  I tried looking through BugParade\n> but nobody has mentioned or discovered this before.\n> \n> I submitted a bug report yesterday, but they have yet to publish it.\n> \n\nmaybe you can also submit it to the OpenJDK folks? I have very bad\nexperiences with reporting java bugs to sun. I reported a few and\n_never_ heard back from them, no publish of the bug, no reject, nothing.\n\n\nGreat work on rewriting this!\nPS. still need your input on ignores :-)\n"},{"id":"113083","messageId":"20090506141526.GA28164@spearce.org","threadId":"19099","inReplyTo":"20090429171659.GF23604@spearce.org","subject":"[PATCH] Link to the Sun JVM bug mentioned in OffsetCache","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-06T14:15:26Z","receivedAt":"2009-05-06T14:15:26Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This bug has now been published by Sun.  We should link to the\ndatabase entry so we can find more detail later.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n  \"Shawn O. Pearce\" <spearce@spearce.org> wrote:\n  > Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:\n  > > > diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\n  > > > +\tprivate void gc() {\n  > > > +\t\tR r;\n  > > > +\t\twhile ((r = (R) queue.poll()) != null) {\n  > > > +\t\t\t// Sun's Java 5 and 6 implementation have a bug where a Reference\n  > > > +\t\t\t// can be enqueued and dequeued twice on the same reference queue\n  > > > +\t\t\t// due to a race condition within ReferenceQueue.enqueue(Reference).\n  > > \n  > > Reference to the official Sun bug? Might help if someone wants to\n  > > implement a flag to avoid this (if necessary...)\n  > \n  > Actually, this is a new bug.  I tried looking through BugParade\n  > but nobody has mentioned or discovered this before.\n  > \n  > I submitted a bug report yesterday, but they have yet to publish it.\n\n  And here it is.\n\n .../src/org/spearce/jgit/lib/OffsetCache.java      |    2 ++\n 1 files changed, 2 insertions(+), 0 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\nindex a1cd4be..b81c7e0 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/OffsetCache.java\n@@ -414,6 +414,8 @@ private void gc() {\n \t\t\t// can be enqueued and dequeued twice on the same reference queue\n \t\t\t// due to a race condition within ReferenceQueue.enqueue(Reference).\n \t\t\t//\n+\t\t\t// http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6837858\n+\t\t\t//\n \t\t\t// We CANNOT permit a Reference to come through us twice, as it will\n \t\t\t// skew the resource counters we maintain. Our canClear() check here\n \t\t\t// provides a way to skip the redundant dequeues, if any.\n-- \n1.6.3.rc4.206.g03e16\n"}]}