{"thread":{"id":"18335","subject":"[JGIT PATCH 0/4] Avoid using 1280 file descriptors","startedAt":"2009-03-17T01:16:06Z","lastAt":"2009-03-18T01:36:16Z","messageCount":9,"participants":["Shawn O. Pearce","Robin Rosenberg"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"108166","messageId":"1237252570-8596-1-git-send-email-spearce@spearce.org","threadId":"18335","inReplyTo":null,"subject":"[JGIT PATCH 0/4] Avoid using 1280 file descriptors","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-17T01:16:06Z","receivedAt":"2009-03-17T01:16:06Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This is a small series to teach WindowCache not to use 1280 file\ndescriptors by default.  Most JVMs won't have a ulimit that large,\nresulting in out of file errors if the WindowCache actually tries\nto fully populate itself.\n\nShawn O. Pearce (4):\n  Refactor WindowCache.reconfigure() to take a configuration object\n  Update EGit plugin to use WindowCacheConfig\n  Cap the number of open files in the WindowCache\n  Teach WindowCacheConfig to read core.packedgit* settings from config\n\n .../spearce/egit/core/project/GitProjectData.java  |   12 +-\n .../org/spearce/jgit/lib/RepositoryTestCase.java   |    8 +-\n .../org/spearce/jgit/lib/UnpackedObjectCache.java  |    7 +-\n .../src/org/spearce/jgit/lib/WindowCache.java      |  119 ++++++++++----\n .../org/spearce/jgit/lib/WindowCacheConfig.java    |  170 ++++++++++++++++++++\n 5 files changed, 271 insertions(+), 45 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\n"},{"id":"108167","messageId":"1237252570-8596-2-git-send-email-spearce@spearce.org","threadId":"18335","inReplyTo":"1237252570-8596-1-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 1/4] Refactor WindowCache.reconfigure() to take a configuration object","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-17T01:16:07Z","receivedAt":"2009-03-17T01:16:07Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This makes it safer for applications to be calling reconfigure()\nshould we ever add more controllable parameters to cache.  It may\nalso leave the door open for more common option handling, such as\nreading from ~/.gitconfig in a standard way across applications\nbuilt upon JGit.  We could also add sanity check functions to the\nnew configuration object, allowing applications to inquire about\nthe validity before installing it.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n .../org/spearce/jgit/lib/RepositoryTestCase.java   |    8 +-\n .../org/spearce/jgit/lib/UnpackedObjectCache.java  |    7 +-\n .../src/org/spearce/jgit/lib/WindowCache.java      |   54 +++++---\n .../org/spearce/jgit/lib/WindowCacheConfig.java    |  134 ++++++++++++++++++++\n 4 files changed, 178 insertions(+), 25 deletions(-)\n create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RepositoryTestCase.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RepositoryTestCase.java\nindex 4e56b38..5d8c056 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RepositoryTestCase.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RepositoryTestCase.java\n@@ -113,8 +113,12 @@ public String getProperty(String key) {\n \t * Configure JGit before setting up test repositories.\n \t */\n \tprotected void configure() {\n-\t\tpackedGitMMAP = \"true\".equals(System.getProperty(\"jgit.junit.usemmmap\"));\n-\t\tWindowCache.reconfigure(128*1024, 8192, packedGitMMAP, 8192);\n+\t\tfinal WindowCacheConfig c = new WindowCacheConfig();\n+\t\tc.setPackedGitLimit(128 * WindowCacheConfig.KB);\n+\t\tc.setPackedGitWindowSize(8 * WindowCacheConfig.KB);\n+\t\tc.setPackedGitMMAP(\"true\".equals(System.getProperty(\"jgit.junit.usemmmap\")));\n+\t\tc.setDeltaBaseCacheLimit(8 * WindowCacheConfig.KB);\n+\t\tWindowCache.reconfigure(c);\n \t}\n \n \t/**\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/UnpackedObjectCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/UnpackedObjectCache.java\nindex 677b3a7..13861bf 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/UnpackedObjectCache.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/UnpackedObjectCache.java\n@@ -42,8 +42,6 @@\n class UnpackedObjectCache {\n \tprivate static final int CACHE_SZ = 1024;\n \n-\tprivate static final int MB = 1024 * 1024;\n-\n \tprivate static final SoftReference<Entry> DEAD;\n \n \tprivate static int hash(final long position) {\n@@ -62,14 +60,15 @@ private static int hash(final long position) {\n \n \tstatic {\n \t\tDEAD = new SoftReference<Entry>(null);\n-\t\tmaxByteCount = 10 * MB;\n+\t\tmaxByteCount = new WindowCacheConfig().getDeltaBaseCacheLimit();\n \n \t\tcache = new Slot[CACHE_SZ];\n \t\tfor (int i = 0; i < CACHE_SZ; i++)\n \t\t\tcache[i] = new Slot();\n \t}\n \n-\tstatic synchronized void reconfigure(final int dbLimit) {\n+\tstatic synchronized void reconfigure(final WindowCacheConfig cfg) {\n+\t\tfinal int dbLimit = cfg.getDeltaBaseCacheLimit();\n \t\tif (maxByteCount != dbLimit) {\n \t\t\tmaxByteCount = dbLimit;\n \t\t\treleaseMemory();\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\nindex 6a650cb..ba1124a 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\n@@ -46,10 +46,6 @@\n  * the other windowed file access classes.\n  */\n public class WindowCache {\n-\tprivate static final int KB = 1024;\n-\n-\tprivate static final int MB = 1024 * KB;\n-\n \tprivate static final int bits(int newSize) {\n \t\tif (newSize < 4096)\n \t\t\tthrow new IllegalArgumentException(\"Invalid window size\");\n@@ -77,10 +73,11 @@ private static final int bits(int newSize) {\n \tprivate static int openByteCount;\n \n \tstatic {\n-\t\tmaxByteCount = 10 * MB;\n-\t\twindowSizeShift = bits(8 * KB);\n+\t\tfinal WindowCacheConfig c = new WindowCacheConfig();\n+\t\tmaxByteCount = c.getPackedGitLimit();\n+\t\twindowSizeShift = bits(c.getPackedGitWindowSize());\n \t\twindowSize = 1 << windowSizeShift;\n-\t\tmmap = false;\n+\t\tmmap = c.isPackedGitMMAP();\n \t\tcache = new ByteWindow[cacheTableSize()];\n \t\tclearedWindowQueue = new ReferenceQueue<Object>();\n \t}\n@@ -104,34 +101,53 @@ private static int cacheTableSize() {\n \t *            true to enable use of mmap when creating windows.\n \t * @param deltaBaseCacheLimit\n \t *            number of bytes to hold in the delta base cache.\n+\t * @deprecated Use {@link WindowCacheConfig} instead.\n \t */\n \tpublic static void reconfigure(final int packedGitLimit,\n \t\t\tfinal int packedGitWindowSize, final boolean packedGitMMAP,\n \t\t\tfinal int deltaBaseCacheLimit) {\n-\t\treconfigureImpl(packedGitLimit, packedGitWindowSize, packedGitMMAP);\n-\t\tUnpackedObjectCache.reconfigure(deltaBaseCacheLimit);\n+\t\tfinal WindowCacheConfig c = new WindowCacheConfig();\n+\t\tc.setPackedGitLimit(packedGitLimit);\n+\t\tc.setPackedGitWindowSize(packedGitWindowSize);\n+\t\tc.setPackedGitMMAP(packedGitMMAP);\n+\t\tc.setDeltaBaseCacheLimit(deltaBaseCacheLimit);\n+\t\treconfigure(c);\n+\t}\n+\n+\t/**\n+\t * Modify the configuration of the window cache.\n+\t * <p>\n+\t * The new configuration is applied immediately. If the new limits are\n+\t * smaller than what what is currently cached, older entries will be purged\n+\t * as soon as possible to allow the cache to meet the new limit.\n+\t *\n+\t * @param cfg\n+\t *            the new window cache configuration.\n+\t */\n+\tpublic static void reconfigure(final WindowCacheConfig cfg) {\n+\t\treconfigureImpl(cfg);\n+\t\tUnpackedObjectCache.reconfigure(cfg);\n \t}\n \n-\tprivate static synchronized void reconfigureImpl(final int packedGitLimit,\n-\t\t\tfinal int packedGitWindowSize, final boolean packedGitMMAP) {\n+\tprivate static synchronized void reconfigureImpl(final WindowCacheConfig cfg) {\n \t\tboolean prune = false;\n \t\tboolean evictAll = false;\n \n-\t\tif (maxByteCount < packedGitLimit) {\n-\t\t\tmaxByteCount = packedGitLimit;\n-\t\t} else if (maxByteCount > packedGitLimit) {\n-\t\t\tmaxByteCount = packedGitLimit;\n+\t\tif (maxByteCount < cfg.getPackedGitLimit()) {\n+\t\t\tmaxByteCount = cfg.getPackedGitLimit();\n+\t\t} else if (maxByteCount > cfg.getPackedGitLimit()) {\n+\t\t\tmaxByteCount = cfg.getPackedGitLimit();\n \t\t\tprune = true;\n \t\t}\n \n-\t\tif (bits(packedGitWindowSize) != windowSizeShift) {\n-\t\t\twindowSizeShift = bits(packedGitWindowSize);\n+\t\tif (bits(cfg.getPackedGitWindowSize()) != windowSizeShift) {\n+\t\t\twindowSizeShift = bits(cfg.getPackedGitWindowSize());\n \t\t\twindowSize = 1 << windowSizeShift;\n \t\t\tevictAll = true;\n \t\t}\n \n-\t\tif (mmap != packedGitMMAP) {\n-\t\t\tmmap = packedGitMMAP;\n+\t\tif (mmap != cfg.isPackedGitMMAP()) {\n+\t\t\tmmap = cfg.isPackedGitMMAP();\n \t\t\tevictAll = true;\n \t\t}\n \ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\nnew file mode 100644\nindex 0000000..b4c4638\n--- /dev/null\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\n@@ -0,0 +1,134 @@\n+/*\n+ * Copyright (C) 2009, Google Inc.\n+ *\n+ * All rights reserved.\n+ *\n+ * Redistribution and use in source and binary forms, with or\n+ * without modification, are permitted provided that the following\n+ * conditions are met:\n+ *\n+ * - Redistributions of source code must retain the above copyright\n+ *   notice, this list of conditions and the following disclaimer.\n+ *\n+ * - Redistributions in binary form must reproduce the above\n+ *   copyright notice, this list of conditions and the following\n+ *   disclaimer in the documentation and/or other materials provided\n+ *   with the distribution.\n+ *\n+ * - Neither the name of the Git Development Community nor the\n+ *   names of its contributors may be used to endorse or promote\n+ *   products derived from this software without specific prior\n+ *   written permission.\n+ *\n+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND\n+ * CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES,\n+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT\n+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF\n+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+ */\n+\n+package org.spearce.jgit.lib;\n+\n+/** Configuration parameters for {@link WindowCache}. */\n+public class WindowCacheConfig {\n+\t/** 1024 (number of bytes in one kibibyte/kilobyte) */\n+\tpublic static final int KB = 1024;\n+\n+\t/** 1024 {@link #KB} (number of bytes in one mebibyte/megabyte) */\n+\tpublic static final int MB = 1024 * KB;\n+\n+\tprivate int packedGitLimit;\n+\n+\tprivate int packedGitWindowSize;\n+\n+\tprivate boolean packedGitMMAP;\n+\n+\tprivate int deltaBaseCacheLimit;\n+\n+\t/** Create a default configuration. */\n+\tpublic WindowCacheConfig() {\n+\t\tpackedGitLimit = 10 * MB;\n+\t\tpackedGitWindowSize = 8 * KB;\n+\t\tpackedGitMMAP = false;\n+\t\tdeltaBaseCacheLimit = 10 * MB;\n+\t}\n+\n+\t/**\n+\t * @return maximum number bytes of heap memory to dedicate to caching pack\n+\t *         file data. <b>Default is 10 MB.</b>\n+\t */\n+\tpublic int getPackedGitLimit() {\n+\t\treturn packedGitLimit;\n+\t}\n+\n+\t/**\n+\t * @param newLimit\n+\t *            maximum number bytes of heap memory to dedicate to caching\n+\t *            pack file data.\n+\t */\n+\tpublic void setPackedGitLimit(final int newLimit) {\n+\t\tpackedGitLimit = newLimit;\n+\t}\n+\n+\t/**\n+\t * @return size in bytes of a single window mapped or read in from the pack\n+\t *         file. <b>Default is 8 KB.</b>\n+\t */\n+\tpublic int getPackedGitWindowSize() {\n+\t\treturn packedGitWindowSize;\n+\t}\n+\n+\t/**\n+\t * @param newSize\n+\t *            size in bytes of a single window read in from the pack file.\n+\t */\n+\tpublic void setPackedGitWindowSize(final int newSize) {\n+\t\tpackedGitWindowSize = newSize;\n+\t}\n+\n+\t/**\n+\t * @return true enables use of Java NIO virtual memory mapping for windows;\n+\t *         false reads entire window into a byte[] with standard read calls.\n+\t *         <b>Default false.</b>\n+\t */\n+\tpublic boolean isPackedGitMMAP() {\n+\t\treturn packedGitMMAP;\n+\t}\n+\n+\t/**\n+\t * @param usemmap\n+\t *            true enables use of Java NIO virtual memory mapping for\n+\t *            windows; false reads entire window into a byte[] with standard\n+\t *            read calls.\n+\t */\n+\tpublic void setPackedGitMMAP(final boolean usemmap) {\n+\t\tpackedGitMMAP = usemmap;\n+\t}\n+\n+\t/**\n+\t * @return maximum number of bytes to cache in {@link UnpackedObjectCache}\n+\t *         for inflated, recently accessed objects, without delta chains.\n+\t *         <b>Default 10 MB.</b>\n+\t */\n+\tpublic int getDeltaBaseCacheLimit() {\n+\t\treturn deltaBaseCacheLimit;\n+\t}\n+\n+\t/**\n+\t * @param newLimit\n+\t *            maximum number of bytes to cache in\n+\t *            {@link UnpackedObjectCache} for inflated, recently accessed\n+\t *            objects, without delta chains.\n+\t */\n+\tpublic void setDeltaBaseCacheLimit(final int newLimit) {\n+\t\tdeltaBaseCacheLimit = newLimit;\n+\t}\n+}\n-- \n1.6.2.1.286.g8173\n"},{"id":"108168","messageId":"1237252570-8596-3-git-send-email-spearce@spearce.org","threadId":"18335","inReplyTo":"1237252570-8596-2-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 2/4] Update EGit plugin to use WindowCacheConfig","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-17T01:16:08Z","receivedAt":"2009-03-17T01:16:08Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Its more flexible for future additions to the parameters.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n .../spearce/egit/core/project/GitProjectData.java  |   12 +++++++-----\n 1 files changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/org.spearce.egit.core/src/org/spearce/egit/core/project/GitProjectData.java b/org.spearce.egit.core/src/org/spearce/egit/core/project/GitProjectData.java\nindex 8b72818..31d5483 100644\n--- a/org.spearce.egit.core/src/org/spearce/egit/core/project/GitProjectData.java\n+++ b/org.spearce.egit.core/src/org/spearce/egit/core/project/GitProjectData.java\n@@ -42,6 +42,7 @@\n import org.spearce.egit.core.GitProvider;\n import org.spearce.jgit.lib.Repository;\n import org.spearce.jgit.lib.WindowCache;\n+import org.spearce.jgit.lib.WindowCacheConfig;\n \n /**\n  * This class keeps information about how a project is mapped to\n@@ -229,12 +230,13 @@ private synchronized static Repository lookupRepository(final File gitDir)\n \t * Update the settings for the global window cache of the workspace.\n \t */\n \tpublic static void reconfigureWindowCache() {\n+\t\tfinal WindowCacheConfig c = new WindowCacheConfig();\n \t\tPreferences p = Activator.getDefault().getPluginPreferences();\n-\t\tint wLimit = p.getInt(GitCorePreferences.core_packedGitLimit);\n-\t\tint wSize = p.getInt(GitCorePreferences.core_packedGitWindowSize);\n-\t\tboolean mmap = p.getBoolean(GitCorePreferences.core_packedGitMMAP);\n-\t\tint dbLimit = p.getInt(GitCorePreferences.core_deltaBaseCacheLimit);\n-\t\tWindowCache.reconfigure(wLimit, wSize, mmap, dbLimit);\n+\t\tc.setPackedGitLimit(p.getInt(GitCorePreferences.core_packedGitLimit));\n+\t\tc.setPackedGitWindowSize(p.getInt(GitCorePreferences.core_packedGitWindowSize));\n+\t\tc.setPackedGitMMAP(p.getBoolean(GitCorePreferences.core_packedGitMMAP));\n+\t\tc.setDeltaBaseCacheLimit(p.getInt(GitCorePreferences.core_deltaBaseCacheLimit));\n+\t\tWindowCache.reconfigure(c);\n \t}\n \n \tprivate final IProject project;\n-- \n1.6.2.1.286.g8173\n"},{"id":"108170","messageId":"1237252570-8596-4-git-send-email-spearce@spearce.org","threadId":"18335","inReplyTo":"1237252570-8596-3-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 3/4] Cap the number of open files in the WindowCache","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-17T01:16:09Z","receivedAt":"2009-03-17T01:16:09Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The default WindowCache configuration is:\n\n  packedGitLimit:      10 MB\n  packedGitWindowSize:  8 KB\n\n10 MB / 8 KB allows up to 1280 windows permitted in the WindowCache\nat any given time.  If every window came from a unique pack file, we\nneed 1280 file descriptors just for the resources in the WindowCache.\nFor most applications this is way beyond the hard limit configured\nfor the host JVM, causing java.io.IOException(\"Too many open files\")\nfrom possibly any part of JGit or the host application.\n\nSpecifically, I ran into this problem in Gerrit Code Review, where we\ncommonly see hundreds of very small pack files spread over hundreds\nof Git repositories, all accessed from a persistent JVM that is also\nhosting an SSH daemon and a web server.  The aggressive caching of\nwindows in the WindowCache and of Repository objects in Gerrit's\nown RepositoryCache caused us to retain far too many tiny pack files.\n\nWe now set the limit at 128 open files, assuming this is a reasonable\nlimit for most applications using the library.  Git repositories tend\nto be in the handful of packs/repository (e.g. <10 packs/repository)\nand applications using JGit tend to access only a handful of Git\nrepositories at a time (e.g. <10 repositories/JVM).\n\nIf we detect a file open failure while opening a pack we halve\nthe number of permitted open files and try again, until we reach\na lower bound of 16 open files.  Needing to go lower may indicate\na file descriptor leak in the host application.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n .../src/org/spearce/jgit/lib/WindowCache.java      |   65 +++++++++++++++-----\n .../org/spearce/jgit/lib/WindowCacheConfig.java    |   20 ++++++\n 2 files changed, 70 insertions(+), 15 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\nindex ba1124a..13912a7 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\n@@ -54,6 +54,8 @@ private static final int bits(int newSize) {\n \t\treturn Integer.numberOfTrailingZeros(newSize);\n \t}\n \n+\tprivate static int maxFileCount;\n+\n \tprivate static int maxByteCount;\n \n \tprivate static int windowSize;\n@@ -70,10 +72,13 @@ private static final int bits(int newSize) {\n \n \tprivate static ByteWindow lruTail;\n \n+\tprivate static int openFileCount;\n+\n \tprivate static int openByteCount;\n \n \tstatic {\n \t\tfinal WindowCacheConfig c = new WindowCacheConfig();\n+\t\tmaxFileCount = c.getPackedGitOpenFiles();\n \t\tmaxByteCount = c.getPackedGitLimit();\n \t\twindowSizeShift = bits(c.getPackedGitWindowSize());\n \t\twindowSize = 1 << windowSizeShift;\n@@ -133,6 +138,13 @@ private static synchronized void reconfigureImpl(final WindowCacheConfig cfg) {\n \t\tboolean prune = false;\n \t\tboolean evictAll = false;\n \n+\t\tif (maxFileCount < cfg.getPackedGitOpenFiles())\n+\t\t\tmaxFileCount = cfg.getPackedGitOpenFiles();\n+\t\telse if (maxFileCount > cfg.getPackedGitOpenFiles()) {\n+\t\t\tmaxFileCount = cfg.getPackedGitOpenFiles();\n+\t\t\tprune = true;\n+\t\t}\n+\n \t\tif (maxByteCount < cfg.getPackedGitLimit()) {\n \t\t\tmaxByteCount = cfg.getPackedGitLimit();\n \t\t} else if (maxByteCount > cfg.getPackedGitLimit()) {\n@@ -229,20 +241,37 @@ private static synchronized final void getImpl(final WindowCursor curs,\n \t\t}\n \n \t\tif (wp.openCount == 0) {\n-\t\t\ttry {\n-\t\t\t\twp.openCount = 1;\n-\t\t\t\twp.cacheOpen();\n-\t\t\t} catch (IOException ioe) {\n-\t\t\t\twp.openCount = 0;\n-\t\t\t\tthrow ioe;\n-\t\t\t} catch (RuntimeException ioe) {\n-\t\t\t\twp.openCount = 0;\n-\t\t\t\tthrow ioe;\n-\t\t\t} catch (Error ioe) {\n-\t\t\t\twp.openCount = 0;\n-\t\t\t\tthrow ioe;\n-\t\t\t} finally {\n-\t\t\t\twp.openCount--;\n+\t\t\tTRY_OPEN: for (;;) {\n+\t\t\t\ttry {\n+\t\t\t\t\topenFileCount++;\n+\t\t\t\t\treleaseMemory();\n+\t\t\t\t\trunClearedWindowQueue();\n+\t\t\t\t\twp.openCount = 1;\n+\t\t\t\t\twp.cacheOpen();\n+\t\t\t\t\tbreak;\n+\t\t\t\t} catch (IOException ioe) {\n+\t\t\t\t\topenFileCount--;\n+\t\t\t\t\tif (\"Too many open files\".equals(ioe.getMessage())\n+\t\t\t\t\t\t\t&& maxFileCount > 16) {\n+\t\t\t\t\t\t// We may be able to recover by halving our limit\n+\t\t\t\t\t\t// and trying again.\n+\t\t\t\t\t\t//\n+\t\t\t\t\t\tmaxFileCount = Math.max(16, maxFileCount >> 2);\n+\t\t\t\t\t\tcontinue TRY_OPEN;\n+\t\t\t\t\t}\n+\t\t\t\t\twp.openCount = 0;\n+\t\t\t\t\tthrow ioe;\n+\t\t\t\t} catch (RuntimeException ioe) {\n+\t\t\t\t\topenFileCount--;\n+\t\t\t\t\twp.openCount = 0;\n+\t\t\t\t\tthrow ioe;\n+\t\t\t\t} catch (Error ioe) {\n+\t\t\t\t\topenFileCount--;\n+\t\t\t\t\twp.openCount = 0;\n+\t\t\t\t\tthrow ioe;\n+\t\t\t\t} finally {\n+\t\t\t\t\twp.openCount--;\n+\t\t\t\t}\n \t\t\t}\n \n \t\t\t// The cacheOpen may have mapped the window we are trying to\n@@ -278,6 +307,7 @@ private static synchronized final void getImpl(final WindowCursor curs,\n \n \tstatic synchronized void markLoaded(final ByteWindow w) {\n \t\tif (--w.provider.openCount == 0) {\n+\t\t\topenFileCount--;\n \t\t\tw.provider.cacheClose();\n \t\t}\n \t}\n@@ -291,13 +321,17 @@ private static void makeMostRecent(ByteWindow<?> e) {\n \n \tprivate static void releaseMemory() {\n \t\tByteWindow<?> e = lruTail;\n-\t\twhile (openByteCount > maxByteCount && e != null) {\n+\t\twhile (isOverLimit() && e != null) {\n \t\t\tfinal ByteWindow<?> p = e.lruPrev;\n \t\t\tclear(e);\n \t\t\te = p;\n \t\t}\n \t}\n \n+\tprivate static boolean isOverLimit() {\n+\t\treturn openByteCount > maxByteCount || openFileCount > maxFileCount;\n+\t}\n+\n \t/**\n \t * Remove all windows associated with a specific provider.\n \t * <p>\n@@ -341,6 +375,7 @@ private static void clear(final ByteWindow<?> e) {\n \tprivate static void unlinkSize(final ByteWindow<?> e) {\n \t\tif (e.sizeActive) {\n \t\t\tif (--e.provider.openCount == 0) {\n+\t\t\t\topenFileCount--;\n \t\t\t\te.provider.cacheClose();\n \t\t\t}\n \t\t\topenByteCount -= e.size;\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\nindex b4c4638..d906a7c 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\n@@ -45,6 +45,8 @@\n \t/** 1024 {@link #KB} (number of bytes in one mebibyte/megabyte) */\n \tpublic static final int MB = 1024 * KB;\n \n+\tprivate int packedGitOpenFiles;\n+\n \tprivate int packedGitLimit;\n \n \tprivate int packedGitWindowSize;\n@@ -55,6 +57,7 @@\n \n \t/** Create a default configuration. */\n \tpublic WindowCacheConfig() {\n+\t\tpackedGitOpenFiles = 128;\n \t\tpackedGitLimit = 10 * MB;\n \t\tpackedGitWindowSize = 8 * KB;\n \t\tpackedGitMMAP = false;\n@@ -62,6 +65,23 @@ public WindowCacheConfig() {\n \t}\n \n \t/**\n+\t * @return maximum number of streams to open at a time. Open packs count\n+\t *         against the process limits. <b>Default is 128.</b>\n+\t */\n+\tpublic int getPackedGitOpenFiles() {\n+\t\treturn packedGitOpenFiles;\n+\t}\n+\n+\t/**\n+\t * @param fdLimit\n+\t *            maximum number of streams to open at a time. Open packs count\n+\t *            against the process limits\n+\t */\n+\tpublic void setPackedGitOpenFiles(final int fdLimit) {\n+\t\tpackedGitOpenFiles = fdLimit;\n+\t}\n+\n+\t/**\n \t * @return maximum number bytes of heap memory to dedicate to caching pack\n \t *         file data. <b>Default is 10 MB.</b>\n \t */\n-- \n1.6.2.1.286.g8173\n"},{"id":"108169","messageId":"1237252570-8596-5-git-send-email-spearce@spearce.org","threadId":"18335","inReplyTo":"1237252570-8596-4-git-send-email-spearce@spearce.org","subject":"[JGIT PATCH 4/4] Teach WindowCacheConfig to read core.packedgit* settings from config","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-17T01:16:10Z","receivedAt":"2009-03-17T01:16:10Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This may help applications which want to read configuration from\nthe current repository, or from ~/.gitconfig when picking the access\nstrategy for the current process.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n .../org/spearce/jgit/lib/WindowCacheConfig.java    |   16 ++++++++++++++++\n 1 files changed, 16 insertions(+), 0 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\nindex d906a7c..ea28164 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\n@@ -151,4 +151,20 @@ public int getDeltaBaseCacheLimit() {\n \tpublic void setDeltaBaseCacheLimit(final int newLimit) {\n \t\tdeltaBaseCacheLimit = newLimit;\n \t}\n+\n+\t/**\n+\t * Update properties by setting fields from the configuration.\n+\t * <p>\n+\t * If a property is not defined in the configuration, then it is left\n+\t * unmodified.\n+\t *\n+\t * @param rc configuration to read properties from.\n+\t */\n+\tpublic void fromConfig(final RepositoryConfig rc) {\n+\t\tsetPackedGitOpenFiles(rc.getInt(\"core\", null, \"packedgitopenfiles\", getPackedGitOpenFiles()));\n+\t\tsetPackedGitLimit(rc.getInt(\"core\", null, \"packedgitlimit\", getPackedGitLimit()));\n+\t\tsetPackedGitWindowSize(rc.getInt(\"core\", null, \"packedgitwindowsize\", getPackedGitWindowSize()));\n+\t\tsetPackedGitMMAP(rc.getBoolean(\"core\", null, \"packedgitmmap\", isPackedGitMMAP()));\n+\t\tsetDeltaBaseCacheLimit(rc.getInt(\"core\", null, \"deltabasecachelimit\", getDeltaBaseCacheLimit()));\n+\t}\n }\n-- \n1.6.2.1.286.g8173\n"},{"id":"108290","messageId":"200903172359.32595.robin.rosenberg.lists@dewire.com","threadId":"18335","inReplyTo":"1237252570-8596-4-git-send-email-spearce@spearce.org","subject":"Re: [JGIT PATCH 3/4] Cap the number of open files in the WindowCache","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-03-17T22:59:32Z","receivedAt":"2009-03-17T22:59:32Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"tisdag 17 mars 2009 02:16:09 skrev \"Shawn O. Pearce\" <spearce@spearce.org>:\n\n> If we detect a file open failure while opening a pack we halve\n> the number of permitted open files and try again, until we reach\n> a lower bound of 16 open files.  Needing to go lower may indicate\n> a file descriptor leak in the host application.\n...\n> +\t\t\tTRY_OPEN: for (;;) {\n> +\t\t\t\ttry {\n> +\t\t\t\t\topenFileCount++;\n> +\t\t\t\t\treleaseMemory();\n> +\t\t\t\t\trunClearedWindowQueue();\n> +\t\t\t\t\twp.openCount = 1;\n> +\t\t\t\t\twp.cacheOpen();\n> +\t\t\t\t\tbreak;\n> +\t\t\t\t} catch (IOException ioe) {\n> +\t\t\t\t\topenFileCount--;\n> +\t\t\t\t\tif (\"Too many open files\".equals(ioe.getMessage())\n> +\t\t\t\t\t\t\t&& maxFileCount > 16) {\n\nThe output of getMessage isn't that simple to interpret. Here it is filename+\" (Too many files open)\",\nand on other platforms it is probably something else. This goes for the message part of most exceptions\nthrown from platform specific code like file i/o socket i/o etc. The type of exception is a FileNotFoundException,\nbtw.\n\nI wonder whether  your code works on any platform.\n"},{"id":"108291","messageId":"20090317230803.GA23521@spearce.org","threadId":"18335","inReplyTo":"200903172359.32595.robin.rosenberg.lists@dewire.com","subject":"Re: [JGIT PATCH 3/4] Cap the number of open files in the WindowCache","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-17T23:08:03Z","receivedAt":"2009-03-17T23:08:03Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:\n> tisdag 17 mars 2009 02:16:09 skrev \"Shawn O. Pearce\" <spearce@spearce.org>:\n> \n> > If we detect a file open failure while opening a pack we halve\n> > the number of permitted open files and try again, [...]\n> \n> The output of getMessage isn't that simple to interpret. Here it is filename+\" (Too many files open)\",\n> and on other platforms it is probably something else. This goes for the message part of most exceptions\n> thrown from platform specific code like file i/o socket i/o etc. The type of exception is a FileNotFoundException,\n> btw.\n> \n> I wonder whether  your code works on any platform.\n\nArrrgh.\n \nOK.  Maybe scrap that part of the patch then?\n\nIts too bad they don't have a specific type of exception for this,\nnor do they have a way to hold onto file descriptors under a type\nlike a SoftReference where the runtime can whack them if you have\ntoo many.\n\nI guess that's why Hadoop HBase just tells you to up your fd ulimit\nto 32767.  :-)\n\n-- \nShawn.\n"},{"id":"108296","messageId":"200903180109.40074.robin.rosenberg.lists@dewire.com","threadId":"18335","inReplyTo":"20090317230803.GA23521@spearce.org","subject":"Re: [JGIT PATCH 3/4] Cap the number of open files in the WindowCache","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-03-18T00:09:39Z","receivedAt":"2009-03-18T00:09:39Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"onsdag 18 mars 2009 00:08:03 skrev \"Shawn O. Pearce\" <spearce@spearce.org>:\n> Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:\n> > tisdag 17 mars 2009 02:16:09 skrev \"Shawn O. Pearce\" <spearce@spearce.org>:\n> > \n> > > If we detect a file open failure while opening a pack we halve\n> > > the number of permitted open files and try again, [...]\n> > \n> > The output of getMessage isn't that simple to interpret. Here it is filename+\" (Too many files open)\",\n> > and on other platforms it is probably something else. This goes for the message part of most exceptions\n> > thrown from platform specific code like file i/o socket i/o etc. The type of exception is a FileNotFoundException,\n> > btw.\n> > \n> > I wonder whether  your code works on any platform.\n> \n> Arrrgh.\n>  \n> OK.  Maybe scrap that part of the patch then?\nYes, I think so, inless you want to try something as ugly as getMessage().toLower().indexof(\"(too many\")? Not\nsure what it looks like in Windows or OSX. We know from the JDK source it's filename + \"(\" + reason +\")\" and\nthe problem here is the reason part.\n\n> Its too bad they don't have a specific type of exception for this,\nFileNotFoundException is a little more specific. Maybe in combinatikon with a file.exists() and file.canRead test...\n(thinking loud now)\n\n> nor do they have a way to hold onto file descriptors under a type\n> like a SoftReference where the runtime can whack them if you have\n> too many.\n\nThe problem is that it's not connected to file descriptors but to memory. Doing a GC on filenotfoundexception\n(here) could help here if one uses soft reference, or one could prune the cache manually. The parameter\ncould also be a \n\n> \n> I guess that's why Hadoop HBase just tells you to up your fd ulimit\n> to 32767.  :-)\n\nYeah, with gigabytes of memory that might not consume too much resources anyway.\n\n-- robin\n"},{"id":"108308","messageId":"20090318013616.GB23521@spearce.org","threadId":"18335","inReplyTo":"200903180109.40074.robin.rosenberg.lists@dewire.com","subject":"[JGIT PATCH 3/4 v2] Cap the number of open files in the WindowCache","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-18T01:36:16Z","receivedAt":"2009-03-18T01:36:16Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The default WindowCache configuration is:\n\n  packedGitLimit:      10 MB\n  packedGitWindowSize:  8 KB\n\n10 MB / 8 KB allows up to 1280 windows permitted in the WindowCache\nat any given time.  If every window came from a unique pack file, we\nneed 1280 file descriptors just for the resources in the WindowCache.\nFor most applications this is way beyond the hard limit configured\nfor the host JVM, causing java.io.IOException(\"Too many open files\")\nfrom possibly any part of JGit or the host application.\n\nSpecifically, I ran into this problem in Gerrit Code Review, where we\ncommonly see hundreds of very small pack files spread over hundreds\nof Git repositories, all accessed from a persistent JVM that is also\nhosting an SSH daemon and a web server.  The aggressive caching of\nwindows in the WindowCache and of Repository objects in Gerrit's\nown RepositoryCache caused us to retain far too many tiny pack files.\n\nWe now set the limit at 128 open files, assuming this is a reasonable\nlimit for most applications using the library.  Git repositories tend\nto be in the handful of packs/repository (e.g. <10 packs/repository)\nand applications using JGit tend to access only a handful of Git\nrepositories at a time (e.g. <10 repositories/JVM).\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n Robin Rosenberg <robin.rosenberg.lists@dewire.com> wrote:\n > onsdag 18 mars 2009 00:08:03 skrev \"Shawn O. Pearce\" <spearce@spearce.org>:\n > > Arrrgh.\n > >  \n > > OK.  Maybe scrap that part of the patch then?\n > Yes, I think so\n\n Scrapped the ugly test for getMessage() when the open fails.\n So we no longer try to reduce fd usage on failure.\n\n .../src/org/spearce/jgit/lib/WindowCache.java      |   26 +++++++++++++++++++-\n .../org/spearce/jgit/lib/WindowCacheConfig.java    |   20 +++++++++++++++\n 2 files changed, 45 insertions(+), 1 deletions(-)\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\nindex ba1124a..f755dad 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCache.java\n@@ -54,6 +54,8 @@ private static final int bits(int newSize) {\n \t\treturn Integer.numberOfTrailingZeros(newSize);\n \t}\n \n+\tprivate static int maxFileCount;\n+\n \tprivate static int maxByteCount;\n \n \tprivate static int windowSize;\n@@ -70,10 +72,13 @@ private static final int bits(int newSize) {\n \n \tprivate static ByteWindow lruTail;\n \n+\tprivate static int openFileCount;\n+\n \tprivate static int openByteCount;\n \n \tstatic {\n \t\tfinal WindowCacheConfig c = new WindowCacheConfig();\n+\t\tmaxFileCount = c.getPackedGitOpenFiles();\n \t\tmaxByteCount = c.getPackedGitLimit();\n \t\twindowSizeShift = bits(c.getPackedGitWindowSize());\n \t\twindowSize = 1 << windowSizeShift;\n@@ -133,6 +138,13 @@ private static synchronized void reconfigureImpl(final WindowCacheConfig cfg) {\n \t\tboolean prune = false;\n \t\tboolean evictAll = false;\n \n+\t\tif (maxFileCount < cfg.getPackedGitOpenFiles())\n+\t\t\tmaxFileCount = cfg.getPackedGitOpenFiles();\n+\t\telse if (maxFileCount > cfg.getPackedGitOpenFiles()) {\n+\t\t\tmaxFileCount = cfg.getPackedGitOpenFiles();\n+\t\t\tprune = true;\n+\t\t}\n+\n \t\tif (maxByteCount < cfg.getPackedGitLimit()) {\n \t\t\tmaxByteCount = cfg.getPackedGitLimit();\n \t\t} else if (maxByteCount > cfg.getPackedGitLimit()) {\n@@ -230,15 +242,21 @@ private static synchronized final void getImpl(final WindowCursor curs,\n \n \t\tif (wp.openCount == 0) {\n \t\t\ttry {\n+\t\t\t\topenFileCount++;\n+\t\t\t\treleaseMemory();\n+\t\t\t\trunClearedWindowQueue();\n \t\t\t\twp.openCount = 1;\n \t\t\t\twp.cacheOpen();\n \t\t\t} catch (IOException ioe) {\n+\t\t\t\topenFileCount--;\n \t\t\t\twp.openCount = 0;\n \t\t\t\tthrow ioe;\n \t\t\t} catch (RuntimeException ioe) {\n+\t\t\t\topenFileCount--;\n \t\t\t\twp.openCount = 0;\n \t\t\t\tthrow ioe;\n \t\t\t} catch (Error ioe) {\n+\t\t\t\topenFileCount--;\n \t\t\t\twp.openCount = 0;\n \t\t\t\tthrow ioe;\n \t\t\t} finally {\n@@ -278,6 +296,7 @@ private static synchronized final void getImpl(final WindowCursor curs,\n \n \tstatic synchronized void markLoaded(final ByteWindow w) {\n \t\tif (--w.provider.openCount == 0) {\n+\t\t\topenFileCount--;\n \t\t\tw.provider.cacheClose();\n \t\t}\n \t}\n@@ -291,13 +310,17 @@ private static void makeMostRecent(ByteWindow<?> e) {\n \n \tprivate static void releaseMemory() {\n \t\tByteWindow<?> e = lruTail;\n-\t\twhile (openByteCount > maxByteCount && e != null) {\n+\t\twhile (isOverLimit() && e != null) {\n \t\t\tfinal ByteWindow<?> p = e.lruPrev;\n \t\t\tclear(e);\n \t\t\te = p;\n \t\t}\n \t}\n \n+\tprivate static boolean isOverLimit() {\n+\t\treturn openByteCount > maxByteCount || openFileCount > maxFileCount;\n+\t}\n+\n \t/**\n \t * Remove all windows associated with a specific provider.\n \t * <p>\n@@ -341,6 +364,7 @@ private static void clear(final ByteWindow<?> e) {\n \tprivate static void unlinkSize(final ByteWindow<?> e) {\n \t\tif (e.sizeActive) {\n \t\t\tif (--e.provider.openCount == 0) {\n+\t\t\t\topenFileCount--;\n \t\t\t\te.provider.cacheClose();\n \t\t\t}\n \t\t\topenByteCount -= e.size;\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\nindex b4c4638..d906a7c 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/WindowCacheConfig.java\n@@ -45,6 +45,8 @@\n \t/** 1024 {@link #KB} (number of bytes in one mebibyte/megabyte) */\n \tpublic static final int MB = 1024 * KB;\n \n+\tprivate int packedGitOpenFiles;\n+\n \tprivate int packedGitLimit;\n \n \tprivate int packedGitWindowSize;\n@@ -55,6 +57,7 @@\n \n \t/** Create a default configuration. */\n \tpublic WindowCacheConfig() {\n+\t\tpackedGitOpenFiles = 128;\n \t\tpackedGitLimit = 10 * MB;\n \t\tpackedGitWindowSize = 8 * KB;\n \t\tpackedGitMMAP = false;\n@@ -62,6 +65,23 @@ public WindowCacheConfig() {\n \t}\n \n \t/**\n+\t * @return maximum number of streams to open at a time. Open packs count\n+\t *         against the process limits. <b>Default is 128.</b>\n+\t */\n+\tpublic int getPackedGitOpenFiles() {\n+\t\treturn packedGitOpenFiles;\n+\t}\n+\n+\t/**\n+\t * @param fdLimit\n+\t *            maximum number of streams to open at a time. Open packs count\n+\t *            against the process limits\n+\t */\n+\tpublic void setPackedGitOpenFiles(final int fdLimit) {\n+\t\tpackedGitOpenFiles = fdLimit;\n+\t}\n+\n+\t/**\n \t * @return maximum number bytes of heap memory to dedicate to caching pack\n \t *         file data. <b>Default is 10 MB.</b>\n \t */\n-- \n1.6.2.1.286.g8173\n"}]}