[jgit] index v2 pull request

From: Shawn O. Pearce <spearce@spearce.org>
Date: Mar 8, 2008, 02:50 UTC
Message-ID: <20080308025027.GZ8410@spearce.org>

So I've started to hack on jgit again, to build out more of the lower level library.

This particular series adds support for pack file index v2, which Nico added to C Git several months ago and is slated to become the default pack index file format in a future release of Git, due to its increased error checking during delta reuse.

I also cleaned up the pack file access code to be shorter and more concise. Unfortunately the overall series diffstat doesn't show it that way, but I think its now easier to follow how the control flows.

I've improved performance for packfile data access by adding a delta base cache, like we did in C Git months ago. This cache is really necessary on modern repositories as pack-objects is tending to use longer delta chains, due to the reader side now having this delta base cache. Profiling was showing jgit was spending a very large amount of time in unpacking tree deltas.

We now actually use the following .git/config options to control how the window cache limits work:

  core.packedGitWindowSize
  core.packedGitLimit
  core.deltaBaseCacheLimit
  core.packedGitMMAP

The configuration file parser now recognizes the "k/m/g" suffixes that are comonly used in the first three parameters. The last option is a boolean and is unique to jgit.

The window cache and delta base cache also exploit SoftReferences to make the caches memory-sensitive. This way the JVM's GC can whack our caches back if its running low on memory. Its not quite the same as C Git's concept of "malloc failed, flush out some windows and retry" but its reasonably close.

Finally we now use a single WindowCache and delta base cache for the entire JVM when running in an Eclipse workbench. If the user has multiple repositories open at once (say different projects) they need to co-habitate in the JVM's virtual address space. By sharing the same caches the repositories can evict each other's windows when they need to make room.

----

The following changes since commit 3cd4f3f9119b79750bbfc542119451a668c462e3:
  Robin Rosenberg (1):
        Readded qualifier to version 0.3.1

are available in the git repository at:

  repo.or.cz:/srv/git/egit/spearce.git master

Shawn O. Pearce (41):
      Make the absolute path of a WindowedFile available to callers
      Include the pack index file name when reporting a bad header
      Report pack file open failures to the console
      Refactor index reading code from PackFile to PackIndex
      Rename PackIndex.readIndexHeader to loadVersion1
      Read pack index files using FileInputStream and not WindowedFile
      Don't bother checking the index file length after reading it
      Move index:pack object count checking into PackFile class
      Refactor PackIndex version 1 handling code into PackIndexV1
      Fully document the PackIndex API
      Hoist readFully and decodeUInt32 up from PackIndexV1 to PackIndex
      Refactor PackIndex.readFully to support reading into the middle
      Read the pack index header to perform automatic version detection
      Add complete reading support for pack index v2
      Stop creating "new format" loose objects as C Git no longer likes them
      Refactor RepositoryConfig to be stacked many levels deep
      Don't make unit tests depend upon compressed file lengths
      Corrected name of the method that reloads the packed-refs data
      Cleanup misspelling in Javadoc of Repository class
      Extract StGitPatch from Repository to its own top-level type
      Refactor RepositoryState into its own top-level type
      Refactor PackFile construction to take the idxFile path from scan
      Shuffle the length of a WindowedFile down into its provider
      Defer opening pack files until we actually need object data
      Use Java 5 bit twiddling APIs rather than hand-rolling bit counting
      Support reading config file integers with k/m/g suffixes
      Honor core.packedGitWindowSize, packedGitLimit, packedGitMMAP config
      Enable easy access to the user's ~/.gitconfig
      Use a single WindowCache for the entire Eclipse workspace
      Modify the WindowedFile API to pass around a WindowCursor during reads
      Change ByteWindow implementations to hold data by SoftReferences
      Refactor WindowedFile/WindowCache so all windows are a uniform size
      Simplify WindowCache and WindowedFile internal APIs to be private
      Use a proper ReferenceQueue to avoid duplpicate window entries
      New RepositoryConfig getInt,getBoolean helper methods with no subsection
      Change delta packed object loaders to use a PackedObjectLoader for base
      Implement core.deltaBaseCacheLimit to improve unpacking performance
      Remove unnecessary bitwise AND masks in readTree
      Precount the number of tree entries during readTree
      Change the ObjectLoader.getType method to return int
      Fix ArrayIndexOutOfBoundsException in WindowCache

 .../egit/core/op/ConnectProviderOperation.java     |    4 +-
 .../spearce/egit/core/project/GitProjectData.java  |   27 +++-
 .../src/org/spearce/egit/ui/GitHistoryPage.java    |    2 +-
 .../internal/decorators/GitResourceDecorator.java  |    2 +-
 .../tst/org/spearce/jgit/lib/ReadTreeTest.java     |    4 +-
 .../org/spearce/jgit/lib/RepositoryConfigTest.java |    9 +-
 .../tst/org/spearce/jgit/lib/T0003_Basic.java      |   90 ---------
 .../tst/org/spearce/jgit/lib/T0004_PackReader.java |   11 +-
 org.spearce.jgit/META-INF/MANIFEST.MF              |    3 +-
 .../src/org/spearce/jgit/lib/ByteArrayWindow.java  |   27 +--
 .../src/org/spearce/jgit/lib/ByteBufferWindow.java |   21 +--
 .../src/org/spearce/jgit/lib/ByteWindow.java       |   42 +++--
 .../src/org/spearce/jgit/lib/Constants.java        |   21 ++
 .../src/org/spearce/jgit/lib/CoreConfig.java       |   61 +++++-
 .../jgit/lib/DeltaOfsPackedObjectLoader.java       |    2 +-
 .../spearce/jgit/lib/DeltaPackedObjectLoader.java  |   29 ++-
 .../jgit/lib/DeltaRefPackedObjectLoader.java       |    4 +-
 .../src/org/spearce/jgit/lib/ObjectLoader.java     |    6 +-
 .../src/org/spearce/jgit/lib/ObjectWriter.java     |   40 +---
 .../src/org/spearce/jgit/lib/PackFile.java         |  155 +++++----------
 .../src/org/spearce/jgit/lib/PackIndex.java        |  165 +++++++++++++++
 .../src/org/spearce/jgit/lib/PackIndexV1.java      |   84 ++++++++
 .../src/org/spearce/jgit/lib/PackIndexV2.java      |  125 ++++++++++++
 .../org/spearce/jgit/lib/PackedObjectLoader.java   |   13 +-
 .../src/org/spearce/jgit/lib/Repository.java       |  211 +++++---------------
 .../src/org/spearce/jgit/lib/RepositoryConfig.java |  172 +++++++++++-----
 .../src/org/spearce/jgit/lib/RepositoryState.java  |  108 ++++++++++
 .../src/org/spearce/jgit/lib/Tree.java             |   40 ++--
 .../org/spearce/jgit/lib/UnpackedObjectCache.java  |  156 +++++++++++++++
 .../org/spearce/jgit/lib/UnpackedObjectLoader.java |   20 +--
 .../spearce/jgit/lib/WholePackedObjectLoader.java  |   12 +-
 .../src/org/spearce/jgit/lib/WindowCache.java      |  151 +++++++++++---
 .../src/org/spearce/jgit/lib/WindowCursor.java     |   67 ++++++
 .../src/org/spearce/jgit/lib/WindowProvider.java   |   75 -------
 .../src/org/spearce/jgit/lib/WindowedFile.java     |  207 +++++++++++--------
 .../src/org/spearce/jgit/stgit/StGitPatch.java     |   52 +++++
 36 files changed, 1459 insertions(+), 759 deletions(-)
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndex.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexV1.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/PackIndexV2.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryState.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/UnpackedObjectCache.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/WindowCursor.java
 delete mode 100644 org.spearce.jgit/src/org/spearce/jgit/lib/WindowProvider.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/stgit/StGitPatch.java

-- 
Shawn.