git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[EGIT PATCH 00/26] New DirCache API

From
Shawn O. Pearce <spearce@spearce.org>
Date
Aug 12, 2008, 01:07 UTC
Message-ID
<1218503293-14057-1-git-send-email-spearce@spearce.org>

This is (finally) my long-promised DirCache API. It has been evolving and growing since before April I think. Its finally cleaned up enough for public viewing and usage.

I started this change because I didn't see an easy way to plug GitIndex into TreeWalk, I found GitIndex to be too slow, and it required direct access to the working directory in order to handle certain operations. DirCache has none of these issues.

The API is meant to replace GitIndex, as we can now merge-sort a 'DIRC' (aka .git/index) file against any other tree as part of any active TreeWalk instance. This makes it really easy to do n-way diffs against commits, one or more index files, and the working directory.

The implementation is leaner and meaner than GitIndex. It avoids a lot of unnecessary conversion overheads, and it tries to reduce the per-entry memory usage as much as possible.

We almost support the 'TREE' cache extension. Updates to the index through DirCacheBuilder or DirCacheEditor wind up throwing away the entire 'TREE' extension, instead of invalidating only those trees we know to be dirty. This is certainly suboptimal and is an area for future improvement.

>From here I'm going to start building 3 way merge support onto

this API, by taking advantage of TreeWalk to do the parallel tree unpacks necessary for the merge operation.

Yes, its ~4.5k lines (excluding some test vectors). However it will allow us to eventually remove about 2k lines from jgit alone:

  AbstractIndexTreeVisitor  74
  GitIndex                 912
  IndexDiff                160
  IndexTreeWalker          241
  IndexTreeVisitor          97
  TreeIterator             203
  WorkDirCheckout          401
  ----------------------------
                          2088

Other classes like GitResourceDecorator simplify I think as a result of this new API being available. I have only started to port that over, but found I was getting side-tracked in porting the existing GitIndex users and I really want to focus on the merge code instead.

So overall we may see about a 2k line increase as a result of this API being used in the tree rather than GitIndex, but since it fits into the overall TreeWalk architecture its better suited for reuse, so we may see more than 2k saved over time.

Shawn O. Pearce (26):
  Force all source code to UTF-8 encoding by default
  Protect WorkingTreeIterator's name encoding from weird ByteBuffers
  Add Constants.encode as a utility for quick encoding in UTF-8
  Rely upon Constants.CHARSET over Constants.CHARACTER_ENCODING
  Allow AbstractTreeIterators to find out about StopWalkExceptions
  Implement a new .git/index (aka dircache) read interface
  Export the new DirCache API to Eclipse plugins using jgit
  Support locking (and unlocking) a .git/index through DirCache
  Support writing a .git/index through DirCache
  Support the 'TREE' extension in .git/index through DirCache
  Support using a DirCache within a TreeWalk
  Support recreating a .git/index through DirCache
  Support iterating and building a DirCache at the same time
  Support creating a new DirCacheEntry for an arbitrary path
  Support a simplified model of editing index entries
  Support recursively getting all entries under a subtree path
  Support copying meta fields from one DirCacheEntry to another
  Add JUnit tests for new DirCache API
  Add JUnit tests for DirCache compatibility with C Git
  Allow the new DirCacheIterator in command line arguments
  Add debugging commands to interact with the new DirCache code
  Add a basic command line implementation of rm
  Rewrite GitMoveDeleteHook to use DirCacheBuilder
  Teach GitMoveDeleteHook how to move a folder recursively
  Rewrite UntrackOperation to use DirCacheBuilder
  Rewrite AssumeUnchangedOperation to use DirCache
 .../.settings/org.eclipse.core.resources.prefs     |    3 +
 .../.settings/org.eclipse.core.resources.prefs     |    3 +
 .../.settings/org.eclipse.core.resources.prefs     |    3 +
 .../.settings/org.eclipse.core.resources.prefs     |    3 +
 .../org/spearce/egit/core/GitMoveDeleteHook.java   |  156 ++-
 .../egit/core/op/AssumeUnchangedOperation.java     |  133 +-
 .../org/spearce/egit/core/op/UntrackOperation.java |   98 +-
 .../.settings/org.eclipse.core.resources.prefs     |    3 +
 .../.settings/org.eclipse.core.resources.prefs     |    3 +
 .../.settings/org.eclipse.core.resources.prefs     |    3 +
 .../services/org.spearce.jgit.pgm.TextBuiltin      |    6 +
 .../src/org/spearce/jgit/pgm/Rm.java               |   92 ++
 .../org/spearce/jgit/pgm/debug/MakeCacheTree.java  |   67 +
 .../org/spearce/jgit/pgm/debug/ReadDirCache.java   |   53 +
 .../org/spearce/jgit/pgm/debug/ShowCacheTree.java  |   69 +
 .../org/spearce/jgit/pgm/debug/ShowDirCache.java   |   72 +
 .../org/spearce/jgit/pgm/debug/WriteDirCache.java  |   54 +
 .../jgit/pgm/opt/AbstractTreeIteratorHandler.java  |   13 +
 .../.settings/org.eclipse.core.resources.prefs     |    3 +
 .../spearce/jgit/dircache/DirCacheBasicTest.java   |  185 +++
 .../jgit/dircache/DirCacheBuilderIteratorTest.java |   91 ++
 .../spearce/jgit/dircache/DirCacheBuilderTest.java |  253 ++++
 .../dircache/DirCacheCGitCompatabilityTest.java    |  204 +++
 .../spearce/jgit/dircache/DirCacheFindTest.java    |   86 ++
 .../jgit/dircache/DirCacheIteratorTest.java        |  273 ++++
 .../spearce/jgit/dircache/DirCacheTreeTest.java    |  150 ++
 .../tst/org/spearce/jgit/dircache/gitgit.index     |  Bin 0 -> 134799 bytes
 .../tst/org/spearce/jgit/dircache/gitgit.lsfiles   | 1437 ++++++++++++++++++++
 .../tst/org/spearce/jgit/dircache/gitgit.lstree    |  331 +++++
 .../spearce/jgit/lib/ConstantsEncodingTest.java    |   89 ++
 .../.settings/org.eclipse.core.resources.prefs     |    3 +
 org.spearce.jgit/META-INF/MANIFEST.MF              |    3 +-
 .../spearce/jgit/dircache/BaseDirCacheEditor.java  |  194 +++
 .../src/org/spearce/jgit/dircache/DirCache.java    |  681 +++++++++
 .../jgit/dircache/DirCacheBuildIterator.java       |  124 ++
 .../org/spearce/jgit/dircache/DirCacheBuilder.java |  175 +++
 .../org/spearce/jgit/dircache/DirCacheEditor.java  |  265 ++++
 .../org/spearce/jgit/dircache/DirCacheEntry.java   |  385 ++++++
 .../spearce/jgit/dircache/DirCacheIterator.java    |  205 +++
 .../org/spearce/jgit/dircache/DirCacheTree.java    |  453 ++++++
 .../src/org/spearce/jgit/lib/Constants.java        |   25 +
 .../src/org/spearce/jgit/lib/ObjectWriter.java     |    2 +-
 .../src/org/spearce/jgit/lib/RefDatabase.java      |    9 +-
 .../src/org/spearce/jgit/lib/RepositoryConfig.java |    4 +-
 .../org/spearce/jgit/transport/PacketLineIn.java   |    3 +-
 .../jgit/transport/SideBandInputStream.java        |    3 +-
 .../spearce/jgit/transport/TransportBundle.java    |    3 +-
 .../spearce/jgit/transport/WalkPushConnection.java |    4 +-
 .../jgit/transport/WalkRemoteObjectDatabase.java   |    5 +-
 .../jgit/treewalk/AbstractTreeIterator.java        |   11 +
 .../spearce/jgit/treewalk/EmptyTreeIterator.java   |    6 +
 .../src/org/spearce/jgit/treewalk/TreeWalk.java    |   12 +-
 .../spearce/jgit/treewalk/WorkingTreeIterator.java |    2 +-
 .../spearce/jgit/treewalk/filter/PathFilter.java   |   10 +-
 54 files changed, 6333 insertions(+), 190 deletions(-)
 create mode 100644 org.spearce.egit-feature/.settings/org.eclipse.core.resources.prefs
 create mode 100644 org.spearce.egit-updatesite/.settings/org.eclipse.core.resources.prefs
 create mode 100644 org.spearce.egit.core.test/.settings/org.eclipse.core.resources.prefs
 create mode 100644 org.spearce.egit.core/.settings/org.eclipse.core.resources.prefs
 create mode 100644 org.spearce.egit.ui/.settings/org.eclipse.core.resources.prefs
 create mode 100644 org.spearce.egit/.settings/org.eclipse.core.resources.prefs
 create mode 100644 org.spearce.jgit.pgm/.settings/org.eclipse.core.resources.prefs
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Rm.java
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/debug/MakeCacheTree.java
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/debug/ReadDirCache.java
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/debug/ShowCacheTree.java
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/debug/ShowDirCache.java
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/debug/WriteDirCache.java
 create mode 100644 org.spearce.jgit.test/.settings/org.eclipse.core.resources.prefs
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/DirCacheBasicTest.java
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/DirCacheBuilderIteratorTest.java
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/DirCacheBuilderTest.java
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/DirCacheCGitCompatabilityTest.java
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/DirCacheFindTest.java
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/DirCacheIteratorTest.java
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/DirCacheTreeTest.java
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/gitgit.index
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/gitgit.lsfiles
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/dircache/gitgit.lstree
 create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/lib/ConstantsEncodingTest.java
 create mode 100644 org.spearce.jgit/.settings/org.eclipse.core.resources.prefs
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/dircache/BaseDirCacheEditor.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/dircache/DirCache.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/dircache/DirCacheBuildIterator.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/dircache/DirCacheBuilder.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/dircache/DirCacheEditor.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/dircache/DirCacheEntry.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/dircache/DirCacheIterator.java
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/dircache/DirCacheTree.java
Next: Shawn O. Pearce
Message 1 of 26 in “New DirCache API”
  1. 00/26 New DirCache APIShawn O. Pearce, Aug 12, 2008
  2. 01/26 Force all source code to UTF-8 encoding by defaultShawn O. Pearce, Aug 12, 2008
  3. 02/26 Protect WorkingTreeIterator's name encoding from weird ByteBuffersShawn O. Pearce, Aug 12, 2008
  4. 03/26 Add Constants.encode as a utility for quick encoding in UTF-8Shawn O. Pearce, Aug 12, 2008
  5. 04/26 Rely upon Constants.CHARSET over Constants.CHARACTER_ENCODINGShawn O. Pearce, Aug 12, 2008
  6. 05/26 Allow AbstractTreeIterators to find out about StopWalkExceptionsShawn O. Pearce, Aug 12, 2008
  7. 06/26 Implement a new .git/index (aka dircache) read interfaceShawn O. Pearce, Aug 12, 2008
  8. 07/26 Export the new DirCache API to Eclipse plugins using jgitShawn O. Pearce, Aug 12, 2008
  9. 08/26 Support locking (and unlocking) a .git/index through DirCacheShawn O. Pearce, Aug 12, 2008
  10. 09/26 Support writing a .git/index through DirCacheShawn O. Pearce, Aug 12, 2008
  11. 10/26 Support the 'TREE' extension in .git/index through DirCacheShawn O. Pearce, Aug 12, 2008
  12. 11/26 Support using a DirCache within a TreeWalkShawn O. Pearce, Aug 12, 2008
  13. 12/26 Support recreating a .git/index through DirCacheShawn O. Pearce, Aug 12, 2008
  14. 13/26 Support iterating and building a DirCache at the same timeShawn O. Pearce, Aug 12, 2008
  15. 14/26 Support creating a new DirCacheEntry for an arbitrary pathShawn O. Pearce, Aug 12, 2008
  16. 15/26 Support a simplified model of editing index entriesShawn O. Pearce, Aug 12, 2008
  17. 16/26 Support recursively getting all entries under a subtree pathShawn O. Pearce, Aug 12, 2008
  18. 17/26 Support copying meta fields from one DirCacheEntry to anotherShawn O. Pearce, Aug 12, 2008
  19. 18/26 Add JUnit tests for new DirCache APIShawn O. Pearce, Aug 12, 2008
  20. 20/26 Allow the new DirCacheIterator in command line argumentsShawn O. Pearce, Aug 12, 2008
  21. 21/26 Add debugging commands to interact with the new DirCache codeShawn O. Pearce, Aug 12, 2008
  22. 22/26 Add a basic command line implementation of rmShawn O. Pearce, Aug 12, 2008
  23. 23/26 Rewrite GitMoveDeleteHook to use DirCacheBuilderShawn O. Pearce, Aug 12, 2008
  24. 24/26 Teach GitMoveDeleteHook how to move a folder recursivelyShawn O. Pearce, Aug 12, 2008
  25. 25/26 Rewrite UntrackOperation to use DirCacheBuilderShawn O. Pearce, Aug 12, 2008
  26. 26/26 Rewrite AssumeUnchangedOperation to use DirCacheShawn O. Pearce, Aug 12, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.