Re: [PATCH 1/3] Lazily open pack index files on demand

Dana How <danahow@gmail.com> wrote:

Yes; in fact this has been raised on the list before. The general idea was to create some sort of "super index" that had a list of all objects and which packfile they could be found in. This way the running process doesn't have to search multiple indexes, and the process doesn't have to be responsible for the merging itself.

See the thing is, if you read all of every .idx file on a simple `git-log` operation you've already lost. The number of trees and blobs tends to far outweigh the number of commits and they really outweigh the number of commits the average user looks at in a `git-log` session before they abort their pager. So sorting all of the available .idx files before we produce even the first commit is a horrible thing to do.

But the problem with a super index is repacking. Every time the user repacks their recent loose objects (or recently fetched packs) we are folding some packfiles together, but may be leaving others alone. The super index would need to account for the packfiles we aren't looking at or repacking. It gets complicated fast.

There's also the problem of alternate ODBs; do we fold the indexes of our alternates into our own super index? Or does each ODB get its own super index and we still have to load multiple super index files?

In pack v4 we're likely to move the SHA-1 table from the .idx file into the front of the .pack file. This makes the .idx file hold only the offsets and the CRC checkums of each object. If we start making a super index, we have to duplicate the SHA-1 table twice (once in the .pack, again in the super index).