Re: [RFC] super indexes to span multiple packfiles

Object's are not accessed in random order with git. Once an object reference hits a pack file it is very likely that following references will hit the same pack file. That's because you always find object SHA's by following the chains.

So first place to look for an object is the same place the previous object was found. If it isn't there order the search of the pack files by creation data (just a heuristic). Make this list a circle and start the search in the pack where the previous object was found. This can all be done with the existing indexes.

I haven't been reading all of the messages on this subject, but is this strategy enough to eliminate the need for a super index?

If you still need a super index, note that it may be good enough for it to only contain the SHA's for objects that are externally referenced. This index would be small and simply point to the correct pack file index to find the object in. You could add a list of dangling links to each packfile index to assist with building this super index.

My work with databases leads me to believe that figuring out how to pack everything into a smaller space always beats efforts put into incrementally improving the indexing scheme. Packing into a smaller space reduces the total IO needs and that's always a winner.