Re: [PATCH v1 1/4] fastindex: speed up index load through parallelization

> Ben Peart <peartben@gmail.com> writes:
> 
>> I have no thoughts or plans for changes in the future of IEOT (which I
>> plan to rename ITOC).  At this point in time, I can't even imagine
>> what else we'd want as the index only contains cache entries, ...
> 
> Yeah, but the thing is that this is open source, and the imagination
> of the originator of the initial idea does not limit how the code
> and data structure evolves.
> 
> Back when I added the index extensions to the system, I didn't have
> perfect foresight, and I didn't have specific plans to add anything
> beyond "TREE" to optimize "write-tree" and "diff-index --cached".
> 
> In hindsight, I got one thing right and one thing wrong.
> 
> Even though I didn't have any plan to add a mandatory extension, I
> made the code to ignore optional ones and error out on mandatory
> ones if an index extension section is not understood.  It turns out
> that later (in fact much later---the "TREE" extension dates back to
> April 2006, while "link" came in June 2014) we could add the
> split-index mode without having to worry about older versions of Git
> doing random wrong things when they see this new extension, thanks
> to that design decision.  That went well.
> 
> On the other hand, I didn't think things through to realize that
> some operations may want to peek only the extensions without ever
> using the main table, that some other operations may want to read
> some extensions first before reading the main table, or more
> importantly, that these limitations would start mattering once Git
> becomes popular enough and starts housing a project tree with very
> many paths in the main table.
> 
> I really wish somebody had brought it up as a potential issue during
> the review---I would have defined the extended index format to have
> the simple extension at the end that points back to the tail end of
> the main table back then, and we wouldn't be having this back and
> forth now.  But I was just too happy and excited that I have found a
> room to squeeze extension sections into the index file format
> without breaking existing implementations of Git (which checked the
> trailer checksum matches to the result of hashing the whole thing,
> and read the recorded number of entries from the main table, without
> even noticing that there is a gap in between), and that excitement
> blinded me.
> 
>> I understand the risk but the list of offsets into the cache entries
>> is pretty simple as well. I prefer the simplicity of a single TOC
>> extension that gives us random access to the entire index rather than
>> having to piece one together using multiple extensions.  That model
>> has its own set of risks and tradeoffs.
> 
> I thought that you are not using the "where does the series of
> extensions begin" information in the first place, no?  That piece of
> information is useful independent of the usefulness of "index into
> the main table to list entries where the prefix-compression is
> reset".  So if anything, I'd prefer the simplicity of introducing
> that "where does the series of extensions begin" that does not say
> anything else, and build on other things like ITOC as mere users of
> the mechanism.
>