Re: [PATCH v2 4/5] reftable/writer: fix writing multi-level indices
- From
jltobler <jltobler@gmail.com>
- Date
- Feb 5, 2024, 23:56 UTC
- Message-ID
- <g5wshgzfv7x6om5zglsiv4bzsmhwcihwrqkmq4ebppiljqbreu@ml5eyr6zhkgu>
- In-Reply-To
- <89a88cf83eeb50542d3878c5c6e56e46f2bc3e73.1706773842.git.ps@pks.im>
On 24/02/01 08:52AM, Patrick Steinhardt wrote:
Show 22 quoted lines
> When finishing a section we will potentially write an index that makes > it more efficient to look up relevant blocks. The index records written > will encode, for each block of the indexed section, what the offset of > that block is as well as the last key of that block. Thus, the reader > would iterate through the index records to find the first key larger or > equal to the wanted key and then use the encoded offset to look up the > desired block. > > When there are a lot of blocks to index though we may end up writing > multiple index blocks, too. To not require a linear search across all > index blocks we instead end up writing a multi-level index. Instead of > referring to the block we are after, an index record may point to > another index block. The reader will then access the highest-level index > and follow down the chain of index blocks until it hits the sought-after > block. > > It has been observed though that it is impossible to seek ref records of > the last ref block when using a multi-level index. While the multi-level > index exists and looks fine for most of the part, the highest-level > index was missing an index record pointing to the last block of the next > index. Thus, every additional level made more refs become unseekable at > the end of the ref section.
Just to clarify, is only the highest-level index not recording the last block when multi-level indexes are being used? Or are the indexes at all levels leaving the last block unreachable?
Show 7 quoted lines
> > The root cause is that we are not flushing the last block of the current > level once done writing the level. Consequently, it wasn't recorded in > the blocks that need to be indexed by the next-higher level and thus we > forgot about it. > > Fix this bug by flushing blocks after we have written all index records.
-Justin