git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 4/5] reftable/writer: fix writing multi-level indices

From
jltobler <jltobler@gmail.com>
Date
Feb 5, 2024, 23:56 UTC
Message-ID
<g5wshgzfv7x6om5zglsiv4bzsmhwcihwrqkmq4ebppiljqbreu@ml5eyr6zhkgu>
In-Reply-To
<89a88cf83eeb50542d3878c5c6e56e46f2bc3e73.1706773842.git.ps@pks.im>
On 24/02/01 08:52AM, Patrick Steinhardt wrote:
Show 22 quoted lines
> When finishing a section we will potentially write an index that makes
> it more efficient to look up relevant blocks. The index records written
> will encode, for each block of the indexed section, what the offset of
> that block is as well as the last key of that block. Thus, the reader
> would iterate through the index records to find the first key larger or
> equal to the wanted key and then use the encoded offset to look up the
> desired block.
> 
> When there are a lot of blocks to index though we may end up writing
> multiple index blocks, too. To not require a linear search across all
> index blocks we instead end up writing a multi-level index. Instead of
> referring to the block we are after, an index record may point to
> another index block. The reader will then access the highest-level index
> and follow down the chain of index blocks until it hits the sought-after
> block.
> 
> It has been observed though that it is impossible to seek ref records of
> the last ref block when using a multi-level index. While the multi-level
> index exists and looks fine for most of the part, the highest-level
> index was missing an index record pointing to the last block of the next
> index. Thus, every additional level made more refs become unseekable at
> the end of the ref section.

Just to clarify, is only the highest-level index not recording the last block when multi-level indexes are being used? Or are the indexes at all levels leaving the last block unreachable?

Show 7 quoted lines
> 
> The root cause is that we are not flushing the last block of the current
> level once done writing the level. Consequently, it wasn't recorded in
> the blocks that need to be indexed by the next-higher level and thus we
> forgot about it.
> 
> Fix this bug by flushing blocks after we have written all index records.
 -Justin
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 17 of 21 in “reftable: fix writing multi-level indices”
  1. 0/5 reftable: fix writing multi-level indicesPatrick Steinhardt, Jan 26, 2024
  2. 1/5 reftable/reader: be more careful about errors in indexed seeksPatrick Steinhardt, Jan 26, 2024
  3. 2/5 reftable/writer: use correct type to iterate through index entriesPatrick Steinhardt, Jan 26, 2024
  4. 3/5 reftable/writer: simplify writing index recordsPatrick Steinhardt, Jan 26, 2024
  5. Toon ClaesJan 31, 2024
  6. Patrick SteinhardtFeb 1, 2024
  7. Kristoffer HaugsbakkJan 31, 2024
  8. Patrick SteinhardtFeb 1, 2024
  9. 4/5 reftable/writer: fix writing multi-level indicesPatrick Steinhardt, Jan 26, 2024
  10. 5/5 reftable: document reading and writing indicesPatrick Steinhardt, Jan 26, 2024
  11. Junio C HamanoJan 26, 2024
  12. 0/5 reftable: fix writing multi-level indicesPatrick Steinhardt, Feb 1, 2024
  13. 1/5 reftable/reader: be more careful about errors in indexed seeksPatrick Steinhardt, Feb 1, 2024
  14. 2/5 reftable/writer: use correct type to iterate through index entriesPatrick Steinhardt, Feb 1, 2024
  15. 3/5 reftable/writer: simplify writing index recordsPatrick Steinhardt, Feb 1, 2024
  16. 4/5 reftable/writer: fix writing multi-level indicesPatrick Steinhardt, Feb 1, 2024
  17. jltoblerFeb 5, 2024
  18. Patrick SteinhardtFeb 6, 2024
  19. 5/5 reftable: document reading and writing indicesPatrick Steinhardt, Feb 1, 2024
  20. jltoblerFeb 6, 2024
  21. Patrick SteinhardtFeb 6, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.