git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 09/10] reftable/block: reuse `zstream` state on inflation

From
Karthik Nayak <karthik.188@gmail.com>
Date
Apr 10, 2024, 10:15 UTC
Message-ID
<CAOLa=ZTnnmBvwXMbHqJ+58kPGe4jwLhu_GaEfkAXyfVa5imeYA@mail.gmail.com>
In-Reply-To
<587b5601c0c51a8f041258a27e693a2590468eed.1712578376.git.ps@pks.im>
Patrick Steinhardt <ps@pks.im> writes:
Show 27 quoted lines
> When calling `inflateInit()` and `inflate()`, the zlib library will
> allocate several data structures for the underlying `zstream` to keep
> track of various information. Thus, when inflating repeatedly, it is
> possible to optimize memory allocation patterns by reusing the `zstream`
> and then calling `inflateReset()` on it to prepare it for the next chunk
> of data to inflate.
>
> This is exactly what the reftable code is doing: when iterating through
> reflogs we need to potentially inflate many log blocks, but we discard
> the `zstream` every single time. Instead, as we reuse the `block_reader`
> for each of the blocks anyway, we can initialize the `zstream` once and
> then reuse it for subsequent inflations.
>
> Refactor the code to do so, which leads to a significant reduction in
> the number of allocations. The following measurements were done when
> iterating through 1 million reflog entries. Before:
>
>   HEAP SUMMARY:
>       in use at exit: 13,473 bytes in 122 blocks
>     total heap usage: 23,028 allocs, 22,906 frees, 162,813,552 bytes allocated
>
> After:
>
>   HEAP SUMMARY:
>       in use at exit: 13,473 bytes in 122 blocks
>     total heap usage: 302 allocs, 180 frees, 88,352 bytes allocated
>

Really nice how just reusing the data structure has such a significant impact.

Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 26 of 31 in “reftable: optimize table and block iterators”
  1. 0/9 reftable: optimize table and block iteratorsPatrick Steinhardt, Mar 27, 2024
  2. 1/9 reftable/block: rename `block_reader_start()`Patrick Steinhardt, Mar 27, 2024
  3. 2/9 reftable/block: merge `block_iter_seek()` and `block_reader_seek()`Patrick Steinhardt, Mar 27, 2024
  4. 3/9 reftable/block: better grouping of functionsPatrick Steinhardt, Mar 27, 2024
  5. 4/9 reftable/block: introduce `block_reader_release()`Patrick Steinhardt, Mar 27, 2024
  6. Karthik NayakApr 3, 2024
  7. Patrick SteinhardtApr 8, 2024
  8. 5/9 reftable/block: move ownership of block reader into `struct table_iter`Patrick Steinhardt, Mar 27, 2024
  9. Justin ToblerApr 3, 2024
  10. Patrick SteinhardtApr 3, 2024
  11. 6/9 reftable/reader: iterate to next block in placePatrick Steinhardt, Mar 27, 2024
  12. 7/9 reftable/block: reuse uncompressed blocksPatrick Steinhardt, Mar 27, 2024
  13. 8/9 reftable/block: open-code call to `uncompress2()`Patrick Steinhardt, Mar 27, 2024
  14. 9/9 reftable/block: reuse `zstream` state on inflationPatrick Steinhardt, Mar 27, 2024
  15. Karthik NayakApr 3, 2024
  16. 00/10 reftable: optimize table and block iteratorsPatrick Steinhardt, Apr 8, 2024
  17. 01/10 reftable/block: rename `block_reader_start()`Patrick Steinhardt, Apr 8, 2024
  18. 02/10 reftable/block: merge `block_iter_seek()` and `block_reader_seek()`Patrick Steinhardt, Apr 8, 2024
  19. 03/10 reftable/block: better grouping of functionsPatrick Steinhardt, Apr 8, 2024
  20. 04/10 reftable/block: introduce `block_reader_release()`Patrick Steinhardt, Apr 8, 2024
  21. 05/10 reftable/block: move ownership of block reader into `struct table_iter`Patrick Steinhardt, Apr 8, 2024
  22. 06/10 reftable/reader: iterate to next block in placePatrick Steinhardt, Apr 8, 2024
  23. 07/10 reftable/block: reuse uncompressed blocksPatrick Steinhardt, Apr 8, 2024
  24. 08/10 reftable/block: open-code call to `uncompress2()`Patrick Steinhardt, Apr 8, 2024
  25. 09/10 reftable/block: reuse `zstream` state on inflationPatrick Steinhardt, Apr 8, 2024
  26. Karthik NayakApr 10, 2024
  27. 10/10 reftable/block: avoid copying block iterators on seekPatrick Steinhardt, Apr 8, 2024
  28. Justin ToblerApr 9, 2024
  29. Patrick SteinhardtApr 9, 2024
  30. Justin ToblerApr 9, 2024
  31. Karthik NayakApr 10, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.