Re: [PATCH v2 09/10] reftable/block: reuse `zstream` state on inflation
- From
Karthik Nayak <karthik.188@gmail.com>
- Date
- Apr 10, 2024, 10:15 UTC
- Message-ID
- <CAOLa=ZTnnmBvwXMbHqJ+58kPGe4jwLhu_GaEfkAXyfVa5imeYA@mail.gmail.com>
- In-Reply-To
- <587b5601c0c51a8f041258a27e693a2590468eed.1712578376.git.ps@pks.im>
Patrick Steinhardt <ps@pks.im> writes:
Show 27 quoted lines
> When calling `inflateInit()` and `inflate()`, the zlib library will > allocate several data structures for the underlying `zstream` to keep > track of various information. Thus, when inflating repeatedly, it is > possible to optimize memory allocation patterns by reusing the `zstream` > and then calling `inflateReset()` on it to prepare it for the next chunk > of data to inflate. > > This is exactly what the reftable code is doing: when iterating through > reflogs we need to potentially inflate many log blocks, but we discard > the `zstream` every single time. Instead, as we reuse the `block_reader` > for each of the blocks anyway, we can initialize the `zstream` once and > then reuse it for subsequent inflations. > > Refactor the code to do so, which leads to a significant reduction in > the number of allocations. The following measurements were done when > iterating through 1 million reflog entries. Before: > > HEAP SUMMARY: > in use at exit: 13,473 bytes in 122 blocks > total heap usage: 23,028 allocs, 22,906 frees, 162,813,552 bytes allocated > > After: > > HEAP SUMMARY: > in use at exit: 13,473 bytes in 122 blocks > total heap usage: 302 allocs, 180 frees, 88,352 bytes allocated >
Really nice how just reusing the data structure has such a significant impact.