[PATCH 0/9] reftable: optimize write performance

Hi,

this is my first patch series taking an actual look at write performance for the reftable backend. This series addresses two major pain points:

  - Duplicate directory/file conflict checks when writing refs.

  - Allocation churn when compressing log blocks.

Overall though I found that there is not much of a point to investigate write performance in the reftable library itself, at least not right now. This is mostly because the write performance is heavily dominated by random ref reads. And while past patch series have optimized scanning through refs linearly, seeking random refs isn't well-optimized yet. So once all in-flight series relating to reftable performance have landed I will focus on random ref reads next.

For the bigger picture, the following benchmarks show perfomance compared to the "files" backend after applying this patch series.

Writing many refs in a single transaction:

  Benchmark 1: update-ref: create many refs (refformat = files, refcount = 100000)
    Time (mean ± σ):     10.085 s ±  0.057 s    [User: 1.876 s, System: 8.161 s]
    Range (min … max):   10.013 s … 10.202 s    10 runs

  Benchmark 2: update-ref: create many refs (refformat = reftable, refcount = 100000)
    Time (mean ± σ):      2.768 s ±  0.018 s    [User: 1.381 s, System: 1.383 s]
    Range (min … max):    2.745 s …  2.804 s    10 runs

  Summary
    update-ref: create many refs (refformat = reftable, refcount = 100000) ran
      3.64 ± 0.03 times faster than update-ref: create many refs (refformat = files, refcount = 100000)

And for writing many refs sequentially in separate transactions:

  Benchmark 1: update-ref: create refs sequentially (refformat = files, refcount = 10000)
    Time (mean ± σ):     40.286 s ±  0.086 s    [User: 22.241 s, System: 17.912 s]
    Range (min … max):   40.166 s … 40.410 s    10 runs

  Benchmark 2: update-ref: create refs sequentially (refformat = reftable, refcount = 10000)
    Time (mean ± σ):     44.046 s ±  0.137 s    [User: 23.790 s, System: 20.146 s]
    Range (min … max):   43.813 s … 44.301 s    10 runs

  Summary
    update-ref: create refs sequentially (refformat = files, refcount = 10000) ran
      1.09 ± 0.00 times faster than update-ref: create refs sequentially (refformat = reftable, refcount = 10000)

This is to the best of my knowledge last area where the "files" backend outperforms the "reftable" backend. This is partially also due to the fact that writes perform auto-compaction with the "reftable" backend.

Patrick

Patrick Steinhardt (9):
  refs/reftable: fix D/F conflict error message on ref copy
  refs/reftable: perform explicit D/F check when writing symrefs
  refs/reftable: skip duplicate name checks
  refs/reftable: don't recompute committer ident
  reftable/writer: refactorings for `writer_add_record()`
  reftable/writer: refactorings for `writer_flush_nonempty_block()`
  reftable/block: reuse zstream when writing log blocks
  reftable/block: reuse compressed array
  reftable/writer: reset `last_key` instead of releasing it

 refs/reftable-backend.c    |  80 ++++++++++++++++++-------
 reftable/block.c           |  83 ++++++++++++++++----------
 reftable/block.h           |   4 ++
 reftable/writer.c          | 119 ++++++++++++++++++++++++-------------
 t/t0610-reftable-basics.sh |  35 ++++++++++-
 5 files changed, 227 insertions(+), 94 deletions(-)