Re: [PATCH 00/10] refs: optimize ref format migrations

> Hi,
>
> I have recently learned that ref format migrations can take a
> significant amount of time in the order of minutes when migrating
> millions of refs. This is probably not entirely surprising: the initial
> focus for the logic to migrate ref backends was mostly focussed on
> getting the basic feature working, and I didn't yet invest any time into
> optimizing the code path at all. But I was still mildly surprised that
> the migration of a couple million refs was taking minutes to finish.
>
> This patch series thus optimizes how we migrate ref formats. This is
> mostly done by expanding upon the "initial transaction" semantics that
> we already use for git-clone(1). These semantics allow us to assume that
> the ref backend is completely empty and that there are no concurrent
> writers, and thus we are free to perform certain optimizations that
> wouldn't have otherwise been possible. On the one hand this allows us to
> drop needless collision checks. On the other hand, it also allows us to
> write regular refs directly into the "packed-refs" file when migrating
> from the "reftable" backend to the "files" backend.
>
> This leads to some significant speedups. Migrating 1 million refs from
> "files" to "reftable":
>
>     Benchmark 1: migrate files:reftable (refcount = 1000000, revision = origin/master)
>       Time (mean ± σ):      4.580 s ±  0.062 s    [User: 1.818 s, System: 2.746 s]
>       Range (min … max):    4.534 s …  4.743 s    10 runs
>
>     Benchmark 2: migrate files:reftable (refcount = 1000000, revision = pks-refs-optimize-migrations)
>       Time (mean ± σ):     767.7 ms ±   9.5 ms    [User: 629.2 ms, System: 126.1 ms]
>       Range (min … max):   755.8 ms … 786.9 ms    10 runs
>
>     Summary
>       migrate files:reftable (refcount = 1000000, revision = pks-refs-optimize-migrations) ran
>         5.97 ± 0.11 times faster than migrate files:reftable (refcount = 1000000, revision = origin/master)
>
> And migrating from "reftable" to "files:
>
>     Benchmark 1: migrate reftable:files (refcount = 1000000, revision = origin/master)
>       Time (mean ± σ):     35.409 s ±  0.302 s    [User: 5.061 s, System: 29.244 s]
>       Range (min … max):   35.055 s … 35.898 s    10 runs
>
>     Benchmark 2: migrate reftable:files (refcount = 1000000, revision = pks-refs-optimize-migrations)
>       Time (mean ± σ):     855.9 ms ±  61.5 ms    [User: 646.7 ms, System: 187.1 ms]
>       Range (min … max):   830.0 ms … 1030.3 ms    10 runs
>
>     Summary
>       migrate reftable:files (refcount = 1000000, revision = pks-refs-optimize-migrations) ran
>        41.37 ± 2.99 times faster than migrate reftable:files (refcount = 1000000, revision = origin/master)
>
> Thanks!
>
> Patrick
>