Continuous Benchmarking

Hi,

due to a couple performance regressions that we have hit over the last couple Git releases at GitLab, we have started to set up an effort to implement continuous benchmarking for the Git project. The intent is to have regular (daily) benchmarking runs against Git's `master` and `next` branches to be able to spot any performance regressions before they make it into the next release.

I have started with a relatively simple setup:

  - I have started collection benchmarks that I myself do regularly [1].
    These benchmarks are built on hyperfine and are thus not part of the
    Git repository itself.

  - GitLab CI runs on a nightly basis, executing a subset of these
    benchmarks [2].

  - Results are uploaded with a hyperfine adaptor to Bencher and are
    summarized in dashboards.

This at least gives us some visibility in severe performance outliers, whether these are improvements or regressions. Some statistics are applied on this data to automatically generate alerts when things are significantly changing.

The setup is of course not perfect. It's built on top of CI jobs, which are by their very nature not really performing consistent. The scripts are hosted outside of Git. And I'm the only one running this.

So I wonder whether there is a wider interest in the Git community to have this infrastructure part of the Git project itself. This may include steps like the following:

  - Extending our performance tests we have in "t/perf" to cover more
    benchmarks.

  - Writing an adaptor that is able to upload the data generated from
    our perf scripts to Bencher.

  - Setting up proper infrastructure to do the benchmarking. We may for
    now also continue to use GitLab CI, but as said they are quite noisy
    overall. Dedicated servers would help here.

  - Sending alerts to the Git mailing list.

I'm happy to hear your thoughts on this. Any ideas are welcome, including "we're not interested at all". In that case, we'd simply continue to maintain the setup ourselves at GitLab.

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/data-access/git/benchmarks [2]: https://gitlab.com/gitlab-org/data-access/git/benchmarks/-/blob/main/.gitlab-ci.yml?ref_type=heads [3]: https://bencher.dev/console/projects/git/plots