Re: [PATCH 4/5] meson: wire up benchmarks

From: Toon Claes <toon@iotcl.com>
Date: Apr 10, 2025, 11:44 UTC
Message-ID: <87r020dyvc.fsf@iotcl.com>
In-Reply-To: <20250331-pks-meson-benchmarks-v1-4-b2ace85616a3@pks.im>

Patrick Steinhardt <ps@pks.im> writes:

Show 20 quoted lines

> Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> up our tests. The only difference is that benchmarks get wired up via
> the `benchmark()` option instead of via `test()`, which gives them a bit
> of special treatment:
>
>   - Benchmarks never run in parallel.
>
>   - Benchmarks aren't run by default when tests are executed.
>
>   - Meson does not inject the `MALLOC_PERTURB` environment variable.
>
> Using benchmarks is quite simple:
>
>     ```
>     $ meson setup build
>     # Run all benchmarks.
>     $ meson test -C build --benchmark
>     # Run a specific benchmark.
>     $ meson test -C build --benchmark p0000-*
>     ```

I really appreciate the efforts, and I'd love to run the perf tests with Meson, but at the moment I don't know how these changes are useful because this doesn't print anything relevant:

    $ meson test -C build --benchmark 'p0005-*'
    ninja: Entering directory `/home/toon/devel/git/build'
    [1/28] Generating GIT-VERSION-FILE with a custom command (wrapped by meson to set env)
    1/1 p0005-status        OK              7.39s

    Ok:                 1
    Expected Fail:      0
    Fail:               0
    Unexpected Pass:    0
    Skipped:            0
    Timeout:            0

    Full log written to /home/toon/devel/git/build/meson-logs/testlog.txt

Show 9 quoted lines

> Other than that the usual command line arguments accepted when running
> tests are also accepted when running benchmarks.
>
> Note that the benchmarking target is somewhat limited because it will
> only run benchmarks for the current build. Other usecases, like running
> benchmarks against multiple different versions of Git, are not currently
> supported. Users should continue to use "t/perf/run" for those usecases.
> The script should get extended at one point in time to support Meson,
> but this is outside of the scope of this series.

Yeah, this is unfortunate, but totally understandable. I've been digging in `t/perf/run` and `f/perf/aggregate.perl` and it doesn't look easy to adapt to Meson.

But I was wondering, instead of trying to fully integrate Meson into those scripts, could we modify the scripts so they work with binaries built by Meson? I mean, if we could run `cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would simply run the benchmarks in those directories (without trying to check out code and build the sources). I think this would help a lot already.

-- 
Toon