patch, 5 partsmeson: wire up support for benchmarks

Hi,

this small patch series implements support for running our benchmarks in "t/perf" via Meson. The series does not aim to replace "t/perf/run", which is more fully-featured and allows running benchmarks against multiple different trees. Instead, this series only allows running the benchmarks against the current tree. Users are thus expected to continue using "t/perf/run" for more advanced usecases.

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build        |  7 ++---
 meson_options.txt  |  6 +++++
 t/meson.build      | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/perf-lib.sh | 32 ++++++++++++++++++++---
 t/perf/run         |  4 +--
 5 files changed, 117 insertions(+), 9 deletions(-)

--- base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

Many of our benchmarks operate on a user-defined repository that we copy over before running the benchmarked logic. To keep unintentional side effects caused by on-disk state at bay we skip copying some files. This includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite easy to inadvertently carry over configuration that may significantly impact the performance measurements. But we cannot fully ignore the configuration either, as it may contain information about the repository format. This will cause failures when for example using a repository with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Our benchmarks use a couple of Perl scripts to compute results. These Perl scripts get executed directly, and as the shebang is hardcoded to "/usr/bin/perl" this will fail on any system where the Perl interpreter is located in a different path.

Our build infrastructure already lets users configure the location of Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file. This file is being sourced by "test-lib.sh", and consequently we already have the "PERL_PATH" variable available that contains its configured location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more esoteric systems like NixOS.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 ++--
 t/perf/run         | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

Patrick Steinhardt <ps@pks.im> writes:

I see in `t/perf/README` there's a mention of running `./aggregate.perl` directly? Shall we inform the user to run that through their Perl as well?

-    $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh
+    $ perl ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh

Or do we expect users to know what they are doing when they don't have Perl installed at /usr/bin/perl?

-- Toon

On Thu, Apr 10, 2025 at 01:43:44PM +0200, Toon Claes wrote:

Another solution could be to switch the shebang to `#!/usr/bin/env perl`. We also do this for "t/chainlint.pl".

Patrick

Patrick Steinhardt <ps@pks.im> writes:

Good.

When you do not have perl installed anywhere, how does this fail? I think you would get

    $ ./aggregate.perl ...
    /usr/bin/env: 'perl': No such file or directory

and compared to that, 

    $ perl ./aggregate.perl ...
    bash: perl: command not found

I think it makes it slightly more obvious to those who lack perl on their $PATH what is going wrong to explicitly tell them to run "perl" like Toon's suggestion above (primarily because use of /usr/bin/env is not obvious to those who are told to run ./aggregate.perl script).

On Mon, Apr 14, 2025 at 12:20:40PM -0700, Junio C Hamano wrote:

The counterargument is that `#!/usr/bin/env perl` works on systems where Perl is in PATH, but in a non-standard location. So isn't it preferable to make things Just Work(TM) on such systems instead of giving slightly better error messages on systems where it wouldn't work anyway because Perl is not available? I certainly think so, but if you or others feel strongly I'll adapt.

Patrick

The "perf-lib.sh" script is sourced by all of our benchmarking suites to make available common infrastructure. The script assumes that build and source directory are the same, which works for our Makefile. But the assumption breaks with both CMake and Meson, where the build directory can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. This prepares us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

Patrick Steinhardt <ps@pks.im> writes:

I'm not sure if it's related to this commit, but this patch series has broken something:

    $ make && cd t/perf && ./run . master p0005-status.sh

    === Running 1 tests in this tree ===
    ok 1 - setup repo
    perf 2 - read-tree status br_ballast (4629): 1 ok
    # passed all 2 test(s)
    1..2
    === Unpacking 485f5f863615e670fd97ae40af744e14072cfe18 in build/485f5f863615e670fd97ae40af744e14072cfe18 ===
    === Building 485f5f863615e670fd97ae40af744e14072cfe18 (master) ===
    GIT_VERSION=2.49.GIT
        * new build flags
        CC daemon.o
        * new link flags
        CC common-main.o
        CC abspath.o
        CC add-interactive.o
        CC add-patch.o
        [snip]
        CC t/unit-tests/unit-test.o
        CC t/unit-tests/lib-oid.o
        LINK t/unit-tests/bin/unit-tests
        GEN gitweb/gitweb.cgi
        GEN gitweb/static/gitweb.js
    === Running 1 tests in /home/toon/devel/git/t/perf/build/485f5f863615e670fd97ae40af744e14072cfe18/bin-wrappers ===
    ok 1 - setup repo
    perf 2 - read-tree status br_ballast (4629): 1 ok
    # passed all 2 test(s)
    1..2
    cannot open test-results/p0005-status.subtests: No such file or directory at ./aggregate.perl line 159.

-- Toon

Toon Claes <toon@iotcl.com> writes:

Euhm, it seems I no longer can reproduce this issue. So you can ignore this message. Sorry for the noice.

-- Toon

On Mon, Apr 14, 2025 at 08:28:53AM +0200, Toon Claes wrote:

Okay, great. I wasn't able to reproduce this locally, so thanks for clarifying!

Patrick

Wire up benchmarks in Meson. The setup is mostly the same as how we wire up our tests. The only difference is that benchmarks get wired up via the `benchmark()` option instead of via `test()`, which gives them a bit of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will only run benchmarks for the current build. Other usecases, like running benchmarks against multiple different versions of Git, are not currently supported. Users should continue to use "t/perf/run" for those usecases. The script should get extended at one point in time to support Meson, but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build   |  1 +
 t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 78 insertions(+)

Patrick Steinhardt <ps@pks.im> writes:

I really appreciate the efforts, and I'd love to run the perf tests with Meson, but at the moment I don't know how these changes are useful because this doesn't print anything relevant:

    $ meson test -C build --benchmark 'p0005-*'
    ninja: Entering directory `/home/toon/devel/git/build'
    [1/28] Generating GIT-VERSION-FILE with a custom command (wrapped by meson to set env)
    1/1 p0005-status        OK              7.39s

    Ok:                 1
    Expected Fail:      0
    Fail:               0
    Unexpected Pass:    0
    Skipped:            0
    Timeout:            0

    Full log written to /home/toon/devel/git/build/meson-logs/testlog.txt

Yeah, this is unfortunate, but totally understandable. I've been digging in `t/perf/run` and `f/perf/aggregate.perl` and it doesn't look easy to adapt to Meson.

But I was wondering, instead of trying to fully integrate Meson into those scripts, could we modify the scripts so they work with binaries built by Meson? I mean, if we could run `cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would simply run the benchmarks in those directories (without trying to check out code and build the sources). I think this would help a lot already.

On Thu, Apr 10, 2025 at 01:44:55PM +0200, Toon Claes wrote:

You can execute with `meson test -i`, which will cause Meson to print the benchmark's output.

This is another thing we could (and eventually should) do, but it's orthogonal to this patch series from my point of view.

Patrick

Patrick Steinhardt <ps@pks.im> writes:

(annoyed grunt) I didn't think about using `-i` here, probably because it wasn't used in the example in the commit message.

That's true. This series already enables me to use benchmarks with Meson and is definetely a decent step in the right direction. Let's get this series merged and we can work on better comparative benchmarks later.

I just reviewed v2 and I approve on this series.

Wire up a couple of benchmarking options that we end up writing into our "GIT-BUILD-OPTIONS" file. These options allow users to control how exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are exposed as a build option. Those options are used by "t/perf/run", which is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

Hi,

this small patch series implements support for running our benchmarks in "t/perf" via Meson. The series does not aim to replace "t/perf/run", which is more fully-featured and allows running benchmarks against multiple different trees. Instead, this series only allows running the benchmarks against the current tree. Users are thus expected to continue using "t/perf/run" for more advanced usecases.

Changes in v2:
  - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
  - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build           |  7 +++--
 meson_options.txt     |  6 ++++
 t/meson.build         | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/aggregate.perl |  2 +-
 t/perf/perf-lib.sh    | 32 ++++++++++++++++++---
 t/perf/run            |  4 +--
 6 files changed, 118 insertions(+), 10 deletions(-)

Range-diff versus v1:

1:  fd734280c1d = 1:  844e8bdf81d t/perf: fix benchmarks with alternate repo formats
2:  89a011eec63 ! 2:  d950c828933 t/perf: use configured PERL_PATH
    @@ Commit message
         location.
     
         Use "PERL_PATH" to execute Perl scripts, which makes them work on more
    -    esoteric systems like NixOS.
    +    esoteric systems like NixOS. Furthermore, adapt the shebang to use
    +    env(1) to execute Perl so that users who have Perl in PATH, but in a
    +    non-standard location can execute the script directly.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    + ## t/perf/aggregate.perl ##
    +@@
    +-#!/usr/bin/perl
    ++#!/usr/bin/env perl
    + 
    + use lib '../../perl/build/lib';
    + use strict;
    +
      ## t/perf/perf-lib.sh ##
     @@ t/perf/perf-lib.sh: test_perf_ () {
      	else
3:  efecbd86cd6 = 3:  dc1adf62a86 t/perf: fix benchmarks with out-of-tree builds
4:  feeaaae3ec1 = 4:  8528a775395 meson: wire up benchmarks
5:  fc594fc3dc5 = 5:  c8a4d0b354e meson: wire up benchmarking options

--- base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

Many of our benchmarks operate on a user-defined repository that we copy over before running the benchmarked logic. To keep unintentional side effects caused by on-disk state at bay we skip copying some files. This includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite easy to inadvertently carry over configuration that may significantly impact the performance measurements. But we cannot fully ignore the configuration either, as it may contain information about the repository format. This will cause failures when for example using a repository with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Our benchmarks use a couple of Perl scripts to compute results. These Perl scripts get executed directly, and as the shebang is hardcoded to "/usr/bin/perl" this will fail on any system where the Perl interpreter is located in a different path.

Our build infrastructure already lets users configure the location of Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file. This file is being sourced by "test-lib.sh", and consequently we already have the "PERL_PATH" variable available that contains its configured location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more esoteric systems like NixOS. Furthermore, adapt the shebang to use env(1) to execute Perl so that users who have Perl in PATH, but in a non-standard location can execute the script directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/aggregate.perl | 2 +-
 t/perf/perf-lib.sh    | 4 ++--
 t/perf/run            | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

The "perf-lib.sh" script is sourced by all of our benchmarking suites to make available common infrastructure. The script assumes that build and source directory are the same, which works for our Makefile. But the assumption breaks with both CMake and Meson, where the build directory can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. This prepares us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

On Mon, Apr 14, 2025 at 8:51 AM Patrick Steinhardt <ps@pks.im> wrote:

Right now on 'master' there is:

GIT_BUILD_DIR="${GIT_BUILD_DIR:-${TEST_DIRECTORY%/t}}"
if test "$TEST_DIRECTORY" = "$GIT_BUILD_DIR"
then
       echo "PANIC: Running in a $TEST_DIRECTORY that doesn't end in '/t'?" >&2
       exit 1
fi

so it's not exactly the same thing, even if it still probably works well.

Future readers might wonder if this discrepancy results from changes that were made to only one of the files or if we really wanted to get rid of the "/t" check here. In case we do want to get rid of the "/t" check, I think it might be worth saying it clearly in the comment.

Maybe something like the following could help debug this:

       echo >&2 "error: GIT-BUILD-OPTIONS file missing from '$GIT_BUILD_DIR'"
       echo >&2 'error: (has Git been built?).'

On Sun, Apr 20, 2025 at 12:00:08PM +0200, Christian Couder wrote:

The "/.." is intentional here due to the way that `TEST_DIRECTORY` is constructed. If you extend the context of this patch a bit, you can see that `TEST_DIRECTORY=$(pwd)/..`. So stripping "/t" from the suffix wouldn't do anything because it never has that suffix in the first place. And neither do we want to strip "/..", because then we'd end up in "t/perf". So the easiest fix is to just append another "/.." to end up where we want to.

I'll try to paraphrase this in the commit message.

I'd rather want to keep this as-is for now as we have the same error message in "test-lib.sh". If we want to change it we should change both errors, but that feels outside of the scope of this patch series.

Patrick

Wire up benchmarks in Meson. The setup is mostly the same as how we wire up our tests. The only difference is that benchmarks get wired up via the `benchmark()` option instead of via `test()`, which gives them a bit of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will only run benchmarks for the current build. Other usecases, like running benchmarks against multiple different versions of Git, are not currently supported. Users should continue to use "t/perf/run" for those usecases. The script should get extended at one point in time to support Meson, but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build   |  1 +
 t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 78 insertions(+)

On Mon, Apr 14, 2025 at 8:51 AM Patrick Steinhardt <ps@pks.im> wrote:

Instead of just here, I would have expected that something like this would appear towards the top of the top level meson.build file, maybe in a "Benchmarking" section, where there is the rest of the meson documentation for Git.

s/usecases/use cases/

Fair enough.

I wonder what happens when we run `meson test -C build --benchmark` but 'time' is not found.

Wire up a couple of benchmarking options that we end up writing into our "GIT-BUILD-OPTIONS" file. These options allow users to control how exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are exposed as a build option. Those options are used by "t/perf/run", which is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

Patrick Steinhardt <ps@pks.im> writes:

The previous iteration of this series has been kept out of 'seen' for some time and I didn't recall why I did so. With this iteration merged, all GitHub Actions CI tasks with "meson" in the name are failing, so does "documentation" jobs (which recently acquired "let's make sure meson-based build does the docs fine" substep).

Can you help seeing where the merge went wrong (yes, I am suspecting that there is some stupid merge mistake there)?

Thanks.

Junio C Hamano <gitster@pobox.com> writes:

Just a few test CI runs.

    https://github.com/git/git/actions/runs/14457387669

is with this topic mergecd in (with alleged mismerge).

This one

    https://github.com/git/git/actions/runs/14406901394

is from last week without the earlier iteration of this topic, which fails a few meson jobs.

linux-meson job that fails with

meson.build:689:19: ERROR: Command `/usr/bin/git -C /__w/git/git ls-files --deduplicate '*.h' ':!contrib' ':!compat/inet_ntop.c' ':!compat/inet_pton.c' ':!compat/nedmalloc' ':!compat/obstack.*' ':!compat/poll' ':!compat/regex' ':!sha1collisiondetection' ':!sha1dc' ':!t/unit-tests/clar' ':!t/unit-tests/clar' ':!t/t[0-9][0-9][0-9][0-9]*'` failed with status 128.

and

win+Meson test(3) that dies inside Python asyncio both look problematic.

Thanks.

On Tue, Apr 15, 2025 at 11:18:42AM -0700, Junio C Hamano wrote:

Hm, curious. These lines have been moved here by kn/meson-hdr-check, but the logic already existed beforehand. I've Cc'd Karthik -- it would probably make sense to investigate (and ideally also fix) the issue as part of that series.

Yeah, this is a result of cancellation of the job. Johannes already mentioned this issue to me in the past, and it's known that Python's asyncio is a bit wonky in Windows. The problem here is mostly that Windows does not have proper signals.

It's not a huge problem given that this only happens when the job is being cancelled, but it's not great, either. I've already invested some time into improving this on Windows [1], but there are a couple of uncertainties still. Anyway, this should hopefully be fixed soonish.

Patrick

[1]: https://github.com/mesonbuild/meson/pull/14311

Patrick Steinhardt <ps@pks.im> writes:

Indeed. I tentatively kicked meson-hdr-check topic out of 'seen' and the CI seems to be a lot happier. What is curious is that in my local environment 'seen' with meson-hdr-check topic did not fail for me.

Hi,

this small patch series implements support for running our benchmarks in "t/perf" via Meson. The series does not aim to replace "t/perf/run", which is more fully-featured and allows running benchmarks against multiple different trees. Instead, this series only allows running the benchmarks against the current tree. Users are thus expected to continue using "t/perf/run" for more advanced usecases.

Changes in v2:
  - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
  - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im

Changes in v3:
  - Document how to run benchmarks in "meson.build".
  - Expand the message for the commit that enables out-of-tree
    benchmarking.
  - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build           | 13 +++++++--
 meson_options.txt     |  6 ++++
 t/meson.build         | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/aggregate.perl |  2 +-
 t/perf/perf-lib.sh    | 32 ++++++++++++++++++---
 t/perf/run            |  4 +--
 6 files changed, 124 insertions(+), 10 deletions(-)

Range-diff versus v2:

1:  dad3c5a3e78 = 1:  85098f35716 t/perf: fix benchmarks with alternate repo formats
2:  42284091b9c = 2:  457ea957acf t/perf: use configured PERL_PATH
3:  d03ca0c0630 ! 3:  a66bc5aaf51 t/perf: fix benchmarks with out-of-tree builds
    @@ Commit message
         assumption breaks with both CMake and Meson, where the build directory
         can be located in an arbitrary place.
     
    -    Adapt the script so that it works with out-of-tree builds. This prepares
    -    us for wiring up benchmarks in Meson.
    +    Adapt the script so that it works with out-of-tree builds. Most
    +    importantly, this requires us to figure out the location of the build
    +    directory:
    +
    +      - When running benchmarks via our Makefile the build directory is the
    +        same as the source directory. We already know to derive the test
    +        directory ("t/") via `$(pwd)/..`, which works because we chdir into
    +        "t/perf" before executing benchmarks. We can thus derive the build
    +        directory by appending another "/.." to that path.
    +
    +      - When running benchmarks via Meson the build directory is located at
    +        an arbitrary location. The build system thus has to make the path
    +        known by exporting the `GIT_BUILD_DIR` environment variable.
    +
    +    This change prepares us for wiring up benchmarks in Meson.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
4:  14945278066 ! 4:  386b960f3e1 meson: wire up benchmarks
    @@ Commit message
         tests are also accepted when running benchmarks.
     
         Note that the benchmarking target is somewhat limited because it will
    -    only run benchmarks for the current build. Other usecases, like running
    +    only run benchmarks for the current build. Other use cases, like running
         benchmarks against multiple different versions of Git, are not currently
    -    supported. Users should continue to use "t/perf/run" for those usecases.
    -    The script should get extended at one point in time to support Meson,
    -    but this is outside of the scope of this series.
    +    supported. Users should continue to use "t/perf/run" for those use
    +    cases. The script should get extended at one point in time to support
    +    Meson, but this is outside of the scope of this series.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## meson.build ##
    +@@
    + #      # Execute single test interactively such that features like `debug ()` work.
    + #      $ meson test -i --test-args='-ix' t1400-update-ref
    + #
    ++#      # Execute all benchmarks.
    ++#      $ meson test -i --benchmark
    ++#
    ++#      # Execute single benchmark.
    ++#      $ meson test -i --benchmark p0000-*
    ++#
    + #   Test execution is parallelized by default and scales with the number of
    + #   processor cores available. You can change the number of processes by passing
    + #   the `-jN` flag to `meson test`.
     @@ meson.build: git = find_program('git', dirs: program_path, required: false)
      sed = find_program('sed', dirs: program_path)
      shell = find_program('sh', dirs: program_path)
5:  101be1e8efc = 5:  cc3e771ada0 meson: wire up benchmarking options

--- base-commit: 683c54c999c301c2cd6f715c411407c413b1d84e change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

Our benchmarks use a couple of Perl scripts to compute results. These Perl scripts get executed directly, and as the shebang is hardcoded to "/usr/bin/perl" this will fail on any system where the Perl interpreter is located in a different path.

Our build infrastructure already lets users configure the location of Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file. This file is being sourced by "test-lib.sh", and consequently we already have the "PERL_PATH" variable available that contains its configured location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more esoteric systems like NixOS. Furthermore, adapt the shebang to use env(1) to execute Perl so that users who have Perl in PATH, but in a non-standard location can execute the script directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/aggregate.perl | 2 +-
 t/perf/perf-lib.sh    | 4 ++--
 t/perf/run            | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

Many of our benchmarks operate on a user-defined repository that we copy over before running the benchmarked logic. To keep unintentional side effects caused by on-disk state at bay we skip copying some files. This includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite easy to inadvertently carry over configuration that may significantly impact the performance measurements. But we cannot fully ignore the configuration either, as it may contain information about the repository format. This will cause failures when for example using a repository with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

The "perf-lib.sh" script is sourced by all of our benchmarking suites to make available common infrastructure. The script assumes that build and source directory are the same, which works for our Makefile. But the assumption breaks with both CMake and Meson, where the build directory can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. Most importantly, this requires us to figure out the location of the build directory:

  - When running benchmarks via our Makefile the build directory is the
    same as the source directory. We already know to derive the test
    directory ("t/") via `$(pwd)/..`, which works because we chdir into
    "t/perf" before executing benchmarks. We can thus derive the build
    directory by appending another "/.." to that path.

  - When running benchmarks via Meson the build directory is located at
    an arbitrary location. The build system thus has to make the path
    known by exporting the `GIT_BUILD_DIR` environment variable.

This change prepares us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

Wire up benchmarks in Meson. The setup is mostly the same as how we wire up our tests. The only difference is that benchmarks get wired up via the `benchmark()` option instead of via `test()`, which gives them a bit of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will only run benchmarks for the current build. Other use cases, like running benchmarks against multiple different versions of Git, are not currently supported. Users should continue to use "t/perf/run" for those use cases. The script should get extended at one point in time to support Meson, but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build   |  7 ++++++
 t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)

Wire up a couple of benchmarking options that we end up writing into our "GIT-BUILD-OPTIONS" file. These options allow users to control how exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are exposed as a build option. Those options are used by "t/perf/run", which is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

On Tue, Apr 22, 2025 at 8:50 AM Patrick Steinhardt <ps@pks.im> wrote:

In https://lore.kernel.org/git/CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com/ I wrote:

"I wonder what happens when we run `meson test -C build --benchmark` but 'time' is not found."

because I wasn't sure if it would just do nothing in that case which might not be very user friendly.

Otherwise it looks like my concerns have been addressed.

Thanks.

On Tue, Apr 22, 2025 at 09:27:57AM +0200, Christian Couder wrote:

Ah, sorry, forgot to answer that question. What Meson does in that case is to print "No tests defined". We could help improve usability a bit by printing benchmarks as part of the auto-detected features after setup of the build directory has finished. That would make it more discoverable that benchmarks have been disabled.

Patrick

On Tue, Apr 22, 2025 at 9:53 AM Patrick Steinhardt <ps@pks.im> wrote:

I was more expecting something simple like:

if time.found()
  ...
else
  error('Benchmarking requires the `time` command')
endif

in the same way as in meson.build elsewhere we have things like:

if not msgfmt.found() and gettext_option.enabled()
  error('Internationalization via libintl requires msgfmt')
endif

On Wed, Apr 23, 2025 at 04:44:20PM +0200, Christian Couder wrote:

But erroring out by default doesn't really feel nice to the general developer. I'd claim that 99% of the time, developers will only end up running the test suite, never the benchmarks. So the default should match that and not require GNU time to be available, if you ask me.

An alternative could be to have a 'benchmarks' feature option wired up. This feature would be set to 'auto', and then we can detect time like this:

    time = find_program('time', dirs: program_path, required: get_option('benchmarks'))

This will then auto-enable or -disable the feature depending on the availability of GNU time. But if the user passes `-Dbenchmarks=enabled`, we would fail the setup in case the program wasn't found.

Patrick

On Thu, Apr 24, 2025 at 6:31 AM Patrick Steinhardt <ps@pks.im> wrote:

Yeah, right.

Yeah, that would be great.

An alternative might be a placeholder benchmark that just reports an error when run, like:

if time.found()
   ...
else
  benchmark('perf-benchmarks-require-time', shell, args: [
    '-c',
    'echo "ERROR: Benchmarks require the \"time\" command to be
available. Please install it and reconfigure." >&2; exit 1'
  ], env: test_environment, )
endif

Patrick Steinhardt <ps@pks.im> writes:

Maybe a stupid question, but does "time" a shell built-in count when "if time.found()" is evaluated?

    $ type --all time
    time is a shell keyword
    time is /usr/bin/time
    time is /bin/time

Sounds quite sensible. Do what most users expect by default without failing, let users give explicit preference, and when the explicitly expressed preference cannot be fulfilled, loudly error out. I like it.

Thanks.

On Thu, Apr 24, 2025 at 04:13:15AM -0700, Junio C Hamano wrote:

No, it doesn't, and that is intentional. Our benchmarks explicitly require GNU time because it surfaces more information than the shell builtin.

We even have a `USR_BIN_TIME` prereq in our test lib, which is unfortunately way too limiting for some platforms like NixOS. Maybe I should use that as an excuse to also refactor this prereq while at it to instead use a path-based lookup of time. On the other hand it's only used by a single test anyway, so it doesn't really feel worth it.

Patrick

Patrick Steinhardt <ps@pks.im> writes:

Thanks for adding this.

I have no further comments. I approve.

-- Toon

Hi,

this small patch series implements support for running our benchmarks in "t/perf" via Meson. The series does not aim to replace "t/perf/run", which is more fully-featured and allows running benchmarks against multiple different trees. Instead, this series only allows running the benchmarks against the current tree. Users are thus expected to continue using "t/perf/run" for more advanced usecases.

Changes in v2:
  - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
  - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im

Changes in v3:
  - Document how to run benchmarks in "meson.build".
  - Expand the message for the commit that enables out-of-tree
    benchmarking.
  - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im

Changes in v4:
  - The patch series was rebased on top of f65182a99e5 (The ninth batch,
    2025-04-24). This is due to a conflict with ps/test-wo-perl-prereq.
  - Introduce a 'benchmarks' option. This allows developers to require
    benchmarks as desired. By default, we auto-detect whether the host
    system has all dependencies available and enable or disable them
    accordingly.
  - Report whether or not benchmarks are enabled via `summary()`.
  - Our benchmarks depend on Perl, so add this dependency accordingly.
  - Link to v3: https://lore.kernel.org/r/20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build           | 16 ++++++++---
 meson_options.txt     |  8 ++++++
 t/meson.build         | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/aggregate.perl |  2 +-
 t/perf/perf-lib.sh    | 32 ++++++++++++++++++---
 t/perf/run            |  4 +--
 6 files changed, 128 insertions(+), 11 deletions(-)

Range-diff versus v3:

1:  2375a16c2c0 = 1:  174804805c2 t/perf: fix benchmarks with alternate repo formats
2:  69782035246 = 2:  41faa3d9b3b t/perf: use configured PERL_PATH
3:  cca40e9dd0e = 3:  87ea3ab1a45 t/perf: fix benchmarks with out-of-tree builds
4:  50e9b507db4 ! 4:  723a0a2e449 meson: wire up benchmarks
    @@ meson.build
      #   Test execution is parallelized by default and scales with the number of
      #   processor cores available. You can change the number of processes by passing
      #   the `-jN` flag to `meson test`.
    -@@ meson.build: git = find_program('git', dirs: program_path, required: false)
    - sed = find_program('sed', dirs: program_path)
    - shell = find_program('sh', dirs: program_path)
    - tar = find_program('tar', dirs: program_path)
    -+time = find_program('time', dirs: program_path, required: false)
    +@@ meson.build: git = find_program('git', dirs: program_path, native: true, required: false)
    + sed = find_program('sed', dirs: program_path, native: true)
    + shell = find_program('sh', dirs: program_path, native: true)
    + tar = find_program('tar', dirs: program_path, native: true)
    ++time = find_program('time', dirs: program_path, required: get_option('benchmarks'))
      
    - # Sanity-check that programs required for the build exist.
    - foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']
    + target_shell = find_program('sh', dirs: program_path, native: false)
    + 
    +@@ meson.build: endif
    + # features. It is optional if you want to neither execute tests nor use any of
    + # these optional features.
    + perl_required = get_option('perl')
    +-if get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []
    ++if get_option('benchmarks').enabled() or get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []
    +   perl_required = true
    + endif
    + 
    +@@ meson.build: meson.add_dist_script(
    + )
    + 
    + summary({
    ++  'benchmarks': get_option('tests') and perl.found() and time.found(),
    +   'curl': curl.found(),
    +   'expat': expat.found(),
    +   'gettext': intl.found(),
    +
    + ## meson_options.txt ##
    +@@ meson_options.txt: option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
    +   description: 'Which backend to use to generate documentation.')
    + 
    + # Testing.
    ++option('benchmarks', type: 'feature', value: 'auto',
    ++  description: 'Enable benchmarks. This requires Perl and GNU time.')
    + option('coccinelle', type: 'feature', value: 'auto',
    +   description: 'Provide a coccicheck target that generates a Coccinelle patch.')
    + option('tests', type: 'boolean', value: true,
     
      ## t/meson.build ##
     @@ t/meson.build: integration_tests = [
    @@ t/meson.build: foreach integration_test : integration_tests
        )
      endforeach
     +
    -+if time.found()
    ++if perl.found() and time.found()
     +  benchmark_environment = test_environment
     +  benchmark_environment.set('GTIME', time.full_path())
     +
5:  52cca4b17ae ! 5:  16c0230656f meson: wire up benchmarking options
    @@ meson.build: builtin_sources += custom_target(
     
      ## meson_options.txt ##
     @@ meson_options.txt: option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto
    -   description: 'Which backend to use to generate documentation.')
    - 
      # Testing.
    + option('benchmarks', type: 'feature', value: 'auto',
    +   description: 'Enable benchmarks. This requires Perl and GNU time.')
     +option('benchmark_repo', type: 'string', value: '',
     +  description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')
     +option('benchmark_large_repo', type: 'string', value: '',

--- base-commit: f65182a99e545d2f2bc22e6c1c2da192133b16a3 change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

Many of our benchmarks operate on a user-defined repository that we copy over before running the benchmarked logic. To keep unintentional side effects caused by on-disk state at bay we skip copying some files. This includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite easy to inadvertently carry over configuration that may significantly impact the performance measurements. But we cannot fully ignore the configuration either, as it may contain information about the repository format. This will cause failures when for example using a repository with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Our benchmarks use a couple of Perl scripts to compute results. These Perl scripts get executed directly, and as the shebang is hardcoded to "/usr/bin/perl" this will fail on any system where the Perl interpreter is located in a different path.

Our build infrastructure already lets users configure the location of Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file. This file is being sourced by "test-lib.sh", and consequently we already have the "PERL_PATH" variable available that contains its configured location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more esoteric systems like NixOS. Furthermore, adapt the shebang to use env(1) to execute Perl so that users who have Perl in PATH, but in a non-standard location can execute the script directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/aggregate.perl | 2 +-
 t/perf/perf-lib.sh    | 4 ++--
 t/perf/run            | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

The "perf-lib.sh" script is sourced by all of our benchmarking suites to make available common infrastructure. The script assumes that build and source directory are the same, which works for our Makefile. But the assumption breaks with both CMake and Meson, where the build directory can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. Most importantly, this requires us to figure out the location of the build directory:

  - When running benchmarks via our Makefile the build directory is the
    same as the source directory. We already know to derive the test
    directory ("t/") via `$(pwd)/..`, which works because we chdir into
    "t/perf" before executing benchmarks. We can thus derive the build
    directory by appending another "/.." to that path.

  - When running benchmarks via Meson the build directory is located at
    an arbitrary location. The build system thus has to make the path
    known by exporting the `GIT_BUILD_DIR` environment variable.

This change prepares us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

Wire up benchmarks in Meson. The setup is mostly the same as how we wire up our tests. The only difference is that benchmarks get wired up via the `benchmark()` option instead of via `test()`, which gives them a bit of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will only run benchmarks for the current build. Other use cases, like running benchmarks against multiple different versions of Git, are not currently supported. Users should continue to use "t/perf/run" for those use cases. The script should get extended at one point in time to support Meson, but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 10 +++++++-
 meson_options.txt |  2 ++
 t/meson.build     | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 88 insertions(+), 1 deletion(-)

On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt <ps@pks.im> wrote:

[...]

Doesn't the above give the impression that benchmarks are also parallelized by default while the commit message says they are not?

On Fri, Apr 25, 2025 at 09:57:33AM +0200, Christian Couder wrote:

To me it doesn't because it talks about test execution, not benchmark execution. The comments for each of the invocations also discern between tests and benchmarks. I'm happy to hear about alternative ways to word this though.

Patrick

On Fri, Apr 25, 2025 at 10:01 AM Patrick Steinhardt <ps@pks.im> wrote:

Maybe something like:

"Test execution (but not benchmark execution) is parallelized by default ..."

On Fri, Apr 25, 2025 at 10:09:18AM +0200, Christian Couder wrote:

Okay. I amended this locally, but will hold off sending another iteration just for this change. Thanks!

Patrick

Wire up a couple of benchmarking options that we end up writing into our "GIT-BUILD-OPTIONS" file. These options allow users to control how exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are exposed as a build option. Those options are used by "t/perf/run", which is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)

On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt <ps@pks.im> wrote:

Nice!

Nice too!

[...]

In https://lore.kernel.org/git/aAc8f52rJ8ATncVc@pks.im/ you said you would try to paraphrase some explanations about how GIT_BUILD_DIR is computed, but it looks like there aren't any changes in the patch since the previous version :-(

On Fri, Apr 25, 2025 at 10:06:12AM +0200, Christian Couder wrote:

There were, but not in this version:

3:  d03ca0c0630 ! 3:  a66bc5aaf51 t/perf: fix benchmarks with out-of-tree builds
    @@ Commit message
         assumption breaks with both CMake and Meson, where the build directory
         can be located in an arbitrary place.

    -    Adapt the script so that it works with out-of-tree builds. This prepares
    -    us for wiring up benchmarks in Meson.
    +    Adapt the script so that it works with out-of-tree builds. Most
    +    importantly, this requires us to figure out the location of the build
    +    directory:
    +
    +      - When running benchmarks via our Makefile the build directory is the
    +        same as the source directory. We already know to derive the test
    +        directory ("t/") via `$(pwd)/..`, which works because we chdir into
    +        "t/perf" before executing benchmarks. We can thus derive the build
    +        directory by appending another "/.." to that path.
    +
    +      - When running benchmarks via Meson the build directory is located at
    +        an arbitrary location. The build system thus has to make the path
    +        known by exporting the `GIT_BUILD_DIR` environment variable.
    +
    +    This change prepares us for wiring up benchmarks in Meson.

The changes were already part of v3, so you have to look at the diff versus v2.

Patrick

On Fri, Apr 25, 2025 at 10:26 AM Patrick Steinhardt <ps@pks.im> wrote:

Oops, sorry for the noise.

Hi,

this small patch series implements support for running our benchmarks in "t/perf" via Meson. The series does not aim to replace "t/perf/run", which is more fully-featured and allows running benchmarks against multiple different trees. Instead, this series only allows running the benchmarks against the current tree. Users are thus expected to continue using "t/perf/run" for more advanced usecases.

Changes in v2:
  - Adapt "aggregate.perl" to use a "/usr/bin/env perl" shebang.
  - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im

Changes in v3:
  - Document how to run benchmarks in "meson.build".
  - Expand the message for the commit that enables out-of-tree
    benchmarking.
  - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im

Changes in v4:
  - The patch series was rebased on top of f65182a99e5 (The ninth batch,
    2025-04-24). This is due to a conflict with ps/test-wo-perl-prereq.
  - Introduce a 'benchmarks' option. This allows developers to require
    benchmarks as desired. By default, we auto-detect whether the host
    system has all dependencies available and enable or disable them
    accordingly.
  - Report whether or not benchmarks are enabled via `summary()`.
  - Our benchmarks depend on Perl, so add this dependency accordingly.
  - Link to v3: https://lore.kernel.org/r/20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im

Changes in v5:
  - Clarify that benchmarks are not run in parallel by default.
  - Link to v4: https://lore.kernel.org/r/20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (5):
      t/perf: fix benchmarks with alternate repo formats
      t/perf: use configured PERL_PATH
      t/perf: fix benchmarks with out-of-tree builds
      meson: wire up benchmarks
      meson: wire up benchmarking options

 meson.build           | 22 ++++++++++-----
 meson_options.txt     |  8 ++++++
 t/meson.build         | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++
 t/perf/aggregate.perl |  2 +-
 t/perf/perf-lib.sh    | 32 ++++++++++++++++++---
 t/perf/run            |  4 +--
 6 files changed, 131 insertions(+), 14 deletions(-)

Range-diff versus v4:

1:  bc19180b174 = 1:  03181f601f2 t/perf: fix benchmarks with alternate repo formats
2:  3e128c09333 = 2:  afb2fe1ce63 t/perf: use configured PERL_PATH
3:  f7adc20d14c = 3:  a61e43c6cfb t/perf: fix benchmarks with out-of-tree builds
4:  8b3ea380503 ! 4:  e51b46296ed meson: wire up benchmarks
    @@ meson.build
      #      # Execute single test interactively such that features like `debug ()` work.
      #      $ meson test -i --test-args='-ix' t1400-update-ref
      #
    +-#   Test execution is parallelized by default and scales with the number of
    +-#   processor cores available. You can change the number of processes by passing
    +-#   the `-jN` flag to `meson test`.
     +#      # Execute all benchmarks.
     +#      $ meson test -i --benchmark
     +#
     +#      # Execute single benchmark.
     +#      $ meson test -i --benchmark p0000-*
     +#
    - #   Test execution is parallelized by default and scales with the number of
    - #   processor cores available. You can change the number of processes by passing
    - #   the `-jN` flag to `meson test`.
    ++#   Test execution (but not benchmark execution) is parallelized by default and
    ++#   scales with the number of processor cores available. You can change the
    ++#   number of processes by passing the `-jN` flag to `meson test`.
    + #
    + # 4. Install the Git distribution. Again, this can be done via Meson, Ninja or
    + #    Samurai:
     @@ meson.build: git = find_program('git', dirs: program_path, native: true, required: false)
      sed = find_program('sed', dirs: program_path, native: true)
      shell = find_program('sh', dirs: program_path, native: true)
5:  a0b06cc7017 = 5:  20f9dd86d66 meson: wire up benchmarking options

--- base-commit: f65182a99e545d2f2bc22e6c1c2da192133b16a3 change-id: 20250328-pks-meson-benchmarks-a8fac5f69467

Many of our benchmarks operate on a user-defined repository that we copy over before running the benchmarked logic. To keep unintentional side effects caused by on-disk state at bay we skip copying some files. This includes for example hooks, but also the repo's configuration.

It is quite sensible to not copy over the configuration, as it is quite easy to inadvertently carry over configuration that may significantly impact the performance measurements. But we cannot fully ignore the configuration either, as it may contain information about the repository format. This will cause failures when for example using a repository with SHA256 object format or the reftable ref format.

Fix the issue by parsing the reference and object formats from the source repository and passing them to git-init(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Our benchmarks use a couple of Perl scripts to compute results. These Perl scripts get executed directly, and as the shebang is hardcoded to "/usr/bin/perl" this will fail on any system where the Perl interpreter is located in a different path.

Our build infrastructure already lets users configure the location of Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file. This file is being sourced by "test-lib.sh", and consequently we already have the "PERL_PATH" variable available that contains its configured location.

Use "PERL_PATH" to execute Perl scripts, which makes them work on more esoteric systems like NixOS. Furthermore, adapt the shebang to use env(1) to execute Perl so that users who have Perl in PATH, but in a non-standard location can execute the script directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/aggregate.perl | 2 +-
 t/perf/perf-lib.sh    | 4 ++--
 t/perf/run            | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

The "perf-lib.sh" script is sourced by all of our benchmarking suites to make available common infrastructure. The script assumes that build and source directory are the same, which works for our Makefile. But the assumption breaks with both CMake and Meson, where the build directory can be located in an arbitrary place.

Adapt the script so that it works with out-of-tree builds. Most importantly, this requires us to figure out the location of the build directory:

  - When running benchmarks via our Makefile the build directory is the
    same as the source directory. We already know to derive the test
    directory ("t/") via `$(pwd)/..`, which works because we chdir into
    "t/perf" before executing benchmarks. We can thus derive the build
    directory by appending another "/.." to that path.

  - When running benchmarks via Meson the build directory is located at
    an arbitrary location. The build system thus has to make the path
    known by exporting the `GIT_BUILD_DIR` environment variable.

This change prepares us for wiring up benchmarks in Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

Wire up benchmarks in Meson. The setup is mostly the same as how we wire up our tests. The only difference is that benchmarks get wired up via the `benchmark()` option instead of via `test()`, which gives them a bit of special treatment:

  - Benchmarks never run in parallel.

  - Benchmarks aren't run by default when tests are executed.

  - Meson does not inject the `MALLOC_PERTURB` environment variable.

Using benchmarks is quite simple:

    ```
    $ meson setup build
    # Run all benchmarks.
    $ meson test -C build --benchmark
    # Run a specific benchmark.
    $ meson test -C build --benchmark p0000-*
    ```

Other than that the usual command line arguments accepted when running tests are also accepted when running benchmarks.

Note that the benchmarking target is somewhat limited because it will only run benchmarks for the current build. Other use cases, like running benchmarks against multiple different versions of Git, are not currently supported. Users should continue to use "t/perf/run" for those use cases. The script should get extended at one point in time to support Meson, but this is outside of the scope of this series.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 16 +++++++++---
 meson_options.txt |  2 ++
 t/meson.build     | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+), 4 deletions(-)

Wire up a couple of benchmarking options that we end up writing into our "GIT-BUILD-OPTIONS" file. These options allow users to control how exactly benchmarks are executed.

Note that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are exposed as a build option. Those options are used by "t/perf/run", which is not used by Meson.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 6 +++---
 meson_options.txt | 6 ++++++
 2 files changed, 9 insertions(+), 3 deletions(-)