{"thread":{"id":"63221","subject":"[PATCH 0/5] meson: wire up support for benchmarks","startedAt":"2025-03-31T06:17:05Z","lastAt":"2025-04-28T07:30:58Z","messageCount":62,"participants":["Patrick Steinhardt","Toon Claes","Junio C Hamano","Christian Couder"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"515315","messageId":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","threadId":"63221","inReplyTo":null,"subject":"[PATCH 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-31T06:16:43Z","receivedAt":"2025-03-31T06:17:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis small patch series implements support for running our benchmarks in\n\"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\nwhich is more fully-featured and allows running benchmarks against\nmultiple different trees. Instead, this series only allows running the\nbenchmarks against the current tree. Users are thus expected to continue\nusing \"t/perf/run\" for more advanced usecases.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (5):\n t/perf: fix benchmarks with alternate repo formats\n t/perf: use configured PERL_PATH\n t/perf: fix benchmarks with out-of-tree builds\n meson: wire up benchmarks\n meson: wire up benchmarking options\n\n meson.build | 7 ++---\n meson_options.txt | 6 +++++\n t/meson.build | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n t/perf/perf-lib.sh | 32 ++++++++++++++++++++---\n t/perf/run | 4 +--\n 5 files changed, 117 insertions(+), 9 deletions(-)\n\n\n---\nbase-commit: 683c54c999c301c2cd6f715c411407c413b1d84e\nchange-id: 20250328-pks-meson-benchmarks-a8fac5f69467\n\n"},{"id":"515317","messageId":"20250331-pks-meson-benchmarks-v1-1-b2ace85616a3@pks.im","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","subject":"[PATCH 1/5] t/perf: fix benchmarks with alternate repo formats","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-31T06:16:44Z","receivedAt":"2025-03-31T06:17:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Many of our benchmarks operate on a user-defined repository that we copy\nover before running the benchmarked logic. To keep unintentional side\neffects caused by on-disk state at bay we skip copying some files. This\nincludes for example hooks, but also the repo's configuration.\n\nIt is quite sensible to not copy over the configuration, as it is quite\neasy to inadvertently carry over configuration that may significantly\nimpact the performance measurements. But we cannot fully ignore the\nconfiguration either, as it may contain information about the repository\nformat. This will cause failures when for example using a repository\nwith SHA256 object format or the reftable ref format.\n\nFix the issue by parsing the reference and object formats from the\nsource repository and passing them to git-init(1).\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 8ab6d9c4694..1a9a51ca3cc 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -98,6 +98,8 @@ test_perf_create_repo_from () {\n \tsource_git=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-dir)\"\n \tobjects_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-path objects)\"\n \tcommon_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-common-dir)\"\n+\trefformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-ref-format)\"\n+\tobjectformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-object-format)\"\n \tmkdir -p \"$repo/.git\"\n \t(\n \t\tcd \"$source\" &&\n@@ -114,7 +116,7 @@ test_perf_create_repo_from () {\n \t) &&\n \t(\n \t\tcd \"$repo\" &&\n-\t\t\"$MODERN_GIT\" init -q &&\n+\t\t\"$MODERN_GIT\" init -q --ref-format=\"$refformat\" --object-format=\"$objectformat\" &&\n \t\ttest_perf_do_repo_symlink_config_ &&\n \t\tmv .git/hooks .git/hooks-disabled 2>/dev/null &&\n \t\tif test -f .git/index.lock\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515316","messageId":"20250331-pks-meson-benchmarks-v1-2-b2ace85616a3@pks.im","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","subject":"[PATCH 2/5] t/perf: use configured PERL_PATH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-31T06:16:45Z","receivedAt":"2025-03-31T06:17:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our benchmarks use a couple of Perl scripts to compute results. These\nPerl scripts get executed directly, and as the shebang is hardcoded to\n\"/usr/bin/perl\" this will fail on any system where the Perl interpreter\nis located in a different path.\n\nOur build infrastructure already lets users configure the location of\nPerl, which ultimately gets written into the GIT-BUILD-OPTIONS file.\nThis file is being sourced by \"test-lib.sh\", and consequently we already\nhave the \"PERL_PATH\" variable available that contains its configured\nlocation.\n\nUse \"PERL_PATH\" to execute Perl scripts, which makes them work on more\nesoteric systems like NixOS.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 4 ++--\n t/perf/run | 4 ++--\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 1a9a51ca3cc..4173eee4def 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -276,7 +276,7 @@ test_perf_ () {\n \telse\n \t\ttest_ok_ \"$1\"\n \tfi\n-\t\"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n \trm test_time.*\n }\n \n@@ -324,7 +324,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\ndiff --git a/t/perf/run b/t/perf/run\nindex 486ead21980..073bcb2afff 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -192,10 +192,10 @@ run_subsection () {\n \n \tif test -z \"$GIT_PERF_SEND_TO_CODESPEED\"\n \tthen\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n \telse\n \t\tjson_res_file=\"\"$TEST_RESULTS_DIR\"/$GIT_PERF_SUBSECTION/aggregate.json\"\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n \t\tsend_data_url=\"$GIT_PERF_SEND_TO_CODESPEED/result/add/json/\"\n \t\tcurl -v --request POST --data-urlencode \"json=$(cat \"$json_res_file\")\" \"$send_data_url\"\n \tfi\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515318","messageId":"20250331-pks-meson-benchmarks-v1-3-b2ace85616a3@pks.im","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","subject":"[PATCH 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-31T06:16:46Z","receivedAt":"2025-03-31T06:17:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"perf-lib.sh\" script is sourced by all of our benchmarking suites to\nmake available common infrastructure. The script assumes that build and\nsource directory are the same, which works for our Makefile. But the\nassumption breaks with both CMake and Meson, where the build directory\ncan be located in an arbitrary place.\n\nAdapt the script so that it works with out-of-tree builds. This prepares\nus for wiring up benchmarks in Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--\n 1 file changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 4173eee4def..5406557b7ca 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)\n TEST_NO_CREATE_REPO=t\n TEST_NO_MALLOC_CHECK=t\n \n-. ../test-lib.sh\n+# While test-lib.sh computes the build directory for us, we also have to do the\n+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first\n+# place.\n+GIT_BUILD_DIR=\"${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}\"\n+if test -f \"$GIT_BUILD_DIR/GIT-BUILD-DIR\"\n+then\n+\tGIT_BUILD_DIR=\"$(cat \"$GIT_BUILD_DIR/GIT-BUILD-DIR\")\" || exit 1\n+\t# On Windows, we must convert Windows paths lest they contain a colon\n+\tcase \"$(uname -s)\" in\n+\t*MINGW*)\n+\t\tGIT_BUILD_DIR=\"$(cygpath -au \"$GIT_BUILD_DIR\")\"\n+\t\t;;\n+\tesac\n+fi\n+\n+if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+then\n+\techo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'\n+\texit 1\n+fi\n+\n+. \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+. \"$GIT_SOURCE_DIR\"/t/test-lib.sh\n \n unset GIT_CONFIG_NOSYSTEM\n GIT_CONFIG_SYSTEM=\"$TEST_DIRECTORY/perf/config\"\n@@ -324,7 +346,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" \"$GIT_SOURCE_DIR\"/t/perf/aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515319","messageId":"20250331-pks-meson-benchmarks-v1-4-b2ace85616a3@pks.im","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","subject":"[PATCH 4/5] meson: wire up benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-31T06:16:47Z","receivedAt":"2025-03-31T06:17:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up benchmarks in Meson. The setup is mostly the same as how we wire\nup our tests. The only difference is that benchmarks get wired up via\nthe `benchmark()` option instead of via `test()`, which gives them a bit\nof special treatment:\n\n - Benchmarks never run in parallel.\n\n - Benchmarks aren't run by default when tests are executed.\n\n - Meson does not inject the `MALLOC_PERTURB` environment variable.\n\nUsing benchmarks is quite simple:\n\n ```\n $ meson setup build\n # Run all benchmarks.\n $ meson test -C build --benchmark\n # Run a specific benchmark.\n $ meson test -C build --benchmark p0000-*\n ```\n\nOther than that the usual command line arguments accepted when running\ntests are also accepted when running benchmarks.\n\nNote that the benchmarking target is somewhat limited because it will\nonly run benchmarks for the current build. Other usecases, like running\nbenchmarks against multiple different versions of Git, are not currently\nsupported. Users should continue to use \"t/perf/run\" for those usecases.\nThe script should get extended at one point in time to support Meson,\nbut this is outside of the scope of this series.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 1 +\n t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 78 insertions(+)\n\ndiff --git a/meson.build b/meson.build\nindex efe2871c9db..9af4a10188d 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -204,6 +204,7 @@ git = find_program('git', dirs: program_path, required: false)\n sed = find_program('sed', dirs: program_path)\n shell = find_program('sh', dirs: program_path)\n tar = find_program('tar', dirs: program_path)\n+time = find_program('time', dirs: program_path, required: false)\n \n # Sanity-check that programs required for the build exist.\n foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']\ndiff --git a/t/meson.build b/t/meson.build\nindex a59da26be3f..dba327fd7ec 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -1096,11 +1096,71 @@ integration_tests = [\n 't9903-bash-prompt.sh',\n ]\n \n+benchmarks = [\n+ 'perf/p0000-perf-lib-sanity.sh',\n+ 'perf/p0001-rev-list.sh',\n+ 'perf/p0002-read-cache.sh',\n+ 'perf/p0003-delta-base-cache.sh',\n+ 'perf/p0004-lazy-init-name-hash.sh',\n+ 'perf/p0005-status.sh',\n+ 'perf/p0006-read-tree-checkout.sh',\n+ 'perf/p0007-write-cache.sh',\n+ 'perf/p0008-odb-fsync.sh',\n+ 'perf/p0071-sort.sh',\n+ 'perf/p0090-cache-tree.sh',\n+ 'perf/p0100-globbing.sh',\n+ 'perf/p1006-cat-file.sh',\n+ 'perf/p1400-update-ref.sh',\n+ 'perf/p1450-fsck.sh',\n+ 'perf/p1451-fsck-skip-list.sh',\n+ 'perf/p1500-graph-walks.sh',\n+ 'perf/p2000-sparse-operations.sh',\n+ 'perf/p3400-rebase.sh',\n+ 'perf/p3404-rebase-interactive.sh',\n+ 'perf/p4000-diff-algorithms.sh',\n+ 'perf/p4001-diff-no-index.sh',\n+ 'perf/p4002-diff-color-moved.sh',\n+ 'perf/p4205-log-pretty-formats.sh',\n+ 'perf/p4209-pickaxe.sh',\n+ 'perf/p4211-line-log.sh',\n+ 'perf/p4220-log-grep-engines.sh',\n+ 'perf/p4221-log-grep-engines-fixed.sh',\n+ 'perf/p5302-pack-index.sh',\n+ 'perf/p5303-many-packs.sh',\n+ 'perf/p5304-prune.sh',\n+ 'perf/p5310-pack-bitmaps.sh',\n+ 'perf/p5311-pack-bitmaps-fetch.sh',\n+ 'perf/p5312-pack-bitmaps-revs.sh',\n+ 'perf/p5313-pack-objects.sh',\n+ 'perf/p5314-name-hash.sh',\n+ 'perf/p5326-multi-pack-bitmaps.sh',\n+ 'perf/p5332-multi-pack-reuse.sh',\n+ 'perf/p5333-pseudo-merge-bitmaps.sh',\n+ 'perf/p5550-fetch-tags.sh',\n+ 'perf/p5551-fetch-rescan.sh',\n+ 'perf/p5600-partial-clone.sh',\n+ 'perf/p5601-clone-reference.sh',\n+ 'perf/p6100-describe.sh',\n+ 'perf/p6300-for-each-ref.sh',\n+ 'perf/p7000-filter-branch.sh',\n+ 'perf/p7102-reset.sh',\n+ 'perf/p7300-clean.sh',\n+ 'perf/p7519-fsmonitor.sh',\n+ 'perf/p7527-builtin-fsmonitor.sh',\n+ 'perf/p7810-grep.sh',\n+ 'perf/p7820-grep-engines.sh',\n+ 'perf/p7821-grep-engines-fixed.sh',\n+ 'perf/p7822-grep-perl-character.sh',\n+ 'perf/p9210-scalar.sh',\n+ 'perf/p9300-fast-import-export.sh',\n+]\n+\n # Sanity check that we are not missing any tests present in 't/'. This check\n # only runs once at configure time and is thus best-effort, only. It is\n # sufficient to catch missing test suites in our CI though.\n foreach glob, tests : {\n 't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,\n+ 'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,\n 'unit-tests/t-*.c': unit_test_programs,\n 'unit-tests/u-*.c': clar_test_suites,\n }\n@@ -1152,3 +1212,20 @@ foreach integration_test : integration_tests\n timeout: 0,\n )\n endforeach\n+\n+if time.found()\n+ benchmark_environment = test_environment\n+ benchmark_environment.set('GTIME', time.full_path())\n+\n+ foreach benchmark : benchmarks\n+ benchmark(fs.stem(benchmark), shell,\n+ args: [\n+ fs.name(benchmark),\n+ ],\n+ workdir: meson.current_source_dir() / 'perf',\n+ env: benchmark_environment,\n+ depends: test_dependencies + bin_wrappers,\n+ timeout: 0,\n+ )\n+ endforeach\n+endif\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515320","messageId":"20250331-pks-meson-benchmarks-v1-5-b2ace85616a3@pks.im","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","subject":"[PATCH 5/5] meson: wire up benchmarking options","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-31T06:16:48Z","receivedAt":"2025-03-31T06:17:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up a couple of benchmarking options that we end up writing into our\n\"GIT-BUILD-OPTIONS\" file. These options allow users to control how\nexactly benchmarks are executed.\n\nNote that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are\nexposed as a build option. Those options are used by \"t/perf/run\", which\nis not used by Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 6 +++---\n meson_options.txt | 6 ++++++\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/meson.build b/meson.build\nindex 9af4a10188d..ba128ab0759 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -660,11 +660,11 @@ builtin_sources += custom_target(\n # build options to our tests.\n build_options_config = configuration_data()\n build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_LARGE_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))\n build_options_config.set('GIT_PERF_MAKE_COMMAND', '')\n build_options_config.set('GIT_PERF_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')\n-build_options_config.set('GIT_PERF_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())\n+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))\n build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')\n build_options_config.set('GIT_TEST_INDEX_VERSION', '')\n build_options_config.set('GIT_TEST_OPTS', '')\ndiff --git a/meson_options.txt b/meson_options.txt\nindex 78d172a7401..ca106c43b2e 100644\n--- a/meson_options.txt\n+++ b/meson_options.txt\n@@ -101,6 +101,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto\n description: 'Which backend to use to generate documentation.')\n \n # Testing.\n+option('benchmark_repo', type: 'string', value: '',\n+ description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')\n+option('benchmark_large_repo', type: 'string', value: '',\n+ description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')\n+option('benchmark_repeat_count', type: 'integer', value: 3,\n+ description: 'Number of times a test should be repeated for best-of-N measurements.')\n option('coccinelle', type: 'feature', value: 'auto',\n description: 'Provide a coccicheck target that generates a Coccinelle patch.')\n option('tests', type: 'boolean', value: true,\n\n-- \n2.49.0.604.gff1f9ca942.dirty\n\n"},{"id":"515953","messageId":"87v7rcdzck.fsf@iotcl.com","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-3-b2ace85616a3@pks.im","subject":"Re: [PATCH 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-04-10T11:34:35Z","receivedAt":"2025-04-10T11:34:54Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt writes:\n\n> The \"perf-lib.sh\" script is sourced by all of our benchmarking suites to\n> make available common infrastructure. The script assumes that build and\n> source directory are the same, which works for our Makefile. But the\n> assumption breaks with both CMake and Meson, where the build directory\n> can be located in an arbitrary place.\n>\n> Adapt the script so that it works with out-of-tree builds. This prepares\n> us for wiring up benchmarks in Meson.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--\n> 1 file changed, 24 insertions(+), 2 deletions(-)\n>\n> diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\n> index 4173eee4def..5406557b7ca 100644\n> --- a/t/perf/perf-lib.sh\n> +++ b/t/perf/perf-lib.sh\n> @@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)\n> TEST_NO_CREATE_REPO=t\n> TEST_NO_MALLOC_CHECK=t\n> \n> -. ../test-lib.sh\n> +# While test-lib.sh computes the build directory for us, we also have to do the\n> +# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first\n> +# place.\n> +GIT_BUILD_DIR=\"${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}\"\n> +if test -f \"$GIT_BUILD_DIR/GIT-BUILD-DIR\"\n> +then\n> +\tGIT_BUILD_DIR=\"$(cat \"$GIT_BUILD_DIR/GIT-BUILD-DIR\")\" || exit 1\n> +\t# On Windows, we must convert Windows paths lest they contain a colon\n> +\tcase \"$(uname -s)\" in\n> +\t*MINGW*)\n> +\t\tGIT_BUILD_DIR=\"$(cygpath -au \"$GIT_BUILD_DIR\")\"\n> +\t\t;;\n> +\tesac\n> +fi\n> +\n> +if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n> +then\n> +\techo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'\n> +\texit 1\n> +fi\n> +\n> +. \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n> +. \"$GIT_SOURCE_DIR\"/t/test-lib.sh\n> \n> unset GIT_CONFIG_NOSYSTEM\n> GIT_CONFIG_SYSTEM=\"$TEST_DIRECTORY/perf/config\"\n> @@ -324,7 +346,7 @@ test_at_end_hook_ () {\n> \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n> \t\t(\n> \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n> -\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n> +\t\t\t\"$PERL_PATH\" \"$GIT_SOURCE_DIR\"/t/perf/aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n> \t\t)\n> \tfi\n> }\n>\n> -- \n> 2.49.0.604.gff1f9ca942.dirty\n\nI'm not sure if it's related to this commit, but this patch series has\nbroken something:\n\n $ make && cd t/perf && ./run . master p0005-status.sh\n\n === Running 1 tests in this tree ===\n ok 1 - setup repo\n perf 2 - read-tree status br_ballast (4629): 1 ok\n # passed all 2 test(s)\n 1..2\n === Unpacking 485f5f863615e670fd97ae40af744e14072cfe18 in build/485f5f863615e670fd97ae40af744e14072cfe18 ===\n === Building 485f5f863615e670fd97ae40af744e14072cfe18 (master) ===\n GIT_VERSION=2.49.GIT\n * new build flags\n CC daemon.o\n * new link flags\n CC common-main.o\n CC abspath.o\n CC add-interactive.o\n CC add-patch.o\n [snip]\n CC t/unit-tests/unit-test.o\n CC t/unit-tests/lib-oid.o\n LINK t/unit-tests/bin/unit-tests\n GEN gitweb/gitweb.cgi\n GEN gitweb/static/gitweb.js\n === Running 1 tests in /home/toon/devel/git/t/perf/build/485f5f863615e670fd97ae40af744e14072cfe18/bin-wrappers ===\n ok 1 - setup repo\n perf 2 - read-tree status br_ballast (4629): 1 ok\n # passed all 2 test(s)\n 1..2\n cannot open test-results/p0005-status.subtests: No such file or directory at ./aggregate.perl line 159.\n\n--\nToon\n"},{"id":"515954","messageId":"87semgdyxb.fsf@iotcl.com","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-2-b2ace85616a3@pks.im","subject":"Re: [PATCH 2/5] t/perf: use configured PERL_PATH","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-04-10T11:43:44Z","receivedAt":"2025-04-10T11:44:00Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt writes:\n\n> Our benchmarks use a couple of Perl scripts to compute results. These\n> Perl scripts get executed directly, and as the shebang is hardcoded to\n> \"/usr/bin/perl\" this will fail on any system where the Perl interpreter\n> is located in a different path.\n>\n> Our build infrastructure already lets users configure the location of\n> Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file.\n> This file is being sourced by \"test-lib.sh\", and consequently we already\n> have the \"PERL_PATH\" variable available that contains its configured\n> location.\n>\n> Use \"PERL_PATH\" to execute Perl scripts, which makes them work on more\n> esoteric systems like NixOS.\n\nI see in `t/perf/README` there's a mention of running `./aggregate.perl`\ndirectly? Shall we inform the user to run that through their Perl as\nwell?\n\n- $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh\n+ $ perl ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh\n\nOr do we expect users to know what they are doing when they don't have\nPerl installed at /usr/bin/perl?\n\n--\nToon\n"},{"id":"515955","messageId":"87r020dyvc.fsf@iotcl.com","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-4-b2ace85616a3@pks.im","subject":"Re: [PATCH 4/5] meson: wire up benchmarks","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-04-10T11:44:55Z","receivedAt":"2025-04-10T11:45:14Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt writes:\n\n> Wire up benchmarks in Meson. The setup is mostly the same as how we wire\n> up our tests. The only difference is that benchmarks get wired up via\n> the `benchmark()` option instead of via `test()`, which gives them a bit\n> of special treatment:\n>\n> - Benchmarks never run in parallel.\n>\n> - Benchmarks aren't run by default when tests are executed.\n>\n> - Meson does not inject the `MALLOC_PERTURB` environment variable.\n>\n> Using benchmarks is quite simple:\n>\n> ```\n> $ meson setup build\n> # Run all benchmarks.\n> $ meson test -C build --benchmark\n> # Run a specific benchmark.\n> $ meson test -C build --benchmark p0000-*\n> ```\n\nI really appreciate the efforts, and I'd love to run the perf tests with\nMeson, but at the moment I don't know how these changes are useful\nbecause this doesn't print anything relevant:\n\n $ meson test -C build --benchmark 'p0005-*'\n ninja: Entering directory `/home/toon/devel/git/build'\n [1/28] Generating GIT-VERSION-FILE with a custom command (wrapped by meson to set env)\n 1/1 p0005-status OK 7.39s\n\n Ok: 1\n Expected Fail: 0\n Fail: 0\n Unexpected Pass: 0\n Skipped: 0\n Timeout: 0\n\n Full log written to /home/toon/devel/git/build/meson-logs/testlog.txt\n\n> Other than that the usual command line arguments accepted when running\n> tests are also accepted when running benchmarks.\n>\n> Note that the benchmarking target is somewhat limited because it will\n> only run benchmarks for the current build. Other usecases, like running\n> benchmarks against multiple different versions of Git, are not currently\n> supported. Users should continue to use \"t/perf/run\" for those usecases.\n> The script should get extended at one point in time to support Meson,\n> but this is outside of the scope of this series.\n\nYeah, this is unfortunate, but totally understandable. I've been digging\nin `t/perf/run` and `f/perf/aggregate.perl` and it doesn't look easy to\nadapt to Meson.\n\nBut I was wondering, instead of trying to fully integrate Meson into\nthose scripts, could we modify the scripts so they work with binaries\nbuilt by Meson? I mean, if we could run\n`cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would\nsimply run the benchmarks in those directories (without trying to check\nout code and build the sources). I think this would help a lot already.\n\n-- \nToon\n"},{"id":"516069","messageId":"87r01v5k9m.fsf@iotcl.com","threadId":"63221","inReplyTo":"87v7rcdzck.fsf@iotcl.com","subject":"Re: [PATCH 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-04-14T06:28:53Z","receivedAt":"2025-04-14T06:29:12Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Toon Claes writes:\n\n> I'm not sure if it's related to this commit, but this patch series has\n> broken something:\n>\n> $ make && cd t/perf && ./run . master p0005-status.sh\n>\n> === Running 1 tests in this tree ===\n> ok 1 - setup repo\n> perf 2 - read-tree status br_ballast (4629): 1 ok\n> # passed all 2 test(s)\n> 1..2\n> === Unpacking 485f5f863615e670fd97ae40af744e14072cfe18 in build/485f5f863615e670fd97ae40af744e14072cfe18 ===\n> === Building 485f5f863615e670fd97ae40af744e14072cfe18 (master) ===\n> GIT_VERSION=2.49.GIT\n> * new build flags\n> CC daemon.o\n> * new link flags\n> CC common-main.o\n> CC abspath.o\n> CC add-interactive.o\n> CC add-patch.o\n> [snip]\n> CC t/unit-tests/unit-test.o\n> CC t/unit-tests/lib-oid.o\n> LINK t/unit-tests/bin/unit-tests\n> GEN gitweb/gitweb.cgi\n> GEN gitweb/static/gitweb.js\n> === Running 1 tests in /home/toon/devel/git/t/perf/build/485f5f863615e670fd97ae40af744e14072cfe18/bin-wrappers ===\n> ok 1 - setup repo\n> perf 2 - read-tree status br_ballast (4629): 1 ok\n> # passed all 2 test(s)\n> 1..2\n> cannot open test-results/p0005-status.subtests: No such file or directory at ./aggregate.perl line 159.\n>\n> --\n> Toon\n\nEuhm, it seems I no longer can reproduce this issue. So you can ignore\nthis message. Sorry for the noice.\n\n--\nToon\n"},{"id":"516070","messageId":"Z_ywLLN5xj6cYFN9@pks.im","threadId":"63221","inReplyTo":"87semgdyxb.fsf@iotcl.com","subject":"Re: [PATCH 2/5] t/perf: use configured PERL_PATH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-14T06:50:20Z","receivedAt":"2025-04-14T06:50:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 10, 2025 at 01:43:44PM +0200, Toon Claes wrote:\n> Patrick Steinhardt writes:\n> \n> > Our benchmarks use a couple of Perl scripts to compute results. These\n> > Perl scripts get executed directly, and as the shebang is hardcoded to\n> > \"/usr/bin/perl\" this will fail on any system where the Perl interpreter\n> > is located in a different path.\n> >\n> > Our build infrastructure already lets users configure the location of\n> > Perl, which ultimately gets written into the GIT-BUILD-OPTIONS file.\n> > This file is being sourced by \"test-lib.sh\", and consequently we already\n> > have the \"PERL_PATH\" variable available that contains its configured\n> > location.\n> >\n> > Use \"PERL_PATH\" to execute Perl scripts, which makes them work on more\n> > esoteric systems like NixOS.\n> \n> I see in `t/perf/README` there's a mention of running `./aggregate.perl`\n> directly? Shall we inform the user to run that through their Perl as\n> well?\n> \n> - $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh\n> + $ perl ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh\n> \n> Or do we expect users to know what they are doing when they don't have\n> Perl installed at /usr/bin/perl?\n\nAnother solution could be to switch the shebang to `#!/usr/bin/env\nperl`. We also do this for \"t/chainlint.pl\".\n\nPatrick\n"},{"id":"516071","messageId":"Z_ywMpJ7PuOvQu2k@pks.im","threadId":"63221","inReplyTo":"87r020dyvc.fsf@iotcl.com","subject":"Re: [PATCH 4/5] meson: wire up benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-14T06:50:26Z","receivedAt":"2025-04-14T06:50:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 10, 2025 at 01:44:55PM +0200, Toon Claes wrote:\n> Patrick Steinhardt writes:\n> \n> > Wire up benchmarks in Meson. The setup is mostly the same as how we wire\n> > up our tests. The only difference is that benchmarks get wired up via\n> > the `benchmark()` option instead of via `test()`, which gives them a bit\n> > of special treatment:\n> >\n> > - Benchmarks never run in parallel.\n> >\n> > - Benchmarks aren't run by default when tests are executed.\n> >\n> > - Meson does not inject the `MALLOC_PERTURB` environment variable.\n> >\n> > Using benchmarks is quite simple:\n> >\n> > ```\n> > $ meson setup build\n> > # Run all benchmarks.\n> > $ meson test -C build --benchmark\n> > # Run a specific benchmark.\n> > $ meson test -C build --benchmark p0000-*\n> > ```\n> \n> I really appreciate the efforts, and I'd love to run the perf tests with\n> Meson, but at the moment I don't know how these changes are useful\n> because this doesn't print anything relevant:\n> \n> $ meson test -C build --benchmark 'p0005-*'\n> ninja: Entering directory `/home/toon/devel/git/build'\n> [1/28] Generating GIT-VERSION-FILE with a custom command (wrapped by meson to set env)\n> 1/1 p0005-status OK 7.39s\n> \n> Ok: 1\n> Expected Fail: 0\n> Fail: 0\n> Unexpected Pass: 0\n> Skipped: 0\n> Timeout: 0\n> \n> Full log written to /home/toon/devel/git/build/meson-logs/testlog.txt\n\nYou can execute with `meson test -i`, which will cause Meson to print\nthe benchmark's output.\n\n> > Other than that the usual command line arguments accepted when running\n> > tests are also accepted when running benchmarks.\n> >\n> > Note that the benchmarking target is somewhat limited because it will\n> > only run benchmarks for the current build. Other usecases, like running\n> > benchmarks against multiple different versions of Git, are not currently\n> > supported. Users should continue to use \"t/perf/run\" for those usecases.\n> > The script should get extended at one point in time to support Meson,\n> > but this is outside of the scope of this series.\n> \n> Yeah, this is unfortunate, but totally understandable. I've been digging\n> in `t/perf/run` and `f/perf/aggregate.perl` and it doesn't look easy to\n> adapt to Meson.\n> \n> But I was wondering, instead of trying to fully integrate Meson into\n> those scripts, could we modify the scripts so they work with binaries\n> built by Meson? I mean, if we could run\n> `cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would\n> simply run the benchmarks in those directories (without trying to check\n> out code and build the sources). I think this would help a lot already.\n\nThis is another thing we could (and eventually should) do, but it's\northogonal to this patch series from my point of view.\n\nPatrick\n"},{"id":"516072","messageId":"Z_yvMp6j091ZXzwC@pks.im","threadId":"63221","inReplyTo":"87r01v5k9m.fsf@iotcl.com","subject":"Re: [PATCH 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-14T06:50:29Z","receivedAt":"2025-04-14T06:50:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 14, 2025 at 08:28:53AM +0200, Toon Claes wrote:\n> Toon Claes writes:\n> \n> > I'm not sure if it's related to this commit, but this patch series has\n> > broken something:\n> >\n> > $ make && cd t/perf && ./run . master p0005-status.sh\n> >\n> > === Running 1 tests in this tree ===\n> > ok 1 - setup repo\n> > perf 2 - read-tree status br_ballast (4629): 1 ok\n> > # passed all 2 test(s)\n> > 1..2\n> > === Unpacking 485f5f863615e670fd97ae40af744e14072cfe18 in build/485f5f863615e670fd97ae40af744e14072cfe18 ===\n> > === Building 485f5f863615e670fd97ae40af744e14072cfe18 (master) ===\n> > GIT_VERSION=2.49.GIT\n> > * new build flags\n> > CC daemon.o\n> > * new link flags\n> > CC common-main.o\n> > CC abspath.o\n> > CC add-interactive.o\n> > CC add-patch.o\n> > [snip]\n> > CC t/unit-tests/unit-test.o\n> > CC t/unit-tests/lib-oid.o\n> > LINK t/unit-tests/bin/unit-tests\n> > GEN gitweb/gitweb.cgi\n> > GEN gitweb/static/gitweb.js\n> > === Running 1 tests in /home/toon/devel/git/t/perf/build/485f5f863615e670fd97ae40af744e14072cfe18/bin-wrappers ===\n> > ok 1 - setup repo\n> > perf 2 - read-tree status br_ballast (4629): 1 ok\n> > # passed all 2 test(s)\n> > 1..2\n> > cannot open test-results/p0005-status.subtests: No such file or directory at ./aggregate.perl line 159.\n> >\n> > --\n> > Toon\n> \n> Euhm, it seems I no longer can reproduce this issue. So you can ignore\n> this message. Sorry for the noice.\n\nOkay, great. I wasn't able to reproduce this locally, so thanks for\nclarifying!\n\nPatrick\n"},{"id":"516073","messageId":"20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","subject":"[PATCH v2 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-14T06:51:10Z","receivedAt":"2025-04-14T06:51:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis small patch series implements support for running our benchmarks in\n\"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\nwhich is more fully-featured and allows running benchmarks against\nmultiple different trees. Instead, this series only allows running the\nbenchmarks against the current tree. Users are thus expected to continue\nusing \"t/perf/run\" for more advanced usecases.\n\nChanges in v2:\n - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (5):\n t/perf: fix benchmarks with alternate repo formats\n t/perf: use configured PERL_PATH\n t/perf: fix benchmarks with out-of-tree builds\n meson: wire up benchmarks\n meson: wire up benchmarking options\n\n meson.build | 7 +++--\n meson_options.txt | 6 ++++\n t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++\n t/perf/aggregate.perl | 2 +-\n t/perf/perf-lib.sh | 32 ++++++++++++++++++---\n t/perf/run | 4 +--\n 6 files changed, 118 insertions(+), 10 deletions(-)\n\nRange-diff versus v1:\n\n1: fd734280c1d = 1: 844e8bdf81d t/perf: fix benchmarks with alternate repo formats\n2: 89a011eec63 ! 2: d950c828933 t/perf: use configured PERL_PATH\n @@ Commit message\n location.\n \n Use \"PERL_PATH\" to execute Perl scripts, which makes them work on more\n - esoteric systems like NixOS.\n + esoteric systems like NixOS. Furthermore, adapt the shebang to use\n + env(1) to execute Perl so that users who have Perl in PATH, but in a\n + non-standard location can execute the script directly.\n \n Signed-off-by: Patrick Steinhardt \n \n + ## t/perf/aggregate.perl ##\n +@@\n +-#!/usr/bin/perl\n ++#!/usr/bin/env perl\n + \n + use lib '../../perl/build/lib';\n + use strict;\n +\n ## t/perf/perf-lib.sh ##\n @@ t/perf/perf-lib.sh: test_perf_ () {\n \telse\n3: efecbd86cd6 = 3: dc1adf62a86 t/perf: fix benchmarks with out-of-tree builds\n4: feeaaae3ec1 = 4: 8528a775395 meson: wire up benchmarks\n5: fc594fc3dc5 = 5: c8a4d0b354e meson: wire up benchmarking options\n\n---\nbase-commit: 683c54c999c301c2cd6f715c411407c413b1d84e\nchange-id: 20250328-pks-meson-benchmarks-a8fac5f69467\n\n"},{"id":"516074","messageId":"20250414-pks-meson-benchmarks-v2-1-04377080a167@pks.im","threadId":"63221","inReplyTo":"20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im","subject":"[PATCH v2 1/5] t/perf: fix benchmarks with alternate repo formats","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-14T06:51:11Z","receivedAt":"2025-04-14T06:51:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Many of our benchmarks operate on a user-defined repository that we copy\nover before running the benchmarked logic. To keep unintentional side\neffects caused by on-disk state at bay we skip copying some files. This\nincludes for example hooks, but also the repo's configuration.\n\nIt is quite sensible to not copy over the configuration, as it is quite\neasy to inadvertently carry over configuration that may significantly\nimpact the performance measurements. But we cannot fully ignore the\nconfiguration either, as it may contain information about the repository\nformat. This will cause failures when for example using a repository\nwith SHA256 object format or the reftable ref format.\n\nFix the issue by parsing the reference and object formats from the\nsource repository and passing them to git-init(1).\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 8ab6d9c4694..1a9a51ca3cc 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -98,6 +98,8 @@ test_perf_create_repo_from () {\n \tsource_git=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-dir)\"\n \tobjects_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-path objects)\"\n \tcommon_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-common-dir)\"\n+\trefformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-ref-format)\"\n+\tobjectformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-object-format)\"\n \tmkdir -p \"$repo/.git\"\n \t(\n \t\tcd \"$source\" &&\n@@ -114,7 +116,7 @@ test_perf_create_repo_from () {\n \t) &&\n \t(\n \t\tcd \"$repo\" &&\n-\t\t\"$MODERN_GIT\" init -q &&\n+\t\t\"$MODERN_GIT\" init -q --ref-format=\"$refformat\" --object-format=\"$objectformat\" &&\n \t\ttest_perf_do_repo_symlink_config_ &&\n \t\tmv .git/hooks .git/hooks-disabled 2>/dev/null &&\n \t\tif test -f .git/index.lock\n\n-- \n2.49.0.805.g082f7c87e0.dirty\n\n"},{"id":"516075","messageId":"20250414-pks-meson-benchmarks-v2-2-04377080a167@pks.im","threadId":"63221","inReplyTo":"20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im","subject":"[PATCH v2 2/5] t/perf: use configured PERL_PATH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-14T06:51:12Z","receivedAt":"2025-04-14T06:51:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our benchmarks use a couple of Perl scripts to compute results. These\nPerl scripts get executed directly, and as the shebang is hardcoded to\n\"/usr/bin/perl\" this will fail on any system where the Perl interpreter\nis located in a different path.\n\nOur build infrastructure already lets users configure the location of\nPerl, which ultimately gets written into the GIT-BUILD-OPTIONS file.\nThis file is being sourced by \"test-lib.sh\", and consequently we already\nhave the \"PERL_PATH\" variable available that contains its configured\nlocation.\n\nUse \"PERL_PATH\" to execute Perl scripts, which makes them work on more\nesoteric systems like NixOS. Furthermore, adapt the shebang to use\nenv(1) to execute Perl so that users who have Perl in PATH, but in a\nnon-standard location can execute the script directly.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/aggregate.perl | 2 +-\n t/perf/perf-lib.sh | 4 ++--\n t/perf/run | 4 ++--\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex 575d2000cc1..1791c7528a9 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -1,4 +1,4 @@\n-#!/usr/bin/perl\n+#!/usr/bin/env perl\n \n use lib '../../perl/build/lib';\n use strict;\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 1a9a51ca3cc..4173eee4def 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -276,7 +276,7 @@ test_perf_ () {\n \telse\n \t\ttest_ok_ \"$1\"\n \tfi\n-\t\"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n \trm test_time.*\n }\n \n@@ -324,7 +324,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\ndiff --git a/t/perf/run b/t/perf/run\nindex 486ead21980..073bcb2afff 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -192,10 +192,10 @@ run_subsection () {\n \n \tif test -z \"$GIT_PERF_SEND_TO_CODESPEED\"\n \tthen\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n \telse\n \t\tjson_res_file=\"\"$TEST_RESULTS_DIR\"/$GIT_PERF_SUBSECTION/aggregate.json\"\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n \t\tsend_data_url=\"$GIT_PERF_SEND_TO_CODESPEED/result/add/json/\"\n \t\tcurl -v --request POST --data-urlencode \"json=$(cat \"$json_res_file\")\" \"$send_data_url\"\n \tfi\n\n-- \n2.49.0.805.g082f7c87e0.dirty\n\n"},{"id":"516076","messageId":"20250414-pks-meson-benchmarks-v2-3-04377080a167@pks.im","threadId":"63221","inReplyTo":"20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im","subject":"[PATCH v2 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-14T06:51:13Z","receivedAt":"2025-04-14T06:51:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"perf-lib.sh\" script is sourced by all of our benchmarking suites to\nmake available common infrastructure. The script assumes that build and\nsource directory are the same, which works for our Makefile. But the\nassumption breaks with both CMake and Meson, where the build directory\ncan be located in an arbitrary place.\n\nAdapt the script so that it works with out-of-tree builds. This prepares\nus for wiring up benchmarks in Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--\n 1 file changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 4173eee4def..5406557b7ca 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)\n TEST_NO_CREATE_REPO=t\n TEST_NO_MALLOC_CHECK=t\n \n-. ../test-lib.sh\n+# While test-lib.sh computes the build directory for us, we also have to do the\n+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first\n+# place.\n+GIT_BUILD_DIR=\"${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}\"\n+if test -f \"$GIT_BUILD_DIR/GIT-BUILD-DIR\"\n+then\n+\tGIT_BUILD_DIR=\"$(cat \"$GIT_BUILD_DIR/GIT-BUILD-DIR\")\" || exit 1\n+\t# On Windows, we must convert Windows paths lest they contain a colon\n+\tcase \"$(uname -s)\" in\n+\t*MINGW*)\n+\t\tGIT_BUILD_DIR=\"$(cygpath -au \"$GIT_BUILD_DIR\")\"\n+\t\t;;\n+\tesac\n+fi\n+\n+if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+then\n+\techo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'\n+\texit 1\n+fi\n+\n+. \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+. \"$GIT_SOURCE_DIR\"/t/test-lib.sh\n \n unset GIT_CONFIG_NOSYSTEM\n GIT_CONFIG_SYSTEM=\"$TEST_DIRECTORY/perf/config\"\n@@ -324,7 +346,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" \"$GIT_SOURCE_DIR\"/t/perf/aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\n\n-- \n2.49.0.805.g082f7c87e0.dirty\n\n"},{"id":"516077","messageId":"20250414-pks-meson-benchmarks-v2-4-04377080a167@pks.im","threadId":"63221","inReplyTo":"20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im","subject":"[PATCH v2 4/5] meson: wire up benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-14T06:51:14Z","receivedAt":"2025-04-14T06:51:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up benchmarks in Meson. The setup is mostly the same as how we wire\nup our tests. The only difference is that benchmarks get wired up via\nthe `benchmark()` option instead of via `test()`, which gives them a bit\nof special treatment:\n\n - Benchmarks never run in parallel.\n\n - Benchmarks aren't run by default when tests are executed.\n\n - Meson does not inject the `MALLOC_PERTURB` environment variable.\n\nUsing benchmarks is quite simple:\n\n ```\n $ meson setup build\n # Run all benchmarks.\n $ meson test -C build --benchmark\n # Run a specific benchmark.\n $ meson test -C build --benchmark p0000-*\n ```\n\nOther than that the usual command line arguments accepted when running\ntests are also accepted when running benchmarks.\n\nNote that the benchmarking target is somewhat limited because it will\nonly run benchmarks for the current build. Other usecases, like running\nbenchmarks against multiple different versions of Git, are not currently\nsupported. Users should continue to use \"t/perf/run\" for those usecases.\nThe script should get extended at one point in time to support Meson,\nbut this is outside of the scope of this series.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 1 +\n t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 78 insertions(+)\n\ndiff --git a/meson.build b/meson.build\nindex efe2871c9db..9af4a10188d 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -204,6 +204,7 @@ git = find_program('git', dirs: program_path, required: false)\n sed = find_program('sed', dirs: program_path)\n shell = find_program('sh', dirs: program_path)\n tar = find_program('tar', dirs: program_path)\n+time = find_program('time', dirs: program_path, required: false)\n \n # Sanity-check that programs required for the build exist.\n foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']\ndiff --git a/t/meson.build b/t/meson.build\nindex a59da26be3f..dba327fd7ec 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -1096,11 +1096,71 @@ integration_tests = [\n 't9903-bash-prompt.sh',\n ]\n \n+benchmarks = [\n+ 'perf/p0000-perf-lib-sanity.sh',\n+ 'perf/p0001-rev-list.sh',\n+ 'perf/p0002-read-cache.sh',\n+ 'perf/p0003-delta-base-cache.sh',\n+ 'perf/p0004-lazy-init-name-hash.sh',\n+ 'perf/p0005-status.sh',\n+ 'perf/p0006-read-tree-checkout.sh',\n+ 'perf/p0007-write-cache.sh',\n+ 'perf/p0008-odb-fsync.sh',\n+ 'perf/p0071-sort.sh',\n+ 'perf/p0090-cache-tree.sh',\n+ 'perf/p0100-globbing.sh',\n+ 'perf/p1006-cat-file.sh',\n+ 'perf/p1400-update-ref.sh',\n+ 'perf/p1450-fsck.sh',\n+ 'perf/p1451-fsck-skip-list.sh',\n+ 'perf/p1500-graph-walks.sh',\n+ 'perf/p2000-sparse-operations.sh',\n+ 'perf/p3400-rebase.sh',\n+ 'perf/p3404-rebase-interactive.sh',\n+ 'perf/p4000-diff-algorithms.sh',\n+ 'perf/p4001-diff-no-index.sh',\n+ 'perf/p4002-diff-color-moved.sh',\n+ 'perf/p4205-log-pretty-formats.sh',\n+ 'perf/p4209-pickaxe.sh',\n+ 'perf/p4211-line-log.sh',\n+ 'perf/p4220-log-grep-engines.sh',\n+ 'perf/p4221-log-grep-engines-fixed.sh',\n+ 'perf/p5302-pack-index.sh',\n+ 'perf/p5303-many-packs.sh',\n+ 'perf/p5304-prune.sh',\n+ 'perf/p5310-pack-bitmaps.sh',\n+ 'perf/p5311-pack-bitmaps-fetch.sh',\n+ 'perf/p5312-pack-bitmaps-revs.sh',\n+ 'perf/p5313-pack-objects.sh',\n+ 'perf/p5314-name-hash.sh',\n+ 'perf/p5326-multi-pack-bitmaps.sh',\n+ 'perf/p5332-multi-pack-reuse.sh',\n+ 'perf/p5333-pseudo-merge-bitmaps.sh',\n+ 'perf/p5550-fetch-tags.sh',\n+ 'perf/p5551-fetch-rescan.sh',\n+ 'perf/p5600-partial-clone.sh',\n+ 'perf/p5601-clone-reference.sh',\n+ 'perf/p6100-describe.sh',\n+ 'perf/p6300-for-each-ref.sh',\n+ 'perf/p7000-filter-branch.sh',\n+ 'perf/p7102-reset.sh',\n+ 'perf/p7300-clean.sh',\n+ 'perf/p7519-fsmonitor.sh',\n+ 'perf/p7527-builtin-fsmonitor.sh',\n+ 'perf/p7810-grep.sh',\n+ 'perf/p7820-grep-engines.sh',\n+ 'perf/p7821-grep-engines-fixed.sh',\n+ 'perf/p7822-grep-perl-character.sh',\n+ 'perf/p9210-scalar.sh',\n+ 'perf/p9300-fast-import-export.sh',\n+]\n+\n # Sanity check that we are not missing any tests present in 't/'. This check\n # only runs once at configure time and is thus best-effort, only. It is\n # sufficient to catch missing test suites in our CI though.\n foreach glob, tests : {\n 't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,\n+ 'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,\n 'unit-tests/t-*.c': unit_test_programs,\n 'unit-tests/u-*.c': clar_test_suites,\n }\n@@ -1152,3 +1212,20 @@ foreach integration_test : integration_tests\n timeout: 0,\n )\n endforeach\n+\n+if time.found()\n+ benchmark_environment = test_environment\n+ benchmark_environment.set('GTIME', time.full_path())\n+\n+ foreach benchmark : benchmarks\n+ benchmark(fs.stem(benchmark), shell,\n+ args: [\n+ fs.name(benchmark),\n+ ],\n+ workdir: meson.current_source_dir() / 'perf',\n+ env: benchmark_environment,\n+ depends: test_dependencies + bin_wrappers,\n+ timeout: 0,\n+ )\n+ endforeach\n+endif\n\n-- \n2.49.0.805.g082f7c87e0.dirty\n\n"},{"id":"516078","messageId":"20250414-pks-meson-benchmarks-v2-5-04377080a167@pks.im","threadId":"63221","inReplyTo":"20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im","subject":"[PATCH v2 5/5] meson: wire up benchmarking options","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-14T06:51:15Z","receivedAt":"2025-04-14T06:51:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up a couple of benchmarking options that we end up writing into our\n\"GIT-BUILD-OPTIONS\" file. These options allow users to control how\nexactly benchmarks are executed.\n\nNote that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are\nexposed as a build option. Those options are used by \"t/perf/run\", which\nis not used by Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 6 +++---\n meson_options.txt | 6 ++++++\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/meson.build b/meson.build\nindex 9af4a10188d..ba128ab0759 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -660,11 +660,11 @@ builtin_sources += custom_target(\n # build options to our tests.\n build_options_config = configuration_data()\n build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_LARGE_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))\n build_options_config.set('GIT_PERF_MAKE_COMMAND', '')\n build_options_config.set('GIT_PERF_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')\n-build_options_config.set('GIT_PERF_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())\n+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))\n build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')\n build_options_config.set('GIT_TEST_INDEX_VERSION', '')\n build_options_config.set('GIT_TEST_OPTS', '')\ndiff --git a/meson_options.txt b/meson_options.txt\nindex 78d172a7401..ca106c43b2e 100644\n--- a/meson_options.txt\n+++ b/meson_options.txt\n@@ -101,6 +101,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto\n description: 'Which backend to use to generate documentation.')\n \n # Testing.\n+option('benchmark_repo', type: 'string', value: '',\n+ description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')\n+option('benchmark_large_repo', type: 'string', value: '',\n+ description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')\n+option('benchmark_repeat_count', type: 'integer', value: 3,\n+ description: 'Number of times a test should be repeated for best-of-N measurements.')\n option('coccinelle', type: 'feature', value: 'auto',\n description: 'Provide a coccicheck target that generates a Coccinelle patch.')\n option('tests', type: 'boolean', value: true,\n\n-- \n2.49.0.805.g082f7c87e0.dirty\n\n"},{"id":"516088","messageId":"87frib5cwi.fsf@iotcl.com","threadId":"63221","inReplyTo":"Z_ywMpJ7PuOvQu2k@pks.im","subject":"Re: [PATCH 4/5] meson: wire up benchmarks","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-04-14T09:07:57Z","receivedAt":"2025-04-14T09:08:10Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt writes:\n\n> You can execute with `meson test -i`, which will cause Meson to print\n> the benchmark's output.\n\n(annoyed grunt) I didn't think about using `-i` here, probably because\nit wasn't used in the example in the commit message.\n\n>> But I was wondering, instead of trying to fully integrate Meson into\n>> those scripts, could we modify the scripts so they work with binaries\n>> built by Meson? I mean, if we could run\n>> `cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would\n>> simply run the benchmarks in those directories (without trying to check\n>> out code and build the sources). I think this would help a lot already.\n>\n> This is another thing we could (and eventually should) do, but it's\n> orthogonal to this patch series from my point of view.\n\nThat's true. This series already enables me to use benchmarks with Meson\nand is definetely a decent step in the right direction. Let's get this\nseries merged and we can work on better comparative benchmarks later.\n\nI just reviewed v2 and I approve on this series.\n\n-- \nToon\n"},{"id":"516121","messageId":"xmqq5xj6r1mf.fsf@gitster.g","threadId":"63221","inReplyTo":"Z_ywLLN5xj6cYFN9@pks.im","subject":"Re: [PATCH 2/5] t/perf: use configured PERL_PATH","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-14T19:20:40Z","receivedAt":"2025-04-14T19:20:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> > Use \"PERL_PATH\" to execute Perl scripts, which makes them work on more\n>> > esoteric systems like NixOS.\n>> \n>> I see in `t/perf/README` there's a mention of running `./aggregate.perl`\n>> directly? Shall we inform the user to run that through their Perl as\n>> well?\n>> \n>> - $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh\n>> + $ perl ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh\n\nGood.\n\n>> \n>> Or do we expect users to know what they are doing when they don't have\n>> Perl installed at /usr/bin/perl?\n>\n> Another solution could be to switch the shebang to `#!/usr/bin/env\n> perl`. We also do this for \"t/chainlint.pl\".\n\nWhen you do not have perl installed anywhere, how does this fail? I\nthink you would get\n\n $ ./aggregate.perl ...\n /usr/bin/env: 'perl': No such file or directory\n\nand compared to that, \n\n $ perl ./aggregate.perl ...\n bash: perl: command not found\n\nI think it makes it slightly more obvious to those who lack perl on\ntheir $PATH what is going wrong to explicitly tell them to run\n\"perl\" like Toon's suggestion above (primarily because use of\n/usr/bin/env is not obvious to those who are told to run\n./aggregate.perl script).\n\n"},{"id":"516184","messageId":"Z_4ugEF1iKC5U7Ry@pks.im","threadId":"63221","inReplyTo":"xmqq5xj6r1mf.fsf@gitster.g","subject":"Re: [PATCH 2/5] t/perf: use configured PERL_PATH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-15T10:01:36Z","receivedAt":"2025-04-15T10:01:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 14, 2025 at 12:20:40PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> > Use \"PERL_PATH\" to execute Perl scripts, which makes them work on more\n> >> > esoteric systems like NixOS.\n> >> \n> >> I see in `t/perf/README` there's a mention of running `./aggregate.perl`\n> >> directly? Shall we inform the user to run that through their Perl as\n> >> well?\n> >> \n> >> - $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh\n> >> + $ perl ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh\n> \n> Good.\n> \n> >> \n> >> Or do we expect users to know what they are doing when they don't have\n> >> Perl installed at /usr/bin/perl?\n> >\n> > Another solution could be to switch the shebang to `#!/usr/bin/env\n> > perl`. We also do this for \"t/chainlint.pl\".\n> \n> When you do not have perl installed anywhere, how does this fail? I\n> think you would get\n> \n> $ ./aggregate.perl ...\n> /usr/bin/env: 'perl': No such file or directory\n> \n> and compared to that, \n> \n> $ perl ./aggregate.perl ...\n> bash: perl: command not found\n> \n> I think it makes it slightly more obvious to those who lack perl on\n> their $PATH what is going wrong to explicitly tell them to run\n> \"perl\" like Toon's suggestion above (primarily because use of\n> /usr/bin/env is not obvious to those who are told to run\n> ./aggregate.perl script).\n\nThe counterargument is that `#!/usr/bin/env perl` works on systems where\nPerl is in PATH, but in a non-standard location. So isn't it preferable\nto make things Just Work(TM) on such systems instead of giving slightly\nbetter error messages on systems where it wouldn't work anyway because\nPerl is not available? I certainly think so, but if you or others feel\nstrongly I'll adapt.\n\nPatrick\n"},{"id":"516198","messageId":"xmqqplhdo5ji.fsf@gitster.g","threadId":"63221","inReplyTo":"20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im","subject":"Re: [PATCH v2 0/5] meson: wire up support for benchmarks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-15T14:36:33Z","receivedAt":"2025-04-15T14:36:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> this small patch series implements support for running our benchmarks in\n> \"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\n> which is more fully-featured and allows running benchmarks against\n> multiple different trees. Instead, this series only allows running the\n> benchmarks against the current tree. Users are thus expected to continue\n> using \"t/perf/run\" for more advanced usecases.\n>\n> Changes in v2:\n> - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n> - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n>\n> Thanks!\n\n\nThe previous iteration of this series has been kept out of 'seen'\nfor some time and I didn't recall why I did so. With this iteration\nmerged, all GitHub Actions CI tasks with \"meson\" in the name are\nfailing, so does \"documentation\" jobs (which recently acquired\n\"let's make sure meson-based build does the docs fine\" substep).\n\nCan you help seeing where the merge went wrong (yes, I am suspecting\nthat there is some stupid merge mistake there)?\n\nThanks.\n"},{"id":"516214","messageId":"xmqq34e9mgot.fsf@gitster.g","threadId":"63221","inReplyTo":"xmqqplhdo5ji.fsf@gitster.g","subject":"Re: [PATCH v2 0/5] meson: wire up support for benchmarks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-15T18:18:42Z","receivedAt":"2025-04-15T18:18:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Patrick Steinhardt writes:\n>\n>> this small patch series implements support for running our benchmarks in\n>> \"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\n>> which is more fully-featured and allows running benchmarks against\n>> multiple different trees. Instead, this series only allows running the\n>> benchmarks against the current tree. Users are thus expected to continue\n>> using \"t/perf/run\" for more advanced usecases.\n>>\n>> Changes in v2:\n>> - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n>> - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n>>\n>> Thanks!\n>\n>\n> The previous iteration of this series has been kept out of 'seen'\n> for some time and I didn't recall why I did so. With this iteration\n> merged, all GitHub Actions CI tasks with \"meson\" in the name are\n> failing, so does \"documentation\" jobs (which recently acquired\n> \"let's make sure meson-based build does the docs fine\" substep).\n>\n> Can you help seeing where the merge went wrong (yes, I am suspecting\n> that there is some stupid merge mistake there)?\n\n\nJust a few test CI runs.\n\n https://github.com/git/git/actions/runs/14457387669\n\nis with this topic mergecd in (with alleged mismerge).\n\nThis one\n\n https://github.com/git/git/actions/runs/14406901394\n\nis from last week without the earlier iteration of this topic, which\nfails a few meson jobs.\n\nlinux-meson job that fails with\n\nmeson.build:689:19: ERROR: Command `/usr/bin/git -C /__w/git/git ls-files --deduplicate '*.h' ':!contrib' ':!compat/inet_ntop.c' ':!compat/inet_pton.c' ':!compat/nedmalloc' ':!compat/obstack.*' ':!compat/poll' ':!compat/regex' ':!sha1collisiondetection' ':!sha1dc' ':!t/unit-tests/clar' ':!t/unit-tests/clar' ':!t/t[0-9][0-9][0-9][0-9]*'` failed with status 128.\n\nand\n\nwin+Meson test(3) that dies inside Python asyncio both look\nproblematic.\n\nThanks.\n"},{"id":"516277","messageId":"Z_-Nu27GeW0o2E0f@pks.im","threadId":"63221","inReplyTo":"xmqq34e9mgot.fsf@gitster.g","subject":"Re: [PATCH v2 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-16T11:00:11Z","receivedAt":"2025-04-16T11:00:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 15, 2025 at 11:18:42AM -0700, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > Patrick Steinhardt writes:\n> >\n> >> this small patch series implements support for running our benchmarks in\n> >> \"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\n> >> which is more fully-featured and allows running benchmarks against\n> >> multiple different trees. Instead, this series only allows running the\n> >> benchmarks against the current tree. Users are thus expected to continue\n> >> using \"t/perf/run\" for more advanced usecases.\n> >>\n> >> Changes in v2:\n> >> - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n> >> - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n> >>\n> >> Thanks!\n> >\n> >\n> > The previous iteration of this series has been kept out of 'seen'\n> > for some time and I didn't recall why I did so. With this iteration\n> > merged, all GitHub Actions CI tasks with \"meson\" in the name are\n> > failing, so does \"documentation\" jobs (which recently acquired\n> > \"let's make sure meson-based build does the docs fine\" substep).\n> >\n> > Can you help seeing where the merge went wrong (yes, I am suspecting\n> > that there is some stupid merge mistake there)?\n> \n> \n> Just a few test CI runs.\n> \n> https://github.com/git/git/actions/runs/14457387669\n> \n> is with this topic mergecd in (with alleged mismerge).\n> \n> This one\n> \n> https://github.com/git/git/actions/runs/14406901394\n> \n> is from last week without the earlier iteration of this topic, which\n> fails a few meson jobs.\n> \n> linux-meson job that fails with\n> \n> meson.build:689:19: ERROR: Command `/usr/bin/git -C /__w/git/git ls-files --deduplicate '*.h' ':!contrib' ':!compat/inet_ntop.c' ':!compat/inet_pton.c' ':!compat/nedmalloc' ':!compat/obstack.*' ':!compat/poll' ':!compat/regex' ':!sha1collisiondetection' ':!sha1dc' ':!t/unit-tests/clar' ':!t/unit-tests/clar' ':!t/t[0-9][0-9][0-9][0-9]*'` failed with status 128.\n\nHm, curious. These lines have been moved here by kn/meson-hdr-check, but\nthe logic already existed beforehand. I've Cc'd Karthik -- it would\nprobably make sense to investigate (and ideally also fix) the issue as\npart of that series.\n\n> and\n> \n> win+Meson test(3) that dies inside Python asyncio both look\n> problematic.\n\nYeah, this is a result of cancellation of the job. Johannes already\nmentioned this issue to me in the past, and it's known that Python's\nasyncio is a bit wonky in Windows. The problem here is mostly that\nWindows does not have proper signals.\n\nIt's not a huge problem given that this only happens when the job is\nbeing cancelled, but it's not great, either. I've already invested some\ntime into improving this on Windows [1], but there are a couple of\nuncertainties still. Anyway, this should hopefully be fixed soonish.\n\nPatrick\n\n[1]: https://github.com/mesonbuild/meson/pull/14311\n"},{"id":"516387","messageId":"xmqqcyd99ipm.fsf@gitster.g","threadId":"63221","inReplyTo":"Z_-Nu27GeW0o2E0f@pks.im","subject":"Re: [PATCH v2 0/5] meson: wire up support for benchmarks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-18T23:02:29Z","receivedAt":"2025-04-18T23:02:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> This one\n>> \n>> https://github.com/git/git/actions/runs/14406901394\n>> \n>> is from last week without the earlier iteration of this topic, which\n>> fails a few meson jobs.\n>> \n>> linux-meson job that fails with\n>> \n>> meson.build:689:19: ERROR: Command `/usr/bin/git -C /__w/git/git ls-files --deduplicate '*.h' ':!contrib' ':!compat/inet_ntop.c' ':!compat/inet_pton.c' ':!compat/nedmalloc' ':!compat/obstack.*' ':!compat/poll' ':!compat/regex' ':!sha1collisiondetection' ':!sha1dc' ':!t/unit-tests/clar' ':!t/unit-tests/clar' ':!t/t[0-9][0-9][0-9][0-9]*'` failed with status 128.\n>\n> Hm, curious. These lines have been moved here by kn/meson-hdr-check, but\n> the logic already existed beforehand. I've Cc'd Karthik -- it would\n> probably make sense to investigate (and ideally also fix) the issue as\n> part of that series.\n\nIndeed. I tentatively kicked meson-hdr-check topic out of 'seen'\nand the CI seems to be a lot happier. What is curious is that in my\nlocal environment 'seen' with meson-hdr-check topic did not fail for\nme.\n\n"},{"id":"516398","messageId":"CAP8UFD3HcAVs351sabU2qd70PmMzTtGbS7P7eKYyDyUQZ4nQ2Q@mail.gmail.com","threadId":"63221","inReplyTo":"20250414-pks-meson-benchmarks-v2-3-04377080a167@pks.im","subject":"Re: [PATCH v2 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-04-20T10:00:08Z","receivedAt":"2025-04-20T10:00:22Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Apr 14, 2025 at 8:51 AM Patrick Steinhardt wrote:\n\n> --- a/t/perf/perf-lib.sh\n> +++ b/t/perf/perf-lib.sh\n> @@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)\n> TEST_NO_CREATE_REPO=t\n> TEST_NO_MALLOC_CHECK=t\n>\n> -. ../test-lib.sh\n> +# While test-lib.sh computes the build directory for us, we also have to do the\n> +# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first\n> +# place.\n> +GIT_BUILD_DIR=\"${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}\"\n\nRight now on 'master' there is:\n\nGIT_BUILD_DIR=\"${GIT_BUILD_DIR:-${TEST_DIRECTORY%/t}}\"\nif test \"$TEST_DIRECTORY\" = \"$GIT_BUILD_DIR\"\nthen\n echo \"PANIC: Running in a $TEST_DIRECTORY that doesn't end in '/t'?\" >&2\n exit 1\nfi\n\nso it's not exactly the same thing, even if it still probably works well.\n\nFuture readers might wonder if this discrepancy results from changes\nthat were made to only one of the files or if we really wanted to get\nrid of the \"/t\" check here. In case we do want to get rid of the \"/t\"\ncheck, I think it might be worth saying it clearly in the comment.\n\n> +if test -f \"$GIT_BUILD_DIR/GIT-BUILD-DIR\"\n> +then\n> + GIT_BUILD_DIR=\"$(cat \"$GIT_BUILD_DIR/GIT-BUILD-DIR\")\" || exit 1\n> + # On Windows, we must convert Windows paths lest they contain a colon\n> + case \"$(uname -s)\" in\n> + *MINGW*)\n> + GIT_BUILD_DIR=\"$(cygpath -au \"$GIT_BUILD_DIR\")\"\n> + ;;\n> + esac\n> +fi\n> +\n> +if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n> +then\n> + echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'\n\nMaybe something like the following could help debug this:\n\n echo >&2 \"error: GIT-BUILD-OPTIONS file missing from '$GIT_BUILD_DIR'\"\n echo >&2 'error: (has Git been built?).'\n\n> + exit 1\n> +fi\n"},{"id":"516399","messageId":"CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com","threadId":"63221","inReplyTo":"20250414-pks-meson-benchmarks-v2-4-04377080a167@pks.im","subject":"Re: [PATCH v2 4/5] meson: wire up benchmarks","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-04-20T10:00:25Z","receivedAt":"2025-04-20T10:00:39Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Apr 14, 2025 at 8:51 AM Patrick Steinhardt wrote:\n>\n> Wire up benchmarks in Meson. The setup is mostly the same as how we wire\n> up our tests. The only difference is that benchmarks get wired up via\n> the `benchmark()` option instead of via `test()`, which gives them a bit\n> of special treatment:\n>\n> - Benchmarks never run in parallel.\n>\n> - Benchmarks aren't run by default when tests are executed.\n>\n> - Meson does not inject the `MALLOC_PERTURB` environment variable.\n>\n> Using benchmarks is quite simple:\n>\n> ```\n> $ meson setup build\n> # Run all benchmarks.\n> $ meson test -C build --benchmark\n> # Run a specific benchmark.\n> $ meson test -C build --benchmark p0000-*\n> ```\n\nInstead of just here, I would have expected that something like this\nwould appear towards the top of the top level meson.build file, maybe\nin a \"Benchmarking\" section, where there is the rest of the meson\ndocumentation for Git.\n\n> Other than that the usual command line arguments accepted when running\n> tests are also accepted when running benchmarks.\n>\n> Note that the benchmarking target is somewhat limited because it will\n> only run benchmarks for the current build. Other usecases, like running\n> benchmarks against multiple different versions of Git, are not currently\n> supported. Users should continue to use \"t/perf/run\" for those usecases.\n\ns/usecases/use cases/\n\n> The script should get extended at one point in time to support Meson,\n> but this is outside of the scope of this series.\n\nFair enough.\n\n> +if time.found()\n> + benchmark_environment = test_environment\n> + benchmark_environment.set('GTIME', time.full_path())\n> +\n> + foreach benchmark : benchmarks\n> + benchmark(fs.stem(benchmark), shell,\n> + args: [\n> + fs.name(benchmark),\n> + ],\n> + workdir: meson.current_source_dir() / 'perf',\n> + env: benchmark_environment,\n> + depends: test_dependencies + bin_wrappers,\n> + timeout: 0,\n> + )\n> + endforeach\n> +endif\n\nI wonder what happens when we run `meson test -C build --benchmark`\nbut 'time' is not found.\n"},{"id":"516452","messageId":"20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","subject":"[PATCH v3 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-22T06:50:10Z","receivedAt":"2025-04-22T06:50:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis small patch series implements support for running our benchmarks in\n\"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\nwhich is more fully-featured and allows running benchmarks against\nmultiple different trees. Instead, this series only allows running the\nbenchmarks against the current tree. Users are thus expected to continue\nusing \"t/perf/run\" for more advanced usecases.\n\nChanges in v2:\n - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n\nChanges in v3:\n - Document how to run benchmarks in \"meson.build\".\n - Expand the message for the commit that enables out-of-tree\n benchmarking.\n - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (5):\n t/perf: fix benchmarks with alternate repo formats\n t/perf: use configured PERL_PATH\n t/perf: fix benchmarks with out-of-tree builds\n meson: wire up benchmarks\n meson: wire up benchmarking options\n\n meson.build | 13 +++++++--\n meson_options.txt | 6 ++++\n t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++\n t/perf/aggregate.perl | 2 +-\n t/perf/perf-lib.sh | 32 ++++++++++++++++++---\n t/perf/run | 4 +--\n 6 files changed, 124 insertions(+), 10 deletions(-)\n\nRange-diff versus v2:\n\n1: dad3c5a3e78 = 1: 85098f35716 t/perf: fix benchmarks with alternate repo formats\n2: 42284091b9c = 2: 457ea957acf t/perf: use configured PERL_PATH\n3: d03ca0c0630 ! 3: a66bc5aaf51 t/perf: fix benchmarks with out-of-tree builds\n @@ Commit message\n assumption breaks with both CMake and Meson, where the build directory\n can be located in an arbitrary place.\n \n - Adapt the script so that it works with out-of-tree builds. This prepares\n - us for wiring up benchmarks in Meson.\n + Adapt the script so that it works with out-of-tree builds. Most\n + importantly, this requires us to figure out the location of the build\n + directory:\n +\n + - When running benchmarks via our Makefile the build directory is the\n + same as the source directory. We already know to derive the test\n + directory (\"t/\") via `$(pwd)/..`, which works because we chdir into\n + \"t/perf\" before executing benchmarks. We can thus derive the build\n + directory by appending another \"/..\" to that path.\n +\n + - When running benchmarks via Meson the build directory is located at\n + an arbitrary location. The build system thus has to make the path\n + known by exporting the `GIT_BUILD_DIR` environment variable.\n +\n + This change prepares us for wiring up benchmarks in Meson.\n \n Signed-off-by: Patrick Steinhardt \n \n4: 14945278066 ! 4: 386b960f3e1 meson: wire up benchmarks\n @@ Commit message\n tests are also accepted when running benchmarks.\n \n Note that the benchmarking target is somewhat limited because it will\n - only run benchmarks for the current build. Other usecases, like running\n + only run benchmarks for the current build. Other use cases, like running\n benchmarks against multiple different versions of Git, are not currently\n - supported. Users should continue to use \"t/perf/run\" for those usecases.\n - The script should get extended at one point in time to support Meson,\n - but this is outside of the scope of this series.\n + supported. Users should continue to use \"t/perf/run\" for those use\n + cases. The script should get extended at one point in time to support\n + Meson, but this is outside of the scope of this series.\n \n Signed-off-by: Patrick Steinhardt \n \n ## meson.build ##\n +@@\n + # # Execute single test interactively such that features like `debug ()` work.\n + # $ meson test -i --test-args='-ix' t1400-update-ref\n + #\n ++# # Execute all benchmarks.\n ++# $ meson test -i --benchmark\n ++#\n ++# # Execute single benchmark.\n ++# $ meson test -i --benchmark p0000-*\n ++#\n + # Test execution is parallelized by default and scales with the number of\n + # processor cores available. You can change the number of processes by passing\n + # the `-jN` flag to `meson test`.\n @@ meson.build: git = find_program('git', dirs: program_path, required: false)\n sed = find_program('sed', dirs: program_path)\n shell = find_program('sh', dirs: program_path)\n5: 101be1e8efc = 5: cc3e771ada0 meson: wire up benchmarking options\n\n---\nbase-commit: 683c54c999c301c2cd6f715c411407c413b1d84e\nchange-id: 20250328-pks-meson-benchmarks-a8fac5f69467\n\n"},{"id":"516453","messageId":"20250422-pks-meson-benchmarks-v3-2-7aad68bac6fd@pks.im","threadId":"63221","inReplyTo":"20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im","subject":"[PATCH v3 2/5] t/perf: use configured PERL_PATH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-22T06:50:12Z","receivedAt":"2025-04-22T06:50:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our benchmarks use a couple of Perl scripts to compute results. These\nPerl scripts get executed directly, and as the shebang is hardcoded to\n\"/usr/bin/perl\" this will fail on any system where the Perl interpreter\nis located in a different path.\n\nOur build infrastructure already lets users configure the location of\nPerl, which ultimately gets written into the GIT-BUILD-OPTIONS file.\nThis file is being sourced by \"test-lib.sh\", and consequently we already\nhave the \"PERL_PATH\" variable available that contains its configured\nlocation.\n\nUse \"PERL_PATH\" to execute Perl scripts, which makes them work on more\nesoteric systems like NixOS. Furthermore, adapt the shebang to use\nenv(1) to execute Perl so that users who have Perl in PATH, but in a\nnon-standard location can execute the script directly.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/aggregate.perl | 2 +-\n t/perf/perf-lib.sh | 4 ++--\n t/perf/run | 4 ++--\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex 575d2000cc1..1791c7528a9 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -1,4 +1,4 @@\n-#!/usr/bin/perl\n+#!/usr/bin/env perl\n \n use lib '../../perl/build/lib';\n use strict;\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 1a9a51ca3cc..4173eee4def 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -276,7 +276,7 @@ test_perf_ () {\n \telse\n \t\ttest_ok_ \"$1\"\n \tfi\n-\t\"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n \trm test_time.*\n }\n \n@@ -324,7 +324,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\ndiff --git a/t/perf/run b/t/perf/run\nindex 486ead21980..073bcb2afff 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -192,10 +192,10 @@ run_subsection () {\n \n \tif test -z \"$GIT_PERF_SEND_TO_CODESPEED\"\n \tthen\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n \telse\n \t\tjson_res_file=\"\"$TEST_RESULTS_DIR\"/$GIT_PERF_SUBSECTION/aggregate.json\"\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n \t\tsend_data_url=\"$GIT_PERF_SEND_TO_CODESPEED/result/add/json/\"\n \t\tcurl -v --request POST --data-urlencode \"json=$(cat \"$json_res_file\")\" \"$send_data_url\"\n \tfi\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516454","messageId":"20250422-pks-meson-benchmarks-v3-1-7aad68bac6fd@pks.im","threadId":"63221","inReplyTo":"20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im","subject":"[PATCH v3 1/5] t/perf: fix benchmarks with alternate repo formats","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-22T06:50:11Z","receivedAt":"2025-04-22T06:50:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Many of our benchmarks operate on a user-defined repository that we copy\nover before running the benchmarked logic. To keep unintentional side\neffects caused by on-disk state at bay we skip copying some files. This\nincludes for example hooks, but also the repo's configuration.\n\nIt is quite sensible to not copy over the configuration, as it is quite\neasy to inadvertently carry over configuration that may significantly\nimpact the performance measurements. But we cannot fully ignore the\nconfiguration either, as it may contain information about the repository\nformat. This will cause failures when for example using a repository\nwith SHA256 object format or the reftable ref format.\n\nFix the issue by parsing the reference and object formats from the\nsource repository and passing them to git-init(1).\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 8ab6d9c4694..1a9a51ca3cc 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -98,6 +98,8 @@ test_perf_create_repo_from () {\n \tsource_git=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-dir)\"\n \tobjects_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-path objects)\"\n \tcommon_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-common-dir)\"\n+\trefformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-ref-format)\"\n+\tobjectformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-object-format)\"\n \tmkdir -p \"$repo/.git\"\n \t(\n \t\tcd \"$source\" &&\n@@ -114,7 +116,7 @@ test_perf_create_repo_from () {\n \t) &&\n \t(\n \t\tcd \"$repo\" &&\n-\t\t\"$MODERN_GIT\" init -q &&\n+\t\t\"$MODERN_GIT\" init -q --ref-format=\"$refformat\" --object-format=\"$objectformat\" &&\n \t\ttest_perf_do_repo_symlink_config_ &&\n \t\tmv .git/hooks .git/hooks-disabled 2>/dev/null &&\n \t\tif test -f .git/index.lock\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516455","messageId":"20250422-pks-meson-benchmarks-v3-3-7aad68bac6fd@pks.im","threadId":"63221","inReplyTo":"20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im","subject":"[PATCH v3 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-22T06:50:13Z","receivedAt":"2025-04-22T06:50:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"perf-lib.sh\" script is sourced by all of our benchmarking suites to\nmake available common infrastructure. The script assumes that build and\nsource directory are the same, which works for our Makefile. But the\nassumption breaks with both CMake and Meson, where the build directory\ncan be located in an arbitrary place.\n\nAdapt the script so that it works with out-of-tree builds. Most\nimportantly, this requires us to figure out the location of the build\ndirectory:\n\n - When running benchmarks via our Makefile the build directory is the\n same as the source directory. We already know to derive the test\n directory (\"t/\") via `$(pwd)/..`, which works because we chdir into\n \"t/perf\" before executing benchmarks. We can thus derive the build\n directory by appending another \"/..\" to that path.\n\n - When running benchmarks via Meson the build directory is located at\n an arbitrary location. The build system thus has to make the path\n known by exporting the `GIT_BUILD_DIR` environment variable.\n\nThis change prepares us for wiring up benchmarks in Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--\n 1 file changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 4173eee4def..5406557b7ca 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)\n TEST_NO_CREATE_REPO=t\n TEST_NO_MALLOC_CHECK=t\n \n-. ../test-lib.sh\n+# While test-lib.sh computes the build directory for us, we also have to do the\n+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first\n+# place.\n+GIT_BUILD_DIR=\"${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}\"\n+if test -f \"$GIT_BUILD_DIR/GIT-BUILD-DIR\"\n+then\n+\tGIT_BUILD_DIR=\"$(cat \"$GIT_BUILD_DIR/GIT-BUILD-DIR\")\" || exit 1\n+\t# On Windows, we must convert Windows paths lest they contain a colon\n+\tcase \"$(uname -s)\" in\n+\t*MINGW*)\n+\t\tGIT_BUILD_DIR=\"$(cygpath -au \"$GIT_BUILD_DIR\")\"\n+\t\t;;\n+\tesac\n+fi\n+\n+if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+then\n+\techo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'\n+\texit 1\n+fi\n+\n+. \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+. \"$GIT_SOURCE_DIR\"/t/test-lib.sh\n \n unset GIT_CONFIG_NOSYSTEM\n GIT_CONFIG_SYSTEM=\"$TEST_DIRECTORY/perf/config\"\n@@ -324,7 +346,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" \"$GIT_SOURCE_DIR\"/t/perf/aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516456","messageId":"20250422-pks-meson-benchmarks-v3-4-7aad68bac6fd@pks.im","threadId":"63221","inReplyTo":"20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im","subject":"[PATCH v3 4/5] meson: wire up benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-22T06:50:14Z","receivedAt":"2025-04-22T06:50:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up benchmarks in Meson. The setup is mostly the same as how we wire\nup our tests. The only difference is that benchmarks get wired up via\nthe `benchmark()` option instead of via `test()`, which gives them a bit\nof special treatment:\n\n - Benchmarks never run in parallel.\n\n - Benchmarks aren't run by default when tests are executed.\n\n - Meson does not inject the `MALLOC_PERTURB` environment variable.\n\nUsing benchmarks is quite simple:\n\n ```\n $ meson setup build\n # Run all benchmarks.\n $ meson test -C build --benchmark\n # Run a specific benchmark.\n $ meson test -C build --benchmark p0000-*\n ```\n\nOther than that the usual command line arguments accepted when running\ntests are also accepted when running benchmarks.\n\nNote that the benchmarking target is somewhat limited because it will\nonly run benchmarks for the current build. Other use cases, like running\nbenchmarks against multiple different versions of Git, are not currently\nsupported. Users should continue to use \"t/perf/run\" for those use\ncases. The script should get extended at one point in time to support\nMeson, but this is outside of the scope of this series.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 7 ++++++\n t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 84 insertions(+)\n\ndiff --git a/meson.build b/meson.build\nindex efe2871c9db..205b1bb301a 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -70,6 +70,12 @@\n # # Execute single test interactively such that features like `debug ()` work.\n # $ meson test -i --test-args='-ix' t1400-update-ref\n #\n+# # Execute all benchmarks.\n+# $ meson test -i --benchmark\n+#\n+# # Execute single benchmark.\n+# $ meson test -i --benchmark p0000-*\n+#\n # Test execution is parallelized by default and scales with the number of\n # processor cores available. You can change the number of processes by passing\n # the `-jN` flag to `meson test`.\n@@ -204,6 +210,7 @@ git = find_program('git', dirs: program_path, required: false)\n sed = find_program('sed', dirs: program_path)\n shell = find_program('sh', dirs: program_path)\n tar = find_program('tar', dirs: program_path)\n+time = find_program('time', dirs: program_path, required: false)\n \n # Sanity-check that programs required for the build exist.\n foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']\ndiff --git a/t/meson.build b/t/meson.build\nindex a59da26be3f..dba327fd7ec 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -1096,11 +1096,71 @@ integration_tests = [\n 't9903-bash-prompt.sh',\n ]\n \n+benchmarks = [\n+ 'perf/p0000-perf-lib-sanity.sh',\n+ 'perf/p0001-rev-list.sh',\n+ 'perf/p0002-read-cache.sh',\n+ 'perf/p0003-delta-base-cache.sh',\n+ 'perf/p0004-lazy-init-name-hash.sh',\n+ 'perf/p0005-status.sh',\n+ 'perf/p0006-read-tree-checkout.sh',\n+ 'perf/p0007-write-cache.sh',\n+ 'perf/p0008-odb-fsync.sh',\n+ 'perf/p0071-sort.sh',\n+ 'perf/p0090-cache-tree.sh',\n+ 'perf/p0100-globbing.sh',\n+ 'perf/p1006-cat-file.sh',\n+ 'perf/p1400-update-ref.sh',\n+ 'perf/p1450-fsck.sh',\n+ 'perf/p1451-fsck-skip-list.sh',\n+ 'perf/p1500-graph-walks.sh',\n+ 'perf/p2000-sparse-operations.sh',\n+ 'perf/p3400-rebase.sh',\n+ 'perf/p3404-rebase-interactive.sh',\n+ 'perf/p4000-diff-algorithms.sh',\n+ 'perf/p4001-diff-no-index.sh',\n+ 'perf/p4002-diff-color-moved.sh',\n+ 'perf/p4205-log-pretty-formats.sh',\n+ 'perf/p4209-pickaxe.sh',\n+ 'perf/p4211-line-log.sh',\n+ 'perf/p4220-log-grep-engines.sh',\n+ 'perf/p4221-log-grep-engines-fixed.sh',\n+ 'perf/p5302-pack-index.sh',\n+ 'perf/p5303-many-packs.sh',\n+ 'perf/p5304-prune.sh',\n+ 'perf/p5310-pack-bitmaps.sh',\n+ 'perf/p5311-pack-bitmaps-fetch.sh',\n+ 'perf/p5312-pack-bitmaps-revs.sh',\n+ 'perf/p5313-pack-objects.sh',\n+ 'perf/p5314-name-hash.sh',\n+ 'perf/p5326-multi-pack-bitmaps.sh',\n+ 'perf/p5332-multi-pack-reuse.sh',\n+ 'perf/p5333-pseudo-merge-bitmaps.sh',\n+ 'perf/p5550-fetch-tags.sh',\n+ 'perf/p5551-fetch-rescan.sh',\n+ 'perf/p5600-partial-clone.sh',\n+ 'perf/p5601-clone-reference.sh',\n+ 'perf/p6100-describe.sh',\n+ 'perf/p6300-for-each-ref.sh',\n+ 'perf/p7000-filter-branch.sh',\n+ 'perf/p7102-reset.sh',\n+ 'perf/p7300-clean.sh',\n+ 'perf/p7519-fsmonitor.sh',\n+ 'perf/p7527-builtin-fsmonitor.sh',\n+ 'perf/p7810-grep.sh',\n+ 'perf/p7820-grep-engines.sh',\n+ 'perf/p7821-grep-engines-fixed.sh',\n+ 'perf/p7822-grep-perl-character.sh',\n+ 'perf/p9210-scalar.sh',\n+ 'perf/p9300-fast-import-export.sh',\n+]\n+\n # Sanity check that we are not missing any tests present in 't/'. This check\n # only runs once at configure time and is thus best-effort, only. It is\n # sufficient to catch missing test suites in our CI though.\n foreach glob, tests : {\n 't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,\n+ 'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,\n 'unit-tests/t-*.c': unit_test_programs,\n 'unit-tests/u-*.c': clar_test_suites,\n }\n@@ -1152,3 +1212,20 @@ foreach integration_test : integration_tests\n timeout: 0,\n )\n endforeach\n+\n+if time.found()\n+ benchmark_environment = test_environment\n+ benchmark_environment.set('GTIME', time.full_path())\n+\n+ foreach benchmark : benchmarks\n+ benchmark(fs.stem(benchmark), shell,\n+ args: [\n+ fs.name(benchmark),\n+ ],\n+ workdir: meson.current_source_dir() / 'perf',\n+ env: benchmark_environment,\n+ depends: test_dependencies + bin_wrappers,\n+ timeout: 0,\n+ )\n+ endforeach\n+endif\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516457","messageId":"20250422-pks-meson-benchmarks-v3-5-7aad68bac6fd@pks.im","threadId":"63221","inReplyTo":"20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im","subject":"[PATCH v3 5/5] meson: wire up benchmarking options","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-22T06:50:15Z","receivedAt":"2025-04-22T06:50:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up a couple of benchmarking options that we end up writing into our\n\"GIT-BUILD-OPTIONS\" file. These options allow users to control how\nexactly benchmarks are executed.\n\nNote that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are\nexposed as a build option. Those options are used by \"t/perf/run\", which\nis not used by Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 6 +++---\n meson_options.txt | 6 ++++++\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/meson.build b/meson.build\nindex 205b1bb301a..5bba743dc0e 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -666,11 +666,11 @@ builtin_sources += custom_target(\n # build options to our tests.\n build_options_config = configuration_data()\n build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_LARGE_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))\n build_options_config.set('GIT_PERF_MAKE_COMMAND', '')\n build_options_config.set('GIT_PERF_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')\n-build_options_config.set('GIT_PERF_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())\n+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))\n build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')\n build_options_config.set('GIT_TEST_INDEX_VERSION', '')\n build_options_config.set('GIT_TEST_OPTS', '')\ndiff --git a/meson_options.txt b/meson_options.txt\nindex 78d172a7401..ca106c43b2e 100644\n--- a/meson_options.txt\n+++ b/meson_options.txt\n@@ -101,6 +101,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto\n description: 'Which backend to use to generate documentation.')\n \n # Testing.\n+option('benchmark_repo', type: 'string', value: '',\n+ description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')\n+option('benchmark_large_repo', type: 'string', value: '',\n+ description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')\n+option('benchmark_repeat_count', type: 'integer', value: 3,\n+ description: 'Number of times a test should be repeated for best-of-N measurements.')\n option('coccinelle', type: 'feature', value: 'auto',\n description: 'Provide a coccicheck target that generates a Coccinelle patch.')\n option('tests', type: 'boolean', value: true,\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516458","messageId":"aAc8f52rJ8ATncVc@pks.im","threadId":"63221","inReplyTo":"CAP8UFD3HcAVs351sabU2qd70PmMzTtGbS7P7eKYyDyUQZ4nQ2Q@mail.gmail.com","subject":"Re: [PATCH v2 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-22T06:51:43Z","receivedAt":"2025-04-22T06:51:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Apr 20, 2025 at 12:00:08PM +0200, Christian Couder wrote:\n> On Mon, Apr 14, 2025 at 8:51 AM Patrick Steinhardt wrote:\n> \n> > --- a/t/perf/perf-lib.sh\n> > +++ b/t/perf/perf-lib.sh\n> > @@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)\n> > TEST_NO_CREATE_REPO=t\n> > TEST_NO_MALLOC_CHECK=t\n> >\n> > -. ../test-lib.sh\n> > +# While test-lib.sh computes the build directory for us, we also have to do the\n> > +# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first\n> > +# place.\n> > +GIT_BUILD_DIR=\"${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}\"\n> \n> Right now on 'master' there is:\n> \n> GIT_BUILD_DIR=\"${GIT_BUILD_DIR:-${TEST_DIRECTORY%/t}}\"\n> if test \"$TEST_DIRECTORY\" = \"$GIT_BUILD_DIR\"\n> then\n> echo \"PANIC: Running in a $TEST_DIRECTORY that doesn't end in '/t'?\" >&2\n> exit 1\n> fi\n> \n> so it's not exactly the same thing, even if it still probably works well.\n> \n> Future readers might wonder if this discrepancy results from changes\n> that were made to only one of the files or if we really wanted to get\n> rid of the \"/t\" check here. In case we do want to get rid of the \"/t\"\n> check, I think it might be worth saying it clearly in the comment.\n\nThe \"/..\" is intentional here due to the way that `TEST_DIRECTORY` is\nconstructed. If you extend the context of this patch a bit, you can see\nthat `TEST_DIRECTORY=$(pwd)/..`. So stripping \"/t\" from the suffix\nwouldn't do anything because it never has that suffix in the first\nplace. And neither do we want to strip \"/..\", because then we'd end up\nin \"t/perf\". So the easiest fix is to just append another \"/..\" to end\nup where we want to.\n\nI'll try to paraphrase this in the commit message.\n\n> > +if test -f \"$GIT_BUILD_DIR/GIT-BUILD-DIR\"\n> > +then\n> > + GIT_BUILD_DIR=\"$(cat \"$GIT_BUILD_DIR/GIT-BUILD-DIR\")\" || exit 1\n> > + # On Windows, we must convert Windows paths lest they contain a colon\n> > + case \"$(uname -s)\" in\n> > + *MINGW*)\n> > + GIT_BUILD_DIR=\"$(cygpath -au \"$GIT_BUILD_DIR\")\"\n> > + ;;\n> > + esac\n> > +fi\n> > +\n> > +if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n> > +then\n> > + echo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'\n> \n> Maybe something like the following could help debug this:\n> \n> echo >&2 \"error: GIT-BUILD-OPTIONS file missing from '$GIT_BUILD_DIR'\"\n> echo >&2 'error: (has Git been built?).'\n\nI'd rather want to keep this as-is for now as we have the same error\nmessage in \"test-lib.sh\". If we want to change it we should change both\nerrors, but that feels outside of the scope of this patch series.\n\nPatrick\n"},{"id":"516468","messageId":"CAP8UFD3Nr_f7WSzjv4AEKzH9Kpsf6soOd+VSO9ng=-ZSws6P3g@mail.gmail.com","threadId":"63221","inReplyTo":"20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im","subject":"Re: [PATCH v3 0/5] meson: wire up support for benchmarks","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-04-22T07:27:57Z","receivedAt":"2025-04-22T07:28:12Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Apr 22, 2025 at 8:50 AM Patrick Steinhardt wrote:\n>\n> Hi,\n>\n> this small patch series implements support for running our benchmarks in\n> \"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\n> which is more fully-featured and allows running benchmarks against\n> multiple different trees. Instead, this series only allows running the\n> benchmarks against the current tree. Users are thus expected to continue\n> using \"t/perf/run\" for more advanced usecases.\n>\n> Changes in v2:\n> - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n> - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n>\n> Changes in v3:\n> - Document how to run benchmarks in \"meson.build\".\n> - Expand the message for the commit that enables out-of-tree\n> benchmarking.\n> - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im\n\nIn https://lore.kernel.org/git/CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com/\nI wrote:\n\n\"I wonder what happens when we run `meson test -C build --benchmark`\nbut 'time' is not found.\"\n\nbecause I wasn't sure if it would just do nothing in that case which\nmight not be very user friendly.\n\nOtherwise it looks like my concerns have been addressed.\n\nThanks.\n"},{"id":"516479","messageId":"aAdK9gnXOO3ERq7w@pks.im","threadId":"63221","inReplyTo":"CAP8UFD3Nr_f7WSzjv4AEKzH9Kpsf6soOd+VSO9ng=-ZSws6P3g@mail.gmail.com","subject":"Re: [PATCH v3 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-22T07:53:26Z","receivedAt":"2025-04-22T07:53:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Apr 22, 2025 at 09:27:57AM +0200, Christian Couder wrote:\n> On Tue, Apr 22, 2025 at 8:50 AM Patrick Steinhardt wrote:\n> >\n> > Hi,\n> >\n> > this small patch series implements support for running our benchmarks in\n> > \"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\n> > which is more fully-featured and allows running benchmarks against\n> > multiple different trees. Instead, this series only allows running the\n> > benchmarks against the current tree. Users are thus expected to continue\n> > using \"t/perf/run\" for more advanced usecases.\n> >\n> > Changes in v2:\n> > - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n> > - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n> >\n> > Changes in v3:\n> > - Document how to run benchmarks in \"meson.build\".\n> > - Expand the message for the commit that enables out-of-tree\n> > benchmarking.\n> > - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im\n> \n> In https://lore.kernel.org/git/CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com/\n> I wrote:\n> \n> \"I wonder what happens when we run `meson test -C build --benchmark`\n> but 'time' is not found.\"\n> \n> because I wasn't sure if it would just do nothing in that case which\n> might not be very user friendly.\n\nAh, sorry, forgot to answer that question. What Meson does in that case\nis to print \"No tests defined\". We could help improve usability a bit by\nprinting benchmarks as part of the auto-detected features after setup of\nthe build directory has finished. That would make it more discoverable\nthat benchmarks have been disabled.\n\nPatrick\n"},{"id":"516606","messageId":"87a587ezlu.fsf@iotcl.com","threadId":"63221","inReplyTo":"20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im","subject":"Re: [PATCH v3 0/5] meson: wire up support for benchmarks","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-04-23T14:12:13Z","receivedAt":"2025-04-23T14:12:30Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this small patch series implements support for running our benchmarks in\n> \"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\n> which is more fully-featured and allows running benchmarks against\n> multiple different trees. Instead, this series only allows running the\n> benchmarks against the current tree. Users are thus expected to continue\n> using \"t/perf/run\" for more advanced usecases.\n>\n> Changes in v2:\n> - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n> - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n>\n> Changes in v3:\n> - Document how to run benchmarks in \"meson.build\".\n\nThanks for adding this.\n\nI have no further comments. I approve.\n\n\n--\nToon\n"},{"id":"516607","messageId":"CAP8UFD1zqAk1_Y0ECu4=iyeF4k4LagfLgHWkEr8iz8ZnwkVT8A@mail.gmail.com","threadId":"63221","inReplyTo":"aAdK9gnXOO3ERq7w@pks.im","subject":"Re: [PATCH v3 0/5] meson: wire up support for benchmarks","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-04-23T14:44:20Z","receivedAt":"2025-04-23T14:44:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Apr 22, 2025 at 9:53 AM Patrick Steinhardt wrote:\n>\n> On Tue, Apr 22, 2025 at 09:27:57AM +0200, Christian Couder wrote:\n\n> > In https://lore.kernel.org/git/CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com/\n> > I wrote:\n> >\n> > \"I wonder what happens when we run `meson test -C build --benchmark`\n> > but 'time' is not found.\"\n> >\n> > because I wasn't sure if it would just do nothing in that case which\n> > might not be very user friendly.\n>\n> Ah, sorry, forgot to answer that question. What Meson does in that case\n> is to print \"No tests defined\". We could help improve usability a bit by\n> printing benchmarks as part of the auto-detected features after setup of\n> the build directory has finished. That would make it more discoverable\n> that benchmarks have been disabled.\n\nI was more expecting something simple like:\n\nif time.found()\n ...\nelse\n error('Benchmarking requires the `time` command')\nendif\n\nin the same way as in meson.build elsewhere we have things like:\n\nif not msgfmt.found() and gettext_option.enabled()\n error('Internationalization via libintl requires msgfmt')\nendif\n"},{"id":"516646","messageId":"aAm-pcIfrFt6mjlV@pks.im","threadId":"63221","inReplyTo":"CAP8UFD1zqAk1_Y0ECu4=iyeF4k4LagfLgHWkEr8iz8ZnwkVT8A@mail.gmail.com","subject":"Re: [PATCH v3 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-24T04:31:33Z","receivedAt":"2025-04-24T04:31:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 23, 2025 at 04:44:20PM +0200, Christian Couder wrote:\n> On Tue, Apr 22, 2025 at 9:53 AM Patrick Steinhardt wrote:\n> >\n> > On Tue, Apr 22, 2025 at 09:27:57AM +0200, Christian Couder wrote:\n> \n> > > In https://lore.kernel.org/git/CAP8UFD2jKwYzmc40knXY7k+FQabjZbGTqs9fowF=-0OqfNYp_w@mail.gmail.com/\n> > > I wrote:\n> > >\n> > > \"I wonder what happens when we run `meson test -C build --benchmark`\n> > > but 'time' is not found.\"\n> > >\n> > > because I wasn't sure if it would just do nothing in that case which\n> > > might not be very user friendly.\n> >\n> > Ah, sorry, forgot to answer that question. What Meson does in that case\n> > is to print \"No tests defined\". We could help improve usability a bit by\n> > printing benchmarks as part of the auto-detected features after setup of\n> > the build directory has finished. That would make it more discoverable\n> > that benchmarks have been disabled.\n> \n> I was more expecting something simple like:\n> \n> if time.found()\n> ...\n> else\n> error('Benchmarking requires the `time` command')\n> endif\n> \n> in the same way as in meson.build elsewhere we have things like:\n> \n> if not msgfmt.found() and gettext_option.enabled()\n> error('Internationalization via libintl requires msgfmt')\n> endif\n\nBut erroring out by default doesn't really feel nice to the general\ndeveloper. I'd claim that 99% of the time, developers will only end up\nrunning the test suite, never the benchmarks. So the default should\nmatch that and not require GNU time to be available, if you ask me.\n\nAn alternative could be to have a 'benchmarks' feature option wired up.\nThis feature would be set to 'auto', and then we can detect time like\nthis:\n\n time = find_program('time', dirs: program_path, required: get_option('benchmarks'))\n\nThis will then auto-enable or -disable the feature depending on the\navailability of GNU time. But if the user passes `-Dbenchmarks=enabled`,\nwe would fail the setup in case the program wasn't found.\n\nPatrick\n"},{"id":"516648","messageId":"CAP8UFD26q4zDTN=NrCtYNFPh+H=f4yU-1+3HxEE-DZD+qO0_Mw@mail.gmail.com","threadId":"63221","inReplyTo":"aAm-pcIfrFt6mjlV@pks.im","subject":"Re: [PATCH v3 0/5] meson: wire up support for benchmarks","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-04-24T06:28:04Z","receivedAt":"2025-04-24T06:28:18Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Apr 24, 2025 at 6:31 AM Patrick Steinhardt wrote:\n>\n> On Wed, Apr 23, 2025 at 04:44:20PM +0200, Christian Couder wrote:\n> > On Tue, Apr 22, 2025 at 9:53 AM Patrick Steinhardt wrote:\n> > >\n> > > On Tue, Apr 22, 2025 at 09:27:57AM +0200, Christian Couder wrote:\n\n> > if time.found()\n> > ...\n> > else\n> > error('Benchmarking requires the `time` command')\n> > endif\n> >\n> > in the same way as in meson.build elsewhere we have things like:\n> >\n> > if not msgfmt.found() and gettext_option.enabled()\n> > error('Internationalization via libintl requires msgfmt')\n> > endif\n>\n> But erroring out by default doesn't really feel nice to the general\n> developer. I'd claim that 99% of the time, developers will only end up\n> running the test suite, never the benchmarks. So the default should\n> match that and not require GNU time to be available, if you ask me.\n\nYeah, right.\n\n> An alternative could be to have a 'benchmarks' feature option wired up.\n> This feature would be set to 'auto', and then we can detect time like\n> this:\n>\n> time = find_program('time', dirs: program_path, required: get_option('benchmarks'))\n>\n> This will then auto-enable or -disable the feature depending on the\n> availability of GNU time. But if the user passes `-Dbenchmarks=enabled`,\n> we would fail the setup in case the program wasn't found.\n\nYeah, that would be great.\n\nAn alternative might be a placeholder benchmark that just reports an\nerror when run, like:\n\nif time.found()\n ...\nelse\n benchmark('perf-benchmarks-require-time', shell, args: [\n '-c',\n 'echo \"ERROR: Benchmarks require the \\\"time\\\" command to be\navailable. Please install it and reconfigure.\" >&2; exit 1'\n ], env: test_environment, )\nendif\n"},{"id":"516656","messageId":"xmqqwmb9vmlw.fsf@gitster.g","threadId":"63221","inReplyTo":"aAm-pcIfrFt6mjlV@pks.im","subject":"Re: [PATCH v3 0/5] meson: wire up support for benchmarks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-24T11:13:15Z","receivedAt":"2025-04-24T11:13:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> I was more expecting something simple like:\n>> \n>> if time.found()\n>> ...\n>> else\n>> error('Benchmarking requires the `time` command')\n>> endif\n>> \n>> in the same way as in meson.build elsewhere we have things like:\n>> \n>> if not msgfmt.found() and gettext_option.enabled()\n>> error('Internationalization via libintl requires msgfmt')\n>> endif\n>\n> But erroring out by default doesn't really feel nice to the general\n> developer. I'd claim that 99% of the time, developers will only end up\n> running the test suite, never the benchmarks. So the default should\n> match that and not require GNU time to be available, if you ask me.\n\nMaybe a stupid question, but does \"time\" a shell built-in count when\n\"if time.found()\" is evaluated?\n\n $ type --all time\n time is a shell keyword\n time is /usr/bin/time\n time is /bin/time\n\n> An alternative could be to have a 'benchmarks' feature option wired up.\n> This feature would be set to 'auto', and then we can detect time like\n> this:\n>\n> time = find_program('time', dirs: program_path, required: get_option('benchmarks'))\n>\n> This will then auto-enable or -disable the feature depending on the\n> availability of GNU time. But if the user passes `-Dbenchmarks=enabled`,\n> we would fail the setup in case the program wasn't found.\n\nSounds quite sensible. Do what most users expect by default without\nfailing, let users give explicit preference, and when the explicitly\nexpressed preference cannot be fulfilled, loudly error out. I like\nit.\n\nThanks.\n"},{"id":"516674","messageId":"aApBbArLEq9dSIYr@pks.im","threadId":"63221","inReplyTo":"xmqqwmb9vmlw.fsf@gitster.g","subject":"Re: [PATCH v3 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-24T13:49:32Z","receivedAt":"2025-04-24T13:49:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 24, 2025 at 04:13:15AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> I was more expecting something simple like:\n> >> \n> >> if time.found()\n> >> ...\n> >> else\n> >> error('Benchmarking requires the `time` command')\n> >> endif\n> >> \n> >> in the same way as in meson.build elsewhere we have things like:\n> >> \n> >> if not msgfmt.found() and gettext_option.enabled()\n> >> error('Internationalization via libintl requires msgfmt')\n> >> endif\n> >\n> > But erroring out by default doesn't really feel nice to the general\n> > developer. I'd claim that 99% of the time, developers will only end up\n> > running the test suite, never the benchmarks. So the default should\n> > match that and not require GNU time to be available, if you ask me.\n> \n> Maybe a stupid question, but does \"time\" a shell built-in count when\n> \"if time.found()\" is evaluated?\n> \n> $ type --all time\n> time is a shell keyword\n> time is /usr/bin/time\n> time is /bin/time\n\nNo, it doesn't, and that is intentional. Our benchmarks explicitly\nrequire GNU time because it surfaces more information than the shell\nbuiltin.\n\nWe even have a `USR_BIN_TIME` prereq in our test lib, which is\nunfortunately way too limiting for some platforms like NixOS. Maybe I\nshould use that as an excuse to also refactor this prereq while at it to\ninstead use a path-based lookup of time. On the other hand it's only\nused by a single test anyway, so it doesn't really feel worth it.\n\nPatrick\n"},{"id":"516752","messageId":"20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","subject":"[PATCH v4 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-25T07:28:12Z","receivedAt":"2025-04-25T07:28:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis small patch series implements support for running our benchmarks in\n\"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\nwhich is more fully-featured and allows running benchmarks against\nmultiple different trees. Instead, this series only allows running the\nbenchmarks against the current tree. Users are thus expected to continue\nusing \"t/perf/run\" for more advanced usecases.\n\nChanges in v2:\n - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n\nChanges in v3:\n - Document how to run benchmarks in \"meson.build\".\n - Expand the message for the commit that enables out-of-tree\n benchmarking.\n - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im\n\nChanges in v4:\n - The patch series was rebased on top of f65182a99e5 (The ninth batch,\n 2025-04-24). This is due to a conflict with ps/test-wo-perl-prereq.\n - Introduce a 'benchmarks' option. This allows developers to require\n benchmarks as desired. By default, we auto-detect whether the host\n system has all dependencies available and enable or disable them\n accordingly.\n - Report whether or not benchmarks are enabled via `summary()`.\n - Our benchmarks depend on Perl, so add this dependency accordingly.\n - Link to v3: https://lore.kernel.org/r/20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (5):\n t/perf: fix benchmarks with alternate repo formats\n t/perf: use configured PERL_PATH\n t/perf: fix benchmarks with out-of-tree builds\n meson: wire up benchmarks\n meson: wire up benchmarking options\n\n meson.build | 16 ++++++++---\n meson_options.txt | 8 ++++++\n t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++\n t/perf/aggregate.perl | 2 +-\n t/perf/perf-lib.sh | 32 ++++++++++++++++++---\n t/perf/run | 4 +--\n 6 files changed, 128 insertions(+), 11 deletions(-)\n\nRange-diff versus v3:\n\n1: 2375a16c2c0 = 1: 174804805c2 t/perf: fix benchmarks with alternate repo formats\n2: 69782035246 = 2: 41faa3d9b3b t/perf: use configured PERL_PATH\n3: cca40e9dd0e = 3: 87ea3ab1a45 t/perf: fix benchmarks with out-of-tree builds\n4: 50e9b507db4 ! 4: 723a0a2e449 meson: wire up benchmarks\n @@ meson.build\n # Test execution is parallelized by default and scales with the number of\n # processor cores available. You can change the number of processes by passing\n # the `-jN` flag to `meson test`.\n -@@ meson.build: git = find_program('git', dirs: program_path, required: false)\n - sed = find_program('sed', dirs: program_path)\n - shell = find_program('sh', dirs: program_path)\n - tar = find_program('tar', dirs: program_path)\n -+time = find_program('time', dirs: program_path, required: false)\n +@@ meson.build: git = find_program('git', dirs: program_path, native: true, required: false)\n + sed = find_program('sed', dirs: program_path, native: true)\n + shell = find_program('sh', dirs: program_path, native: true)\n + tar = find_program('tar', dirs: program_path, native: true)\n ++time = find_program('time', dirs: program_path, required: get_option('benchmarks'))\n \n - # Sanity-check that programs required for the build exist.\n - foreach tool : ['cat', 'cut', 'grep', 'sort', 'tr', 'uname']\n + target_shell = find_program('sh', dirs: program_path, native: false)\n + \n +@@ meson.build: endif\n + # features. It is optional if you want to neither execute tests nor use any of\n + # these optional features.\n + perl_required = get_option('perl')\n +-if get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []\n ++if get_option('benchmarks').enabled() or get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []\n + perl_required = true\n + endif\n + \n +@@ meson.build: meson.add_dist_script(\n + )\n + \n + summary({\n ++ 'benchmarks': get_option('tests') and perl.found() and time.found(),\n + 'curl': curl.found(),\n + 'expat': expat.found(),\n + 'gettext': intl.found(),\n +\n + ## meson_options.txt ##\n +@@ meson_options.txt: option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto\n + description: 'Which backend to use to generate documentation.')\n + \n + # Testing.\n ++option('benchmarks', type: 'feature', value: 'auto',\n ++ description: 'Enable benchmarks. This requires Perl and GNU time.')\n + option('coccinelle', type: 'feature', value: 'auto',\n + description: 'Provide a coccicheck target that generates a Coccinelle patch.')\n + option('tests', type: 'boolean', value: true,\n \n ## t/meson.build ##\n @@ t/meson.build: integration_tests = [\n @@ t/meson.build: foreach integration_test : integration_tests\n )\n endforeach\n +\n -+if time.found()\n ++if perl.found() and time.found()\n + benchmark_environment = test_environment\n + benchmark_environment.set('GTIME', time.full_path())\n +\n5: 52cca4b17ae ! 5: 16c0230656f meson: wire up benchmarking options\n @@ meson.build: builtin_sources += custom_target(\n \n ## meson_options.txt ##\n @@ meson_options.txt: option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto\n - description: 'Which backend to use to generate documentation.')\n - \n # Testing.\n + option('benchmarks', type: 'feature', value: 'auto',\n + description: 'Enable benchmarks. This requires Perl and GNU time.')\n +option('benchmark_repo', type: 'string', value: '',\n + description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')\n +option('benchmark_large_repo', type: 'string', value: '',\n\n---\nbase-commit: f65182a99e545d2f2bc22e6c1c2da192133b16a3\nchange-id: 20250328-pks-meson-benchmarks-a8fac5f69467\n\n"},{"id":"516753","messageId":"20250425-pks-meson-benchmarks-v4-1-6b89555052c3@pks.im","threadId":"63221","inReplyTo":"20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im","subject":"[PATCH v4 1/5] t/perf: fix benchmarks with alternate repo formats","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-25T07:28:13Z","receivedAt":"2025-04-25T07:28:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Many of our benchmarks operate on a user-defined repository that we copy\nover before running the benchmarked logic. To keep unintentional side\neffects caused by on-disk state at bay we skip copying some files. This\nincludes for example hooks, but also the repo's configuration.\n\nIt is quite sensible to not copy over the configuration, as it is quite\neasy to inadvertently carry over configuration that may significantly\nimpact the performance measurements. But we cannot fully ignore the\nconfiguration either, as it may contain information about the repository\nformat. This will cause failures when for example using a repository\nwith SHA256 object format or the reftable ref format.\n\nFix the issue by parsing the reference and object formats from the\nsource repository and passing them to git-init(1).\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 8ab6d9c4694..1a9a51ca3cc 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -98,6 +98,8 @@ test_perf_create_repo_from () {\n \tsource_git=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-dir)\"\n \tobjects_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-path objects)\"\n \tcommon_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-common-dir)\"\n+\trefformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-ref-format)\"\n+\tobjectformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-object-format)\"\n \tmkdir -p \"$repo/.git\"\n \t(\n \t\tcd \"$source\" &&\n@@ -114,7 +116,7 @@ test_perf_create_repo_from () {\n \t) &&\n \t(\n \t\tcd \"$repo\" &&\n-\t\t\"$MODERN_GIT\" init -q &&\n+\t\t\"$MODERN_GIT\" init -q --ref-format=\"$refformat\" --object-format=\"$objectformat\" &&\n \t\ttest_perf_do_repo_symlink_config_ &&\n \t\tmv .git/hooks .git/hooks-disabled 2>/dev/null &&\n \t\tif test -f .git/index.lock\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516754","messageId":"20250425-pks-meson-benchmarks-v4-2-6b89555052c3@pks.im","threadId":"63221","inReplyTo":"20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im","subject":"[PATCH v4 2/5] t/perf: use configured PERL_PATH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-25T07:28:14Z","receivedAt":"2025-04-25T07:28:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our benchmarks use a couple of Perl scripts to compute results. These\nPerl scripts get executed directly, and as the shebang is hardcoded to\n\"/usr/bin/perl\" this will fail on any system where the Perl interpreter\nis located in a different path.\n\nOur build infrastructure already lets users configure the location of\nPerl, which ultimately gets written into the GIT-BUILD-OPTIONS file.\nThis file is being sourced by \"test-lib.sh\", and consequently we already\nhave the \"PERL_PATH\" variable available that contains its configured\nlocation.\n\nUse \"PERL_PATH\" to execute Perl scripts, which makes them work on more\nesoteric systems like NixOS. Furthermore, adapt the shebang to use\nenv(1) to execute Perl so that users who have Perl in PATH, but in a\nnon-standard location can execute the script directly.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/aggregate.perl | 2 +-\n t/perf/perf-lib.sh | 4 ++--\n t/perf/run | 4 ++--\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex 575d2000cc1..1791c7528a9 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -1,4 +1,4 @@\n-#!/usr/bin/perl\n+#!/usr/bin/env perl\n \n use lib '../../perl/build/lib';\n use strict;\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 1a9a51ca3cc..4173eee4def 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -276,7 +276,7 @@ test_perf_ () {\n \telse\n \t\ttest_ok_ \"$1\"\n \tfi\n-\t\"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n \trm test_time.*\n }\n \n@@ -324,7 +324,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\ndiff --git a/t/perf/run b/t/perf/run\nindex 486ead21980..073bcb2afff 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -192,10 +192,10 @@ run_subsection () {\n \n \tif test -z \"$GIT_PERF_SEND_TO_CODESPEED\"\n \tthen\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n \telse\n \t\tjson_res_file=\"\"$TEST_RESULTS_DIR\"/$GIT_PERF_SUBSECTION/aggregate.json\"\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n \t\tsend_data_url=\"$GIT_PERF_SEND_TO_CODESPEED/result/add/json/\"\n \t\tcurl -v --request POST --data-urlencode \"json=$(cat \"$json_res_file\")\" \"$send_data_url\"\n \tfi\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516755","messageId":"20250425-pks-meson-benchmarks-v4-3-6b89555052c3@pks.im","threadId":"63221","inReplyTo":"20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im","subject":"[PATCH v4 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-25T07:28:15Z","receivedAt":"2025-04-25T07:28:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"perf-lib.sh\" script is sourced by all of our benchmarking suites to\nmake available common infrastructure. The script assumes that build and\nsource directory are the same, which works for our Makefile. But the\nassumption breaks with both CMake and Meson, where the build directory\ncan be located in an arbitrary place.\n\nAdapt the script so that it works with out-of-tree builds. Most\nimportantly, this requires us to figure out the location of the build\ndirectory:\n\n - When running benchmarks via our Makefile the build directory is the\n same as the source directory. We already know to derive the test\n directory (\"t/\") via `$(pwd)/..`, which works because we chdir into\n \"t/perf\" before executing benchmarks. We can thus derive the build\n directory by appending another \"/..\" to that path.\n\n - When running benchmarks via Meson the build directory is located at\n an arbitrary location. The build system thus has to make the path\n known by exporting the `GIT_BUILD_DIR` environment variable.\n\nThis change prepares us for wiring up benchmarks in Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--\n 1 file changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 4173eee4def..5406557b7ca 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)\n TEST_NO_CREATE_REPO=t\n TEST_NO_MALLOC_CHECK=t\n \n-. ../test-lib.sh\n+# While test-lib.sh computes the build directory for us, we also have to do the\n+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first\n+# place.\n+GIT_BUILD_DIR=\"${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}\"\n+if test -f \"$GIT_BUILD_DIR/GIT-BUILD-DIR\"\n+then\n+\tGIT_BUILD_DIR=\"$(cat \"$GIT_BUILD_DIR/GIT-BUILD-DIR\")\" || exit 1\n+\t# On Windows, we must convert Windows paths lest they contain a colon\n+\tcase \"$(uname -s)\" in\n+\t*MINGW*)\n+\t\tGIT_BUILD_DIR=\"$(cygpath -au \"$GIT_BUILD_DIR\")\"\n+\t\t;;\n+\tesac\n+fi\n+\n+if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+then\n+\techo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'\n+\texit 1\n+fi\n+\n+. \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+. \"$GIT_SOURCE_DIR\"/t/test-lib.sh\n \n unset GIT_CONFIG_NOSYSTEM\n GIT_CONFIG_SYSTEM=\"$TEST_DIRECTORY/perf/config\"\n@@ -324,7 +346,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" \"$GIT_SOURCE_DIR\"/t/perf/aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516756","messageId":"20250425-pks-meson-benchmarks-v4-4-6b89555052c3@pks.im","threadId":"63221","inReplyTo":"20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im","subject":"[PATCH v4 4/5] meson: wire up benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-25T07:28:16Z","receivedAt":"2025-04-25T07:28:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up benchmarks in Meson. The setup is mostly the same as how we wire\nup our tests. The only difference is that benchmarks get wired up via\nthe `benchmark()` option instead of via `test()`, which gives them a bit\nof special treatment:\n\n - Benchmarks never run in parallel.\n\n - Benchmarks aren't run by default when tests are executed.\n\n - Meson does not inject the `MALLOC_PERTURB` environment variable.\n\nUsing benchmarks is quite simple:\n\n ```\n $ meson setup build\n # Run all benchmarks.\n $ meson test -C build --benchmark\n # Run a specific benchmark.\n $ meson test -C build --benchmark p0000-*\n ```\n\nOther than that the usual command line arguments accepted when running\ntests are also accepted when running benchmarks.\n\nNote that the benchmarking target is somewhat limited because it will\nonly run benchmarks for the current build. Other use cases, like running\nbenchmarks against multiple different versions of Git, are not currently\nsupported. Users should continue to use \"t/perf/run\" for those use\ncases. The script should get extended at one point in time to support\nMeson, but this is outside of the scope of this series.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 10 +++++++-\n meson_options.txt | 2 ++\n t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 88 insertions(+), 1 deletion(-)\n\ndiff --git a/meson.build b/meson.build\nindex 410bbf93dad..2cd05a9b1ba 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -70,6 +70,12 @@\n # # Execute single test interactively such that features like `debug ()` work.\n # $ meson test -i --test-args='-ix' t1400-update-ref\n #\n+# # Execute all benchmarks.\n+# $ meson test -i --benchmark\n+#\n+# # Execute single benchmark.\n+# $ meson test -i --benchmark p0000-*\n+#\n # Test execution is parallelized by default and scales with the number of\n # processor cores available. You can change the number of processes by passing\n # the `-jN` flag to `meson test`.\n@@ -235,6 +241,7 @@ git = find_program('git', dirs: program_path, native: true, required: false)\n sed = find_program('sed', dirs: program_path, native: true)\n shell = find_program('sh', dirs: program_path, native: true)\n tar = find_program('tar', dirs: program_path, native: true)\n+time = find_program('time', dirs: program_path, required: get_option('benchmarks'))\n \n target_shell = find_program('sh', dirs: program_path, native: false)\n \n@@ -836,7 +843,7 @@ endif\n # features. It is optional if you want to neither execute tests nor use any of\n # these optional features.\n perl_required = get_option('perl')\n-if get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []\n+if get_option('benchmarks').enabled() or get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []\n perl_required = true\n endif\n \n@@ -2082,6 +2089,7 @@ meson.add_dist_script(\n )\n \n summary({\n+ 'benchmarks': get_option('tests') and perl.found() and time.found(),\n 'curl': curl.found(),\n 'expat': expat.found(),\n 'gettext': intl.found(),\ndiff --git a/meson_options.txt b/meson_options.txt\nindex 8ac30a52231..7f5bca5c029 100644\n--- a/meson_options.txt\n+++ b/meson_options.txt\n@@ -101,6 +101,8 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto\n description: 'Which backend to use to generate documentation.')\n \n # Testing.\n+option('benchmarks', type: 'feature', value: 'auto',\n+ description: 'Enable benchmarks. This requires Perl and GNU time.')\n option('coccinelle', type: 'feature', value: 'auto',\n description: 'Provide a coccicheck target that generates a Coccinelle patch.')\n option('tests', type: 'boolean', value: true,\ndiff --git a/t/meson.build b/t/meson.build\nindex bfb744e8863..59438b06266 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -1097,11 +1097,71 @@ integration_tests = [\n 't9903-bash-prompt.sh',\n ]\n \n+benchmarks = [\n+ 'perf/p0000-perf-lib-sanity.sh',\n+ 'perf/p0001-rev-list.sh',\n+ 'perf/p0002-read-cache.sh',\n+ 'perf/p0003-delta-base-cache.sh',\n+ 'perf/p0004-lazy-init-name-hash.sh',\n+ 'perf/p0005-status.sh',\n+ 'perf/p0006-read-tree-checkout.sh',\n+ 'perf/p0007-write-cache.sh',\n+ 'perf/p0008-odb-fsync.sh',\n+ 'perf/p0071-sort.sh',\n+ 'perf/p0090-cache-tree.sh',\n+ 'perf/p0100-globbing.sh',\n+ 'perf/p1006-cat-file.sh',\n+ 'perf/p1400-update-ref.sh',\n+ 'perf/p1450-fsck.sh',\n+ 'perf/p1451-fsck-skip-list.sh',\n+ 'perf/p1500-graph-walks.sh',\n+ 'perf/p2000-sparse-operations.sh',\n+ 'perf/p3400-rebase.sh',\n+ 'perf/p3404-rebase-interactive.sh',\n+ 'perf/p4000-diff-algorithms.sh',\n+ 'perf/p4001-diff-no-index.sh',\n+ 'perf/p4002-diff-color-moved.sh',\n+ 'perf/p4205-log-pretty-formats.sh',\n+ 'perf/p4209-pickaxe.sh',\n+ 'perf/p4211-line-log.sh',\n+ 'perf/p4220-log-grep-engines.sh',\n+ 'perf/p4221-log-grep-engines-fixed.sh',\n+ 'perf/p5302-pack-index.sh',\n+ 'perf/p5303-many-packs.sh',\n+ 'perf/p5304-prune.sh',\n+ 'perf/p5310-pack-bitmaps.sh',\n+ 'perf/p5311-pack-bitmaps-fetch.sh',\n+ 'perf/p5312-pack-bitmaps-revs.sh',\n+ 'perf/p5313-pack-objects.sh',\n+ 'perf/p5314-name-hash.sh',\n+ 'perf/p5326-multi-pack-bitmaps.sh',\n+ 'perf/p5332-multi-pack-reuse.sh',\n+ 'perf/p5333-pseudo-merge-bitmaps.sh',\n+ 'perf/p5550-fetch-tags.sh',\n+ 'perf/p5551-fetch-rescan.sh',\n+ 'perf/p5600-partial-clone.sh',\n+ 'perf/p5601-clone-reference.sh',\n+ 'perf/p6100-describe.sh',\n+ 'perf/p6300-for-each-ref.sh',\n+ 'perf/p7000-filter-branch.sh',\n+ 'perf/p7102-reset.sh',\n+ 'perf/p7300-clean.sh',\n+ 'perf/p7519-fsmonitor.sh',\n+ 'perf/p7527-builtin-fsmonitor.sh',\n+ 'perf/p7810-grep.sh',\n+ 'perf/p7820-grep-engines.sh',\n+ 'perf/p7821-grep-engines-fixed.sh',\n+ 'perf/p7822-grep-perl-character.sh',\n+ 'perf/p9210-scalar.sh',\n+ 'perf/p9300-fast-import-export.sh',\n+]\n+\n # Sanity check that we are not missing any tests present in 't/'. This check\n # only runs once at configure time and is thus best-effort, only. It is\n # sufficient to catch missing test suites in our CI though.\n foreach glob, tests : {\n 't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,\n+ 'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,\n 'unit-tests/t-*.c': unit_test_programs,\n 'unit-tests/u-*.c': clar_test_suites,\n }\n@@ -1153,3 +1213,20 @@ foreach integration_test : integration_tests\n timeout: 0,\n )\n endforeach\n+\n+if perl.found() and time.found()\n+ benchmark_environment = test_environment\n+ benchmark_environment.set('GTIME', time.full_path())\n+\n+ foreach benchmark : benchmarks\n+ benchmark(fs.stem(benchmark), shell,\n+ args: [\n+ fs.name(benchmark),\n+ ],\n+ workdir: meson.current_source_dir() / 'perf',\n+ env: benchmark_environment,\n+ depends: test_dependencies + bin_wrappers,\n+ timeout: 0,\n+ )\n+ endforeach\n+endif\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516757","messageId":"20250425-pks-meson-benchmarks-v4-5-6b89555052c3@pks.im","threadId":"63221","inReplyTo":"20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im","subject":"[PATCH v4 5/5] meson: wire up benchmarking options","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-25T07:28:17Z","receivedAt":"2025-04-25T07:28:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up a couple of benchmarking options that we end up writing into our\n\"GIT-BUILD-OPTIONS\" file. These options allow users to control how\nexactly benchmarks are executed.\n\nNote that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are\nexposed as a build option. Those options are used by \"t/perf/run\", which\nis not used by Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 6 +++---\n meson_options.txt | 6 ++++++\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/meson.build b/meson.build\nindex 2cd05a9b1ba..5e073403632 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -706,11 +706,11 @@ builtin_sources += custom_target(\n # build options to our tests.\n build_options_config = configuration_data()\n build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_LARGE_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))\n build_options_config.set('GIT_PERF_MAKE_COMMAND', '')\n build_options_config.set('GIT_PERF_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')\n-build_options_config.set('GIT_PERF_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())\n+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))\n build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')\n build_options_config.set('GIT_TEST_INDEX_VERSION', '')\n build_options_config.set('GIT_TEST_OPTS', '')\ndiff --git a/meson_options.txt b/meson_options.txt\nindex 7f5bca5c029..8547c0eb47f 100644\n--- a/meson_options.txt\n+++ b/meson_options.txt\n@@ -103,6 +103,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto\n # Testing.\n option('benchmarks', type: 'feature', value: 'auto',\n description: 'Enable benchmarks. This requires Perl and GNU time.')\n+option('benchmark_repo', type: 'string', value: '',\n+ description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')\n+option('benchmark_large_repo', type: 'string', value: '',\n+ description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')\n+option('benchmark_repeat_count', type: 'integer', value: 3,\n+ description: 'Number of times a test should be repeated for best-of-N measurements.')\n option('coccinelle', type: 'feature', value: 'auto',\n description: 'Provide a coccicheck target that generates a Coccinelle patch.')\n option('tests', type: 'boolean', value: true,\n\n-- \n2.49.0.901.g37484f566f.dirty\n\n"},{"id":"516767","messageId":"CAP8UFD0rq7Z=6PUPyPZOXaKyXzbWW7Eb0rDfoTxrd7k4Qo8ciQ@mail.gmail.com","threadId":"63221","inReplyTo":"20250425-pks-meson-benchmarks-v4-4-6b89555052c3@pks.im","subject":"Re: [PATCH v4 4/5] meson: wire up benchmarks","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-04-25T07:57:33Z","receivedAt":"2025-04-25T07:57:48Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt wrote:\n>\n> Wire up benchmarks in Meson. The setup is mostly the same as how we wire\n> up our tests. The only difference is that benchmarks get wired up via\n> the `benchmark()` option instead of via `test()`, which gives them a bit\n> of special treatment:\n>\n> - Benchmarks never run in parallel.\n>\n> - Benchmarks aren't run by default when tests are executed.\n>\n> - Meson does not inject the `MALLOC_PERTURB` environment variable.\n\n[...]\n\n> diff --git a/meson.build b/meson.build\n> index 410bbf93dad..2cd05a9b1ba 100644\n> --- a/meson.build\n> +++ b/meson.build\n> @@ -70,6 +70,12 @@\n> # # Execute single test interactively such that features like `debug ()` work.\n> # $ meson test -i --test-args='-ix' t1400-update-ref\n> #\n> +# # Execute all benchmarks.\n> +# $ meson test -i --benchmark\n> +#\n> +# # Execute single benchmark.\n> +# $ meson test -i --benchmark p0000-*\n> +#\n> # Test execution is parallelized by default and scales with the number of\n> # processor cores available. You can change the number of processes by passing\n> # the `-jN` flag to `meson test`.\n\nDoesn't the above give the impression that benchmarks are also\nparallelized by default while the commit message says they are not?\n"},{"id":"516768","messageId":"aAtBQ67Vveg1f598@pks.im","threadId":"63221","inReplyTo":"CAP8UFD0rq7Z=6PUPyPZOXaKyXzbWW7Eb0rDfoTxrd7k4Qo8ciQ@mail.gmail.com","subject":"Re: [PATCH v4 4/5] meson: wire up benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-25T08:01:07Z","receivedAt":"2025-04-25T08:01:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Apr 25, 2025 at 09:57:33AM +0200, Christian Couder wrote:\n> On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt wrote:\n> >\n> > Wire up benchmarks in Meson. The setup is mostly the same as how we wire\n> > up our tests. The only difference is that benchmarks get wired up via\n> > the `benchmark()` option instead of via `test()`, which gives them a bit\n> > of special treatment:\n> >\n> > - Benchmarks never run in parallel.\n> >\n> > - Benchmarks aren't run by default when tests are executed.\n> >\n> > - Meson does not inject the `MALLOC_PERTURB` environment variable.\n> \n> [...]\n> \n> > diff --git a/meson.build b/meson.build\n> > index 410bbf93dad..2cd05a9b1ba 100644\n> > --- a/meson.build\n> > +++ b/meson.build\n> > @@ -70,6 +70,12 @@\n> > # # Execute single test interactively such that features like `debug ()` work.\n> > # $ meson test -i --test-args='-ix' t1400-update-ref\n> > #\n> > +# # Execute all benchmarks.\n> > +# $ meson test -i --benchmark\n> > +#\n> > +# # Execute single benchmark.\n> > +# $ meson test -i --benchmark p0000-*\n> > +#\n> > # Test execution is parallelized by default and scales with the number of\n> > # processor cores available. You can change the number of processes by passing\n> > # the `-jN` flag to `meson test`.\n> \n> Doesn't the above give the impression that benchmarks are also\n> parallelized by default while the commit message says they are not?\n\nTo me it doesn't because it talks about test execution, not benchmark\nexecution. The comments for each of the invocations also discern between\ntests and benchmarks. I'm happy to hear about alternative ways to word\nthis though.\n\nPatrick\n"},{"id":"516769","messageId":"CAP8UFD1ovdQdfvLMvfPi8fz5S82n+8mfjK7ECqoQUMW2p-khjA@mail.gmail.com","threadId":"63221","inReplyTo":"20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im","subject":"Re: [PATCH v4 0/5] meson: wire up support for benchmarks","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-04-25T08:06:12Z","receivedAt":"2025-04-25T08:06:26Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt wrote:\n\n> Changes in v4:\n> - The patch series was rebased on top of f65182a99e5 (The ninth batch,\n> 2025-04-24). This is due to a conflict with ps/test-wo-perl-prereq.\n> - Introduce a 'benchmarks' option. This allows developers to require\n> benchmarks as desired. By default, we auto-detect whether the host\n> system has all dependencies available and enable or disable them\n> accordingly.\n\nNice!\n\n> - Report whether or not benchmarks are enabled via `summary()`.\n> - Our benchmarks depend on Perl, so add this dependency accordingly.\n\nNice too!\n\n[...]\n\n> Range-diff versus v3:\n>\n> 1: 2375a16c2c0 = 1: 174804805c2 t/perf: fix benchmarks with alternate repo formats\n> 2: 69782035246 = 2: 41faa3d9b3b t/perf: use configured PERL_PATH\n> 3: cca40e9dd0e = 3: 87ea3ab1a45 t/perf: fix benchmarks with out-of-tree builds\n\nIn https://lore.kernel.org/git/aAc8f52rJ8ATncVc@pks.im/ you said you\nwould try to paraphrase some explanations about how GIT_BUILD_DIR is\ncomputed, but it looks like there aren't any changes in the patch\nsince the previous version :-(\n"},{"id":"516770","messageId":"CAP8UFD37tRiCyZSzJwSbbL=D5b61TGOBMgM5qzeHOQaP-hzbow@mail.gmail.com","threadId":"63221","inReplyTo":"aAtBQ67Vveg1f598@pks.im","subject":"Re: [PATCH v4 4/5] meson: wire up benchmarks","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-04-25T08:09:18Z","receivedAt":"2025-04-25T08:09:31Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Apr 25, 2025 at 10:01 AM Patrick Steinhardt wrote:\n>\n> On Fri, Apr 25, 2025 at 09:57:33AM +0200, Christian Couder wrote:\n> > On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt wrote:\n> > >\n> > > Wire up benchmarks in Meson. The setup is mostly the same as how we wire\n> > > up our tests. The only difference is that benchmarks get wired up via\n> > > the `benchmark()` option instead of via `test()`, which gives them a bit\n> > > of special treatment:\n> > >\n> > > - Benchmarks never run in parallel.\n> > >\n> > > - Benchmarks aren't run by default when tests are executed.\n> > >\n> > > - Meson does not inject the `MALLOC_PERTURB` environment variable.\n> >\n> > [...]\n> >\n> > > diff --git a/meson.build b/meson.build\n> > > index 410bbf93dad..2cd05a9b1ba 100644\n> > > --- a/meson.build\n> > > +++ b/meson.build\n> > > @@ -70,6 +70,12 @@\n> > > # # Execute single test interactively such that features like `debug ()` work.\n> > > # $ meson test -i --test-args='-ix' t1400-update-ref\n> > > #\n> > > +# # Execute all benchmarks.\n> > > +# $ meson test -i --benchmark\n> > > +#\n> > > +# # Execute single benchmark.\n> > > +# $ meson test -i --benchmark p0000-*\n> > > +#\n> > > # Test execution is parallelized by default and scales with the number of\n> > > # processor cores available. You can change the number of processes by passing\n> > > # the `-jN` flag to `meson test`.\n> >\n> > Doesn't the above give the impression that benchmarks are also\n> > parallelized by default while the commit message says they are not?\n>\n> To me it doesn't because it talks about test execution, not benchmark\n> execution. The comments for each of the invocations also discern between\n> tests and benchmarks. I'm happy to hear about alternative ways to word\n> this though.\n\nMaybe something like:\n\n\"Test execution (but not benchmark execution) is parallelized by default ...\"\n"},{"id":"516771","messageId":"aAtHG38GPdbhOCop@pks.im","threadId":"63221","inReplyTo":"CAP8UFD1ovdQdfvLMvfPi8fz5S82n+8mfjK7ECqoQUMW2p-khjA@mail.gmail.com","subject":"Re: [PATCH v4 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-25T08:26:03Z","receivedAt":"2025-04-25T08:26:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Apr 25, 2025 at 10:06:12AM +0200, Christian Couder wrote:\n> On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt wrote:\n> > Range-diff versus v3:\n> >\n> > 1: 2375a16c2c0 = 1: 174804805c2 t/perf: fix benchmarks with alternate repo formats\n> > 2: 69782035246 = 2: 41faa3d9b3b t/perf: use configured PERL_PATH\n> > 3: cca40e9dd0e = 3: 87ea3ab1a45 t/perf: fix benchmarks with out-of-tree builds\n> \n> In https://lore.kernel.org/git/aAc8f52rJ8ATncVc@pks.im/ you said you\n> would try to paraphrase some explanations about how GIT_BUILD_DIR is\n> computed, but it looks like there aren't any changes in the patch\n> since the previous version :-(\n\nThere were, but not in this version:\n\n3: d03ca0c0630 ! 3: a66bc5aaf51 t/perf: fix benchmarks with out-of-tree builds\n @@ Commit message\n assumption breaks with both CMake and Meson, where the build directory\n can be located in an arbitrary place.\n\n - Adapt the script so that it works with out-of-tree builds. This prepares\n - us for wiring up benchmarks in Meson.\n + Adapt the script so that it works with out-of-tree builds. Most\n + importantly, this requires us to figure out the location of the build\n + directory:\n +\n + - When running benchmarks via our Makefile the build directory is the\n + same as the source directory. We already know to derive the test\n + directory (\"t/\") via `$(pwd)/..`, which works because we chdir into\n + \"t/perf\" before executing benchmarks. We can thus derive the build\n + directory by appending another \"/..\" to that path.\n +\n + - When running benchmarks via Meson the build directory is located at\n + an arbitrary location. The build system thus has to make the path\n + known by exporting the `GIT_BUILD_DIR` environment variable.\n +\n + This change prepares us for wiring up benchmarks in Meson.\n\nThe changes were already part of v3, so you have to look at the diff\nversus v2.\n\nPatrick\n"},{"id":"516772","messageId":"aAtHV9rk30j739Kx@pks.im","threadId":"63221","inReplyTo":"CAP8UFD37tRiCyZSzJwSbbL=D5b61TGOBMgM5qzeHOQaP-hzbow@mail.gmail.com","subject":"Re: [PATCH v4 4/5] meson: wire up benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-25T08:27:03Z","receivedAt":"2025-04-25T08:27:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Apr 25, 2025 at 10:09:18AM +0200, Christian Couder wrote:\n> On Fri, Apr 25, 2025 at 10:01 AM Patrick Steinhardt wrote:\n> >\n> > On Fri, Apr 25, 2025 at 09:57:33AM +0200, Christian Couder wrote:\n> > > On Fri, Apr 25, 2025 at 9:28 AM Patrick Steinhardt wrote:\n> > > >\n> > > > Wire up benchmarks in Meson. The setup is mostly the same as how we wire\n> > > > up our tests. The only difference is that benchmarks get wired up via\n> > > > the `benchmark()` option instead of via `test()`, which gives them a bit\n> > > > of special treatment:\n> > > >\n> > > > - Benchmarks never run in parallel.\n> > > >\n> > > > - Benchmarks aren't run by default when tests are executed.\n> > > >\n> > > > - Meson does not inject the `MALLOC_PERTURB` environment variable.\n> > >\n> > > [...]\n> > >\n> > > > diff --git a/meson.build b/meson.build\n> > > > index 410bbf93dad..2cd05a9b1ba 100644\n> > > > --- a/meson.build\n> > > > +++ b/meson.build\n> > > > @@ -70,6 +70,12 @@\n> > > > # # Execute single test interactively such that features like `debug ()` work.\n> > > > # $ meson test -i --test-args='-ix' t1400-update-ref\n> > > > #\n> > > > +# # Execute all benchmarks.\n> > > > +# $ meson test -i --benchmark\n> > > > +#\n> > > > +# # Execute single benchmark.\n> > > > +# $ meson test -i --benchmark p0000-*\n> > > > +#\n> > > > # Test execution is parallelized by default and scales with the number of\n> > > > # processor cores available. You can change the number of processes by passing\n> > > > # the `-jN` flag to `meson test`.\n> > >\n> > > Doesn't the above give the impression that benchmarks are also\n> > > parallelized by default while the commit message says they are not?\n> >\n> > To me it doesn't because it talks about test execution, not benchmark\n> > execution. The comments for each of the invocations also discern between\n> > tests and benchmarks. I'm happy to hear about alternative ways to word\n> > this though.\n> \n> Maybe something like:\n> \n> \"Test execution (but not benchmark execution) is parallelized by default ...\"\n\nOkay. I amended this locally, but will hold off sending another\niteration just for this change. Thanks!\n\nPatrick\n"},{"id":"516774","messageId":"CAP8UFD1y=rB1PC8M5aN4qz8EbEdiXuVOgoFaYG1tVw9NoV0rnA@mail.gmail.com","threadId":"63221","inReplyTo":"aAtHG38GPdbhOCop@pks.im","subject":"Re: [PATCH v4 0/5] meson: wire up support for benchmarks","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-04-25T08:33:16Z","receivedAt":"2025-04-25T08:33:30Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Apr 25, 2025 at 10:26 AM Patrick Steinhardt wrote:\n>\n> On Fri, Apr 25, 2025 at 10:06:12AM +0200, Christian Couder wrote:\n\n> > In https://lore.kernel.org/git/aAc8f52rJ8ATncVc@pks.im/ you said you\n> > would try to paraphrase some explanations about how GIT_BUILD_DIR is\n> > computed, but it looks like there aren't any changes in the patch\n> > since the previous version :-(\n>\n> There were, but not in this version:\n>\n> 3: d03ca0c0630 ! 3: a66bc5aaf51 t/perf: fix benchmarks with out-of-tree builds\n> @@ Commit message\n> assumption breaks with both CMake and Meson, where the build directory\n> can be located in an arbitrary place.\n>\n> - Adapt the script so that it works with out-of-tree builds. This prepares\n> - us for wiring up benchmarks in Meson.\n> + Adapt the script so that it works with out-of-tree builds. Most\n> + importantly, this requires us to figure out the location of the build\n> + directory:\n> +\n> + - When running benchmarks via our Makefile the build directory is the\n> + same as the source directory. We already know to derive the test\n> + directory (\"t/\") via `$(pwd)/..`, which works because we chdir into\n> + \"t/perf\" before executing benchmarks. We can thus derive the build\n> + directory by appending another \"/..\" to that path.\n> +\n> + - When running benchmarks via Meson the build directory is located at\n> + an arbitrary location. The build system thus has to make the path\n> + known by exporting the `GIT_BUILD_DIR` environment variable.\n> +\n> + This change prepares us for wiring up benchmarks in Meson.\n>\n> The changes were already part of v3, so you have to look at the diff\n> versus v2.\n\nOops, sorry for the noise.\n"},{"id":"516874","messageId":"20250428-pks-meson-benchmarks-v5-0-5010dd014d1d@pks.im","threadId":"63221","inReplyTo":"20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im","subject":"[PATCH v5 0/5] meson: wire up support for benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-28T07:30:45Z","receivedAt":"2025-04-28T07:30:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis small patch series implements support for running our benchmarks in\n\"t/perf\" via Meson. The series does not aim to replace \"t/perf/run\",\nwhich is more fully-featured and allows running benchmarks against\nmultiple different trees. Instead, this series only allows running the\nbenchmarks against the current tree. Users are thus expected to continue\nusing \"t/perf/run\" for more advanced usecases.\n\nChanges in v2:\n - Adapt \"aggregate.perl\" to use a \"/usr/bin/env perl\" shebang.\n - Link to v1: https://lore.kernel.org/r/20250331-pks-meson-benchmarks-v1-0-b2ace85616a3@pks.im\n\nChanges in v3:\n - Document how to run benchmarks in \"meson.build\".\n - Expand the message for the commit that enables out-of-tree\n benchmarking.\n - Link to v2: https://lore.kernel.org/r/20250414-pks-meson-benchmarks-v2-0-04377080a167@pks.im\n\nChanges in v4:\n - The patch series was rebased on top of f65182a99e5 (The ninth batch,\n 2025-04-24). This is due to a conflict with ps/test-wo-perl-prereq.\n - Introduce a 'benchmarks' option. This allows developers to require\n benchmarks as desired. By default, we auto-detect whether the host\n system has all dependencies available and enable or disable them\n accordingly.\n - Report whether or not benchmarks are enabled via `summary()`.\n - Our benchmarks depend on Perl, so add this dependency accordingly.\n - Link to v3: https://lore.kernel.org/r/20250422-pks-meson-benchmarks-v3-0-7aad68bac6fd@pks.im\n\nChanges in v5:\n - Clarify that benchmarks are not run in parallel by default.\n - Link to v4: https://lore.kernel.org/r/20250425-pks-meson-benchmarks-v4-0-6b89555052c3@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (5):\n t/perf: fix benchmarks with alternate repo formats\n t/perf: use configured PERL_PATH\n t/perf: fix benchmarks with out-of-tree builds\n meson: wire up benchmarks\n meson: wire up benchmarking options\n\n meson.build | 22 ++++++++++-----\n meson_options.txt | 8 ++++++\n t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++\n t/perf/aggregate.perl | 2 +-\n t/perf/perf-lib.sh | 32 ++++++++++++++++++---\n t/perf/run | 4 +--\n 6 files changed, 131 insertions(+), 14 deletions(-)\n\nRange-diff versus v4:\n\n1: bc19180b174 = 1: 03181f601f2 t/perf: fix benchmarks with alternate repo formats\n2: 3e128c09333 = 2: afb2fe1ce63 t/perf: use configured PERL_PATH\n3: f7adc20d14c = 3: a61e43c6cfb t/perf: fix benchmarks with out-of-tree builds\n4: 8b3ea380503 ! 4: e51b46296ed meson: wire up benchmarks\n @@ meson.build\n # # Execute single test interactively such that features like `debug ()` work.\n # $ meson test -i --test-args='-ix' t1400-update-ref\n #\n +-# Test execution is parallelized by default and scales with the number of\n +-# processor cores available. You can change the number of processes by passing\n +-# the `-jN` flag to `meson test`.\n +# # Execute all benchmarks.\n +# $ meson test -i --benchmark\n +#\n +# # Execute single benchmark.\n +# $ meson test -i --benchmark p0000-*\n +#\n - # Test execution is parallelized by default and scales with the number of\n - # processor cores available. You can change the number of processes by passing\n - # the `-jN` flag to `meson test`.\n ++# Test execution (but not benchmark execution) is parallelized by default and\n ++# scales with the number of processor cores available. You can change the\n ++# number of processes by passing the `-jN` flag to `meson test`.\n + #\n + # 4. Install the Git distribution. Again, this can be done via Meson, Ninja or\n + # Samurai:\n @@ meson.build: git = find_program('git', dirs: program_path, native: true, required: false)\n sed = find_program('sed', dirs: program_path, native: true)\n shell = find_program('sh', dirs: program_path, native: true)\n5: a0b06cc7017 = 5: 20f9dd86d66 meson: wire up benchmarking options\n\n---\nbase-commit: f65182a99e545d2f2bc22e6c1c2da192133b16a3\nchange-id: 20250328-pks-meson-benchmarks-a8fac5f69467\n\n"},{"id":"516875","messageId":"20250428-pks-meson-benchmarks-v5-1-5010dd014d1d@pks.im","threadId":"63221","inReplyTo":"20250428-pks-meson-benchmarks-v5-0-5010dd014d1d@pks.im","subject":"[PATCH v5 1/5] t/perf: fix benchmarks with alternate repo formats","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-28T07:30:46Z","receivedAt":"2025-04-28T07:30:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Many of our benchmarks operate on a user-defined repository that we copy\nover before running the benchmarked logic. To keep unintentional side\neffects caused by on-disk state at bay we skip copying some files. This\nincludes for example hooks, but also the repo's configuration.\n\nIt is quite sensible to not copy over the configuration, as it is quite\neasy to inadvertently carry over configuration that may significantly\nimpact the performance measurements. But we cannot fully ignore the\nconfiguration either, as it may contain information about the repository\nformat. This will cause failures when for example using a repository\nwith SHA256 object format or the reftable ref format.\n\nFix the issue by parsing the reference and object formats from the\nsource repository and passing them to git-init(1).\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 8ab6d9c4694..1a9a51ca3cc 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -98,6 +98,8 @@ test_perf_create_repo_from () {\n \tsource_git=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-dir)\"\n \tobjects_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-path objects)\"\n \tcommon_dir=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --git-common-dir)\"\n+\trefformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-ref-format)\"\n+\tobjectformat=\"$(\"$MODERN_GIT\" -C \"$source\" rev-parse --show-object-format)\"\n \tmkdir -p \"$repo/.git\"\n \t(\n \t\tcd \"$source\" &&\n@@ -114,7 +116,7 @@ test_perf_create_repo_from () {\n \t) &&\n \t(\n \t\tcd \"$repo\" &&\n-\t\t\"$MODERN_GIT\" init -q &&\n+\t\t\"$MODERN_GIT\" init -q --ref-format=\"$refformat\" --object-format=\"$objectformat\" &&\n \t\ttest_perf_do_repo_symlink_config_ &&\n \t\tmv .git/hooks .git/hooks-disabled 2>/dev/null &&\n \t\tif test -f .git/index.lock\n\n-- \n2.49.0.967.g6a0df3ecc3.dirty\n\n"},{"id":"516876","messageId":"20250428-pks-meson-benchmarks-v5-2-5010dd014d1d@pks.im","threadId":"63221","inReplyTo":"20250428-pks-meson-benchmarks-v5-0-5010dd014d1d@pks.im","subject":"[PATCH v5 2/5] t/perf: use configured PERL_PATH","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-28T07:30:47Z","receivedAt":"2025-04-28T07:30:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our benchmarks use a couple of Perl scripts to compute results. These\nPerl scripts get executed directly, and as the shebang is hardcoded to\n\"/usr/bin/perl\" this will fail on any system where the Perl interpreter\nis located in a different path.\n\nOur build infrastructure already lets users configure the location of\nPerl, which ultimately gets written into the GIT-BUILD-OPTIONS file.\nThis file is being sourced by \"test-lib.sh\", and consequently we already\nhave the \"PERL_PATH\" variable available that contains its configured\nlocation.\n\nUse \"PERL_PATH\" to execute Perl scripts, which makes them work on more\nesoteric systems like NixOS. Furthermore, adapt the shebang to use\nenv(1) to execute Perl so that users who have Perl in PATH, but in a\nnon-standard location can execute the script directly.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/aggregate.perl | 2 +-\n t/perf/perf-lib.sh | 4 ++--\n t/perf/run | 4 ++--\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl\nindex 575d2000cc1..1791c7528a9 100755\n--- a/t/perf/aggregate.perl\n+++ b/t/perf/aggregate.perl\n@@ -1,4 +1,4 @@\n-#!/usr/bin/perl\n+#!/usr/bin/env perl\n \n use lib '../../perl/build/lib';\n use strict;\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 1a9a51ca3cc..4173eee4def 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -276,7 +276,7 @@ test_perf_ () {\n \telse\n \t\ttest_ok_ \"$1\"\n \tfi\n-\t\"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n+\t\"$PERL_PATH\" \"$TEST_DIRECTORY\"/perf/min_time.perl test_time.* >\"$base\".result\n \trm test_time.*\n }\n \n@@ -324,7 +324,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\ndiff --git a/t/perf/run b/t/perf/run\nindex 486ead21980..073bcb2afff 100755\n--- a/t/perf/run\n+++ b/t/perf/run\n@@ -192,10 +192,10 @@ run_subsection () {\n \n \tif test -z \"$GIT_PERF_SEND_TO_CODESPEED\"\n \tthen\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $codespeed_opt \"$@\"\n \telse\n \t\tjson_res_file=\"\"$TEST_RESULTS_DIR\"/$GIT_PERF_SUBSECTION/aggregate.json\"\n-\t\t./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n+\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" --codespeed \"$@\" | tee \"$json_res_file\"\n \t\tsend_data_url=\"$GIT_PERF_SEND_TO_CODESPEED/result/add/json/\"\n \t\tcurl -v --request POST --data-urlencode \"json=$(cat \"$json_res_file\")\" \"$send_data_url\"\n \tfi\n\n-- \n2.49.0.967.g6a0df3ecc3.dirty\n\n"},{"id":"516877","messageId":"20250428-pks-meson-benchmarks-v5-3-5010dd014d1d@pks.im","threadId":"63221","inReplyTo":"20250428-pks-meson-benchmarks-v5-0-5010dd014d1d@pks.im","subject":"[PATCH v5 3/5] t/perf: fix benchmarks with out-of-tree builds","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-28T07:30:48Z","receivedAt":"2025-04-28T07:30:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"perf-lib.sh\" script is sourced by all of our benchmarking suites to\nmake available common infrastructure. The script assumes that build and\nsource directory are the same, which works for our Makefile. But the\nassumption breaks with both CMake and Meson, where the build directory\ncan be located in an arbitrary place.\n\nAdapt the script so that it works with out-of-tree builds. Most\nimportantly, this requires us to figure out the location of the build\ndirectory:\n\n - When running benchmarks via our Makefile the build directory is the\n same as the source directory. We already know to derive the test\n directory (\"t/\") via `$(pwd)/..`, which works because we chdir into\n \"t/perf\" before executing benchmarks. We can thus derive the build\n directory by appending another \"/..\" to that path.\n\n - When running benchmarks via Meson the build directory is located at\n an arbitrary location. The build system thus has to make the path\n known by exporting the `GIT_BUILD_DIR` environment variable.\n\nThis change prepares us for wiring up benchmarks in Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/perf-lib.sh | 26 ++++++++++++++++++++++++--\n 1 file changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh\nindex 4173eee4def..5406557b7ca 100644\n--- a/t/perf/perf-lib.sh\n+++ b/t/perf/perf-lib.sh\n@@ -25,7 +25,29 @@ TEST_OUTPUT_DIRECTORY=$(pwd)\n TEST_NO_CREATE_REPO=t\n TEST_NO_MALLOC_CHECK=t\n \n-. ../test-lib.sh\n+# While test-lib.sh computes the build directory for us, we also have to do the\n+# same thing in order to locate the script via GIT-BUILD-OPTIONS in the first\n+# place.\n+GIT_BUILD_DIR=\"${GIT_BUILD_DIR:-$TEST_DIRECTORY/..}\"\n+if test -f \"$GIT_BUILD_DIR/GIT-BUILD-DIR\"\n+then\n+\tGIT_BUILD_DIR=\"$(cat \"$GIT_BUILD_DIR/GIT-BUILD-DIR\")\" || exit 1\n+\t# On Windows, we must convert Windows paths lest they contain a colon\n+\tcase \"$(uname -s)\" in\n+\t*MINGW*)\n+\t\tGIT_BUILD_DIR=\"$(cygpath -au \"$GIT_BUILD_DIR\")\"\n+\t\t;;\n+\tesac\n+fi\n+\n+if test ! -f \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+then\n+\techo >&2 'error: GIT-BUILD-OPTIONS missing (has Git been built?).'\n+\texit 1\n+fi\n+\n+. \"$GIT_BUILD_DIR\"/GIT-BUILD-OPTIONS\n+. \"$GIT_SOURCE_DIR\"/t/test-lib.sh\n \n unset GIT_CONFIG_NOSYSTEM\n GIT_CONFIG_SYSTEM=\"$TEST_DIRECTORY/perf/config\"\n@@ -324,7 +346,7 @@ test_at_end_hook_ () {\n \tif test -z \"$GIT_PERF_AGGREGATING_LATER\"; then\n \t\t(\n \t\t\tcd \"$TEST_DIRECTORY\"/perf &&\n-\t\t\t\"$PERL_PATH\" ./aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n+\t\t\t\"$PERL_PATH\" \"$GIT_SOURCE_DIR\"/t/perf/aggregate.perl --results-dir=\"$TEST_RESULTS_DIR\" $(basename \"$0\")\n \t\t)\n \tfi\n }\n\n-- \n2.49.0.967.g6a0df3ecc3.dirty\n\n"},{"id":"516878","messageId":"20250428-pks-meson-benchmarks-v5-4-5010dd014d1d@pks.im","threadId":"63221","inReplyTo":"20250428-pks-meson-benchmarks-v5-0-5010dd014d1d@pks.im","subject":"[PATCH v5 4/5] meson: wire up benchmarks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-28T07:30:49Z","receivedAt":"2025-04-28T07:30:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up benchmarks in Meson. The setup is mostly the same as how we wire\nup our tests. The only difference is that benchmarks get wired up via\nthe `benchmark()` option instead of via `test()`, which gives them a bit\nof special treatment:\n\n - Benchmarks never run in parallel.\n\n - Benchmarks aren't run by default when tests are executed.\n\n - Meson does not inject the `MALLOC_PERTURB` environment variable.\n\nUsing benchmarks is quite simple:\n\n ```\n $ meson setup build\n # Run all benchmarks.\n $ meson test -C build --benchmark\n # Run a specific benchmark.\n $ meson test -C build --benchmark p0000-*\n ```\n\nOther than that the usual command line arguments accepted when running\ntests are also accepted when running benchmarks.\n\nNote that the benchmarking target is somewhat limited because it will\nonly run benchmarks for the current build. Other use cases, like running\nbenchmarks against multiple different versions of Git, are not currently\nsupported. Users should continue to use \"t/perf/run\" for those use\ncases. The script should get extended at one point in time to support\nMeson, but this is outside of the scope of this series.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 16 +++++++++---\n meson_options.txt | 2 ++\n t/meson.build | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 91 insertions(+), 4 deletions(-)\n\ndiff --git a/meson.build b/meson.build\nindex 410bbf93dad..939a2eae326 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -70,9 +70,15 @@\n # # Execute single test interactively such that features like `debug ()` work.\n # $ meson test -i --test-args='-ix' t1400-update-ref\n #\n-# Test execution is parallelized by default and scales with the number of\n-# processor cores available. You can change the number of processes by passing\n-# the `-jN` flag to `meson test`.\n+# # Execute all benchmarks.\n+# $ meson test -i --benchmark\n+#\n+# # Execute single benchmark.\n+# $ meson test -i --benchmark p0000-*\n+#\n+# Test execution (but not benchmark execution) is parallelized by default and\n+# scales with the number of processor cores available. You can change the\n+# number of processes by passing the `-jN` flag to `meson test`.\n #\n # 4. Install the Git distribution. Again, this can be done via Meson, Ninja or\n # Samurai:\n@@ -235,6 +241,7 @@ git = find_program('git', dirs: program_path, native: true, required: false)\n sed = find_program('sed', dirs: program_path, native: true)\n shell = find_program('sh', dirs: program_path, native: true)\n tar = find_program('tar', dirs: program_path, native: true)\n+time = find_program('time', dirs: program_path, required: get_option('benchmarks'))\n \n target_shell = find_program('sh', dirs: program_path, native: false)\n \n@@ -836,7 +843,7 @@ endif\n # features. It is optional if you want to neither execute tests nor use any of\n # these optional features.\n perl_required = get_option('perl')\n-if get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []\n+if get_option('benchmarks').enabled() or get_option('gitweb').enabled() or 'netrc' in get_option('credential_helpers') or get_option('docs') != []\n perl_required = true\n endif\n \n@@ -2082,6 +2089,7 @@ meson.add_dist_script(\n )\n \n summary({\n+ 'benchmarks': get_option('tests') and perl.found() and time.found(),\n 'curl': curl.found(),\n 'expat': expat.found(),\n 'gettext': intl.found(),\ndiff --git a/meson_options.txt b/meson_options.txt\nindex 8ac30a52231..7f5bca5c029 100644\n--- a/meson_options.txt\n+++ b/meson_options.txt\n@@ -101,6 +101,8 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto\n description: 'Which backend to use to generate documentation.')\n \n # Testing.\n+option('benchmarks', type: 'feature', value: 'auto',\n+ description: 'Enable benchmarks. This requires Perl and GNU time.')\n option('coccinelle', type: 'feature', value: 'auto',\n description: 'Provide a coccicheck target that generates a Coccinelle patch.')\n option('tests', type: 'boolean', value: true,\ndiff --git a/t/meson.build b/t/meson.build\nindex bfb744e8863..59438b06266 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -1097,11 +1097,71 @@ integration_tests = [\n 't9903-bash-prompt.sh',\n ]\n \n+benchmarks = [\n+ 'perf/p0000-perf-lib-sanity.sh',\n+ 'perf/p0001-rev-list.sh',\n+ 'perf/p0002-read-cache.sh',\n+ 'perf/p0003-delta-base-cache.sh',\n+ 'perf/p0004-lazy-init-name-hash.sh',\n+ 'perf/p0005-status.sh',\n+ 'perf/p0006-read-tree-checkout.sh',\n+ 'perf/p0007-write-cache.sh',\n+ 'perf/p0008-odb-fsync.sh',\n+ 'perf/p0071-sort.sh',\n+ 'perf/p0090-cache-tree.sh',\n+ 'perf/p0100-globbing.sh',\n+ 'perf/p1006-cat-file.sh',\n+ 'perf/p1400-update-ref.sh',\n+ 'perf/p1450-fsck.sh',\n+ 'perf/p1451-fsck-skip-list.sh',\n+ 'perf/p1500-graph-walks.sh',\n+ 'perf/p2000-sparse-operations.sh',\n+ 'perf/p3400-rebase.sh',\n+ 'perf/p3404-rebase-interactive.sh',\n+ 'perf/p4000-diff-algorithms.sh',\n+ 'perf/p4001-diff-no-index.sh',\n+ 'perf/p4002-diff-color-moved.sh',\n+ 'perf/p4205-log-pretty-formats.sh',\n+ 'perf/p4209-pickaxe.sh',\n+ 'perf/p4211-line-log.sh',\n+ 'perf/p4220-log-grep-engines.sh',\n+ 'perf/p4221-log-grep-engines-fixed.sh',\n+ 'perf/p5302-pack-index.sh',\n+ 'perf/p5303-many-packs.sh',\n+ 'perf/p5304-prune.sh',\n+ 'perf/p5310-pack-bitmaps.sh',\n+ 'perf/p5311-pack-bitmaps-fetch.sh',\n+ 'perf/p5312-pack-bitmaps-revs.sh',\n+ 'perf/p5313-pack-objects.sh',\n+ 'perf/p5314-name-hash.sh',\n+ 'perf/p5326-multi-pack-bitmaps.sh',\n+ 'perf/p5332-multi-pack-reuse.sh',\n+ 'perf/p5333-pseudo-merge-bitmaps.sh',\n+ 'perf/p5550-fetch-tags.sh',\n+ 'perf/p5551-fetch-rescan.sh',\n+ 'perf/p5600-partial-clone.sh',\n+ 'perf/p5601-clone-reference.sh',\n+ 'perf/p6100-describe.sh',\n+ 'perf/p6300-for-each-ref.sh',\n+ 'perf/p7000-filter-branch.sh',\n+ 'perf/p7102-reset.sh',\n+ 'perf/p7300-clean.sh',\n+ 'perf/p7519-fsmonitor.sh',\n+ 'perf/p7527-builtin-fsmonitor.sh',\n+ 'perf/p7810-grep.sh',\n+ 'perf/p7820-grep-engines.sh',\n+ 'perf/p7821-grep-engines-fixed.sh',\n+ 'perf/p7822-grep-perl-character.sh',\n+ 'perf/p9210-scalar.sh',\n+ 'perf/p9300-fast-import-export.sh',\n+]\n+\n # Sanity check that we are not missing any tests present in 't/'. This check\n # only runs once at configure time and is thus best-effort, only. It is\n # sufficient to catch missing test suites in our CI though.\n foreach glob, tests : {\n 't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,\n+ 'perf/p[0-9][0-9][0-9][0-9]-*.sh': benchmarks,\n 'unit-tests/t-*.c': unit_test_programs,\n 'unit-tests/u-*.c': clar_test_suites,\n }\n@@ -1153,3 +1213,20 @@ foreach integration_test : integration_tests\n timeout: 0,\n )\n endforeach\n+\n+if perl.found() and time.found()\n+ benchmark_environment = test_environment\n+ benchmark_environment.set('GTIME', time.full_path())\n+\n+ foreach benchmark : benchmarks\n+ benchmark(fs.stem(benchmark), shell,\n+ args: [\n+ fs.name(benchmark),\n+ ],\n+ workdir: meson.current_source_dir() / 'perf',\n+ env: benchmark_environment,\n+ depends: test_dependencies + bin_wrappers,\n+ timeout: 0,\n+ )\n+ endforeach\n+endif\n\n-- \n2.49.0.967.g6a0df3ecc3.dirty\n\n"},{"id":"516879","messageId":"20250428-pks-meson-benchmarks-v5-5-5010dd014d1d@pks.im","threadId":"63221","inReplyTo":"20250428-pks-meson-benchmarks-v5-0-5010dd014d1d@pks.im","subject":"[PATCH v5 5/5] meson: wire up benchmarking options","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-28T07:30:50Z","receivedAt":"2025-04-28T07:30:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Wire up a couple of benchmarking options that we end up writing into our\n\"GIT-BUILD-OPTIONS\" file. These options allow users to control how\nexactly benchmarks are executed.\n\nNote that neither `GIT_PERF_MAKE_COMMAND` nor `GIT_PERF_MAKE_OPTS` are\nexposed as a build option. Those options are used by \"t/perf/run\", which\nis not used by Meson.\n\nSigned-off-by: Patrick Steinhardt \n---\n meson.build | 6 +++---\n meson_options.txt | 6 ++++++\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/meson.build b/meson.build\nindex 939a2eae326..d06f82ff9f0 100644\n--- a/meson.build\n+++ b/meson.build\n@@ -706,11 +706,11 @@ builtin_sources += custom_target(\n # build options to our tests.\n build_options_config = configuration_data()\n build_options_config.set('GIT_INTEROP_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_LARGE_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_LARGE_REPO', get_option('benchmark_large_repo'))\n build_options_config.set('GIT_PERF_MAKE_COMMAND', '')\n build_options_config.set('GIT_PERF_MAKE_OPTS', '')\n-build_options_config.set('GIT_PERF_REPEAT_COUNT', '')\n-build_options_config.set('GIT_PERF_REPO', '')\n+build_options_config.set_quoted('GIT_PERF_REPEAT_COUNT', get_option('benchmark_repeat_count').to_string())\n+build_options_config.set_quoted('GIT_PERF_REPO', get_option('benchmark_repo'))\n build_options_config.set('GIT_TEST_CMP_USE_COPIED_CONTEXT', '')\n build_options_config.set('GIT_TEST_INDEX_VERSION', '')\n build_options_config.set('GIT_TEST_OPTS', '')\ndiff --git a/meson_options.txt b/meson_options.txt\nindex 7f5bca5c029..8547c0eb47f 100644\n--- a/meson_options.txt\n+++ b/meson_options.txt\n@@ -103,6 +103,12 @@ option('docs_backend', type: 'combo', choices: ['asciidoc', 'asciidoctor', 'auto\n # Testing.\n option('benchmarks', type: 'feature', value: 'auto',\n description: 'Enable benchmarks. This requires Perl and GNU time.')\n+option('benchmark_repo', type: 'string', value: '',\n+ description: 'Repository to copy for the performance tests. Should be at least the size of the Git repository.')\n+option('benchmark_large_repo', type: 'string', value: '',\n+ description: 'Large repository to copy for the performance tests. Should be at least the size of the Linux repository.')\n+option('benchmark_repeat_count', type: 'integer', value: 3,\n+ description: 'Number of times a test should be repeated for best-of-N measurements.')\n option('coccinelle', type: 'feature', value: 'auto',\n description: 'Provide a coccicheck target that generates a Coccinelle patch.')\n option('tests', type: 'boolean', value: true,\n\n-- \n2.49.0.967.g6a0df3ecc3.dirty\n\n"}]}