Re: [PATCH 4/8] meson: detect missing tests at configure time

From: Toon Claes <toon@iotcl.com>
Date: Dec 13, 2024, 09:58 UTC
Message-ID: <87bjxfj44o.fsf@iotcl.com>
In-Reply-To: <20241211-pks-meson-ci-v1-4-28d18b494374@pks.im>

Patrick Steinhardt <ps@pks.im> writes:

Show 39 quoted lines

> It is quite easy for the list of integration tests to go out-of-sync
> without anybody noticing. Introduce a new configure-time check that
> verifies that all tests are wired up properly.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  t/meson.build | 36 ++++++++++++++++++++++++++++++++++++
>  1 file changed, 36 insertions(+)
>
> diff --git a/t/meson.build b/t/meson.build
> index 9e676e69363ed6311426500d98fe281e30d26bcb..f1fbc6ae179079f4d5d86f9a60956fad84d0495c 100644
> --- a/t/meson.build
> +++ b/t/meson.build
> @@ -1092,6 +1092,42 @@ integration_tests = [
>    't9903-bash-prompt.sh',
>  ]
>  
> +# Sanity check that we are not missing any tests present in 't/'. This check
> +# only runs once at configure time and is thus best-effort, only. It is
> +# sufficient to catch missing test suites in our CI though.
> +foreach glob, tests : {
> +  't[0-9][0-9][0-9][0-9]-*.sh': integration_tests,
> +  'unit-tests/t-*.c': unit_test_programs,
> +  'unit-tests/u-*.c': clar_test_suites,
> +}
> +  actual_tests = run_command(shell, '-c', 'ls ' + glob,
> +    check: true,
> +    env: script_environment,
> +  ).stdout().strip().split('\n')
> +
> +  if tests != actual_tests
> +    missing_tests = [ ]
> +    foreach actual_test : actual_tests
> +      if actual_test not in tests
> +        missing_tests += actual_test
> +      endif
> +    endforeach
> +    if missing_tests.length() > 0
> +      error('Missing tests:\n\n - ' + '\n - '.join(missing_tests))

This gives nice output:

    $ touch t/unit-tests/u-bar.c t/unit-tests/u-foo.c

    $ meson setup builddir --reconfigure

    The Meson build system
    Version: 1.4.1
    [snip]
    Configuring update.sample using configuration
    Configuring exclude using configuration

    t/meson.build:1116:6: ERROR: Problem encountered: Missing tests:

     - unit-tests/u-bar.c
     - unit-tests/u-foo.c

    A full log can be found at git/builddir/meson-logs/meson-log.txt

But I think the error message is a little bit confusing. It sounds like the test file is missing, but it's the configuration of the test that is missing.

Now I've realized it hard to write a good error message here. But I would suggest something like "Tests files found, but not configured".

Show 10 quoted lines

> +    endif
> +
> +    superfluous_tests = [ ]
> +    foreach integration_test : tests
> +      if integration_test not in actual_tests
> +        superfluous_tests += integration_test
> +      endif
> +    endforeach
> +    if superfluous_tests.length() > 0
> +      error('Superfluous tests:\n\n - ' + '\n - '.join(superfluous_tests))

Also here I would suggest different error message, maybe something like "Tests configured, but file not found"

-- 
Toon