From: Patrick Steinhardt <ps@pks.im>
Date: Thu, 15 May 2025 08:20:49 GMT
Subject: Re: [PATCH 4/4] meson: parse TAP output generated by our tests
Message-ID: <aCWj4Xi7SuG22_kK@pks.im>
In-Reply-To: <CAOLa=ZSxPtMQdb8Hne6-WMEd9gKwWUC5hnCUCAaS2FfifqZHHA@mail.gmail.com>

On Thu, May 15, 2025 at 12:48:23AM -0700, Karthik Nayak wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > diff --git a/meson.build b/meson.build
> > index 94bd525dd7b..cd8df189d79 100644
> > --- a/meson.build
> > +++ b/meson.build
> > @@ -2031,6 +2031,14 @@ if get_option('tests')
> >      'timeout': 0,
> >    }
> >
> > +  # The TAP protocol was already understood by previous versions of Meson, but
> > +  # it was incompatible with the `meson test --interactive` flag.
> > +  if meson.version().version_compare('>=1.8.0')
> > +    test_kwargs += {
> > +      'protocol': 'tap',
> > +    }
> > +  endif
> > +
> >
> 
> The change itself looks good. But I do have a question about this:
> 
[snip]
> 
> Shouldn't the '--interactive' flag also produce 'Ok: 1'. Instead it is
> printing out 'Ignored: 1'. This is while I was testing on your series.
> Seems to be fine on master.

The answer is unfortunately "no". In interactive mode the expectation is
that the user will, well, interact with the test. This has two
consequences:

  - The standard streams will be directly connected to the user's
    console. This has the consequence that Meson won't be able to parse
    the generated output anymore, and thus it labels the tests as
    "ignored" because it cannot derive their status.

  - Even if Meson intercepted the output it very likely wouldn't be able
    to parse it. After all, we're in interactive mode, which means that
    the user may be directly communicating with the tests. E.g. if you
    use `test_pause`, then you'll be dropped to a shell and communicate
    with it. The consequence is that the output won't follow proper TAP
    format anymore.

So it's basically impossible to parse TAP in interactive mode, which is
why Meson then ignores the results. If all you want is to see verbose
output you can do that with `meson test -v`. But interactive mode really
indicates that you _want_ to get hands-on with the tests.

Patrick