Re: [RFC PATCH 1/2] Add C TAP harness

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: May 2, 2023, 16:39 UTC
Message-ID: <230502.86sfcehecl.gmgdl@evledraar.gmail.com>
In-Reply-To: <6451324ed84e2_1ba2d29454@chronos.notmuch>

On Tue, May 02 2023, Felipe Contreras wrote:

Show 30 quoted lines

> Phillip Wood wrote:
>
>> Unfortunately this library doesn't seem to offer any of those features. 
>> It does support a lazy test plan but uses atexit() so will not detect if 
>> the test program exits before all the tests have run.
>
> I think there's a fundamental misunderstanding of how we use TAP.
>
> If a program generates this output:
>
>   1..3
>   ok 1 - test 1
>   ok 2 - test 2
>
> That's clearly not complete. It shouldn't be the job a test script to check for
> those cases.
>
> If you run the programm through a TAP harness such as prove, you get:
>
>   foo.t .. Failed 1/3 subtests 
>
>   Test Summary Report
>   -------------------
>   foo.t (Wstat: 0 Tests: 2 Failed: 0)
>     Parse errors: Bad plan.  You planned 3 tests but ran 2.
>   Files=1, Tests=2,  0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
>   Result: FAIL
>
> Why do we bother generaing a TAP output if we are not going to take advantage
> of it?

(As the person who added the TAP output to git.git)

Yeah, we could do the "plan ahead", but it would mean that tests would need to pre-declare the number of tests they have.

In the Perl world that's the usual pattern, but as it involves having a:

	plan tests => 123;

At the top of the file that's guaranteed to give you merge conflicts if two topics add/remove tests in different parts of the file.

It also doesn't work well in cases where the number of tests is hard to determine in advance, i.e. when they're determined programatically.

I don't think there's any practical downside to using the "test_done" pattern to print a plan at the end as far as missing tests go.

There *is* a minor practical downside to it though, which is that we'll get output like "25/?" or whatever, but not "25/100", as we don't know yet that we've got a total of 100 tests.

But I think that's a minor drawback, and really only matters if you're eyeballing the prove(1) output of a very slow test as it scrolls by.

I think on balance the "plan at the end" approach we're using now is much better, and would also be better in a future (or hypothetical) pure-C test framework.

Well, there are ways to avoid the painful conflicts, e.g. by mandating that all tests are driven by callbacks in an array or something, so then you won't get merge conflicts on the "plan" line, as it'll just be "the number of tests is the number of items in this array".

But such a thing is painful to mandate, and has its own drawbacks, i.e. not being able to do a "test" at anything less than a function callback level of granularity.