Re: [RFC PATCH 1/2] Add C TAP harness

From: Felipe Contreras <felipe.contreras@gmail.com>
Date: May 2, 2023, 18:11 UTC
Message-ID: <6451523ec8b47_1ba2d29427@chronos.notmuch>
In-Reply-To: <230502.86sfcehecl.gmgdl@evledraar.gmail.com>

Ævar Arnfjörð Bjarmason wrote:

Show 37 quoted lines

> On Tue, May 02 2023, Felipe Contreras wrote:
> 
> > Phillip Wood wrote:
> >
> >> Unfortunately this library doesn't seem to offer any of those features. 
> >> It does support a lazy test plan but uses atexit() so will not detect if 
> >> the test program exits before all the tests have run.
> >
> > I think there's a fundamental misunderstanding of how we use TAP.
> >
> > If a program generates this output:
> >
> >   1..3
> >   ok 1 - test 1
> >   ok 2 - test 2
> >
> > That's clearly not complete. It shouldn't be the job a test script to check for
> > those cases.
> >
> > If you run the programm through a TAP harness such as prove, you get:
> >
> >   foo.t .. Failed 1/3 subtests 
> >
> >   Test Summary Report
> >   -------------------
> >   foo.t (Wstat: 0 Tests: 2 Failed: 0)
> >     Parse errors: Bad plan.  You planned 3 tests but ran 2.
> >   Files=1, Tests=2,  0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
> >   Result: FAIL
> >
> > Why do we bother generaing a TAP output if we are not going to take advantage
> > of it?
> 
> (As the person who added the TAP output to git.git)
> 
> Yeah, we could do the "plan ahead", but it would mean that tests would
> need to pre-declare the number of tests they have.

I'm not advocating for planning ahead.

In the case of the shell testing framework it makes sense to not plan ahead and just print the test plan at the end with test_done(), as that's very convenient.

But $subject is a new proposal for C unit tests with a test plan specified ahead, therefore if the program exits before all the tests have run, that should be obvious from the output. I'm not saying that is good or desirable, merely that it is the case.

I personaly don't even think C is the way to write unit tests.

Show 32 quoted lines

> In the Perl world that's the usual pattern, but as it involves having a:
> 
> 	plan tests => 123;
> 
> At the top of the file that's guaranteed to give you merge conflicts if
> two topics add/remove tests in different parts of the file.
> 
> It also doesn't work well in cases where the number of tests is hard to
> determine in advance, i.e. when they're determined programatically.
> 
> I don't think there's any practical downside to using the "test_done"
> pattern to print a plan at the end as far as missing tests go.
> 
> There *is* a minor practical downside to it though, which is that we'll
> get output like "25/?" or whatever, but not "25/100", as we don't know
> yet that we've got a total of 100 tests.
> 
> But I think that's a minor drawback, and really only matters if you're
> eyeballing the prove(1) output of a very slow test as it scrolls by.
> 
> I think on balance the "plan at the end" approach we're using now is
> much better, and would also be better in a future (or hypothetical)
> pure-C test framework.
>
> Well, there are ways to avoid the painful conflicts, e.g. by mandating
> that all tests are driven by callbacks in an array or something, so then
> you won't get merge conflicts on the "plan" line, as it'll just be "the
> number of tests is the number of items in this array".
> 
> But such a thing is painful to mandate, and has its own drawbacks,
> i.e. not being able to do a "test" at anything less than a function
> callback level of granularity.

If you are using an old-school language that doesn't have support for anonymous functions, then yes, that is pretty much your only nice maintenable option.

But the list of languages that don't have support for that is rather small, it's pretty much limited to shell, C, Fortran and COBOL. Even Perl has support for anonymous functions.

In all modern languages you can write test cases using anonymous functions such as (JavaScript's ava [1]):

  test('test case', t => {
    t.is('actual', 'expected');
  });

This can be done in Perl as well (see Test::Functional [2]).

So at the end of the script it is known how many test cases there are *before* running any of them.

This is precisely the approach I followed in my new testing framework, which I sent as a proposal to the ml [3].

  require 'testox'

  test('one') { ok(true) }
  test('two') { ok(true) }
  test('three') { ok(true) }
  test('four') { ok(true) }

  puts '# num tests: %s' % TestOx.tests.length

No test case is run until the script ends in at_exit:

  # num tests: 4
  1..4
  ok 1 - one
  ok 2 - two
  ok 3 - three
  ok 4 - four

There's no need to specify the number of test cases.

I don't see why we insist in using ancient languages like C and shell for things that neither C nor shell are good at. Other languages exist in 2023. Other languages exist for a reason.

Cheers.

[1] https://github.com/avajs/ava [2] https://metacpan.org/pod/Test::Functional [3] https://lore.kernel.org/git/20230502041113.103385-1-felipe.contreras@gmail.com/T/#t

-- 
Felipe Contreras