From: Phillip Wood <phillip.wood123@gmail.com>
Date: Wed, 10 May 2023 08:18:35 GMT
Subject: Re: [RFC PATCH 1/2] Add C TAP harness
Message-ID: <3808fc35-6d3d-5a91-3eac-9428ff0e8045@gmail.com>
In-Reply-To: <230502.86wn1qhemd.gmgdl@evledraar.gmail.com>

Hi Ævar

On 02/05/2023 17:34, Ævar Arnfjörð Bjarmason wrote:
> 
> On Thu, Apr 27 2023, Phillip Wood wrote:
> 
>> Hi Calvin
>>
>> On 27/04/2023 18:50, Calvin Wan wrote:
>>> Introduces the C TAP harness from https://github.com/rra/c-tap-harness/
>>> There is also more complete documentation at
>>> https://www.eyrie.org/~eagle/software/c-tap-harness/
>>
>> I'm afraid this reply is rather briefer than I'd like but I'm short of
>> time and about to go off-list for a couple of weeks. My ideal unit
>> test library would
>>
>>   - print the file and line number of failed assertions
>>   - allow the test plan to be omitted by calling test_done() at the end
>>     of the test file as we do in our main test suite.
>>   - support the TODO directive
>>   - allow named tests (this maybe more trouble that it is worth as I
>>     think it inevitably leads to more boilerplate code calling the named
>>     tests)
>>
>> Unfortunately this library doesn't seem to offer any of those
>> features. It does support a lazy test plan but uses atexit() so will
>> not detect if the test program exits before all the tests have run. I
>> think it would be useful to add some unit tests to our test suite and
>> maybe this library could form the basis of that but I think printing
>> the file and line number of failed assertions is pretty essential.
> 
> Other things aside, I prefer our explicit "test_done", but I don't see
> why you think an atexit() isn't enough to catch incomplete tests.
>
> For a C program you'd just do something like this (somewhat pseudocode,
> I didn't check if it compiled etc):
> 	
>          static int done; /* read by atexit() handler */
> 
>          void on_atexit(void)
> 	{
> 		if (!done)
> 			BUG();
>          	print_plan_line();
> 	}
> 
> 	int main(void)
> 	{
>                  int ret;
> 
> 	        setup_atexit(a_handler);
> 		ret = do_tests();
> 	        done = 1;
> 
>                  return ret;
> 	}
> 
> If I'm understanding you correctly you're concerned that if some user
> code within do_test() calls exit() we won't return from "do_test()", but
> we *would* call print_plan_line().

Exactly

> That's a valid concern, we want to distinguish such "early return" from
> cases where we run to completion, that's why we use "test_done" in the
> shell code.
> 
> But in the C case I think just using something like the "done" variable
> pattern above should cover that, without the need for an explicit
> "test_done".

We could do that. My complaint is that the code being proposed does not 
and so prints a valid plan if any code being tested calls exit()

Best Wishes

Phillip