Re: [RFC PATCH 0/2] add an external testing library for unit tests

From: Felipe Contreras <felipe.contreras@gmail.com>
Date: May 2, 2023, 15:28 UTC
Message-ID: <64512c0b1237b_1ba2d29426@chronos.notmuch>
In-Reply-To: <230502.861qjyj0cb.gmgdl@evledraar.gmail.com>

Ævar Arnfjörð Bjarmason wrote:

Show 16 quoted lines

> We already have pure-C libraries that we add a really shim to unit test,
> the most extreme example of this is t0032-reftable-unittest.sh, whose
> body is simply (excluding comments):
> 	
> 	#!/bin/sh
> 	test_description='reftable unittests'
> 	
> 	TEST_PASSES_SANITIZE_LEAK=true
> 	. ./test-lib.sh
> 	
> 	test_expect_success 'unittests' '
> 		TMPDIR=$(pwd) && export TMPDIR &&
> 		test-tool reftable
> 	'
> 	
> 	test_done

Yeah, but an output of 'ok 1 - unittests' is not very useful, neither is an output of 'not ok 1 - unittests'.

This completely misses the point of a TAP interface, which is to parse the status of individual test cases, or even individual assertions.

If all we are doing is check the exit code of some program, then we don't need TAP.

Show 5 quoted lines

> Now, that goes into reftable/test_framework.h which basically implements
> its own mini-test framework, so that's at least a *partial* argument for
> what you're suggesting here, but note that it doesn't emit TAP, it just
> returns an exit code, the EXPECT() etc. is purely internal. I.e. "what
> should we return?".

You are just describing the status quo.

I think this is a naturaistic fallacy: confusing what is versus what ought to be.

Can we test C code with our current testing framework? Yes.

Should we? That's the actual question.

> None of those are perfect, but I think the current arrangement is rather
> ideal.

I think misuing TAP is far from ideal.

In my view an ideal framework would:

 1. Be able to test C code
 2. Report individual test cases success/failure
 3. Report relevant context in the case of failure (actual vs. expected)
 4. Don't create forks on every individual test case or assertion
 5. Properly handle crashes

Doing basically the following is not an ideal framework:

  echo '1..1'
  test-tool reftable > /dev/null 2>&1 && echo 'ok 1' || { echo 'not ok 1'; false; }

Yes, it works. But "it works" has never been a good enough reason to stay on the status quo.

> We can write most or all of the test in C, but we just do so by
> calling a function that returns an exit code.

Yes, we *can*, but should we?

---

If we can test C code in a way that individual test case failures are reported (as is the intention with TAP), why would we reject that in favor of the status quo which basically is just reporting the exit code of the whole test file?

-- 
Felipe Contreras