Show 105 quoted lines
> On Jun 30, 2023, at 6:08 AM, Phillip Wood <phillip.wood123@gmail.com> wrote:
>
> Hi Josh
>
> Thanks for putting this together, I think it is really helpful to have a comparison of the various options. Sorry for the slow reply, I was off the list for a couple of weeks.
>
>> On 10/06/2023 00:25, Josh Steadmon wrote:
>> diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt
>> new file mode 100644
>> index 0000000000..dac8062a43
>> --- /dev/null
>> +++ b/Documentation/technical/unit-tests.txt
>> @@ -0,0 +1,141 @@
>> += Unit Testing
>
> I've deleted the sections I agree with to avoid quoting parts that are not relevant to my comments.
>
>> +== Definitions
>> +
>> +For the purposes of this document, we'll use *test framework* to refer to
>> +projects that support writing test cases and running tests within the context
>> +of a single executable. *Test harness* will refer to projects that manage
>> +running multiple executables (each of which may contain multiple test cases) and
>> +aggregating their results.
>
> Thanks for adding this, it is really helpful to have definitions for what we mean by "test framework" and "test harness" within the git project. It might be worth mentioning somewhere that we already use prove as a test harness when running our integration tests.
>
>> +In reality, these terms are not strictly defined, and many of the projects
>> +discussed below contain features from both categories.
>
>> +
>> +== Choosing a framework & harness
>> +
>> +=== Desired features
>> +
>> [...]
>> +==== Parallel execution
>> +
>> +Ideally, we will build up a significant collection of unit tests cases, most
>> +likely split across multiple executables. It will be necessary to run these
>> +tests in parallel to enable fast develop-test-debug cycles.
>
> This is a good point, though I think it is really a property of the harness rather than the framework so we might want to indicate in the table whether a framework provides parallelism itself or relies on the harness providing it.
>
> > [...]
>> +==== Major platform support
>> +
>> +At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.
>
> I think we'd want to be able to run unit tests on *BSD and NonStop as well, especially as I think some of the platform dependent code probably lends itself to being unit tested. I suspect a framework that covers Linux and MacOS would probably run on those platforms as well (I don't think NonStop has complete POSIX support but it is hard to imagine a test framework doing anything very exotic)
>
>> [...]
>> +==== Mock support
>> +
>> +Unit test authors may wish to test code that interacts with objects that may be
>> +inconvenient to handle in a test (e.g. interacting with a network service).
>> +Mocking allows test authors to provide a fake implementation of these objects
>> +for more convenient tests.
>
> Do we have any idea what sort of thing we're likely to want to mock and what we want that support to look like?
>
>> +==== Signal & exception handling
>> +
>> +The test framework must fail gracefully when test cases are themselves buggy or
>> +when they are interrupted by signals during runtime.
>
> I had assumed that it would be enough for the test harness to detect if a test executable was killed by a signal or exited early due to a bug in the test script. That requires the framework to have robust support for lazy test plans but I'm not sure that we need it to catch and recover from things like SIGSEGV.
>
>> +==== Coverage reports
>> +
>> +It may be convenient to generate coverage reports when running unit tests
>> +(although it may be possible to accomplish this regardless of test framework /
>> +harness support).
>
> I agree this would be useful, though perhaps we should build it on our existing gcov usage.
>
> Related to this do we want timing reports from the harness or the framework?
>
>> +
>> +=== Comparison
>> +
>> +[format="csv",options="header",width="75%"]
>> +|=====
>> +Framework,"TAP support","Diagnostic output","Parallel execution","Vendorable / ubiquitous","Maintainable / extensible","Major platform support","Lazy test planning","Runtime- skippable tests","Scheduling / re-running",Mocks,"Signal & exception handling","Coverage reports"
>> +https://lore.kernel.org/git/c902a166-98ce-afba-93f2-ea6027557176@gmail.com/[Custom Git impl.],[lime-background]#True#,[lime-background]#True#,?,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,[lime-background]#True#,?,?,[red-background]#False#,?,?
>> +https://cmocka.org/[cmocka],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[lime-background]#True#,?,?
>> +https://libcheck.github.io/check/[Check],[lime-background]#True#,[lime-background]#True#,?,[red-background]#False#,[yellow-background]#Partial#,[lime-background]#True#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
>> +https://github.com/rra/c-tap-harness/[C TAP],[lime-background]#True#,[red-background]#False#,?,[lime-background]#True#,[yellow-background]#Partial#,[yellow-background]#Partial#,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
>> +https://github.com/silentbicycle/greatest[Greatest],[yellow-background]#Partial#,?,?,[lime-background]#True#,[yellow-background]#Partial#,?,[yellow-background]#Partial#,?,?,[red-background]#False#,?,?
>> +https://github.com/Snaipe/Criterion[Criterion],[lime-background]#True#,?,?,[red-background]#False#,?,[lime-background]#True#,?,?,?,[red-background]#False#,?,?
>> +https://github.com/zorgnax/libtap[libtap],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?
>> +https://nemequ.github.io/munit/[µnit],?,?,?,?,?,?,?,?,?,?,?,?
>> +https://github.com/google/cmockery[cmockery],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,?
>> +https://github.com/lpabon/cmockery2[cmockery2],?,?,?,?,?,?,?,?,?,[lime-background]#True#,?,?
>> +https://github.com/ThrowTheSwitch/Unity[Unity],?,?,?,?,?,?,?,?,?,?,?,?
>> +https://github.com/siu/minunit[minunit],?,?,?,?,?,?,?,?,?,?,?,?
>> +https://cunit.sourceforge.net/[CUnit],?,?,?,?,?,?,?,?,?,?,?,?
>> +https://www.kindahl.net/mytap/doc/index.html[MyTAP],[lime-background]#True#,?,?,?,?,?,?,?,?,?,?,?
>> +|=====
>
> Thanks for going through these projects, hopefully we can use this information to make a decision on a framework soon.
>
> Best Wishes
>
> Phillip