Re: [PATCH RFC v2 2/4] unit tests: Add a project plan document

From: Phillip Wood <phillip.wood123@gmail.com>
Date: Jun 1, 2023, 09:19 UTC
Message-ID: <73580406-f2cd-12ff-954a-2adb73b0edc8@gmail.com>
In-Reply-To: <kl6lzg61xuox.fsf@chooglen-macbookpro.roam.corp.google.com>

Hi Glen

On 18/05/2023 21:15, Glen Choo wrote:

> steadmon@google.com writes:

Show 30 quoted lines

> A third option would be to pick another, more mature third party testing
> library. As I mentioned in
> 
>    https://lore.kernel.org/git/kl6lpm76zcg7.fsf@chooglen-macbookpro.roam.corp.google.com
> 
> my primary concern is the maintainability and extensibility of a third
> party library that (no offense to the original author) is not used very
> widely, is relatively underdocumented, is missing features that we want,
> and whose maintenance policy is relatively unknown to us. I'm not
> opposed to taking in a third party testing framework, but we need to be
> sure that we can rely on it instead of being something that requires
> active upkeep.
> 
> I don't what sorts of testing libraries exist for C or how widely they
> are used, but a quick web search gives some candidates that seem like
> plausible alternatives to C TAP Harness:
> 
> - cmocka https://cmocka.org/ supports TAP, assert macros and mocking,
>    and is used by other projects (their website indicates Samba, OpenVPN,
>    etc). The documentation is a bit lacking IMO. It's apparently a fork
>    of cmockery, which I'm not familiar with.
> 
> - Check https://libcheck.github.io/check/ supports TAP and has
>    relatively good documentation, though the last release seems to have
>    been 3 years ago.
> 
> - µnit https://nemequ.github.io/munit/ has a shiny website with nice
>    docs (and a handy list of other unit test frameworks we can look at).
>    The last release also seems to be ~3 years ago. Not sure if this
>    supports TAP.

Thanks for finding those. I'd add

  - libtap https://github.com/zorgnax/libtap which seems to have nice
      assertions and TAP output. It is licensed under LGPL3 but used
      to be GPL2 and there don't seem to have been many changes since
      the license change so we could just use the older version.

I think it would be helpful to have a section in the plan which clearly states the features we want (TAP output, helpers that make it easy to write tests with good diagnostic output, ...) in our unit test framework and explains how the selected solution meets those criteria.

It would also be helpful to have a guide to writing effective tests that sets out some guidelines on where unit tests are appropriate and outlines our expectations in terms of style, coverage, leak cleanliness etc.

Best Wishes

Phillip

> For flexibility, I also think it's reasonable for us to roll our own
> testing library. I think it is perfectly fine for our unit test
> framework to be suboptimal at the beginning, but owning the code makes
> it relatively easy to fix bugs and extend it to our liking.