Re: [PATCH RFC v2 2/4] unit tests: Add a project plan document

From: Glen Choo <chooglen@google.com>
Date: May 18, 2023, 20:15 UTC
Message-ID: <kl6lzg61xuox.fsf@chooglen-macbookpro.roam.corp.google.com>
In-Reply-To: <20230517-unit-tests-v2-v2-2-21b5b60f4b32@google.com>

steadmon@google.com writes:

> Describe what we hope to accomplish by implementing unit tests, and
> explain some open questions and milestones.

Thanks! I found this very helpful.

Show 16 quoted lines

> diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt
> new file mode 100644
> index 0000000000..7c575e6ef7
> --- /dev/null
> +++ b/Documentation/technical/unit-tests.txt
> @@ -0,0 +1,47 @@
> += Unit Testing
> +
> +In our current testing environment, we spend a significant amount of effort
> +crafting end-to-end tests for error conditions that could easily be captured by
> +unit tests (or we simply forgo some hard-to-setup and rare error conditions).
> +Unit tests additionally provide stability to the codebase and can simplify
> +debugging through isolation. Writing unit tests in pure C, rather than with our
> +current shell/test-tool helper setup, simplifies test setup, simplifies passing
> +data around (no shell-isms required), and reduces testing runtime by not
> +spawning a separate process for every test invocation.

The stated goals make sense to me, and I believe they are worth restating. I believe this is mostly taken from Calvin's v1 cover letter

  https://lore.kernel.org/git/20230427175007.902278-1-calvinwan@google.com

so perhaps he should receive some writing credit in a commit trailer (Helped-by?).

Show 15 quoted lines

> +== Open questions
> +
> +=== TAP harness
> +
> +We'll need to decide on a TAP harness. The C TAP library is easy to integrate,
> +but has a few drawbacks:
> +* (copy objections from lore thread)
> +* We may need to carry local patches against C TAP. We'll need to decide how to
> +  manage these. We could vendor the code in and modify them directly, or use a
> +  submodule (but then we'll need to decide on where to host the submodule with
> +  our patches on top).
> +
> +Phillip Wood has also proposed a new implementation of a TAP harness (linked
> +above). While it hasn't been thoroughly reviewed yet, it looks to support a few
> +nice features that C TAP does not, e.g. lazy test plans and skippable tests.

A third option would be to pick another, more mature third party testing library. As I mentioned in

  https://lore.kernel.org/git/kl6lpm76zcg7.fsf@chooglen-macbookpro.roam.corp.google.com

my primary concern is the maintainability and extensibility of a third party library that (no offense to the original author) is not used very widely, is relatively underdocumented, is missing features that we want, and whose maintenance policy is relatively unknown to us. I'm not opposed to taking in a third party testing framework, but we need to be sure that we can rely on it instead of being something that requires active upkeep.

I don't what sorts of testing libraries exist for C or how widely they are used, but a quick web search gives some candidates that seem like plausible alternatives to C TAP Harness:

- cmocka https://cmocka.org/ supports TAP, assert macros and mocking,
  and is used by other projects (their website indicates Samba, OpenVPN,
  etc). The documentation is a bit lacking IMO. It's apparently a fork
  of cmockery, which I'm not familiar with.

- Check https://libcheck.github.io/check/ supports TAP and has
  relatively good documentation, though the last release seems to have
  been 3 years ago.

- µnit https://nemequ.github.io/munit/ has a shiny website with nice
  docs (and a handy list of other unit test frameworks we can look at).
  The last release also seems to be ~3 years ago. Not sure if this
  supports TAP.

For flexibility, I also think it's reasonable for us to roll our own testing library. I think it is perfectly fine for our unit test framework to be suboptimal at the beginning, but owning the code makes it relatively easy to fix bugs and extend it to our liking.