[RFC PATCH 0/3] Introduce clar testing framework

Hi,

there's been some discussion around extending our unit testing framework to avoid duplication when declaring test functions. Right now, a testing function has to be declared and then wired up via the test's main function, which can be a bit annoying. In the thread, René proposes an alternative that gets rid of this duplication by using macros. And while that does solve the issue, there were some concerns about things being too much "magic" while at the same time not being flexible enough.

Part of the discussion revolved around whether we maybe want to have a proper unit testing framework in our codebase instead of reinventing the wheel. As I quite liked the "clar" [2] testing framework from back when I was still developing libgit2 I proposed it as a possible alternative. This patch series wires up the clar framework as a proof of concept and converts the strvec test suite to use it.

The magic to avoid the above code duplication is quite self-contained in a "generate.py" file. This script extracts function declarations from all unit test suites and then writes those into a "clar.suite" header file. All that one needs to do is thus to declare a function with a specific name "test_<suite>__<name>" and then everything else gets wired up automatically.

Whether this is better than the solution proposed by René is probably a matter of taste. While I'm not a huge fan of the macro-based solution, I don't want to veto it either (not that I'd have that power anyway). So please, you should rather read this as a proof of concept to see how alternatives could look like such that we have a better picture of where we want to end up.

Some random thoughts:

  - The mandated Python dependency is suboptimal in my opinion.
    Rewriting the script in e.g. Perl should be easy enough though, it's
    no rocket science.

  - I prefer that the proposed solution results in a single binary as
    compared to one binary per test system.

  - The clar gives us the ability to pick which tests to run via command
    line parameters, which I personally like more than picking the
    specific binary to run.

  - The clar replaces some test assertions that we already have. They
    feel a bit more mature, but overall there aren't all that many
    assertions available. If we wanted to pick it up, then we'd likely
    have to add some more wrappers.

  - The clar uses longjmp instead of manually having to `return` from
    functions in case there was an assertion failure. This is easier to
    work with in my opinion.

Also, note that I only tested this on my Linux machine. I have no clue whether this works as-is on Windows, but I do know that libgit2 tests run on Linux, macOS and Windows. So it should work in theory, it's just a matter of polishing this series.

I'm happy to hear your thoughts on this, even if it ultimately ends up being shot down.

Patrick

[1]: <85b6b8a9-ee5f-42ab-bcbc-49976b30ef33@web.de> [2]: https://github.com/clar-test/clar

Patrick Steinhardt (3):
  t: import the clar unit testing framework
  Makefile: wire up the clar unit testing framework
  t/unit-tests: convert strvec tests to use clar

 .gitignore                                 |   1 +
 Makefile                                   |  36 +-
 t/Makefile                                 |   1 +
 t/unit-tests/.gitignore                    |   3 +
 t/unit-tests/clar/.github/workflows/ci.yml |  23 +
 t/unit-tests/clar/COPYING                  |  15 +
 t/unit-tests/clar/README.md                | 329 ++++++++
 t/unit-tests/clar/clar.c                   | 842 +++++++++++++++++++++
 t/unit-tests/clar/clar.h                   | 173 +++++
 t/unit-tests/clar/clar/fixtures.h          |  50 ++
 t/unit-tests/clar/clar/fs.h                | 522 +++++++++++++
 t/unit-tests/clar/clar/print.h             | 211 ++++++
 t/unit-tests/clar/clar/sandbox.h           | 154 ++++
 t/unit-tests/clar/clar/summary.h           | 143 ++++
 t/unit-tests/clar/generate.py              | 267 +++++++
 t/unit-tests/clar/test/.gitignore          |   5 +
 t/unit-tests/clar/test/Makefile            |  39 +
 t/unit-tests/clar/test/clar_test.h         |  16 +
 t/unit-tests/clar/test/main.c              |  40 +
 t/unit-tests/clar/test/main.c.sample       |  27 +
 t/unit-tests/clar/test/resources/test/file |   1 +
 t/unit-tests/clar/test/sample.c            |  84 ++
 t/unit-tests/{t-strvec.c => strvec.c}      | 124 ++-
 t/unit-tests/unit-test.c                   |  16 +
 t/unit-tests/unit-test.h                   |   3 +
 25 files changed, 3041 insertions(+), 84 deletions(-)
 create mode 100644 t/unit-tests/clar/.github/workflows/ci.yml
 create mode 100644 t/unit-tests/clar/COPYING
 create mode 100644 t/unit-tests/clar/README.md
 create mode 100644 t/unit-tests/clar/clar.c
 create mode 100644 t/unit-tests/clar/clar.h
 create mode 100644 t/unit-tests/clar/clar/fixtures.h
 create mode 100644 t/unit-tests/clar/clar/fs.h
 create mode 100644 t/unit-tests/clar/clar/print.h
 create mode 100644 t/unit-tests/clar/clar/sandbox.h
 create mode 100644 t/unit-tests/clar/clar/summary.h
 create mode 100755 t/unit-tests/clar/generate.py
 create mode 100644 t/unit-tests/clar/test/.gitignore
 create mode 100644 t/unit-tests/clar/test/Makefile
 create mode 100644 t/unit-tests/clar/test/clar_test.h
 create mode 100644 t/unit-tests/clar/test/main.c
 create mode 100644 t/unit-tests/clar/test/main.c.sample
 create mode 100644 t/unit-tests/clar/test/resources/test/file
 create mode 100644 t/unit-tests/clar/test/sample.c
 rename t/unit-tests/{t-strvec.c => strvec.c} (54%)
 create mode 100644 t/unit-tests/unit-test.c
 create mode 100644 t/unit-tests/unit-test.h