git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v8 2/3] unit tests: add TAP unit test framework

From
Christian Couder <christian.couder@gmail.com>
Date
Oct 27, 2023, 20:15 UTC
Message-ID
<CAP8UFD3eY_i36YO0OcpAp9ey5KO0q-PrwvjSLRXKYQb=iZ8JCA@mail.gmail.com>
In-Reply-To
<00d3c95a81449bf49c4ce992d862d7a858691840.1696889530.git.steadmon@google.com>
On Tue, Oct 10, 2023 at 12:22 AM Josh Steadmon <steadmon@google.com> wrote:
Show 38 quoted lines
>
> From: Phillip Wood <phillip.wood@dunelm.org.uk>
>
> This patch contains an implementation for writing unit tests with TAP
> output. Each test is a function that contains one or more checks. The
> test is run with the TEST() macro and if any of the checks fail then the
> test will fail. A complete program that tests STRBUF_INIT would look
> like
>
>      #include "test-lib.h"
>      #include "strbuf.h"
>
>      static void t_static_init(void)
>      {
>              struct strbuf buf = STRBUF_INIT;
>
>              check_uint(buf.len, ==, 0);
>              check_uint(buf.alloc, ==, 0);
>              check_char(buf.buf[0], ==, '\0');
>      }
>
>      int main(void)
>      {
>              TEST(t_static_init(), "static initialization works);
>
>              return test_done();
>      }
>
> The output of this program would be
>
>      ok 1 - static initialization works
>      1..1
>
> If any of the checks in a test fail then they print a diagnostic message
> to aid debugging and the test will be reported as failing. For example a
> failing integer check would look like
>
>      # check "x >= 3" failed at my-test.c:102

I wonder if it would be a bit better to say that the test was an integer test for example with "check_int(x >= 3) failed ..."

>      #    left: 2
>      #   right: 3

I like "expected" and "actual" better than "left" and "right", not sure how it's possible to have that in a way consistent with the shell tests though.

Show 7 quoted lines
>      not ok 1 - x is greater than or equal to three
>
> There are a number of check functions implemented so far. check() checks
> a boolean condition, check_int(), check_uint() and check_char() take two
> values to compare and a comparison operator. check_str() will check if
> two strings are equal. Custom checks are simple to implement as shown in
> the comments above test_assert() in test-lib.h.
Yeah, nice.
> Tests can be skipped with test_skip() which can be supplied with a
> reason for skipping which it will print. Tests can print diagnostic
> messages with test_msg().  Checks that are known to fail can be wrapped
> in TEST_TODO().

Maybe TEST_TOFIX() would be a bit more clear, but "TODO" is something that is more likely to be searched for than "TOFIX", so Ok.

Show 10 quoted lines
> There are a couple of example test programs included in this
> patch. t-basic.c implements some self-tests and demonstrates the
> diagnostic output for failing test. The output of this program is
> checked by t0080-unit-test-output.sh. t-strbuf.c shows some example
> unit tests for strbuf.c
>
> The unit tests will be built as part of the default "make all" target,
> to avoid bitrot. If you wish to build just the unit tests, you can run
> "make build-unit-tests". To run the tests, you can use "make unit-tests"
> or run the test binaries directly, as in "./t/unit-tests/bin/t-strbuf".
Nice!
> +unit-tests-prove:
> +       @echo "*** prove - unit tests ***"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS)

Nice, but DEFAULT_TEST_TARGET=prove isn't used. So not sure how important or relevant the 'prove' related sections are in the Documentation/technical/unit-tests.txt file introduced by the previous patch.

Show 8 quoted lines
> +int test_assert(const char *location, const char *check, int ok)
> +{
> +       assert(ctx.running);
> +
> +       if (ctx.result == RESULT_SKIP) {
> +               test_msg("skipping check '%s' at %s", check, location);
> +               return 1;
> +       } else if (!ctx.todo) {

I think it would be a bit clearer without the "else" above and with the "if (!ctx.todo) {" starting on a new line.

Show 10 quoted lines
> +               if (ok) {
> +                       test_pass();
> +               } else {
> +                       test_msg("check \"%s\" failed at %s", check, location);
> +                       test_fail();
> +               }
> +       }
> +
> +       return !!ok;
> +}
Otherwise it looks good to me.
Previous: Junio C HamanoNext: Josh Steadmon
Message 45 of 67 in “unit tests: Add a project plan document”
  1. unit tests: Add a project plan documentJosh Steadmon, Jun 30, 2023
  2. Junio C HamanoJul 1, 2023
  3. Junio C HamanoJul 1, 2023
  4. unit tests: Add a project plan documentJosh Steadmon, Aug 7, 2023
  5. Phillip WoodAug 14, 2023
  6. Josh SteadmonAug 15, 2023
  7. Phillip WoodAug 17, 2023
  8. 0/3 Add unit test framework and project plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bitJosh Steadmon, Aug 16, 2023
  9. 3/3 ci: run unit tests in CIJosh Steadmon, Aug 16, 2023
  10. 1/3 unit tests: Add a project plan documentJosh Steadmon, Aug 16, 2023
  11. 2/3 unit tests: add TAP unit test frameworkJosh Steadmon, Aug 16, 2023
  12. Junio C HamanoAug 17, 2023
  13. Junio C HamanoAug 17, 2023
  14. Josh SteadmonAug 17, 2023
  15. 0/3 Add unit test framework and project planJosh Steadmon, Aug 17, 2023
  16. 3/3 ci: run unit tests in CIJosh Steadmon, Aug 17, 2023
  17. 1/3 unit tests: Add a project plan documentJosh Steadmon, Aug 17, 2023
  18. 2/3 unit tests: add TAP unit test frameworkJosh Steadmon, Aug 17, 2023
  19. Junio C HamanoAug 18, 2023
  20. Junio C HamanoSep 22, 2023
  21. phillip.wood123@gmail.comSep 24, 2023
  22. Junio C HamanoSep 25, 2023
  23. Josh SteadmonOct 6, 2023
  24. Josh SteadmonOct 9, 2023
  25. Junio C HamanoAug 17, 2023
  26. Josh SteadmonAug 24, 2023
  27. Junio C HamanoSep 13, 2023
  28. 0/3 Add unit test framework and project planJosh Steadmon, Oct 9, 2023
  29. 1/3 unit tests: Add a project plan documentJosh Steadmon, Oct 9, 2023
  30. Oswald BuddenhagenOct 10, 2023
  31. Josh SteadmonOct 11, 2023
  32. Oswald BuddenhagenOct 11, 2023
  33. Josh SteadmonNov 1, 2023
  34. Christian CouderOct 27, 2023
  35. Josh SteadmonNov 1, 2023
  36. Junio C HamanoNov 1, 2023
  37. 2/3 unit tests: add TAP unit test frameworkJosh Steadmon, Oct 9, 2023
  38. Junio C HamanoOct 11, 2023
  39. 5/3 fixup! unit tests: add TAP unit test frameworkPhillip Wood, Oct 16, 2023
  40. Junio C HamanoOct 16, 2023
  41. Josh SteadmonNov 1, 2023
  42. Junio C HamanoNov 1, 2023
  43. Josh SteadmonNov 1, 2023
  44. Junio C HamanoNov 1, 2023
  45. Christian CouderOct 27, 2023
  46. Josh SteadmonNov 1, 2023
  47. 3/3 ci: run unit tests in CIJosh Steadmon, Oct 9, 2023
  48. Junio C HamanoOct 9, 2023
  49. 0/3 CMake unit test fixupsPhillip Wood, Oct 19, 2023
  50. 1/3 fixup! cmake: also build unit testsPhillip Wood, Oct 19, 2023
  51. 2/3 fixup! artifacts-tar: when including `.dll` files, don't forget the unit-testsPhillip Wood, Oct 19, 2023
  52. 3/3 fixup! cmake: handle also unit testsPhillip Wood, Oct 19, 2023
  53. Junio C HamanoOct 19, 2023
  54. phillip.wood123@gmail.comOct 16, 2023
  55. Josh SteadmonNov 1, 2023
  56. Christian CouderOct 27, 2023
  57. 0/3 Add unit test framework and project planJosh Steadmon, Nov 1, 2023
  58. 1/3 unit tests: Add a project plan documentJosh Steadmon, Nov 1, 2023
  59. 3/3 ci: run unit tests in CIJosh Steadmon, Nov 1, 2023
  60. 2/3 unit tests: add TAP unit test frameworkJosh Steadmon, Nov 1, 2023
  61. Christian CouderNov 3, 2023
  62. Josh SteadmonNov 9, 2023
  63. 0/3 Add unit test framework and project planJosh Steadmon, Nov 9, 2023
  64. 1/3 unit tests: Add a project plan documentJosh Steadmon, Nov 9, 2023
  65. Junio C HamanoNov 9, 2023
  66. 2/3 unit tests: add TAP unit test frameworkJosh Steadmon, Nov 9, 2023
  67. 3/3 ci: run unit tests in CIJosh Steadmon, Nov 9, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.