git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: my complaints with clar

From
PWPhillip Wood <phillip.wood123@gmail.com>
Date
Dec 1, 2025, 14:16 UTC
Message-ID
<bd0a8a76-fccb-4b6c-abb7-b53dd890e9e0@gmail.com>
In-Reply-To
<20251130134625.GA199421@coredump.intra.peff.net>
Hi Peff
On 30/11/2025 13:46, Jeff King wrote:
Show 9 quoted lines
> 
>    cl_invoke(check_int_full("0", 0));
>    cl_invoke(check_int_full("11", 11));
>    cl_invoke(check_int_full("-23", -23));
>    cl_invoke(check_int_full("+23", 23));
> 
> gives you the line number in the caller. Better, but there's a lot of
> cross-referencing the line numbers (plus sprinkling cl_invoke everywhere
> is ugly).

The README for the old unit-testing framework recommended wrapping helper functions in a macro to pass the file and line numbers from the calling site. Perhaps we should do the same with clar

#define check_int_full(input, expect) cl_invoke(check_int_full(input, expect))

Show 8 quoted lines
> What I really would have liked is some notion of "context". If the
> helper could have done:
> 
>    cl_context("input: %.*s", (int)len, buf);
> 
> or similar, and failed assertions print that context, then that would
> have made the failing part of the test easy to see, even without using
> cl_invoke() at all.

If you're writing a helper function you might want to use cl_failf() instead of cl_assert_* to provide more context but it's a pain that you can't just use the builtin assertions. I've not used them but there are assertions named cl_assert_*_ which I think let you add some context. One of the features of the conversion of our unit tests from the old framework to clar has been a degradation of the diagnostic messages when a test fails.

Show 18 quoted lines
>> +void test_parse_int__basic(void)
>> +{
>> +	cl_invoke(check_int_full("0", 0));
>> +	cl_invoke(check_int_full("11", 11));
>> +	cl_invoke(check_int_full("-23", -23));
>> +	cl_invoke(check_int_full("+23", 23));
>> +	cl_invoke(check_int_str("  31337  ", 7, 0, 31337));
>> +
>> +	cl_invoke(check_int_err("  garbage", EINVAL));
>> +	cl_invoke(check_int_err("", EINVAL));
>> +	cl_invoke(check_int_err("-", EINVAL));
>> +
>> +	cl_invoke(check_int("123", 2, 2, 0, 12));
>> +}
> 
> into a series of nine separate tests, each of which gets a name. But
> each of those tests is at least five lines of boilerplate, which sucks
> (plus you have to come up with syntactically valid C names for them).

Yes that's a pain. One of the nice things about the old framework was the the TEST() macro just took an expression and created a test case out of it which worked well for tests like this and meant you could have table driven tests where each entry in the table was a separate test case.

Show 22 quoted lines
>> +	/*
>> +	 * Do not use cl_assert_equal_i_fmt(..., PRIuMAX) here. The macro
>> +	 * casts to int under the hood, corrupting the values.
>> +	 */
>> +	clar__assert_equal(CLAR_CURRENT_FILE, CLAR_CURRENT_FUNC,
>> +			   CLAR_CURRENT_LINE,
>> +			   "expect_result != result", 1,
>> +			   "%"PRIuMAX, expect_result, result);
>> +}
> 
> This was an exciting bug to track down. If you use i_fmt() here, you get
> some neat undefined behavior. It worked for gcc, but failed with clang
> (but only with -O2!).
> 
> Obviously this was me using it wrong, and the "i" in the macro should
> have been a hint. But this invocation is kind of ugly, with the explicit
> mentions of internal CLAR variables. clar__assert_equal() understands
> PRIuMAX as a comparator, but there doesn't appear to be any macro to use
> it nicely.
> 
> Should there be a generic cl_assert_equal() that fills in the first
> few parameters but is otherwise type-agnostic?

Patrick's got a PR open for that at https://github.com/clar-test/clar/pull/117 it seems to have got stuck because of a lack of review.

Show 26 quoted lines
>    # start of suite 10: parse_int
>    not ok 59 - parse_int::basic
>        ---
>        reason: |
>          expect_result != result
>          10 != 11
>        at:
>          file: 't/unit-tests/u-parse-int.c'
>          line: 41
>          function: 'test_parse_int__basic'
>        ---
> 
> OK, but "prove t/unit-tests/bin/unit-tests" gives me:
> 
>    t/unit-tests/bin/unit-tests .. Failed 1/59 subtests
>    
>    Test Summary Report
>    -------------------
>    t/unit-tests/bin/unit-tests (Wstat: (none) Tests: 59 Failed: 1)
>      Failed test:  59
>      Parse errors: Badly formed hash line: '---' at /usr/share/perl/5.40/TAP/Parser/YAMLish/Reader.pm line 244.
> 
> Yuck. It actually does have what I need (that test 59 was the failure),
> so the extra parse error is mostly a red herring (though it does prevent
> us finding any further failures). I think in TAP that arbitrary output
> is supposed to be prefixed with a "#".

TAP also allows you to embed YAML and unfortunately that's what clar tries to do but that last " ---" line should be " ...". With the diff below (which I'm afraid thunderbird will probably mangle) prove parses the output correctly but still does not print the error message. I'll update clar's self tests and open a PR later this week.

---- 8< ----
diff --git a/t/unit-tests/clar/clar/print.h b/t/unit-tests/clar/clar/print.h
index 89b66591d75..6a2321b399d 100644
--- a/t/unit-tests/clar/clar/print.h
+++ b/t/unit-tests/clar/clar/print.h
@@ -164,7 +164,7 @@ static void clar_print_tap_ontest(const char 
*suite_name, const char *test_name,
                          printf("      file: '"); 
print_escaped(error->file); printf("'\n");
                          printf("      line: %" PRIuMAX "\n", 
error->line_number);
                          printf("      function: '%s'\n", error->function);
-                        printf("    ---\n");
+                        printf("    ...\n");
                  }

                  break;
---- >8 ----

> In test-lib.sh, we solve this by
> only allowing "--verbose-log", not regular "-v", under a TAP harness.
> 
> I kind of wonder if we should have t0011-unit-tests.sh that simply runs
> unit-tests and filters the output into stdout and stderr.

I don't have a strong opinion on this but now that we don't run the unit 
tests in parallel because clar links them all into a single executable 
there is less reason to use prove.


Thanks

Phillip
Previous: Jeff KingNext: Patrick Steinhardt
Message 50 of 64 in “asan bonanza”
  1. 0/9 asan bonanzaJeff King, Nov 12, 2025
  2. 1/9 compat/mmap: mark unused argument in git_munmap()Jeff King, Nov 12, 2025
  3. 2/9 pack-bitmap: handle name-hash lookups in incremental bitmapsJeff King, Nov 12, 2025
  4. Patrick SteinhardtNov 12, 2025
  5. Taylor BlauNov 13, 2025
  6. Jeff KingNov 18, 2025
  7. 3/9 Makefile: turn on NO_MMAP when building with ASanJeff King, Nov 12, 2025
  8. Collin FunkNov 12, 2025
  9. Jeff KingNov 12, 2025
  10. Collin FunkNov 12, 2025
  11. Patrick SteinhardtNov 12, 2025
  12. Taylor BlauNov 13, 2025
  13. Patrick SteinhardtNov 13, 2025
  14. Jeff KingNov 18, 2025
  15. Junio C HamanoNov 13, 2025
  16. Patrick SteinhardtNov 14, 2025
  17. Jeff KingNov 15, 2025
  18. 4/9 cache-tree: avoid strtol() on non-string bufferJeff King, Nov 12, 2025
  19. Patrick SteinhardtNov 12, 2025
  20. Taylor BlauNov 13, 2025
  21. Jeff KingNov 18, 2025
  22. Jeff KingNov 18, 2025
  23. 5/9 fsck: assert newline presence in fsck_ident()Jeff King, Nov 12, 2025
  24. 6/9 fsck: avoid strcspn() in fsck_ident()Jeff King, Nov 12, 2025
  25. 7/9 fsck: remove redundant date timestamp checkJeff King, Nov 12, 2025
  26. 8/9 fsck: avoid parse_timestamp() on buffer that isn't NUL-terminatedJeff King, Nov 12, 2025
  27. Patrick SteinhardtNov 12, 2025
  28. Junio C HamanoNov 12, 2025
  29. Jeff KingNov 15, 2025
  30. 9/9 t: enable ASan's strict_string_checks optionJeff King, Nov 12, 2025
  31. Taylor BlauNov 13, 2025
  32. 0/9 asan bonanzaJeff King, Nov 18, 2025
  33. 1/9 compat/mmap: mark unused argument in git_munmap()Jeff King, Nov 18, 2025
  34. 2/9 pack-bitmap: handle name-hash lookups in incremental bitmapsJeff King, Nov 18, 2025
  35. 3/9 Makefile: turn on NO_MMAP when building with ASanJeff King, Nov 18, 2025
  36. 4/9 cache-tree: avoid strtol() on non-string bufferJeff King, Nov 18, 2025
  37. Phillip WoodNov 18, 2025
  38. Junio C HamanoNov 23, 2025
  39. Phillip WoodNov 23, 2025
  40. Junio C HamanoNov 23, 2025
  41. Jeff KingNov 24, 2025
  42. Junio C HamanoNov 24, 2025
  43. Jeff KingNov 26, 2025
  44. Junio C HamanoNov 26, 2025
  45. 0/4 more robust functions for parsing int from bufJeff King, Nov 30, 2025
  46. 1/4 parse: prefer bool to int for boolean returnsJeff King, Nov 30, 2025
  47. Patrick SteinhardtDec 4, 2025
  48. 2/4 parse: add functions for parsing from non-string buffersJeff King, Nov 30, 2025
  49. my complaints with clarJeff King, Nov 30, 2025
  50. Phillip WoodDec 1, 2025
  51. Patrick SteinhardtDec 4, 2025
  52. Jeff KingDec 5, 2025
  53. Patrick SteinhardtDec 4, 2025
  54. Phillip WoodDec 5, 2025
  55. Junio C HamanoJan 20, 2026
  56. Jeff KingJan 21, 2026
  57. 3/4 cache-tree: use parse_int_from_buf()Jeff King, Nov 30, 2025
  58. 4/4 fsck: use parse_unsigned_from_buf() for parsing timestampJeff King, Nov 30, 2025
  59. 5/9 fsck: assert newline presence in fsck_ident()Jeff King, Nov 18, 2025
  60. 6/9 fsck: avoid strcspn() in fsck_ident()Jeff King, Nov 18, 2025
  61. 7/9 fsck: remove redundant date timestamp checkJeff King, Nov 18, 2025
  62. 8/9 fsck: avoid parse_timestamp() on buffer that isn't NUL-terminatedJeff King, Nov 18, 2025
  63. 9/9 t: enable ASan's strict_string_checks optionJeff King, Nov 18, 2025
  64. Junio C HamanoNov 23, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.