my complaints with clar

This was my first time writing unit tests with clar, so I thought I'd document a few rough edges I found. It's possible I'm holding it wrong in some areas.

The error messages I got on failure from this function were not super informative. Naively, if you do something like:

  check_int_full("0", 0);
  check_int_full("11", 11);
  check_int_full("-23", -23);
  check_int_full("+23", 23);

and it fails, you'll get not much beyond "expected ok, but it's not true" with a line number in the helper, but no idea which input failed. So OK, we have cl_invoke for that, and:

  cl_invoke(check_int_full("0", 0));
  cl_invoke(check_int_full("11", 11));
  cl_invoke(check_int_full("-23", -23));
  cl_invoke(check_int_full("+23", 23));

gives you the line number in the caller. Better, but there's a lot of cross-referencing the line numbers (plus sprinkling cl_invoke everywhere is ugly).

What I really would have liked is some notion of "context". If the helper could have done:

  cl_context("input: %.*s", (int)len, buf);

or similar, and failed assertions print that context, then that would have made the failing part of the test easy to see, even without using cl_invoke() at all.

Alternatively, I kind of wonder if cl_invoke() could just stringify the entire macro argument and shove that into the context. That helps for:

  cl_invoke(check_int_full("11", 11));

though not if parameters are opaque in that line, like:

  cl_invoke(check_int_full(str, expect));

But I think boilerplate-saving helpers tend to be written more like the first way.

In a more general sense, what I'd really have loved is an automatic backtrace, but I suspect getting a readable one is impossible. Even if we knew the called function and the parameters, a generic backtracer cannot know the meaning of "buf" and "len" enough to show what was in the buffer.

And of course the more obvious way to avoid that is to break this:

into a series of nine separate tests, each of which gets a name. But each of those tests is at least five lines of boilerplate, which sucks (plus you have to come up with syntactically valid C names for them).

This was an exciting bug to track down. If you use i_fmt() here, you get some neat undefined behavior. It worked for gcc, but failed with clang (but only with -O2!).

Obviously this was me using it wrong, and the "i" in the macro should have been a hint. But this invocation is kind of ugly, with the explicit mentions of internal CLAR variables. clar__assert_equal() understands PRIuMAX as a comparator, but there doesn't appear to be any macro to use it nicely.

Should there be a generic cl_assert_equal() that fills in the first few parameters but is otherwise type-agnostic?

It also looked error-prone to me that if you pass in an unknown format specifier, clar__assert_equal() will assume you want integers. So a typo, or using an unknown-but-equivalent specifier will give you weird undefined behavior bugs. These are just tests, so we can perhaps a bit more loose, but these kinds of things can be hard to track down (especially if they trigger only in certain compiler combos via CI, which is what happened to me).

And that brings me to my final complaint. ;)

When the unit-tests fail in CI, you get very little useful feedback, because the output is eaten by "prove", and the unit tests don't understand --verbose-log at all. And then to make it more exciting, the output that clar produces actually chokes prove. For example, if I do this: