Feasibility of folding `unit-tests` into `make test`, was Re: [PATCH] ci: avoid running the test suite _twice_

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Nov 16, 2023, 08:42 UTC
Message-ID: <850ea42c-f103-68d5-896b-9120e2628686@gmx.de>
In-Reply-To: <ZVU4EVcj0MDrSNcG@google.com>

Hi Josh,

On Wed, 15 Nov 2023, Josh Steadmon wrote:

Show 17 quoted lines

> On 2023.11.13 13:49, Jeff King wrote:
> >
> > why are the unit tests totally separate from the rest of the suite? I
> > would think we'd want them run from one or more t/t*.sh scripts. That
> > would make bugs like this impossible, but also:
> >
> >   1. They'd be run via "make test", so developers don't have to remember
> >      to run them separately.
> >
> >   2. They can be run in parallel with all of the other tests when using
> >      "prove -j", etc.
>
> The first part is easy, but I don't see a good way to get both shell
> tests and unit tests executing under the same `prove` process. For shell
> tests, we pass `--exec '$(TEST_SHELL_PATH_SQ)'` to prove, meaning that
> we use the specified shell as an interpreter for the test files. That
> will not work for unit test executables.

Probably my favorite aspect about the new unit tests is that they avoid using the error-prone, unintuitive and slow shell scripts and stay within the programming language of the code that is to be tested: C.

Show 6 quoted lines

> We could bundle all the unit tests into a single shell script, but then
> we lose parallelization and add hoops to jump through to determine what
> breaks. Or we could autogenerate a corresponding shell script to run
> each individual unit test, but that seems gross. Of course, these are
> hypothetical concerns for now, since we only have a single unit test at
> the moment.

I totally agree with you, Josh, that it makes little sense to try to contort the unit tests to be run in the same `prove` run as the regression tests that need to be invoked so totally differently.

Show 9 quoted lines

> There's also the issue that the shell test arguments we pass on from
> prove would be shared with the unit tests. That's fine for now, as
> t-strbuf doesn't accept any runtime arguments, but it's possible that
> either the framework or individual unit tests might grow to need
> arguments, and it might not be convenient to stay compatible with the
> shell tests.
>
> Personally, I lean towards keeping things simple and just running a
> second `prove` process as part of `make test`.

Agreed.

> If I was forced to pick a way to get everything under one process, I'd
> lean towards autogenerating individual shell script wrappers for each
> unit test. But I'm open to discussion, especially if people have other
> approaches I haven't thought of.

One alternative would be to avoid running the unit tests via `prove` in the first place.

For example, we could use the helper from be5d88e11280 (test-tool run-command: learn to run (parts of) the testsuite, 2019-10-04) [*1*]. It would probably need a few improvements, but certainly no wizardry nor witchcraft would be required. It would also help on Windows, where running a simple test helper written in C is vastly faster than running a complex Perl script (which `prove` is).

Ciao, Johannes

Footnote *1*: I had always wanted to improve that test helper to the point where it could replace our use of `prove`, at least on Windows. It seems, however, that as of 4c2c38e800f3 (ci: modification of main.yml to use cmake for vs-build job, 2020-06-26) we do not use the helper at all anymore. Hopefully it can still be useful. 🤞