[PATCH v4 00/10] meson: parse TAP output generated by our tests
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Jun 2, 2025, 06:44 UTC
- Message-ID
- <20250602-pks-meson-tap-v4-0-052dfde0818a@pks.im>
- In-Reply-To
- <20250506-pks-meson-tap-v1-0-5aaab2942a4c@pks.im>
Hi,
this patch series starts to parse TAP output generated by our tests when executing them via Meson. This has the benefit that Meson starts to understand skipped tests and reports how many subtests have been executed:
```
$ meson test t002*
ninja: Entering directory `/home/pks/Development/git/build'
1/10 t0024-crlf-archive OK 0.17s 2 subtests passed
2/10 t0022-crlf-rename OK 0.18s 2 subtests passed
3/10 t0029-core-unsetenvvars SKIP 0.15s
4/10 t0023-crlf-am OK 0.18s 2 subtests passed
5/10 t0025-crlf-renormalize OK 0.21s 3 subtests passed
6/10 t0026-eol-config OK 0.25s 5 subtests passed
7/10 t0020-crlf OK 0.81s 36 subtests passed
8/10 t0028-working-tree-encoding OK 0.85s 22 subtests passed
9/10 t0021-conversion OK 3.45s 38 subtests passed
10/10 t0027-auto-crlf OK 26.35s 2600 subtests passed
Ok: 9
Fail: 0
Skipped: 1
```This new feature is only enabled with Meson 1.8 and newer, which contains a bugfix that we have upstreamed [1] to make the TAP parser work in `meson test --interactive` mode.
Despite the changes to Meson itself, this patch series also contains a couple of fixes for our test suite that caused us to not generate proper TAP output.
Changes in v2: - Add a patch to fix an unexpectedly passing test on macOS. - A couple more fixes for broken TAP output. - Link to v1: https://lore.kernel.org/r/20250506-pks-meson-tap-v1-0-5aaab2942a4c@pks.im
Changes in v3:
- Split up the patch that silences output into multiple patches and
rework them a bit.
- Remove redirect that was retained by accident from an earlier
version.
- Slight rewording of a commit message.
- Treat unexpected passes as failure in prove(1) and when executing
the test directly.
- Link to v2: https://lore.kernel.org/r/20250527-pks-meson-tap-v2-0-ae360f77786e@pks.imChanges in v4: - Fix references to git-p4(1), which was mistyped as git-b4(1). - A couple of commit message improvements. - Avoid duplicate test names in t4041. - Link to v3: https://lore.kernel.org/r/20250530-pks-meson-tap-v3-0-676f5e41f2e4@pks.im
Thanks!
Patrick
[1]: https://github.com/mesonbuild/meson/pull/13980
---
Patrick Steinhardt (10):
t: stop announcing prereqs
t: silence output from `test_create_repo()`
t9822: use prereq to check for ISO-8859-1 support
t983*: use prereq to check for Python-specific git-p4(1) support
t/test-lib: don't print shell traces to stdout
t/test-lib: fix TAP format for BASH_XTRACEFD warning
t7815: fix unexpectedly passing test on macOS
test-lib: fail on unexpectedly passing tests
meson: introduce kwargs variable for tests
meson: parse TAP output generated by our testscontrib/credential/netrc/meson.build | 2 +- contrib/subtree/meson.build | 2 +- meson.build | 12 +++++++++ t/meson.build | 6 ++--- t/t0000-basic.sh | 39 +++++++++++++++------------- t/t0050-filesystem.sh | 30 +++++---------------- t/t1007-hash-object.sh | 2 +- t/t3600-rm.sh | 5 ---- t/t4000-diff-format.sh | 2 +- t/t4041-diff-submodule-option.sh | 22 +++++++++------- t/t4060-diff-submodule-option-diff-format.sh | 9 ++++--- t/t7401-submodule-summary.sh | 18 ++++++++----- t/t7815-grep-binary.sh | 2 +- t/t9500-gitweb-standalone-no-errors.sh | 16 +++++------- t/t9822-git-p4-path-encoding.sh | 13 +++++++--- t/t9835-git-p4-metadata-encoding-python2.sh | 24 +++++++++-------- t/t9836-git-p4-metadata-encoding-python3.sh | 24 +++++++++-------- t/t9903-bash-prompt.sh | 4 --- t/test-lib.sh | 18 ++++++++++--- 19 files changed, 133 insertions(+), 117 deletions(-)
Range-diff versus v3:
1: 0e9aac3c63f = 1: 05d16d9d7ac t: stop announcing prereqs
2: 2c3bd12eb5d ! 2: ed6a8b205f0 t: silence output from `test_create_repo()`
@@ t/t4041-diff-submodule-option.sh: test_expect_success 'typechanged submodule(sub
-test_create_repo sm1 &&
-head6=$(add_file sm1 foo6 foo7)
-fullhead6=$(cd sm1; git rev-parse --verify HEAD)
-+test_expect_success 'setup submodule' '
++test_expect_success 'setup submodule anew' '
+ rm -f sm1 &&
+ git init sm1 &&
+ head6=$(add_file sm1 foo6 foo7) &&
3: c659a0ce551 = 3: cf05118aeae t9822: use prereq to check for ISO-8859-1 support
4: e7141b15b56 ! 4: 4d41989afe6 t983*: use prereq to check for Python-specific git-b4(1) support
@@ Metadata
Author: Patrick Steinhardt <ps@pks.im>
## Commit message ##
- t983*: use prereq to check for Python-specific git-b4(1) support
+ t983*: use prereq to check for Python-specific git-p4(1) support
- The tests in t9835 and t9836 verify that git-b4(1) works with both
+ The tests in t9835 and t9836 verify that git-p4(1) works with both
Python 2 and 3, respectively. To determine whether we have those Python
versions in the first place we create a wrapper script that directly
- executes the git-b4(1) script with `python2` or `python3` binaries. We
+ executes the git-p4(1) script with `python2` or `python3` binaries. We
then condition the execution of tests on whether that wrapper script can
be executed successfully.
5: 71b76db40e4 = 5: b3f5f4e0d4d t/test-lib: don't print shell traces to stdout
6: b60daf5ac69 ! 6: 78ab5b1d331 t/test-lib: fix TAP format for BASH_XTRACEFD warning
@@ Commit message
t/test-lib: fix TAP format for BASH_XTRACEFD warning
When the Bash version is too old to support BASH_XTRACEFD we print a
- warning to stderr. This warning breaks the TAP format because it is not
- prefixed with a "#". Fix this.
+ warning to stderr. This warning is not prefixed with "#", which causes
+ TAP parsers to (wrongly) interpret the warning as part of the protocol.
+
+ Fix this issue by prefixing the warning with a "#" so that it is treated
+ as comment.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
7: ce55bee9a12 ! 7: 61b8b7640b7 t7815: fix unexpectedly passing test on macOS
@@ Commit message
The test passes if '.' matches a NUL byte, which we expect to only
happen on Cygwin. The upcoming changes to support parsing TAP output in
- Meson surface that this test is also unexpectedly passing on macOS
- though.
+ Meson surface that this test, surprisingly, passes on macOS as well.
It is unclear how long the test has been passing on macOS already.
064eed36c7f (config.mak.uname: only set NO_REGEX on cygwin for v1.7,
- 2025-04-17) mentions that the test started to pass for Cygwin once it
- has imported a newer implementation of regcomp(3p) et all, which was
+ 2025-04-17) mentions that the test started to pass for Cygwin. This was
+ attributed to a new implementation of regcomp(3p) and friends, which was
inherited from FreeBSD. Given the BSD lineage of macOS it is feasible
that it also inherited similar code eventually that made the test pass
now.
@@ Commit message
same in this edge case and mark the new behaviour as expected.
This change is outside of the scope of this patch series, which only
- introduce support for TAP. So instead of fixing the bigger issue, ignore
- the test on Darwin like we already do for Cygwin.
+ introduces support for TAP. So instead of fixing the bigger issue,
+ ignore the test on Darwin like we already do for Cygwin.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
8: 80dcf1d5979 = 8: 02011b7017c test-lib: fail on unexpectedly passing tests
9: dca5730ab18 = 9: 540741acc80 meson: introduce kwargs variable for tests
10: 60393aa4af9 = 10: 8417d0ed94c meson: parse TAP output generated by our tests--- base-commit: 845c48a16a7f7b2c44d8cb137b16a4a1f0140229 change-id: 20250429-pks-meson-tap-1eed604a02a3