git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 4/4] t: let prove fail when parsing invalid TAP output

From
Patrick Steinhardt <ps@pks.im>
Date
Jun 3, 2026, 05:39 UTC
Message-ID
<20260603-pks-t7527-fix-tap-output-v2-4-cf3af5694e20@pks.im>
In-Reply-To
<20260603-pks-t7527-fix-tap-output-v2-0-cf3af5694e20@pks.im>

To make the result of our tests accessible we use the TAP protocol. This protocol is parsed by either prove or by Meson. Unfortunately, these two tools differ when it comes to their strictness when parsing the protocol:

  - Prove by default happily accepts lines not specified by the
    protocol.
  - Meson will also accept such lines, but prints a big and ugly warning
    message.

We have fixed our test suite in the past to not print invalid TAP lines anymore via b1dc2e796e (Merge branch 'ps/meson-tap-parse', 2025-06-17). But as none of our tools perform a strict check it's still possible for broken tests to sneak back in, like for example in 362f69547f (Merge branch 'ps/t1006-tap-fix', 2025-07-16). This doesn't hurt at all when using prove, but it's quite annoying when using Meson due to the generated warnings.

Unfortunately, there doesn't seem to be a portable way to make all tools complain about violations of the TAP format. The TAP 14 specification has added pragmas to the protocol that would allow us to say `pragma +strict`, and the effect of that would be to treat invalid TAP lines as a test failure. But the release of TAP 14 is still rather recent, and Test-Harness for example only gained support for it in version 3.48, which was released in 2023.

In fact though, this pragma was already introduced as an inofficial extension of the TAP protocol with Test-Harness 3.10, released in 2008. So while not all tools understand the pragma, at least prove does for a long time.

Unconditionally enable the pragma when using prove so that we'll detect tests that emit broken TAP output right away. This would have detected the issues fixed in preceding commits:

    $ prove t7527-builtin-fsmonitor.sh
    t7527-builtin-fsmonitor.sh .. All 69 subtests passed
            (less 6 skipped subtests: 63 okay)
    Test Summary Report
    -------------------
    t7527-builtin-fsmonitor.sh (Wstat: 0 Tests: 69 Failed: 0)
      Parse errors: Unknown TAP token: "Initialized empty Git repository in /tmp/git/test_fsmonitor_smoke/.git/"
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/test-lib.sh | 6 ++++++
 1 file changed, 6 insertions(+)
diff --git a/t/test-lib.sh b/t/test-lib.sh
index d1d24c4124..ceefb99bff 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -1532,6 +1532,12 @@ then
 	BAIL_OUT 'You need to build test-tool; Run "make t/helper/test-tool" in the source (toplevel) directory'
 fi
 
+if test -n "$HARNESS_ACTIVE"
+then
+	say "TAP version 13"
+	say "pragma +strict"
+fi
+
 # Are we running this test at all?
 remove_trash=
 this_test=${0##*/}
-- 
2.54.0.1064.gd145956f57.dirty
Previous: Patrick Steinhardt
Message 5 of 5 in “t: fix broken TAP output”
  1. 0/4 t: fix broken TAP outputPatrick Steinhardt, Jun 3, 2026
  2. 1/4 t7527: fix broken TAP outputPatrick Steinhardt, Jun 3, 2026
  3. 2/4 t/test-lib: silence EBUSY errors on Windows during test cleanupPatrick Steinhardt, Jun 3, 2026
  4. 3/4 t/lib-git-p4: silence output when killing p4d and its watchdogPatrick Steinhardt, Jun 3, 2026
  5. 4/4 t: let prove fail when parsing invalid TAP outputPatrick Steinhardt, Jun 3, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.