Re: [PATCH v4 1/5] test-lib: Adjust output to be valid TAP format
- From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
- Date
- Jun 24, 2010, 19:17 UTC
- Message-ID
- <AANLkTinVp6kivNzbdUj-7EXjKrolBqvtsNROo-dCSydn@mail.gmail.com>
- In-Reply-To
- <m3mxuknumn.fsf@localhost.localdomain>
On Thu, Jun 24, 2010 at 18:39, Jakub Narebski <jnareb@gmail.com> wrote:
Show 77 quoted lines
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > >> TAP, the Test Anything Protocol, is a simple text-based interface >> between testing modules in a test harness. test-lib.sh's output was >> already very close to being valid TAP. This change brings it all the >> way there. Before: >> >> $ ./t0005-signals.sh >> * ok 1: sigchain works >> * passed all 1 test(s) >> >> And after: >> >> $ ./t0005-signals.sh >> ok 1 - sigchain works >> # passed all 1 test(s) >> 1..1 > > This is incomplete information. It only tells us how the output is > changed for the case when all test_expect_success tests passes. This > commit message doesn't tell us how failing tests looks like, and how > test_expect_failure results gets translated. > > Take for example a following test script: > -- >8 -- > #!/bin/sh > > test_description='this is a sample test. > > This test is here to see various test outputs.' > > . ./test-lib.sh > > say 'diagnostic message' > > test_expect_success 'true test' 'true' > test_expect_success 'false test' 'false' > > test_expect_failure 'true test (todo)' 'true' > test_expect_failure 'false test (todo)' 'false' > > test_debug 'echo "debug message"' > > test_done > -- 8< -- > > This test script output looks like the following (the comments are not > part of output, but denote color of given line of test output): > ---- > * diagnostic message # yellow > * ok 1: true test > * FAIL 2: false test # bold red > false > * FIXED 3: true test (todo) > * still broken 4: false test (todo) # bold green > * fixed 1 known breakage(s) # green > * still have 1 known breakage(s) # bold red > * failed 1 among remaining 3 test(s) # bold red > ---- > > How would the output of this test look like after TAP-ification? > Would it still provide color output when run on terminal? > > This test also generates summary of test run in a file in > test-results/ subdirectory. Currently such file can be e.g. named > test-results/test_test-8030, and consist of > -- >8 -- > total 4 > success 1 > fixed 1 > broken 1 > failed 1 > > -- 8< -- > > Would TAP-ification change that? Even if there is no change, this is > worth a sentence or a few words in a commit message.
To all of the above: I could be more verbose in the commit message in needed, but after you raised similar questions in a previous version of this series I added this to the above commit message:
All this patch does is re-arrange the output a bit so that it conforms
with the TAP spec, everything that the test suite did before continues
to work. That includes aggregating results in t/test-results/, the
--verbose, --debug and other options for tests, and the test color
output.I.e. aside from turning "FAIL" into "not ok" and removing the "* " prefix from lines everything else works exactly as before, that includes color output and the aggregation in t/test-results/.
I can expand on that paragraph with more examples if you think it's necessary.
Here's how the output from the above test script looks under TAP:
diagnostic message
ok 1 - true test
not ok - 2 false test
# false
ok 3 - true test (todo) # TODO known breakage
not ok 4 - false test (todo) # TODO known breakage
# fixed 1 known breakage(s)
# still have 1 known breakage(s)
# failed 1 among remaining 3 test(s)
1..4> P.S. With current output pass / fail results of test_expect_failure > are justified on ':' separator. This is not the case of TAP output.
TAP output is conventionally not justified, unlike the existing output. I don't know if anything would break if it were, i.e.:
ok 1 - foo
not ok 2 - bar