Re: [PATCH 8/8] t/README: Document the dangers of printing "ok" / "not ok"

From: Junio C Hamano <gitster@pobox.com>
Date: Jul 1, 2010, 17:16 UTC
Message-ID: <7vhbkj2kcr.fsf@alter.siamese.dyndns.org>
In-Reply-To: <1277997004-29504-9-git-send-email-avarab@gmail.com>

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

Show 13 quoted lines

> index 42a4c25..29f795e 100644
> --- a/t/README
> +++ b/t/README
> @@ -50,6 +50,12 @@ prove and other harnesses come with a lot of useful options. The
>      # Repeat until no more failures
>      $ prove -j 15 --state=failed,save ./t[0-9]*.sh
>  
> +The TAP support is completely incidental. A TAP harness is just a
> +program that interprets the "ok"/"not ok" (and some other strings) in
> +a special way. The only limitation this imposes is that you shouldn't
> +echo "ok" or "not ok" by yourself at the beginning of a line, that'll
> +confuse the TAP harness.
> +

I think the above is not suitable for the target audience of the section ("Running Tests"); they are not _writing_ tests.

This information (or an enhanced version of it) should come between "Source 'test-lib.sh'" that talks about what the harness gives, and "Skipping tests". Perhaps name that to "Test your stuff". An example...

    Test your stuff
    ---------------

    After sourcing test-lib.sh, you test the parts of the system using the
    test harness.  Your tests will be done in the <script> part of
    test_expect_success helper function, and the exit status of the last
    command in that <script> signals the success/failure of a test.

    You are not supposed to "exit" from the <script> part; the harness
    will catch this as a programming error of the test.  Use test_done
    instead, if you need to stop the tests early (see "Skipping tests"
    below).

    Inside <script> part, the standard output and standard error streams
    are discarded, and the test harness only reports "ok" or "not ok" to
    the end user running the tests.  Under --verbose, they are shown to
    help debugging the tests.

    The output from the test suite is designed to be TAP compliant (see
    http://..... to learn more about TAP), to allow programs by other
    people to parse our output.  This further places restriction on what
    your tests can say to the standard output and standard error.  The TAP
    readers expect a line that begins with either "ok" and "not ok" to
    signal a test passed or failed (and our harness already produces such
    lines), so your script shouldn't emit such lines to their output.

I suspect the last paragraph needs a bit more work. For example, don't you have some restriction not to say things like "1..43"???