Re: [PATCH v2] tests: Use skip_all=<reason> to skip tests

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Jul 9, 2010, 23:28 UTC
Message-ID: <AANLkTinAG4FgzsWyLtcVPu9Lz3e9Jp_PLlGRCuJeIWPN@mail.gmail.com>
In-Reply-To: <7vlj9kl1bv.fsf@alter.siamese.dyndns.org>

On Fri, Jul 9, 2010 at 22:47, Junio C Hamano <gitster@pobox.com> wrote:

Show 28 quoted lines

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> On Fri, Jul 9, 2010 at 17:49, Junio C Hamano <gitster@pobox.com> wrote:
>> ...
>>> Among 63 assignments to $skip_all that are all over in t/*.sh scripts, the
>>> only ones that are not immediately followed by test_done are in
>>> lib-git-svn.sh (chooses one among 3 messages), lib-httpd.sh (sets a trap
>>> before calling test_done), and t3600-rm (makes a mental note to report
>>> that one test was skipped long before all the tests run).
>> ...
>>     $ perl -MTest::More -E 'plan tests => 6; pass "gettext stuff ok"
>> for 1..2; SKIP: { skip "Can not test without locale files", 2 } pass
>> "moo" for 1..2'
>>     1..6
>>     ok 1 - gettext stuff ok
>>     ok 2 - gettext stuff ok
>>     ok 3 # skip Can not test without locale files
>>     ok 4 # skip Can not test without locale files
>>     ok 5 - moo
>>     ok 6 - moo
>
> Now you are talking.
>
> What t3600-rm does becomes a lot more natural to express with something
> like this.  Any test with "prerequisite" missing will automatically get
> "this test was skipped because you lack this prerequisite" for free.  We
> can lose skip_all= assignment there but move the logic to test-lib.sh,
> which is a good thing.

Right we already do this in the case of t3600-rm.sh. I think in the general case that using test_set_prereq + 3 arg test_expect_success (i.e. declare prereq) is a lot better than arranging tests so that you can skip a lot of dependency-heavy tests at the end.

Show 6 quoted lines

> But that is orthogonal to what you call an API, i.e. your assignment to
> the global variable $skip_all that is immediately followed by test_done,
> no?  The conversion you did for that does not help counting the remainder
> of the tests that are skipped anyway, so you will need to redo that
> conversion altogether if you ever want to be able to show "these tests
> through the end of the scripts were skipped".

Right, those two are completely orthogonal. That sort of API would look something like this:

    #!/bin/sh

    plan=6

    . ./test-lib.sh

    for i in {1..2}; do
        test_expect_success 'gettext stuff ok' 'true'
    done

    # something came up, skip the remainder
    skip_all="Oh noes"
    test_done

(or test_done 'Oh noes', or something).

At that point it knows to output 4 skip lines with the message "Oh noes" because we've only run two tests so far.