From: Sverre Rabbelier <srabbelier@gmail.com>
Date: Tue, 17 Aug 2010 17:57:01 GMT
Subject: Re: [PATCH] t/t0000-basic.sh: Don't run a passing TODO unless TEST_PASSING_TODO=1
Message-ID: <AANLkTimx_kOr6UptsVOCdMHjxQqcde4oaDgrN2amnuYe@mail.gmail.com>
In-Reply-To: <1282038234-1880-1-git-send-email-avarab@gmail.com>

Heya,

On Tue, Aug 17, 2010 at 04:43, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> Change the sanity tests in t/t0000-basic.sh to not to run a passing
> TODO test unless the TEST_PASSING_TODO environment variable is set.
>
> The motivation is to have nothing out of the ordinary on a normal test
> run for test smoking purposes.
>
> If every normal test run has a passing TODO you're more likely to turn
> a blind eye to it and not to investigate cases where things really are
> passing unexpectedly.

Thanks, this has bothered me ever since I first ran the test suite.
While I understand that the test suite needs testing to, it should be
done in such a way that it doesn't produce false positives. That is,
my preference would be to still keep the test in the suite as-is, but
somehow get it to show up as a passing test instead of a fixed one.
E.g., something like:

test_in_new_harness 0 1 0 0 '
    test_expect_failure 'pretend we have fixed a known breakage' '
           :
     '
'

Where there '0 1 0 0' stands for '0 failures, 1 fixed, 0 broken, 0
success'. Not sure there's any other use for a test_in_new_harness
function though...

PS: To clarify, test_in_new_harness would not change any of the
counters (test_failure, test_count, test_fixed, test_broken,
test_success), but would instead compare those against the values
specified as argument to test_in_new_harness.

PSPS: I have no idea how easy it would be to implement something like this.

-- 
Cheers,

Sverre Rabbelier