Re: test-lib.sh musings: test_expect_failure considered harmful

From: Junio C Hamano <gitster@pobox.com>
Date: Oct 12, 2021, 16:45 UTC
Message-ID: <xmqq4k9m6vkw.fsf@gitster.g>
In-Reply-To: <87tuhmk19c.fsf@evledraar.gmail.com>

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> On Mon, Oct 11 2021, Junio C Hamano wrote:
>
> [Removed "In-reply-to: <xmqq5yu3b80j.fsf@gitster.g>" with the Subject
> change]

Please do not do the former, although it is welcome to change Subject.

> Presumably with test_expect_failure.
>
> I'll change it, in this case we'd end up with a test_expect_success at
> the end, so it doesn't matter much & I don't care.

I do agree with you that compared to expect_success, which requires _all_ steps to succeed, so an failure in any of its steps is immediately noticeable, it is harder to write and keep expect_failure useful, because it is not like we are happy to see any failure in any step. We do not expect a failure in many preparation and conclusion steps in the &&-chain in expect_failure block, and we consider it is an error if these steps fail. We only want to mark only a single step to exhibit an expected but undesirable behaviour.

But even with the shortcomings of expect_failure, it still is much better than claiming that we expect a bogus outcome.

Improving the shortcomings of expect_failure would be a much better use of our time than advocating an abuse of expect_sucess, I would think.