Re: [PATCH v2 2/4] t1517: test commands that are designed to be run outside repository

From: Junio C Hamano <gitster@pobox.com>
Date: May 13, 2024, 21:00 UTC
Message-ID: <xmqqseyle8aw.fsf@gitster.g>
In-Reply-To: <xmqqy18de9kq.fsf@gitster.g>

Junio C Hamano <gitster@pobox.com> writes:

Show 8 quoted lines

> Kyle Lippincott <spectral@google.com> writes:
>
>> Do we only expect failure because of a temporary condition (the bug
>> that is mentioned in the commit message)? If so, we should probably
>> add a TODO, FIXME, or some other similar style of comment that
>> describes that this should be fixed.
>
> test_expect_failure is description enough for that purpose.

We say this in t/README:

 - test_expect_failure [<prereq>] <message> <script>

   This is NOT the opposite of test_expect_success, but is used
   to mark a test that demonstrates a known breakage.  Unlike
   the usual test_expect_success tests, which say "ok" on
   success and "FAIL" on failure, this will say "FIXED" on
   success and "still broken" on failure.  Failures from these
   tests won't cause -i (immediate) to stop.

Which means that when somebody rans out of things to do, grepping for test_expect_failure may give them a good place to start ;-).

Note that there were a few very rare occasions that what was marked as "known breakage" with test_expect_failure turned out to be what was working as intended.

Thanks.