From: Kyle Lippincott <spectral@google.com>
Date: Mon, 13 May 2024 21:07:38 GMT
Subject: Re: [PATCH v2 2/4] t1517: test commands that are designed to be run outside repository
Message-ID: <CAO_smVgj85+LdGyhwgyzrCV-KYJ7r+zhJq-cBq_2wa+Qs1bbjQ@mail.gmail.com>
In-Reply-To: <xmqqseyle8aw.fsf@gitster.g>

On Mon, May 13, 2024 at 2:00 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Junio C Hamano <gitster@pobox.com> writes:
>
> > Kyle Lippincott <spectral@google.com> writes:
> >
> >> Do we only expect failure because of a temporary condition (the bug
> >> that is mentioned in the commit message)? If so, we should probably
> >> add a TODO, FIXME, or some other similar style of comment that
> >> describes that this should be fixed.
> >
> > test_expect_failure is description enough for that purpose.
>
> We say this in t/README:
>
>  - test_expect_failure [<prereq>] <message> <script>
>
>    This is NOT the opposite of test_expect_success, but is used
>    to mark a test that demonstrates a known breakage.  Unlike
>    the usual test_expect_success tests, which say "ok" on
>    success and "FAIL" on failure, this will say "FIXED" on
>    success and "still broken" on failure.  Failures from these
>    tests won't cause -i (immediate) to stop.

Got it, thanks for explaining. With that, this change looks good to me.

>
> Which means that when somebody rans out of things to do, grepping
> for test_expect_failure may give them a good place to start ;-).
>
> Note that there were a few very rare occasions that what was marked
> as "known breakage" with test_expect_failure turned out to be what
> was working as intended.
>
> Thanks.