Re: [PATCH 17/29] t: use test_must_fail() instead of checking exit code manually

From: Johannes Sixt <j6t@kdbg.org>
Date: Jun 26, 2018, 18:05 UTC
Message-ID: <29d090a2-bbe4-0d87-6dda-037ae675d4a3@kdbg.org>
In-Reply-To: <CAPig+cRTG625H3CF1Zw30vQt2W8uKf1xLxVaQni2YbJ=xAif2g@mail.gmail.com>

Am 26.06.2018 um 11:21 schrieb Eric Sunshine:

Show 18 quoted lines

> On Tue, Jun 26, 2018 at 4:58 AM Elijah Newren <newren@gmail.com> wrote:
>> On Tue, Jun 26, 2018 at 12:29 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>>> +               p4 help client &&
>>> +               test_must_fail p4 help nosuchcommand
>>
>> same question?
> 
> Same answer. Not shown in this patch, but just above the context lines
> you will find this comment in the file:
> 
>      # We rely on this behavior to detect for p4 move availability.
> 
> which means that the test is really interested in being able to
> reliably detect if a sub-command is or is not available. So, despite
> the (somewhat) misleading test title, this test doesn't care about the
> exact error code but rather cares only that "p4 help nosuchcommand"
> errors out, period. Hence, test_must_fail() again agrees with the
> spirit of the test.

test_must_fail ensures that only "proper" failures are diagnosed as expected; failures due to signals such as SEGV are not expected failures.

In the test suite we expect all programs that are not our "git" to work correctly; in particular, that they do not crash on anything that we ask them to operate on. Under this assumption, the protection given by test_must_fail is not needed.

Hence, these lines should actually be

		p4 help client &&
		! p4 help nosuchcommand

-- Hannes