From: Johannes Sixt <j6t@kdbg.org>
Date: Tue, 26 Jun 2018 18:05:55 GMT
Subject: Re: [PATCH 17/29] t: use test_must_fail() instead of checking exit code manually
Message-ID: <29d090a2-bbe4-0d87-6dda-037ae675d4a3@kdbg.org>
In-Reply-To: <CAPig+cRTG625H3CF1Zw30vQt2W8uKf1xLxVaQni2YbJ=xAif2g@mail.gmail.com>

Am 26.06.2018 um 11:21 schrieb Eric Sunshine:
> On Tue, Jun 26, 2018 at 4:58 AM Elijah Newren <newren@gmail.com> wrote:
>> On Tue, Jun 26, 2018 at 12:29 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>>> +               p4 help client &&
>>> +               test_must_fail p4 help nosuchcommand
>>
>> same question?
> 
> Same answer. Not shown in this patch, but just above the context lines
> you will find this comment in the file:
> 
>      # We rely on this behavior to detect for p4 move availability.
> 
> which means that the test is really interested in being able to
> reliably detect if a sub-command is or is not available. So, despite
> the (somewhat) misleading test title, this test doesn't care about the
> exact error code but rather cares only that "p4 help nosuchcommand"
> errors out, period. Hence, test_must_fail() again agrees with the
> spirit of the test.

test_must_fail ensures that only "proper" failures are diagnosed as 
expected; failures due to signals such as SEGV are not expected failures.

In the test suite we expect all programs that are not our "git" to work 
correctly; in particular, that they do not crash on anything that we ask 
them to operate on. Under this assumption, the protection given by 
test_must_fail is not needed.

Hence, these lines should actually be

		p4 help client &&
		! p4 help nosuchcommand

-- Hannes