From: Junio C Hamano <gitster@pobox.com>
Date: Tue, 20 Jul 2010 18:34:52 GMT
Subject: Re: [PATCH] t/README: clarify test_must_fail description
Message-ID: <7v1vaym27n.fsf@alter.siamese.dyndns.org>
In-Reply-To: <AANLkTinLOLzmA9XSDYKsKwxV1Byvp-hd82JbjuSTNWb3@mail.gmail.com>

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> On Tue, Jul 20, 2010 at 18:00, Junio C Hamano <gitster@pobox.com> wrote:
>
>>    Run a git command and ensure it fails in a controlled way.  Use
>>    this instead of "! <git-command>".  When git-command dies due to a
>>    segfault, test_must_fail diagnoses it as an error; "! <git-command>"
>>    treats it as just another expected failure. letting such a bug go
>>    unnoticed.
>
> To add to that:
>
>     Don't use test_must_fail to negate the return values of commands
>     on the system like grep, sed etc. If we can't trust that the core
>     utilities won't randomly segfault we might as well die horribly.

I think you are being incoherent.  If we can't trust system "grep" and it
randomly segfaults, then a test:

    git some-command >actual &&
    ! grep string-that-should-not-be-in-the-output actual

would _pass_ when the command segfaults.  I do agree with you that "We
might as well die horribly", and the way you do so is by protecting the
test with test_must_fail, like this:

    git some-command >actual &&
    test_must_fail grep string-that-should-not-be-in-the-output actual

Having said that, as we _do_ trust system tools to a certain degree, we do
not care very deeply about this.  IOW, I wouldn't want to see a patch that
rewrites "! grep" to "test_must_fail grep".

Thanks.