From: Ævar Arnfjörð Bjarmason Date: Tue, 20 Jul 2010 18:43:29 GMT Subject: Re: [PATCH] t/README: clarify test_must_fail description Message-ID: In-Reply-To: <7v1vaym27n.fsf@alter.siamese.dyndns.org> On Tue, Jul 20, 2010 at 18:34, Junio C Hamano wrote: > Ævar Arnfjörð Bjarmason writes: > >> On Tue, Jul 20, 2010 at 18:00, Junio C Hamano wrote: >> >>> Run a git command and ensure it fails in a controlled way. Use >>> this instead of "! ". When git-command dies due to a >>> segfault, test_must_fail diagnoses it as an error; "! " >>> treats it as just another expected failure. letting such a bug go >>> unnoticed. >> >> To add to that: >> >> Don't use test_must_fail to negate the return values of commands >> on the system like grep, sed etc. If we can't trust that the core >> utilities won't randomly segfault we might as well die horribly. > > I think you are being incoherent. If we can't trust system "grep" and it > randomly segfaults, then a test: > > git some-command >actual && > ! grep string-that-should-not-be-in-the-output actual > > would _pass_ when the command segfaults. I do agree with you that "We > might as well die horribly", and the way you do so is by protecting the > test with test_must_fail, like this: > > git some-command >actual && > test_must_fail grep string-that-should-not-be-in-the-output actual > > Having said that, as we _do_ trust system tools to a certain degree, we do > not care very deeply about this. IOW, I wouldn't want to see a patch that > rewrites "! grep" to "test_must_fail grep". An individual test would pass, yes. But if test or grep are segfaulting we're going to bail out horribly eventually anyway, so I don't think it's worth the effort to guard them with test_must_fail, and I wouldn't write tests to do that. I'd just use !. That's what we seem to be doing in the tests so far, i.e. test_must_fail is reserved for git commands only.