Re: [PATCH 1/2] tests: use shell negation instead of test_must_fail for test_cmp

From: Junio C Hamano <gitster@pobox.com>
Date: Oct 7, 2017, 02:12 UTC
Message-ID: <xmqq1smfvh8c.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <20171006192252.3a27enzim5yurhe5@sigill.intra.peff.net>

Jeff King <peff@peff.net> writes:

Show 15 quoted lines

> Hmph. "test_must_fail test_cmp" is a weird thing for somebody to write.
> And your patch is obviously an improvement, but I have to wonder if some
> of these make any sense.
>
> If we're expecting some outcome, then it's reasonable to say:
>
>   1. The output should look exactly like this. (test_cmp)
>
>   2. The output should look something like this. (grep)
>
>   3. The output should _not_ mention this (! grep)
>
> But "the output should not look exactly like this" doesn't seem very
> robust. It's likely to give a false success due to small changes (or
> translations), or even bugs in the script.

Yeah, thanks for stepping back and looking at it from higher level.

$ git grep -e 'test_must_fail test_cmp' -e '! test_cmp' t/

gives 21 hits, in addition to the 5 Stefan identified, and it would be a promising hunt to go through each one of them to see if they make sense.