Re: [PATCH 2/2] test-lib: exhaustively insert non-alnum ASCII into the TRASH_DIRECTORY name

From: Jeff King <peff@peff.net>
Date: Apr 10, 2017, 19:22 UTC
Message-ID: <20170410192223.dbcysbbq5ei3wglj@sigill.intra.peff.net>
In-Reply-To: <d1a71326-06bc-4133-de2b-7d42f525f7a4@durchholz.org>

On Mon, Apr 10, 2017 at 08:19:40PM +0200, Joachim Durchholz wrote:

Show 7 quoted lines

> > I very much disagree with that. Git's test operate under a set of
> > assumptions, and if you violate those assumptions, then the failures are
> > not meaningful.
> 
> In that case the tests do not validate that git can properly work with
> special characters.
> That's a pretty big coverage gap.

That's not necessarily true either; there may be specific tests that create exotic paths and check them. My point is that the outcome depends on that paths. So you cannot just take a test which runs "git clone -s" and expect it to work both with and without paths with newlines. You need two tests, because there are two different outcomes, depending on the test environment.

So if you're proposing to write a bunch of new tests that check the proper behavior under various conditions, go for it. But I don't think running the entire existing test suite with exotic paths tells you much. A failure might be a bug, or it might be that the thing is untestable given the environment.

Show 6 quoted lines

> > Sure, and I'd encourage people who are interested to dig through the
> > results and see if they can find a real problem. I looked at several and
> > didn't find anything that wasn't an example of the "test assumptions"
> > thing above.
> 
> Don't assume that there's no risk just because you didn't find anything.

I'm not assuming that at all. Didn't I say somebody would need to dig into all of these to find out the real answer? I'm only arguing that blindly adding this feature to the test suite has no value. It's the digging that has value, and you do not even need to modify the test suite to do it (you can just use --root).

I've been trying to invite you to do that digging, if it's something you care about.

-Peff