Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run

From: Junio C Hamano <gitster@pobox.com>
Date: Oct 16, 2020, 16:16 UTC
Message-ID: <xmqq7drqdu2u.fsf@gitster.c.googlers.com>
In-Reply-To: <20201016003847.GB2932796@coredump.intra.peff.net>

Jeff King <peff@peff.net> writes:

Show 9 quoted lines

>> Ah, I see.  I actually do use "git ls-files t/ | grep ..." to look
>> for tests that are relevant to the issue I have at hand quite often,
>> so unlike what Jonathan said in the thread, having a good name does
>> matter to me.
>
> I'm not sure where he suggests worse names. I'd think if anything we'd
> have better names, because they'd be even more meaningful (if people
> start using them for test selectors). FWIW, I also grep like that when
> looking for scripts.

I didn't mean Jonathan suggested worse names. Unlike "I don't tend to discover test scripts based on their filename", which was what Jonathan said, I do look for tests based on their filename, so having a good name matters (on the other hand, if you are the kind of person who does not look for them by name, the naming may not matter to you).

Show 7 quoted lines

> My plan is that you'd be able to say:
>
>   Meta/make --test=filesystem,update-ref
>
> for t0050 and t1400 respectively. Or more interesting things like
> "git-svn-*" to skip all of the t91xx tests (or "git-svn/*" if we split
> them by directories).

Most of the time, I am focused on running only the selected few, not excluding a few known-broken ones.

For example, with an round of integration, I know only two tests fail, and after I tried to fix the bug in the series, I want to retry these two tests and nothing else to see if the attempted fix patches the breakage up (of course I need to run the remainder of the tests to ensure there is no regression, but that is just the matter of running "make test" to run the whole thing again).

As long as it is known that "filesystem" and "update-ref" can serve as tokens to uniquely identify these two tests, it would be fine for my purpose. But 0050 (under the rule that numbering must be unique) would give me such an assurance much better without having to look at any other test file.

The word "filesystem"? Unless we have a rule that we can use each unique word in test names only once (which of course is impractical) I am not sure I can use it in place of 0050 without checking names of other tests first.

> I definitely don't want to make anybody's life harder. But if the
> numbers aren't unique and the text connected to them is, then it seems
> like we should just use the text primarily.

True in principle, but it is harder to come up with unique substring of text, knowing only that the whole string is unique.

Thanks.