Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests

From: Ilya Bobyr <ilya.bobir@gmail.com>
Date: Mar 3, 2014, 23:48 UTC
Message-ID: <531514D9.4000101@gmail.com>
In-Reply-To: <xmqqob1mn9t6.fsf@gitster.dls.corp.google.com>

On 3/3/2014 3:26 PM, Junio C Hamano wrote:

Show 29 quoted lines

> Eric Sunshine <sunshine@sunshineco.com> writes:
>
>> On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr <ilya.bobyr@gmail.com> wrote:
>>> This is a counterpart to GIT_SKIP_TESTS.  Mostly useful when debugging.
>> To be grammatically similar to GIT_SKIP_TESTS, perhaps name it GIT_RUN_TESTS?
> I actually do not like the interface to use two variables very much.
> Can't we just allow negative entries on "to be skipped" list?
>
> That is
>
> 	GIT_SKIP_TESTS='t9??? !t91??'
>
> would skip nine-thousand series, but would run 91xx series, and all
> the others are not excluded.
>
> Simple rules to consider:
>
>   - If the list consists of _only_ negated patterns, pretend that
>     there is "unless otherwise specified with negatives, skip all
>     tests", i.e. treat GIT_SKIP_TESTS='!t91??' just the same way you
>     would treat GIT_SKIP_TESTS='* !t91??'.
>
>   - The orders should not matter for simplicity of the semantics;
>     before running each test, check if it matches any negative (and
>     run it if it matches, without looking at any positives), and
>     otherwise check if it matches any positive (and skip it if it
>     does not).
>
> Hmm?

I can do that. But I am not sure that matches the use cases I had in mind the best.

First use case is that while developing I want to run tests frequently and I have a specific test that I am working on at the moment. That test is broken and I am trying to fix it (TDD). I want to run just the initialization test(s) and then that specific test. Running everything is quite slow.

GIT_RUN_ONLY addresses the TDD case.

Second case is when I broke one or more tests and want to figure out what is wrong. In this case running tests after the broken one will clutter the output directory and will make debugging somewhat harder, especially if I am not familiar with all the tests.

For the second case I was actually thinking that something like "<t9100.32" would be useful, where 32 is the broken test.

Maybe we can come up with an interface that covers all 3 cases?

While exclusion can be used it adds an extra step to both cases, as you need to mentally negate what you want first.

It might be that we are looking at different use cases, as you are talking about whole test suits.