Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests

From: Ilya Bobyr <ilya.bobir@gmail.com>
Date: Mar 4, 2014, 07:06 UTC
Message-ID: <53157B72.3000205@gmail.com>
In-Reply-To: <xmqq4n3en7un.fsf@gitster.dls.corp.google.com>

On 3/3/2014 4:08 PM, Junio C Hamano wrote:

Show 5 quoted lines

> Ilya Bobyr <ilya.bobir@gmail.com> writes:
>
>> It might be that we are looking at different use cases, as you are
>> talking about whole test suits.
> I do not think so.

Good :) I am trying to understand the use cases. And make sure we are talking about the same ones. I am not sure what are the use cases for GIT_SKIP_TESTS.

I think that while in it really nice when an interface allows to do new things, the main use case (or use cases) should be as easy and obvious in the first place. If the target is the TDD use case I described, then it appears that a user needs to do double negation. That is what concerns me.

Show 7 quoted lines

> I do not see anything prevents you from saying
>
> 	GIT_SKIP_TESTS='t0000 !t0000.1 !t0000.4'
>
> to specify test-pieces in individual tests so that you can run the
> setup step (step .1) and the specific test (step .4) without running
> two tests in between.

While it could be done, it looks less obvious than this:

     GIT_TEST_ONLY='1 4' ./t0001-init.sh

What if we do what you proposed, but with GIT_RUN_TESTS? If we want to have one interface, maybe, building on top of a "negation" (GIT_SKIP_TESTS) is not very good. At the same time, GIT_TEST_ONLY is also too specific for a generic interface. I could add GIT_RUN_TESTS and allow it to have all of the features, thus making GIT_SKIP_TESTS a working but deprecated tool.

Running specific tests:
     GIT_RUN_TESTS='t0000'
     GIT_RUN_TESTS='t0000.1 t0002'
     GIT_RUN_TESTS='1 3 7'

Negating some tests:
     GIT_RUN_TESTS='!t0000'
     GIT_RUN_TESTS='!t0000.1'
     GIT_RUN_TESTS='!1 !3 !7'

The above would work exactly as you described but with one less level of negation. Default is everything, unless at least one positive pattern is given.

Later on range specification could added:
     GIT_RUN_TESTS='<11'

At least for now that would cover most use cases that I can think of that look reasonable.