Re: [PATCH 3/3] test-lib: '--run' to run only specific tests

From: Junio C Hamano <gitster@pobox.com>
Date: May 6, 2014, 20:53 UTC
Message-ID: <xmqqk39yeiic.fsf@gitster.dls.corp.google.com>
In-Reply-To: <1398851444-2492-4-git-send-email-ilya.bobyr@gmail.com>

Ilya Bobyr <ilya.bobyr@gmail.com> writes:

Show 7 quoted lines

> Allow better control of the set of tests that will be executed for a
> single test suite.  Mostly useful while debugging or developing as it
> allows to focus on a specific test.
>
> Signed-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>
> ---
> A number of minor changes according to the review comments.

I think the interaction between multiple selectors, especially when some of them are negated, are much better explained in this version, compared to the previous round in the README.

But I still think that the negation a feature that is unnecessary and having it makes it harder to understand for users, especially after reading this part:

Show 17 quoted lines

> +If --run starts with an unprefixed number or range the initial
> +set of tests to run is empty. If the first item starts with '!'
> +all the tests are added to the initial set.  After initial set is
> +determined every test number or range is added or excluded from
> +the set one by one, from left to right.
> ...
> +As noted above, the test set is built going though items left to
> +right, so this:
> +
> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
> +
> +will run tests 1, 2, and 4.  Items that comes later have higher
> +precendence.  It means that this:
> +
> +    $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'
> +
> +would just run tests from 1 to 4, including 3.

The initial !3 means the same thing as "1-2,4-", and then 1-4 will do what to that set? The answer is "It is added"... wouldn't the reader expect then that the result should be "1-", not "1-4"? I myself wondered what would happen to the fifth test from your description. Has the text told the reader that t9200 test has only four tests?

The need to explain better with longer description will reduce the likelyhood that the feature is understood and correctly used. When you can write "1-2,4-", why accept "1-4 !3" and force yourself to explain to people why that is different from "!3 1-4"?