[RFC/PATCH v3] Better control of the tests run by a test suite

From: Ilya Bobyr <ilya.bobyr@gmail.com>
Date: Apr 22, 2014, 08:19 UTC
Message-ID: <1398154767-1276-1-git-send-email-ilya.bobyr@gmail.com>

This patches add `--run` option to the test suites to allow one to run individual tests out of the test suite. Like this:

    ./t0000-basic.sh --run='-4,7,9-12,15-'

Both spaces and commas are accepted as separators for the ranges (In previous versions only spaces were accepted).

Two previous versions are here:

    [RFC/PATCH] Better control of the tests run by a test suite
    http://www.mail-archive.com/git@vger.kernel.org/msg46419.html

    [RFC/PATCH v2] Better control of the tests run by a test suite
    http://www.mail-archive.com/git@vger.kernel.org/msg46877.html

In this version I have removed mathematical operators and used ranges as suggested by Junio[1] and Eric Sunshine[2].

[1] http://www.mail-archive.com/git@vger.kernel.org/msg47098.html [2] http://www.mail-archive.com/git@vger.kernel.org/msg46960.html

This version also includes changes according to the comments from Eric Sunshine in the documentation. But as this version has slightly different documentation, it would be nice if someone would read it once again :)

Shell patterns are not allowed any more. I think they are not that useful and ranges cover almost the same functionality. Also with patterns like '[8-9]', it is harder to produce good error messages for invalid range ends.

This conversion is a bit unfinished:

On 3/31/2014 10:09 AM, Junio C Hamano wrote:

Show 8 quoted lines

> I would have to say that there is already an established pattern to
> pick ranges that normal people understand well and it would be silly
> to invent another more verbose way to express the same thing.  You
> tell your Print Dialog which page to print with e.g. "-4,7,9-12,15-",
> not ">=4 7 ...".  
>
> Would the same notation be insufficient for our purpose?  You do not
> even have to worry about negation that way.

    http://www.mail-archive.com/git@vger.kernel.org/msg47098.html

Negation was not necessary for my use cases even in the first version. I've added it more because it seemed to be very close to the functionality I was adding and not that complicated.

So, I've left the negation in the new version as well.

I am actually thinking now that --verbose-only= and --valgrind= could be switched to use the same syntax as in --run.

I also noticed that I am doing the following quite often:

    ./t0000-basic.sh --run=1-4,27 --verbose-only=27

Maybe it would be better to support 'v' suffix as a flag to indicate what a test needs to be run in verbose mode:

    ./t0000-basic.sh --run=1-4,27v

Ilya Bobyr (3):
  test-lib: Document short options in t/README
  test-lib: tests skipped by GIT_SKIP_TESTS say so
  test-lib: '--run' to run only specific tests

 t/README         |   81 ++++++++++-
 t/t0000-basic.sh |  419 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 t/test-lib.sh    |  120 +++++++++++++++-
 3 files changed, 604 insertions(+), 16 deletions(-)