{"thread":{"id":"36472","subject":"[RFC/PATCH v3] Better control of the tests run by a test suite","startedAt":"2014-04-22T08:19:24Z","lastAt":"2014-05-06T21:02:31Z","messageCount":17,"participants":["Ilya Bobyr","Junio C Hamano","Eric Sunshine"],"isPatch":true,"patchVersion":3,"patchTotal":null},"messages":[{"id":"239310","messageId":"1398154767-1276-1-git-send-email-ilya.bobyr@gmail.com","threadId":"36472","inReplyTo":null,"subject":"[RFC/PATCH v3] Better control of the tests run by a test suite","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-22T08:19:24Z","receivedAt":"2014-04-22T08:19:24Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"This patches add `--run` option to the test suites to allow one to run\nindividual tests out of the test suite.  Like this:\n\n    ./t0000-basic.sh --run='-4,7,9-12,15-'\n\nBoth spaces and commas are accepted as separators for the ranges (In\nprevious versions only spaces were accepted).\n\nTwo previous versions are here:\n\n    [RFC/PATCH] Better control of the tests run by a test suite\n    http://www.mail-archive.com/git@vger.kernel.org/msg46419.html\n\n    [RFC/PATCH v2] Better control of the tests run by a test suite\n    http://www.mail-archive.com/git@vger.kernel.org/msg46877.html\n\nIn this version I have removed mathematical operators and used ranges as\nsuggested by Junio[1] and Eric Sunshine[2].\n\n[1] http://www.mail-archive.com/git@vger.kernel.org/msg47098.html\n[2] http://www.mail-archive.com/git@vger.kernel.org/msg46960.html\n\nThis version also includes changes according to the comments from Eric\nSunshine in the documentation.  But as this version has slightly different\ndocumentation, it would be nice if someone would read it once again :)\n\nShell patterns are not allowed any more.  I think they are not that useful\nand ranges cover almost the same functionality.  Also with patterns like\n'[8-9]', it is harder to produce good error messages for invalid range\nends.\n\nThis conversion is a bit unfinished:\n\nOn 3/31/2014 10:09 AM, Junio C Hamano wrote:\n> I would have to say that there is already an established pattern to\n> pick ranges that normal people understand well and it would be silly\n> to invent another more verbose way to express the same thing.  You\n> tell your Print Dialog which page to print with e.g. \"-4,7,9-12,15-\",\n> not \">=4 7 ...\".  \n>\n> Would the same notation be insufficient for our purpose?  You do not\n> even have to worry about negation that way.\n\n    http://www.mail-archive.com/git@vger.kernel.org/msg47098.html\n\nNegation was not necessary for my use cases even in the first version.\nI've added it more because it seemed to be very close to the functionality\nI was adding and not that complicated.\n\nSo, I've left the negation in the new version as well.\n\n\nI am actually thinking now that --verbose-only= and --valgrind= could be\nswitched to use the same syntax as in --run.\n\nI also noticed that I am doing the following quite often:\n\n    ./t0000-basic.sh --run=1-4,27 --verbose-only=27\n\nMaybe it would be better to support 'v' suffix as a flag to indicate what\na test needs to be run in verbose mode:\n\n    ./t0000-basic.sh --run=1-4,27v\n\n\nIlya Bobyr (3):\n  test-lib: Document short options in t/README\n  test-lib: tests skipped by GIT_SKIP_TESTS say so\n  test-lib: '--run' to run only specific tests\n\n t/README         |   81 ++++++++++-\n t/t0000-basic.sh |  419 +++++++++++++++++++++++++++++++++++++++++++++++++++++-\n t/test-lib.sh    |  120 +++++++++++++++-\n 3 files changed, 604 insertions(+), 16 deletions(-)\n"},{"id":"239312","messageId":"1398154767-1276-2-git-send-email-ilya.bobyr@gmail.com","threadId":"36472","inReplyTo":"1398154767-1276-1-git-send-email-ilya.bobyr@gmail.com","subject":"[PATCH 1/3] test-lib: Document short options in t/README","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-22T08:19:25Z","receivedAt":"2014-04-22T08:19:25Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"Most arguments that could be provided to a test have short forms.\nUnless documented, the only way to learn them is to read the code.\n\nSigned-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n---\n t/README |    8 ++++----\n 1 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex caeeb9d..6b93aca 100644\n--- a/t/README\n+++ b/t/README\n@@ -71,7 +71,7 @@ You can pass --verbose (or -v), --debug (or -d), and --immediate\n (or -i) command line argument to the test, or by setting GIT_TEST_OPTS\n appropriately before running \"make\".\n \n---verbose::\n+-v,--verbose::\n \tThis makes the test more verbose.  Specifically, the\n \tcommand being run and their output if any are also\n \toutput.\n@@ -81,7 +81,7 @@ appropriately before running \"make\".\n \tnumbers matching <pattern>.  The number matched against is\n \tsimply the running count of the test within the file.\n \n---debug::\n+-d,--debug::\n \tThis may help the person who is developing a new test.\n \tIt causes the command defined with test_debug to run.\n \tThe \"trash\" directory (used to store all temporary data\n@@ -89,14 +89,14 @@ appropriately before running \"make\".\n \tfailed tests so that you can inspect its contents after\n \tthe test finished.\n \n---immediate::\n+-i,--immediate::\n \tThis causes the test to immediately exit upon the first\n \tfailed test. Cleanup commands requested with\n \ttest_when_finished are not executed if the test failed,\n \tin order to keep the state for inspection by the tester\n \tto diagnose the bug.\n \n---long-tests::\n+-l,--long-tests::\n \tThis causes additional long-running tests to be run (where\n \tavailable), for more exhaustive testing.\n \n-- \n1.7.9\n"},{"id":"239311","messageId":"1398154767-1276-3-git-send-email-ilya.bobyr@gmail.com","threadId":"36472","inReplyTo":"1398154767-1276-1-git-send-email-ilya.bobyr@gmail.com","subject":"[PATCH 2/3] test-lib: tests skipped by GIT_SKIP_TESTS say so","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-22T08:19:26Z","receivedAt":"2014-04-22T08:19:26Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"We used to show \"(missing )\" next to tests skipped because they are\nspecified in GIT_SKIP_TESTS.  Use \"(GIT_SKIP_TESTS)\" instead.\n\nPlus tests that check basic GIT_SKIP_TESTS functions.\n\nSigned-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n---\n t/t0000-basic.sh |   63 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n t/test-lib.sh    |   13 ++++++----\n 2 files changed, 71 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex a2bb63c..ae8874e 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -270,6 +270,69 @@ test_expect_success 'test --verbose-only' '\n \tEOF\n '\n \n+test_expect_success 'GIT_SKIP_TESTS' \"\n+\tGIT_SKIP_TESTS='git.2' \\\n+\t\trun_sub_test_lib_test git-skip-tests-basic \\\n+\t\t'GIT_SKIP_TESTS' <<-\\\\EOF &&\n+\tfor i in 1 2 3\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test git-skip-tests-basic <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)\n+\t> ok 3 - passing test #3\n+\t> # passed all 3 test(s)\n+\t> 1..3\n+\tEOF\n+\"\n+\n+test_expect_success 'GIT_SKIP_TESTS several tests' \"\n+\tGIT_SKIP_TESTS='git.2 git.5' \\\n+\t\trun_sub_test_lib_test git-skip-tests-several \\\n+\t\t'GIT_SKIP_TESTS several tests' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test git-skip-tests-several <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)\n+\t> ok 3 - passing test #3\n+\t> ok 4 - passing test #4\n+\t> ok 5 # skip passing test #5 (GIT_SKIP_TESTS)\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success 'GIT_SKIP_TESTS sh pattern' \"\n+\tGIT_SKIP_TESTS='git.[2-5]' \\\n+\t\trun_sub_test_lib_test git-skip-tests-sh-pattern \\\n+\t\t'GIT_SKIP_TESTS sh pattern' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test git-skip-tests-sh-pattern <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)\n+\t> ok 3 # skip passing test #3 (GIT_SKIP_TESTS)\n+\t> ok 4 # skip passing test #4 (GIT_SKIP_TESTS)\n+\t> ok 5 # skip passing test #5 (GIT_SKIP_TESTS)\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n test_set_prereq HAVEIT\n haveit=no\n test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex c081668..e7d9c51 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -452,25 +452,28 @@ test_finish_ () {\n \n test_skip () {\n \tto_skip=\n+\tskipped_reason=\n \tif match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS\n \tthen\n \t\tto_skip=t\n+\t\tskipped_reason=\"GIT_SKIP_TESTS\"\n \tfi\n \tif test -z \"$to_skip\" && test -n \"$test_prereq\" &&\n \t   ! test_have_prereq \"$test_prereq\"\n \tthen\n \t\tto_skip=t\n-\tfi\n-\tcase \"$to_skip\" in\n-\tt)\n+\n \t\tof_prereq=\n \t\tif test \"$missing_prereq\" != \"$test_prereq\"\n \t\tthen\n \t\t\tof_prereq=\" of $test_prereq\"\n \t\tfi\n-\n+\t\tskipped_reason=\"missing $missing_prereq${of_prereq}\"\n+\tfi\n+\tcase \"$to_skip\" in\n+\tt)\n \t\tsay_color skip >&3 \"skipping test: $@\"\n-\t\tsay_color skip \"ok $test_count # skip $1 (missing $missing_prereq${of_prereq})\"\n+\t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n \t*)\n-- \n1.7.9\n"},{"id":"239313","messageId":"1398154767-1276-4-git-send-email-ilya.bobyr@gmail.com","threadId":"36472","inReplyTo":"1398154767-1276-1-git-send-email-ilya.bobyr@gmail.com","subject":"[PATCH 3/3] test-lib: '--run' to run only specific tests","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-22T08:19:27Z","receivedAt":"2014-04-22T08:19:27Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"Allow better control of the set of tests that will be executed for a\nsingle test suite.  Mostly useful while debugging or developing as it\nallows to focus on a specific test.\n\nSigned-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n---\n t/README         |   73 +++++++++++-\n t/t0000-basic.sh |  356 +++++++++++++++++++++++++++++++++++++++++++++++++++++-\n t/test-lib.sh    |  109 +++++++++++++++++\n 3 files changed, 530 insertions(+), 8 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 6b93aca..2dac619 100644\n--- a/t/README\n+++ b/t/README\n@@ -100,6 +100,11 @@ appropriately before running \"make\".\n \tThis causes additional long-running tests to be run (where\n \tavailable), for more exhaustive testing.\n \n+-r,--run=<test-selector>::\n+\tRun only the subset of tests indicated by\n+\t<test-selector>.  See section \"Skipping Tests\" below for\n+\t<test-selector> syntax.\n+\n --valgrind=<tool>::\n \tExecute all Git binaries under valgrind tool <tool> and exit\n \twith status 126 on errors (just like regular tests, this will\n@@ -187,10 +192,70 @@ and either can match the \"t[0-9]{4}\" part to skip the whole\n test, or t[0-9]{4} followed by \".$number\" to say which\n particular test to skip.\n \n-Note that some tests in the existing test suite rely on previous\n-test item, so you cannot arbitrarily disable one and expect the\n-remainder of test to check what the test originally was intended\n-to check.\n+For an individual test suite --run could be used to specify that\n+only some tests should be run or that some tests should be\n+excluded from a run.\n+\n+The argument for --run is a list of individual test numbers or\n+ranges with an optional negation prefix that define what tests in\n+a test suite to include in the run.  A range is two numbers\n+separated with a dash and matches a range of tests with both ends\n+been included.  You may omit the first or the second number to\n+mean \"from the first test\" or \"up to the very last test\"\n+respectively.\n+\n+Optional prefix of '!' means that the test or a range of tests\n+should be excluded from the run.\n+\n+If --run starts with an unprefixed number or range the initial\n+set of tests to run is empty. If the first item starts with '!'\n+all the tests are added to the initial set.  After initial set is\n+determined every test number or range is added or excluded from\n+the set one by one, from left to right.\n+\n+Individual numbers or ranges could be separated either by a space\n+or a comma.\n+\n+For example, common case is to run several setup tests (1, 2, 3)\n+and then a specific test (21) that relies on that setup:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'\n+\n+or:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run=1,2,3,21\n+\n+or:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'\n+\n+To run only tests up to a specific test (21), one could do this:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='1-21'\n+\n+or this:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='-21'\n+\n+As noted above, the test set is built going though items left to\n+right, so this:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n+\n+will run tests 1, 2, and 4.\n+\n+You may use negation with ranges.  The following will run all\n+test as a test suite except from 7 upto 11:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n+\n+Some tests in a test suite rely on the previous tests performing\n+certain actions, specifically some tests are designated as\n+\"setup\" test, so you cannot _arbitrarily_ disable one test and\n+expect the rest to function correctly.\n+--run is mostly useful when you want to focus on a specific test\n+and know what you are doing.  Or when you want to run up to a\n+certain test.\n \n \n Naming Tests\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex ae8874e..e2589cc 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -42,9 +42,9 @@ test_expect_success 'success is reported like this' '\n \t:\n '\n \n-run_sub_test_lib_test () {\n-\tname=\"$1\" descr=\"$2\" # stdin is the body of the test code\n-\tshift 2\n+_run_sub_test_lib_test_common () {\n+\tneg=\"$1\" name=\"$2\" descr=\"$3\" # stdin is the body of the test code\n+\tshift 3\n \tmkdir \"$name\" &&\n \t(\n \t\t# Pretend we're not running under a test harness, whether we\n@@ -70,10 +70,23 @@ run_sub_test_lib_test () {\n \t\texport TEST_DIRECTORY &&\n \t\tTEST_OUTPUT_DIRECTORY=$(pwd) &&\n \t\texport TEST_OUTPUT_DIRECTORY &&\n-\t\t./\"$name.sh\" \"$@\" >out 2>err\n+\t\tif test -z \"$neg\"\n+\t\tthen\n+\t\t\t./\"$name.sh\" \"$@\" >out 2>err\n+\t\telse\n+\t\t\t!  ./\"$name.sh\" \"$@\" >out 2>err\n+\t\tfi\n \t)\n }\n \n+run_sub_test_lib_test () {\n+\t_run_sub_test_lib_test_common '' \"$@\"\n+}\n+\n+run_sub_test_lib_test_err () {\n+\t_run_sub_test_lib_test_common '!' \"$@\"\n+}\n+\n check_sub_test_lib_test () {\n \tname=\"$1\" # stdin is the expected output from the test\n \t(\n@@ -84,6 +97,18 @@ check_sub_test_lib_test () {\n \t)\n }\n \n+check_sub_test_lib_test_err () {\n+\tname=\"$1\" # stdin is the expected output output from the test\n+\t# expecte error output is in descriptior 3\n+\t(\n+\t\tcd \"$name\" &&\n+\t\tsed -e 's/^> //' -e 's/Z$//' >expect.out &&\n+\t\ttest_cmp expect.out out &&\n+\t\tsed -e 's/^> //' -e 's/Z$//' <&3 >expect.err &&\n+\t\ttest_cmp expect.err err\n+\t)\n+}\n+\n test_expect_success 'pretend we have a fully passing test suite' \"\n \trun_sub_test_lib_test full-pass '3 passing tests' <<-\\\\EOF &&\n \tfor i in 1 2 3\n@@ -333,6 +358,329 @@ test_expect_success 'GIT_SKIP_TESTS sh pattern' \"\n \tEOF\n \"\n \n+test_expect_success '--run basic' \"\n+\trun_sub_test_lib_test run-basic \\\n+\t\t'--run basic' --run='1 3 5' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-basic <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 - passing test #3\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with a range' \"\n+\trun_sub_test_lib_test run-range \\\n+\t\t'--run with a range' --run='1-3' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-range <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 - passing test #3\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 # skip passing test #5 (--run)\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with two ranges' \"\n+\trun_sub_test_lib_test run-two-ranges \\\n+\t\t'--run with two ranges' --run='1-2 5-6' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-two-ranges <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 - passing test #5\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with a left open range' \"\n+\trun_sub_test_lib_test run-left-open-range \\\n+\t\t'--run with a left open range' --run='-3' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-left-open-range <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 - passing test #3\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 # skip passing test #5 (--run)\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with a right open range' \"\n+\trun_sub_test_lib_test run-right-open-range \\\n+\t\t'--run with a right open range' --run='4-' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-right-open-range <<-\\\\EOF\n+\t> ok 1 # skip passing test #1 (--run)\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with basic negation' \"\n+\trun_sub_test_lib_test run-basic-neg \\\n+\t\t'--run with basic negation' --run='\"'!3'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-basic-neg <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with two negations' \"\n+\trun_sub_test_lib_test run-two-neg \\\n+\t\t'--run with two negations' --run='\"'!3 !6'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-two-neg <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run a range and negation' \"\n+\trun_sub_test_lib_test run-range-and-neg \\\n+\t\t'--run a range and negation' --run='\"'-4 !2'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-range-and-neg <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 - passing test #3\n+\t> ok 4 - passing test #4\n+\t> ok 5 # skip passing test #5 (--run)\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run range negation' \"\n+\trun_sub_test_lib_test run-range-neg \\\n+\t\t'--run range negation' --run='\"'!1-3'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-range-neg <<-\\\\EOF\n+\t> ok 1 # skip passing test #1 (--run)\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run include, exclude and include' \"\n+\trun_sub_test_lib_test run-inc-neg-inc \\\n+\t\t'--run include, exclude and include' \\\n+\t\t--run='\"'1-5 !1-3 2'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-inc-neg-inc <<-\\\\EOF\n+\t> ok 1 # skip passing test #1 (--run)\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run include, exclude and include, comma separated' \"\n+\trun_sub_test_lib_test run-inc-neg-inc-comma \\\n+\t\t'--run include, exclude and include, comma separated' \\\n+\t\t--run=1-5,\\!1-3,2 <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-inc-neg-inc-comma <<-\\\\EOF\n+\t> ok 1 # skip passing test #1 (--run)\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run exclude and include' \"\n+\trun_sub_test_lib_test run-neg-inc \\\n+\t\t'--run exclude and include' \\\n+\t\t--run='\"'!3- 5'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-neg-inc <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run empty selectors' \"\n+\trun_sub_test_lib_test run-empty-sel \\\n+\t\t'--run empty selectors' \\\n+\t\t--run='1,,3,,,5' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-empty-sel <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 - passing test #3\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run invalid range start' \"\n+\trun_sub_test_lib_test_err run-inv-range-start \\\n+\t\t'--run invalid range start' \\\n+\t\t--run='a-5' <<-\\\\EOF &&\n+\ttest_expect_success \\\"passing test #1\\\" 'true'\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test_err run-inv-range-start \\\n+\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n+\t> FATAL: Unexpected exit with code 1\n+\tEOF_OUT\n+\t> error: --run: range start should contain only digits: 'a-5'\n+\tEOF_ERR\n+\"\n+\n+test_expect_success '--run invalid range end' \"\n+\trun_sub_test_lib_test_err run-inv-range-end \\\n+\t\t'--run invalid range end' \\\n+\t\t--run='1-z' <<-\\\\EOF &&\n+\ttest_expect_success \\\"passing test #1\\\" 'true'\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test_err run-inv-range-end \\\n+\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n+\t> FATAL: Unexpected exit with code 1\n+\tEOF_OUT\n+\t> error: --run: range end should contain only digits: '1-z'\n+\tEOF_ERR\n+\"\n+\n+test_expect_success '--run invalid selector' \"\n+\trun_sub_test_lib_test_err run-inv-selector \\\n+\t\t'--run invalid selector' \\\n+\t\t--run='1?' <<-\\\\EOF &&\n+\ttest_expect_success \\\"passing test #1\\\" 'true'\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test_err run-inv-selector \\\n+\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n+\t> FATAL: Unexpected exit with code 1\n+\tEOF_OUT\n+\t> error: --run: test selector should contain only digits: '1?'\n+\tEOF_ERR\n+\"\n+\n+\n test_set_prereq HAVEIT\n haveit=no\n test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e7d9c51..46ba513 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -191,6 +191,14 @@ do\n \t\timmediate=t; shift ;;\n \t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n \t\tGIT_TEST_LONG=t; export GIT_TEST_LONG; shift ;;\n+\t-r)\n+\t\tshift; test \"$#\" -ne 0 || {\n+\t\t\techo 'error: -r requires an argument' >&2;\n+\t\t\texit 1;\n+\t\t}\n+\t\trun_list=$1; shift ;;\n+\t--run=*)\n+\t\trun_list=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)'); shift ;;\n \t-h|--h|--he|--hel|--help)\n \t\thelp=t; shift ;;\n \t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n@@ -366,6 +374,100 @@ match_pattern_list () {\n \treturn 1\n }\n \n+match_test_selector_list () {\n+\ttitle=\"$1\"\n+\tshift\n+\targ=\"$1\"\n+\tshift\n+\ttest -z \"$1\" && return 0\n+\n+\t# Both commas and spaces are accepted as separators\n+\tOLDIFS=$IFS\n+\tIFS=' \t,'\n+\tset -- $1\n+\tIFS=$OLDIFS\n+\n+\t# If the first selector is negative we include by default.\n+\tinclude=\n+\tcase \"$1\" in\n+\t\t!*) include=t ;;\n+\tesac\n+\n+\tfor selector\n+\tdo\n+\t\torig_selector=$selector\n+\n+\n+\t\tpositive=t\n+\t\tcase \"$selector\" in\n+\t\t\t!*)\n+\t\t\t\tpositive=\n+\t\t\t\tselector=${selector##?}\n+\t\t\t\t;;\n+\t\tesac\n+\n+\t\ttest -z \"$selector\" && continue\n+\n+\t\tcase \"$selector\" in\n+\t\t\t*-*)\n+\t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n+\t\t\t\tthen\n+\t\t\t\t\techo \"error: $title: range start should contain only\" \\\n+\t\t\t\t\t\t\"digits: '$orig_selector'\" >&2\n+\t\t\t\t\texit 1\n+\t\t\t\tfi\n+\t\t\t\tif expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n+\t\t\t\tthen\n+\t\t\t\t\techo \"error: $title: range end should contain only\" \\\n+\t\t\t\t\t\t\"digits: '$orig_selector'\" >&2\n+\t\t\t\t\texit 1\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\t\t*)\n+\t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n+\t\t\t\tthen\n+\t\t\t\t\techo \"error: $title: test selector should contain\" \\\n+\t\t\t\t\t\t\"only digits: '$orig_selector'\" >&2\n+\t\t\t\t\texit 1\n+\t\t\t\tfi\n+\t\tesac\n+\n+\t\t# Short cut for \"obvious\" cases\n+\t\ttest -z \"$include\" && test -z \"$positive\" && continue\n+\t\ttest -n \"$include\" && test -n \"$positive\" && continue\n+\n+\t\tcase \"$selector\" in\n+\t\t\t-*)\n+\t\t\t\tif test $arg -le ${selector#-}\n+\t\t\t\tthen\n+\t\t\t\t\tinclude=$positive\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\t\t*-)\n+\t\t\t\tif test $arg -ge ${selector%-}\n+\t\t\t\tthen\n+\t\t\t\t\tinclude=$positive\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\t\t*-*)\n+\t\t\t\tif test ${selector%%-*} -le $arg \\\n+\t\t\t\t\t-a $arg -le ${selector#*-}\n+\t\t\t\tthen\n+\t\t\t\t\tinclude=$positive\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\t\t*)\n+\t\t\t\tif test $arg -eq $selector\n+\t\t\t\tthen\n+\t\t\t\t\tinclude=$positive\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\tesac\n+\tdone\n+\n+\ttest -n \"$include\"\n+}\n+\n maybe_teardown_verbose () {\n \ttest -z \"$verbose_only\" && return\n \texec 4>/dev/null 3>/dev/null\n@@ -470,6 +572,13 @@ test_skip () {\n \t\tfi\n \t\tskipped_reason=\"missing $missing_prereq${of_prereq}\"\n \tfi\n+\tif test -z \"$to_skip\" && test -n \"$run_list\" &&\n+\t\t! match_test_selector_list '--run' $test_count \"$run_list\"\n+\tthen\n+\t\tto_skip=t\n+\t\tskipped_reason=\"--run\"\n+\tfi\n+\n \tcase \"$to_skip\" in\n \tt)\n \t\tsay_color skip >&3 \"skipping test: $@\"\n-- \n1.7.9\n"},{"id":"239461","messageId":"xmqqr44nzyx6.fsf@gitster.dls.corp.google.com","threadId":"36472","inReplyTo":"1398154767-1276-2-git-send-email-ilya.bobyr@gmail.com","subject":"Re: [PATCH 1/3] test-lib: Document short options in t/README","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-23T18:24:53Z","receivedAt":"2014-04-23T18:24:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ilya Bobyr <ilya.bobyr@gmail.com> writes:\n\n> Most arguments that could be provided to a test have short forms.\n> Unless documented, the only way to learn them is to read the code.\n>\n> Signed-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n> ---\n>  t/README |    8 ++++----\n>  1 files changed, 4 insertions(+), 4 deletions(-)\n>\n> diff --git a/t/README b/t/README\n> index caeeb9d..6b93aca 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -71,7 +71,7 @@ You can pass --verbose (or -v), --debug (or -d), and --immediate\n>  (or -i) command line argument to the test, or by setting GIT_TEST_OPTS\n>  appropriately before running \"make\".\n>  \n> ---verbose::\n> +-v,--verbose::\n>  \tThis makes the test more verbose.  Specifically, the\n>  \tcommand being run and their output if any are also\n>  \toutput.\n\nI was debating myself if the result should look more like this:\n\n\t-v::\n\t--verbose::\n\t\tThis makes the test more verbose.  Specifically, the\n\t\tcommand being run and their output if any are also\n\t\toutput.\n\nAs a straight text file, your version is certainly a lot easier to\nread, but at the same time, the entire file is written in more or\nless AsciiDoc format (the list of prerequisites and the list of\nharness library functions need to be converted to the \"item::\" form\nfor the text to format well, though) and I've seen some efforts by\nothers to run text files in Documentation/ that were originally\nmeant to be consumed as straight text thru AsciiDoc, so the latter\nform might be a small step for futureproofing.\n\nMy conclusion at this point is that the original is good for the\ncurrent need of the project; if somebody wants to include this file\nfrom somewhere in Documentation/technical, a conversion to use\nmultiple \"item1::<newline>item2::<newline>description\" headers can\nbe done by that person as part of the \"make it fully AsciiDoc\"\neffort.\n\nThanks.\n"},{"id":"239462","messageId":"xmqqlhuvzy6r.fsf@gitster.dls.corp.google.com","threadId":"36472","inReplyTo":"1398154767-1276-4-git-send-email-ilya.bobyr@gmail.com","subject":"Re: [PATCH 3/3] test-lib: '--run' to run only specific tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-23T18:40:44Z","receivedAt":"2014-04-23T18:40:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ilya Bobyr <ilya.bobyr@gmail.com> writes:\n\n> @@ -187,10 +192,70 @@ and either can match the \"t[0-9]{4}\" part to skip the whole\n>  test, or t[0-9]{4} followed by \".$number\" to say which\n>  particular test to skip.\n>  \n> -Note that some tests in the existing test suite rely on previous\n> -test item, so you cannot arbitrarily disable one and expect the\n> -remainder of test to check what the test originally was intended\n> -to check.\n> +For an individual test suite --run could be used to specify that\n> +only some tests should be run or that some tests should be\n> +excluded from a run.\n> +\n> +The argument for --run is a list of individual test numbers or\n> +ranges with an optional negation prefix that define what tests in\n> +a test suite to include in the run.  A range is two numbers\n> +separated with a dash and matches a range of tests with both ends\n> +been included.  You may omit the first or the second number to\n> +mean \"from the first test\" or \"up to the very last test\"\n> +respectively.\n> +\n> +Optional prefix of '!' means that the test or a range of tests\n> +should be excluded from the run.\n> +\n> +If --run starts with an unprefixed number or range the initial\n> +set of tests to run is empty. If the first item starts with '!'\n> +all the tests are added to the initial set.  After initial set is\n> +determined every test number or range is added or excluded from\n> +the set one by one, from left to right.\n> +\n> +Individual numbers or ranges could be separated either by a space\n> +or a comma.\n> +\n> +For example, common case is to run several setup tests (1, 2, 3)\n> +and then a specific test (21) that relies on that setup:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'\n> +\n> +or:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run=1,2,3,21\n> +\n> +or:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'\n\nGood and easily understandable examples. \n\n> +To run only tests up to a specific test (21), one could do this:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1-21'\n> +\n> +or this:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='-21'\n\nThese may be redundant, given that the reader would have to have\ngrokked the earlier \"-3 21\" already at this point.\n\n> +As noted above, the test set is built going though items left to\n> +right, so this:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n> +\n> +will run tests 1, 2, and 4.\n\nI do not quite understand what you mean by \"left to right\"; is that\nimplementation detail necessary for the user of the feature, or is\nit talking about some limitation coming from the implementation?\ne.g. perhaps \"!3 1-4\" would not work as people would expect \"do not\nrun 3, but run tests from 1 thru 4 otherwise\", and warning against\nhaving such an expectation that cannot be fulfilled?\n\n> +You may use negation with ranges.  The following will run all\n> +test as a test suite except from 7 upto 11:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n\nHmm, that is somewhat counter-intuitive or at least ambiguous.  I\nfirst thought you would be running everything but skipping 7 thru\n11, but your explanation is that it is equivalent to \"-6,8-11\" (that\nis, to intersect set \"-11\" and set \"!7\").\n\nThe above two illustrate the reason rather well why I said it would\nbe better to avoid negation because it would complicate the mental\nmodel the user needs to form when using the feature.\n\n> +Some tests in a test suite rely on the previous tests performing\n> +certain actions, specifically some tests are designated as\n> +\"setup\" test, so you cannot _arbitrarily_ disable one test and\n> +expect the rest to function correctly.\n\nWhat this text (moved from the top of this hunk) tells the reader\napplies to both the traditional t0123.4 and the new \"--run=1-3,5-\"\nsyntaxes, but the new placement of it make it sound as if it is only\nfor skipping with \"--run\", especially because the text before this\nparagraph and also after this paragraph both apply only to \"--run\".\n\n> +--run is mostly useful when you want to focus on a specific test\n> +and know what you are doing.  Or when you want to run up to a\n> +certain test.\n\nLikewise for \"and know what you are doing\" part.  I'd suggest\ndropping that phrase from here, and/or make it part of the \"you\ncannot randomly omit and expect later ones to work\" that covers both\nways to skip tests.\n\nThanks.\n"},{"id":"239470","messageId":"CAPig+cTK8i6rGYmum0m-wCmTUE4oqMVEdryiAt29vJXYdgSuYQ@mail.gmail.com","threadId":"36472","inReplyTo":"1398154767-1276-4-git-send-email-ilya.bobyr@gmail.com","subject":"Re: [PATCH 3/3] test-lib: '--run' to run only specific tests","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2014-04-23T19:51:28Z","receivedAt":"2014-04-23T19:51:28Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Apr 22, 2014 at 4:19 AM, Ilya Bobyr <ilya.bobyr@gmail.com> wrote:\n> Allow better control of the set of tests that will be executed for a\n> single test suite.  Mostly useful while debugging or developing as it\n> allows to focus on a specific test.\n>\n> Signed-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n> ---\n> diff --git a/t/README b/t/README\n> index 6b93aca..2dac619 100644\n> --- a/t/README\n> +++ b/t/README\n> +As noted above, the test set is built going though items left to\n> +right, so this:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n> +\n> +will run tests 1, 2, and 4.\n> +\n> +You may use negation with ranges.  The following will run all\n> +test as a test suite except from 7 upto 11:\n\ns/upto/up to/\n...or...\ns/upto/through/\n\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n> +\n> +Some tests in a test suite rely on the previous tests performing\n> +certain actions, specifically some tests are designated as\n> +\"setup\" test, so you cannot _arbitrarily_ disable one test and\n> +expect the rest to function correctly.\n> +--run is mostly useful when you want to focus on a specific test\n> +and know what you are doing.  Or when you want to run up to a\n> +certain test.\n>\n>\n>  Naming Tests\n> diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\n> index ae8874e..e2589cc 100755\n> --- a/t/t0000-basic.sh\n> +++ b/t/t0000-basic.sh\n> @@ -84,6 +97,18 @@ check_sub_test_lib_test () {\n>         )\n>  }\n>\n> +check_sub_test_lib_test_err () {\n> +       name=\"$1\" # stdin is the expected output output from the test\n> +       # expecte error output is in descriptor 3\n\ns/expecte/expected/\n\n> +       (\n> +               cd \"$name\" &&\n> +               sed -e 's/^> //' -e 's/Z$//' >expect.out &&\n> +               test_cmp expect.out out &&\n> +               sed -e 's/^> //' -e 's/Z$//' <&3 >expect.err &&\n> +               test_cmp expect.err err\n> +       )\n> +}\n> +\n>  test_expect_success 'pretend we have a fully passing test suite' \"\n>         run_sub_test_lib_test full-pass '3 passing tests' <<-\\\\EOF &&\n>         for i in 1 2 3\n> @@ -333,6 +358,329 @@ test_expect_success 'GIT_SKIP_TESTS sh pattern' \"\n> +test_expect_success '--run invalid range start' \"\n> +       run_sub_test_lib_test_err run-inv-range-start \\\n> +               '--run invalid range start' \\\n> +               --run='a-5' <<-\\\\EOF &&\n> +       test_expect_success \\\"passing test #1\\\" 'true'\n> +       test_done\n> +       EOF\n> +       check_sub_test_lib_test_err run-inv-range-start \\\n> +               <<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n> +       > FATAL: Unexpected exit with code 1\n> +       EOF_OUT\n> +       > error: --run: range start should contain only digits: 'a-5'\n\nThis reads rather strangely, as if it's attempting to give an example\n(after the colon) of a valid digit range, but then shows something\nthat is not valid. Rewording it slightly can eliminate the ambiguity:\n\n    error: --run: invalid non-numeric range start: 'a-5'\n\n> +       EOF_ERR\n> +\"\n> +\n> +test_expect_success '--run invalid range end' \"\n> +       run_sub_test_lib_test_err run-inv-range-end \\\n> +               '--run invalid range end' \\\n> +               --run='1-z' <<-\\\\EOF &&\n> +       test_expect_success \\\"passing test #1\\\" 'true'\n> +       test_done\n> +       EOF\n> +       check_sub_test_lib_test_err run-inv-range-end \\\n> +               <<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n> +       > FATAL: Unexpected exit with code 1\n> +       EOF_OUT\n> +       > error: --run: range end should contain only digits: '1-z'\n\nDitto.\n\n> +       EOF_ERR\n> +\"\n> +\n> +test_expect_success '--run invalid selector' \"\n> +       run_sub_test_lib_test_err run-inv-selector \\\n> +               '--run invalid selector' \\\n> +               --run='1?' <<-\\\\EOF &&\n> +       test_expect_success \\\"passing test #1\\\" 'true'\n> +       test_done\n> +       EOF\n> +       check_sub_test_lib_test_err run-inv-selector \\\n> +               <<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n> +       > FATAL: Unexpected exit with code 1\n> +       EOF_OUT\n> +       > error: --run: test selector should contain only digits: '1?'\n\nAnd here:\n\n    error: --run: invalid non-digit in range selector: '1?'\n\nor something.\n\n> +       EOF_ERR\n> +\"\n> +\n> +\n>  test_set_prereq HAVEIT\n>  haveit=no\n>  test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index e7d9c51..46ba513 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -366,6 +374,100 @@ match_pattern_list () {\n>         return 1\n>  }\n>\n> +match_test_selector_list () {\n> +       title=\"$1\"\n> +       shift\n> +       arg=\"$1\"\n> +       shift\n> +       test -z \"$1\" && return 0\n> +\n> +       # Both commas and spaces are accepted as separators\n> +       OLDIFS=$IFS\n> +       IFS='   ,'\n\nThe comment mentions only space and comma, but the actual assigned IFS\nvalue also treats tabs as separators. Perhaps update the comment to\nsay \"commas and whitespace\".\n\n> +       set -- $1\n> +       IFS=$OLDIFS\n> +\n> +       # If the first selector is negative we include by default.\n> +       include=\n> +       case \"$1\" in\n> +               !*) include=t ;;\n> +       esac\n> +\n> +       for selector\n> +       do\n> +               orig_selector=$selector\n> +\n> +\n\nUnnecessary extra blank line.\n\n> +               positive=t\n> +               case \"$selector\" in\n> +                       !*)\n> +                               positive=\n> +                               selector=${selector##?}\n> +                               ;;\n> +               esac\n> +\n> +               test -z \"$selector\" && continue\n> +\n> +               case \"$selector\" in\n> +                       *-*)\n> +                               if expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n> +                               then\n> +                                       echo \"error: $title: range start should contain only\" \\\n> +                                               \"digits: '$orig_selector'\" >&2\n> +                                       exit 1\n> +                               fi\n> +                               if expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n> +                               then\n> +                                       echo \"error: $title: range end should contain only\" \\\n> +                                               \"digits: '$orig_selector'\" >&2\n> +                                       exit 1\n> +                               fi\n\nWeird ranges like \"1-4-6\" and \"1-!5\" will be caught by the \"error:\nrange end\" clause. Okay.\n\n> +                               ;;\n> +                       *)\n> +                               if expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n> +                               then\n> +                                       echo \"error: $title: test selector should contain\" \\\n> +                                               \"only digits: '$orig_selector'\" >&2\n> +                                       exit 1\n> +                               fi\n> +               esac\n> +\n> +               # Short cut for \"obvious\" cases\n> +               test -z \"$include\" && test -z \"$positive\" && continue\n> +               test -n \"$include\" && test -n \"$positive\" && continue\n> +\n> +               case \"$selector\" in\n> +                       -*)\n> +                               if test $arg -le ${selector#-}\n> +                               then\n> +                                       include=$positive\n> +                               fi\n> +                               ;;\n> +                       *-)\n> +                               if test $arg -ge ${selector%-}\n> +                               then\n> +                                       include=$positive\n> +                               fi\n> +                               ;;\n> +                       *-*)\n> +                               if test ${selector%%-*} -le $arg \\\n> +                                       -a $arg -le ${selector#*-}\n\nThe -a option to 'test' is not portable [1] and is considered obsolete\nby POSIX [2]. Use \"test foo && test bar\" instead.\n\n[1]: http://www.gnu.org/software/autoconf/manual/autoconf.html#index-g_t_0040command_007btest_007d-1793\n[2]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html\n\n> +                               then\n> +                                       include=$positive\n> +                               fi\n> +                               ;;\n> +                       *)\n> +                               if test $arg -eq $selector\n> +                               then\n> +                                       include=$positive\n> +                               fi\n> +                               ;;\n> +               esac\n> +       done\n> +\n> +       test -n \"$include\"\n> +}\n> +\n>  maybe_teardown_verbose () {\n>         test -z \"$verbose_only\" && return\n>         exec 4>/dev/null 3>/dev/null\n> @@ -470,6 +572,13 @@ test_skip () {\n>                 fi\n>                 skipped_reason=\"missing $missing_prereq${of_prereq}\"\n>         fi\n> +       if test -z \"$to_skip\" && test -n \"$run_list\" &&\n> +               ! match_test_selector_list '--run' $test_count \"$run_list\"\n> +       then\n> +               to_skip=t\n> +               skipped_reason=\"--run\"\n> +       fi\n> +\n>         case \"$to_skip\" in\n>         t)\n>                 say_color skip >&3 \"skipping test: $@\"\n> --\n> 1.7.9\n>\n"},{"id":"240280","messageId":"5360C4B1.10701@gmail.com","threadId":"36472","inReplyTo":"xmqqr44nzyx6.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 1/3] test-lib: Document short options in t/README","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-30T09:38:57Z","receivedAt":"2014-04-30T09:38:57Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"On 4/23/2014 11:24 AM, Junio C Hamano wrote:\n> Ilya Bobyr <ilya.bobyr@gmail.com> writes:\n>> Most arguments that could be provided to a test have short forms.\n>> Unless documented, the only way to learn them is to read the code.\n>>\n>> Signed-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n>> ---\n>>  t/README |    8 ++++----\n>>  1 files changed, 4 insertions(+), 4 deletions(-)\n>>\n>> diff --git a/t/README b/t/README\n>> index caeeb9d..6b93aca 100644\n>> --- a/t/README\n>> +++ b/t/README\n>> @@ -71,7 +71,7 @@ You can pass --verbose (or -v), --debug (or -d), and --immediate\n>>  (or -i) command line argument to the test, or by setting GIT_TEST_OPTS\n>>  appropriately before running \"make\".\n>>  \n>> ---verbose::\n>> +-v,--verbose::\n>>  \tThis makes the test more verbose.  Specifically, the\n>>  \tcommand being run and their output if any are also\n>>  \toutput.\n> I was debating myself if the result should look more like this:\n>\n> \t-v::\n> \t--verbose::\n> \t\tThis makes the test more verbose.  Specifically, the\n> \t\tcommand being run and their output if any are also\n> \t\toutput.\n>\n> As a straight text file, your version is certainly a lot easier to\n> read, but at the same time, the entire file is written in more or\n> less AsciiDoc format (the list of prerequisites and the list of\n> harness library functions need to be converted to the \"item::\" form\n> for the text to format well, though) and I've seen some efforts by\n> others to run text files in Documentation/ that were originally\n> meant to be consumed as straight text thru AsciiDoc, so the latter\n> form might be a small step for futureproofing.\n>\n> My conclusion at this point is that the original is good for the\n> current need of the project; if somebody wants to include this file\n> from somewhere in Documentation/technical, a conversion to use\n> multiple \"item1::<newline>item2::<newline>description\" headers can\n> be done by that person as part of the \"make it fully AsciiDoc\"\n> effort.\n>\n> Thanks.\n\nI've changed it.\nIt is a trivial change and it does not seem to be that bad in plain text\nform either.\n\nI do not know the AsciiDoc conventions as  have not read its spec.  If\nthere are any other conventions I am breaking - let me know.\nI will read the spec if I will be contributing more to the documentation.\n\nP.S.  Sorry it takes me this long to reply %)\n"},{"id":"240281","messageId":"5360C50F.7070505@gmail.com","threadId":"36472","inReplyTo":"xmqqlhuvzy6r.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 3/3] test-lib: '--run' to run only specific tests","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-30T09:40:31Z","receivedAt":"2014-04-30T09:40:31Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"On 4/23/2014 11:40 AM, Junio C Hamano wrote:\n> Ilya Bobyr <ilya.bobyr@gmail.com> writes:\n>\n>> @@ -187,10 +192,70 @@ and either can match the \"t[0-9]{4}\" part to skip the whole\n>>  test, or t[0-9]{4} followed by \".$number\" to say which\n>>  particular test to skip.\n>>  \n>> -Note that some tests in the existing test suite rely on previous\n>> -test item, so you cannot arbitrarily disable one and expect the\n>> -remainder of test to check what the test originally was intended\n>> -to check.\n>> +For an individual test suite --run could be used to specify that\n>> +only some tests should be run or that some tests should be\n>> +excluded from a run.\n>> +\n>> +The argument for --run is a list of individual test numbers or\n>> +ranges with an optional negation prefix that define what tests in\n>> +a test suite to include in the run.  A range is two numbers\n>> +separated with a dash and matches a range of tests with both ends\n>> +been included.  You may omit the first or the second number to\n>> +mean \"from the first test\" or \"up to the very last test\"\n>> +respectively.\n>> +\n>> +Optional prefix of '!' means that the test or a range of tests\n>> +should be excluded from the run.\n>> +\n>> +If --run starts with an unprefixed number or range the initial\n>> +set of tests to run is empty. If the first item starts with '!'\n>> +all the tests are added to the initial set.  After initial set is\n>> +determined every test number or range is added or excluded from\n>> +the set one by one, from left to right.\n>> +\n>> +Individual numbers or ranges could be separated either by a space\n>> +or a comma.\n>> +\n>> +For example, common case is to run several setup tests (1, 2, 3)\n>> +and then a specific test (21) that relies on that setup:\n>> +\n>> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'\n>> +\n>> +or:\n>> +\n>> +    $ sh ./t9200-git-cvsexport-commit.sh --run=1,2,3,21\n>> +\n>> +or:\n>> +\n>> +    $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'\n> Good and easily understandable examples. \n>\n>> +To run only tests up to a specific test (21), one could do this:\n>> +\n>> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1-21'\n>> +\n>> +or this:\n>> +\n>> +    $ sh ./t9200-git-cvsexport-commit.sh --run='-21'\n> These may be redundant, given that the reader would have to have\n> grokked the earlier \"-3 21\" already at this point.\n\nThe original idea was to show two most common use cases in the examples,\nso that one could just copy/paste it.\nI guess you are right that the second is a bit redundant now from the\nstandpoint of a person who is reading all of it.\n\nI have reordered the examples.  Single range is simpler, it comes first\nand then a more complicated example.\n\n>> +As noted above, the test set is built going though items left to\n>> +right, so this:\n>> +\n>> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n>> +\n>> +will run tests 1, 2, and 4.\n> I do not quite understand what you mean by \"left to right\"; is that\n> implementation detail necessary for the user of the feature, or is\n> it talking about some limitation coming from the implementation?\n> e.g. perhaps \"!3 1-4\" would not work as people would expect \"do not\n> run 3, but run tests from 1 thru 4 otherwise\", and warning against\n> having such an expectation that cannot be fulfilled?\n\nI thought that it is something that you may want to understand if you\nare going to build something complicated.  As I do not have a specific\nuse case, this is kind of a made up example.\nThe idea is that what is on the right overwrites what is on the left. \nI've added that sentence as an additional clarification, and your example.\n\n>> +You may use negation with ranges.  The following will run all\n>> +test as a test suite except from 7 upto 11:\n>> +\n>> +    $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n> Hmm, that is somewhat counter-intuitive or at least ambiguous.  I\n> first thought you would be running everything but skipping 7 thru\n> 11, but your explanation is that it is equivalent to \"-6,8-11\" (that\n> is, to intersect set \"-11\" and set \"!7\").\n\nYour expectation is correct.\nA space or a comma is needed in order for \"!7\" and \"-11\" to be treated\nseparately.\nI am not sure why did you read the description as \"-6,8-11\".  There is a\ntypo in the sentence: s/as a/in the/.\nI've changed that, but I would not object a better explanation of cause :)\n\n> The above two illustrate the reason rather well why I said it would\n> be better to avoid negation because it would complicate the mental\n> model the user needs to form when using the feature.\n\nI think that you do not have to use it if you do not need it.\nIt adds some expressiveness, is rather easy to implement and is already\nthere :)\nI can remove it, of cause, but is it really necessary?\n\n>> +Some tests in a test suite rely on the previous tests performing\n>> +certain actions, specifically some tests are designated as\n>> +\"setup\" test, so you cannot _arbitrarily_ disable one test and\n>> +expect the rest to function correctly.\n> What this text (moved from the top of this hunk) tells the reader\n> applies to both the traditional t0123.4 and the new \"--run=1-3,5-\"\n> syntaxes, but the new placement of it make it sound as if it is only\n> for skipping with \"--run\", especially because the text before this\n> paragraph and also after this paragraph both apply only to \"--run\".\n\nTrue, but there is another paragraph at the beginning of the section\nthat talks why would you want to use GIT_SKIP_TESTS:\n\n> In some environments, certain tests have no way of succeeding\n> due to platform limitation, such as lack of 'unzip' program, or\n> filesystem that do not allow arbitrary sequence of non-NUL bytes\n> as pathnames.\n\nI was thinking that if you would be working with individual test suits\nyou would use '--run'.\nAnd this is where you more likely to think about setup tests.\nI could move that paragraph just after the GIT_SKIP_TESTS description. \nThen it would apply more to both.\nI am not sure it is needed.  Let me know if you think otherwise.\n\n>> +--run is mostly useful when you want to focus on a specific test\n>> +and know what you are doing.  Or when you want to run up to a\n>> +certain test.\n> Likewise for \"and know what you are doing\" part.  I'd suggest\n> dropping that phrase from here, and/or make it part of the \"you\n> cannot randomly omit and expect later ones to work\" that covers both\n> ways to skip tests.\n\nI've made this part a bit less wage.\n\nThank you for reviewing it :)\n"},{"id":"240282","messageId":"5360C53F.5080909@gmail.com","threadId":"36472","inReplyTo":"CAPig+cTK8i6rGYmum0m-wCmTUE4oqMVEdryiAt29vJXYdgSuYQ@mail.gmail.com","subject":"Re: [PATCH 3/3] test-lib: '--run' to run only specific tests","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-30T09:41:19Z","receivedAt":"2014-04-30T09:41:19Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"On 4/23/2014 12:51 PM, Eric Sunshine wrote:\n> On Tue, Apr 22, 2014 at 4:19 AM, Ilya Bobyr <ilya.bobyr@gmail.com> wrote:\n>> Allow better control of the set of tests that will be executed for a\n>> single test suite.  Mostly useful while debugging or developing as it\n>> allows to focus on a specific test.\n>>\n>> Signed-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n>> ---\n>> diff --git a/t/README b/t/README\n>> index 6b93aca..2dac619 100644\n>> --- a/t/README\n>> +++ b/t/README\n>> +As noted above, the test set is built going though items left to\n>> +right, so this:\n>> +\n>> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n>> +\n>> +will run tests 1, 2, and 4.\n>> +\n>> +You may use negation with ranges.  The following will run all\n>> +test as a test suite except from 7 upto 11:\n> s/upto/up to/\n> ...or...\n> s/upto/through/\n\nFixed.  Thanks.\n\n>> +    $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n>> +\n>> +Some tests in a test suite rely on the previous tests performing\n>> +certain actions, specifically some tests are designated as\n>> +\"setup\" test, so you cannot _arbitrarily_ disable one test and\n>> +expect the rest to function correctly.\n>> +--run is mostly useful when you want to focus on a specific test\n>> +and know what you are doing.  Or when you want to run up to a\n>> +certain test.\n>>\n>>\n>>  Naming Tests\n>> diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\n>> index ae8874e..e2589cc 100755\n>> --- a/t/t0000-basic.sh\n>> +++ b/t/t0000-basic.sh\n>> @@ -84,6 +97,18 @@ check_sub_test_lib_test () {\n>>         )\n>>  }\n>>\n>> +check_sub_test_lib_test_err () {\n>> +       name=\"$1\" # stdin is the expected output output from the test\n>> +       # expecte error output is in descriptor 3\n> s/expecte/expected/\n\nFixed.\n\n>> +       (\n>> +               cd \"$name\" &&\n>> +               sed -e 's/^> //' -e 's/Z$//' >expect.out &&\n>> +               test_cmp expect.out out &&\n>> +               sed -e 's/^> //' -e 's/Z$//' <&3 >expect.err &&\n>> +               test_cmp expect.err err\n>> +       )\n>> +}\n>> +\n>>  test_expect_success 'pretend we have a fully passing test suite' \"\n>>         run_sub_test_lib_test full-pass '3 passing tests' <<-\\\\EOF &&\n>>         for i in 1 2 3\n>> @@ -333,6 +358,329 @@ test_expect_success 'GIT_SKIP_TESTS sh pattern' \"\n>> +test_expect_success '--run invalid range start' \"\n>> +       run_sub_test_lib_test_err run-inv-range-start \\\n>> +               '--run invalid range start' \\\n>> +               --run='a-5' <<-\\\\EOF &&\n>> +       test_expect_success \\\"passing test #1\\\" 'true'\n>> +       test_done\n>> +       EOF\n>> +       check_sub_test_lib_test_err run-inv-range-start \\\n>> +               <<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n>> +       > FATAL: Unexpected exit with code 1\n>> +       EOF_OUT\n>> +       > error: --run: range start should contain only digits: 'a-5'\n> This reads rather strangely, as if it's attempting to give an example\n> (after the colon) of a valid digit range, but then shows something\n> that is not valid. Rewording it slightly can eliminate the ambiguity:\n>\n>     error: --run: invalid non-numeric range start: 'a-5'\n\nChanged.\n\n>> +       EOF_ERR\n>> +\"\n>> +\n>> +test_expect_success '--run invalid range end' \"\n>> +       run_sub_test_lib_test_err run-inv-range-end \\\n>> +               '--run invalid range end' \\\n>> +               --run='1-z' <<-\\\\EOF &&\n>> +       test_expect_success \\\"passing test #1\\\" 'true'\n>> +       test_done\n>> +       EOF\n>> +       check_sub_test_lib_test_err run-inv-range-end \\\n>> +               <<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n>> +       > FATAL: Unexpected exit with code 1\n>> +       EOF_OUT\n>> +       > error: --run: range end should contain only digits: '1-z'\n> Ditto.\n\nFixed.\n\n>> +       EOF_ERR\n>> +\"\n>> +\n>> +test_expect_success '--run invalid selector' \"\n>> +       run_sub_test_lib_test_err run-inv-selector \\\n>> +               '--run invalid selector' \\\n>> +               --run='1?' <<-\\\\EOF &&\n>> +       test_expect_success \\\"passing test #1\\\" 'true'\n>> +       test_done\n>> +       EOF\n>> +       check_sub_test_lib_test_err run-inv-selector \\\n>> +               <<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n>> +       > FATAL: Unexpected exit with code 1\n>> +       EOF_OUT\n>> +       > error: --run: test selector should contain only digits: '1?'\n> And here:\n>\n>     error: --run: invalid non-digit in range selector: '1?'\n>\n> or something.\n\nChanged to \"invalid non-digit in test selector\".  This one is only shown\nif it does not have a \"-\", so it is probably not a range.\n\n>> +       EOF_ERR\n>> +\"\n>> +\n>> +\n>>  test_set_prereq HAVEIT\n>>  haveit=no\n>>  test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '\n>> diff --git a/t/test-lib.sh b/t/test-lib.sh\n>> index e7d9c51..46ba513 100644\n>> --- a/t/test-lib.sh\n>> +++ b/t/test-lib.sh\n>> @@ -366,6 +374,100 @@ match_pattern_list () {\n>>         return 1\n>>  }\n>>\n>> +match_test_selector_list () {\n>> +       title=\"$1\"\n>> +       shift\n>> +       arg=\"$1\"\n>> +       shift\n>> +       test -z \"$1\" && return 0\n>> +\n>> +       # Both commas and spaces are accepted as separators\n>> +       OLDIFS=$IFS\n>> +       IFS='   ,'\n> The comment mentions only space and comma, but the actual assigned IFS\n> value also treats tabs as separators. Perhaps update the comment to\n> say \"commas and whitespace\".\n\nI thought that tab is a space character =)  Changed it.\n\n>> +       set -- $1\n>> +       IFS=$OLDIFS\n>> +\n>> +       # If the first selector is negative we include by default.\n>> +       include=\n>> +       case \"$1\" in\n>> +               !*) include=t ;;\n>> +       esac\n>> +\n>> +       for selector\n>> +       do\n>> +               orig_selector=$selector\n>> +\n>> +\n> Unnecessary extra blank line.\n\nThanks.\n\n> [...]\n>> +                               ;;\n>> +                       *)\n>> +                               if expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n>> +                               then\n>> +                                       echo \"error: $title: test selector should contain\" \\\n>> +                                               \"only digits: '$orig_selector'\" >&2\n>> +                                       exit 1\n>> +                               fi\n>> +               esac\n>> +\n>> +               # Short cut for \"obvious\" cases\n>> +               test -z \"$include\" && test -z \"$positive\" && continue\n>> +               test -n \"$include\" && test -n \"$positive\" && continue\n>> +\n>> +               case \"$selector\" in\n>> +                       -*)\n>> +                               if test $arg -le ${selector#-}\n>> +                               then\n>> +                                       include=$positive\n>> +                               fi\n>> +                               ;;\n>> +                       *-)\n>> +                               if test $arg -ge ${selector%-}\n>> +                               then\n>> +                                       include=$positive\n>> +                               fi\n>> +                               ;;\n>> +                       *-*)\n>> +                               if test ${selector%%-*} -le $arg \\\n>> +                                       -a $arg -le ${selector#*-}\n> The -a option to 'test' is not portable [1] and is considered obsolete\n> by POSIX [2]. Use \"test foo && test bar\" instead.\n>\n> [1]: http://www.gnu.org/software/autoconf/manual/autoconf.html#index-g_t_0040command_007btest_007d-1793\n> [2]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html\n\nDid not know that.  Thanks.  Changed it.\n\nIt is used a number of times thought:\n\n$ git grep '\\<test\\>.*-a\\>' | wc -l\n72\n\nAbout 10 matches are accidental, but the rest are '-a' uses in 'test'.\n\n>> [...]\n\nThanks a lot for looking into it :)\n"},{"id":"240285","messageId":"1398851444-2492-1-git-send-email-ilya.bobyr@gmail.com","threadId":"36472","inReplyTo":"1398154767-1276-1-git-send-email-ilya.bobyr@gmail.com","subject":"[RFC/PATCH v4] Better control of the tests run by a test suite","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-30T09:50:41Z","receivedAt":"2014-04-30T09:50:41Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"This patches add `--run` option to the test suites to allow one to run\nindividual tests out of the test suite.  Like this:\n\n    ./t0000-basic.sh --run='-4,7,9-12,15-'\n\nPrevious version:\n\n    [RFC/PATCH v3] Better control of the tests run by a test suite\n    http://www.mail-archive.com/git@vger.kernel.org/msg48304.html\n\nThis version addresses comments by Junio[1] and Eric Sunshine[2].  I think that\nthey are mostly minor changes.  \n\n[1] http://www.mail-archive.com/git@vger.kernel.org/msg48454.html\n    http://www.mail-archive.com/git@vger.kernel.org/msg48455.html\n\n[2] http://www.mail-archive.com/git@vger.kernel.org/msg48463.html\n\nI've replied to those messages. There are two comments from Junio that did not\nresult in changed.  All the others have been addressed I hope.\n\nIlya Bobyr (3):\n  test-lib: Document short options in t/README\n  test-lib: tests skipped by GIT_SKIP_TESTS say so\n  test-lib: '--run' to run only specific tests\n\n t/README         |   85 +++++++++++-\n t/t0000-basic.sh |  419 +++++++++++++++++++++++++++++++++++++++++++++++++++++-\n t/test-lib.sh    |  119 +++++++++++++++-\n 3 files changed, 611 insertions(+), 12 deletions(-)\n"},{"id":"240284","messageId":"1398851444-2492-2-git-send-email-ilya.bobyr@gmail.com","threadId":"36472","inReplyTo":"1398851444-2492-1-git-send-email-ilya.bobyr@gmail.com","subject":"[PATCH 1/3] test-lib: Document short options in t/README","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-30T09:50:42Z","receivedAt":"2014-04-30T09:50:42Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"Most arguments that could be provided to a test have short forms.\nUnless documented, the only way to learn them is to read the code.\n\nSigned-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n---\n\nChanged to use AsciiDoc format.\n\n t/README |    4 ++++\n 1 files changed, 4 insertions(+), 0 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex caeeb9d..eaf6ecd 100644\n--- a/t/README\n+++ b/t/README\n@@ -71,6 +71,7 @@ You can pass --verbose (or -v), --debug (or -d), and --immediate\n (or -i) command line argument to the test, or by setting GIT_TEST_OPTS\n appropriately before running \"make\".\n \n+-v::\n --verbose::\n \tThis makes the test more verbose.  Specifically, the\n \tcommand being run and their output if any are also\n@@ -81,6 +82,7 @@ appropriately before running \"make\".\n \tnumbers matching <pattern>.  The number matched against is\n \tsimply the running count of the test within the file.\n \n+-d::\n --debug::\n \tThis may help the person who is developing a new test.\n \tIt causes the command defined with test_debug to run.\n@@ -89,6 +91,7 @@ appropriately before running \"make\".\n \tfailed tests so that you can inspect its contents after\n \tthe test finished.\n \n+-i::\n --immediate::\n \tThis causes the test to immediately exit upon the first\n \tfailed test. Cleanup commands requested with\n@@ -96,6 +99,7 @@ appropriately before running \"make\".\n \tin order to keep the state for inspection by the tester\n \tto diagnose the bug.\n \n+-l::\n --long-tests::\n \tThis causes additional long-running tests to be run (where\n \tavailable), for more exhaustive testing.\n-- \n1.7.9\n"},{"id":"240286","messageId":"1398851444-2492-3-git-send-email-ilya.bobyr@gmail.com","threadId":"36472","inReplyTo":"1398851444-2492-1-git-send-email-ilya.bobyr@gmail.com","subject":"[PATCH 2/3] test-lib: tests skipped by GIT_SKIP_TESTS say so","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-30T09:50:43Z","receivedAt":"2014-04-30T09:50:43Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"We used to show \"(missing )\" next to tests skipped because they are\nspecified in GIT_SKIP_TESTS.  Use \"(GIT_SKIP_TESTS)\" instead.\n\nPlus tests that check basic GIT_SKIP_TESTS functions.\n\nSigned-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n---\nNo changes.\n\n t/t0000-basic.sh |   63 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n t/test-lib.sh    |   13 ++++++----\n 2 files changed, 71 insertions(+), 5 deletions(-)\n\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex a2bb63c..ae8874e 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -270,6 +270,69 @@ test_expect_success 'test --verbose-only' '\n \tEOF\n '\n \n+test_expect_success 'GIT_SKIP_TESTS' \"\n+\tGIT_SKIP_TESTS='git.2' \\\n+\t\trun_sub_test_lib_test git-skip-tests-basic \\\n+\t\t'GIT_SKIP_TESTS' <<-\\\\EOF &&\n+\tfor i in 1 2 3\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test git-skip-tests-basic <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)\n+\t> ok 3 - passing test #3\n+\t> # passed all 3 test(s)\n+\t> 1..3\n+\tEOF\n+\"\n+\n+test_expect_success 'GIT_SKIP_TESTS several tests' \"\n+\tGIT_SKIP_TESTS='git.2 git.5' \\\n+\t\trun_sub_test_lib_test git-skip-tests-several \\\n+\t\t'GIT_SKIP_TESTS several tests' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test git-skip-tests-several <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)\n+\t> ok 3 - passing test #3\n+\t> ok 4 - passing test #4\n+\t> ok 5 # skip passing test #5 (GIT_SKIP_TESTS)\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success 'GIT_SKIP_TESTS sh pattern' \"\n+\tGIT_SKIP_TESTS='git.[2-5]' \\\n+\t\trun_sub_test_lib_test git-skip-tests-sh-pattern \\\n+\t\t'GIT_SKIP_TESTS sh pattern' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test git-skip-tests-sh-pattern <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (GIT_SKIP_TESTS)\n+\t> ok 3 # skip passing test #3 (GIT_SKIP_TESTS)\n+\t> ok 4 # skip passing test #4 (GIT_SKIP_TESTS)\n+\t> ok 5 # skip passing test #5 (GIT_SKIP_TESTS)\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n test_set_prereq HAVEIT\n haveit=no\n test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex c081668..e7d9c51 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -452,25 +452,28 @@ test_finish_ () {\n \n test_skip () {\n \tto_skip=\n+\tskipped_reason=\n \tif match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS\n \tthen\n \t\tto_skip=t\n+\t\tskipped_reason=\"GIT_SKIP_TESTS\"\n \tfi\n \tif test -z \"$to_skip\" && test -n \"$test_prereq\" &&\n \t   ! test_have_prereq \"$test_prereq\"\n \tthen\n \t\tto_skip=t\n-\tfi\n-\tcase \"$to_skip\" in\n-\tt)\n+\n \t\tof_prereq=\n \t\tif test \"$missing_prereq\" != \"$test_prereq\"\n \t\tthen\n \t\t\tof_prereq=\" of $test_prereq\"\n \t\tfi\n-\n+\t\tskipped_reason=\"missing $missing_prereq${of_prereq}\"\n+\tfi\n+\tcase \"$to_skip\" in\n+\tt)\n \t\tsay_color skip >&3 \"skipping test: $@\"\n-\t\tsay_color skip \"ok $test_count # skip $1 (missing $missing_prereq${of_prereq})\"\n+\t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n \t*)\n-- \n1.7.9\n"},{"id":"240283","messageId":"1398851444-2492-4-git-send-email-ilya.bobyr@gmail.com","threadId":"36472","inReplyTo":"1398851444-2492-1-git-send-email-ilya.bobyr@gmail.com","subject":"[PATCH 3/3] test-lib: '--run' to run only specific tests","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-04-30T09:50:44Z","receivedAt":"2014-04-30T09:50:44Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"Allow better control of the set of tests that will be executed for a\nsingle test suite.  Mostly useful while debugging or developing as it\nallows to focus on a specific test.\n\nSigned-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n---\nA number of minor changes according to the review comments.\n\n t/README         |   81 ++++++++++++-\n t/t0000-basic.sh |  356 +++++++++++++++++++++++++++++++++++++++++++++++++++++-\n t/test-lib.sh    |  108 ++++++++++++++++\n 3 files changed, 537 insertions(+), 8 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex eaf6ecd..cd99d21 100644\n--- a/t/README\n+++ b/t/README\n@@ -104,6 +104,12 @@ appropriately before running \"make\".\n \tThis causes additional long-running tests to be run (where\n \tavailable), for more exhaustive testing.\n \n+-r::\n+--run=<test-selector>::\n+\tRun only the subset of tests indicated by\n+\t<test-selector>.  See section \"Skipping Tests\" below for\n+\t<test-selector> syntax.\n+\n --valgrind=<tool>::\n \tExecute all Git binaries under valgrind tool <tool> and exit\n \twith status 126 on errors (just like regular tests, this will\n@@ -191,10 +197,77 @@ and either can match the \"t[0-9]{4}\" part to skip the whole\n test, or t[0-9]{4} followed by \".$number\" to say which\n particular test to skip.\n \n-Note that some tests in the existing test suite rely on previous\n-test item, so you cannot arbitrarily disable one and expect the\n-remainder of test to check what the test originally was intended\n-to check.\n+For an individual test suite --run could be used to specify that\n+only some tests should be run or that some tests should be\n+excluded from a run.\n+\n+The argument for --run is a list of individual test numbers or\n+ranges with an optional negation prefix that define what tests in\n+a test suite to include in the run.  A range is two numbers\n+separated with a dash and matches a range of tests with both ends\n+been included.  You may omit the first or the second number to\n+mean \"from the first test\" or \"up to the very last test\"\n+respectively.\n+\n+Optional prefix of '!' means that the test or a range of tests\n+should be excluded from the run.\n+\n+If --run starts with an unprefixed number or range the initial\n+set of tests to run is empty. If the first item starts with '!'\n+all the tests are added to the initial set.  After initial set is\n+determined every test number or range is added or excluded from\n+the set one by one, from left to right.\n+\n+Individual numbers or ranges could be separated either by a space\n+or a comma.\n+\n+For example, to run only tests up to a specific test (21), one\n+could do this:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='1-21'\n+\n+or this:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='-21'\n+\n+Common case is to run several setup tests (1, 2, 3) and then a\n+specific test (21) that relies on that setup:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'\n+\n+or:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run=1,2,3,21\n+\n+or:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'\n+\n+As noted above, the test set is built going though items left to\n+right, so this:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n+\n+will run tests 1, 2, and 4.  Items that comes later have higher\n+precendence.  It means that this:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'\n+\n+would just run tests from 1 to 4, including 3.\n+\n+You may use negation with ranges.  The following will run all\n+test in the test suite except from 7 up to 11:\n+\n+    $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n+\n+Some tests in a test suite rely on the previous tests performing\n+certain actions, specifically some tests are designated as\n+\"setup\" test, so you cannot _arbitrarily_ disable one test and\n+expect the rest to function correctly.\n+\n+--run is mostly useful when you want to focus on a specific test\n+and know what setup is needed for it.  Or when you want to run\n+everything up to a certain test.\n \n \n Naming Tests\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex ae8874e..8345c8a 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -42,9 +42,9 @@ test_expect_success 'success is reported like this' '\n \t:\n '\n \n-run_sub_test_lib_test () {\n-\tname=\"$1\" descr=\"$2\" # stdin is the body of the test code\n-\tshift 2\n+_run_sub_test_lib_test_common () {\n+\tneg=\"$1\" name=\"$2\" descr=\"$3\" # stdin is the body of the test code\n+\tshift 3\n \tmkdir \"$name\" &&\n \t(\n \t\t# Pretend we're not running under a test harness, whether we\n@@ -70,10 +70,23 @@ run_sub_test_lib_test () {\n \t\texport TEST_DIRECTORY &&\n \t\tTEST_OUTPUT_DIRECTORY=$(pwd) &&\n \t\texport TEST_OUTPUT_DIRECTORY &&\n-\t\t./\"$name.sh\" \"$@\" >out 2>err\n+\t\tif test -z \"$neg\"\n+\t\tthen\n+\t\t\t./\"$name.sh\" \"$@\" >out 2>err\n+\t\telse\n+\t\t\t!  ./\"$name.sh\" \"$@\" >out 2>err\n+\t\tfi\n \t)\n }\n \n+run_sub_test_lib_test () {\n+\t_run_sub_test_lib_test_common '' \"$@\"\n+}\n+\n+run_sub_test_lib_test_err () {\n+\t_run_sub_test_lib_test_common '!' \"$@\"\n+}\n+\n check_sub_test_lib_test () {\n \tname=\"$1\" # stdin is the expected output from the test\n \t(\n@@ -84,6 +97,18 @@ check_sub_test_lib_test () {\n \t)\n }\n \n+check_sub_test_lib_test_err () {\n+\tname=\"$1\" # stdin is the expected output output from the test\n+\t# expected error output is in descriptior 3\n+\t(\n+\t\tcd \"$name\" &&\n+\t\tsed -e 's/^> //' -e 's/Z$//' >expect.out &&\n+\t\ttest_cmp expect.out out &&\n+\t\tsed -e 's/^> //' -e 's/Z$//' <&3 >expect.err &&\n+\t\ttest_cmp expect.err err\n+\t)\n+}\n+\n test_expect_success 'pretend we have a fully passing test suite' \"\n \trun_sub_test_lib_test full-pass '3 passing tests' <<-\\\\EOF &&\n \tfor i in 1 2 3\n@@ -333,6 +358,329 @@ test_expect_success 'GIT_SKIP_TESTS sh pattern' \"\n \tEOF\n \"\n \n+test_expect_success '--run basic' \"\n+\trun_sub_test_lib_test run-basic \\\n+\t\t'--run basic' --run='1 3 5' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-basic <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 - passing test #3\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with a range' \"\n+\trun_sub_test_lib_test run-range \\\n+\t\t'--run with a range' --run='1-3' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-range <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 - passing test #3\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 # skip passing test #5 (--run)\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with two ranges' \"\n+\trun_sub_test_lib_test run-two-ranges \\\n+\t\t'--run with two ranges' --run='1-2 5-6' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-two-ranges <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 - passing test #5\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with a left open range' \"\n+\trun_sub_test_lib_test run-left-open-range \\\n+\t\t'--run with a left open range' --run='-3' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-left-open-range <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 - passing test #3\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 # skip passing test #5 (--run)\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with a right open range' \"\n+\trun_sub_test_lib_test run-right-open-range \\\n+\t\t'--run with a right open range' --run='4-' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-right-open-range <<-\\\\EOF\n+\t> ok 1 # skip passing test #1 (--run)\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with basic negation' \"\n+\trun_sub_test_lib_test run-basic-neg \\\n+\t\t'--run with basic negation' --run='\"'!3'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-basic-neg <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run with two negations' \"\n+\trun_sub_test_lib_test run-two-neg \\\n+\t\t'--run with two negations' --run='\"'!3 !6'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-two-neg <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run a range and negation' \"\n+\trun_sub_test_lib_test run-range-and-neg \\\n+\t\t'--run a range and negation' --run='\"'-4 !2'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-range-and-neg <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 - passing test #3\n+\t> ok 4 - passing test #4\n+\t> ok 5 # skip passing test #5 (--run)\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run range negation' \"\n+\trun_sub_test_lib_test run-range-neg \\\n+\t\t'--run range negation' --run='\"'!1-3'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-range-neg <<-\\\\EOF\n+\t> ok 1 # skip passing test #1 (--run)\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 - passing test #6\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run include, exclude and include' \"\n+\trun_sub_test_lib_test run-inc-neg-inc \\\n+\t\t'--run include, exclude and include' \\\n+\t\t--run='\"'1-5 !1-3 2'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-inc-neg-inc <<-\\\\EOF\n+\t> ok 1 # skip passing test #1 (--run)\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run include, exclude and include, comma separated' \"\n+\trun_sub_test_lib_test run-inc-neg-inc-comma \\\n+\t\t'--run include, exclude and include, comma separated' \\\n+\t\t--run=1-5,\\!1-3,2 <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-inc-neg-inc-comma <<-\\\\EOF\n+\t> ok 1 # skip passing test #1 (--run)\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 - passing test #4\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run exclude and include' \"\n+\trun_sub_test_lib_test run-neg-inc \\\n+\t\t'--run exclude and include' \\\n+\t\t--run='\"'!3- 5'\"' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-neg-inc <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 - passing test #2\n+\t> ok 3 # skip passing test #3 (--run)\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run empty selectors' \"\n+\trun_sub_test_lib_test run-empty-sel \\\n+\t\t'--run empty selectors' \\\n+\t\t--run='1,,3,,,5' <<-\\\\EOF &&\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-empty-sel <<-\\\\EOF\n+\t> ok 1 - passing test #1\n+\t> ok 2 # skip passing test #2 (--run)\n+\t> ok 3 - passing test #3\n+\t> ok 4 # skip passing test #4 (--run)\n+\t> ok 5 - passing test #5\n+\t> ok 6 # skip passing test #6 (--run)\n+\t> # passed all 6 test(s)\n+\t> 1..6\n+\tEOF\n+\"\n+\n+test_expect_success '--run invalid range start' \"\n+\trun_sub_test_lib_test_err run-inv-range-start \\\n+\t\t'--run invalid range start' \\\n+\t\t--run='a-5' <<-\\\\EOF &&\n+\ttest_expect_success \\\"passing test #1\\\" 'true'\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test_err run-inv-range-start \\\n+\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n+\t> FATAL: Unexpected exit with code 1\n+\tEOF_OUT\n+\t> error: --run: invalid non-numeric in range start: 'a-5'\n+\tEOF_ERR\n+\"\n+\n+test_expect_success '--run invalid range end' \"\n+\trun_sub_test_lib_test_err run-inv-range-end \\\n+\t\t'--run invalid range end' \\\n+\t\t--run='1-z' <<-\\\\EOF &&\n+\ttest_expect_success \\\"passing test #1\\\" 'true'\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test_err run-inv-range-end \\\n+\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n+\t> FATAL: Unexpected exit with code 1\n+\tEOF_OUT\n+\t> error: --run: invalid non-numeric in range end: '1-z'\n+\tEOF_ERR\n+\"\n+\n+test_expect_success '--run invalid selector' \"\n+\trun_sub_test_lib_test_err run-inv-selector \\\n+\t\t'--run invalid selector' \\\n+\t\t--run='1?' <<-\\\\EOF &&\n+\ttest_expect_success \\\"passing test #1\\\" 'true'\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test_err run-inv-selector \\\n+\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n+\t> FATAL: Unexpected exit with code 1\n+\tEOF_OUT\n+\t> error: --run: invalid non-numeric in test selector: '1?'\n+\tEOF_ERR\n+\"\n+\n+\n test_set_prereq HAVEIT\n haveit=no\n test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e7d9c51..91000fe 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -191,6 +191,14 @@ do\n \t\timmediate=t; shift ;;\n \t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n \t\tGIT_TEST_LONG=t; export GIT_TEST_LONG; shift ;;\n+\t-r)\n+\t\tshift; test \"$#\" -ne 0 || {\n+\t\t\techo 'error: -r requires an argument' >&2;\n+\t\t\texit 1;\n+\t\t}\n+\t\trun_list=$1; shift ;;\n+\t--run=*)\n+\t\trun_list=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)'); shift ;;\n \t-h|--h|--he|--hel|--help)\n \t\thelp=t; shift ;;\n \t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n@@ -366,6 +374,99 @@ match_pattern_list () {\n \treturn 1\n }\n \n+match_test_selector_list () {\n+\ttitle=\"$1\"\n+\tshift\n+\targ=\"$1\"\n+\tshift\n+\ttest -z \"$1\" && return 0\n+\n+\t# Both commas and whitespace are accepted as separators.\n+\tOLDIFS=$IFS\n+\tIFS=' \t,'\n+\tset -- $1\n+\tIFS=$OLDIFS\n+\n+\t# If the first selector is negative we include by default.\n+\tinclude=\n+\tcase \"$1\" in\n+\t\t!*) include=t ;;\n+\tesac\n+\n+\tfor selector\n+\tdo\n+\t\torig_selector=$selector\n+\n+\t\tpositive=t\n+\t\tcase \"$selector\" in\n+\t\t\t!*)\n+\t\t\t\tpositive=\n+\t\t\t\tselector=${selector##?}\n+\t\t\t\t;;\n+\t\tesac\n+\n+\t\ttest -z \"$selector\" && continue\n+\n+\t\tcase \"$selector\" in\n+\t\t\t*-*)\n+\t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n+\t\t\t\tthen\n+\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\t\t\"start: '$orig_selector'\" >&2\n+\t\t\t\t\texit 1\n+\t\t\t\tfi\n+\t\t\t\tif expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n+\t\t\t\tthen\n+\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\t\t\"end: '$orig_selector'\" >&2\n+\t\t\t\t\texit 1\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\t\t*)\n+\t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n+\t\t\t\tthen\n+\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n+\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n+\t\t\t\t\texit 1\n+\t\t\t\tfi\n+\t\tesac\n+\n+\t\t# Short cut for \"obvious\" cases\n+\t\ttest -z \"$include\" && test -z \"$positive\" && continue\n+\t\ttest -n \"$include\" && test -n \"$positive\" && continue\n+\n+\t\tcase \"$selector\" in\n+\t\t\t-*)\n+\t\t\t\tif test $arg -le ${selector#-}\n+\t\t\t\tthen\n+\t\t\t\t\tinclude=$positive\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\t\t*-)\n+\t\t\t\tif test $arg -ge ${selector%-}\n+\t\t\t\tthen\n+\t\t\t\t\tinclude=$positive\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\t\t*-*)\n+\t\t\t\tif test ${selector%%-*} -le $arg \\\n+\t\t\t\t\t&& test $arg -le ${selector#*-}\n+\t\t\t\tthen\n+\t\t\t\t\tinclude=$positive\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\t\t*)\n+\t\t\t\tif test $arg -eq $selector\n+\t\t\t\tthen\n+\t\t\t\t\tinclude=$positive\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\tesac\n+\tdone\n+\n+\ttest -n \"$include\"\n+}\n+\n maybe_teardown_verbose () {\n \ttest -z \"$verbose_only\" && return\n \texec 4>/dev/null 3>/dev/null\n@@ -470,6 +571,13 @@ test_skip () {\n \t\tfi\n \t\tskipped_reason=\"missing $missing_prereq${of_prereq}\"\n \tfi\n+\tif test -z \"$to_skip\" && test -n \"$run_list\" &&\n+\t\t! match_test_selector_list '--run' $test_count \"$run_list\"\n+\tthen\n+\t\tto_skip=t\n+\t\tskipped_reason=\"--run\"\n+\tfi\n+\n \tcase \"$to_skip\" in\n \tt)\n \t\tsay_color skip >&3 \"skipping test: $@\"\n-- \n1.7.9\n"},{"id":"240305","messageId":"xmqqtx9ahpfp.fsf@gitster.dls.corp.google.com","threadId":"36472","inReplyTo":"5360C50F.7070505@gmail.com","subject":"Re: [PATCH 3/3] test-lib: '--run' to run only specific tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-30T14:17:30Z","receivedAt":"2014-04-30T14:17:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ilya Bobyr <ilya.bobyr@gmail.com> writes:\n\n>> The above two illustrate the reason rather well why I said it would\n>> be better to avoid negation because it would complicate the mental\n>> model the user needs to form when using the feature.\n>\n> I think that you do not have to use it if you do not need it.\n> It adds some expressiveness, is rather easy to implement and is already\n> there :)\n> I can remove it, of cause, but is it really necessary?\n\nAn extra \"expressiveness\" that needs explanation and careful\nthinking on the part of the user to pick the same world model you\npicked among multiple valid world models is not necessarily a good\naddition, so none of \"you do not have to use it\", \"it's already\nthere\", \"it is easy to implement\" is a valid argument.\n\nIf it weren't there, I wouldn't have had to wonder what the notation\nmeant, you wouldn't have had to explain it to me, and the most\nimportantly, nobody has to learn there is a subtle distinction\nbetween \"!7 -11\", \"!7-11\" and \"!7- 11\".\n"},{"id":"240838","messageId":"xmqqk39yeiic.fsf@gitster.dls.corp.google.com","threadId":"36472","inReplyTo":"1398851444-2492-4-git-send-email-ilya.bobyr@gmail.com","subject":"Re: [PATCH 3/3] test-lib: '--run' to run only specific tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-06T20:53:31Z","receivedAt":"2014-05-06T20:53:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ilya Bobyr <ilya.bobyr@gmail.com> writes:\n\n> Allow better control of the set of tests that will be executed for a\n> single test suite.  Mostly useful while debugging or developing as it\n> allows to focus on a specific test.\n>\n> Signed-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>\n> ---\n> A number of minor changes according to the review comments.\n\nI think the interaction between multiple selectors, especially when\nsome of them are negated, are much better explained in this version,\ncompared to the previous round in the README.\n\nBut I still think that the negation a feature that is unnecessary\nand having it makes it harder to understand for users, especially\nafter reading this part:\n\n> +If --run starts with an unprefixed number or range the initial\n> +set of tests to run is empty. If the first item starts with '!'\n> +all the tests are added to the initial set.  After initial set is\n> +determined every test number or range is added or excluded from\n> +the set one by one, from left to right.\n> ...\n> +As noted above, the test set is built going though items left to\n> +right, so this:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n> +\n> +will run tests 1, 2, and 4.  Items that comes later have higher\n> +precendence.  It means that this:\n> +\n> +    $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'\n> +\n> +would just run tests from 1 to 4, including 3.\n\nThe initial !3 means the same thing as \"1-2,4-\", and then 1-4 will\ndo what to that set?  The answer is \"It is added\"... wouldn't the\nreader expect then that the result should be \"1-\", not \"1-4\"?  I\nmyself wondered what would happen to the fifth test from your\ndescription.  Has the text told the reader that t9200 test has only\nfour tests?\n\nThe need to explain better with longer description will reduce the\nlikelyhood that the feature is understood and correctly used.  When\nyou can write \"1-2,4-\", why accept \"1-4 !3\" and force yourself to\nexplain to people why that is different from \"!3 1-4\"?\n"},{"id":"240839","messageId":"xmqqfvkmei3c.fsf@gitster.dls.corp.google.com","threadId":"36472","inReplyTo":"xmqqk39yeiic.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 3/3] test-lib: '--run' to run only specific tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-06T21:02:31Z","receivedAt":"2014-05-06T21:02:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> The need to explain better with longer description will reduce the\n> likelyhood that the feature is understood and correctly used.  When\n> you can write \"1-2,4-\", why accept \"1-4 !3\" and force yourself to\n> explain to people why that is different from \"!3 1-4\"?\n\nBy the way, having said all that, I would understand if the result\nis made not depend on the order of selectors given to the option.\n\nThat is:\n\n - Find all the positive ones and form the \"positive\" set; if there is\n   no positive ones, then make the \"positive\" set include everything.\n\n - Find all the negative ones and form a \"negative\" set.  The\n   \"negative\" set can be an empty set if there is no negative\n   selector.\n\n - Subtract the \"negative\" set from the \"positive\" set, and run only\n   the tests in the resulting set.\n\nThen \"1-4 !3\" and \"!3 1-4\" would mean the same thing.  Explanation\nof the semantics would be far simpler than \"we do it from left to\nright\".\n\nOne reason why the orders shouldn't matter is, unlike the \"print\ndialog\" case, we won't run tests out of order when given \"1 4 3 2\".\nWe will still run \"1-4\".  So it would be natural for the readers to\nexpect that the orders they give selectors would not matter.\n"}]}