{"thread":{"id":"54403","subject":"[PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","startedAt":"2020-10-12T23:26:13Z","lastAt":"2020-10-18T00:24:02Z","messageCount":71,"participants":["Elijah Newren via GitGitGadget","Phillip Wood","Taylor Blau","Elijah Newren","Junio C Hamano","Jeff King","Johannes Schindelin","Andrei Rybak"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"407402","messageId":"pull.878.git.git.1602545164.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":null,"subject":"[PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-12T23:26:01Z","receivedAt":"2020-10-12T23:26:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series enables me to change\n\n ./t9999-my-test.sh --run=1-3,5,17,19\n\ninto\n\n ./t9999-my-test.sh --run=setup,rename,symlink\n\nand have it pick out tests with \"setup\", \"rename\", or \"symlink\" in their\ndescription and run those. Saves me a lot of time, especially since numbers\nfor tests aren't readily accessible. The easiest way for me to get the\nnumbers corresponding to the tests I want to run, is to run all the tests\nand look at the output to match up the descriptions with their numbers --\nthus defeating the point of selecting just a subset of the tests to run in\nthe first place.\n\nElijah Newren (3):\n test-lib: allow selecting tests by substring/regex with --run\n t6006, t6012: adjust tests to use 'setup' instead of synonyms\n test-lib: reduce verbosity of skipped tests\n\n t/README | 26 +++++++++++++++++------\n t/t0000-basic.sh | 41 ++++++++++++++++++++++--------------\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n t/test-lib.sh | 15 +++++++------\n 5 files changed, 54 insertions(+), 32 deletions(-)\n\n\nbase-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v1\nPull-Request: https://github.com/git/git/pull/878\n-- \ngitgitgadget\n"},{"id":"407403","messageId":"46fce3a844c90b4078578f5aa2058bd6825af1d6.1602545164.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.git.git.1602545164.gitgitgadget@gmail.com","subject":"[PATCH 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-12T23:26:02Z","receivedAt":"2020-10-12T23:26:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany of our test scripts have several \"setup\" tests. It's a lot easier\nto say\n\n ./t0050-filesystem.sh --run=setup,9\n\nin order to run all the setup tests as well as test #9, than it is to\ntrack down what all the setup tests are and enter all their numbers in\nthe list. Also, I often find myself wanting to run just one or a couple\ntests from the test file, but I don't know the numbering of any of the\ntests -- to get it I either have to first run the whole test file (or\nstart counting by hand or figure out some other clever but non-obvious\ntricks). It's really convenient to be able to just look at the test\ndescription(s) and then run\n\n ./t6416-recursive-corner-cases.sh --run=symlink\n\nor\n\n ./t6402-merge-rename.sh --run='setup,unnecessary update'\n\nAdd such an ability to test selection which relies on merely matching\nagainst the test description.\n\nSigned-off-by: Elijah Newren \n---\n t/README | 26 +++++++++++++++++++-------\n t/t0000-basic.sh | 41 +++++++++++++++++++++++++----------------\n t/test-lib.sh | 13 +++++++------\n 3 files changed, 51 insertions(+), 29 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 2adaf7c2d2..23639c5bcf 100644\n--- a/t/README\n+++ b/t/README\n@@ -258,13 +258,13 @@ For an individual test suite --run could be used to specify that\n only some tests should be run or that some tests should be\n excluded from a run.\n \n-The argument for --run is a list of individual test numbers or\n-ranges with an optional negation prefix that define what tests in\n-a test suite to include in the run. A range is two numbers\n-separated with a dash and matches a range of tests with both ends\n-been included. You may omit the first or the second number to\n-mean \"from the first test\" or \"up to the very last test\"\n-respectively.\n+The argument for --run, , is a list of description\n+substrings or regexes or individual test numbers or ranges with an\n+optional negation prefix that define what tests in a test suite to\n+include in the run. A range is two numbers separated with a dash and\n+matches a range of tests with both ends been included. You may omit\n+the first or the second number to mean \"from the first test\" or \"up to\n+the very last test\" respectively.\n \n Optional prefix of '!' means that the test or a range of tests\n should be excluded from the run.\n@@ -317,6 +317,18 @@ test in the test suite except from 7 up to 11:\n \n $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n \n+Sometimes there may be multiple tests with e.g. \"setup\" in their name\n+that are needed and rather than figuring out the number for all of them\n+we can just use \"setup\" as a substring/regex to match against the test\n+description:\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,9-11\n+\n+or one could select both the setup tests and the rename ones (assuming all\n+relevant tests had those words in their descriptions):\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,rename\n+\n Some tests in a test suite rely on the previous tests performing\n certain actions, specifically some tests are designated as\n \"setup\" test, so you cannot _arbitrarily_ disable one test and\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 923281af93..07105b2078 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' \"\n \tEOF\n \"\n \n-test_expect_success '--run invalid range start' \"\n+test_expect_success '--run substring selector' \"\n+\trun_sub_test_lib_test run-substring-selector \\\n+\t\t'--run empty selectors' \\\n+\t\t--run='relevant' <<-\\\\EOF &&\n+\ttest_expect_success \\\"relevant test\\\" 'true'\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"other test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-substring-selector <<-\\\\EOF\n+\t> ok 1 - relevant test\n+\t> ok 2 # skip other test #1 (--run)\n+\t> ok 3 # skip other test #2 (--run)\n+\t> ok 4 # skip other test #3 (--run)\n+\t> ok 5 # skip other test #4 (--run)\n+\t> ok 6 # skip other test #5 (--run)\n+\t> ok 7 # skip other test #6 (--run)\n+\t> # passed all 7 test(s)\n+\t> 1..7\n+\tEOF\n+\"\n+\n+test_expect_success '--run keyword selection' \"\n \trun_sub_test_lib_test_err run-inv-range-start \\\n \t\t'--run invalid range start' \\\n \t\t--run='a-5' <<-\\\\EOF &&\n@@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' \"\n \tEOF_ERR\n \"\n \n-test_expect_success '--run invalid selector' \"\n-\trun_sub_test_lib_test_err run-inv-selector \\\n-\t\t'--run invalid selector' \\\n-\t\t--run='1?' <<-\\\\EOF &&\n-\ttest_expect_success \\\"passing test #1\\\" 'true'\n-\ttest_done\n-\tEOF\n-\tcheck_sub_test_lib_test_err run-inv-selector \\\n-\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n-\t> FATAL: Unexpected exit with code 1\n-\tEOF_OUT\n-\t> error: --run: invalid non-numeric in test selector: '1?'\n-\tEOF_ERR\n-\"\n-\n \n test_set_prereq HAVEIT\n haveit=no\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..2aca398e1e 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -769,6 +769,8 @@ match_pattern_list () {\n }\n \n match_test_selector_list () {\n+\toperation=\"$1\"\n+\tshift\n \ttitle=\"$1\"\n \tshift\n \targ=\"$1\"\n@@ -805,13 +807,13 @@ match_test_selector_list () {\n \t\t\t*-*)\n \t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"start: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n \t\t\t\tif expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"end: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n@@ -819,9 +821,8 @@ match_test_selector_list () {\n \t\t\t*)\n \t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n-\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n-\t\t\t\t\texit 1\n+\t\t\t\t\techo \"$title\" | grep -q \"$selector\" && return\n+\t\t\t\t\tcontinue\n \t\t\t\tfi\n \t\tesac\n \n@@ -1031,7 +1032,7 @@ test_skip () {\n \t\tskipped_reason=\"GIT_SKIP_TESTS\"\n \tfi\n \tif test -z \"$to_skip\" && test -n \"$run_list\" &&\n-\t ! match_test_selector_list '--run' $test_count \"$run_list\"\n+\t ! match_test_selector_list '--run' \"$1\" $test_count \"$run_list\"\n \tthen\n \t\tto_skip=t\n \t\tskipped_reason=\"--run\"\n-- \ngitgitgadget\n\n"},{"id":"407404","messageId":"bb2317972a8faf0358aaad1247fdfd2af2ea142d.1602545164.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.git.git.1602545164.gitgitgadget@gmail.com","subject":"[PATCH 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-12T23:26:04Z","receivedAt":"2020-10-12T23:26:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen using the --run flag to run just two or three tests from a test\nfile which contains several dozen tests, having every skipped test print\nout dozens of lines of output for the test code for that skipped test\nadds up to hundreds or thousands of lines of irrelevant output that make\nit very hard to fish out the relevant results you were looking for.\nSimplify the output for skipped tests down to just showing the one-line\ndescriptions.\n\nSigned-off-by: Elijah Newren \n---\n t/test-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 2aca398e1e..7602bbe9e9 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1059,7 +1059,7 @@ test_skip () {\n \t\t\t\t\" \"\n \t\tfi\n \n-\t\tsay_color skip >&3 \"skipping test: $@\"\n+\t\tsay_color skip >&3 \"skipping test: $1\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n-- \ngitgitgadget\n"},{"id":"407405","messageId":"d1d73400e838107b7a9a8562a20ec24a2a9a2ba6.1602545164.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.git.git.1602545164.gitgitgadget@gmail.com","subject":"[PATCH 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-12T23:26:03Z","receivedAt":"2020-10-12T23:27:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWith the new ability to pass --run=setup to select which tests to run,\nit is more convenient if tests use the term \"setup\" instead of synonyms\nlike 'prepare' or 'rebuild'. There are undoubtedly many other tests in\nour testsuite that could be changed over too, these are just a couple\nthat I ran into.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex bc95da8a5f..99a1eaf332 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -339,7 +339,7 @@ commit $head1\n .. (hinzugef${added_utf8_part_iso88591}gt) foo\n EOF\n \n-test_expect_success 'prepare expected messages (for test %b)' '\n+test_expect_success 'setup expected messages (for test %b)' '\n \tcat <<-EOF >expected.utf-8 &&\n \tcommit $head3\n \tThis commit message is much longer than the others,\ndiff --git a/t/t6012-rev-list-simplify.sh b/t/t6012-rev-list-simplify.sh\nindex b6fa43ace0..7254060240 100755\n--- a/t/t6012-rev-list-simplify.sh\n+++ b/t/t6012-rev-list-simplify.sh\n@@ -168,7 +168,7 @@ test_expect_success '--full-diff is not affected by --parents' '\n #\n # This example is explained in Documentation/rev-list-options.txt\n \n-test_expect_success 'rebuild repo' '\n+test_expect_success 'setup rebuild repo' '\n \trm -rf .git * &&\n \tgit init &&\n \tgit switch -c main &&\n-- \ngitgitgadget\n\n"},{"id":"407439","messageId":"fba0a914-1818-8a24-41b0-35740a7c2d2d@gmail.com","threadId":"54403","inReplyTo":"pull.878.git.git.1602545164.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-10-13T13:08:16Z","receivedAt":"2020-10-13T13:08:21Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Elijah\n\nI'm worryingly excited by this series\n\nOn 13/10/2020 00:26, Elijah Newren via GitGitGadget wrote:\n> This patch series enables me to change\n> \n> ./t9999-my-test.sh --run=1-3,5,17,19\n> \n> into\n> \n> ./t9999-my-test.sh --run=setup,rename,symlink\n> \n> and have it pick out tests with \"setup\", \"rename\", or \"symlink\" in their\n> description and run those. Saves me a lot of time, especially since numbers\n> for tests aren't readily accessible. The easiest way for me to get the\n> numbers corresponding to the tests I want to run, is to run all the tests\n> and look at the output to match up the descriptions with their numbers --\n> thus defeating the point of selecting just a subset of the tests to run in\n> the first place.\n\nYes it's a real pain to have to run all the tests to find the numbers of \nthe ones you actually wanted to run. I like patch 3 as well, without it \nif you're only running a couple of tests it can be hard to find then \namongst all the skipped tests. I had a quick read of the patches and \nthey looked fine to me.\n\nBest Wishes\n\nPhillip\n\n> Elijah Newren (3):\n> test-lib: allow selecting tests by substring/regex with --run\n> t6006, t6012: adjust tests to use 'setup' instead of synonyms\n> test-lib: reduce verbosity of skipped tests\n> \n> t/README | 26 +++++++++++++++++------\n> t/t0000-basic.sh | 41 ++++++++++++++++++++++--------------\n> t/t6006-rev-list-format.sh | 2 +-\n> t/t6012-rev-list-simplify.sh | 2 +-\n> t/test-lib.sh | 15 +++++++------\n> 5 files changed, 54 insertions(+), 32 deletions(-)\n> \n> \n> base-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v1\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v1\n> Pull-Request: https://github.com/git/git/pull/878\n> \n"},{"id":"407454","messageId":"20201013153901.GA1913367@nand.local","threadId":"54403","inReplyTo":"46fce3a844c90b4078578f5aa2058bd6825af1d6.1602545164.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-10-13T15:39:01Z","receivedAt":"2020-10-13T15:39:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 12, 2020 at 11:26:02PM +0000, Elijah Newren via GitGitGadget wrote:\n> diff --git a/t/README b/t/README\n> index 2adaf7c2d2..23639c5bcf 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -258,13 +258,13 @@ For an individual test suite --run could be used to specify that\n> only some tests should be run or that some tests should be\n> excluded from a run.\n>\n> -The argument for --run is a list of individual test numbers or\n> -ranges with an optional negation prefix that define what tests in\n> -a test suite to include in the run. A range is two numbers\n> -separated with a dash and matches a range of tests with both ends\n> -been included. You may omit the first or the second number to\n> -mean \"from the first test\" or \"up to the very last test\"\n> -respectively.\n> +The argument for --run, , is a list of description\n> +substrings or regexes or individual test numbers or ranges with an\n> +optional negation prefix that define what tests in a test suite to\n> +include in the run. A range is two numbers separated with a dash and\n> +matches a range of tests with both ends been included. You may omit\n> +the first or the second number to mean \"from the first test\" or \"up to\n> +the very last test\" respectively.\n>\n> Optional prefix of '!' means that the test or a range of tests\n> should be excluded from the run.\n\nThis piece of documentation looks to be now out of date; it mentions a\n\"test or range of tests\", but that only covers a two of the now four\nways to describe a test.\n\nThe rest of the patch looks great to me; and I, too, am very excited\nabout it.\n\nThanks,\nTaylor\n"},{"id":"407455","messageId":"20201013154439.GB1913367@nand.local","threadId":"54403","inReplyTo":"bb2317972a8faf0358aaad1247fdfd2af2ea142d.1602545164.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-10-13T15:44:39Z","receivedAt":"2020-10-13T15:44:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 12, 2020 at 11:26:04PM +0000, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n>\n> When using the --run flag to run just two or three tests from a test\n> file which contains several dozen tests, having every skipped test print\n> out dozens of lines of output for the test code for that skipped test\n> adds up to hundreds or thousands of lines of irrelevant output that make\n> it very hard to fish out the relevant results you were looking for.\n> Simplify the output for skipped tests down to just showing the one-line\n> descriptions.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> t/test-lib.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 2aca398e1e..7602bbe9e9 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -1059,7 +1059,7 @@ test_skip () {\n> \t\t\t\t\" \"\n> \t\tfi\n>\n> -\t\tsay_color skip >&3 \"skipping test: $@\"\n> +\t\tsay_color skip >&3 \"skipping test: $1\"\n\nI would be comfortable going further than this and removing this line\nentirely. We need the \"ok $test_count # skip\" below since it's part of\nour TAP output, but now the output looks somewhat redundant.\n\nWith this patch running a test that I'm working on with `--run=...`, I\nget output that looks like:\n\n skipping test: writing bitmaps via command-line can duplicate .keep objects\n ok 2 # skip writing bitmaps via command-line can duplicate .keep objects (--run)\n\n skipping test: writing bitmaps via config can duplicate .keep objects\n ok 3 # skip writing bitmaps via config can duplicate .keep objects (--run)\n\nScanning over the same test description twice per skipped test makes the\noutput difficult (but still much easier than before) to scan. What do\nyou think about either of the following:\n\n skipping test: writing bitmaps via command-line can duplicate .keep objects\n ok 2 # skip (--run)\n\nor:\n\n ok 2 # skip writing bitmaps via command-line can duplicate .keep objects (--run)\n\nI have a slight preference towards the latter, since it keeps more of\nthe information in the TAP line, and it cuts the total line count of\noutput from skipped tests in half.\n\nFor what it's worth, I'd also be fine with the patch as-is.\n\nThanks,\nTaylor\n"},{"id":"407460","messageId":"CABPp-BF4bqNXGusoUgzKUTecaR9bzXJbVODGVB2kqX5qTp2Fug@mail.gmail.com","threadId":"54403","inReplyTo":"20201013153901.GA1913367@nand.local","subject":"Re: [PATCH 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-13T17:23:32Z","receivedAt":"2020-10-13T17:23:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 13, 2020 at 8:39 AM Taylor Blau wrote:\n>\n> On Mon, Oct 12, 2020 at 11:26:02PM +0000, Elijah Newren via GitGitGadget wrote:\n> > diff --git a/t/README b/t/README\n> > index 2adaf7c2d2..23639c5bcf 100644\n> > --- a/t/README\n> > +++ b/t/README\n> > @@ -258,13 +258,13 @@ For an individual test suite --run could be used to specify that\n> > only some tests should be run or that some tests should be\n> > excluded from a run.\n> >\n> > -The argument for --run is a list of individual test numbers or\n> > -ranges with an optional negation prefix that define what tests in\n> > -a test suite to include in the run. A range is two numbers\n> > -separated with a dash and matches a range of tests with both ends\n> > -been included. You may omit the first or the second number to\n> > -mean \"from the first test\" or \"up to the very last test\"\n> > -respectively.\n> > +The argument for --run, , is a list of description\n> > +substrings or regexes or individual test numbers or ranges with an\n> > +optional negation prefix that define what tests in a test suite to\n> > +include in the run. A range is two numbers separated with a dash and\n> > +matches a range of tests with both ends been included. You may omit\n> > +the first or the second number to mean \"from the first test\" or \"up to\n> > +the very last test\" respectively.\n> >\n> > Optional prefix of '!' means that the test or a range of tests\n> > should be excluded from the run.\n>\n> This piece of documentation looks to be now out of date; it mentions a\n> \"test or range of tests\", but that only covers a two of the now four\n> ways to describe a test.\n\nPersonally, I had read \"test or range of tests\" as general enough to\ncover all the cases (I would have agreed that \"test *number* or range\nof tests\" would sound out of date), and just left it. However, if\nthere's something clearer, I'm happy to adopt it. Do you have an\nalternate wording suggestion?\n\n> The rest of the patch looks great to me; and I, too, am very excited\n> about it.\n>\n> Thanks,\n> Taylor\n"},{"id":"407461","messageId":"20201013172835.GA1917721@nand.local","threadId":"54403","inReplyTo":"CABPp-BF4bqNXGusoUgzKUTecaR9bzXJbVODGVB2kqX5qTp2Fug@mail.gmail.com","subject":"Re: [PATCH 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-10-13T17:28:35Z","receivedAt":"2020-10-13T17:28:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Oct 13, 2020 at 10:23:32AM -0700, Elijah Newren wrote:\n> On Tue, Oct 13, 2020 at 8:39 AM Taylor Blau wrote:\n> > This piece of documentation looks to be now out of date; it mentions a\n> > \"test or range of tests\", but that only covers a two of the now four\n> > ways to describe a test.\n>\n> Personally, I had read \"test or range of tests\" as general enough to\n> cover all the cases (I would have agreed that \"test *number* or range\n> of tests\" would sound out of date), and just left it. However, if\n> there's something clearer, I'm happy to adopt it. Do you have an\n> alternate wording suggestion?\n\nMaybe \"test selectors\"? You could even say something like \"test\nselectors (e.g., the test number, a range, a substring/regular\nexpression)\". I don't know, though; I don't feel strongly about it.\n\nThanks,\nTaylor\n"},{"id":"407465","messageId":"CABPp-BFj0S1Z7rgNGsUuRurNgKFOqtD_Nm0Dr3fZt-RFMTwoDw@mail.gmail.com","threadId":"54403","inReplyTo":"20201013154439.GB1913367@nand.local","subject":"Re: [PATCH 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-13T17:56:54Z","receivedAt":"2020-10-13T17:57:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 13, 2020 at 8:44 AM Taylor Blau wrote:\n>\n> On Mon, Oct 12, 2020 at 11:26:04PM +0000, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > When using the --run flag to run just two or three tests from a test\n> > file which contains several dozen tests, having every skipped test print\n> > out dozens of lines of output for the test code for that skipped test\n> > adds up to hundreds or thousands of lines of irrelevant output that make\n> > it very hard to fish out the relevant results you were looking for.\n> > Simplify the output for skipped tests down to just showing the one-line\n> > descriptions.\n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > t/test-lib.sh | 2 +-\n> > 1 file changed, 1 insertion(+), 1 deletion(-)\n> >\n> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> > index 2aca398e1e..7602bbe9e9 100644\n> > --- a/t/test-lib.sh\n> > +++ b/t/test-lib.sh\n> > @@ -1059,7 +1059,7 @@ test_skip () {\n> > \" \"\n> > fi\n> >\n> > - say_color skip >&3 \"skipping test: $@\"\n> > + say_color skip >&3 \"skipping test: $1\"\n>\n> I would be comfortable going further than this and removing this line\n> entirely. We need the \"ok $test_count # skip\" below since it's part of\n> our TAP output, but now the output looks somewhat redundant.\n>\n> With this patch running a test that I'm working on with `--run=...`, I\n> get output that looks like:\n>\n> skipping test: writing bitmaps via command-line can duplicate .keep objects\n> ok 2 # skip writing bitmaps via command-line can duplicate .keep objects (--run)\n>\n> skipping test: writing bitmaps via config can duplicate .keep objects\n> ok 3 # skip writing bitmaps via config can duplicate .keep objects (--run)\n>\n> Scanning over the same test description twice per skipped test makes the\n> output difficult (but still much easier than before) to scan. What do\n> you think about either of the following:\n>\n> skipping test: writing bitmaps via command-line can duplicate .keep objects\n> ok 2 # skip (--run)\n>\n> or:\n>\n> ok 2 # skip writing bitmaps via command-line can duplicate .keep objects (--run)\n>\n> I have a slight preference towards the latter, since it keeps more of\n> the information in the TAP line, and it cuts the total line count of\n> output from skipped tests in half.\n>\n> For what it's worth, I'd also be fine with the patch as-is.\n\nI was worried that shortening it as much as I did would run into\nobjections for some obscure reason. But if no one objects, I think\nI'd also prefer your latter suggestion for shortening it even more.\nIt looks like the dual output for skipped tests comes from commit\n04ece59399 (\"GIT_SKIP_TESTS: allow users to omit tests that are known\nto break\", 2006-12-28) by Junio, so it'd be nice to hear his opinion\non how much we shorten it.\n"},{"id":"407466","messageId":"CABPp-BHN6s6W03uH3WssUk0ukzgkBSiVXg3_K5m_XKNGtCSxiQ@mail.gmail.com","threadId":"54403","inReplyTo":"20201013172835.GA1917721@nand.local","subject":"Re: [PATCH 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-13T18:21:36Z","receivedAt":"2020-10-13T18:21:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 13, 2020 at 10:28 AM Taylor Blau wrote:\n>\n> On Tue, Oct 13, 2020 at 10:23:32AM -0700, Elijah Newren wrote:\n> > On Tue, Oct 13, 2020 at 8:39 AM Taylor Blau wrote:\n> > > This piece of documentation looks to be now out of date; it mentions a\n> > > \"test or range of tests\", but that only covers a two of the now four\n> > > ways to describe a test.\n> >\n> > Personally, I had read \"test or range of tests\" as general enough to\n> > cover all the cases (I would have agreed that \"test *number* or range\n> > of tests\" would sound out of date), and just left it. However, if\n> > there's something clearer, I'm happy to adopt it. Do you have an\n> > alternate wording suggestion?\n>\n> Maybe \"test selectors\"? You could even say something like \"test\n> selectors (e.g., the test number, a range, a substring/regular\n> expression)\". I don't know, though; I don't feel strongly about it.\n\nEarlier in the document, the --run option is defined as\n --run=::\nwhich might suggest to some readers that \"test-selector\" is the\ncombination of all the selections and thus that you can only negate\nthe whole series rather than individual components of it.\n\nPerhaps we can just nuke this sentence with a slight tweak of the\nparagraph before it to include slightly more about the \"optional\nnegation prefix\". I'll send an updated series that does that.\n"},{"id":"407477","messageId":"41e2528e83ba7087c9d21f0b15efed416f1512f8.1602616786.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v2.git.git.1602616786.gitgitgadget@gmail.com","subject":"[PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-13T19:19:44Z","receivedAt":"2020-10-13T19:19:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany of our test scripts have several \"setup\" tests. It's a lot easier\nto say\n\n ./t0050-filesystem.sh --run=setup,9\n\nin order to run all the setup tests as well as test #9, than it is to\ntrack down what all the setup tests are and enter all their numbers in\nthe list. Also, I often find myself wanting to run just one or a couple\ntests from the test file, but I don't know the numbering of any of the\ntests -- to get it I either have to first run the whole test file (or\nstart counting by hand or figure out some other clever but non-obvious\ntricks). It's really convenient to be able to just look at the test\ndescription(s) and then run\n\n ./t6416-recursive-corner-cases.sh --run=symlink\n\nor\n\n ./t6402-merge-rename.sh --run='setup,unnecessary update'\n\nAdd such an ability to test selection which relies on merely matching\nagainst the test description.\n\nSigned-off-by: Elijah Newren \n---\n t/README | 29 +++++++++++++++++++----------\n t/t0000-basic.sh | 41 +++++++++++++++++++++++++----------------\n t/test-lib.sh | 13 +++++++------\n 3 files changed, 51 insertions(+), 32 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 2adaf7c2d2..5fd8eaf595 100644\n--- a/t/README\n+++ b/t/README\n@@ -258,16 +258,13 @@ For an individual test suite --run could be used to specify that\n only some tests should be run or that some tests should be\n excluded from a run.\n \n-The argument for --run is a list of individual test numbers or\n-ranges with an optional negation prefix that define what tests in\n-a test suite to include in the run. A range is two numbers\n-separated with a dash and matches a range of tests with both ends\n-been included. You may omit the first or the second number to\n-mean \"from the first test\" or \"up to the very last test\"\n-respectively.\n-\n-Optional prefix of '!' means that the test or a range of tests\n-should be excluded from the run.\n+The argument for --run, , is a list of description\n+substrings or regexes or individual test numbers or ranges with an\n+optional negation prefix (of '!') that define what tests in a test\n+suite to include (or exclude, if negated) in the run. A range is two\n+numbers separated with a dash and matches a range of tests with both\n+ends been included. You may omit the first or the second number to\n+mean \"from the first test\" or \"up to the very last test\" respectively.\n \n If --run starts with an unprefixed number or range the initial\n set of tests to run is empty. If the first item starts with '!'\n@@ -317,6 +314,18 @@ test in the test suite except from 7 up to 11:\n \n $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n \n+Sometimes there may be multiple tests with e.g. \"setup\" in their name\n+that are needed and rather than figuring out the number for all of them\n+we can just use \"setup\" as a substring/regex to match against the test\n+description:\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,9-11\n+\n+or one could select both the setup tests and the rename ones (assuming all\n+relevant tests had those words in their descriptions):\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,rename\n+\n Some tests in a test suite rely on the previous tests performing\n certain actions, specifically some tests are designated as\n \"setup\" test, so you cannot _arbitrarily_ disable one test and\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 923281af93..07105b2078 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' \"\n \tEOF\n \"\n \n-test_expect_success '--run invalid range start' \"\n+test_expect_success '--run substring selector' \"\n+\trun_sub_test_lib_test run-substring-selector \\\n+\t\t'--run empty selectors' \\\n+\t\t--run='relevant' <<-\\\\EOF &&\n+\ttest_expect_success \\\"relevant test\\\" 'true'\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"other test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-substring-selector <<-\\\\EOF\n+\t> ok 1 - relevant test\n+\t> ok 2 # skip other test #1 (--run)\n+\t> ok 3 # skip other test #2 (--run)\n+\t> ok 4 # skip other test #3 (--run)\n+\t> ok 5 # skip other test #4 (--run)\n+\t> ok 6 # skip other test #5 (--run)\n+\t> ok 7 # skip other test #6 (--run)\n+\t> # passed all 7 test(s)\n+\t> 1..7\n+\tEOF\n+\"\n+\n+test_expect_success '--run keyword selection' \"\n \trun_sub_test_lib_test_err run-inv-range-start \\\n \t\t'--run invalid range start' \\\n \t\t--run='a-5' <<-\\\\EOF &&\n@@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' \"\n \tEOF_ERR\n \"\n \n-test_expect_success '--run invalid selector' \"\n-\trun_sub_test_lib_test_err run-inv-selector \\\n-\t\t'--run invalid selector' \\\n-\t\t--run='1?' <<-\\\\EOF &&\n-\ttest_expect_success \\\"passing test #1\\\" 'true'\n-\ttest_done\n-\tEOF\n-\tcheck_sub_test_lib_test_err run-inv-selector \\\n-\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n-\t> FATAL: Unexpected exit with code 1\n-\tEOF_OUT\n-\t> error: --run: invalid non-numeric in test selector: '1?'\n-\tEOF_ERR\n-\"\n-\n \n test_set_prereq HAVEIT\n haveit=no\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..2aca398e1e 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -769,6 +769,8 @@ match_pattern_list () {\n }\n \n match_test_selector_list () {\n+\toperation=\"$1\"\n+\tshift\n \ttitle=\"$1\"\n \tshift\n \targ=\"$1\"\n@@ -805,13 +807,13 @@ match_test_selector_list () {\n \t\t\t*-*)\n \t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"start: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n \t\t\t\tif expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"end: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n@@ -819,9 +821,8 @@ match_test_selector_list () {\n \t\t\t*)\n \t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n-\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n-\t\t\t\t\texit 1\n+\t\t\t\t\techo \"$title\" | grep -q \"$selector\" && return\n+\t\t\t\t\tcontinue\n \t\t\t\tfi\n \t\tesac\n \n@@ -1031,7 +1032,7 @@ test_skip () {\n \t\tskipped_reason=\"GIT_SKIP_TESTS\"\n \tfi\n \tif test -z \"$to_skip\" && test -n \"$run_list\" &&\n-\t ! match_test_selector_list '--run' $test_count \"$run_list\"\n+\t ! match_test_selector_list '--run' \"$1\" $test_count \"$run_list\"\n \tthen\n \t\tto_skip=t\n \t\tskipped_reason=\"--run\"\n-- \ngitgitgadget\n\n"},{"id":"407478","messageId":"8f1a4420f4eca64dece1cc40e20134001eb1dd74.1602616786.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v2.git.git.1602616786.gitgitgadget@gmail.com","subject":"[PATCH v2 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-13T19:19:45Z","receivedAt":"2020-10-13T19:19:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWith the new ability to pass --run=setup to select which tests to run,\nit is more convenient if tests use the term \"setup\" instead of synonyms\nlike 'prepare' or 'rebuild'. There are undoubtedly many other tests in\nour testsuite that could be changed over too, these are just a couple\nthat I ran into.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex bc95da8a5f..99a1eaf332 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -339,7 +339,7 @@ commit $head1\n .. (hinzugef${added_utf8_part_iso88591}gt) foo\n EOF\n \n-test_expect_success 'prepare expected messages (for test %b)' '\n+test_expect_success 'setup expected messages (for test %b)' '\n \tcat <<-EOF >expected.utf-8 &&\n \tcommit $head3\n \tThis commit message is much longer than the others,\ndiff --git a/t/t6012-rev-list-simplify.sh b/t/t6012-rev-list-simplify.sh\nindex b6fa43ace0..7254060240 100755\n--- a/t/t6012-rev-list-simplify.sh\n+++ b/t/t6012-rev-list-simplify.sh\n@@ -168,7 +168,7 @@ test_expect_success '--full-diff is not affected by --parents' '\n #\n # This example is explained in Documentation/rev-list-options.txt\n \n-test_expect_success 'rebuild repo' '\n+test_expect_success 'setup rebuild repo' '\n \trm -rf .git * &&\n \tgit init &&\n \tgit switch -c main &&\n-- \ngitgitgadget\n\n"},{"id":"407479","messageId":"pull.878.v2.git.git.1602616786.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.git.git.1602545164.gitgitgadget@gmail.com","subject":"[PATCH v2 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-13T19:19:43Z","receivedAt":"2020-10-13T19:19:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series enables me to change\n\n ./t9999-my-test.sh --run=1-3,5,17,19\n\ninto\n\n ./t9999-my-test.sh --run=setup,rename,symlink\n\nand have it pick out tests with \"setup\", \"rename\", or \"symlink\" in their\ndescription and run those. Saves me a lot of time, especially since numbers\nfor tests aren't readily accessible. The easiest way for me to get the\nnumbers corresponding to the tests I want to run, is to run all the tests\nand look at the output to match up the descriptions with their numbers --\nthus defeating the point of selecting just a subset of the tests to run in\nthe first place.\n\nChanges since v1:\n\n * Modified the documentation slightly to combine a separate sentence into\n the previous paragraph and avoid confusion around whether negation\n applies to substring selections (pointed out by Taylor)\n * Simplified the output for skipped tests even further, as suggested by\n Taylor.\n\nElijah Newren (3):\n test-lib: allow selecting tests by substring/regex with --run\n t6006, t6012: adjust tests to use 'setup' instead of synonyms\n test-lib: reduce verbosity of skipped tests\n\n t/README | 29 ++++++++++++++++---------\n t/t0000-basic.sh | 41 ++++++++++++++++++++++--------------\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n t/test-lib.sh | 14 ++++++------\n 5 files changed, 53 insertions(+), 35 deletions(-)\n\n\nbase-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v2\nPull-Request: https://github.com/git/git/pull/878\n\nRange-diff vs v1:\n\n 1: 46fce3a844 ! 1: 41e2528e83 test-lib: allow selecting tests by substring/regex with --run\n @@ t/README: For an individual test suite --run could be used to specify that\n -been included. You may omit the first or the second number to\n -mean \"from the first test\" or \"up to the very last test\"\n -respectively.\n +-\n +-Optional prefix of '!' means that the test or a range of tests\n +-should be excluded from the run.\n +The argument for --run, , is a list of description\n +substrings or regexes or individual test numbers or ranges with an\n -+optional negation prefix that define what tests in a test suite to\n -+include in the run. A range is two numbers separated with a dash and\n -+matches a range of tests with both ends been included. You may omit\n -+the first or the second number to mean \"from the first test\" or \"up to\n -+the very last test\" respectively.\n ++optional negation prefix (of '!') that define what tests in a test\n ++suite to include (or exclude, if negated) in the run. A range is two\n ++numbers separated with a dash and matches a range of tests with both\n ++ends been included. You may omit the first or the second number to\n ++mean \"from the first test\" or \"up to the very last test\" respectively.\n \n - Optional prefix of '!' means that the test or a range of tests\n - should be excluded from the run.\n + If --run starts with an unprefixed number or range the initial\n + set of tests to run is empty. If the first item starts with '!'\n @@ t/README: test in the test suite except from 7 up to 11:\n \n $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n 2: d1d73400e8 = 2: 8f1a4420f4 t6006, t6012: adjust tests to use 'setup' instead of synonyms\n 3: bb2317972a ! 3: 85a4ca164a test-lib: reduce verbosity of skipped tests\n @@ t/test-lib.sh: test_skip () {\n \t\tfi\n \n -\t\tsay_color skip >&3 \"skipping test: $@\"\n -+\t\tsay_color skip >&3 \"skipping test: $1\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n\n-- \ngitgitgadget\n"},{"id":"407480","messageId":"85a4ca164a9f665016d4aad0f29cbef6f62f36b0.1602616786.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v2.git.git.1602616786.gitgitgadget@gmail.com","subject":"[PATCH v2 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-13T19:19:46Z","receivedAt":"2020-10-13T19:19:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen using the --run flag to run just two or three tests from a test\nfile which contains several dozen tests, having every skipped test print\nout dozens of lines of output for the test code for that skipped test\nadds up to hundreds or thousands of lines of irrelevant output that make\nit very hard to fish out the relevant results you were looking for.\nSimplify the output for skipped tests down to just showing the one-line\ndescriptions.\n\nSigned-off-by: Elijah Newren \n---\n t/test-lib.sh | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 2aca398e1e..fe81d9c733 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1059,7 +1059,6 @@ test_skip () {\n \t\t\t\t\" \"\n \t\tfi\n \n-\t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n-- \ngitgitgadget\n"},{"id":"407481","messageId":"xmqqpn5mj59r.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"CABPp-BFj0S1Z7rgNGsUuRurNgKFOqtD_Nm0Dr3fZt-RFMTwoDw@mail.gmail.com","subject":"Re: [PATCH 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-13T19:27:12Z","receivedAt":"2020-10-13T19:27:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n>> skipping test: writing bitmaps via command-line can duplicate .keep objects\n>> ok 2 # skip (--run)\n>>\n>> or:\n>>\n>> ok 2 # skip writing bitmaps via command-line can duplicate .keep objects (--run)\n>>\n>> I have a slight preference towards the latter, since it keeps more of\n>> the information in the TAP line, and it cuts the total line count of\n>> output from skipped tests in half.\n>>\n>> For what it's worth, I'd also be fine with the patch as-is.\n>\n> I was worried that shortening it as much as I did would run into\n> objections for some obscure reason. But if no one objects, I think\n> I'd also prefer your latter suggestion for shortening it even more.\n> It looks like the dual output for skipped tests comes from commit\n> 04ece59399 (\"GIT_SKIP_TESTS: allow users to omit tests that are known\n> to break\", 2006-12-28) by Junio, so it'd be nice to hear his opinion\n> on how much we shorten it.\n\nI am afraid that the surrounding code has changed so heavily that\nexpertise as the author of 04ece593 would not count that much in the\ncontext of today's code. Back in the late 2006 version of the code,\ntext_expect_success asked test_skip to see if a test should be\nskipped, and when the latter says yes (which also gave \"skipping\ntest\" to the verbose output and \"skip $test_count: $1\" to normal\noutput), test_expect_success did not even call test_ok_ or\ntest_failure_. So \"ok 2 # skip ...\" we see in the quoted part of\nyour exchange above is probably a much more recent invention.\n\nAs long as we can convince ourselves that writing similar things to\nfd #1 (for tap in \"ok $count # skip ($why)\" format) and fd #3 is\nredundant and not useful, I think removing the utterance to fd #3\nin this codepath may be a reasonable thing to do.\n\n ... goes and scans for \">&3\" in test-lib.sh and\n ... test-lib-functions.sh\n\nOK, I think we can safely lose the one we send for the \"verbose\"\ncase here and that would turn our use of fd #3 more consistent with\nhow it is used by other parts of the test framework.\n\nThanks.\n\n\n"},{"id":"407482","messageId":"20201013194913.GA1921887@nand.local","threadId":"54403","inReplyTo":"pull.878.v2.git.git.1602616786.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-10-13T19:49:13Z","receivedAt":"2020-10-13T19:49:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Oct 13, 2020 at 07:19:43PM +0000, Elijah Newren via GitGitGadget wrote:\n> Changes since v1:\n>\n> * Modified the documentation slightly to combine a separate sentence into\n> the previous paragraph and avoid confusion around whether negation\n> applies to substring selections (pointed out by Taylor)\n> * Simplified the output for skipped tests even further, as suggested by\n> Taylor.\n\nThanks, this version looks very good to me. With Junio's ack on the\nchanges in test-lib, I feel much better about my own suggestion. So,\nthis all looks good to me.\n\nThanks,\nTaylor\n"},{"id":"407522","messageId":"20201014165329.GA21687@coredump.intra.peff.net","threadId":"54403","inReplyTo":"85a4ca164a9f665016d4aad0f29cbef6f62f36b0.1602616786.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-14T16:53:29Z","receivedAt":"2020-10-14T16:53:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 13, 2020 at 07:19:46PM +0000, Elijah Newren via GitGitGadget wrote:\n\n> From: Elijah Newren \n> \n> When using the --run flag to run just two or three tests from a test\n> file which contains several dozen tests, having every skipped test print\n> out dozens of lines of output for the test code for that skipped test\n> adds up to hundreds or thousands of lines of irrelevant output that make\n> it very hard to fish out the relevant results you were looking for.\n> Simplify the output for skipped tests down to just showing the one-line\n> descriptions.\n\nThis last sentence is inaccurate in this version, isn't it?\n\nOther than that, I think this is a good change (I admit I never noticed\nthe irrelevant output because it is only shown with \"-v\", and that is\nalready full of irrelevant bits. But I have trouble imagining how it\nwould be useful).\n\nEarlier discussion mentioned shortening the \"ok 2 # skip\" line, but I\nthink removing this fd-3 version is the only sane direction. The other\none gets parsed by TAP tools like prove, and may be shown. E.g., \"prove\n--directive t0003-attributes.sh\" shows the description of the skipped\ntests (likewise, \"-v\" shows all tests but highlights the skipped ones).\n\n-Peff\n"},{"id":"407525","messageId":"20201014170413.GB21687@coredump.intra.peff.net","threadId":"54403","inReplyTo":"41e2528e83ba7087c9d21f0b15efed416f1512f8.1602616786.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-14T17:04:13Z","receivedAt":"2020-10-14T17:04:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 13, 2020 at 07:19:44PM +0000, Elijah Newren via GitGitGadget wrote:\n\n> Many of our test scripts have several \"setup\" tests. It's a lot easier\n> to say\n> \n> ./t0050-filesystem.sh --run=setup,9\n\nI like this direction very well.\n\nThere was a small discussion recently that we might be better off\ndropping test script numbers entirely, and allowing selection of script\nnames by word-hierarchy or regex, too. That's mostly orthogonal to what\nyou're doing here, but I think this is taking us in a good direction\nthere.\n\n> @@ -819,9 +821,8 @@ match_test_selector_list () {\n> \t\t\t*)\n> \t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n> \t\t\t\tthen\n> -\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n> -\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n> -\t\t\t\t\texit 1\n> +\t\t\t\t\techo \"$title\" | grep -q \"$selector\" && return\n> +\t\t\t\t\tcontinue\n> \t\t\t\tfi\n\nI like that you allow regexes. It's unfortunate that the skip-check\ncosts us a process in every test. It may not be that big a deal since we\nonly pay it if you use a non-numeric selector. But I wonder if there's\nany reason not to use \"expr\" here, as well.\n\n-Peff\n"},{"id":"407531","messageId":"CABPp-BERKaLTLFjXYSo2mbT+3RSMR+5M7pzPmHH-0hNP2KFMOA@mail.gmail.com","threadId":"54403","inReplyTo":"20201014165329.GA21687@coredump.intra.peff.net","subject":"Re: [PATCH v2 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-14T17:39:01Z","receivedAt":"2020-10-14T17:39:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Oct 14, 2020 at 9:53 AM Jeff King wrote:\n>\n> On Tue, Oct 13, 2020 at 07:19:46PM +0000, Elijah Newren via GitGitGadget wrote:\n>\n> > From: Elijah Newren \n> >\n> > When using the --run flag to run just two or three tests from a test\n> > file which contains several dozen tests, having every skipped test print\n> > out dozens of lines of output for the test code for that skipped test\n> > adds up to hundreds or thousands of lines of irrelevant output that make\n> > it very hard to fish out the relevant results you were looking for.\n> > Simplify the output for skipped tests down to just showing the one-line\n> > descriptions.\n>\n> This last sentence is inaccurate in this version, isn't it?\n\nMaybe I could make it clearer, but I think that it is accurate[1]. If\nthis wording seems confusing, though, I could simplify the commit\nmessage by reducing the sentence to \"Simplify the output for skipped\ntests.\"\n\n[1] The original code showed the one-line description (with a\n\"skipping test:\" prefix) AND the full test code AND then repeated the\none-line description on an \"ok\" line. My v1 version of this patch\nripped out the long middle portion -- the full test code -- and thus\nonly showed the other bits, namely showing the one-line descriptions\ntwice (once with the \"skipping test:\" prefix, and the second time with\nan \"ok # \" prefix). This v2 version of the patch shows the\none-line description only once with the \"ok #\" prefix. So,\nI'd say the commit message is still accurate: the code only shows the\none-line description in both v1 and v2, though there is a question of\nhow many times it is shown.\n\n> Other than that, I think this is a good change (I admit I never noticed\n> the irrelevant output because it is only shown with \"-v\", and that is\n> already full of irrelevant bits. But I have trouble imagining how it\n> would be useful).\n>\n> Earlier discussion mentioned shortening the \"ok 2 # skip\" line, but I\n> think removing this fd-3 version is the only sane direction. The other\n> one gets parsed by TAP tools like prove, and may be shown. E.g., \"prove\n> --directive t0003-attributes.sh\" shows the description of the skipped\n> tests (likewise, \"-v\" shows all tests but highlights the skipped ones).\n\nMake sense, thanks.\n"},{"id":"407532","messageId":"CAPc5daVa4j5mDvTzCFCX2bLr1qVFLO=P=nSakT+mDG1kqeXa+g@mail.gmail.com","threadId":"54403","inReplyTo":"20201014170413.GB21687@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-14T17:46:15Z","receivedAt":"2020-10-14T17:46:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"> > @@ -819,9 +821,8 @@ match_test_selector_list () {\n> > *)\n> > if expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n> > then\n> > - echo \"error: $title: invalid non-numeric in test\" \\\n> > - \"selector: '$orig_selector'\" >&2\n> > - exit 1\n> > + echo \"$title\" | grep -q \"$selector\" && return\n> > + continue\n> > fi\n>\n> I like that you allow regexes. It's unfortunate that the skip-check\n> costs us a process in every test. It may not be that big a deal since we\n> only pay it if you use a non-numeric selector. But I wonder if there's\n> any reason not to use \"expr\" here, as well.\n\nIf you define the pattern is not regexp but is glob, you can use\ncase/esac to do this without any forking.\n\nYour expr may well be built-in, though.\n"},{"id":"407533","messageId":"CABPp-BEOrYhUBZ112Z7qFgwUYojmus6_Ra0G0Ts4y+n=-D5vMw@mail.gmail.com","threadId":"54403","inReplyTo":"20201014170413.GB21687@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-14T17:50:03Z","receivedAt":"2020-10-14T17:50:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Oct 14, 2020 at 10:04 AM Jeff King wrote:\n>\n> On Tue, Oct 13, 2020 at 07:19:44PM +0000, Elijah Newren via GitGitGadget wrote:\n>\n> > Many of our test scripts have several \"setup\" tests. It's a lot easier\n> > to say\n> >\n> > ./t0050-filesystem.sh --run=setup,9\n>\n> I like this direction very well.\n>\n> There was a small discussion recently that we might be better off\n> dropping test script numbers entirely, and allowing selection of script\n> names by word-hierarchy or regex, too. That's mostly orthogonal to what\n> you're doing here, but I think this is taking us in a good direction\n> there.\n>\n> > @@ -819,9 +821,8 @@ match_test_selector_list () {\n> > *)\n> > if expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n> > then\n> > - echo \"error: $title: invalid non-numeric in test\" \\\n> > - \"selector: '$orig_selector'\" >&2\n> > - exit 1\n> > + echo \"$title\" | grep -q \"$selector\" && return\n> > + continue\n> > fi\n>\n> I like that you allow regexes. It's unfortunate that the skip-check\n> costs us a process in every test. It may not be that big a deal since we\n> only pay it if you use a non-numeric selector. But I wonder if there's\n> any reason not to use \"expr\" here, as well.\n\nI originally used [[ $title =~ \"$selector\" ]] in order to avoid the\nsub-shell...but that was bash-specific. I briefly looked to see if\nthere was a shell portable way to handle regexes, wasn't having much\nluck, and decided that this is only likely to arise when people are\npassing --run and thus only running a single script and they avoid all\nthe subprocesses that would have been invoked inside the test, so it's\nstill a big net win overall. Does expr handle regexes, portably? Or\nare you suggesting dropping the regex handling and limit it to\nsubstring matching? In either case, does using expr save us anything\n(isn't expr a shell command)?\n"},{"id":"407534","messageId":"20201014175512.GA24279@coredump.intra.peff.net","threadId":"54403","inReplyTo":"CABPp-BERKaLTLFjXYSo2mbT+3RSMR+5M7pzPmHH-0hNP2KFMOA@mail.gmail.com","subject":"Re: [PATCH v2 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-14T17:55:12Z","receivedAt":"2020-10-14T17:55:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 14, 2020 at 10:39:01AM -0700, Elijah Newren wrote:\n\n> On Wed, Oct 14, 2020 at 9:53 AM Jeff King wrote:\n> >\n> > On Tue, Oct 13, 2020 at 07:19:46PM +0000, Elijah Newren via GitGitGadget wrote:\n> >\n> > > From: Elijah Newren \n> > >\n> > > When using the --run flag to run just two or three tests from a test\n> > > file which contains several dozen tests, having every skipped test print\n> > > out dozens of lines of output for the test code for that skipped test\n> > > adds up to hundreds or thousands of lines of irrelevant output that make\n> > > it very hard to fish out the relevant results you were looking for.\n> > > Simplify the output for skipped tests down to just showing the one-line\n> > > descriptions.\n> >\n> > This last sentence is inaccurate in this version, isn't it?\n> \n> Maybe I could make it clearer, but I think that it is accurate[1]. If\n> this wording seems confusing, though, I could simplify the commit\n> message by reducing the sentence to \"Simplify the output for skipped\n> tests.\"\n\nYeah, I wondered if you might have been thinking that. It makes sense in\nthe context of the other discussion, but the single-line TAP output is\nnot even mentioned here. And it might be worth doing so, because the\nreal reason it is OK to delete this line entirely is that it is\nredundant with that line.\n\n-Peff\n"},{"id":"407536","messageId":"CAPc5daU+w2U_5FBrSYEVO5PB2sOV_fxvBQ1=51_=NMTZooZGmA@mail.gmail.com","threadId":"54403","inReplyTo":"CABPp-BEOrYhUBZ112Z7qFgwUYojmus6_Ra0G0Ts4y+n=-D5vMw@mail.gmail.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-14T17:57:43Z","receivedAt":"2020-10-14T17:58:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"> are you suggesting dropping the regex handling and limit it to\n> substring matching? In either case, does using expr save us anything\n> (isn't expr a shell command)?\n\nI had something along this line in mind, not to do a regex but a glob.\n\ncase \"$title\" in $selector) found=1 ;; esac\n"},{"id":"407537","messageId":"CABPp-BHM3PdYGv12wAVuo22V2L97=9Sp3mpWPbqFeKDbGV24Ng@mail.gmail.com","threadId":"54403","inReplyTo":"20201014175512.GA24279@coredump.intra.peff.net","subject":"Re: [PATCH v2 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-14T17:57:51Z","receivedAt":"2020-10-14T17:58:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Oct 14, 2020 at 10:55 AM Jeff King wrote:\n>\n> On Wed, Oct 14, 2020 at 10:39:01AM -0700, Elijah Newren wrote:\n>\n> > On Wed, Oct 14, 2020 at 9:53 AM Jeff King wrote:\n> > >\n> > > On Tue, Oct 13, 2020 at 07:19:46PM +0000, Elijah Newren via GitGitGadget wrote:\n> > >\n> > > > From: Elijah Newren \n> > > >\n> > > > When using the --run flag to run just two or three tests from a test\n> > > > file which contains several dozen tests, having every skipped test print\n> > > > out dozens of lines of output for the test code for that skipped test\n> > > > adds up to hundreds or thousands of lines of irrelevant output that make\n> > > > it very hard to fish out the relevant results you were looking for.\n> > > > Simplify the output for skipped tests down to just showing the one-line\n> > > > descriptions.\n> > >\n> > > This last sentence is inaccurate in this version, isn't it?\n> >\n> > Maybe I could make it clearer, but I think that it is accurate[1]. If\n> > this wording seems confusing, though, I could simplify the commit\n> > message by reducing the sentence to \"Simplify the output for skipped\n> > tests.\"\n>\n> Yeah, I wondered if you might have been thinking that. It makes sense in\n> the context of the other discussion, but the single-line TAP output is\n> not even mentioned here. And it might be worth doing so, because the\n> real reason it is OK to delete this line entirely is that it is\n> redundant with that line.\n\nMakes sense; I'll update it to mention that, once the discussion on\nhow we want to handle regexes/globs/subshells for patch 1/3 is\nresolved.\n"},{"id":"407538","messageId":"20201014180707.GB24279@coredump.intra.peff.net","threadId":"54403","inReplyTo":"CAPc5daVa4j5mDvTzCFCX2bLr1qVFLO=P=nSakT+mDG1kqeXa+g@mail.gmail.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-14T18:07:07Z","receivedAt":"2020-10-14T18:07:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 14, 2020 at 10:46:15AM -0700, Junio C Hamano wrote:\n\n> > I like that you allow regexes. It's unfortunate that the skip-check\n> > costs us a process in every test. It may not be that big a deal since we\n> > only pay it if you use a non-numeric selector. But I wonder if there's\n> > any reason not to use \"expr\" here, as well.\n> \n> If you define the pattern is not regexp but is glob, you can use\n> case/esac to do this without any forking.\n\nYes, that would probably be OK for most purposes, though I admit my real\nlove for regex support is the ability to use \".\" instead of space to\navoid quoting arguments. ;)\n\nGlobs may make some real patterns slightly simpler, though. I imagine\nthat the \"setup\" example may need to be \"set.?up\" or \"set.*up\" in\npractice. Which is only \"set*up\" as a glob (I also don't have a problem\nstandardizing on one spelling as people find cases).\n\n> Your expr may well be built-in, though.\n\nYeah, that was my assumption, though I didn't bother to test it. Having\ndone so, it looks like it's not a built-in either in dash or bash. So\nswitching to it from grep may be buying less in practice than I thought.\n\nWe're also running a ton of exprs earlier in the function. Running:\n\n strace -f -e execve -o foo.out ./t0003-attributes.sh --run=10\n\nappears to exec expr 65 times. There are only 103 execves total in the\nwhole run, so that's more than half of them! It might be worth seeing if\nsome of those could do globbing via case/esac.\n\nRepeating without \"--run\" yields 39 exprs out of 492 execs. So that's\nless abysmal. Most of those are from test_oid_cache.\n\n-Peff\n"},{"id":"407539","messageId":"20201014180959.GC24279@coredump.intra.peff.net","threadId":"54403","inReplyTo":"CABPp-BEOrYhUBZ112Z7qFgwUYojmus6_Ra0G0Ts4y+n=-D5vMw@mail.gmail.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-14T18:09:59Z","receivedAt":"2020-10-14T18:10:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 14, 2020 at 10:50:03AM -0700, Elijah Newren wrote:\n\n> > > @@ -819,9 +821,8 @@ match_test_selector_list () {\n> > > *)\n> > > if expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n> > > then\n> > > - echo \"error: $title: invalid non-numeric in test\" \\\n> > > - \"selector: '$orig_selector'\" >&2\n> > > - exit 1\n> > > + echo \"$title\" | grep -q \"$selector\" && return\n> > > + continue\n> > > fi\n> >\n> > I like that you allow regexes. It's unfortunate that the skip-check\n> > costs us a process in every test. It may not be that big a deal since we\n> > only pay it if you use a non-numeric selector. But I wonder if there's\n> > any reason not to use \"expr\" here, as well.\n> \n> I originally used [[ $title =~ \"$selector\" ]] in order to avoid the\n> sub-shell...but that was bash-specific. I briefly looked to see if\n> there was a shell portable way to handle regexes, wasn't having much\n> luck, and decided that this is only likely to arise when people are\n> passing --run and thus only running a single script and they avoid all\n> the subprocesses that would have been invoked inside the test, so it's\n> still a big net win overall. Does expr handle regexes, portably? Or\n> are you suggesting dropping the regex handling and limit it to\n> substring matching? In either case, does using expr save us anything\n> (isn't expr a shell command)?\n\nThere's an expr command doing a regex match in the diff context quoted\nabove. :)\n\nI was wrong that it would save us a process, though. I thought it was a\nbuiltin in modern shells, but it doesn't appear to be in either dash or\nbash. So there's little reason to prefer it over grep here (switching to\nglobs could be done with case and would save a process).\n\n-Peff\n"},{"id":"407543","messageId":"CABPp-BHV4WGygzyw6RUsJqCGkvmc-f4dseFjbAG9hUvJgkYx+Q@mail.gmail.com","threadId":"54403","inReplyTo":"20201014180959.GC24279@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-14T19:02:07Z","receivedAt":"2020-10-14T19:02:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Oct 14, 2020 at 11:09 AM Jeff King wrote:\n>\n> On Wed, Oct 14, 2020 at 10:50:03AM -0700, Elijah Newren wrote:\n>\n> > > > @@ -819,9 +821,8 @@ match_test_selector_list () {\n> > > > *)\n> > > > if expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n> > > > then\n> > > > - echo \"error: $title: invalid non-numeric in test\" \\\n> > > > - \"selector: '$orig_selector'\" >&2\n> > > > - exit 1\n> > > > + echo \"$title\" | grep -q \"$selector\" && return\n> > > > + continue\n> > > > fi\n> > >\n> > > I like that you allow regexes. It's unfortunate that the skip-check\n> > > costs us a process in every test. It may not be that big a deal since we\n> > > only pay it if you use a non-numeric selector. But I wonder if there's\n> > > any reason not to use \"expr\" here, as well.\n> >\n> > I originally used [[ $title =~ \"$selector\" ]] in order to avoid the\n> > sub-shell...but that was bash-specific. I briefly looked to see if\n> > there was a shell portable way to handle regexes, wasn't having much\n> > luck, and decided that this is only likely to arise when people are\n> > passing --run and thus only running a single script and they avoid all\n> > the subprocesses that would have been invoked inside the test, so it's\n> > still a big net win overall. Does expr handle regexes, portably? Or\n> > are you suggesting dropping the regex handling and limit it to\n> > substring matching? In either case, does using expr save us anything\n> > (isn't expr a shell command)?\n>\n> There's an expr command doing a regex match in the diff context quoted\n> above. :)\n\nOh, indeed. How'd I miss that?\n\n> I was wrong that it would save us a process, though. I thought it was a\n> builtin in modern shells, but it doesn't appear to be in either dash or\n> bash. So there's little reason to prefer it over grep here (switching to\n> globs could be done with case and would save a process).\n\nI guess it's at least slightly more consistent with the surrounding code.\n\nOh, and I found a bug in that my code was not handling negated\nsubstrings correctly, e.g.\n ./test-script.sh --run=\"!rename\"\nwould run all the tests instead of excluding the ones that had\n\"rename\" as a substring. So I'll send a reroll fixing that.\n"},{"id":"407546","messageId":"CABPp-BGBG=wh41BrvFZdzsN6biUPvuDkrA9EZURM4GKLx8kC0A@mail.gmail.com","threadId":"54403","inReplyTo":"CAPc5daU+w2U_5FBrSYEVO5PB2sOV_fxvBQ1=51_=NMTZooZGmA@mail.gmail.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-14T19:12:22Z","receivedAt":"2020-10-14T19:12:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Oct 14, 2020 at 10:57 AM Junio C Hamano wrote:\n>\n> > are you suggesting dropping the regex handling and limit it to\n> > substring matching? In either case, does using expr save us anything\n> > (isn't expr a shell command)?\n>\n> I had something along this line in mind, not to do a regex but a glob.\n>\n> case \"$title\" in $selector) found=1 ;; esac\n\nInteresting. Since it needs to handle substring searching (e.g.\n./test-script.sh --run=setup,rename), I think this would need to be\ntweaked to be\n case \"$title\" in *${selector}*) include=$positive ;; esac\n\nThat'd probably be good enough for most cases, but I'm still inclined\nto just pay the subprocess cost in order to allow regexes. If someone\nis specifying --run, they are going to be skipping many of the tests\n(and thus a whole code block with dozens or even hundreds of execs per\ntest skipped), more than making up for the one extra exec per test.\nAnd if someone doesn't specify --run, there is no extra cost.\n"},{"id":"407547","messageId":"CAPc5daU_aw=yWPKHG1XGt4X5A1AkEXeLfCmHqqVfZ+FzBD7nxg@mail.gmail.com","threadId":"54403","inReplyTo":"20201014180707.GB24279@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-14T19:24:22Z","receivedAt":"2020-10-14T19:24:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"> > If you define the pattern is not regexp but is glob, you can use\n> > case/esac to do this without any forking.\n>\n> Yes, that would probably be OK for most purposes, though I admit my real\n> love for regex support is the ability to use \".\" instead of space to\n> avoid quoting arguments. ;)\n\nI use \"?\" for the same purpose for globs. For things that are casual,\nI find that\nit tends to make the end-user (read: my) experience simpler to use globs than\nto use regexp, largely for your \".*\" vs \"*\" reasons.\n"},{"id":"407549","messageId":"CABPp-BE0_u=TAJKUGUTLrZbg8vk2MFYmfscT_+MgYwAS2rV8Aw@mail.gmail.com","threadId":"54403","inReplyTo":"CAPc5daU_aw=yWPKHG1XGt4X5A1AkEXeLfCmHqqVfZ+FzBD7nxg@mail.gmail.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-14T19:41:00Z","receivedAt":"2020-10-14T19:41:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Oct 14, 2020 at 12:24 PM Junio C Hamano wrote:\n>\n> > > If you define the pattern is not regexp but is glob, you can use\n> > > case/esac to do this without any forking.\n> >\n> > Yes, that would probably be OK for most purposes, though I admit my real\n> > love for regex support is the ability to use \".\" instead of space to\n> > avoid quoting arguments. ;)\n>\n> I use \"?\" for the same purpose for globs. For things that are casual,\n> I find that\n> it tends to make the end-user (read: my) experience simpler to use globs than\n> to use regexp, largely for your \".*\" vs \"*\" reasons.\n\nOh, I thought you were arguing for globs over regexes here just due to\nperformance reasons. Most of my uses of this feature so far are just\nsubstring matches, so either globs or regexes would be fine for what\nI've done so far. Mostly, I was leaning towards regexes because I\nfigured they were a bit more flexible and perhaps there might arise a\nusecase where that would be handy (plus Peff seemed to like them).\nBut if you find the normal usecase easier with globs, maybe I should\nlet you and Peff argue it out and then I'll resubmit with the\npreferred solution?\n"},{"id":"407550","messageId":"20201014194926.GB365553@coredump.intra.peff.net","threadId":"54403","inReplyTo":"CABPp-BE0_u=TAJKUGUTLrZbg8vk2MFYmfscT_+MgYwAS2rV8Aw@mail.gmail.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-14T19:49:26Z","receivedAt":"2020-10-14T19:49:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 14, 2020 at 12:41:00PM -0700, Elijah Newren wrote:\n\n> > I use \"?\" for the same purpose for globs. For things that are casual,\n> > I find that\n> > it tends to make the end-user (read: my) experience simpler to use globs than\n> > to use regexp, largely for your \".*\" vs \"*\" reasons.\n> \n> Oh, I thought you were arguing for globs over regexes here just due to\n> performance reasons. Most of my uses of this feature so far are just\n> substring matches, so either globs or regexes would be fine for what\n> I've done so far. Mostly, I was leaning towards regexes because I\n> figured they were a bit more flexible and perhaps there might arise a\n> usecase where that would be handy (plus Peff seemed to like them).\n> But if you find the normal usecase easier with globs, maybe I should\n> let you and Peff argue it out and then I'll resubmit with the\n> preferred solution?\n\nThinking on it more, globs are probably a more natural fit here.\nEspecially if we do eventually allow selection of scripts by name, too\n(e.g., for GIT_SKIP_TESTS), matching those filenames with a glob would\nmatch what happens on the command-line.\n\n-Peff\n"},{"id":"407571","messageId":"pull.878.v3.git.git.1602710025.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v2.git.git.1602616786.gitgitgadget@gmail.com","subject":"[PATCH v3 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-14T21:13:42Z","receivedAt":"2020-10-15T01:59:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series enables me to change\n\n ./t9999-my-test.sh --run=1-3,5,17,19\n\ninto\n\n ./t9999-my-test.sh --run=setup,rename,symlink\n\nand have it pick out tests with \"setup\", \"rename\", or \"symlink\" in their\ndescription and run those. Saves me a lot of time, especially since numbers\nfor tests aren't readily accessible. The easiest way for me to get the\nnumbers corresponding to the tests I want to run, is to run all the tests\nand look at the output to match up the descriptions with their numbers --\nthus defeating the point of selecting just a subset of the tests to run in\nthe first place.\n\nChanges since v2:\n\n * Use a glob rather than a regex for selecting tests\n * Touched up the commit message in patch 3 to mention that the TAP output\n line is still included for skipped tests and already included the word\n \"skip\" to highlight the fact that the test was skipped.\n * Fixed a bug where -run=\"!rename\" would run all tests instead of skipping\n the ones with 'rename' in the description\n\nElijah Newren (3):\n test-lib: allow selecting tests by substring/glob with --run\n t6006, t6012: adjust tests to use 'setup' instead of synonyms\n test-lib: reduce verbosity of skipped tests\n\n t/README | 29 ++++++++++++++++---------\n t/t0000-basic.sh | 41 ++++++++++++++++++++++--------------\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n t/test-lib.sh | 17 +++++++++------\n 5 files changed, 56 insertions(+), 35 deletions(-)\n\n\nbase-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v3\nPull-Request: https://github.com/git/git/pull/878\n\nRange-diff vs v2:\n\n 1: 41e2528e83 ! 1: 9c8b6a1a94 test-lib: allow selecting tests by substring/regex with --run\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - test-lib: allow selecting tests by substring/regex with --run\n + test-lib: allow selecting tests by substring/glob with --run\n \n Many of our test scripts have several \"setup\" tests. It's a lot easier\n to say\n @@ t/README: For an individual test suite --run could be used to specify that\n -Optional prefix of '!' means that the test or a range of tests\n -should be excluded from the run.\n +The argument for --run, , is a list of description\n -+substrings or regexes or individual test numbers or ranges with an\n ++substrings or globs or individual test numbers or ranges with an\n +optional negation prefix (of '!') that define what tests in a test\n +suite to include (or exclude, if negated) in the run. A range is two\n +numbers separated with a dash and matches a range of tests with both\n @@ t/README: test in the test suite except from 7 up to 11:\n \n +Sometimes there may be multiple tests with e.g. \"setup\" in their name\n +that are needed and rather than figuring out the number for all of them\n -+we can just use \"setup\" as a substring/regex to match against the test\n ++we can just use \"setup\" as a substring/glob to match against the test\n +description:\n +\n + $ sh ./t0050-filesystem.sh --run=setup,9-11\n @@ t/test-lib.sh: match_test_selector_list () {\n -\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n -\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n -\t\t\t\t\texit 1\n -+\t\t\t\t\techo \"$title\" | grep -q \"$selector\" && return\n ++\t\t\t\t\tcase \"$title\" in *${selector}*)\n ++\t\t\t\t\t\tinclude=$positive\n ++\t\t\t\t\t\t;;\n ++\t\t\t\t\tesac\n +\t\t\t\t\tcontinue\n \t\t\t\tfi\n \t\tesac\n 2: 8f1a4420f4 = 2: 6479d73603 t6006, t6012: adjust tests to use 'setup' instead of synonyms\n 3: 85a4ca164a ! 3: 24f6febde6 test-lib: reduce verbosity of skipped tests\n @@ Commit message\n When using the --run flag to run just two or three tests from a test\n file which contains several dozen tests, having every skipped test print\n out dozens of lines of output for the test code for that skipped test\n - adds up to hundreds or thousands of lines of irrelevant output that make\n - it very hard to fish out the relevant results you were looking for.\n - Simplify the output for skipped tests down to just showing the one-line\n - descriptions.\n + (in addition to the TAP output line) adds up to hundreds or thousands of\n + lines of irrelevant output that make it very hard to fish out the\n + relevant results you were looking for. Simplify the output for skipped\n + tests to remove this extra output, leaving only the TAP output line\n + (i.e. the line reading \"ok # skip \", which\n + already mentions that the test was \"skip\"ped).\n \n Signed-off-by: Elijah Newren \n \n\n-- \ngitgitgadget\n"},{"id":"407572","messageId":"6479d73603d538d2b97d209eec5c5e570fde1269.1602710025.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v3.git.git.1602710025.gitgitgadget@gmail.com","subject":"[PATCH v3 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-14T21:13:44Z","receivedAt":"2020-10-15T01:59:44Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWith the new ability to pass --run=setup to select which tests to run,\nit is more convenient if tests use the term \"setup\" instead of synonyms\nlike 'prepare' or 'rebuild'. There are undoubtedly many other tests in\nour testsuite that could be changed over too, these are just a couple\nthat I ran into.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex bc95da8a5f..99a1eaf332 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -339,7 +339,7 @@ commit $head1\n .. (hinzugef${added_utf8_part_iso88591}gt) foo\n EOF\n \n-test_expect_success 'prepare expected messages (for test %b)' '\n+test_expect_success 'setup expected messages (for test %b)' '\n \tcat <<-EOF >expected.utf-8 &&\n \tcommit $head3\n \tThis commit message is much longer than the others,\ndiff --git a/t/t6012-rev-list-simplify.sh b/t/t6012-rev-list-simplify.sh\nindex b6fa43ace0..7254060240 100755\n--- a/t/t6012-rev-list-simplify.sh\n+++ b/t/t6012-rev-list-simplify.sh\n@@ -168,7 +168,7 @@ test_expect_success '--full-diff is not affected by --parents' '\n #\n # This example is explained in Documentation/rev-list-options.txt\n \n-test_expect_success 'rebuild repo' '\n+test_expect_success 'setup rebuild repo' '\n \trm -rf .git * &&\n \tgit init &&\n \tgit switch -c main &&\n-- \ngitgitgadget\n\n"},{"id":"407573","messageId":"24f6febde649d8fbe847cbba95171e6fec923e2a.1602710025.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v3.git.git.1602710025.gitgitgadget@gmail.com","subject":"[PATCH v3 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-14T21:13:45Z","receivedAt":"2020-10-15T01:59:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen using the --run flag to run just two or three tests from a test\nfile which contains several dozen tests, having every skipped test print\nout dozens of lines of output for the test code for that skipped test\n(in addition to the TAP output line) adds up to hundreds or thousands of\nlines of irrelevant output that make it very hard to fish out the\nrelevant results you were looking for. Simplify the output for skipped\ntests to remove this extra output, leaving only the TAP output line\n(i.e. the line reading \"ok # skip \", which\nalready mentions that the test was \"skip\"ped).\n\nSigned-off-by: Elijah Newren \n---\n t/test-lib.sh | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a040d54a76..3f8c404335 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1062,7 +1062,6 @@ test_skip () {\n \t\t\t\t\" \"\n \t\tfi\n \n-\t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n-- \ngitgitgadget\n"},{"id":"407574","messageId":"9c8b6a1a943261635fa09bed22ae36e368686f15.1602710025.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v3.git.git.1602710025.gitgitgadget@gmail.com","subject":"[PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-14T21:13:43Z","receivedAt":"2020-10-15T02:00:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany of our test scripts have several \"setup\" tests. It's a lot easier\nto say\n\n ./t0050-filesystem.sh --run=setup,9\n\nin order to run all the setup tests as well as test #9, than it is to\ntrack down what all the setup tests are and enter all their numbers in\nthe list. Also, I often find myself wanting to run just one or a couple\ntests from the test file, but I don't know the numbering of any of the\ntests -- to get it I either have to first run the whole test file (or\nstart counting by hand or figure out some other clever but non-obvious\ntricks). It's really convenient to be able to just look at the test\ndescription(s) and then run\n\n ./t6416-recursive-corner-cases.sh --run=symlink\n\nor\n\n ./t6402-merge-rename.sh --run='setup,unnecessary update'\n\nAdd such an ability to test selection which relies on merely matching\nagainst the test description.\n\nSigned-off-by: Elijah Newren \n---\n t/README | 29 +++++++++++++++++++----------\n t/t0000-basic.sh | 41 +++++++++++++++++++++++++----------------\n t/test-lib.sh | 16 ++++++++++------\n 3 files changed, 54 insertions(+), 32 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 2adaf7c2d2..0a8b60b5c7 100644\n--- a/t/README\n+++ b/t/README\n@@ -258,16 +258,13 @@ For an individual test suite --run could be used to specify that\n only some tests should be run or that some tests should be\n excluded from a run.\n \n-The argument for --run is a list of individual test numbers or\n-ranges with an optional negation prefix that define what tests in\n-a test suite to include in the run. A range is two numbers\n-separated with a dash and matches a range of tests with both ends\n-been included. You may omit the first or the second number to\n-mean \"from the first test\" or \"up to the very last test\"\n-respectively.\n-\n-Optional prefix of '!' means that the test or a range of tests\n-should be excluded from the run.\n+The argument for --run, , is a list of description\n+substrings or globs or individual test numbers or ranges with an\n+optional negation prefix (of '!') that define what tests in a test\n+suite to include (or exclude, if negated) in the run. A range is two\n+numbers separated with a dash and matches a range of tests with both\n+ends been included. You may omit the first or the second number to\n+mean \"from the first test\" or \"up to the very last test\" respectively.\n \n If --run starts with an unprefixed number or range the initial\n set of tests to run is empty. If the first item starts with '!'\n@@ -317,6 +314,18 @@ test in the test suite except from 7 up to 11:\n \n $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n \n+Sometimes there may be multiple tests with e.g. \"setup\" in their name\n+that are needed and rather than figuring out the number for all of them\n+we can just use \"setup\" as a substring/glob to match against the test\n+description:\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,9-11\n+\n+or one could select both the setup tests and the rename ones (assuming all\n+relevant tests had those words in their descriptions):\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,rename\n+\n Some tests in a test suite rely on the previous tests performing\n certain actions, specifically some tests are designated as\n \"setup\" test, so you cannot _arbitrarily_ disable one test and\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 923281af93..07105b2078 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' \"\n \tEOF\n \"\n \n-test_expect_success '--run invalid range start' \"\n+test_expect_success '--run substring selector' \"\n+\trun_sub_test_lib_test run-substring-selector \\\n+\t\t'--run empty selectors' \\\n+\t\t--run='relevant' <<-\\\\EOF &&\n+\ttest_expect_success \\\"relevant test\\\" 'true'\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"other test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-substring-selector <<-\\\\EOF\n+\t> ok 1 - relevant test\n+\t> ok 2 # skip other test #1 (--run)\n+\t> ok 3 # skip other test #2 (--run)\n+\t> ok 4 # skip other test #3 (--run)\n+\t> ok 5 # skip other test #4 (--run)\n+\t> ok 6 # skip other test #5 (--run)\n+\t> ok 7 # skip other test #6 (--run)\n+\t> # passed all 7 test(s)\n+\t> 1..7\n+\tEOF\n+\"\n+\n+test_expect_success '--run keyword selection' \"\n \trun_sub_test_lib_test_err run-inv-range-start \\\n \t\t'--run invalid range start' \\\n \t\t--run='a-5' <<-\\\\EOF &&\n@@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' \"\n \tEOF_ERR\n \"\n \n-test_expect_success '--run invalid selector' \"\n-\trun_sub_test_lib_test_err run-inv-selector \\\n-\t\t'--run invalid selector' \\\n-\t\t--run='1?' <<-\\\\EOF &&\n-\ttest_expect_success \\\"passing test #1\\\" 'true'\n-\ttest_done\n-\tEOF\n-\tcheck_sub_test_lib_test_err run-inv-selector \\\n-\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n-\t> FATAL: Unexpected exit with code 1\n-\tEOF_OUT\n-\t> error: --run: invalid non-numeric in test selector: '1?'\n-\tEOF_ERR\n-\"\n-\n \n test_set_prereq HAVEIT\n haveit=no\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..a040d54a76 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -769,6 +769,8 @@ match_pattern_list () {\n }\n \n match_test_selector_list () {\n+\toperation=\"$1\"\n+\tshift\n \ttitle=\"$1\"\n \tshift\n \targ=\"$1\"\n@@ -805,13 +807,13 @@ match_test_selector_list () {\n \t\t\t*-*)\n \t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"start: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n \t\t\t\tif expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"end: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n@@ -819,9 +821,11 @@ match_test_selector_list () {\n \t\t\t*)\n \t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n-\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n-\t\t\t\t\texit 1\n+\t\t\t\t\tcase \"$title\" in *${selector}*)\n+\t\t\t\t\t\tinclude=$positive\n+\t\t\t\t\t\t;;\n+\t\t\t\t\tesac\n+\t\t\t\t\tcontinue\n \t\t\t\tfi\n \t\tesac\n \n@@ -1031,7 +1035,7 @@ test_skip () {\n \t\tskipped_reason=\"GIT_SKIP_TESTS\"\n \tfi\n \tif test -z \"$to_skip\" && test -n \"$run_list\" &&\n-\t ! match_test_selector_list '--run' $test_count \"$run_list\"\n+\t ! match_test_selector_list '--run' \"$1\" $test_count \"$run_list\"\n \tthen\n \t\tto_skip=t\n \t\tskipped_reason=\"--run\"\n-- \ngitgitgadget\n\n"},{"id":"407595","messageId":"nycvar.QRO.7.76.6.2010151347240.56@tvgsbejvaqbjf.bet","threadId":"54403","inReplyTo":"fba0a914-1818-8a24-41b0-35740a7c2d2d@gmail.com","subject":"Re: [PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-10-15T11:48:54Z","receivedAt":"2020-10-15T11:49:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 13 Oct 2020, Phillip Wood wrote:\n\n> I'm worryingly excited by this series\n\nFWIW I am in the same boat.\n\n> On 13/10/2020 00:26, Elijah Newren via GitGitGadget wrote:\n> > This patch series enables me to change\n> >\n> > ./t9999-my-test.sh --run=1-3,5,17,19\n> >\n> > into\n> >\n> > ./t9999-my-test.sh --run=setup,rename,symlink\n> >\n> > and have it pick out tests with \"setup\", \"rename\", or \"symlink\" in their\n> > description and run those. Saves me a lot of time, especially since numbers\n> > for tests aren't readily accessible. The easiest way for me to get the\n> > numbers corresponding to the tests I want to run, is to run all the tests\n> > and look at the output to match up the descriptions with their numbers --\n> > thus defeating the point of selecting just a subset of the tests to run in\n> > the first place.\n>\n> Yes it's a real pain to have to run all the tests to find the numbers of the\n> ones you actually wanted to run. I like patch 3 as well, without it if you're\n> only running a couple of tests it can be hard to find then amongst all the\n> skipped tests. I had a quick read of the patches and they looked fine to me.\n\nI learned the trick to run with `--run=0` just to figure out the numbers\nin a recent Outreachy project, I think it was Slavica who taught me.\n\nBut yes, without patch 3, the output looks awfully verbose and you have to\nscroll back an insane amount of lines for long test scripts.\n\nGiddy,\nDscho\n"},{"id":"407609","messageId":"4890563a-4860-5958-8984-2d784b2a481a@gmail.com","threadId":"54403","inReplyTo":"nycvar.QRO.7.76.6.2010151347240.56@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-10-15T13:42:23Z","receivedAt":"2020-10-15T13:42:29Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 15/10/2020 12:48, Johannes Schindelin wrote:\n> Hi,\n> \n> On Tue, 13 Oct 2020, Phillip Wood wrote:\n> \n>> I'm worryingly excited by this series\n> \n> FWIW I am in the same boat.\n\nI'm glad it's not just me!\n\n>> On 13/10/2020 00:26, Elijah Newren via GitGitGadget wrote:\n>>> This patch series enables me to change\n>>>\n>>> ./t9999-my-test.sh --run=1-3,5,17,19\n>>>\n>>> into\n>>>\n>>> ./t9999-my-test.sh --run=setup,rename,symlink\n>>>\n>>> and have it pick out tests with \"setup\", \"rename\", or \"symlink\" in their\n>>> description and run those. Saves me a lot of time, especially since numbers\n>>> for tests aren't readily accessible. The easiest way for me to get the\n>>> numbers corresponding to the tests I want to run, is to run all the tests\n>>> and look at the output to match up the descriptions with their numbers --\n>>> thus defeating the point of selecting just a subset of the tests to run in\n>>> the first place.\n>>\n>> Yes it's a real pain to have to run all the tests to find the numbers of the\n>> ones you actually wanted to run. I like patch 3 as well, without it if you're\n>> only running a couple of tests it can be hard to find then amongst all the\n>> skipped tests. I had a quick read of the patches and they looked fine to me.\n> \n> I learned the trick to run with `--run=0` just to figure out the numbers\n> in a recent Outreachy project, I think it was Slavica who taught me.\n\nThat's a handy tip, thanks (though hopefully I soon wont need to use it)\n\nPhillip\n\n> But yes, without patch 3, the output looks awfully verbose and you have to\n> scroll back an insane amount of lines for long test scripts.\n> \n> Giddy,\n> Dscho\n> \n"},{"id":"407614","messageId":"xmqqeelziip2.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"CABPp-BE0_u=TAJKUGUTLrZbg8vk2MFYmfscT_+MgYwAS2rV8Aw@mail.gmail.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-15T15:59:21Z","receivedAt":"2020-10-15T15:59:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> On Wed, Oct 14, 2020 at 12:24 PM Junio C Hamano wrote:\n>>\n>> > > If you define the pattern is not regexp but is glob, you can use\n>> > > case/esac to do this without any forking.\n>> >\n>> > Yes, that would probably be OK for most purposes, though I admit my real\n>> > love for regex support is the ability to use \".\" instead of space to\n>> > avoid quoting arguments. ;)\n>>\n>> I use \"?\" for the same purpose for globs. For things that are casual,\n>> I find that\n>> it tends to make the end-user (read: my) experience simpler to use globs than\n>> to use regexp, largely for your \".*\" vs \"*\" reasons.\n>\n> Oh, I thought you were arguing for globs over regexes here just due to\n> performance reasons.\n\nHeh, but no, not me.\n\nYou and Peff were the ones who were talking about performance by\ncounting forks. I more often than others come from usability's\npoint of view. When we are not dealing with uncontrollable and\nunbounded possiblity of end-user generated contents, I find it\neasier to forego the power and flexibility of regexp and instead\nsettle on simpler globs---matching against the test titles is a\ngood example use case, I would imagine.\n\nBut if we are exposing regexp to those who run and debug test\nscripts already, I am perfectly fine with using regexp with 'expr'.\n\nThanks.\n\n\n"},{"id":"407616","messageId":"xmqqa6wnii80.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"20201014194926.GB365553@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-15T16:09:35Z","receivedAt":"2020-10-15T16:09:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Thinking on it more, globs are probably a more natural fit here.\n> Especially if we do eventually allow selection of scripts by name, too\n> (e.g., for GIT_SKIP_TESTS), matching those filenames with a glob would\n> match what happens on the command-line.\n\nAh, I didn't think of \"match with filenames\" but yes that is a good\ncomplement to our \"match with test titles\" to be consistent with.\n\nThanks.\n"},{"id":"407617","messageId":"xmqq5z7bii6r.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"CABPp-BGBG=wh41BrvFZdzsN6biUPvuDkrA9EZURM4GKLx8kC0A@mail.gmail.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-15T16:10:20Z","receivedAt":"2020-10-15T16:10:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> On Wed, Oct 14, 2020 at 10:57 AM Junio C Hamano wrote:\n>>\n>> > are you suggesting dropping the regex handling and limit it to\n>> > substring matching? In either case, does using expr save us anything\n>> > (isn't expr a shell command)?\n>>\n>> I had something along this line in mind, not to do a regex but a glob.\n>>\n>> case \"$title\" in $selector) found=1 ;; esac\n>\n> Interesting. Since it needs to handle substring searching (e.g.\n> ./test-script.sh --run=setup,rename), I think this would need to be\n> tweaked to be\n> case \"$title\" in *${selector}*) include=$positive ;; esac\n\nYup, of course we need to un-anchor with surrounding asterisks.\nThanks.\n"},{"id":"407620","messageId":"xmqq1rhzihqa.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"20201014170413.GB21687@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-15T16:20:13Z","receivedAt":"2020-10-15T16:20:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Tue, Oct 13, 2020 at 07:19:44PM +0000, Elijah Newren via GitGitGadget wrote:\n>\n>> Many of our test scripts have several \"setup\" tests. It's a lot easier\n>> to say\n>> \n>> ./t0050-filesystem.sh --run=setup,9\n>\n> I like this direction very well.\n>\n> There was a small discussion recently that we might be better off\n> dropping test script numbers entirely,...\n\nI think I missed that one. A pointer, if you have one handy?\n\nThanks.\n"},{"id":"407670","messageId":"20201015194648.GB1490964@coredump.intra.peff.net","threadId":"54403","inReplyTo":"xmqq1rhzihqa.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-15T19:46:48Z","receivedAt":"2020-10-15T19:46:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 15, 2020 at 09:20:13AM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > On Tue, Oct 13, 2020 at 07:19:44PM +0000, Elijah Newren via GitGitGadget wrote:\n> >\n> >> Many of our test scripts have several \"setup\" tests. It's a lot easier\n> >> to say\n> >> \n> >> ./t0050-filesystem.sh --run=setup,9\n> >\n> > I like this direction very well.\n> >\n> > There was a small discussion recently that we might be better off\n> > dropping test script numbers entirely,...\n> \n> I think I missed that one. A pointer, if you have one handy?\n\nThe sub-thread between me and Jonathan starting here:\n\n https://lore.kernel.org/git/20201005082448.GB2862927@coredump.intra.peff.net/\n\nbut specifically:\n\n https://lore.kernel.org/git/20201005084946.GE2862927@coredump.intra.peff.net/\n\n-Peff\n"},{"id":"407671","messageId":"20201015194823.GC1490964@coredump.intra.peff.net","threadId":"54403","inReplyTo":"pull.878.v3.git.git.1602710025.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-15T19:48:23Z","receivedAt":"2020-10-15T19:48:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 14, 2020 at 09:13:42PM +0000, Elijah Newren via GitGitGadget wrote:\n\n> Changes since v2:\n> \n> * Use a glob rather than a regex for selecting tests\n> * Touched up the commit message in patch 3 to mention that the TAP output\n> line is still included for skipped tests and already included the word\n> \"skip\" to highlight the fact that the test was skipped.\n> * Fixed a bug where -run=\"!rename\" would run all tests instead of skipping\n> the ones with 'rename' in the description\n\nThanks, this one addresses all my nits. I'm looking forward to using it\nin practice. :)\n\n-Peff\n"},{"id":"407673","messageId":"xmqq7drrfe15.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"20201015194648.GB1490964@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-15T20:08:22Z","receivedAt":"2020-10-15T20:08:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Thu, Oct 15, 2020 at 09:20:13AM -0700, Junio C Hamano wrote:\n>\n>> Jeff King writes:\n>> \n>> > On Tue, Oct 13, 2020 at 07:19:44PM +0000, Elijah Newren via GitGitGadget wrote:\n>> >\n>> >> Many of our test scripts have several \"setup\" tests. It's a lot easier\n>> >> to say\n>> >> \n>> >> ./t0050-filesystem.sh --run=setup,9\n>> >\n>> > I like this direction very well.\n>> >\n>> > There was a small discussion recently that we might be better off\n>> > dropping test script numbers entirely,...\n>> \n>> I think I missed that one. A pointer, if you have one handy?\n>\n> The sub-thread between me and Jonathan starting here:\n>\n> https://lore.kernel.org/git/20201005082448.GB2862927@coredump.intra.peff.net/\n>\n> but specifically:\n>\n> https://lore.kernel.org/git/20201005084946.GE2862927@coredump.intra.peff.net/\n\nAh, I see. I actually do use \"git ls-files t/ | grep ...\" to look\nfor tests that are relevant to the issue I have at hand quite often,\nso unlike what Jonathan said in the thread, having a good name does\nmatter to me.\n\nAs far as I can tell, the numbers in the test names serve only two\npurposes.\n\nOne is as a unique key to avoid colliding in the test result\naggrevation database, and the other is as a unique key to use in\nGIT_SKIP_TESTS (which in turn is used by the Meta/Make wrapper I\nuse, found on the todo branch, like 'Meta/Make --test=0050,1400\ntest').\n\nI would be heavily inconvenienced if we decide to remove numbers\nbecuase it would rob the latter use case from me, but I'd survive if\nwe just are going to lift the requirement that numbers must be\nunique. I may end up running irrelevant 0050 and 1400 when the\ntests I really want to run are the other 0050 and 1400 with\n\"--test=0050,1400\", but when I am trying to run only 2 among 900+\nscripts, running 2 extra ones I didn't have to run only because\ntheir prefix collide is still much better and tolerable.\n\nThanks.\n"},{"id":"407693","messageId":"20201016003847.GB2932796@coredump.intra.peff.net","threadId":"54403","inReplyTo":"xmqq7drrfe15.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-16T00:38:47Z","receivedAt":"2020-10-16T00:38:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 15, 2020 at 01:08:22PM -0700, Junio C Hamano wrote:\n\n> > The sub-thread between me and Jonathan starting here:\n> >\n> > https://lore.kernel.org/git/20201005082448.GB2862927@coredump.intra.peff.net/\n> >\n> > but specifically:\n> >\n> > https://lore.kernel.org/git/20201005084946.GE2862927@coredump.intra.peff.net/\n> \n> Ah, I see. I actually do use \"git ls-files t/ | grep ...\" to look\n> for tests that are relevant to the issue I have at hand quite often,\n> so unlike what Jonathan said in the thread, having a good name does\n> matter to me.\n\nI'm not sure where he suggests worse names. I'd think if anything we'd\nhave better names, because they'd be even more meaningful (if people\nstart using them for test selectors). FWIW, I also grep like that when\nlooking for scripts.\n\n> As far as I can tell, the numbers in the test names serve only two\n> purposes.\n> \n> One is as a unique key to avoid colliding in the test result\n> aggrevation database, and the other is as a unique key to use in\n> GIT_SKIP_TESTS (which in turn is used by the Meta/Make wrapper I\n> use, found on the todo branch, like 'Meta/Make --test=0050,1400\n> test').\n\nYeah, that sounds right. We also frequently talk about \"t5601\" in\nhuman-readable messages. I often use that to either tab-complete or run\n\"./t5601-*\". So they do serve as short unique identifiers there. You\ncould tell me \"t-foo-bar-baz\" and that would similarly be unique, but\nit's not as short. :)\n\nThey do serve as unique keys in places like test-results/, but really\nthe whole test name is the key. Most of them have unique text\ndescriptions. If I do:\n\n git ls-files t/ |\n perl -lne '/t[0-9]{4}-(.*).sh$/ and print \"$1\"' |\n sort | uniq -d\n\nthat only yields two duplicates (and we'd probably be better off giving\nthem unique names anyway).\n\n> I would be heavily inconvenienced if we decide to remove numbers\n> becuase it would rob the latter use case from me, but I'd survive if\n> we just are going to lift the requirement that numbers must be\n> unique.\n\nMy plan is that you'd be able to say:\n\n Meta/make --test=filesystem,update-ref\n\nfor t0050 and t1400 respectively. Or more interesting things like\n\"git-svn-*\" to skip all of the t91xx tests (or \"git-svn/*\" if we split\nthem by directories).\n\n> I may end up running irrelevant 0050 and 1400 when the\n> tests I really want to run are the other 0050 and 1400 with\n> \"--test=0050,1400\", but when I am trying to run only 2 among 900+\n> scripts, running 2 extra ones I didn't have to run only because\n> their prefix collide is still much better and tolerable.\n\nI think you could already do this in general. The only thing standing in\nthe way is test-lint-duplicates. I added that because I thought it would\nbe better to catch the duplicates early rather than get confused by them\nlater when you can't uniquely identify a test. But in practice I do\nfeel like I spend more time de-duping tests when I rebase or merge\n(because my branch raced with somebody else allocating the test number)\nthan I ever did dealing with unexpectedly skipped tests.\n\nI definitely don't want to make anybody's life harder. But if the\nnumbers aren't unique and the text connected to them is, then it seems\nlike we should just use the text primarily.\n\n-Peff\n"},{"id":"407712","messageId":"2b757512-793d-a6e0-0a50-368061e122dd@gmail.com","threadId":"54403","inReplyTo":"9c8b6a1a943261635fa09bed22ae36e368686f15.1602710025.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-10-16T11:40:58Z","receivedAt":"2020-10-16T11:41:07Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Elijah\n\nOn 14/10/2020 22:13, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Many of our test scripts have several \"setup\" tests. It's a lot easier\n> to say\n> \n> ./t0050-filesystem.sh --run=setup,9\n> \n> in order to run all the setup tests as well as test #9, than it is to\n> track down what all the setup tests are and enter all their numbers in\n> the list. Also, I often find myself wanting to run just one or a couple\n> tests from the test file, but I don't know the numbering of any of the\n> tests -- to get it I either have to first run the whole test file (or\n> start counting by hand or figure out some other clever but non-obvious\n> tricks). It's really convenient to be able to just look at the test\n> description(s) and then run\n> \n> ./t6416-recursive-corner-cases.sh --run=symlink\n> \n> or\n> \n> ./t6402-merge-rename.sh --run='setup,unnecessary update'\n\nThe beginning of match_test_selector_list() looks like\n\nmatch_test_selector_list () {\n\ttitle=\"$1\"\n\tshift\n\targ=\"$1\"\n\tshift\n\ttest -z \"$1\" && return 0\n\n\t# Both commas and whitespace are accepted as separators.\n\tOLDIFS=$IFS\n\tIFS=' \t,'\n\tset -- $1\n\tIFS=$OLDIFS\n\n\t# If the first selector is negative we include by default.\n\tinclude=\n\tcase \"$1\" in\n\t\t!*) include=t ;;\n\tesac\n\n\tfor selector\n\tdo\n\nIf I'm reading it correctly the selectors are split on commas and \nwhitespace which would mean we cannot match on \"unnecessary update\". I \nthink we definitely want the ability to include whitespace in the \nselectors in order to be able to narrow down the tests that are run. I'm \nnot sure that there is much value in splitting numbers on whitespace as \nit would mean the user has to quote them on the command line so we can \nprobably just do 'IFS=,'. We'd also need to keep IFS as ',' in the case \nstatement you add below as well rather than restoring it straight after \nthe 'set' statement above.\n\nBest Wishes\n\nPhillip\n\n> Add such an ability to test selection which relies on merely matching\n> against the test description.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> t/README | 29 +++++++++++++++++++----------\n> t/t0000-basic.sh | 41 +++++++++++++++++++++++++----------------\n> t/test-lib.sh | 16 ++++++++++------\n> 3 files changed, 54 insertions(+), 32 deletions(-)\n> \n> diff --git a/t/README b/t/README\n> index 2adaf7c2d2..0a8b60b5c7 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -258,16 +258,13 @@ For an individual test suite --run could be used to specify that\n> only some tests should be run or that some tests should be\n> excluded from a run.\n> \n> -The argument for --run is a list of individual test numbers or\n> -ranges with an optional negation prefix that define what tests in\n> -a test suite to include in the run. A range is two numbers\n> -separated with a dash and matches a range of tests with both ends\n> -been included. You may omit the first or the second number to\n> -mean \"from the first test\" or \"up to the very last test\"\n> -respectively.\n> -\n> -Optional prefix of '!' means that the test or a range of tests\n> -should be excluded from the run.\n> +The argument for --run, , is a list of description\n> +substrings or globs or individual test numbers or ranges with an\n> +optional negation prefix (of '!') that define what tests in a test\n> +suite to include (or exclude, if negated) in the run. A range is two\n> +numbers separated with a dash and matches a range of tests with both\n> +ends been included. You may omit the first or the second number to\n> +mean \"from the first test\" or \"up to the very last test\" respectively.\n> \n> If --run starts with an unprefixed number or range the initial\n> set of tests to run is empty. If the first item starts with '!'\n> @@ -317,6 +314,18 @@ test in the test suite except from 7 up to 11:\n> \n> $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n> \n> +Sometimes there may be multiple tests with e.g. \"setup\" in their name\n> +that are needed and rather than figuring out the number for all of them\n> +we can just use \"setup\" as a substring/glob to match against the test\n> +description:\n> +\n> + $ sh ./t0050-filesystem.sh --run=setup,9-11\n> +\n> +or one could select both the setup tests and the rename ones (assuming all\n> +relevant tests had those words in their descriptions):\n> +\n> + $ sh ./t0050-filesystem.sh --run=setup,rename\n> +\n> Some tests in a test suite rely on the previous tests performing\n> certain actions, specifically some tests are designated as\n> \"setup\" test, so you cannot _arbitrarily_ disable one test and\n> diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\n> index 923281af93..07105b2078 100755\n> --- a/t/t0000-basic.sh\n> +++ b/t/t0000-basic.sh\n> @@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' \"\n> \tEOF\n> \"\n> \n> -test_expect_success '--run invalid range start' \"\n> +test_expect_success '--run substring selector' \"\n> +\trun_sub_test_lib_test run-substring-selector \\\n> +\t\t'--run empty selectors' \\\n> +\t\t--run='relevant' <<-\\\\EOF &&\n> +\ttest_expect_success \\\"relevant test\\\" 'true'\n> +\tfor i in 1 2 3 4 5 6\n> +\tdo\n> +\t\ttest_expect_success \\\"other test #\\$i\\\" 'true'\n> +\tdone\n> +\ttest_done\n> +\tEOF\n> +\tcheck_sub_test_lib_test run-substring-selector <<-\\\\EOF\n> +\t> ok 1 - relevant test\n> +\t> ok 2 # skip other test #1 (--run)\n> +\t> ok 3 # skip other test #2 (--run)\n> +\t> ok 4 # skip other test #3 (--run)\n> +\t> ok 5 # skip other test #4 (--run)\n> +\t> ok 6 # skip other test #5 (--run)\n> +\t> ok 7 # skip other test #6 (--run)\n> +\t> # passed all 7 test(s)\n> +\t> 1..7\n> +\tEOF\n> +\"\n> +\n> +test_expect_success '--run keyword selection' \"\n> \trun_sub_test_lib_test_err run-inv-range-start \\\n> \t\t'--run invalid range start' \\\n> \t\t--run='a-5' <<-\\\\EOF &&\n> @@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' \"\n> \tEOF_ERR\n> \"\n> \n> -test_expect_success '--run invalid selector' \"\n> -\trun_sub_test_lib_test_err run-inv-selector \\\n> -\t\t'--run invalid selector' \\\n> -\t\t--run='1?' <<-\\\\EOF &&\n> -\ttest_expect_success \\\"passing test #1\\\" 'true'\n> -\ttest_done\n> -\tEOF\n> -\tcheck_sub_test_lib_test_err run-inv-selector \\\n> -\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n> -\t> FATAL: Unexpected exit with code 1\n> -\tEOF_OUT\n> -\t> error: --run: invalid non-numeric in test selector: '1?'\n> -\tEOF_ERR\n> -\"\n> -\n> \n> test_set_prereq HAVEIT\n> haveit=no\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index ef31f40037..a040d54a76 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -769,6 +769,8 @@ match_pattern_list () {\n> }\n> \n> match_test_selector_list () {\n> +\toperation=\"$1\"\n> +\tshift\n> \ttitle=\"$1\"\n> \tshift\n> \targ=\"$1\"\n> @@ -805,13 +807,13 @@ match_test_selector_list () {\n> \t\t\t*-*)\n> \t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n> \t\t\t\tthen\n> -\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n> +\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n> \t\t\t\t\t\t\"start: '$orig_selector'\" >&2\n> \t\t\t\t\texit 1\n> \t\t\t\tfi\n> \t\t\t\tif expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n> \t\t\t\tthen\n> -\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n> +\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n> \t\t\t\t\t\t\"end: '$orig_selector'\" >&2\n> \t\t\t\t\texit 1\n> \t\t\t\tfi\n> @@ -819,9 +821,11 @@ match_test_selector_list () {\n> \t\t\t*)\n> \t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n> \t\t\t\tthen\n> -\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n> -\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n> -\t\t\t\t\texit 1\n> +\t\t\t\t\tcase \"$title\" in *${selector}*)\n> +\t\t\t\t\t\tinclude=$positive\n> +\t\t\t\t\t\t;;\n> +\t\t\t\t\tesac\n> +\t\t\t\t\tcontinue\n> \t\t\t\tfi\n> \t\tesac\n> \n> @@ -1031,7 +1035,7 @@ test_skip () {\n> \t\tskipped_reason=\"GIT_SKIP_TESTS\"\n> \tfi\n> \tif test -z \"$to_skip\" && test -n \"$run_list\" &&\n> -\t ! match_test_selector_list '--run' $test_count \"$run_list\"\n> +\t ! match_test_selector_list '--run' \"$1\" $test_count \"$run_list\"\n> \tthen\n> \t\tto_skip=t\n> \t\tskipped_reason=\"--run\"\n> \n"},{"id":"407723","messageId":"xmqq7drqdu2u.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"20201016003847.GB2932796@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-16T16:16:57Z","receivedAt":"2020-10-16T16:17:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> Ah, I see. I actually do use \"git ls-files t/ | grep ...\" to look\n>> for tests that are relevant to the issue I have at hand quite often,\n>> so unlike what Jonathan said in the thread, having a good name does\n>> matter to me.\n>\n> I'm not sure where he suggests worse names. I'd think if anything we'd\n> have better names, because they'd be even more meaningful (if people\n> start using them for test selectors). FWIW, I also grep like that when\n> looking for scripts.\n\nI didn't mean Jonathan suggested worse names. Unlike \"I don't tend\nto discover test scripts based on their filename\", which was what\nJonathan said, I do look for tests based on their filename, so\nhaving a good name matters (on the other hand, if you are the kind\nof person who does not look for them by name, the naming may not\nmatter to you).\n\n\n> My plan is that you'd be able to say:\n>\n> Meta/make --test=filesystem,update-ref\n>\n> for t0050 and t1400 respectively. Or more interesting things like\n> \"git-svn-*\" to skip all of the t91xx tests (or \"git-svn/*\" if we split\n> them by directories).\n\nMost of the time, I am focused on running only the selected few, not\nexcluding a few known-broken ones.\n\nFor example, with an round of integration, I know only two tests\nfail, and after I tried to fix the bug in the series, I want to\nretry these two tests and nothing else to see if the attempted fix\npatches the breakage up (of course I need to run the remainder of\nthe tests to ensure there is no regression, but that is just the\nmatter of running \"make test\" to run the whole thing again).\n\nAs long as it is known that \"filesystem\" and \"update-ref\" can serve\nas tokens to uniquely identify these two tests, it would be fine for\nmy purpose. But 0050 (under the rule that numbering must be unique)\nwould give me such an assurance much better without having to look\nat any other test file.\n\nThe word \"filesystem\"? Unless we have a rule that we can use each\nunique word in test names only once (which of course is impractical)\nI am not sure I can use it in place of 0050 without checking names\nof other tests first.\n\n> I definitely don't want to make anybody's life harder. But if the\n> numbers aren't unique and the text connected to them is, then it seems\n> like we should just use the text primarily.\n\nTrue in principle, but it is harder to come up with unique substring\nof text, knowing only that the whole string is unique.\n\nThanks.\n"},{"id":"407724","messageId":"xmqq362edtgi.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"xmqq7drqdu2u.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-16T16:30:21Z","receivedAt":"2020-10-16T16:30:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Jeff King writes:\n>\n>> My plan is that you'd be able to say:\n>>\n>> Meta/make --test=filesystem,update-ref\n>>\n>> for t0050 and t1400 respectively. Or more interesting things like\n>> \"git-svn-*\" to skip all of the t91xx tests (or \"git-svn/*\" if we split\n>> them by directories).\n> ...\n> The word \"filesystem\"? Unless we have a rule that we can use each\n> unique word in test names only once (which of course is impractical)\n> I am not sure I can use it in place of 0050 without checking names\n> of other tests first.\n>\n>> I definitely don't want to make anybody's life harder. But if the\n>> numbers aren't unique and the text connected to them is, then it seems\n>> like we should just use the text primarily.\n>\n> True in principle, but it is harder to come up with unique substring\n> of text, knowing only that the whole string is unique.\n\nI think there was a mismatch in assumptions and expectations. I\nthink your \"plan\" was to use the whole name (what comes after \\d+-\nin the current naming scheme, except for the .sh at the end), and I\nwas assuming a glob with implicit * on both ends would be used, i.e.\n\n $ ls -d t/*update-ref*\n\nwould give me 1404 (update-ref-errors) in addition to 1400 (update-ref).\n\nSo if the rule is to always spell out the full name if I wanted\nuniqueness, then it would work. In the cited usecase of knowing\nwhich two failed the first time, I should have the full filename\nof them available.\n\nThanks.\n"},{"id":"407729","messageId":"CABPp-BHkykWh8L_FYhLR1BCCpPDmc_2q+Tccg_yZ7W8ZHZ4WsA@mail.gmail.com","threadId":"54403","inReplyTo":"2b757512-793d-a6e0-0a50-368061e122dd@gmail.com","subject":"Re: [PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-16T17:27:56Z","receivedAt":"2020-10-16T17:28:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Phillip,\n\nOn Fri, Oct 16, 2020 at 4:41 AM Phillip Wood wrote:\n>\n> Hi Elijah\n>\n> On 14/10/2020 22:13, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Many of our test scripts have several \"setup\" tests. It's a lot easier\n> > to say\n> >\n> > ./t0050-filesystem.sh --run=setup,9\n> >\n> > in order to run all the setup tests as well as test #9, than it is to\n> > track down what all the setup tests are and enter all their numbers in\n> > the list. Also, I often find myself wanting to run just one or a couple\n> > tests from the test file, but I don't know the numbering of any of the\n> > tests -- to get it I either have to first run the whole test file (or\n> > start counting by hand or figure out some other clever but non-obvious\n> > tricks). It's really convenient to be able to just look at the test\n> > description(s) and then run\n> >\n> > ./t6416-recursive-corner-cases.sh --run=symlink\n> >\n> > or\n> >\n> > ./t6402-merge-rename.sh --run='setup,unnecessary update'\n>\n> The beginning of match_test_selector_list() looks like\n>\n> match_test_selector_list () {\n> title=\"$1\"\n> shift\n> arg=\"$1\"\n> shift\n> test -z \"$1\" && return 0\n>\n> # Both commas and whitespace are accepted as separators.\n> OLDIFS=$IFS\n> IFS=' ,'\n> set -- $1\n> IFS=$OLDIFS\n>\n> # If the first selector is negative we include by default.\n> include=\n> case \"$1\" in\n> !*) include=t ;;\n> esac\n>\n> for selector\n> do\n>\n> If I'm reading it correctly the selectors are split on commas and\n> whitespace which would mean we cannot match on \"unnecessary update\". I\n> think we definitely want the ability to include whitespace in the\n> selectors in order to be able to narrow down the tests that are run. I'm\n> not sure that there is much value in splitting numbers on whitespace as\n> it would mean the user has to quote them on the command line so we can\n> probably just do 'IFS=,'. We'd also need to keep IFS as ',' in the case\n> statement you add below as well rather than restoring it straight after\n> the 'set' statement above.\n\nGiven that t/README explicitly shows examples of space-separated lists\nof numbers, I'm worried we're breaking long-built expectations of\nother developers by changing IFS here. Perhaps I could instead add\nthe following paragraph to t/README:\n\nNote: The argument to --run is split on commas and whitespace into\nseparate strings, numbers, and ranges, and picks all tests that match\nany of individual selection criteria. If the substring you want to\nmatch from the description text includes a comma or space, use the\nglob character '?' instead. For example --run='unnecessary?update\ntiming' would match on all tests that match either the glob\n*unnecessary?update* or the glob *timing*.\n\nDoes that address your concern? The '?' will of course match on\ncharacters other than a space or comma, but the odds that it actually\nmatches anything other than what you want is pretty slim, so I suspect\nthat is good enough.\n"},{"id":"407734","messageId":"2fd1231c-9df3-623d-9ae1-0d66526db618@gmail.com","threadId":"54403","inReplyTo":"CABPp-BHkykWh8L_FYhLR1BCCpPDmc_2q+Tccg_yZ7W8ZHZ4WsA@mail.gmail.com","subject":"Re: [PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-10-16T18:24:15Z","receivedAt":"2020-10-16T18:24:20Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Elijah\n\nOn 16/10/2020 18:27, Elijah Newren wrote:\n> Hi Phillip,\n> \n> On Fri, Oct 16, 2020 at 4:41 AM Phillip Wood wrote:\n>>\n>> Hi Elijah\n>>\n>> On 14/10/2020 22:13, Elijah Newren via GitGitGadget wrote:\n>>> From: Elijah Newren \n>>>\n>>> Many of our test scripts have several \"setup\" tests. It's a lot easier\n>>> to say\n>>>\n>>> ./t0050-filesystem.sh --run=setup,9\n>>>\n>>> in order to run all the setup tests as well as test #9, than it is to\n>>> track down what all the setup tests are and enter all their numbers in\n>>> the list. Also, I often find myself wanting to run just one or a couple\n>>> tests from the test file, but I don't know the numbering of any of the\n>>> tests -- to get it I either have to first run the whole test file (or\n>>> start counting by hand or figure out some other clever but non-obvious\n>>> tricks). It's really convenient to be able to just look at the test\n>>> description(s) and then run\n>>>\n>>> ./t6416-recursive-corner-cases.sh --run=symlink\n>>>\n>>> or\n>>>\n>>> ./t6402-merge-rename.sh --run='setup,unnecessary update'\n>>\n>> The beginning of match_test_selector_list() looks like\n>>\n>> match_test_selector_list () {\n>> title=\"$1\"\n>> shift\n>> arg=\"$1\"\n>> shift\n>> test -z \"$1\" && return 0\n>>\n>> # Both commas and whitespace are accepted as separators.\n>> OLDIFS=$IFS\n>> IFS=' ,'\n>> set -- $1\n>> IFS=$OLDIFS\n>>\n>> # If the first selector is negative we include by default.\n>> include=\n>> case \"$1\" in\n>> !*) include=t ;;\n>> esac\n>>\n>> for selector\n>> do\n>>\n>> If I'm reading it correctly the selectors are split on commas and\n>> whitespace which would mean we cannot match on \"unnecessary update\". I\n>> think we definitely want the ability to include whitespace in the\n>> selectors in order to be able to narrow down the tests that are run. I'm\n>> not sure that there is much value in splitting numbers on whitespace as\n>> it would mean the user has to quote them on the command line so we can\n>> probably just do 'IFS=,'. We'd also need to keep IFS as ',' in the case\n>> statement you add below as well rather than restoring it straight after\n>> the 'set' statement above.\n> \n> Given that t/README explicitly shows examples of space-separated lists\n> of numbers,\n\nThat's a shame\n\n> I'm worried we're breaking long-built expectations of\n> other developers by changing IFS here. \n\nI agree\n\n> Perhaps I could instead add\n> the following paragraph to t/README:\n> \n> Note: The argument to --run is split on commas and whitespace into\n> separate strings, numbers, and ranges, and picks all tests that match\n> any of individual selection criteria. If the substring you want to\n> match from the description text includes a comma or space, use the\n> glob character '?' instead. For example --run='unnecessary?update\n> timing' would match on all tests that match either the glob\n> *unnecessary?update* or the glob *timing*.\n> \n> Does that address your concern? The '?' will of course match on\n> characters other than a space or comma, but the odds that it actually\n> matches anything other than what you want is pretty slim, so I suspect\n> that is good enough.\n\n'unnecessary?update' is pretty ugly but I agree false matches are \nunlikely to be a problem it practice. Your suggested paragraph looks \ngood to me\n\nThanks\n\nPhillip\n\n"},{"id":"407741","messageId":"8f5d4ad31804e456d134a09f11d82aed14f7141d.1602876532.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v4.git.git.1602876532.gitgitgadget@gmail.com","subject":"[PATCH v4 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-16T19:28:50Z","receivedAt":"2020-10-16T19:28:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany of our test scripts have several \"setup\" tests. It's a lot easier\nto say\n\n ./t0050-filesystem.sh --run=setup,9\n\nin order to run all the setup tests as well as test #9, than it is to\ntrack down what all the setup tests are and enter all their numbers in\nthe list. Also, I often find myself wanting to run just one or a couple\ntests from the test file, but I don't know the numbering of any of the\ntests -- to get it I either have to first run the whole test file (or\nstart counting by hand or figure out some other clever but non-obvious\ntricks). It's really convenient to be able to just look at the test\ndescription(s) and then run\n\n ./t6416-recursive-corner-cases.sh --run=symlink\n\nor\n\n ./t6402-merge-rename.sh --run='setup,unnecessary update'\n\nAdd such an ability to test selection which relies on merely matching\nagainst the test description.\n\nSigned-off-by: Elijah Newren \n---\n t/README | 37 +++++++++++++++++++++++++++----------\n t/t0000-basic.sh | 41 +++++++++++++++++++++++++----------------\n t/test-lib.sh | 16 ++++++++++------\n 3 files changed, 62 insertions(+), 32 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 2adaf7c2d2..4594f3ab7b 100644\n--- a/t/README\n+++ b/t/README\n@@ -258,16 +258,21 @@ For an individual test suite --run could be used to specify that\n only some tests should be run or that some tests should be\n excluded from a run.\n \n-The argument for --run is a list of individual test numbers or\n-ranges with an optional negation prefix that define what tests in\n-a test suite to include in the run. A range is two numbers\n-separated with a dash and matches a range of tests with both ends\n-been included. You may omit the first or the second number to\n-mean \"from the first test\" or \"up to the very last test\"\n-respectively.\n-\n-Optional prefix of '!' means that the test or a range of tests\n-should be excluded from the run.\n+The argument for --run, , is a list of description\n+substrings or globs or individual test numbers or ranges with an\n+optional negation prefix (of '!') that define what tests in a test\n+suite to include (or exclude, if negated) in the run. A range is two\n+numbers separated with a dash and matches a range of tests with both\n+ends been included. You may omit the first or the second number to\n+mean \"from the first test\" or \"up to the very last test\" respectively.\n+\n+The argument to --run is split on commas and whitespace into separate\n+strings, numbers, and ranges, and picks all tests that match any of\n+the individual selection criteria. If the substring of the\n+description text that you want to match includes a comma or space, use\n+the glob character '?' instead. For example --run='unnecessary?update\n+timing' would match on all tests that match either the glob\n+*unnecessary?update* or the glob *timing*.\n \n If --run starts with an unprefixed number or range the initial\n set of tests to run is empty. If the first item starts with '!'\n@@ -317,6 +322,18 @@ test in the test suite except from 7 up to 11:\n \n $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n \n+Sometimes there may be multiple tests with e.g. \"setup\" in their name\n+that are needed and rather than figuring out the number for all of them\n+we can just use \"setup\" as a substring/glob to match against the test\n+description:\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,9-11\n+\n+or one could select both the setup tests and the rename ones (assuming all\n+relevant tests had those words in their descriptions):\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,rename\n+\n Some tests in a test suite rely on the previous tests performing\n certain actions, specifically some tests are designated as\n \"setup\" test, so you cannot _arbitrarily_ disable one test and\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 923281af93..07105b2078 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' \"\n \tEOF\n \"\n \n-test_expect_success '--run invalid range start' \"\n+test_expect_success '--run substring selector' \"\n+\trun_sub_test_lib_test run-substring-selector \\\n+\t\t'--run empty selectors' \\\n+\t\t--run='relevant' <<-\\\\EOF &&\n+\ttest_expect_success \\\"relevant test\\\" 'true'\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"other test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-substring-selector <<-\\\\EOF\n+\t> ok 1 - relevant test\n+\t> ok 2 # skip other test #1 (--run)\n+\t> ok 3 # skip other test #2 (--run)\n+\t> ok 4 # skip other test #3 (--run)\n+\t> ok 5 # skip other test #4 (--run)\n+\t> ok 6 # skip other test #5 (--run)\n+\t> ok 7 # skip other test #6 (--run)\n+\t> # passed all 7 test(s)\n+\t> 1..7\n+\tEOF\n+\"\n+\n+test_expect_success '--run keyword selection' \"\n \trun_sub_test_lib_test_err run-inv-range-start \\\n \t\t'--run invalid range start' \\\n \t\t--run='a-5' <<-\\\\EOF &&\n@@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' \"\n \tEOF_ERR\n \"\n \n-test_expect_success '--run invalid selector' \"\n-\trun_sub_test_lib_test_err run-inv-selector \\\n-\t\t'--run invalid selector' \\\n-\t\t--run='1?' <<-\\\\EOF &&\n-\ttest_expect_success \\\"passing test #1\\\" 'true'\n-\ttest_done\n-\tEOF\n-\tcheck_sub_test_lib_test_err run-inv-selector \\\n-\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n-\t> FATAL: Unexpected exit with code 1\n-\tEOF_OUT\n-\t> error: --run: invalid non-numeric in test selector: '1?'\n-\tEOF_ERR\n-\"\n-\n \n test_set_prereq HAVEIT\n haveit=no\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..a040d54a76 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -769,6 +769,8 @@ match_pattern_list () {\n }\n \n match_test_selector_list () {\n+\toperation=\"$1\"\n+\tshift\n \ttitle=\"$1\"\n \tshift\n \targ=\"$1\"\n@@ -805,13 +807,13 @@ match_test_selector_list () {\n \t\t\t*-*)\n \t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"start: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n \t\t\t\tif expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"end: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n@@ -819,9 +821,11 @@ match_test_selector_list () {\n \t\t\t*)\n \t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n-\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n-\t\t\t\t\texit 1\n+\t\t\t\t\tcase \"$title\" in *${selector}*)\n+\t\t\t\t\t\tinclude=$positive\n+\t\t\t\t\t\t;;\n+\t\t\t\t\tesac\n+\t\t\t\t\tcontinue\n \t\t\t\tfi\n \t\tesac\n \n@@ -1031,7 +1035,7 @@ test_skip () {\n \t\tskipped_reason=\"GIT_SKIP_TESTS\"\n \tfi\n \tif test -z \"$to_skip\" && test -n \"$run_list\" &&\n-\t ! match_test_selector_list '--run' $test_count \"$run_list\"\n+\t ! match_test_selector_list '--run' \"$1\" $test_count \"$run_list\"\n \tthen\n \t\tto_skip=t\n \t\tskipped_reason=\"--run\"\n-- \ngitgitgadget\n\n"},{"id":"407742","messageId":"pull.878.v4.git.git.1602876532.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v3.git.git.1602710025.gitgitgadget@gmail.com","subject":"[PATCH v4 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-16T19:28:49Z","receivedAt":"2020-10-16T19:28:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series enables me to change\n\n ./t9999-my-test.sh --run=1-3,5,17,19\n\ninto\n\n ./t9999-my-test.sh --run=setup,rename,symlink\n\nand have it pick out tests with \"setup\", \"rename\", or \"symlink\" in their\ndescription and run those. Saves me a lot of time, especially since numbers\nfor tests aren't readily accessible. The easiest way for me to get the\nnumbers corresponding to the tests I want to run, is to run all the tests\nand look at the output to match up the descriptions with their numbers --\nthus defeating the point of selecting just a subset of the tests to run in\nthe first place.\n\nChanges since v3:\n\n * Added text to the README explaining what to do if the substring you want\n to search on includes a comma or whitespace (namely, use the '?' glob\n character instead)\n\nElijah Newren (3):\n test-lib: allow selecting tests by substring/glob with --run\n t6006, t6012: adjust tests to use 'setup' instead of synonyms\n test-lib: reduce verbosity of skipped tests\n\n t/README | 37 +++++++++++++++++++++++---------\n t/t0000-basic.sh | 41 ++++++++++++++++++++++--------------\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n t/test-lib.sh | 17 +++++++++------\n 5 files changed, 64 insertions(+), 35 deletions(-)\n\n\nbase-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v4\nPull-Request: https://github.com/git/git/pull/878\n\nRange-diff vs v3:\n\n 1: 9c8b6a1a94 ! 1: 8f5d4ad318 test-lib: allow selecting tests by substring/glob with --run\n @@ t/README: For an individual test suite --run could be used to specify that\n +numbers separated with a dash and matches a range of tests with both\n +ends been included. You may omit the first or the second number to\n +mean \"from the first test\" or \"up to the very last test\" respectively.\n ++\n ++The argument to --run is split on commas and whitespace into separate\n ++strings, numbers, and ranges, and picks all tests that match any of\n ++the individual selection criteria. If the substring of the\n ++description text that you want to match includes a comma or space, use\n ++the glob character '?' instead. For example --run='unnecessary?update\n ++timing' would match on all tests that match either the glob\n ++*unnecessary?update* or the glob *timing*.\n \n If --run starts with an unprefixed number or range the initial\n set of tests to run is empty. If the first item starts with '!'\n 2: 6479d73603 = 2: bbae029cc8 t6006, t6012: adjust tests to use 'setup' instead of synonyms\n 3: 24f6febde6 = 3: 0e1a1125f7 test-lib: reduce verbosity of skipped tests\n\n-- \ngitgitgadget\n"},{"id":"407743","messageId":"0e1a1125f7ffc4d97e470474c2c3ee05a25efdeb.1602876532.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v4.git.git.1602876532.gitgitgadget@gmail.com","subject":"[PATCH v4 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-16T19:28:52Z","receivedAt":"2020-10-16T19:29:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen using the --run flag to run just two or three tests from a test\nfile which contains several dozen tests, having every skipped test print\nout dozens of lines of output for the test code for that skipped test\n(in addition to the TAP output line) adds up to hundreds or thousands of\nlines of irrelevant output that make it very hard to fish out the\nrelevant results you were looking for. Simplify the output for skipped\ntests to remove this extra output, leaving only the TAP output line\n(i.e. the line reading \"ok # skip \", which\nalready mentions that the test was \"skip\"ped).\n\nSigned-off-by: Elijah Newren \n---\n t/test-lib.sh | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a040d54a76..3f8c404335 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1062,7 +1062,6 @@ test_skip () {\n \t\t\t\t\" \"\n \t\tfi\n \n-\t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n-- \ngitgitgadget\n"},{"id":"407744","messageId":"bbae029cc868fbc7e628706ba60dfa2c7fd6192d.1602876532.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v4.git.git.1602876532.gitgitgadget@gmail.com","subject":"[PATCH v4 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-16T19:28:51Z","receivedAt":"2020-10-16T19:29:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWith the new ability to pass --run=setup to select which tests to run,\nit is more convenient if tests use the term \"setup\" instead of synonyms\nlike 'prepare' or 'rebuild'. There are undoubtedly many other tests in\nour testsuite that could be changed over too, these are just a couple\nthat I ran into.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex bc95da8a5f..99a1eaf332 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -339,7 +339,7 @@ commit $head1\n .. (hinzugef${added_utf8_part_iso88591}gt) foo\n EOF\n \n-test_expect_success 'prepare expected messages (for test %b)' '\n+test_expect_success 'setup expected messages (for test %b)' '\n \tcat <<-EOF >expected.utf-8 &&\n \tcommit $head3\n \tThis commit message is much longer than the others,\ndiff --git a/t/t6012-rev-list-simplify.sh b/t/t6012-rev-list-simplify.sh\nindex b6fa43ace0..7254060240 100755\n--- a/t/t6012-rev-list-simplify.sh\n+++ b/t/t6012-rev-list-simplify.sh\n@@ -168,7 +168,7 @@ test_expect_success '--full-diff is not affected by --parents' '\n #\n # This example is explained in Documentation/rev-list-options.txt\n \n-test_expect_success 'rebuild repo' '\n+test_expect_success 'setup rebuild repo' '\n \trm -rf .git * &&\n \tgit init &&\n \tgit switch -c main &&\n-- \ngitgitgadget\n\n"},{"id":"407748","messageId":"xmqqd01iaqif.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"CABPp-BHkykWh8L_FYhLR1BCCpPDmc_2q+Tccg_yZ7W8ZHZ4WsA@mail.gmail.com","subject":"Re: [PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-16T20:02:16Z","receivedAt":"2020-10-16T20:02:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Given that t/README explicitly shows examples of space-separated lists\n> of numbers, I'm worried we're breaking long-built expectations of\n> other developers by changing IFS here. Perhaps I could instead add\n> the following paragraph to t/README:\n\nUnlike something that would affect end-users, I think it is OK to\nbreak backward compatibility one-way. If you suddenly forbid spaces\nand force our developers to use comma and nothing else, in muscle\nmemory of their fingers and/or in their scripts, in a version merged\nto 'master', as long as their new habit and updated scripts use\ncomma consistently, they work fine on 'maint', right?\n\nIf there is no such \"works on both sides of the flag day\" choice, it\nis a different story, of course, but comma should work fine for us\nin this case.\n\nThanks.\n"},{"id":"407749","messageId":"20201016200642.GB3355643@coredump.intra.peff.net","threadId":"54403","inReplyTo":"xmqq7drqdu2u.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-16T20:06:42Z","receivedAt":"2020-10-16T20:06:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 16, 2020 at 09:16:57AM -0700, Junio C Hamano wrote:\n\n> > I'm not sure where he suggests worse names. I'd think if anything we'd\n> > have better names, because they'd be even more meaningful (if people\n> > start using them for test selectors). FWIW, I also grep like that when\n> > looking for scripts.\n> \n> I didn't mean Jonathan suggested worse names. Unlike \"I don't tend\n> to discover test scripts based on their filename\", which was what\n> Jonathan said, I do look for tests based on their filename, so\n> having a good name matters (on the other hand, if you are the kind\n> of person who does not look for them by name, the naming may not\n> matter to you).\n\nAh, I misunderstood. Thanks for clarifying (and I am very much in your\ncamp that the names are useful).\n\n> As long as it is known that \"filesystem\" and \"update-ref\" can serve\n> as tokens to uniquely identify these two tests, it would be fine for\n> my purpose. But 0050 (under the rule that numbering must be unique)\n> would give me such an assurance much better without having to look\n> at any other test file.\n> \n> The word \"filesystem\"? Unless we have a rule that we can use each\n> unique word in test names only once (which of course is impractical)\n> I am not sure I can use it in place of 0050 without checking names\n> of other tests first.\n\nWith your follow-up response:\n\n> So if the rule is to always spell out the full name if I wanted\n> uniqueness, then it would work.\n\nI think we are on the same page, and my intent was to match full names.\n\nSo now you get \"t0050\" from some failed-test output (prove, or just the\noutput from make failing), and you copy it into the command-line to use\nwith \"--test\". And instead you'd just copy the full text name. It's a\nlittle less convenient because t1234 is short enough that I'd type it,\nand I'd probably cut-and-paste the text name. But other than that, I'd\nexpect the procedure to be the same.\n\nThe substring matches added by Elijah's series make sense for individual\ntest snippets within a script, I think. And I think we could even add\nscript-name matching now[1], without getting rid of the numbers. But if\nwe do so, we should be careful to introduce it as an anchored match and\nnot a substring match, to avoid having to switch it later.\n\n-Peff\n\n[1] And by \"now\" I don't mean we should hold up Elijah's patches for this\n feature, but that anybody who wishes to build it on top is free to\n do so without us having to make a decision on ditching the numbers\n entirely.\n"},{"id":"407750","messageId":"20201016200744.GC3355643@coredump.intra.peff.net","threadId":"54403","inReplyTo":"xmqqd01iaqif.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-10-16T20:07:44Z","receivedAt":"2020-10-16T20:07:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 16, 2020 at 01:02:16PM -0700, Junio C Hamano wrote:\n\n> Elijah Newren writes:\n> \n> > Given that t/README explicitly shows examples of space-separated lists\n> > of numbers, I'm worried we're breaking long-built expectations of\n> > other developers by changing IFS here. Perhaps I could instead add\n> > the following paragraph to t/README:\n> \n> Unlike something that would affect end-users, I think it is OK to\n> break backward compatibility one-way. If you suddenly forbid spaces\n> and force our developers to use comma and nothing else, in muscle\n> memory of their fingers and/or in their scripts, in a version merged\n> to 'master', as long as their new habit and updated scripts use\n> comma consistently, they work fine on 'maint', right?\n> \n> If there is no such \"works on both sides of the flag day\" choice, it\n> is a different story, of course, but comma should work fine for us\n> in this case.\n\nI was just about to write the same argument. :)\n\n-Peff\n"},{"id":"407753","messageId":"xmqq4kmtc45q.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"20201016200642.GB3355643@coredump.intra.peff.net","subject":"Re: [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-16T20:22:09Z","receivedAt":"2020-10-16T20:22:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I think we are on the same page, and my intent was to match full names.\n>\n> So now you get \"t0050\" from some failed-test output (prove, or just the\n> output from make failing), and you copy it into the command-line to use\n> with \"--test\". And instead you'd just copy the full text name. It's a\n> little less convenient because t1234 is short enough that I'd type it,\n> and I'd probably cut-and-paste the text name. But other than that, I'd\n> expect the procedure to be the same.\n\nMost often, they are no longer in my scroll buffer, though. Two\n4-digit integers I can easily hold in my head and type. Full names\nof two test scripts? That's a bit taxing.\n\nSo, not really.\n"},{"id":"407755","messageId":"xmqqzh4lapfr.fsf@gitster.c.googlers.com","threadId":"54403","inReplyTo":"pull.878.v4.git.git.1602876532.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-10-16T20:25:28Z","receivedAt":"2020-10-16T20:25:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> Changes since v3:\n>\n> * Added text to the README explaining what to do if the substring you want\n> to search on includes a comma or whitespace (namely, use the '?' glob\n> character instead)\n\nThat would work just as well with or without forbidding spaces.\n\n> ++The argument to --run is split on commas and whitespace into separate\n> ++strings, numbers, and ranges, and picks all tests that match any of\n> ++the individual selection criteria. If the substring of the\n> ++description text that you want to match includes a comma or space, use\n> ++the glob character '?' instead. For example --run='unnecessary?update\n> ++timing' would match on all tests that match either the glob\n> ++*unnecessary?update* or the glob *timing*.\n\nOK.\n\nWill queue.\n"},{"id":"407761","messageId":"CABPp-BERdr-UvMEe=kjeRjtXfeV0uu2LOk6G3oYGMWe=gxKxeg@mail.gmail.com","threadId":"54403","inReplyTo":"xmqqzh4lapfr.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v4 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-16T20:56:59Z","receivedAt":"2020-10-16T20:57:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Oct 16, 2020 at 1:25 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > Changes since v3:\n> >\n> > * Added text to the README explaining what to do if the substring you want\n> > to search on includes a comma or whitespace (namely, use the '?' glob\n> > character instead)\n>\n> That would work just as well with or without forbidding spaces.\n>\n> > ++The argument to --run is split on commas and whitespace into separate\n> > ++strings, numbers, and ranges, and picks all tests that match any of\n> > ++the individual selection criteria. If the substring of the\n> > ++description text that you want to match includes a comma or space, use\n> > ++the glob character '?' instead. For example --run='unnecessary?update\n> > ++timing' would match on all tests that match either the glob\n> > ++*unnecessary?update* or the glob *timing*.\n>\n> OK.\n>\n> Will queue.\n\nSince both you and Peff argued for just changing IFS, I'll do that,\nand update this text and elsewhere in t/README to reflect that we only\nsplit on commas and send that in as v4.\n"},{"id":"407772","messageId":"7c7a3d9a348075e0aad7790019c0a44f0d306cb0.1602888652.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v5.git.git.1602888652.gitgitgadget@gmail.com","subject":"[PATCH v5 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-16T22:50:51Z","receivedAt":"2020-10-16T22:50:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWith the new ability to pass --run=setup to select which tests to run,\nit is more convenient if tests use the term \"setup\" instead of synonyms\nlike 'prepare' or 'rebuild'. There are undoubtedly many other tests in\nour testsuite that could be changed over too, these are just a couple\nthat I ran into.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex bc95da8a5f..99a1eaf332 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -339,7 +339,7 @@ commit $head1\n .. (hinzugef${added_utf8_part_iso88591}gt) foo\n EOF\n \n-test_expect_success 'prepare expected messages (for test %b)' '\n+test_expect_success 'setup expected messages (for test %b)' '\n \tcat <<-EOF >expected.utf-8 &&\n \tcommit $head3\n \tThis commit message is much longer than the others,\ndiff --git a/t/t6012-rev-list-simplify.sh b/t/t6012-rev-list-simplify.sh\nindex b6fa43ace0..7254060240 100755\n--- a/t/t6012-rev-list-simplify.sh\n+++ b/t/t6012-rev-list-simplify.sh\n@@ -168,7 +168,7 @@ test_expect_success '--full-diff is not affected by --parents' '\n #\n # This example is explained in Documentation/rev-list-options.txt\n \n-test_expect_success 'rebuild repo' '\n+test_expect_success 'setup rebuild repo' '\n \trm -rf .git * &&\n \tgit init &&\n \tgit switch -c main &&\n-- \ngitgitgadget\n\n"},{"id":"407773","messageId":"01494bc0ba05cf9258c9445e20d316a4efb32092.1602888652.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v5.git.git.1602888652.gitgitgadget@gmail.com","subject":"[PATCH v5 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-16T22:50:52Z","receivedAt":"2020-10-16T22:51:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen using the --run flag to run just two or three tests from a test\nfile which contains several dozen tests, having every skipped test print\nout dozens of lines of output for the test code for that skipped test\n(in addition to the TAP output line) adds up to hundreds or thousands of\nlines of irrelevant output that make it very hard to fish out the\nrelevant results you were looking for. Simplify the output for skipped\ntests to remove this extra output, leaving only the TAP output line\n(i.e. the line reading \"ok # skip \", which\nalready mentions that the test was \"skip\"ped).\n\nSigned-off-by: Elijah Newren \n---\n t/test-lib.sh | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex debfd73fd6..e020bcc5e7 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1062,7 +1062,6 @@ test_skip () {\n \t\t\t\t\" \"\n \t\tfi\n \n-\t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n-- \ngitgitgadget\n"},{"id":"407774","messageId":"389c2c171a07b156fe956ac535033a8b67af51a9.1602888652.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v5.git.git.1602888652.gitgitgadget@gmail.com","subject":"[PATCH v5 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-16T22:50:50Z","receivedAt":"2020-10-16T22:51:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany of our test scripts have several \"setup\" tests. It's a lot easier\nto say\n\n ./t0050-filesystem.sh --run=setup,9\n\nin order to run all the setup tests as well as test #9, than it is to\ntrack down what all the setup tests are and enter all their numbers in\nthe list. Also, I often find myself wanting to run just one or a couple\ntests from the test file, but I don't know the numbering of any of the\ntests -- to get it I either have to first run the whole test file (or\nstart counting by hand or figure out some other clever but non-obvious\ntricks). It's really convenient to be able to just look at the test\ndescription(s) and then run\n\n ./t6416-recursive-corner-cases.sh --run=symlink\n\nor\n\n ./t6402-merge-rename.sh --run='setup,unnecessary update'\n\nAdd such an ability to test selection which relies on merely matching\nagainst the test description.\n\nSigned-off-by: Elijah Newren \n---\n t/README | 48 +++++++++++++++++++++++++++----------------\n t/t0000-basic.sh | 53 ++++++++++++++++++++++++++++--------------------\n t/test-lib.sh | 18 +++++++++-------\n 3 files changed, 73 insertions(+), 46 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 2adaf7c2d2..c730a70770 100644\n--- a/t/README\n+++ b/t/README\n@@ -258,16 +258,21 @@ For an individual test suite --run could be used to specify that\n only some tests should be run or that some tests should be\n excluded from a run.\n \n-The argument for --run is a list of individual test numbers or\n-ranges with an optional negation prefix that define what tests in\n-a test suite to include in the run. A range is two numbers\n-separated with a dash and matches a range of tests with both ends\n-been included. You may omit the first or the second number to\n-mean \"from the first test\" or \"up to the very last test\"\n-respectively.\n-\n-Optional prefix of '!' means that the test or a range of tests\n-should be excluded from the run.\n+The argument for --run, , is a list of description\n+substrings or globs or individual test numbers or ranges with an\n+optional negation prefix (of '!') that define what tests in a test\n+suite to include (or exclude, if negated) in the run. A range is two\n+numbers separated with a dash and matches a range of tests with both\n+ends been included. You may omit the first or the second number to\n+mean \"from the first test\" or \"up to the very last test\" respectively.\n+\n+The argument to --run is split on commas into separate strings,\n+numbers, and ranges, and picks all tests that match any of the\n+individual selection criteria. If the substring of the description\n+text that you want to match includes a comma, use the glob character\n+'?' instead. For example --run='rebase,merge?cherry-pick' would match\n+on all tests that match either the glob *rebase* or the glob\n+*merge?cherry-pick*.\n \n If --run starts with an unprefixed number or range the initial\n set of tests to run is empty. If the first item starts with '!'\n@@ -275,9 +280,6 @@ all the tests are added to the initial set. After initial set is\n determined every test number or range is added or excluded from\n the set one by one, from left to right.\n \n-Individual numbers or ranges could be separated either by a space\n-or a comma.\n-\n For example, to run only tests up to a specific test (21), one\n could do this:\n \n@@ -290,7 +292,7 @@ or this:\n Common case is to run several setup tests (1, 2, 3) and then a\n specific test (21) that relies on that setup:\n \n- $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'\n+ $ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21'\n \n or:\n \n@@ -298,17 +300,17 @@ or:\n \n or:\n \n- $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'\n+ $ sh ./t9200-git-cvsexport-commit.sh --run='-3,21'\n \n As noted above, the test set is built by going through the items\n from left to right, so this:\n \n- $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n+ $ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3'\n \n will run tests 1, 2, and 4. Items that come later have higher\n precedence. It means that this:\n \n- $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'\n+ $ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4'\n \n would just run tests from 1 to 4, including 3.\n \n@@ -317,6 +319,18 @@ test in the test suite except from 7 up to 11:\n \n $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n \n+Sometimes there may be multiple tests with e.g. \"setup\" in their name\n+that are needed and rather than figuring out the number for all of them\n+we can just use \"setup\" as a substring/glob to match against the test\n+description:\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,9-11\n+\n+or one could select both the setup tests and the rename ones (assuming all\n+relevant tests had those words in their descriptions):\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,rename\n+\n Some tests in a test suite rely on the previous tests performing\n certain actions, specifically some tests are designated as\n \"setup\" test, so you cannot _arbitrarily_ disable one test and\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 923281af93..62a16ed4c8 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -430,7 +430,7 @@ test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' \"\n \n test_expect_success '--run basic' \"\n \trun_sub_test_lib_test run-basic \\\n-\t\t'--run basic' --run='1 3 5' <<-\\\\EOF &&\n+\t\t'--run basic' --run='1,3,5' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -472,7 +472,7 @@ test_expect_success '--run with a range' \"\n \n test_expect_success '--run with two ranges' \"\n \trun_sub_test_lib_test run-two-ranges \\\n-\t\t'--run with two ranges' --run='1-2 5-6' <<-\\\\EOF &&\n+\t\t'--run with two ranges' --run='1-2,5-6' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -556,7 +556,7 @@ test_expect_success '--run with basic negation' \"\n \n test_expect_success '--run with two negations' \"\n \trun_sub_test_lib_test run-two-neg \\\n-\t\t'--run with two negations' --run='\"'!3 !6'\"' <<-\\\\EOF &&\n+\t\t'--run with two negations' --run='\"'!3,!6'\"' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -577,7 +577,7 @@ test_expect_success '--run with two negations' \"\n \n test_expect_success '--run a range and negation' \"\n \trun_sub_test_lib_test run-range-and-neg \\\n-\t\t'--run a range and negation' --run='\"'-4 !2'\"' <<-\\\\EOF &&\n+\t\t'--run a range and negation' --run='\"'-4,!2'\"' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -620,7 +620,7 @@ test_expect_success '--run range negation' \"\n test_expect_success '--run include, exclude and include' \"\n \trun_sub_test_lib_test run-inc-neg-inc \\\n \t\t'--run include, exclude and include' \\\n-\t\t--run='\"'1-5 !1-3 2'\"' <<-\\\\EOF &&\n+\t\t--run='\"'1-5,!1-3,2'\"' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -664,7 +664,7 @@ test_expect_success '--run include, exclude and include, comma separated' \"\n test_expect_success '--run exclude and include' \"\n \trun_sub_test_lib_test run-neg-inc \\\n \t\t'--run exclude and include' \\\n-\t\t--run='\"'!3- 5'\"' <<-\\\\EOF &&\n+\t\t--run='\"'!3-,5'\"' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' \"\n \tEOF\n \"\n \n-test_expect_success '--run invalid range start' \"\n+test_expect_success '--run substring selector' \"\n+\trun_sub_test_lib_test run-substring-selector \\\n+\t\t'--run empty selectors' \\\n+\t\t--run='relevant' <<-\\\\EOF &&\n+\ttest_expect_success \\\"relevant test\\\" 'true'\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"other test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-substring-selector <<-\\\\EOF\n+\t> ok 1 - relevant test\n+\t> ok 2 # skip other test #1 (--run)\n+\t> ok 3 # skip other test #2 (--run)\n+\t> ok 4 # skip other test #3 (--run)\n+\t> ok 5 # skip other test #4 (--run)\n+\t> ok 6 # skip other test #5 (--run)\n+\t> ok 7 # skip other test #6 (--run)\n+\t> # passed all 7 test(s)\n+\t> 1..7\n+\tEOF\n+\"\n+\n+test_expect_success '--run keyword selection' \"\n \trun_sub_test_lib_test_err run-inv-range-start \\\n \t\t'--run invalid range start' \\\n \t\t--run='a-5' <<-\\\\EOF &&\n@@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' \"\n \tEOF_ERR\n \"\n \n-test_expect_success '--run invalid selector' \"\n-\trun_sub_test_lib_test_err run-inv-selector \\\n-\t\t'--run invalid selector' \\\n-\t\t--run='1?' <<-\\\\EOF &&\n-\ttest_expect_success \\\"passing test #1\\\" 'true'\n-\ttest_done\n-\tEOF\n-\tcheck_sub_test_lib_test_err run-inv-selector \\\n-\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n-\t> FATAL: Unexpected exit with code 1\n-\tEOF_OUT\n-\t> error: --run: invalid non-numeric in test selector: '1?'\n-\tEOF_ERR\n-\"\n-\n \n test_set_prereq HAVEIT\n haveit=no\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..debfd73fd6 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -769,6 +769,8 @@ match_pattern_list () {\n }\n \n match_test_selector_list () {\n+\toperation=\"$1\"\n+\tshift\n \ttitle=\"$1\"\n \tshift\n \targ=\"$1\"\n@@ -777,7 +779,7 @@ match_test_selector_list () {\n \n \t# Both commas and whitespace are accepted as separators.\n \tOLDIFS=$IFS\n-\tIFS=' \t,'\n+\tIFS=','\n \tset -- $1\n \tIFS=$OLDIFS\n \n@@ -805,13 +807,13 @@ match_test_selector_list () {\n \t\t\t*-*)\n \t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"start: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n \t\t\t\tif expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"end: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n@@ -819,9 +821,11 @@ match_test_selector_list () {\n \t\t\t*)\n \t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n-\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n-\t\t\t\t\texit 1\n+\t\t\t\t\tcase \"$title\" in *${selector}*)\n+\t\t\t\t\t\tinclude=$positive\n+\t\t\t\t\t\t;;\n+\t\t\t\t\tesac\n+\t\t\t\t\tcontinue\n \t\t\t\tfi\n \t\tesac\n \n@@ -1031,7 +1035,7 @@ test_skip () {\n \t\tskipped_reason=\"GIT_SKIP_TESTS\"\n \tfi\n \tif test -z \"$to_skip\" && test -n \"$run_list\" &&\n-\t ! match_test_selector_list '--run' $test_count \"$run_list\"\n+\t ! match_test_selector_list '--run' \"$1\" $test_count \"$run_list\"\n \tthen\n \t\tto_skip=t\n \t\tskipped_reason=\"--run\"\n-- \ngitgitgadget\n\n"},{"id":"407775","messageId":"pull.878.v5.git.git.1602888652.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v4.git.git.1602876532.gitgitgadget@gmail.com","subject":"[PATCH v5 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-16T22:50:49Z","receivedAt":"2020-10-16T22:51:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series enables me to change\n\n ./t9999-my-test.sh --run=1-3,5,17,19\n\ninto\n\n ./t9999-my-test.sh --run=setup,rename,symlink\n\nand have it pick out tests with \"setup\", \"rename\", or \"symlink\" in their\ndescription and run those. Saves me a lot of time, especially since numbers\nfor tests aren't readily accessible. The easiest way for me to get the\nnumbers corresponding to the tests I want to run, is to run all the tests\nand look at the output to match up the descriptions with their numbers --\nthus defeating the point of selecting just a subset of the tests to run in\nthe first place.\n\nChanges since v4:\n\n * Changed to only split tests on commas\n\nElijah Newren (3):\n test-lib: allow selecting tests by substring/glob with --run\n t6006, t6012: adjust tests to use 'setup' instead of synonyms\n test-lib: reduce verbosity of skipped tests\n\n t/README | 48 ++++++++++++++++++++------------\n t/t0000-basic.sh | 53 +++++++++++++++++++++---------------\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n t/test-lib.sh | 19 +++++++------\n 5 files changed, 75 insertions(+), 49 deletions(-)\n\n\nbase-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v5\nPull-Request: https://github.com/git/git/pull/878\n\nRange-diff vs v4:\n\n 1: 8f5d4ad318 ! 1: 389c2c171a test-lib: allow selecting tests by substring/glob with --run\n @@ t/README: For an individual test suite --run could be used to specify that\n +ends been included. You may omit the first or the second number to\n +mean \"from the first test\" or \"up to the very last test\" respectively.\n +\n -+The argument to --run is split on commas and whitespace into separate\n -+strings, numbers, and ranges, and picks all tests that match any of\n -+the individual selection criteria. If the substring of the\n -+description text that you want to match includes a comma or space, use\n -+the glob character '?' instead. For example --run='unnecessary?update\n -+timing' would match on all tests that match either the glob\n -+*unnecessary?update* or the glob *timing*.\n ++The argument to --run is split on commas into separate strings,\n ++numbers, and ranges, and picks all tests that match any of the\n ++individual selection criteria. If the substring of the description\n ++text that you want to match includes a comma, use the glob character\n ++'?' instead. For example --run='rebase,merge?cherry-pick' would match\n ++on all tests that match either the glob *rebase* or the glob\n ++*merge?cherry-pick*.\n \n If --run starts with an unprefixed number or range the initial\n set of tests to run is empty. If the first item starts with '!'\n +@@ t/README: all the tests are added to the initial set. After initial set is\n + determined every test number or range is added or excluded from\n + the set one by one, from left to right.\n + \n +-Individual numbers or ranges could be separated either by a space\n +-or a comma.\n +-\n + For example, to run only tests up to a specific test (21), one\n + could do this:\n + \n +@@ t/README: or this:\n + Common case is to run several setup tests (1, 2, 3) and then a\n + specific test (21) that relies on that setup:\n + \n +- $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'\n ++ $ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21'\n + \n + or:\n + \n +@@ t/README: or:\n + \n + or:\n + \n +- $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'\n ++ $ sh ./t9200-git-cvsexport-commit.sh --run='-3,21'\n + \n + As noted above, the test set is built by going through the items\n + from left to right, so this:\n + \n +- $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n ++ $ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3'\n + \n + will run tests 1, 2, and 4. Items that come later have higher\n + precedence. It means that this:\n + \n +- $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'\n ++ $ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4'\n + \n + would just run tests from 1 to 4, including 3.\n + \n @@ t/README: test in the test suite except from 7 up to 11:\n \n $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n @@ t/README: test in the test suite except from 7 up to 11:\n \"setup\" test, so you cannot _arbitrarily_ disable one test and\n \n ## t/t0000-basic.sh ##\n +@@ t/t0000-basic.sh: test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' \"\n + \n + test_expect_success '--run basic' \"\n + \trun_sub_test_lib_test run-basic \\\n +-\t\t'--run basic' --run='1 3 5' <<-\\\\EOF &&\n ++\t\t'--run basic' --run='1,3,5' <<-\\\\EOF &&\n + \tfor i in 1 2 3 4 5 6\n + \tdo\n + \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n +@@ t/t0000-basic.sh: test_expect_success '--run with a range' \"\n + \n + test_expect_success '--run with two ranges' \"\n + \trun_sub_test_lib_test run-two-ranges \\\n +-\t\t'--run with two ranges' --run='1-2 5-6' <<-\\\\EOF &&\n ++\t\t'--run with two ranges' --run='1-2,5-6' <<-\\\\EOF &&\n + \tfor i in 1 2 3 4 5 6\n + \tdo\n + \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n +@@ t/t0000-basic.sh: test_expect_success '--run with basic negation' \"\n + \n + test_expect_success '--run with two negations' \"\n + \trun_sub_test_lib_test run-two-neg \\\n +-\t\t'--run with two negations' --run='\"'!3 !6'\"' <<-\\\\EOF &&\n ++\t\t'--run with two negations' --run='\"'!3,!6'\"' <<-\\\\EOF &&\n + \tfor i in 1 2 3 4 5 6\n + \tdo\n + \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n +@@ t/t0000-basic.sh: test_expect_success '--run with two negations' \"\n + \n + test_expect_success '--run a range and negation' \"\n + \trun_sub_test_lib_test run-range-and-neg \\\n +-\t\t'--run a range and negation' --run='\"'-4 !2'\"' <<-\\\\EOF &&\n ++\t\t'--run a range and negation' --run='\"'-4,!2'\"' <<-\\\\EOF &&\n + \tfor i in 1 2 3 4 5 6\n + \tdo\n + \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n +@@ t/t0000-basic.sh: test_expect_success '--run range negation' \"\n + test_expect_success '--run include, exclude and include' \"\n + \trun_sub_test_lib_test run-inc-neg-inc \\\n + \t\t'--run include, exclude and include' \\\n +-\t\t--run='\"'1-5 !1-3 2'\"' <<-\\\\EOF &&\n ++\t\t--run='\"'1-5,!1-3,2'\"' <<-\\\\EOF &&\n + \tfor i in 1 2 3 4 5 6\n + \tdo\n + \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n +@@ t/t0000-basic.sh: test_expect_success '--run include, exclude and include, comma separated' \"\n + test_expect_success '--run exclude and include' \"\n + \trun_sub_test_lib_test run-neg-inc \\\n + \t\t'--run exclude and include' \\\n +-\t\t--run='\"'!3- 5'\"' <<-\\\\EOF &&\n ++\t\t--run='\"'!3-,5'\"' <<-\\\\EOF &&\n + \tfor i in 1 2 3 4 5 6\n + \tdo\n + \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n @@ t/t0000-basic.sh: test_expect_success '--run empty selectors' \"\n \tEOF\n \"\n @@ t/test-lib.sh: match_pattern_list () {\n \ttitle=\"$1\"\n \tshift\n \targ=\"$1\"\n +@@ t/test-lib.sh: match_test_selector_list () {\n + \n + \t# Both commas and whitespace are accepted as separators.\n + \tOLDIFS=$IFS\n +-\tIFS=' \t,'\n ++\tIFS=','\n + \tset -- $1\n + \tIFS=$OLDIFS\n + \n @@ t/test-lib.sh: match_test_selector_list () {\n \t\t\t*-*)\n \t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n 2: bbae029cc8 = 2: 7c7a3d9a34 t6006, t6012: adjust tests to use 'setup' instead of synonyms\n 3: 0e1a1125f7 = 3: 01494bc0ba test-lib: reduce verbosity of skipped tests\n\n-- \ngitgitgadget\n"},{"id":"407778","messageId":"89279674-a1b5-930a-e682-b47aafeb11b7@gmail.com","threadId":"54403","inReplyTo":"389c2c171a07b156fe956ac535033a8b67af51a9.1602888652.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Andrei Rybak","fromEmail":"rybak.a.v@gmail.com","sentAt":"2020-10-16T23:25:57Z","receivedAt":"2020-10-16T23:26:05Z","isPatch":true,"sender":{"key":"rybak.a.v@gmail.com","avatar":"https://avatars.githubusercontent.com/u/624072?v=4"},"body":"On 2020-10-17 00:50, Elijah Newren via GitGitGadget wrote:\n\n[trim]\n\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index ef31f40037..debfd73fd6 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -769,6 +769,8 @@ match_pattern_list () {\n> }\n> \n> match_test_selector_list () {\n> +\toperation=\"$1\"\n> +\tshift\n> \ttitle=\"$1\"\n> \tshift\n> \targ=\"$1\"\n> @@ -777,7 +779,7 @@ match_test_selector_list () {\n> \n> \t# Both commas and whitespace are accepted as separators.\n\nThis comment becomes outdated with the change below.\n\n> \tOLDIFS=$IFS\n> -\tIFS=' \t,'\n> +\tIFS=','\n> \tset -- $1\n> \tIFS=$OLDIFS\n> \n> @@ -805,13 +807,13 @@ match_test_selector_list () {\n\n"},{"id":"407824","messageId":"CABPp-BGXpjYkHEYb_WR0Miy+dHLtZWLjFb3G-EhSi+AOx0voHg@mail.gmail.com","threadId":"54403","inReplyTo":"89279674-a1b5-930a-e682-b47aafeb11b7@gmail.com","subject":"Re: [PATCH v5 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-10-17T23:43:10Z","receivedAt":"2020-10-17T23:43:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Oct 16, 2020 at 4:25 PM Andrei Rybak wrote:\n>\n> On 2020-10-17 00:50, Elijah Newren via GitGitGadget wrote:\n>\n> [trim]\n>\n> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> > index ef31f40037..debfd73fd6 100644\n> > --- a/t/test-lib.sh\n> > +++ b/t/test-lib.sh\n> > @@ -769,6 +769,8 @@ match_pattern_list () {\n> > }\n> >\n> > match_test_selector_list () {\n> > + operation=\"$1\"\n> > + shift\n> > title=\"$1\"\n> > shift\n> > arg=\"$1\"\n> > @@ -777,7 +779,7 @@ match_test_selector_list () {\n> >\n> > # Both commas and whitespace are accepted as separators.\n>\n> This comment becomes outdated with the change below.\n\nSigh...indeed. Thanks for pointing it out; I'll send out v6 soon with the fix.\n"},{"id":"407825","messageId":"pull.878.v6.git.git.1602980628.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v5.git.git.1602888652.gitgitgadget@gmail.com","subject":"[PATCH v6 0/3] Make test selection easier by specifying description substrings instead of just numeric counters","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-18T00:23:44Z","receivedAt":"2020-10-18T00:23:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series enables me to change\n\n ./t9999-my-test.sh --run=1-3,5,17,19\n\ninto\n\n ./t9999-my-test.sh --run=setup,rename,symlink\n\nand have it pick out tests with \"setup\", \"rename\", or \"symlink\" in their\ndescription and run those. Saves me a lot of time, especially since numbers\nfor tests aren't readily accessible. The easiest way for me to get the\nnumbers corresponding to the tests I want to run, is to run all the tests\nand look at the output to match up the descriptions with their numbers --\nthus defeating the point of selecting just a subset of the tests to run in\nthe first place.\n\nChanges since v5:\n\n * Update comment to match the code (selectors are now only split on commas,\n not on whitespace)\n\nElijah Newren (3):\n test-lib: allow selecting tests by substring/glob with --run\n t6006, t6012: adjust tests to use 'setup' instead of synonyms\n test-lib: reduce verbosity of skipped tests\n\n t/README | 48 ++++++++++++++++++++------------\n t/t0000-basic.sh | 53 +++++++++++++++++++++---------------\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n t/test-lib.sh | 21 ++++++++------\n 5 files changed, 76 insertions(+), 50 deletions(-)\n\n\nbase-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v6\nPull-Request: https://github.com/git/git/pull/878\n\nRange-diff vs v5:\n\n 1: 389c2c171a ! 1: 0355c88882 test-lib: allow selecting tests by substring/glob with --run\n @@ t/test-lib.sh: match_pattern_list () {\n \ttitle=\"$1\"\n \tshift\n \targ=\"$1\"\n -@@ t/test-lib.sh: match_test_selector_list () {\n + \tshift\n + \ttest -z \"$1\" && return 0\n \n - \t# Both commas and whitespace are accepted as separators.\n +-\t# Both commas and whitespace are accepted as separators.\n ++\t# Commas are accepted as separators.\n \tOLDIFS=$IFS\n -\tIFS=' \t,'\n +\tIFS=','\n 2: 7c7a3d9a34 = 2: 3cc9abf461 t6006, t6012: adjust tests to use 'setup' instead of synonyms\n 3: 01494bc0ba = 3: 0af7835487 test-lib: reduce verbosity of skipped tests\n\n-- \ngitgitgadget\n"},{"id":"407827","messageId":"0355c888828aeec331a6b99a26d8aa04cff59859.1602980628.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v6.git.git.1602980628.gitgitgadget@gmail.com","subject":"[PATCH v6 1/3] test-lib: allow selecting tests by substring/glob with --run","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-18T00:23:45Z","receivedAt":"2020-10-18T00:23:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany of our test scripts have several \"setup\" tests. It's a lot easier\nto say\n\n ./t0050-filesystem.sh --run=setup,9\n\nin order to run all the setup tests as well as test #9, than it is to\ntrack down what all the setup tests are and enter all their numbers in\nthe list. Also, I often find myself wanting to run just one or a couple\ntests from the test file, but I don't know the numbering of any of the\ntests -- to get it I either have to first run the whole test file (or\nstart counting by hand or figure out some other clever but non-obvious\ntricks). It's really convenient to be able to just look at the test\ndescription(s) and then run\n\n ./t6416-recursive-corner-cases.sh --run=symlink\n\nor\n\n ./t6402-merge-rename.sh --run='setup,unnecessary update'\n\nAdd such an ability to test selection which relies on merely matching\nagainst the test description.\n\nSigned-off-by: Elijah Newren \n---\n t/README | 48 +++++++++++++++++++++++++++----------------\n t/t0000-basic.sh | 53 ++++++++++++++++++++++++++++--------------------\n t/test-lib.sh | 20 ++++++++++--------\n 3 files changed, 74 insertions(+), 47 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 2adaf7c2d2..c730a70770 100644\n--- a/t/README\n+++ b/t/README\n@@ -258,16 +258,21 @@ For an individual test suite --run could be used to specify that\n only some tests should be run or that some tests should be\n excluded from a run.\n \n-The argument for --run is a list of individual test numbers or\n-ranges with an optional negation prefix that define what tests in\n-a test suite to include in the run. A range is two numbers\n-separated with a dash and matches a range of tests with both ends\n-been included. You may omit the first or the second number to\n-mean \"from the first test\" or \"up to the very last test\"\n-respectively.\n-\n-Optional prefix of '!' means that the test or a range of tests\n-should be excluded from the run.\n+The argument for --run, , is a list of description\n+substrings or globs or individual test numbers or ranges with an\n+optional negation prefix (of '!') that define what tests in a test\n+suite to include (or exclude, if negated) in the run. A range is two\n+numbers separated with a dash and matches a range of tests with both\n+ends been included. You may omit the first or the second number to\n+mean \"from the first test\" or \"up to the very last test\" respectively.\n+\n+The argument to --run is split on commas into separate strings,\n+numbers, and ranges, and picks all tests that match any of the\n+individual selection criteria. If the substring of the description\n+text that you want to match includes a comma, use the glob character\n+'?' instead. For example --run='rebase,merge?cherry-pick' would match\n+on all tests that match either the glob *rebase* or the glob\n+*merge?cherry-pick*.\n \n If --run starts with an unprefixed number or range the initial\n set of tests to run is empty. If the first item starts with '!'\n@@ -275,9 +280,6 @@ all the tests are added to the initial set. After initial set is\n determined every test number or range is added or excluded from\n the set one by one, from left to right.\n \n-Individual numbers or ranges could be separated either by a space\n-or a comma.\n-\n For example, to run only tests up to a specific test (21), one\n could do this:\n \n@@ -290,7 +292,7 @@ or this:\n Common case is to run several setup tests (1, 2, 3) and then a\n specific test (21) that relies on that setup:\n \n- $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'\n+ $ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21'\n \n or:\n \n@@ -298,17 +300,17 @@ or:\n \n or:\n \n- $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'\n+ $ sh ./t9200-git-cvsexport-commit.sh --run='-3,21'\n \n As noted above, the test set is built by going through the items\n from left to right, so this:\n \n- $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'\n+ $ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3'\n \n will run tests 1, 2, and 4. Items that come later have higher\n precedence. It means that this:\n \n- $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'\n+ $ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4'\n \n would just run tests from 1 to 4, including 3.\n \n@@ -317,6 +319,18 @@ test in the test suite except from 7 up to 11:\n \n $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'\n \n+Sometimes there may be multiple tests with e.g. \"setup\" in their name\n+that are needed and rather than figuring out the number for all of them\n+we can just use \"setup\" as a substring/glob to match against the test\n+description:\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,9-11\n+\n+or one could select both the setup tests and the rename ones (assuming all\n+relevant tests had those words in their descriptions):\n+\n+ $ sh ./t0050-filesystem.sh --run=setup,rename\n+\n Some tests in a test suite rely on the previous tests performing\n certain actions, specifically some tests are designated as\n \"setup\" test, so you cannot _arbitrarily_ disable one test and\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 923281af93..62a16ed4c8 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -430,7 +430,7 @@ test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' \"\n \n test_expect_success '--run basic' \"\n \trun_sub_test_lib_test run-basic \\\n-\t\t'--run basic' --run='1 3 5' <<-\\\\EOF &&\n+\t\t'--run basic' --run='1,3,5' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -472,7 +472,7 @@ test_expect_success '--run with a range' \"\n \n test_expect_success '--run with two ranges' \"\n \trun_sub_test_lib_test run-two-ranges \\\n-\t\t'--run with two ranges' --run='1-2 5-6' <<-\\\\EOF &&\n+\t\t'--run with two ranges' --run='1-2,5-6' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -556,7 +556,7 @@ test_expect_success '--run with basic negation' \"\n \n test_expect_success '--run with two negations' \"\n \trun_sub_test_lib_test run-two-neg \\\n-\t\t'--run with two negations' --run='\"'!3 !6'\"' <<-\\\\EOF &&\n+\t\t'--run with two negations' --run='\"'!3,!6'\"' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -577,7 +577,7 @@ test_expect_success '--run with two negations' \"\n \n test_expect_success '--run a range and negation' \"\n \trun_sub_test_lib_test run-range-and-neg \\\n-\t\t'--run a range and negation' --run='\"'-4 !2'\"' <<-\\\\EOF &&\n+\t\t'--run a range and negation' --run='\"'-4,!2'\"' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -620,7 +620,7 @@ test_expect_success '--run range negation' \"\n test_expect_success '--run include, exclude and include' \"\n \trun_sub_test_lib_test run-inc-neg-inc \\\n \t\t'--run include, exclude and include' \\\n-\t\t--run='\"'1-5 !1-3 2'\"' <<-\\\\EOF &&\n+\t\t--run='\"'1-5,!1-3,2'\"' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -664,7 +664,7 @@ test_expect_success '--run include, exclude and include, comma separated' \"\n test_expect_success '--run exclude and include' \"\n \trun_sub_test_lib_test run-neg-inc \\\n \t\t'--run exclude and include' \\\n-\t\t--run='\"'!3- 5'\"' <<-\\\\EOF &&\n+\t\t--run='\"'!3-,5'\"' <<-\\\\EOF &&\n \tfor i in 1 2 3 4 5 6\n \tdo\n \t\ttest_expect_success \\\"passing test #\\$i\\\" 'true'\n@@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' \"\n \tEOF\n \"\n \n-test_expect_success '--run invalid range start' \"\n+test_expect_success '--run substring selector' \"\n+\trun_sub_test_lib_test run-substring-selector \\\n+\t\t'--run empty selectors' \\\n+\t\t--run='relevant' <<-\\\\EOF &&\n+\ttest_expect_success \\\"relevant test\\\" 'true'\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\ttest_expect_success \\\"other test #\\$i\\\" 'true'\n+\tdone\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test run-substring-selector <<-\\\\EOF\n+\t> ok 1 - relevant test\n+\t> ok 2 # skip other test #1 (--run)\n+\t> ok 3 # skip other test #2 (--run)\n+\t> ok 4 # skip other test #3 (--run)\n+\t> ok 5 # skip other test #4 (--run)\n+\t> ok 6 # skip other test #5 (--run)\n+\t> ok 7 # skip other test #6 (--run)\n+\t> # passed all 7 test(s)\n+\t> 1..7\n+\tEOF\n+\"\n+\n+test_expect_success '--run keyword selection' \"\n \trun_sub_test_lib_test_err run-inv-range-start \\\n \t\t'--run invalid range start' \\\n \t\t--run='a-5' <<-\\\\EOF &&\n@@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' \"\n \tEOF_ERR\n \"\n \n-test_expect_success '--run invalid selector' \"\n-\trun_sub_test_lib_test_err run-inv-selector \\\n-\t\t'--run invalid selector' \\\n-\t\t--run='1?' <<-\\\\EOF &&\n-\ttest_expect_success \\\"passing test #1\\\" 'true'\n-\ttest_done\n-\tEOF\n-\tcheck_sub_test_lib_test_err run-inv-selector \\\n-\t\t<<-\\\\EOF_OUT 3<<-\\\\EOF_ERR\n-\t> FATAL: Unexpected exit with code 1\n-\tEOF_OUT\n-\t> error: --run: invalid non-numeric in test selector: '1?'\n-\tEOF_ERR\n-\"\n-\n \n test_set_prereq HAVEIT\n haveit=no\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..72b88dffa1 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -769,15 +769,17 @@ match_pattern_list () {\n }\n \n match_test_selector_list () {\n+\toperation=\"$1\"\n+\tshift\n \ttitle=\"$1\"\n \tshift\n \targ=\"$1\"\n \tshift\n \ttest -z \"$1\" && return 0\n \n-\t# Both commas and whitespace are accepted as separators.\n+\t# Commas are accepted as separators.\n \tOLDIFS=$IFS\n-\tIFS=' \t,'\n+\tIFS=','\n \tset -- $1\n \tIFS=$OLDIFS\n \n@@ -805,13 +807,13 @@ match_test_selector_list () {\n \t\t\t*-*)\n \t\t\t\tif expr \"z${selector%%-*}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"start: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n \t\t\t\tif expr \"z${selector#*-}\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in range\" \\\n+\t\t\t\t\techo \"error: $operation: invalid non-numeric in range\" \\\n \t\t\t\t\t\t\"end: '$orig_selector'\" >&2\n \t\t\t\t\texit 1\n \t\t\t\tfi\n@@ -819,9 +821,11 @@ match_test_selector_list () {\n \t\t\t*)\n \t\t\t\tif expr \"z$selector\" : \"z[0-9]*[^0-9]\" >/dev/null\n \t\t\t\tthen\n-\t\t\t\t\techo \"error: $title: invalid non-numeric in test\" \\\n-\t\t\t\t\t\t\"selector: '$orig_selector'\" >&2\n-\t\t\t\t\texit 1\n+\t\t\t\t\tcase \"$title\" in *${selector}*)\n+\t\t\t\t\t\tinclude=$positive\n+\t\t\t\t\t\t;;\n+\t\t\t\t\tesac\n+\t\t\t\t\tcontinue\n \t\t\t\tfi\n \t\tesac\n \n@@ -1031,7 +1035,7 @@ test_skip () {\n \t\tskipped_reason=\"GIT_SKIP_TESTS\"\n \tfi\n \tif test -z \"$to_skip\" && test -n \"$run_list\" &&\n-\t ! match_test_selector_list '--run' $test_count \"$run_list\"\n+\t ! match_test_selector_list '--run' \"$1\" $test_count \"$run_list\"\n \tthen\n \t\tto_skip=t\n \t\tskipped_reason=\"--run\"\n-- \ngitgitgadget\n\n"},{"id":"407826","messageId":"3cc9abf461e54857aba87ce98b6c64d6b5e76c91.1602980628.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v6.git.git.1602980628.gitgitgadget@gmail.com","subject":"[PATCH v6 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-18T00:23:46Z","receivedAt":"2020-10-18T00:23:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWith the new ability to pass --run=setup to select which tests to run,\nit is more convenient if tests use the term \"setup\" instead of synonyms\nlike 'prepare' or 'rebuild'. There are undoubtedly many other tests in\nour testsuite that could be changed over too, these are just a couple\nthat I ran into.\n\nSigned-off-by: Elijah Newren \n---\n t/t6006-rev-list-format.sh | 2 +-\n t/t6012-rev-list-simplify.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh\nindex bc95da8a5f..99a1eaf332 100755\n--- a/t/t6006-rev-list-format.sh\n+++ b/t/t6006-rev-list-format.sh\n@@ -339,7 +339,7 @@ commit $head1\n .. (hinzugef${added_utf8_part_iso88591}gt) foo\n EOF\n \n-test_expect_success 'prepare expected messages (for test %b)' '\n+test_expect_success 'setup expected messages (for test %b)' '\n \tcat <<-EOF >expected.utf-8 &&\n \tcommit $head3\n \tThis commit message is much longer than the others,\ndiff --git a/t/t6012-rev-list-simplify.sh b/t/t6012-rev-list-simplify.sh\nindex b6fa43ace0..7254060240 100755\n--- a/t/t6012-rev-list-simplify.sh\n+++ b/t/t6012-rev-list-simplify.sh\n@@ -168,7 +168,7 @@ test_expect_success '--full-diff is not affected by --parents' '\n #\n # This example is explained in Documentation/rev-list-options.txt\n \n-test_expect_success 'rebuild repo' '\n+test_expect_success 'setup rebuild repo' '\n \trm -rf .git * &&\n \tgit init &&\n \tgit switch -c main &&\n-- \ngitgitgadget\n\n"},{"id":"407828","messageId":"0af783548744dc5e2a0e92040f166f80ea0500bf.1602980628.git.gitgitgadget@gmail.com","threadId":"54403","inReplyTo":"pull.878.v6.git.git.1602980628.gitgitgadget@gmail.com","subject":"[PATCH v6 3/3] test-lib: reduce verbosity of skipped tests","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-18T00:23:47Z","receivedAt":"2020-10-18T00:24:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen using the --run flag to run just two or three tests from a test\nfile which contains several dozen tests, having every skipped test print\nout dozens of lines of output for the test code for that skipped test\n(in addition to the TAP output line) adds up to hundreds or thousands of\nlines of irrelevant output that make it very hard to fish out the\nrelevant results you were looking for. Simplify the output for skipped\ntests to remove this extra output, leaving only the TAP output line\n(i.e. the line reading \"ok # skip \", which\nalready mentions that the test was \"skip\"ped).\n\nSigned-off-by: Elijah Newren \n---\n t/test-lib.sh | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 72b88dffa1..f68bca745a 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1062,7 +1062,6 @@ test_skip () {\n \t\t\t\t\" \"\n \t\tfi\n \n-\t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n-- \ngitgitgadget\n"}]}