{"thread":{"id":"36034","subject":"[PATCH 1/2] test-lib: tests skipped by GIT_SKIP_TESTS say so","startedAt":"2014-03-03T10:24:57Z","lastAt":"2014-03-04T19:01:51Z","messageCount":17,"participants":["Ilya Bobyr","Philip Oakley","Eric Sunshine","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"235864","messageId":"1393842298-5944-1-git-send-email-ilya.bobyr@gmail.com","threadId":"36034","inReplyTo":null,"subject":"[PATCH 1/2] test-lib: tests skipped by GIT_SKIP_TESTS say so","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-03-03T10:24:57Z","receivedAt":"2014-03-03T10:24:57Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"We used to show \"(missing )\" next to tests skipped because they are\nspecified in GIT_SKIP_TESTS. Use \"(matched by GIT_SKIP_TESTS)\" instead.\n---\n t/test-lib.sh | 13 ++++++++-----\n 1 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 1531c24..89a405b 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -446,25 +446,28 @@ test_finish_ () {\n \n test_skip () {\n \tto_skip=\n+\tskipped_reason=\n \tif match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS\n \tthen\n \t\tto_skip=t\n+\t\tskipped_reason=\"matched GIT_SKIP_TESTS\"\n \tfi\n \tif test -z \"$to_skip\" && test -n \"$test_prereq\" &&\n \t ! test_have_prereq \"$test_prereq\"\n \tthen\n \t\tto_skip=t\n-\tfi\n-\tcase \"$to_skip\" in\n-\tt)\n+\n \t\tof_prereq=\n \t\tif test \"$missing_prereq\" != \"$test_prereq\"\n \t\tthen\n \t\t\tof_prereq=\" of $test_prereq\"\n \t\tfi\n-\n+\t\tskipped_reason=\"missing $missing_prereq${of_prereq}\"\n+\tfi\n+\tcase \"$to_skip\" in\n+\tt)\n \t\tsay_color skip >&3 \"skipping test: $@\"\n-\t\tsay_color skip \"ok $test_count # skip $1 (missing $missing_prereq${of_prereq})\"\n+\t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n \t\t;;\n \t*)\n-- \n1.7.9\n"},{"id":"235865","messageId":"1393842298-5944-2-git-send-email-ilya.bobyr@gmail.com","threadId":"36034","inReplyTo":"1393842298-5944-1-git-send-email-ilya.bobyr@gmail.com","subject":"[PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Ilya Bobyr","fromEmail":"ilya.bobyr@gmail.com","sentAt":"2014-03-03T10:24:58Z","receivedAt":"2014-03-03T10:24:58Z","isPatch":true,"sender":{"key":"ilya.bobyr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/694419?v=4"},"body":"This is a counterpart to GIT_SKIP_TESTS. Mostly useful when debugging.\n---\n t/README | 15 +++++++++++++++\n t/test-lib.sh | 8 ++++++++\n 2 files changed, 23 insertions(+), 0 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex caeeb9d..f939987 100644\n--- a/t/README\n+++ b/t/README\n@@ -187,6 +187,21 @@ and either can match the \"t[0-9]{4}\" part to skip the whole\n test, or t[0-9]{4} followed by \".$number\" to say which\n particular test to skip.\n \n+Sometimes the opposite is desired - ability to execute only one or\n+several tests. Mostly while debugging tests. For that you can say\n+\n+ $ GIT_TEST_ONLY=t9200.8 sh ./t9200-git-cvsexport-commit.sh\n+\n+or, similrary to GIT_SKIP_TESTS\n+\n+ $ GIT_TEST_ONLY='t[0-4]??? t91?? t9200.8' make\n+\n+In additiona to matching against \".\"\n+GIT_TEST_ONLY is matched against just the test numbes. This comes\n+handy when you are running only one test:\n+\n+ $ GIT_TEST_ONLY='[0-8]' sh ./t9200-git-cvsexport-commit.sh\n+\n Note that some tests in the existing test suite rely on previous\n test item, so you cannot arbitrarily disable one and expect the\n remainder of test to check what the test originally was intended\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 89a405b..12bf436 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -464,6 +464,14 @@ test_skip () {\n \t\tfi\n \t\tskipped_reason=\"missing $missing_prereq${of_prereq}\"\n \tfi\n+\tif test -z \"$to_skip\" && test -n \"$GIT_TEST_ONLY\" &&\n+\t\t! match_pattern_list $this_test.$test_count $GIT_TEST_ONLY &&\n+\t\t! match_pattern_list $test_count $GIT_TEST_ONLY\n+\tthen\n+\t\tto_skip=t\n+\t\tskipped_reason=\"not in GIT_TEST_ONLY\"\n+\tfi\n+\n \tcase \"$to_skip\" in\n \tt)\n \t\tsay_color skip >&3 \"skipping test: $@\"\n-- \n1.7.9\n"},{"id":"235878","messageId":"C65A743B56974711A3806272E9B2C1DD@PhilipOakley","threadId":"36034","inReplyTo":"1393842298-5944-1-git-send-email-ilya.bobyr@gmail.com","subject":"Re: [PATCH 1/2] test-lib: tests skipped by GIT_SKIP_TESTS say so","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2014-03-03T14:47:46Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Ilya Bobyr\" \n> We used to show \"(missing )\" next to tests skipped because they are\n> specified in GIT_SKIP_TESTS. Use \"(matched by GIT_SKIP_TESTS)\" \n> instead.\n\nThe message below forgets the \"by\". Otherwise looks sensible.\n\n> ---\n> t/test-lib.sh | 13 ++++++++-----\n> 1 files changed, 8 insertions(+), 5 deletions(-)\n>\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 1531c24..89a405b 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -446,25 +446,28 @@ test_finish_ () {\n>\n> test_skip () {\n> to_skip=\n> + skipped_reason=\n> if match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS\n> then\n> to_skip=t\n> + skipped_reason=\"matched GIT_SKIP_TESTS\"\n\ns/matched GIT_SKIP_TESTS/matched by GIT_SKIP_TESTS/\n\n> fi\n> if test -z \"$to_skip\" && test -n \"$test_prereq\" &&\n> ! test_have_prereq \"$test_prereq\"\n> then\n> to_skip=t\n> - fi\n> - case \"$to_skip\" in\n> - t)\n> +\n> of_prereq=\n> if test \"$missing_prereq\" != \"$test_prereq\"\n> then\n> of_prereq=\" of $test_prereq\"\n> fi\n> -\n> + skipped_reason=\"missing $missing_prereq${of_prereq}\"\n> + fi\n> + case \"$to_skip\" in\n> + t)\n> say_color skip >&3 \"skipping test: $@\"\n> - say_color skip \"ok $test_count # skip $1 (missing \n> $missing_prereq${of_prereq})\"\n> + say_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n> : true\n> ;;\n> *)\n> -- \n> 1.7.9\n"},{"id":"235880","messageId":"A85165F7C59F4572AB131A02D0E09698@PhilipOakley","threadId":"36034","inReplyTo":"1393842298-5944-2-git-send-email-ilya.bobyr@gmail.com","subject":"Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2014-03-03T15:23:47Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Minor nits.\n\nFrom: \"Ilya Bobyr\" \n> This is a counterpart to GIT_SKIP_TESTS. Mostly useful when \n> debugging.\n> ---\n> t/README | 15 +++++++++++++++\n> t/test-lib.sh | 8 ++++++++\n> 2 files changed, 23 insertions(+), 0 deletions(-)\n>\n> diff --git a/t/README b/t/README\n> index caeeb9d..f939987 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -187,6 +187,21 @@ and either can match the \"t[0-9]{4}\" part to skip \n> the whole\n> test, or t[0-9]{4} followed by \".$number\" to say which\n> particular test to skip.\n>\n> +Sometimes the opposite is desired - ability to execute only one or\n> +several tests. Mostly while debugging tests. For that you can say\n> +\n> + $ GIT_TEST_ONLY=t9200.8 sh ./t9200-git-cvsexport-commit.sh\n> +\n> +or, similrary to GIT_SKIP_TESTS\n> +\n> + $ GIT_TEST_ONLY='t[0-4]??? t91?? t9200.8' make\n> +\n> +In additiona to matching against \".\"\n> +GIT_TEST_ONLY is matched against just the test . This comes\ns/numbes/numbers/\n\n> +handy when you are running only one test:\ns/handy/in handy/\n\n> +\n> + $ GIT_TEST_ONLY='[0-8]' sh ./t9200-git-cvsexport-commit.sh\n> +\n> Note that some tests in the existing test suite rely on previous\n> test item, so you cannot arbitrarily disable one and expect the\n> remainder of test to check what the test originally was intended\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 89a405b..12bf436 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -464,6 +464,14 @@ test_skip () {\n> fi\n> skipped_reason=\"missing $missing_prereq${of_prereq}\"\n> fi\n> + if test -z \"$to_skip\" && test -n \"$GIT_TEST_ONLY\" &&\n> + ! match_pattern_list $this_test.$test_count $GIT_TEST_ONLY &&\n> + ! match_pattern_list $test_count $GIT_TEST_ONLY\n> + then\n> + to_skip=t\n> + skipped_reason=\"not in GIT_TEST_ONLY\"\n> + fi\n> +\n> case \"$to_skip\" in\n> t)\n> say_color skip >&3 \"skipping test: $@\"\n> -- \nPhilip \n"},{"id":"235948","messageId":"CAPig+cQgYOz7rDax=HFNG9mr-H1FaoL-ss2mgdpMazPS4pWbNQ@mail.gmail.com","threadId":"36034","inReplyTo":"1393842298-5944-1-git-send-email-ilya.bobyr@gmail.com","subject":"Re: [PATCH 1/2] test-lib: tests skipped by GIT_SKIP_TESTS say so","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2014-03-03T22:59:15Z","receivedAt":"2014-03-03T22:59:15Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr wrote:\n> We used to show \"(missing )\" next to tests skipped because they are\n> specified in GIT_SKIP_TESTS. Use \"(matched by GIT_SKIP_TESTS)\" instead.\n\nBikeshedding: That's pretty verbose. Perhaps just say \"(excluded)\"?\n\n> ---\n> t/test-lib.sh | 13 ++++++++-----\n> 1 files changed, 8 insertions(+), 5 deletions(-)\n>\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 1531c24..89a405b 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -446,25 +446,28 @@ test_finish_ () {\n>\n> test_skip () {\n> to_skip=\n> + skipped_reason=\n> if match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS\n> then\n> to_skip=t\n> + skipped_reason=\"matched GIT_SKIP_TESTS\"\n> fi\n> if test -z \"$to_skip\" && test -n \"$test_prereq\" &&\n> ! test_have_prereq \"$test_prereq\"\n> then\n> to_skip=t\n> - fi\n> - case \"$to_skip\" in\n> - t)\n> +\n> of_prereq=\n> if test \"$missing_prereq\" != \"$test_prereq\"\n> then\n> of_prereq=\" of $test_prereq\"\n> fi\n> -\n> + skipped_reason=\"missing $missing_prereq${of_prereq}\"\n> + fi\n> + case \"$to_skip\" in\n> + t)\n> say_color skip >&3 \"skipping test: $@\"\n> - say_color skip \"ok $test_count # skip $1 (missing $missing_prereq${of_prereq})\"\n> + say_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n> : true\n> ;;\n> *)\n> --\n> 1.7.9\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"235951","messageId":"CAPig+cT46ekT87TuoTtwvt0G+DraB9cdgW1wd6NsodjJ5FmJrA@mail.gmail.com","threadId":"36034","inReplyTo":"1393842298-5944-2-git-send-email-ilya.bobyr@gmail.com","subject":"Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2014-03-03T23:03:55Z","receivedAt":"2014-03-03T23:03:55Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr wrote:\n> This is a counterpart to GIT_SKIP_TESTS. Mostly useful when debugging.\n\nTo be grammatically similar to GIT_SKIP_TESTS, perhaps name it GIT_RUN_TESTS?\n\n> ---\n> t/README | 15 +++++++++++++++\n> t/test-lib.sh | 8 ++++++++\n> 2 files changed, 23 insertions(+), 0 deletions(-)\n>\n> diff --git a/t/README b/t/README\n> index caeeb9d..f939987 100644\n> --- a/t/README\n> +++ b/t/README\n> @@ -187,6 +187,21 @@ and either can match the \"t[0-9]{4}\" part to skip the whole\n> test, or t[0-9]{4} followed by \".$number\" to say which\n> particular test to skip.\n>\n> +Sometimes the opposite is desired - ability to execute only one or\n> +several tests. Mostly while debugging tests. For that you can say\n> +\n> + $ GIT_TEST_ONLY=t9200.8 sh ./t9200-git-cvsexport-commit.sh\n> +\n> +or, similrary to GIT_SKIP_TESTS\n> +\n> + $ GIT_TEST_ONLY='t[0-4]??? t91?? t9200.8' make\n> +\n> +In additiona to matching against \".\"\n\ns/additiona/addition/\n\nPlus the other typos already mentioned by Philip...\n\n> +GIT_TEST_ONLY is matched against just the test numbes. This comes\n> +handy when you are running only one test:\n> +\n> + $ GIT_TEST_ONLY='[0-8]' sh ./t9200-git-cvsexport-commit.sh\n> +\n> Note that some tests in the existing test suite rely on previous\n> test item, so you cannot arbitrarily disable one and expect the\n> remainder of test to check what the test originally was intended\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 89a405b..12bf436 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -464,6 +464,14 @@ test_skip () {\n> fi\n> skipped_reason=\"missing $missing_prereq${of_prereq}\"\n> fi\n> + if test -z \"$to_skip\" && test -n \"$GIT_TEST_ONLY\" &&\n> + ! match_pattern_list $this_test.$test_count $GIT_TEST_ONLY &&\n> + ! match_pattern_list $test_count $GIT_TEST_ONLY\n> + then\n> + to_skip=t\n> + skipped_reason=\"not in GIT_TEST_ONLY\"\n> + fi\n> +\n> case \"$to_skip\" in\n> t)\n> say_color skip >&3 \"skipping test: $@\"\n> --\n> 1.7.9\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"235952","messageId":"53150B8A.9050709@gmail.com","threadId":"36034","inReplyTo":"C65A743B56974711A3806272E9B2C1DD@PhilipOakley","subject":"Re: [PATCH 1/2] test-lib: tests skipped by GIT_SKIP_TESTS say so","fromName":"Ilya Bobyr","fromEmail":"ilya.bobir@gmail.com","sentAt":"2014-03-03T23:08:58Z","receivedAt":"2014-03-03T23:08:58Z","isPatch":true,"sender":{"key":"ilya.bobir@gmail.com","avatar":null},"body":"On 3/3/2014 7:11 AM, Philip Oakley wrote:\n> From: \"Ilya Bobyr\" \n>> We used to show \"(missing )\" next to tests skipped because they are\n>> specified in GIT_SKIP_TESTS. Use \"(matched by GIT_SKIP_TESTS)\" instead.\n>\n> The message below forgets the \"by\".\n\nI'll fix the commit message. I think the output is long enough, while \n\"by\" does not add any information.\n\n> Otherwise looks sensible.\n\nThanks for looking at it :)\n\n>\n>> ---\n>> t/test-lib.sh | 13 ++++++++-----\n>> 1 files changed, 8 insertions(+), 5 deletions(-)\n>>\n>> diff --git a/t/test-lib.sh b/t/test-lib.sh\n>> index 1531c24..89a405b 100644\n>> --- a/t/test-lib.sh\n>> +++ b/t/test-lib.sh\n>> @@ -446,25 +446,28 @@ test_finish_ () {\n>>\n>> test_skip () {\n>> to_skip=\n>> + skipped_reason=\n>> if match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS\n>> then\n>> to_skip=t\n>> + skipped_reason=\"matched GIT_SKIP_TESTS\"\n>\n> s/matched GIT_SKIP_TESTS/matched by GIT_SKIP_TESTS/\n>\n>> [...]\n"},{"id":"235953","messageId":"53150C61.90500@gmail.com","threadId":"36034","inReplyTo":"CAPig+cQgYOz7rDax=HFNG9mr-H1FaoL-ss2mgdpMazPS4pWbNQ@mail.gmail.com","subject":"Re: [PATCH 1/2] test-lib: tests skipped by GIT_SKIP_TESTS say so","fromName":"Ilya Bobyr","fromEmail":"ilya.bobir@gmail.com","sentAt":"2014-03-03T23:12:33Z","receivedAt":"2014-03-03T23:12:33Z","isPatch":true,"sender":{"key":"ilya.bobir@gmail.com","avatar":null},"body":"On 3/3/2014 2:59 PM, Eric Sunshine wrote:\n> On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr wrote:\n>> We used to show \"(missing )\" next to tests skipped because they are\n>> specified in GIT_SKIP_TESTS. Use \"(matched by GIT_SKIP_TESTS)\" instead.\n> Bikeshedding: That's pretty verbose. Perhaps just say \"(excluded)\"?\n\nThe next patch adds another reason for the test to be skipped, so it \nseems reasonable to say why exactly.\nThe patch actually makes it say \"matched GIT_SKIP_TESTS\".\nIt looks OK on the console.\n"},{"id":"235954","messageId":"xmqqsiqynae4.fsf@gitster.dls.corp.google.com","threadId":"36034","inReplyTo":"CAPig+cQgYOz7rDax=HFNG9mr-H1FaoL-ss2mgdpMazPS4pWbNQ@mail.gmail.com","subject":"Re: [PATCH 1/2] test-lib: tests skipped by GIT_SKIP_TESTS say so","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-03T23:13:39Z","receivedAt":"2014-03-03T23:13:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr wrote:\n>> We used to show \"(missing )\" next to tests skipped because they are\n>> specified in GIT_SKIP_TESTS. Use \"(matched by GIT_SKIP_TESTS)\" instead.\n>\n> Bikeshedding: That's pretty verbose. Perhaps just say \"(excluded)\"?\n\nSounds good, or at least better than matched GIT_SKIP_TESTS, to me ;-).\n\nThanks.\n"},{"id":"235955","messageId":"53150D68.7070803@gmail.com","threadId":"36034","inReplyTo":"CAPig+cT46ekT87TuoTtwvt0G+DraB9cdgW1wd6NsodjJ5FmJrA@mail.gmail.com","subject":"Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Ilya Bobyr","fromEmail":"ilya.bobir@gmail.com","sentAt":"2014-03-03T23:16:56Z","receivedAt":"2014-03-03T23:16:56Z","isPatch":true,"sender":{"key":"ilya.bobir@gmail.com","avatar":null},"body":"On 3/3/2014 3:03 PM, Eric Sunshine wrote:\n> On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr wrote:\n>> This is a counterpart to GIT_SKIP_TESTS. Mostly useful when debugging.\n> To be grammatically similar to GIT_SKIP_TESTS, perhaps name it GIT_RUN_TESTS?\n\nThere is actually an upside in the fact that the name is \"different \nenough\". When you pull a command from a history it is easier to see if \nit is the excluding or the including one.\nMaybe we can have a third opinion here?\n\n>> ---\n>> t/README | 15 +++++++++++++++\n>> t/test-lib.sh | 8 ++++++++\n>> 2 files changed, 23 insertions(+), 0 deletions(-)\n>>\n>> diff --git a/t/README b/t/README\n>> index caeeb9d..f939987 100644\n>> --- a/t/README\n>> +++ b/t/README\n>> @@ -187,6 +187,21 @@ and either can match the \"t[0-9]{4}\" part to skip the whole\n>> test, or t[0-9]{4} followed by \".$number\" to say which\n>> particular test to skip.\n>>\n>> +Sometimes the opposite is desired - ability to execute only one or\n>> +several tests. Mostly while debugging tests. For that you can say\n>> +\n>> + $ GIT_TEST_ONLY=t9200.8 sh ./t9200-git-cvsexport-commit.sh\n>> +\n>> +or, similrary to GIT_SKIP_TESTS\n>> +\n>> + $ GIT_TEST_ONLY='t[0-4]??? t91?? t9200.8' make\n>> +\n>> +In additiona to matching against \".\"\n> s/additiona/addition/\n>\n> Plus the other typos already mentioned by Philip...\n\nThank you. I will include all of those in the next version of the patch.\n\n> [...]\n"},{"id":"235959","messageId":"xmqqob1mn9t6.fsf@gitster.dls.corp.google.com","threadId":"36034","inReplyTo":"CAPig+cT46ekT87TuoTtwvt0G+DraB9cdgW1wd6NsodjJ5FmJrA@mail.gmail.com","subject":"Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-03T23:26:13Z","receivedAt":"2014-03-03T23:26:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr wrote:\n>> This is a counterpart to GIT_SKIP_TESTS. Mostly useful when debugging.\n>\n> To be grammatically similar to GIT_SKIP_TESTS, perhaps name it GIT_RUN_TESTS?\n\nI actually do not like the interface to use two variables very much.\nCan't we just allow negative entries on \"to be skipped\" list?\n\nThat is\n\n\tGIT_SKIP_TESTS='t9??? !t91??'\n\nwould skip nine-thousand series, but would run 91xx series, and all\nthe others are not excluded.\n\nSimple rules to consider:\n\n - If the list consists of _only_ negated patterns, pretend that\n there is \"unless otherwise specified with negatives, skip all\n tests\", i.e. treat GIT_SKIP_TESTS='!t91??' just the same way you\n would treat GIT_SKIP_TESTS='* !t91??'.\n\n - The orders should not matter for simplicity of the semantics;\n before running each test, check if it matches any negative (and\n run it if it matches, without looking at any positives), and\n otherwise check if it matches any positive (and skip it if it\n does not).\n\nHmm?\n\n>> ---\n>> t/README | 15 +++++++++++++++\n>> t/test-lib.sh | 8 ++++++++\n>> 2 files changed, 23 insertions(+), 0 deletions(-)\n>>\n>> diff --git a/t/README b/t/README\n>> index caeeb9d..f939987 100644\n>> --- a/t/README\n>> +++ b/t/README\n>> @@ -187,6 +187,21 @@ and either can match the \"t[0-9]{4}\" part to skip the whole\n>> test, or t[0-9]{4} followed by \".$number\" to say which\n>> particular test to skip.\n>>\n>> +Sometimes the opposite is desired - ability to execute only one or\n>> +several tests. Mostly while debugging tests. For that you can say\n>> +\n>> + $ GIT_TEST_ONLY=t9200.8 sh ./t9200-git-cvsexport-commit.sh\n>> +\n>> +or, similrary to GIT_SKIP_TESTS\n>> +\n>> + $ GIT_TEST_ONLY='t[0-4]??? t91?? t9200.8' make\n>> +\n>> +In additiona to matching against \".\"\n>\n> s/additiona/addition/\n>\n> Plus the other typos already mentioned by Philip...\n>\n>> +GIT_TEST_ONLY is matched against just the test numbes. This comes\n>> +handy when you are running only one test:\n>> +\n>> + $ GIT_TEST_ONLY='[0-8]' sh ./t9200-git-cvsexport-commit.sh\n>> +\n>> Note that some tests in the existing test suite rely on previous\n>> test item, so you cannot arbitrarily disable one and expect the\n>> remainder of test to check what the test originally was intended\n>> diff --git a/t/test-lib.sh b/t/test-lib.sh\n>> index 89a405b..12bf436 100644\n>> --- a/t/test-lib.sh\n>> +++ b/t/test-lib.sh\n>> @@ -464,6 +464,14 @@ test_skip () {\n>> fi\n>> skipped_reason=\"missing $missing_prereq${of_prereq}\"\n>> fi\n>> + if test -z \"$to_skip\" && test -n \"$GIT_TEST_ONLY\" &&\n>> + ! match_pattern_list $this_test.$test_count $GIT_TEST_ONLY &&\n>> + ! match_pattern_list $test_count $GIT_TEST_ONLY\n>> + then\n>> + to_skip=t\n>> + skipped_reason=\"not in GIT_TEST_ONLY\"\n>> + fi\n>> +\n>> case \"$to_skip\" in\n>> t)\n>> say_color skip >&3 \"skipping test: $@\"\n>> --\n>> 1.7.9\n>>\n>> --\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"235964","messageId":"531514D9.4000101@gmail.com","threadId":"36034","inReplyTo":"xmqqob1mn9t6.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Ilya Bobyr","fromEmail":"ilya.bobir@gmail.com","sentAt":"2014-03-03T23:48:41Z","receivedAt":"2014-03-03T23:48:41Z","isPatch":true,"sender":{"key":"ilya.bobir@gmail.com","avatar":null},"body":"On 3/3/2014 3:26 PM, Junio C Hamano wrote:\n> Eric Sunshine writes:\n>\n>> On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr wrote:\n>>> This is a counterpart to GIT_SKIP_TESTS. Mostly useful when debugging.\n>> To be grammatically similar to GIT_SKIP_TESTS, perhaps name it GIT_RUN_TESTS?\n> I actually do not like the interface to use two variables very much.\n> Can't we just allow negative entries on \"to be skipped\" list?\n>\n> That is\n>\n> \tGIT_SKIP_TESTS='t9??? !t91??'\n>\n> would skip nine-thousand series, but would run 91xx series, and all\n> the others are not excluded.\n>\n> Simple rules to consider:\n>\n> - If the list consists of _only_ negated patterns, pretend that\n> there is \"unless otherwise specified with negatives, skip all\n> tests\", i.e. treat GIT_SKIP_TESTS='!t91??' just the same way you\n> would treat GIT_SKIP_TESTS='* !t91??'.\n>\n> - The orders should not matter for simplicity of the semantics;\n> before running each test, check if it matches any negative (and\n> run it if it matches, without looking at any positives), and\n> otherwise check if it matches any positive (and skip it if it\n> does not).\n>\n> Hmm?\n\nI can do that. But I am not sure that matches the use cases I had in \nmind the best.\n\nFirst use case is that while developing I want to run tests frequently \nand I have a specific test that I am working on at the moment.\nThat test is broken and I am trying to fix it (TDD).\nI want to run just the initialization test(s) and then that specific test.\nRunning everything is quite slow.\n\nGIT_RUN_ONLY addresses the TDD case.\n\nSecond case is when I broke one or more tests and want to figure out \nwhat is wrong.\nIn this case running tests after the broken one will clutter the output \ndirectory and will make debugging somewhat harder, especially if I am \nnot familiar with all the tests.\n\nFor the second case I was actually thinking that something like \n\" writes:\n\n> It might be that we are looking at different use cases, as you are\n> talking about whole test suits.\n\nI do not think so.\n\nI do not see anything prevents you from saying\n\n\tGIT_SKIP_TESTS='t0000 !t0000.1 !t0000.4'\n\nto specify test-pieces in individual tests so that you can run the\nsetup step (step .1) and the specific test (step .4) without running\ntwo tests in between.\n"},{"id":"235975","messageId":"53157B72.3000205@gmail.com","threadId":"36034","inReplyTo":"xmqq4n3en7un.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Ilya Bobyr","fromEmail":"ilya.bobir@gmail.com","sentAt":"2014-03-04T07:06:26Z","receivedAt":"2014-03-04T07:06:26Z","isPatch":true,"sender":{"key":"ilya.bobir@gmail.com","avatar":null},"body":"On 3/3/2014 4:08 PM, Junio C Hamano wrote:\n> Ilya Bobyr writes:\n>\n>> It might be that we are looking at different use cases, as you are\n>> talking about whole test suits.\n> I do not think so.\n\nGood :)\nI am trying to understand the use cases. And make sure we are talking \nabout the same ones.\nI am not sure what are the use cases for GIT_SKIP_TESTS.\n\nI think that while in it really nice when an interface allows to do new \nthings, the main use case (or use cases) should be as easy and obvious \nin the first place.\nIf the target is the TDD use case I described, then it appears that a \nuser needs to do double negation. That is what concerns me.\n\n> I do not see anything prevents you from saying\n>\n> \tGIT_SKIP_TESTS='t0000 !t0000.1 !t0000.4'\n>\n> to specify test-pieces in individual tests so that you can run the\n> setup step (step .1) and the specific test (step .4) without running\n> two tests in between.\n\nWhile it could be done, it looks less obvious than this:\n\n GIT_TEST_ONLY='1 4' ./t0001-init.sh\n\nWhat if we do what you proposed, but with GIT_RUN_TESTS?\nIf we want to have one interface, maybe, building on top of a \"negation\" \n(GIT_SKIP_TESTS) is not very good.\nAt the same time, GIT_TEST_ONLY is also too specific for a generic \ninterface.\nI could add GIT_RUN_TESTS and allow it to have all of the features, thus \nmaking GIT_SKIP_TESTS a working but deprecated tool.\n\nRunning specific tests:\n GIT_RUN_TESTS='t0000'\n GIT_RUN_TESTS='t0000.1 t0002'\n GIT_RUN_TESTS='1 3 7'\n\nNegating some tests:\n GIT_RUN_TESTS='!t0000'\n GIT_RUN_TESTS='!t0000.1'\n GIT_RUN_TESTS='!1 !3 !7'\n\nThe above would work exactly as you described but with one less level of \nnegation. Default is everything, unless at least one positive pattern \nis given.\n\nLater on range specification could added:\n GIT_RUN_TESTS='<11'\n\nAt least for now that would cover most use cases that I can think of \nthat look reasonable.\n"},{"id":"235977","messageId":"xmqqvbvul62n.fsf@gitster.dls.corp.google.com","threadId":"36034","inReplyTo":"53157B72.3000205@gmail.com","subject":"Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-04T08:29:52Z","receivedAt":"2014-03-04T08:29:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ilya Bobyr writes:\n\n> While it could be done, it looks less obvious than this:\n>\n> GIT_TEST_ONLY='1 4' ./t0001-init.sh\n\nIf you are thinking about affecting only one test, then you\nshouldn't be mucking with environment variables in the first place,\nprimarily because running:\n\n $ GIT_TEST_ONLY='1 4' make test\n\nto run test .1 and .4 of all the test scripts would not make any\nsense. I think your \"simplicity\" argument is a total red-herring.\nOf course if you do not have to say the test script name, your\nspecification would be shorter, but that is only because your\nspecification is not specific enough to be useful.\n\nGiving that as a command line argument to the specific script, e.g.\n\n $ sh ./t0001-init.sh --only='1 4'\n\nmight make some sense, but the above GIT_TEST_ONLY does not make any\nsense from the UI point of view.\n\nThere are many reasons that makes me unenthused about this line of\nchange in the first place:\n\n * Both at the philosophical level and at the practical level, I've\n found that it always makes sense to run most of the tests, i.e.\n skipping ought to be an exception not the norm. Over the course\n of this project, I often saw an alleged fix to one part of the\n system introduces breakages that are caught by tests that checks\n parts of the system that does not have any superficial link to it\n (e.g. update the refs code and find a rebase test break).\n\n * Even though GIT_SKIP_TESTS mechanism still allows you to skip\n individual test pieces, it has never been a serious \"feature\" in\n the first place. Many of the tests unfortunately do rely on state\n previous sequences of tests left behind, so it is not realistic\n to expect that you can skip test pieces randomly and exercise\n later test pieces reliably.\n\n * The numbering of individual test pieces can easily change by new\n tests inserted in the middle; again, many tests do take advantge\n of the states earlier tests leave behind, so \"do not add new\n tests in the middle\" is not a realistic rule to enforce, unless\n you are willing to clean up existing test scripts so that each\n test piece is independent from all the previous ones.\n\nThe latter two makes the ability to skip individual test pieces a\n\"theoretically it could be made useful but practically not so much\"\nmisfeature. I am very hesitant to see the test framework code\nchurned only to enhance its \"usefulness\" when there isn't any in the\nfirst place, without first making changes that fundamentally\nimproves its usefulness (e.g. to solve \"test numbering is not\nstable\" problem, you could identify the tests with test names\ninstead of numbers to make it more stable, but that is not what your\npatch is even attempting to do).\n\nI view such a change as merely a code churn, damaging the codebase\nthat is already less nice than ideal and turning it more difficult\nto fix properly to make it truly nicer later.\n\nMy suggestion to cram everything into GIT_SKIP_TESTS is primarily\nbecause it is one way I can easily see how it allows us to limit the\n\"damage\" to the codebase--the suggested change could be contained\nwithin a single shell function \"match_pattern_list\" and no other\ncode has to change to support it. I am not saying it is the only\nway, but glancing at your patch, adding an extra environment\nvariable would need to also modify its callers, so the extent of the\ndamage to the codebase seemed somewhat larger.\n\nSo, I dunno. I am not yet enthused.\n"},{"id":"235981","messageId":"53159A55.70105@gmail.com","threadId":"36034","inReplyTo":"xmqqvbvul62n.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Ilya Bobyr","fromEmail":"ilya.bobir@gmail.com","sentAt":"2014-03-04T09:18:13Z","receivedAt":"2014-03-04T09:18:13Z","isPatch":true,"sender":{"key":"ilya.bobir@gmail.com","avatar":null},"body":"On 3/4/2014 12:29 AM, Junio C Hamano wrote:\n> Ilya Bobyr writes:\n>\n>> While it could be done, it looks less obvious than this:\n>>\n>> GIT_TEST_ONLY='1 4' ./t0001-init.sh\n> If you are thinking about affecting only one test,\n\nYes, that is the use case: when I am developing a specific feature I \nwant to run just one test for that feature over and over, while I am \nworking on that specific thing.\nNot the whole test suite (like \"t0001\"), but just the new case that I've \nadded to the end, for example. Plus one or more tests that setup enough \nenvironment for it.\n\n> then you\n> shouldn't be mucking with environment variables in the first place,\n> primarily because running:\n>\n> $ GIT_TEST_ONLY='1 4' make test\n>\n> to run test .1 and .4 of all the test scripts would not make any\n> sense.\n\nNo it does not. It only makes sense for one test suite.\n\n> I think your \"simplicity\" argument is a total red-herring.\n> Of course if you do not have to say the test script name, your\n> specification would be shorter, but that is only because your\n> specification is not specific enough to be useful.\n\nIn my case it is very useful :)\nThis is why I am saying that we might be talking about different cases: \nyou are talking about the test suite level, while the issue I am trying \nto address an issue at an individual test level.\n\n> Giving that as a command line argument to the specific script, e.g.\n>\n> $ sh ./t0001-init.sh --only='1 4'\n>\n> might make some sense, but the above GIT_TEST_ONLY does not make any\n> sense from the UI point of view.\n\nNo problem, I guess I can make it look like that - with '--only'.\nMaybe '--tests'? Then the same negation syntax could be used as \npreviously discussed.\nAs well as range syntax.\n\n> There are many reasons that makes me unenthused about this line of\n> change in the first place:\n>\n> * Both at the philosophical level and at the practical level, I've\n> found that it always makes sense to run most of the tests, i.e.\n> skipping ought to be an exception not the norm. Over the course\n> of this project, I often saw an alleged fix to one part of the\n> system introduces breakages that are caught by tests that checks\n> parts of the system that does not have any superficial link to it\n> (e.g. update the refs code and find a rebase test break).\n\nMy main argument is the time. When testing Git as a whole or a feature \nas a whole there is no reason to skip some tests.\nWhen working on a specific piece I may run the same test 100 times easily.\nHere is what I see on my Cygwin:\n\n $ time ./t0001-init.sh\n [...]\n 1..36\n\n real 0m6.693s\n user 0m1.505s\n sys 0m3.937s\n\n\n $ time GIT_SKIP_TESTS='t0001.[36789] t0001.??' ./t0001-init.sh\n [...]\n 1..36\n\n real 0m3.313s\n user 0m0.769s\n sys 0m1.844s\n\nSo skipping 34 tests that I am not interested in save a bit more that \n50% of the time.\nWhile it would be really nice if it would be faster, this speedup is a \npretty simple one.\n\n> * Even though GIT_SKIP_TESTS mechanism still allows you to skip\n> individual test pieces, it has never been a serious \"feature\" in\n> the first place. Many of the tests unfortunately do rely on state\n> previous sequences of tests left behind, so it is not realistic\n> to expect that you can skip test pieces randomly and exercise\n> later test pieces reliably.\n>\n> * The numbering of individual test pieces can easily change by new\n> tests inserted in the middle; again, many tests do take advantge\n> of the states earlier tests leave behind, so \"do not add new\n> tests in the middle\" is not a realistic rule to enforce, unless\n> you are willing to clean up existing test scripts so that each\n> test piece is independent from all the previous ones.\n\nBoth are true, but do not apply to the TDD case.\nNeither they apply to a case when a test is broken and I want to execute \neverything up to that test.\n\n> The latter two makes the ability to skip individual test pieces a\n> \"theoretically it could be made useful but practically not so much\"\n> misfeature. I am very hesitant to see the test framework code\n> churned only to enhance its \"usefulness\" when there isn't any in the\n> first place, without first making changes that fundamentally\n> improves its usefulness (e.g. to solve \"test numbering is not\n> stable\" problem, you could identify the tests with test names\n> instead of numbers to make it more stable, but that is not what your\n> patch is even attempting to do).\n\nIf you see a way to address my problems, I might be able to code it the \nway you want it to be.\n\n> [...]\n"},{"id":"236020","messageId":"xmqq1tyhlrds.fsf@gitster.dls.corp.google.com","threadId":"36034","inReplyTo":"53159A55.70105@gmail.com","subject":"Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-04T19:01:51Z","receivedAt":"2014-03-04T19:01:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ilya Bobyr writes:\n\n> On 3/4/2014 12:29 AM, Junio C Hamano wrote:\n> ...\n>> then you\n>> shouldn't be mucking with environment variables in the first place,\n>> primarily because running:\n>>\n>> $ GIT_TEST_ONLY='1 4' make test\n>>\n>> to run test .1 and .4 of all the test scripts would not make any\n>> sense.\n>\n> No it does not. It only makes sense for one test suite.\n>\n>> I think your \"simplicity\" argument is a total red-herring.\n>> Of course if you do not have to say the test script name, your\n>> specification would be shorter, but that is only because your\n>> specification is not specific enough to be useful.\n>\n> In my case it is very useful :)\n\nIt invites a nonsense usage (i.e. running \"make test\" under that\nenvironment variable setting); that is not a good trade-off.\n\n>> * Even though GIT_SKIP_TESTS mechanism still allows you to skip\n>> individual test pieces, it has never been a serious \"feature\" in\n>> the first place. Many of the tests unfortunately do rely on state\n>> previous sequences of tests left behind, so it is not realistic\n>> to expect that you can skip test pieces randomly and exercise\n>> later test pieces reliably.\n>>\n>> * The numbering of individual test pieces can easily change by new\n>> tests inserted in the middle; again, many tests do take advantge\n>> of the states earlier tests leave behind, so \"do not add new\n>> tests in the middle\" is not a realistic rule to enforce, unless\n>> you are willing to clean up existing test scripts so that each\n>> test piece is independent from all the previous ones.\n>\n> Both are true, but do not apply to the TDD case.\n\nThe existing tests are designed to be black-box tests, not function\nlevel unit tests, and touching lower level code carelessly affects\nother parts of the system you did not know the interactions about.\n\nWhat does \"TDD case\" change anything in that equation?\n"}]}