{"thread":{"id":"34258","subject":"[PATCH v4 0/8] --valgrind improvements","startedAt":"2013-06-23T18:12:51Z","lastAt":"2013-06-23T18:12:59Z","messageCount":9,"participants":["Thomas Rast"],"isPatch":true,"patchVersion":4,"patchTotal":8},"messages":[{"id":"221729","messageId":"cover.1372010917.git.trast@inf.ethz.ch","threadId":"34258","inReplyTo":null,"subject":"[PATCH v4 0/8] --valgrind improvements","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-23T18:12:51Z","receivedAt":"2013-06-23T18:12:51Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"> Please hold off; Fredrik found an issue with the test automation that\n> bisects to \"verbose mode for only tests matching a pattern\"; when\n> running with the 'test' target (not with prove however), the suite\n> reports\n> \n> failed test(s): \n> \n> fixed 0\n> success 9788\n> failed 2\n> broken 69\n> total 9989\n> \n> Even worse, I botched a rebase that makes the commit before --\n> \"self-test that --verbose works\" untestable. I'm pretty sure that it\n> has the same problem, too, though I'm still investigating the actual\n> issue.\n\nThis should fix it. The culprit was that the test suite generates\n.counts files when not under $TEST_HARNESS, and these are seen and\nreported by aggregate-results. So this version changes 4/8 to set\nTEST_HARNESS, instead of unsetting it.\n\nSorry for the trouble.\n\n\nThomas Rast (8):\n test-lib: enable MALLOC_* for the actual tests\n test-lib: refactor $GIT_SKIP_TESTS matching\n test-lib: rearrange start/end of test_expect_* and test_skip\n test-lib: self-test that --verbose works\n test-lib: verbose mode for only tests matching a pattern\n test-lib: valgrind for only tests matching a pattern\n test-lib: allow prefixing a custom string before \"ok N\" etc.\n test-lib: support running tests under valgrind in parallel\n\n t/README | 10 ++\n t/t0000-basic.sh | 61 +++++++++++-\n t/test-lib-functions.sh | 6 +-\n t/test-lib.sh | 246 ++++++++++++++++++++++++++++++++++++++----------\n t/valgrind/valgrind.sh | 3 +\n 5 files changed, 274 insertions(+), 52 deletions(-)\n\n-- \n1.8.3.1.727.gcbe3af3\n"},{"id":"221730","messageId":"d2b5d1eac050319b4ed24506e3d8b1a07d88aa32.1372010917.git.trast@inf.ethz.ch","threadId":"34258","inReplyTo":"cover.1372010917.git.trast@inf.ethz.ch","subject":"[PATCH v4 1/8] test-lib: enable MALLOC_* for the actual tests","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-23T18:12:52Z","receivedAt":"2013-06-23T18:12:52Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"1b3185f (MALLOC_CHECK: various clean-ups, 2012-09-14) moved around the\nMALLOC_CHECK_ and MALLOC_PERTURB_ assignments, intending to limit\ntheir effect to only the test runs. However, they were actually\nenabled only during test cleanup. Call setup/teardown_malloc_check\nalso around the evaluation of the actual test snippet.\n\nSigned-off-by: Thomas Rast \n---\n t/test-lib.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex eff3a65..35da859 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -337,8 +337,10 @@ test_eval_ () {\n test_run_ () {\n \ttest_cleanup=:\n \texpecting_failure=$2\n+\tsetup_malloc_check\n \ttest_eval_ \"$1\"\n \teval_ret=$?\n+\tteardown_malloc_check\n \n \tif test -z \"$immediate\" || test $eval_ret = 0 || test -n \"$expecting_failure\"\n \tthen\n-- \n1.8.3.1.727.gcbe3af3\n"},{"id":"221733","messageId":"271eaadb01fd16324eea440e56c404266fcce2ff.1372010917.git.trast@inf.ethz.ch","threadId":"34258","inReplyTo":"cover.1372010917.git.trast@inf.ethz.ch","subject":"[PATCH v4 2/8] test-lib: refactor $GIT_SKIP_TESTS matching","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-23T18:12:53Z","receivedAt":"2013-06-23T18:12:53Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"It's already used twice, and we will have more of the same kind of\nmatching in a minute.\n\nHelped-by: Johannes Sixt \nSigned-off-by: Thomas Rast \n---\n t/test-lib.sh | 41 ++++++++++++++++++++++++-----------------\n 1 file changed, 24 insertions(+), 17 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 35da859..4fa141a 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -328,6 +328,20 @@ test_debug () {\n \ttest \"$debug\" = \"\" || eval \"$1\"\n }\n \n+match_pattern_list () {\n+\targ=\"$1\"\n+\tshift\n+\ttest -z \"$*\" && return 1\n+\tfor pattern_\n+\tdo\n+\t\tcase \"$arg\" in\n+\t\t$pattern_)\n+\t\t\treturn 0\n+\t\tesac\n+\tdone\n+\treturn 1\n+}\n+\n test_eval_ () {\n \t# This is a separate function because some tests use\n \t# \"return\" to end a test_expect_success block early.\n@@ -358,14 +372,10 @@ test_run_ () {\n test_skip () {\n \ttest_count=$(($test_count+1))\n \tto_skip=\n-\tfor skp in $GIT_SKIP_TESTS\n-\tdo\n-\t\tcase $this_test.$test_count in\n-\t\t$skp)\n-\t\t\tto_skip=t\n-\t\t\tbreak\n-\t\tesac\n-\tdone\n+\tif match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS\n+\tthen\n+\t\tto_skip=t\n+\tfi\n \tif test -z \"$to_skip\" && test -n \"$test_prereq\" &&\n \t ! test_have_prereq \"$test_prereq\"\n \tthen\n@@ -630,15 +640,12 @@ cd -P \"$TRASH_DIRECTORY\" || exit 1\n \n this_test=${0##*/}\n this_test=${this_test%%-*}\n-for skp in $GIT_SKIP_TESTS\n-do\n-\tcase \"$this_test\" in\n-\t$skp)\n-\t\tsay_color info >&3 \"skipping test $this_test altogether\"\n-\t\tskip_all=\"skip all tests in $this_test\"\n-\t\ttest_done\n-\tesac\n-done\n+if match_pattern_list \"$this_test\" $GIT_SKIP_TESTS\n+then\n+\tsay_color info >&3 \"skipping test $this_test altogether\"\n+\tskip_all=\"skip all tests in $this_test\"\n+\ttest_done\n+fi\n \n # Provide an implementation of the 'yes' utility\n yes () {\n-- \n1.8.3.1.727.gcbe3af3\n"},{"id":"221731","messageId":"6768d2c43cc7f34b70a493f5ac5fee1571b60f78.1372010917.git.trast@inf.ethz.ch","threadId":"34258","inReplyTo":"cover.1372010917.git.trast@inf.ethz.ch","subject":"[PATCH v4 3/8] test-lib: rearrange start/end of test_expect_* and test_skip","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-23T18:12:54Z","receivedAt":"2013-06-23T18:12:54Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"This moves\n\n* the early setup part from test_skip to a new function test_start_\n\n* the final common parts of test_expect_* to a new function\n test_finish_\n\nto make the next commit more obvious.\n\nSigned-off-by: Thomas Rast \n---\n t/test-lib-functions.sh | 6 ++++--\n t/test-lib.sh | 9 ++++++++-\n 2 files changed, 12 insertions(+), 3 deletions(-)\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 8828ff7..a7e9aac 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -343,6 +343,7 @@ test_declared_prereq () {\n }\n \n test_expect_failure () {\n+\ttest_start_\n \ttest \"$#\" = 3 && { test_prereq=$1; shift; } || test_prereq=\n \ttest \"$#\" = 2 ||\n \terror \"bug in the test script: not 2 or 3 parameters to test-expect-failure\"\n@@ -357,10 +358,11 @@ test_expect_failure () {\n \t\t\ttest_known_broken_failure_ \"$1\"\n \t\tfi\n \tfi\n-\techo >&3 \"\"\n+\ttest_finish_\n }\n \n test_expect_success () {\n+\ttest_start_\n \ttest \"$#\" = 3 && { test_prereq=$1; shift; } || test_prereq=\n \ttest \"$#\" = 2 ||\n \terror \"bug in the test script: not 2 or 3 parameters to test-expect-success\"\n@@ -375,7 +377,7 @@ test_expect_success () {\n \t\t\ttest_failure_ \"$@\"\n \t\tfi\n \tfi\n-\techo >&3 \"\"\n+\ttest_finish_\n }\n \n # test_external runs external test scripts that provide continuous\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 4fa141a..e99b0ea 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -369,8 +369,15 @@ test_run_ () {\n \treturn \"$eval_ret\"\n }\n \n-test_skip () {\n+test_start_ () {\n \ttest_count=$(($test_count+1))\n+}\n+\n+test_finish_ () {\n+\techo >&3 \"\"\n+}\n+\n+test_skip () {\n \tto_skip=\n \tif match_pattern_list $this_test.$test_count $GIT_SKIP_TESTS\n \tthen\n-- \n1.8.3.1.727.gcbe3af3\n"},{"id":"221732","messageId":"9c6ca931db165476d6fb449632884f6515b0840f.1372010917.git.trast@inf.ethz.ch","threadId":"34258","inReplyTo":"cover.1372010917.git.trast@inf.ethz.ch","subject":"[PATCH v4 4/8] test-lib: self-test that --verbose works","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-23T18:12:55Z","receivedAt":"2013-06-23T18:12:55Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"t0000 contains some light self-tests of test-lib.sh, but --verbose was\nnot covered. Add a test.\n\nThe only catch is that the presence of a test harness influences the\noutput (specifically, the presence of some empty lines). So we need\nto unset TEST_HARNESS or set it to a known value. Leaving it unset\nleads to spurious test failures in the final summary, which come from\nthe subtest. So we always set it.\n\nSigned-off-by: Thomas Rast \n---\n t/t0000-basic.sh | 37 ++++++++++++++++++++++++++++++++++++-\n t/test-lib.sh | 2 ++\n 2 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 0f13180..4b4103f 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -47,8 +47,13 @@ test_expect_failure 'pretend we have a known breakage' '\n \n run_sub_test_lib_test () {\n \tname=\"$1\" descr=\"$2\" # stdin is the body of the test code\n+\tshift 2\n \tmkdir \"$name\" &&\n \t(\n+\t\t# Pretend we're a test harness. This prevents\n+\t\t# test-lib from writing the counts to a file that will\n+\t\t# later be summarized, showing spurious \"failed\" tests\n+\t\texport HARNESS_ACTIVE=t &&\n \t\tcd \"$name\" &&\n \t\tcat >\"$name.sh\" <<-EOF &&\n \t\t#!$SHELL_PATH\n@@ -65,7 +70,7 @@ run_sub_test_lib_test () {\n \t\tcat >>\"$name.sh\" &&\n \t\tchmod +x \"$name.sh\" &&\n \t\texport TEST_DIRECTORY &&\n-\t\t./\"$name.sh\" >out 2>err\n+\t\t./\"$name.sh\" \"$@\" >out 2>err\n \t)\n }\n \n@@ -215,6 +220,36 @@ test_expect_success 'pretend we have a mix of all possible results' \"\n \tEOF\n \"\n \n+test_expect_success 'test --verbose' '\n+\ttest_must_fail run_sub_test_lib_test \\\n+\t\ttest-verbose \"test verbose\" --verbose <<-\\EOF &&\n+\ttest_expect_success \"passing test\" true\n+\ttest_expect_success \"test with output\" \"echo foo\"\n+\ttest_expect_success \"failing test\" false\n+\ttest_done\n+\tEOF\n+\tmv test-verbose/out test-verbose/out+\n+\tgrep -v \"^Initialized empty\" test-verbose/out+ >test-verbose/out &&\n+\tcheck_sub_test_lib_test test-verbose <<-\\EOF\n+\t> expecting success: true\n+\t> Z\n+\t> ok 1 - passing test\n+\t> Z\n+\t> expecting success: echo foo\n+\t> foo\n+\t> Z\n+\t> ok 2 - test with output\n+\t> Z\n+\t> expecting success: false\n+\t> Z\n+\t> not ok 3 - failing test\n+\t> #\tfalse\n+\t> Z\n+\t> # failed 1 among 3 test(s)\n+\t> 1..3\n+\tEOF\n+'\n+\n test_set_prereq HAVEIT\n haveit=no\n test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e99b0ea..10827a4 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -414,6 +414,8 @@ test_at_end_hook_ () {\n test_done () {\n \tGIT_EXIT_OK=t\n \n+\t# Note: t0000 relies on $HARNESS_ACTIVE disabling the .counts\n+\t# output file\n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n \t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n-- \n1.8.3.1.727.gcbe3af3\n"},{"id":"221734","messageId":"a586b657c16affc72742153f55ae28dce6545c11.1372010917.git.trast@inf.ethz.ch","threadId":"34258","inReplyTo":"cover.1372010917.git.trast@inf.ethz.ch","subject":"[PATCH v4 5/8] test-lib: verbose mode for only tests matching a pattern","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-23T18:12:56Z","receivedAt":"2013-06-23T18:12:56Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"With the new --verbose-only= option, one can enable --verbose\nat a per-test granularity. The pattern is matched against the test\nnumber, e.g.\n\n ./t0000-basic.sh --verbose-only='2[0-2]'\n\nto see only the full output of test 20-22, while showing the rest in the\none-liner format.\n\nAs suggested by Jeff King, this takes care to wrap the entire\ntest_expect_* block, but nothing else, in the verbose toggling. We\ncan use the test_start/end functions from the previous commit for the\npurpose.\n\nThis is arguably not *too* useful on its own, but makes the next patch\neasier to follow.\n\nHelped-by: Jeff King \nSigned-off-by: Thomas Rast \n---\n t/README | 5 +++++\n t/t0000-basic.sh | 24 ++++++++++++++++++++++++\n t/test-lib.sh | 31 +++++++++++++++++++++++++++++++\n 3 files changed, 60 insertions(+)\n\ndiff --git a/t/README b/t/README\nindex ec52468..ec8ab79 100644\n--- a/t/README\n+++ b/t/README\n@@ -76,6 +76,11 @@ appropriately before running \"make\".\n \tcommand being run and their output if any are also\n \toutput.\n \n+--verbose-only=::\n+\tLike --verbose, but the effect is limited to tests with\n+\tnumbers matching . The number matched against is\n+\tsimply the running count of the test within the file.\n+\n --debug::\n \tThis may help the person who is developing a new test.\n \tIt causes the command defined with test_debug to run.\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 4b4103f..5c32288 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -250,6 +250,30 @@ test_expect_success 'test --verbose' '\n \tEOF\n '\n \n+test_expect_success 'test --verbose-only' '\n+\ttest_must_fail run_sub_test_lib_test \\\n+\t\ttest-verbose-only-2 \"test verbose-only=2\" \\\n+\t\t--verbose-only=2 <<-\\EOF &&\n+\ttest_expect_success \"passing test\" true\n+\ttest_expect_success \"test with output\" \"echo foo\"\n+\ttest_expect_success \"failing test\" false\n+\ttest_done\n+\tEOF\n+\tcheck_sub_test_lib_test test-verbose-only-2 <<-\\EOF\n+\t> ok 1 - passing test\n+\t> Z\n+\t> expecting success: echo foo\n+\t> foo\n+\t> Z\n+\t> ok 2 - test with output\n+\t> Z\n+\t> not ok 3 - failing test\n+\t> #\tfalse\n+\t> # failed 1 among 3 test(s)\n+\t> 1..3\n+\tEOF\n+'\n+\n test_set_prereq HAVEIT\n haveit=no\n test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 10827a4..5729702 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -184,6 +184,9 @@ do\n \t\thelp=t; shift ;;\n \t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n \t\tverbose=t; shift ;;\n+\t--verbose-only=*)\n+\t\tverbose_only=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n+\t\tshift ;;\n \t-q|--q|--qu|--qui|--quie|--quiet)\n \t\t# Ignore --quiet under a TAP::Harness. Saying how many tests\n \t\t# passed without the ok/not ok details is always an error.\n@@ -342,6 +345,32 @@ match_pattern_list () {\n \treturn 1\n }\n \n+maybe_teardown_verbose () {\n+\ttest -z \"$verbose_only\" && return\n+\texec 4>/dev/null 3>/dev/null\n+\tverbose=\n+}\n+\n+last_verbose=t\n+maybe_setup_verbose () {\n+\ttest -z \"$verbose_only\" && return\n+\tif match_pattern_list $test_count $verbose_only\n+\tthen\n+\t\texec 4>&2 3>&1\n+\t\t# Emit a delimiting blank line when going from\n+\t\t# non-verbose to verbose. Within verbose mode the\n+\t\t# delimiter is printed by test_expect_*. The choice\n+\t\t# of the initial $last_verbose is such that before\n+\t\t# test 1, we do not print it.\n+\t\ttest -z \"$last_verbose\" && echo >&3 \"\"\n+\t\tverbose=t\n+\telse\n+\t\texec 4>/dev/null 3>/dev/null\n+\t\tverbose=\n+\tfi\n+\tlast_verbose=$verbose\n+}\n+\n test_eval_ () {\n \t# This is a separate function because some tests use\n \t# \"return\" to end a test_expect_success block early.\n@@ -371,10 +400,12 @@ test_run_ () {\n \n test_start_ () {\n \ttest_count=$(($test_count+1))\n+\tmaybe_setup_verbose\n }\n \n test_finish_ () {\n \techo >&3 \"\"\n+\tmaybe_teardown_verbose\n }\n \n test_skip () {\n-- \n1.8.3.1.727.gcbe3af3\n"},{"id":"221735","messageId":"369715b1226286a9334205cbecedfde007874281.1372010917.git.trast@inf.ethz.ch","threadId":"34258","inReplyTo":"cover.1372010917.git.trast@inf.ethz.ch","subject":"[PATCH v4 6/8] test-lib: valgrind for only tests matching a pattern","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-23T18:12:57Z","receivedAt":"2013-06-23T18:12:57Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"With the new --valgrind-only= option, one can enable\n--valgrind at a per-test granularity, exactly analogous to\n--verbose-only from the previous commit.\n\nThe options are wired such that --valgrind implies --verbose (as\nbefore), but --valgrind-only= implies\n--verbose-only= unless --verbose is also in effect.\n\nSigned-off-by: Thomas Rast \n---\n t/README | 5 +++++\n t/test-lib.sh | 36 +++++++++++++++++++++++++++++++++++-\n t/valgrind/valgrind.sh | 3 +++\n 3 files changed, 43 insertions(+), 1 deletion(-)\n\ndiff --git a/t/README b/t/README\nindex ec8ab79..2167125 100644\n--- a/t/README\n+++ b/t/README\n@@ -126,6 +126,11 @@ appropriately before running \"make\".\n \tthe 't/valgrind/' directory and use the commands under\n \t't/valgrind/bin/'.\n \n+--valgrind-only=::\n+\tLike --valgrind, but the effect is limited to tests with\n+\tnumbers matching . The number matched against is\n+\tsimply the running count of the test within the file.\n+\n --tee::\n \tIn addition to printing the test output to the terminal,\n \twrite it to files named 't/test-results/$TEST_NAME.out'.\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 5729702..a926828 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -201,6 +201,9 @@ do\n \t--valgrind=*)\n \t\tvalgrind=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n \t\tshift ;;\n+\t--valgrind-only=*)\n+\t\tvalgrind_only=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n+\t\tshift ;;\n \t--tee)\n \t\tshift ;; # was handled already\n \t--root=*)\n@@ -211,7 +214,14 @@ do\n \tesac\n done\n \n-test -n \"$valgrind\" && verbose=t\n+if test -n \"$valgrind_only\"\n+then\n+\ttest -z \"$valgrind\" && valgrind=memcheck\n+\ttest -z \"$verbose\" && verbose_only=\"$valgrind_only\"\n+elif test -n \"$valgrind\"\n+then\n+\tverbose=t\n+fi\n \n if test -n \"$color\"\n then\n@@ -371,6 +381,25 @@ maybe_setup_verbose () {\n \tlast_verbose=$verbose\n }\n \n+maybe_teardown_valgrind () {\n+\ttest -z \"$GIT_VALGRIND\" && return\n+\tGIT_VALGRIND_ENABLED=\n+}\n+\n+maybe_setup_valgrind () {\n+\ttest -z \"$GIT_VALGRIND\" && return\n+\tif test -z \"$valgrind_only\"\n+\tthen\n+\t\tGIT_VALGRIND_ENABLED=t\n+\t\treturn\n+\tfi\n+\tGIT_VALGRIND_ENABLED=\n+\tif match_pattern_list $test_count $valgrind_only\n+\tthen\n+\t\tGIT_VALGRIND_ENABLED=t\n+\tfi\n+}\n+\n test_eval_ () {\n \t# This is a separate function because some tests use\n \t# \"return\" to end a test_expect_success block early.\n@@ -401,10 +430,12 @@ test_run_ () {\n test_start_ () {\n \ttest_count=$(($test_count+1))\n \tmaybe_setup_verbose\n+\tmaybe_setup_valgrind\n }\n \n test_finish_ () {\n \techo >&3 \"\"\n+\tmaybe_teardown_valgrind\n \tmaybe_teardown_verbose\n }\n \n@@ -590,6 +621,9 @@ then\n \texport GIT_VALGRIND\n \tGIT_VALGRIND_MODE=\"$valgrind\"\n \texport GIT_VALGRIND_MODE\n+\tGIT_VALGRIND_ENABLED=t\n+\ttest -n \"$valgrind_only\" && GIT_VALGRIND_ENABLED=\n+\texport GIT_VALGRIND_ENABLED\n elif test -n \"$GIT_TEST_INSTALLED\"\n then\n \tGIT_EXEC_PATH=$($GIT_TEST_INSTALLED/git --exec-path) ||\ndiff --git a/t/valgrind/valgrind.sh b/t/valgrind/valgrind.sh\nindex 6b87c91..4215303 100755\n--- a/t/valgrind/valgrind.sh\n+++ b/t/valgrind/valgrind.sh\n@@ -4,6 +4,9 @@ base=$(basename \"$0\")\n \n TOOL_OPTIONS='--leak-check=no'\n \n+test -z \"$GIT_VALGRIND_ENABLED\" &&\n+exec \"$GIT_VALGRIND\"/../../\"$base\" \"$@\"\n+\n case \"$GIT_VALGRIND_MODE\" in\n memcheck-fast)\n \t;;\n-- \n1.8.3.1.727.gcbe3af3\n"},{"id":"221737","messageId":"46f94f8f48ebf4a42d3f0f3dbf1566caacd9720c.1372010917.git.trast@inf.ethz.ch","threadId":"34258","inReplyTo":"cover.1372010917.git.trast@inf.ethz.ch","subject":"[PATCH v4 7/8] test-lib: allow prefixing a custom string before \"ok N\" etc.","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-23T18:12:58Z","receivedAt":"2013-06-23T18:12:58Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"This is not really meant for external use, and thus not documented. It\nallows the next commit to neatly distinguish between sub-tests and the\nmain run.\n\nThe format is intentionally not valid TAP. The use in the next commit\nwould not result in anything valid either way, and it seems better to\nmake it obvious.\n\nSigned-off-by: Thomas Rast \n---\n t/test-lib.sh | 27 +++++++++++++++------------\n 1 file changed, 15 insertions(+), 12 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a926828..682459b 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -209,6 +209,9 @@ do\n \t--root=*)\n \t\troot=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n \t\tshift ;;\n+\t--statusprefix=*)\n+\t\tstatusprefix=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n+\t\tshift ;;\n \t*)\n \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n \tesac\n@@ -316,12 +319,12 @@ trap 'die' EXIT\n \n test_ok_ () {\n \ttest_success=$(($test_success + 1))\n-\tsay_color \"\" \"ok $test_count - $@\"\n+\tsay_color \"\" \"${statusprefix}ok $test_count - $@\"\n }\n \n test_failure_ () {\n \ttest_failure=$(($test_failure + 1))\n-\tsay_color error \"not ok $test_count - $1\"\n+\tsay_color error \"${statusprefix}not ok $test_count - $1\"\n \tshift\n \techo \"$@\" | sed -e 's/^/#\t/'\n \ttest \"$immediate\" = \"\" || { GIT_EXIT_OK=t; exit 1; }\n@@ -329,12 +332,12 @@ test_failure_ () {\n \n test_known_broken_ok_ () {\n \ttest_fixed=$(($test_fixed+1))\n-\tsay_color error \"ok $test_count - $@ # TODO known breakage vanished\"\n+\tsay_color error \"${statusprefix}ok $test_count - $@ # TODO known breakage vanished\"\n }\n \n test_known_broken_failure_ () {\n \ttest_broken=$(($test_broken+1))\n-\tsay_color warn \"not ok $test_count - $@ # TODO known breakage\"\n+\tsay_color warn \"${statusprefix}not ok $test_count - $@ # TODO known breakage\"\n }\n \n test_debug () {\n@@ -458,8 +461,8 @@ test_skip () {\n \t\t\tof_prereq=\" of $test_prereq\"\n \t\tfi\n \n-\t\tsay_color skip >&3 \"skipping test: $@\"\n-\t\tsay_color skip \"ok $test_count # skip $1 (missing $missing_prereq${of_prereq})\"\n+\t\tsay_color skip >&3 \"${statusprefix}skipping test: $@\"\n+\t\tsay_color skip \"${statusprefix}ok $test_count # skip $1 (missing $missing_prereq${of_prereq})\"\n \t\t: true\n \t\t;;\n \t*)\n@@ -497,11 +500,11 @@ test_done () {\n \n \tif test \"$test_fixed\" != 0\n \tthen\n-\t\tsay_color error \"# $test_fixed known breakage(s) vanished; please update test(s)\"\n+\t\tsay_color error \"${statusprefix}# $test_fixed known breakage(s) vanished; please update test(s)\"\n \tfi\n \tif test \"$test_broken\" != 0\n \tthen\n-\t\tsay_color warn \"# still have $test_broken known breakage(s)\"\n+\t\tsay_color warn \"${statusprefix}# still have $test_broken known breakage(s)\"\n \tfi\n \tif test \"$test_broken\" != 0 || test \"$test_fixed\" != 0\n \tthen\n@@ -524,9 +527,9 @@ test_done () {\n \t\tthen\n \t\t\tif test $test_remaining -gt 0\n \t\t\tthen\n-\t\t\t\tsay_color pass \"# passed all $msg\"\n+\t\t\t\tsay_color pass \"${statusprefix}# passed all $msg\"\n \t\t\tfi\n-\t\t\tsay \"1..$test_count$skip_all\"\n+\t\t\tsay \"${statusprefix}1..$test_count$skip_all\"\n \t\tfi\n \n \t\ttest -d \"$remove_trash\" &&\n@@ -540,8 +543,8 @@ test_done () {\n \t*)\n \t\tif test $test_external_has_tap -eq 0\n \t\tthen\n-\t\t\tsay_color error \"# failed $test_failure among $msg\"\n-\t\t\tsay \"1..$test_count\"\n+\t\t\tsay_color error \"${statusprefix}# failed $test_failure among $msg\"\n+\t\t\tsay \"${statusprefix}1..$test_count\"\n \t\tfi\n \n \t\texit 1 ;;\n-- \n1.8.3.1.727.gcbe3af3\n"},{"id":"221736","messageId":"7aa3fb8f092eed1890701301d8ad415e59d0676b.1372010917.git.trast@inf.ethz.ch","threadId":"34258","inReplyTo":"cover.1372010917.git.trast@inf.ethz.ch","subject":"[PATCH v4 8/8] test-lib: support running tests under valgrind in parallel","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-06-23T18:12:59Z","receivedAt":"2013-06-23T18:12:59Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"With the new --valgrind-parallel= option, we support running the\ntests in a single test script under valgrind in parallel using 'n'\nprocesses.\n\nThis really follows the dumbest approach possible, as follows:\n\n* We spawn the test script 'n' times, using a throw-away\n TEST_OUTPUT_DIRECTORY. Each of the instances is given options that\n ensures that it only runs every n-th test under valgrind, but\n together they cover the entire range.\n\n* We add up the numbers from the individual tests, and provide the\n usual output.\n\nThis is really a gross hack at this point, and should be improved. In\nparticular we should keep the actual outputs somewhere more easily\ndiscoverable, and summarize them to the user.\n\nNevertheless, this is already workable and gives a speedup of more\nthan 2 on a dual-core (hyperthreaded) machine, using n=4. This is\nexpected since the overhead of valgrind is so big (on the order of 20x\nunder good conditions, and a large startup overhead at every git\ninvocation) that redundantly running the non-valgrind tests in between\nis not that expensive.\n\nHelped-by: Jeff King \nSigned-off-by: Thomas Rast \n---\n t/test-lib.sh | 106 ++++++++++++++++++++++++++++++++++++++++++++++------------\n 1 file changed, 84 insertions(+), 22 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 682459b..9753641 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -204,6 +204,15 @@ do\n \t--valgrind-only=*)\n \t\tvalgrind_only=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n \t\tshift ;;\n+\t--valgrind-parallel=*)\n+\t\tvalgrind_parallel=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n+\t\tshift ;;\n+\t--valgrind-only-stride=*)\n+\t\tvalgrind_only_stride=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n+\t\tshift ;;\n+\t--valgrind-only-offset=*)\n+\t\tvalgrind_only_offset=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n+\t\tshift ;;\n \t--tee)\n \t\tshift ;; # was handled already\n \t--root=*)\n@@ -217,7 +226,7 @@ do\n \tesac\n done\n \n-if test -n \"$valgrind_only\"\n+if test -n \"$valgrind_only\" || test -n \"$valgrind_only_stride\"\n then\n \ttest -z \"$valgrind\" && valgrind=memcheck\n \ttest -z \"$verbose\" && verbose_only=\"$valgrind_only\"\n@@ -367,7 +376,9 @@ maybe_teardown_verbose () {\n last_verbose=t\n maybe_setup_verbose () {\n \ttest -z \"$verbose_only\" && return\n-\tif match_pattern_list $test_count $verbose_only\n+\tif match_pattern_list $test_count $verbose_only ||\n+\t\t{ test -n \"$valgrind_only_stride\" &&\n+\t\texpr $test_count \"%\" $valgrind_only_stride - $valgrind_only_offset = 0 >/dev/null; }\n \tthen\n \t\texec 4>&2 3>&1\n \t\t# Emit a delimiting blank line when going from\n@@ -391,7 +402,7 @@ maybe_teardown_valgrind () {\n \n maybe_setup_valgrind () {\n \ttest -z \"$GIT_VALGRIND\" && return\n-\tif test -z \"$valgrind_only\"\n+\tif test -z \"$valgrind_only\" && test -z \"$valgrind_only_stride\"\n \tthen\n \t\tGIT_VALGRIND_ENABLED=t\n \t\treturn\n@@ -400,6 +411,10 @@ maybe_setup_valgrind () {\n \tif match_pattern_list $test_count $valgrind_only\n \tthen\n \t\tGIT_VALGRIND_ENABLED=t\n+\telif test -n \"$valgrind_only_stride\" &&\n+\t\texpr $test_count \"%\" $valgrind_only_stride - $valgrind_only_offset = 0 >/dev/null\n+\tthen\n+\t\tGIT_VALGRIND_ENABLED=t\n \tfi\n }\n \n@@ -552,6 +567,9 @@ test_done () {\n \tesac\n }\n \n+\n+# Set up a directory that we can put in PATH which redirects all git\n+# calls to 'valgrind git ...'.\n if test -n \"$valgrind\"\n then\n \tmake_symlink () {\n@@ -599,33 +617,42 @@ then\n \t\tmake_symlink \"$symlink_target\" \"$GIT_VALGRIND/bin/$base\" || exit\n \t}\n \n-\t# override all git executables in TEST_DIRECTORY/..\n-\tGIT_VALGRIND=$TEST_DIRECTORY/valgrind\n-\tmkdir -p \"$GIT_VALGRIND\"/bin\n-\tfor file in $GIT_BUILD_DIR/git* $GIT_BUILD_DIR/test-*\n-\tdo\n-\t\tmake_valgrind_symlink $file\n-\tdone\n-\t# special-case the mergetools loadables\n-\tmake_symlink \"$GIT_BUILD_DIR\"/mergetools \"$GIT_VALGRIND/bin/mergetools\"\n-\tOLDIFS=$IFS\n-\tIFS=:\n-\tfor path in $PATH\n-\tdo\n-\t\tls \"$path\"/git-* 2> /dev/null |\n-\t\twhile read file\n+\t# In the case of --valgrind-parallel, we only need to do the\n+\t# wrapping once, in the main script. The worker children all\n+\t# have $valgrind_only_stride set, so we can skip based on that.\n+\tif test -z \"$valgrind_only_stride\"\n+\tthen\n+\t\t# override all git executables in TEST_DIRECTORY/..\n+\t\tGIT_VALGRIND=$TEST_DIRECTORY/valgrind\n+\t\tmkdir -p \"$GIT_VALGRIND\"/bin\n+\t\tfor file in $GIT_BUILD_DIR/git* $GIT_BUILD_DIR/test-*\n \t\tdo\n-\t\t\tmake_valgrind_symlink \"$file\"\n+\t\t\tmake_valgrind_symlink $file\n \t\tdone\n-\tdone\n-\tIFS=$OLDIFS\n+\t\t# special-case the mergetools loadables\n+\t\tmake_symlink \"$GIT_BUILD_DIR\"/mergetools \"$GIT_VALGRIND/bin/mergetools\"\n+\t\tOLDIFS=$IFS\n+\t\tIFS=:\n+\t\tfor path in $PATH\n+\t\tdo\n+\t\t\tls \"$path\"/git-* 2> /dev/null |\n+\t\t\twhile read file\n+\t\t\tdo\n+\t\t\t\tmake_valgrind_symlink \"$file\"\n+\t\t\tdone\n+\t\tdone\n+\t\tIFS=$OLDIFS\n+\tfi\n \tPATH=$GIT_VALGRIND/bin:$PATH\n \tGIT_EXEC_PATH=$GIT_VALGRIND/bin\n \texport GIT_VALGRIND\n \tGIT_VALGRIND_MODE=\"$valgrind\"\n \texport GIT_VALGRIND_MODE\n \tGIT_VALGRIND_ENABLED=t\n-\ttest -n \"$valgrind_only\" && GIT_VALGRIND_ENABLED=\n+\tif test -n \"$valgrind_only\" || test -n \"$valgrind_only_stride\"\n+\tthen\n+\t\tGIT_VALGRIND_ENABLED=\n+\tfi\n \texport GIT_VALGRIND_ENABLED\n elif test -n \"$GIT_TEST_INSTALLED\"\n then\n@@ -711,6 +738,41 @@ then\n else\n \tmkdir -p \"$TRASH_DIRECTORY\"\n fi\n+\n+# Gross hack to spawn N sub-instances of the tests in parallel, and\n+# summarize the results. Note that if this is enabled, the script\n+# terminates at the end of this 'if' block.\n+if test -n \"$valgrind_parallel\"\n+then\n+\tfor i in $(test_seq 1 $valgrind_parallel)\n+\tdo\n+\t\troot=\"$TRASH_DIRECTORY/vgparallel-$i\"\n+\t\tmkdir \"$root\"\n+\t\tTEST_OUTPUT_DIRECTORY=\"$root\" \\\n+\t\t\t${SHELL_PATH} \"$0\" \\\n+\t\t\t--root=\"$root\" --statusprefix=\"[$i] \" \\\n+\t\t\t--valgrind=\"$valgrind\" \\\n+\t\t\t--valgrind-only-stride=\"$valgrind_parallel\" \\\n+\t\t\t--valgrind-only-offset=\"$i\" &\n+\t\tpids=\"$pids $!\"\n+\tdone\n+\ttrap \"kill $pids\" INT TERM HUP\n+\twait $pids\n+\ttrap - INT TERM HUP\n+\tfor i in $(test_seq 1 $valgrind_parallel)\n+\tdo\n+\t\troot=\"$TRASH_DIRECTORY/vgparallel-$i\"\n+\t\teval \"$(cat \"$root/test-results/$(basename \"$0\" .sh)\"-*.counts |\n+\t\t\tsed 's/^\\([a-z][a-z]*\\) \\([0-9][0-9]*\\)/inner_\\1=\\2/')\"\n+\t\ttest_count=$(expr $test_count + $inner_total)\n+\t\ttest_success=$(expr $test_success + $inner_success)\n+\t\ttest_fixed=$(expr $test_fixed + $inner_fixed)\n+\t\ttest_broken=$(expr $test_broken + $inner_broken)\n+\t\ttest_failure=$(expr $test_failure + $inner_failed)\n+\tdone\n+\ttest_done\n+fi\n+\n # Use -P to resolve symlinks in our working directory so that the cwd\n # in subprocesses like git equals our $PWD (for pathname comparisons).\n cd -P \"$TRASH_DIRECTORY\" || exit 1\n-- \n1.8.3.1.727.gcbe3af3\n"}]}