{"thread":{"id":"49946","subject":"[RFC PATCH 0/3] test-lib: add the '--stress' option to help reproduce occasional failures in flaky tests","startedAt":"2018-12-04T16:35:19Z","lastAt":"2019-01-07T08:49:09Z","messageCount":67,"participants":["SZEDER Gábor","Ævar Arnfjörð Bjarmason","Jeff King","Junio C Hamano","Carlo Arenas"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"364560","messageId":"20181204163457.15717-1-szeder.dev@gmail.com","threadId":"49946","inReplyTo":null,"subject":"[RFC PATCH 0/3] test-lib: add the '--stress' option to help reproduce occasional failures in flaky tests","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-04T16:34:54Z","receivedAt":"2018-12-04T16:35:19Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Inspired by Peff's 'stress' script mentioned in:\n\n https://public-inbox.org/git/20181122161722.GC28192@sigill.intra.peff.net/\n\nthe last patch in this series brings that functionality to our test\nlibrary to help to reproduce failures in flaky tests. So\n\n ./t1234-foo --stress\n \nwill run that test script repeatedly in multiple parallel invocations,\nin the hope that the increased load creates enough variance in the\ntiming of the test's commands that a failure is evenually triggered.\n\n\nSZEDER Gábor (3):\n test-lib: consolidate naming of test-results paths\n test-lib-functions: introduce the 'test_set_port' helper function\n test-lib: add the '--stress' option to run a test repeatedly under\n load\n\n t/README | 13 +++++-\n t/lib-git-daemon.sh | 2 +-\n t/lib-git-p4.sh | 9 +---\n t/lib-git-svn.sh | 2 +-\n t/lib-httpd.sh | 2 +-\n t/t0410-partial-clone.sh | 1 -\n t/t5512-ls-remote.sh | 2 +-\n t/test-lib-functions.sh | 39 ++++++++++++++++\n t/test-lib.sh | 99 +++++++++++++++++++++++++++++++++++-----\n 9 files changed, 143 insertions(+), 26 deletions(-)\n\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364561","messageId":"20181204163457.15717-2-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181204163457.15717-1-szeder.dev@gmail.com","subject":"[PATCH 1/3] test-lib: consolidate naming of test-results paths","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-04T16:34:55Z","receivedAt":"2018-12-04T16:35:20Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"There are two places where we strip off any leading path components\nand the '.sh' suffix from the test script's pathname, and there are\ntwo places where we construct the filename of test output files in\n't/test-results/'. The last patch in this series will add even more.\n\nFactor these out into helper variables to avoid repeating ourselves.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 21 ++++++++++-----------\n 1 file changed, 10 insertions(+), 11 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 0f1faa24b2..49e4563405 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -71,6 +71,9 @@ then\n \texit 1\n fi\n \n+TEST_NAME=\"$(basename \"$0\" .sh)\"\n+TEST_RESULTS_BASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$TEST_NAME\"\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n case \"$GIT_TEST_TEE_STARTED, $* \" in\n@@ -78,12 +81,11 @@ done,*)\n \t# do not redirect again\n \t;;\n *' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*)\n-\tmkdir -p \"$TEST_OUTPUT_DIRECTORY/test-results\"\n-\tBASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$(basename \"$0\" .sh)\"\n+\tmkdir -p \"$(dirname \"$TEST_RESULTS_BASE\")\"\n \n \t# Make this filename available to the sub-process in case it is using\n \t# --verbose-log.\n-\tGIT_TEST_TEE_OUTPUT_FILE=$BASE.out\n+\tGIT_TEST_TEE_OUTPUT_FILE=$TEST_RESULTS_BASE.out\n \texport GIT_TEST_TEE_OUTPUT_FILE\n \n \t# Truncate before calling \"tee -a\" to get rid of the results\n@@ -91,8 +93,8 @@ done,*)\n \t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \n \t(GIT_TEST_TEE_STARTED=done ${TEST_SHELL_PATH} \"$0\" \"$@\" 2>&1;\n-\t echo $? >\"$BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n-\ttest \"$(cat \"$BASE.exit\")\" = 0\n+\t echo $? >\"$TEST_RESULTS_BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\ttest \"$(cat \"$TEST_RESULTS_BASE.exit\")\" = 0\n \texit\n \t;;\n esac\n@@ -818,12 +820,9 @@ test_done () {\n \n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n-\t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n-\t\tmkdir -p \"$test_results_dir\"\n-\t\tbase=${0##*/}\n-\t\ttest_results_path=\"$test_results_dir/${base%.sh}.counts\"\n+\t\tmkdir -p \"$(dirname \"$TEST_RESULTS_BASE\")\"\n \n-\t\tcat >\"$test_results_path\" <<-EOF\n+\t\tcat >\"$TEST_RESULTS_BASE.counts\" <<-EOF\n \t\ttotal $test_count\n \t\tsuccess $test_success\n \t\tfixed $test_fixed\n@@ -1029,7 +1028,7 @@ then\n fi\n \n # Test repository\n-TRASH_DIRECTORY=\"trash directory.$(basename \"$0\" .sh)\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n case \"$TRASH_DIRECTORY\" in\n /*) ;; # absolute path is good\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364562","messageId":"20181204163457.15717-3-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181204163457.15717-1-szeder.dev@gmail.com","subject":"[PATCH 2/3] test-lib-functions: introduce the 'test_set_port' helper function","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-04T16:34:56Z","receivedAt":"2018-12-04T16:35:22Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Several test scripts run daemons like 'git-daemon' or Apache, and\ncommunicate with them through TCP sockets. To have unique ports where\nthese daemons are accessible, the ports are usually the number of the\ncorresponding test scripts, unless the user overrides them via\nenvironment variables, and thus all those tests and test libs contain\nmore or less the same bit of one-liner boilerplate code to find out\nthe port. The last patch in this series will make this a bit more\ncomplicated.\n\nFactor out finding the port for a daemon into the common helper\nfunction 'test_set_port' to avoid repeating ourselves.\n\nTake special care of test scripts with \"low\" numbers:\n\n - Test numbers below 1024 would result in a port that's only usable\n as root, so set their port to '10000 + test-nr' to make sure it\n doesn't interfere with other tests in the test suite. This makes\n the hardcoded port number in 't0410-partial-clone.sh' unnecessary,\n remove it.\n\n - The shell's arithmetic evaluation interprets numbers with leading\n zeros as octal values, which means that test number below 1000 and\n containing the digits 8 or 9 will trigger an error. Remove all\n leading zeros from the test numbers to prevent this.\n\nNote that the Perforce tests are unlike the other tests involving\ndaemons in that:\n\n - 'lib-git-p4.sh' doesn't use the test's number for unique port as\n is, but does a bit of additional arithmetic on top [1].\n\n - The port is not overridable via an environment variable.\n\nWith this patch even Perforce tests will use the test's number as\ndefault port, and it will be overridable via the P4DPORT environment\nvariable.\n\n[1] Commit fc00233071 (git-p4 tests: refactor and cleanup, 2011-08-22)\n introduced that \"unusual\" unique port computation without\n explaining why it was necessary (as opposed to simply using the\n test number as is). It seems to be just unnecessary complication,\n and in any case that commit came way before the \"test nr as unique\n port\" got \"standardized\" for other daemons in commits c44132fcf3\n (tests: auto-set git-daemon port, 2014-02-10), 3bb486e439 (tests:\n auto-set LIB_HTTPD_PORT from test name, 2014-02-10), and\n bf9d7df950 (t/lib-git-svn.sh: improve svnserve tests with parallel\n make test, 2017-12-01).\n\nSigned-off-by: SZEDER Gábor \n---\n t/lib-git-daemon.sh | 2 +-\n t/lib-git-p4.sh | 9 +--------\n t/lib-git-svn.sh | 2 +-\n t/lib-httpd.sh | 2 +-\n t/t0410-partial-clone.sh | 1 -\n t/t5512-ls-remote.sh | 2 +-\n t/test-lib-functions.sh | 36 ++++++++++++++++++++++++++++++++++++\n 7 files changed, 41 insertions(+), 13 deletions(-)\n\ndiff --git a/t/lib-git-daemon.sh b/t/lib-git-daemon.sh\nindex f98de95c15..41eb1e3ae8 100644\n--- a/t/lib-git-daemon.sh\n+++ b/t/lib-git-daemon.sh\n@@ -28,7 +28,7 @@ then\n \ttest_skip_or_die $GIT_TEST_GIT_DAEMON \"file system does not support FIFOs\"\n fi\n \n-LIB_GIT_DAEMON_PORT=${LIB_GIT_DAEMON_PORT-${this_test#t}}\n+test_set_port LIB_GIT_DAEMON_PORT\n \n GIT_DAEMON_PID=\n GIT_DAEMON_DOCUMENT_ROOT_PATH=\"$PWD\"/repo\ndiff --git a/t/lib-git-p4.sh b/t/lib-git-p4.sh\nindex c27599474c..b3be3ba011 100644\n--- a/t/lib-git-p4.sh\n+++ b/t/lib-git-p4.sh\n@@ -53,14 +53,7 @@ time_in_seconds () {\n \t(cd / && \"$PYTHON_PATH\" -c 'import time; print(int(time.time()))')\n }\n \n-# Try to pick a unique port: guess a large number, then hope\n-# no more than one of each test is running.\n-#\n-# This does not handle the case where somebody else is running the\n-# same tests and has chosen the same ports.\n-testid=${this_test#t}\n-git_p4_test_start=9800\n-P4DPORT=$((10669 + ($testid - $git_p4_test_start)))\n+test_set_port P4DPORT\n \n P4PORT=localhost:$P4DPORT\n P4CLIENT=client\ndiff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh\nindex a8130f9119..f3b478c307 100644\n--- a/t/lib-git-svn.sh\n+++ b/t/lib-git-svn.sh\n@@ -13,6 +13,7 @@ fi\n GIT_DIR=$PWD/.git\n GIT_SVN_DIR=$GIT_DIR/svn/refs/remotes/git-svn\n SVN_TREE=$GIT_SVN_DIR/svn-tree\n+test_set_port SVNSERVE_PORT\n \n svn >/dev/null 2>&1\n if test $? -ne 1\n@@ -119,7 +120,6 @@ require_svnserve () {\n }\n \n start_svnserve () {\n-\tSVNSERVE_PORT=${SVNSERVE_PORT-${this_test#t}}\n \tsvnserve --listen-port $SVNSERVE_PORT \\\n \t\t --root \"$rawsvnrepo\" \\\n \t\t --listen-once \\\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex a8729f8232..e465116ef9 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -82,7 +82,7 @@ case $(uname) in\n esac\n \n LIB_HTTPD_PATH=${LIB_HTTPD_PATH-\"$DEFAULT_HTTPD_PATH\"}\n-LIB_HTTPD_PORT=${LIB_HTTPD_PORT-${this_test#t}}\n+test_set_port LIB_HTTPD_PORT\n \n TEST_PATH=\"$TEST_DIRECTORY\"/lib-httpd\n HTTPD_ROOT_PATH=\"$PWD\"/httpd\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex ba3887f178..0aca8d7588 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -480,7 +480,6 @@ test_expect_success 'gc stops traversal when a missing but promised object is re\n \t! grep \"$TREE_HASH\" out\n '\n \n-LIB_HTTPD_PORT=12345 # default port, 410, cannot be used as non-root\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \ndiff --git a/t/t5512-ls-remote.sh b/t/t5512-ls-remote.sh\nindex 32e722db2e..cd9e60632d 100755\n--- a/t/t5512-ls-remote.sh\n+++ b/t/t5512-ls-remote.sh\n@@ -260,7 +260,7 @@ test_lazy_prereq GIT_DAEMON '\n # This test spawns a daemon, so run it only if the user would be OK with\n # testing with git-daemon.\n test_expect_success PIPE,JGIT,GIT_DAEMON 'indicate no refs in standards-compliant empty remote' '\n-\tJGIT_DAEMON_PORT=${JGIT_DAEMON_PORT-${this_test#t}} &&\n+\ttest_set_port JGIT_DAEMON_PORT &&\n \tJGIT_DAEMON_PID= &&\n \tgit init --bare empty.git &&\n \t>empty.git/git-daemon-export-ok &&\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6b3bbf99e4..d9a602cd0f 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1263,3 +1263,39 @@ test_oid () {\n \tfi &&\n \teval \"printf '%s' \\\"\\${$var}\\\"\"\n }\n+\n+# Choose a port number based on the test script's number and store it in\n+# the given variable name, unless that variable already contains a number.\n+test_set_port () {\n+\tlocal var=$1 port\n+\n+\tif test $# -ne 1 || test -z \"$var\"\n+\tthen\n+\t\tBUG \"test_set_port requires a variable name\"\n+\tfi\n+\n+\teval port=\\\"\\${$var}\\\"\n+\tcase \"$port\" in\n+\t\"\")\n+\t\t# No port is set in the given env var, use the test\n+\t\t# number as port number instead.\n+\t\t# Remove not only the leading 't', but all leading zeros\n+\t\t# as well, so the arithmetic below won't (mis)interpret\n+\t\t# a test number like '0123' as an octal value.\n+\t\tport=${this_test#${this_test%%[1-9]*}}\n+\t\tif test \"${port:-0}\" -lt 1024\n+\t\tthen\n+\t\t\t# root-only port, use a larger one instead.\n+\t\t\tport=$(($port + 10000))\n+\t\tfi\n+\n+\t\teval $var=$port\n+\t\t;;\n+\t*[^0-9]*)\n+\t\terror >&7 \"invalid port number: $port\"\n+\t\t;;\n+\t*)\n+\t\t# The user has specified the port.\n+\t\t;;\n+\tesac\n+}\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364563","messageId":"20181204163457.15717-4-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181204163457.15717-1-szeder.dev@gmail.com","subject":"[RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-04T16:34:57Z","receivedAt":"2018-12-04T16:35:24Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Unfortunately, we have a few flaky tests, whose failures tend to be\nhard to reproduce. We've found that the best we can do to reproduce\nsuch a failure is to run the test repeatedly while the machine is\nunder load, and wait in the hope that the load creates enough variance\nin the timing of the test's commands that a failure is evenually\ntriggered. I have a command to do that, and I noticed that two other\ncontributors have rolled their own scripts to do the same, all\nchoosing slightly different approaches.\n\nTo help reproduce failures in flaky tests, introduce the '--stress'\noption to run a test script repeatedly in multiple parallel\ninvocations until one of them fails, thereby using the test script\nitself to increase the load on the machine.\n\nThe number of parallel invocations is determined by, in order of\nprecedence: the number specified as '--stress=', or the value of\nthe GIT_TEST_STRESS_LOAD environment variable, or twice the number of\navailable processors in '/proc/cpuinfo', or 8.\n\nTo prevent the several parallel invocations of the same test from\ninterfering with each other:\n\n - Include the parallel job's number in the name of the trash\n directory and the various output files under 't/test-results/' as\n a '.stress-' suffix.\n\n - Add the parallel job's number to the port number specified by the\n user or to the test number, so even tests involving daemons\n listening on a TCP socket can be stressed.\n\n - Make '--stress' imply '--verbose-log' and discard the test's\n standard ouput and error; dumping the output of several parallel\n tests to the terminal would create a big ugly mess.\n\n'wait' for all parallel jobs before exiting (either because a failure\nwas found or because the user lost patience and aborted the stress\ntest), allowing the still running tests to finish. Otherwise the \"OK\nX.Y\" progress output from the last iteration would likely arrive after\nthe user got back the shell prompt, interfering with typing in the\nnext command. OTOH, this waiting might induce a considerable delay\nbetween hitting ctrl-C and the test actually exiting; I'm not sure\nthis is the right tradeoff.\n\nBased on Jeff King's 'stress' script.\n\nSigned-off-by: SZEDER Gábor \n---\n t/README | 13 ++++++-\n t/test-lib-functions.sh | 7 +++-\n t/test-lib.sh | 82 +++++++++++++++++++++++++++++++++++++++--\n 3 files changed, 96 insertions(+), 6 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 28711cc508..9851de25c2 100644\n--- a/t/README\n+++ b/t/README\n@@ -186,6 +186,16 @@ appropriately before running \"make\".\n \tthis feature by setting the GIT_TEST_CHAIN_LINT environment\n \tvariable to \"1\" or \"0\", respectively.\n \n+--stress::\n+--stress=::\n+\tRun the test script repeatedly in multiple parallel\n+\tinvocations until one of them fails. Useful for reproducing\n+\trare failures in flaky tests. The number of parallel\n+\tinvocations is, in order of precedence: , or the value of\n+\tthe GIT_TEST_STRESS_LOAD environment variable, or twice the\n+\tnumber of available processors in '/proc/cpuinfo', or 8.\n+\tImplies `--verbose-log`.\n+\n You can also set the GIT_TEST_INSTALLED environment variable to\n the bindir of an existing git installation to test that installation.\n You still need to have built this git sandbox, from which various\n@@ -425,7 +435,8 @@ This test harness library does the following things:\n - Creates an empty test directory with an empty .git/objects database\n and chdir(2) into it. This directory is 't/trash\n directory.$test_name_without_dotsh', with t/ subject to change by\n- the --root option documented above.\n+ the --root option documented above, and a '.stress-' suffix\n+ appended by the --stress option.\n \n - Defines standard test helper functions for your scripts to\n use. These functions are designed to make all scripts behave\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex d9a602cd0f..9af11e3eed 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1288,8 +1288,6 @@ test_set_port () {\n \t\t\t# root-only port, use a larger one instead.\n \t\t\tport=$(($port + 10000))\n \t\tfi\n-\n-\t\teval $var=$port\n \t\t;;\n \t*[^0-9]*)\n \t\terror >&7 \"invalid port number: $port\"\n@@ -1298,4 +1296,9 @@ test_set_port () {\n \t\t# The user has specified the port.\n \t\t;;\n \tesac\n+\n+\t# Make sure that parallel '--stress' test jobs get different\n+\t# ports.\n+\tport=$(($port + ${GIT_TEST_STRESS_JOB_NR:-0}))\n+\teval $var=$port\n }\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 49e4563405..9b7f687396 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -71,8 +71,81 @@ then\n \texit 1\n fi\n \n+TEST_STRESS_SFX=\"${GIT_TEST_STRESS_JOB_NR:+.stress-$GIT_TEST_STRESS_JOB_NR}\"\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n-TEST_RESULTS_BASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$TEST_NAME\"\n+TEST_RESULTS_BASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$TEST_NAME$TEST_STRESS_SFX\"\n+\n+# If --stress was passed, run this test repeatedly in several parallel loops.\n+case \"$GIT_TEST_STRESS_STARTED, $* \" in\n+done,*)\n+\t# Don't stress test again.\n+\t;;\n+*' --stress '*|*' '--stress=*' '*)\n+\tjob_count=${*##*--stress=}\n+\tif test \"$job_count\" != \"$*\"\n+\tthen\n+\t\tjob_count=${job_count%% *}\n+\telif test -n \"$GIT_TEST_STRESS_LOAD\"\n+\tthen\n+\t\tjob_count=\"$GIT_TEST_STRESS_LOAD\"\n+\telif test -r /proc/cpuinfo\n+\tthen\n+\t\tjob_count=$((2 * $(grep -c ^processor /proc/cpuinfo)))\n+\telse\n+\t\tjob_count=8\n+\tfi\n+\n+\tmkdir -p \"$(dirname \"$TEST_RESULTS_BASE\")\"\n+\tstressfail=\"$TEST_RESULTS_BASE.stress-failed\"\n+\trm -f \"$stressfail\"\n+\ttrap 'echo aborted >\"$stressfail\"' TERM INT HUP\n+\n+\tjob_nr=0\n+\twhile test $job_nr -lt \"$job_count\"\n+\tdo\n+\t\t(\n+\t\t\tGIT_TEST_STRESS_STARTED=done\n+\t\t\tGIT_TEST_STRESS_JOB_NR=$job_nr\n+\t\t\texport GIT_TEST_STRESS_STARTED GIT_TEST_STRESS_JOB_NR\n+\n+\t\t\tcnt=0\n+\t\t\twhile ! test -e \"$stressfail\"\n+\t\t\tdo\n+\t\t\t\tif $TEST_SHELL_PATH \"$0\" \"$@\" >/dev/null 2>&1\n+\t\t\t\tthen\n+\t\t\t\t\tprintf >&2 \"OK %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n+\t\t\t\telif test -f \"$stressfail\" &&\n+\t\t\t\t test \"$(cat \"$stressfail\")\" = \"aborted\"\n+\t\t\t\tthen\n+\t\t\t\t\tprintf >&2 \"ABORTED %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n+\t\t\t\telse\n+\t\t\t\t\tprintf >&2 \"FAIL %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n+\t\t\t\t\techo $GIT_TEST_STRESS_JOB_NR >>\"$stressfail\"\n+\t\t\t\tfi\n+\t\t\t\tcnt=$(($cnt + 1))\n+\t\t\tdone\n+\t\t) &\n+\t\tjob_nr=$(($job_nr + 1))\n+\tdone\n+\n+\tjob_nr=0\n+\twhile test $job_nr -lt \"$job_count\"\n+\tdo\n+\t\twait\n+\t\tjob_nr=$(($job_nr + 1))\n+\tdone\n+\n+\tif test -f \"$stressfail\" && test \"$(cat \"$stressfail\")\" != \"aborted\"\n+\tthen\n+\t\techo \"Log(s) of failed test run(s) be found in:\"\n+\t\tfor f in $(cat \"$stressfail\")\n+\t\tdo\n+\t\t\techo \" $TEST_RESULTS_BASE.stress-$f.out\"\n+\t\tdone\n+\tfi\n+\texit\n+\t;;\n+esac\n \n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n@@ -80,7 +153,7 @@ case \"$GIT_TEST_TEE_STARTED, $* \" in\n done,*)\n \t# do not redirect again\n \t;;\n-*' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*)\n+*' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*|*' --stress '*|*' '--stress=*' '*)\n \tmkdir -p \"$(dirname \"$TEST_RESULTS_BASE\")\"\n \n \t# Make this filename available to the sub-process in case it is using\n@@ -341,6 +414,9 @@ do\n \t-V|--verbose-log)\n \t\tverbose_log=t\n \t\tshift ;;\n+\t--stress|--stress=*)\n+\t\tverbose_log=t\n+\t\tshift ;;\n \t*)\n \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n \tesac\n@@ -1028,7 +1104,7 @@ then\n fi\n \n # Test repository\n-TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME$TEST_STRESS_SFX\"\n test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n case \"$TRASH_DIRECTORY\" in\n /*) ;; # absolute path is good\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364564","messageId":"87o9a1z0j5.fsf@evledraar.gmail.com","threadId":"49946","inReplyTo":"20181204163457.15717-4-szeder.dev@gmail.com","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-12-04T17:04:14Z","receivedAt":"2018-12-04T17:04:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Dec 04 2018, SZEDER Gábor wrote:\n\n> The number of parallel invocations is determined by, in order of\n> precedence: the number specified as '--stress=', or the value of\n> the GIT_TEST_STRESS_LOAD environment variable, or twice the number of\n> available processors in '/proc/cpuinfo', or 8.\n\nWith this series we have at least 3 ways to get this number. First\nonline_cpus() in the C code, then Peff's recent `getconf\n_NPROCESSORS_ONLN` in doc-diff, and now this /proc/cpuinfo parsing.\n\nPerhaps it makes sense to split online_cpus() into a helper to use from\nthe shellscripts instead?\n"},{"id":"364565","messageId":"20181204173721.GI30222@szeder.dev","threadId":"49946","inReplyTo":"87o9a1z0j5.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-04T17:37:21Z","receivedAt":"2018-12-04T17:37:27Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Dec 04, 2018 at 06:04:14PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Dec 04 2018, SZEDER Gábor wrote:\n> \n> > The number of parallel invocations is determined by, in order of\n> > precedence: the number specified as '--stress=', or the value of\n> > the GIT_TEST_STRESS_LOAD environment variable, or twice the number of\n> > available processors in '/proc/cpuinfo', or 8.\n> \n> With this series we have at least 3 ways to get this number. First\n> online_cpus() in the C code, then Peff's recent `getconf\n> _NPROCESSORS_ONLN` in doc-diff, and now this /proc/cpuinfo parsing.\n> \n> Perhaps it makes sense to split online_cpus() into a helper to use from\n> the shellscripts instead?\n\nI don't think so, but I will update this patch to use 'getconf'\ninstead.\n\n"},{"id":"364568","messageId":"87muplyxfn.fsf@evledraar.gmail.com","threadId":"49946","inReplyTo":"20181204163457.15717-4-szeder.dev@gmail.com","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-12-04T18:11:08Z","receivedAt":"2018-12-04T18:11:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Dec 04 2018, SZEDER Gábor wrote:\n\n> Unfortunately, we have a few flaky tests, whose failures tend to be\n> hard to reproduce. We've found that the best we can do to reproduce\n> such a failure is to run the test repeatedly while the machine is\n> under load, and wait in the hope that the load creates enough variance\n> in the timing of the test's commands that a failure is evenually\n> triggered. I have a command to do that, and I noticed that two other\n> contributors have rolled their own scripts to do the same, all\n> choosing slightly different approaches.\n>\n> To help reproduce failures in flaky tests, introduce the '--stress'\n> option to run a test script repeatedly in multiple parallel\n> invocations until one of them fails, thereby using the test script\n> itself to increase the load on the machine.\n>\n> The number of parallel invocations is determined by, in order of\n> precedence: the number specified as '--stress=', or the value of\n> the GIT_TEST_STRESS_LOAD environment variable, or twice the number of\n> available processors in '/proc/cpuinfo', or 8.\n>\n> To prevent the several parallel invocations of the same test from\n> interfering with each other:\n>\n> - Include the parallel job's number in the name of the trash\n> directory and the various output files under 't/test-results/' as\n> a '.stress-' suffix.\n>\n> - Add the parallel job's number to the port number specified by the\n> user or to the test number, so even tests involving daemons\n> listening on a TCP socket can be stressed.\n>\n> - Make '--stress' imply '--verbose-log' and discard the test's\n> standard ouput and error; dumping the output of several parallel\n> tests to the terminal would create a big ugly mess.\n>\n> 'wait' for all parallel jobs before exiting (either because a failure\n> was found or because the user lost patience and aborted the stress\n> test), allowing the still running tests to finish. Otherwise the \"OK\n> X.Y\" progress output from the last iteration would likely arrive after\n> the user got back the shell prompt, interfering with typing in the\n> next command. OTOH, this waiting might induce a considerable delay\n> between hitting ctrl-C and the test actually exiting; I'm not sure\n> this is the right tradeoff.\n\nI think it makes sense to generalize this and split it up into two\nfeatures.\n\nIt's a frequent annoyance of mine in the test suite that I'm\ne.g. running t*.sh with some parallel \"prove\" in one screen, and then I\nrun tABCD*.sh manually, and get unlucky because they use the same trash\ndir, and both tests go boom.\n\nYou can fix that with --root, which is much of what this patch does. My\none-liner for doing --stress has been something like:\n\n perl -E 'say ++$_ while 1' | parallel --jobs=100% --halt-on-error soon,fail=1 './t0000-basic.sh --root=trash-{} -v'\n\nBut it would be great if I didn't have to worry about that and could\njust run two concurrent:\n\n ./t0000-basic.sh\n\nSo I think we could just set some env variable where instead of having\nthe predictable trash directory we have a $TRASHDIR.$N as this patch\ndoes, except we pick $N by locking some test-runs/tABCD.Nth file with\nflock() during the run.\n\nThen a stress mode like this would just be:\n\n GIT_TEST_FLOCKED_TRASH_DIRS=1 perl -E 'say ++$_ while 1' | parallel --jobs=100% --halt-on-error soon,fail=1 './t0000-basic.sh'\n\nAnd sure, we could ship a --stress option too, but it wouldn't be\nmagical in any way, just another \"spawn N in a loop\" implementation, but\nyou could also e.g. use GNU parallel to drive it, and without needing to\ndecide to stress test in advance, since we'd either flock() the trash\ndir, or just mktemp(1)-it.\n"},{"id":"364603","messageId":"20181205045714.GC12284@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181204163457.15717-2-szeder.dev@gmail.com","subject":"Re: [PATCH 1/3] test-lib: consolidate naming of test-results paths","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T04:57:15Z","receivedAt":"2018-12-05T04:57:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 04, 2018 at 05:34:55PM +0100, SZEDER Gábor wrote:\n\n> There are two places where we strip off any leading path components\n> and the '.sh' suffix from the test script's pathname, and there are\n> two places where we construct the filename of test output files in\n> 't/test-results/'. The last patch in this series will add even more.\n> \n> Factor these out into helper variables to avoid repeating ourselves.\n\nMakes sense.\n\n> +TEST_NAME=\"$(basename \"$0\" .sh)\"\n> +TEST_RESULTS_BASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$TEST_NAME\"\n\nHmm, since we are building up this BASE variable anyway, why not:\n\n TEST_RESULTS_DIR=$TEST_OUTPUT_DIRECTORY/test-results\n TEST_RESULTS_BASE=$TEST_RESULTS_DIR/$TEST_NAME\n\n? That saves having to run `dirname` on it later.\n\nI guess one could argue that saying \"the directory name of the file I'm\nwriting\" is more readable. I just generally try to avoid extra\nmanipulation of the strings when possible (especially in shell).\n\n-Peff\n"},{"id":"364605","messageId":"20181205051709.GD12284@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181204163457.15717-3-szeder.dev@gmail.com","subject":"Re: [PATCH 2/3] test-lib-functions: introduce the 'test_set_port' helper function","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T05:17:09Z","receivedAt":"2018-12-05T05:17:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 04, 2018 at 05:34:56PM +0100, SZEDER Gábor wrote:\n\n> Several test scripts run daemons like 'git-daemon' or Apache, and\n> communicate with them through TCP sockets. To have unique ports where\n> these daemons are accessible, the ports are usually the number of the\n> corresponding test scripts, unless the user overrides them via\n> environment variables, and thus all those tests and test libs contain\n> more or less the same bit of one-liner boilerplate code to find out\n> the port. The last patch in this series will make this a bit more\n> complicated.\n> \n> Factor out finding the port for a daemon into the common helper\n> function 'test_set_port' to avoid repeating ourselves.\n\nOK. Looks like this should keep the same port numbers for all of the\nexisting tests, which I think is good. As nice as choosing random high\nport numbers might be for some cases, it can also be confusing when\nthere are random conflicts.\n\nI've also run into non-random conflicts, but at least once you figure\nthem out they're predictable (the most confusing I've seen is that adb,\nthe android debugger, sometimes but not always leaves a daemon hanging\naround on port 5561).\n\n> Take special care of test scripts with \"low\" numbers:\n> \n> - Test numbers below 1024 would result in a port that's only usable\n> as root, so set their port to '10000 + test-nr' to make sure it\n> doesn't interfere with other tests in the test suite. This makes\n> the hardcoded port number in 't0410-partial-clone.sh' unnecessary,\n> remove it.\n\nThis part is what made me think a bit more about conflicting with\ndynamically allocated ports. Arguably the http parts of t0410 ought to\nbe in a much higher-numbered script, but I don't know that we've held\nover the years very well to the idea that scripts should only depend on\nthe bits from lower numbered scripts.\n\n> ---\n> t/lib-git-daemon.sh | 2 +-\n> t/lib-git-p4.sh | 9 +--------\n> t/lib-git-svn.sh | 2 +-\n> t/lib-httpd.sh | 2 +-\n> t/t0410-partial-clone.sh | 1 -\n> t/t5512-ls-remote.sh | 2 +-\n> t/test-lib-functions.sh | 36 ++++++++++++++++++++++++++++++++++++\n> 7 files changed, 41 insertions(+), 13 deletions(-)\n\nThe patch itself looks good to me.\n\n> +\teval port=\\\"\\${$var}\\\"\n> +\tcase \"$port\" in\n\nThe quotes in the eval'd variable assignment aren't necessary. Usually I\ndon't mind them in a simple:\n\n FOO=\"$bar\"\n\neven though they're redundant (and there were a few instances in the\nprior patch, I think). But here the escaping makes it harder to read,\ncompared to:\n\n eval port=\\$$var\n\nIt might be worth simplifying, but I don't feel strongly about it (we'd\nrun into problems if $var contained spaces with either variant, but I\ndon't think that's worth caring about).\n\n-Peff\n"},{"id":"364606","messageId":"20181205054408.GE12284@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181204163457.15717-4-szeder.dev@gmail.com","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T05:44:09Z","receivedAt":"2018-12-05T05:44:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 04, 2018 at 05:34:57PM +0100, SZEDER Gábor wrote:\n\n> To prevent the several parallel invocations of the same test from\n> interfering with each other:\n> \n> - Include the parallel job's number in the name of the trash\n> directory and the various output files under 't/test-results/' as\n> a '.stress-' suffix.\n\nMakes sense.\n\n> - Add the parallel job's number to the port number specified by the\n> user or to the test number, so even tests involving daemons\n> listening on a TCP socket can be stressed.\n\nIn my script I just use an arbitrary sequence of ports. That means two\nstress runs may stomp on each other, but stress runs tend not to\ninterfere with regular runs (whereas with your scheme, a stress run of\nt5561 is going to stomp on t5562). It probably doesn't matter much\neither way, as I tend not to do too much with the machine during a\nstress run.\n\n> - Make '--stress' imply '--verbose-log' and discard the test's\n> standard ouput and error; dumping the output of several parallel\n> tests to the terminal would create a big ugly mess.\n\nMakes sense. My script just redirects the output to a file, but it\npredates --verbose-log.\n\nMy script always runs with \"-x\". I guess it's not that hard to add it if\nyou want, but it is annoying to finally get a failure after several\nminutes only to realize that your are missing some important\ninformation. ;)\n\nDitto for \"-i\", which leaves more readable output (you know the\ninteresting part is at the end of the log), and leaves a trash directory\nstate that is more likely to be useful for inspecting.\n\nMy script also implies \"--root\". That's not strictly necessary, though I\nsuspect it is much more likely to catch races when it's run on a ram\ndisk (simply because it leaves the CPU as the main bottleneck).\n\n> 'wait' for all parallel jobs before exiting (either because a failure\n> was found or because the user lost patience and aborted the stress\n> test), allowing the still running tests to finish. Otherwise the \"OK\n> X.Y\" progress output from the last iteration would likely arrive after\n> the user got back the shell prompt, interfering with typing in the\n> next command. OTOH, this waiting might induce a considerable delay\n> between hitting ctrl-C and the test actually exiting; I'm not sure\n> this is the right tradeoff.\n\nIf there is a delay, it's actually quite annoying to _not_ wait; you\nstart doing something in the terminal, and then a bunch of extra test\nstatuses print at a random time.\n\n> +\tjob_nr=0\n> +\twhile test $job_nr -lt \"$job_count\"\n> +\tdo\n> +\t\t(\n> +\t\t\tGIT_TEST_STRESS_STARTED=done\n> +\t\t\tGIT_TEST_STRESS_JOB_NR=$job_nr\n> +\t\t\texport GIT_TEST_STRESS_STARTED GIT_TEST_STRESS_JOB_NR\n> +\n> +\t\t\tcnt=0\n> +\t\t\twhile ! test -e \"$stressfail\"\n> +\t\t\tdo\n> +\t\t\t\tif $TEST_SHELL_PATH \"$0\" \"$@\" >/dev/null 2>&1\n> +\t\t\t\tthen\n> +\t\t\t\t\tprintf >&2 \"OK %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n\nOK, this adds a counter for each job number (compared to my script).\nSeems helpful.\n\n> +\t\t\t\telif test -f \"$stressfail\" &&\n> +\t\t\t\t test \"$(cat \"$stressfail\")\" = \"aborted\"\n> +\t\t\t\tthen\n> +\t\t\t\t\tprintf >&2 \"ABORTED %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n> +\t\t\t\telse\n> +\t\t\t\t\tprintf >&2 \"FAIL %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n> +\t\t\t\t\techo $GIT_TEST_STRESS_JOB_NR >>\"$stressfail\"\n> +\t\t\t\tfi\n\nHmm. What happens if we see a failure _and_ there's an abort? That's\nactually pretty plausible if you see a failure whiz by, and you want to\nabort the remaining scripts because they take a long time to run.\n\nIf the abort happens first, then the goal is to assume any errors are\ndue to the abort. But there's a race between the individual jobs reading\n$stressfail and the parent signal handler writing it. So you may end up\nwith \"aborted\\n5\" or similar (after which any other jobs would fail to\nmatch \"aborted\" and declare themselves failures, as well). I think my\nscript probably also has this race, too (it doesn't check the abort case\nexplicitly, but it would race in checking \"test -e fail\").\n\nIf the fail happens first (which I think is the more likely case), then\nthe abort handler will overwrite the file with \"aborted\", and we'll\nforget that there was a real failure. This works in my script because of\nthe symlinking (if a real failure symlinked $fail to a directory, then\nthe attempt to write \"aborted\" will just be a noop).\n\n> +\tjob_nr=0\n> +\twhile test $job_nr -lt \"$job_count\"\n> +\tdo\n> +\t\twait\n> +\t\tjob_nr=$(($job_nr + 1))\n> +\tdone\n\nDo we need to loop? Calling \"wait\" with no arguments should wait for all\nchildren.\n\n> +\tif test -f \"$stressfail\" && test \"$(cat \"$stressfail\")\" != \"aborted\"\n> +\tthen\n> +\t\techo \"Log(s) of failed test run(s) be found in:\"\n> +\t\tfor f in $(cat \"$stressfail\")\n> +\t\tdo\n> +\t\t\techo \" $TEST_RESULTS_BASE.stress-$f.out\"\n> +\t\tdone\n> +\tfi\n\nIn my experience, outputting the failed log saves a step (especially\nwith \"-i\"), since seeing the failed test and its output is often\nsufficient.\n\nI'm also sad to lose the symlink to the failed trash directory, which\nsaves cutting and pasting when you have to inspect it. But I guess we\ncan't rely on symlinks everywhere.\n\n-Peff\n"},{"id":"364607","messageId":"20181205054637.GF12284@sigill.intra.peff.net","threadId":"49946","inReplyTo":"87o9a1z0j5.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T05:46:37Z","receivedAt":"2018-12-05T05:46:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 04, 2018 at 06:04:14PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Tue, Dec 04 2018, SZEDER Gábor wrote:\n> \n> > The number of parallel invocations is determined by, in order of\n> > precedence: the number specified as '--stress=', or the value of\n> > the GIT_TEST_STRESS_LOAD environment variable, or twice the number of\n> > available processors in '/proc/cpuinfo', or 8.\n> \n> With this series we have at least 3 ways to get this number. First\n> online_cpus() in the C code, then Peff's recent `getconf\n> _NPROCESSORS_ONLN` in doc-diff, and now this /proc/cpuinfo parsing.\n\nTo be fair, I only added the \"getconf\" thing because somebody complained\nthat I was parsing /proc/cpuinfo, and suggested it instead. ;)\n\nI don't think it's especially portable, but it should work on Linux and\nmodern BSD/macOS, which may be enough (unlike doc-diff, I'd be a little\nmore inclined to ask somebody on a random platform to stress test if\nthey're hitting a bug).\n\n> Perhaps it makes sense to split online_cpus() into a helper to use from\n> the shellscripts instead?\n\nI think somebody proposed that in a recent thread for other reasons,\ntoo. I'd be OK with that, but probably just using getconf is reasonable\nin the meantime.\n\n-Peff\n"},{"id":"364608","messageId":"20181205055024.GG12284@sigill.intra.peff.net","threadId":"49946","inReplyTo":"87muplyxfn.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T05:50:24Z","receivedAt":"2018-12-05T05:50:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 04, 2018 at 07:11:08PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> It's a frequent annoyance of mine in the test suite that I'm\n> e.g. running t*.sh with some parallel \"prove\" in one screen, and then I\n> run tABCD*.sh manually, and get unlucky because they use the same trash\n> dir, and both tests go boom.\n> \n> You can fix that with --root, which is much of what this patch does. My\n> one-liner for doing --stress has been something like:\n> \n> perl -E 'say ++$_ while 1' | parallel --jobs=100% --halt-on-error soon,fail=1 './t0000-basic.sh --root=trash-{} -v'\n> \n> But it would be great if I didn't have to worry about that and could\n> just run two concurrent:\n> \n> ./t0000-basic.sh\n> \n> So I think we could just set some env variable where instead of having\n> the predictable trash directory we have a $TRASHDIR.$N as this patch\n> does, except we pick $N by locking some test-runs/tABCD.Nth file with\n> flock() during the run.\n\nThat actually sounds kind of annoying when doing a single run, since now\nyou have this extra \".N\". I guess it would at least be predictable, and\nI tend to tab-complete the trash dirs anyway.\n\nI accomplish the same thing by doing my \"big\" runs using --root\nspecified in config.mak (which points to a RAM disk -- if you're not\nusing one to run the tests, you really should look into it, as it's way\nfaster). And then for one-offs, investigating failures, etc, I do \"cd t\n&& ./t0000-basic.sh -v -i\", which naturally runs in the local directory.\n\n-Peff\n"},{"id":"364620","messageId":"20181205103454.GJ30222@szeder.dev","threadId":"49946","inReplyTo":"20181205054408.GE12284@sigill.intra.peff.net","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-05T10:34:54Z","receivedAt":"2018-12-05T10:35:01Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\nJust a quick reply to this one point for now:\n\nOn Wed, Dec 05, 2018 at 12:44:09AM -0500, Jeff King wrote:\n> On Tue, Dec 04, 2018 at 05:34:57PM +0100, SZEDER Gábor wrote:\n> > +\tjob_nr=0\n> > +\twhile test $job_nr -lt \"$job_count\"\n> > +\tdo\n> > +\t\twait\n> > +\t\tjob_nr=$(($job_nr + 1))\n> > +\tdone\n> \n> Do we need to loop? Calling \"wait\" with no arguments should wait for all\n> children.\n\nIt should, but in dash, ksh, ksh93, Bash v4.2 or older it doesn't seem\nto do so, at least not on my machine, and I get back my shell prompt\nright after hitting ctrl-C or the first failed test, and then get the\nprogress output from the remaining jobs later.\n\nBash 4.3 or later are strange: I get back the shell prompt immediately\nafter ctrl-C as well, so it doesn't appear to be waiting for all\nremaining jobs to finish either, but! I don't get any of the progress\noutput from those jobs to mess up my next command.\n\nAnd mksh and zsh can't run our tests, and I don't have any more shells\nat hand to try.\n\n"},{"id":"364621","messageId":"20181205120725.GK30222@szeder.dev","threadId":"49946","inReplyTo":"87muplyxfn.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-05T12:07:25Z","receivedAt":"2018-12-05T12:07:32Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Dec 04, 2018 at 07:11:08PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> It's a frequent annoyance of mine in the test suite that I'm\n> e.g. running t*.sh with some parallel \"prove\" in one screen, and then I\n> run tABCD*.sh manually, and get unlucky because they use the same trash\n> dir, and both tests go boom.\n> \n> You can fix that with --root, which is much of what this patch does. My\n> one-liner for doing --stress has been something like:\n> \n> perl -E 'say ++$_ while 1' | parallel --jobs=100% --halt-on-error soon,fail=1 './t0000-basic.sh --root=trash-{} -v'\n> \n> But it would be great if I didn't have to worry about that and could\n> just run two concurrent:\n> \n> ./t0000-basic.sh\n> \n> So I think we could just set some env variable where instead of having\n> the predictable trash directory we have a $TRASHDIR.$N as this patch\n> does, except we pick $N by locking some test-runs/tABCD.Nth file with\n> flock() during the run.\n\nIs 'flock' portable? It doesn't appear so.\n\n> Then a stress mode like this would just be:\n> \n> GIT_TEST_FLOCKED_TRASH_DIRS=1 perl -E 'say ++$_ while 1' | parallel --jobs=100% --halt-on-error soon,fail=1 './t0000-basic.sh'\n\nThis doesn't address the issue of TCP ports for various daemons.\n\n> And sure, we could ship a --stress option too, but it wouldn't be\n> magical in any way, just another \"spawn N in a loop\" implementation, but\n> you could also e.g. use GNU parallel to drive it, and without needing to\n\nGNU 'parallel' may or may not be available on the platform, that's why\nI stuck with the barebones shell-loops-in-the-background approach.\n\n> decide to stress test in advance, since we'd either flock() the trash\n> dir, or just mktemp(1)-it.\n\nWhile 'mktemp' seems to be more portable than 'flock', it doesn't seem\nto be portable enough; at least it's not in POSIX.\n\nAnd in general I'd prefer deterministic trash directory names. If I\nre-run a failed test after fixing the bug, then currently the trash\ndirectory will be overwritten, and that's good. With 'mktemp's junk\nin the trash direcory's name it won't be overwritten, and if my bugfix\ndoesn't work, then I'll start accumulating trash directories and won't\neven know which one is from the last run.\n\n"},{"id":"364622","messageId":"20181205122035.GL30222@szeder.dev","threadId":"49946","inReplyTo":"20181205051709.GD12284@sigill.intra.peff.net","subject":"Re: [PATCH 2/3] test-lib-functions: introduce the 'test_set_port' helper function","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-05T12:20:35Z","receivedAt":"2018-12-05T12:20:43Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Dec 05, 2018 at 12:17:09AM -0500, Jeff King wrote:\n> On Tue, Dec 04, 2018 at 05:34:56PM +0100, SZEDER Gábor wrote:\n> \n> > Several test scripts run daemons like 'git-daemon' or Apache, and\n> > communicate with them through TCP sockets. To have unique ports where\n> > these daemons are accessible, the ports are usually the number of the\n> > corresponding test scripts, unless the user overrides them via\n> > environment variables, and thus all those tests and test libs contain\n> > more or less the same bit of one-liner boilerplate code to find out\n> > the port. The last patch in this series will make this a bit more\n> > complicated.\n> > \n> > Factor out finding the port for a daemon into the common helper\n> > function 'test_set_port' to avoid repeating ourselves.\n> \n> OK. Looks like this should keep the same port numbers for all of the\n> existing tests, which I think is good.\n\nNot for all existing tests, though: t0410 and the 'git p4' tests do\nget different ports.\n\n\nHrm, speaking of affecting 'git p4' tests... I should have put Luke\non Cc, so doing it now.\n\n"},{"id":"364623","messageId":"20181205140106.GM30222@szeder.dev","threadId":"49946","inReplyTo":"20181205054408.GE12284@sigill.intra.peff.net","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-05T14:01:06Z","receivedAt":"2018-12-05T14:01:14Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Dec 05, 2018 at 12:44:09AM -0500, Jeff King wrote:\n> On Tue, Dec 04, 2018 at 05:34:57PM +0100, SZEDER Gábor wrote:\n> \n> > To prevent the several parallel invocations of the same test from\n> > interfering with each other:\n> > \n> > - Include the parallel job's number in the name of the trash\n> > directory and the various output files under 't/test-results/' as\n> > a '.stress-' suffix.\n> \n> Makes sense.\n> \n> > - Add the parallel job's number to the port number specified by the\n> > user or to the test number, so even tests involving daemons\n> > listening on a TCP socket can be stressed.\n> \n> In my script I just use an arbitrary sequence of ports. That means two\n> stress runs may stomp on each other, but stress runs tend not to\n> interfere with regular runs (whereas with your scheme, a stress run of\n> t5561 is going to stomp on t5562). It probably doesn't matter much\n> either way, as I tend not to do too much with the machine during a\n> stress run.\n\nA custom port can always be set via environment variables, though the\nuser has to remember to set it (I doubt that I would remember it until\nreminded by test failures...)\n\n> > - Make '--stress' imply '--verbose-log' and discard the test's\n> > standard ouput and error; dumping the output of several parallel\n> > tests to the terminal would create a big ugly mess.\n> \n> Makes sense. My script just redirects the output to a file, but it\n> predates --verbose-log.\n> \n> My script always runs with \"-x\". I guess it's not that hard to add it if\n> you want, but it is annoying to finally get a failure after several\n> minutes only to realize that your are missing some important\n> information. ;)\n> \n> Ditto for \"-i\", which leaves more readable output (you know the\n> interesting part is at the end of the log), and leaves a trash directory\n> state that is more likely to be useful for inspecting.\n\nI wanted to imply only the bare minimum of options that are necessary\nto prevent the tests from stomping on each other. Other than that I\nagree and wouldn't run '--stress' without '-x -i' myself, and at one\npoint considered to recommend '-x -i' in the description of\n'--stress'.\n\n> My script also implies \"--root\". That's not strictly necessary, though I\n> suspect it is much more likely to catch races when it's run on a ram\n> disk (simply because it leaves the CPU as the main bottleneck).\n> \n> > 'wait' for all parallel jobs before exiting (either because a failure\n> > was found or because the user lost patience and aborted the stress\n> > test), allowing the still running tests to finish. Otherwise the \"OK\n> > X.Y\" progress output from the last iteration would likely arrive after\n> > the user got back the shell prompt, interfering with typing in the\n> > next command. OTOH, this waiting might induce a considerable delay\n> > between hitting ctrl-C and the test actually exiting; I'm not sure\n> > this is the right tradeoff.\n> \n> If there is a delay, it's actually quite annoying to _not_ wait; you\n> start doing something in the terminal, and then a bunch of extra test\n> statuses print at a random time.\n\nAt least the INT/TERM/etc. handler should say something like \"Waiting\nfor background jobs to finish\", to let the user know why it doesn't\nexit right away.\n\nAn alternative would be to exit right away, and make the background\nloops a tad bit smarter to not print these progress lines when\naborting.\n\n> > +\tjob_nr=0\n> > +\twhile test $job_nr -lt \"$job_count\"\n> > +\tdo\n> > +\t\t(\n> > +\t\t\tGIT_TEST_STRESS_STARTED=done\n> > +\t\t\tGIT_TEST_STRESS_JOB_NR=$job_nr\n> > +\t\t\texport GIT_TEST_STRESS_STARTED GIT_TEST_STRESS_JOB_NR\n> > +\n> > +\t\t\tcnt=0\n> > +\t\t\twhile ! test -e \"$stressfail\"\n> > +\t\t\tdo\n> > +\t\t\t\tif $TEST_SHELL_PATH \"$0\" \"$@\" >/dev/null 2>&1\n> > +\t\t\t\tthen\n> > +\t\t\t\t\tprintf >&2 \"OK %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n> \n> OK, this adds a counter for each job number (compared to my script).\n> Seems helpful.\n> \n> > +\t\t\t\telif test -f \"$stressfail\" &&\n> > +\t\t\t\t test \"$(cat \"$stressfail\")\" = \"aborted\"\n> > +\t\t\t\tthen\n> > +\t\t\t\t\tprintf >&2 \"ABORTED %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n> > +\t\t\t\telse\n> > +\t\t\t\t\tprintf >&2 \"FAIL %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n> > +\t\t\t\t\techo $GIT_TEST_STRESS_JOB_NR >>\"$stressfail\"\n> > +\t\t\t\tfi\n> \n> Hmm. What happens if we see a failure _and_ there's an abort? That's\n> actually pretty plausible if you see a failure whiz by, and you want to\n> abort the remaining scripts because they take a long time to run.\n> \n> If the abort happens first, then the goal is to assume any errors are\n> due to the abort.\n\nYeah, that's why I added this ABORTED case. When I hit ctrl-C, a\ncouple of the tests tend to fail, but that's not the rare failure we\nare hoping to find when stress testing, so printing FAIL would be\nmisleading. The test script exits with 1 all the same, so we can't\ntell the difference, but since rare failures are, well, rare, this\nfailure is most likely due to the abort.\n\n> But there's a race between the individual jobs reading\n> $stressfail and the parent signal handler writing it. So you may end up\n> with \"aborted\\n5\" or similar (after which any other jobs would fail to\n> match \"aborted\" and declare themselves failures, as well). I think my\n> script probably also has this race, too (it doesn't check the abort case\n> explicitly, but it would race in checking \"test -e fail\").\n>\n> If the fail happens first (which I think is the more likely case), then\n> the abort handler will overwrite the file with \"aborted\", and we'll\n> forget that there was a real failure. This works in my script because of\n> the symlinking (if a real failure symlinked $fail to a directory, then\n> the attempt to write \"aborted\" will just be a noop).\n\nOK, I think the abort handler should append, not overwrite, and then\nthe file's contents should be checked e.g. with a case statement\nlooking for *aborted*.\n\nOr use two separate files for signaling abort and test failure.\n\nIn any case, you could always grab the job number from the \"FAIL X.Y\"\nline, and then look at its logs and trash direcory.\n\n> > +\tjob_nr=0\n> > +\twhile test $job_nr -lt \"$job_count\"\n> > +\tdo\n> > +\t\twait\n> > +\t\tjob_nr=$(($job_nr + 1))\n> > +\tdone\n> \n> Do we need to loop? Calling \"wait\" with no arguments should wait for all\n> children.\n> \n> > +\tif test -f \"$stressfail\" && test \"$(cat \"$stressfail\")\" != \"aborted\"\n> > +\tthen\n> > +\t\techo \"Log(s) of failed test run(s) be found in:\"\n> > +\t\tfor f in $(cat \"$stressfail\")\n> > +\t\tdo\n> > +\t\t\techo \" $TEST_RESULTS_BASE.stress-$f.out\"\n> > +\t\tdone\n> > +\tfi\n> \n> In my experience, outputting the failed log saves a step (especially\n> with \"-i\"), since seeing the failed test and its output is often\n> sufficient.\n\nI just don't like when test output, especially with '-x', is dumped on\nmy terminal :)\n\n> I'm also sad to lose the symlink to the failed trash directory, which\n> saves cutting and pasting when you have to inspect it. But I guess we\n> can't rely on symlinks everywhere.\n\nYou can tab-complete most of the trash directory, and then there are\nonly those 1-2-3 digits of the job number that you have to type. That\nworked out well enough for me so far (but I've only used it for a few\ndays, so...).\n\nWe could rename the trash directory of the failed test to something\nshort, sweet, and common, but that could be racy in the unlikely case\nthat two tests fail at the same time. And I like the 'trash\ndirectory.' prefix, because then 'make clean' will delete that one,\ntoo, without modifying 't/Makefile' (which is also the reason for\nadding the 'stress-N' suffix instead of a prefix, and putting the fail\nfile into the 'test-results' directory).\n\n"},{"id":"364624","messageId":"87wooof4xm.fsf@evledraar.gmail.com","threadId":"49946","inReplyTo":"20181205120725.GK30222@szeder.dev","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-12-05T14:01:41Z","receivedAt":"2018-12-05T14:01:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Dec 05 2018, SZEDER Gábor wrote:\n\n> On Tue, Dec 04, 2018 at 07:11:08PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> It's a frequent annoyance of mine in the test suite that I'm\n>> e.g. running t*.sh with some parallel \"prove\" in one screen, and then I\n>> run tABCD*.sh manually, and get unlucky because they use the same trash\n>> dir, and both tests go boom.\n>>\n>> You can fix that with --root, which is much of what this patch does. My\n>> one-liner for doing --stress has been something like:\n>>\n>> perl -E 'say ++$_ while 1' | parallel --jobs=100% --halt-on-error soon,fail=1 './t0000-basic.sh --root=trash-{} -v'\n>>\n>> But it would be great if I didn't have to worry about that and could\n>> just run two concurrent:\n>>\n>> ./t0000-basic.sh\n>>\n>> So I think we could just set some env variable where instead of having\n>> the predictable trash directory we have a $TRASHDIR.$N as this patch\n>> does, except we pick $N by locking some test-runs/tABCD.Nth file with\n>> flock() during the run.\n>\n> Is 'flock' portable? It doesn't appear so.\n\nPortable enough, and since it's just an alias for \"grab lock atomically\"\nit can devolve into other more basic FS functions on systems that don't\nhave it.\n\n>> Then a stress mode like this would just be:\n>>\n>> GIT_TEST_FLOCKED_TRASH_DIRS=1 perl -E 'say ++$_ while 1' | parallel --jobs=100% --halt-on-error soon,fail=1 './t0000-basic.sh'\n>\n> This doesn't address the issue of TCP ports for various daemons.\n\nOnce we have a test ABCD and slot offset N (for a fixed size of N) we\ncan pick a port from that.\n\n>> And sure, we could ship a --stress option too, but it wouldn't be\n>> magical in any way, just another \"spawn N in a loop\" implementation, but\n>> you could also e.g. use GNU parallel to drive it, and without needing to\n>\n> GNU 'parallel' may or may not be available on the platform, that's why\n> I stuck with the barebones shell-loops-in-the-background approach.\n\nYes, my point is not that you should drop that part of your patch for\nusing GNU parallel, but just to demonstrate that we can get pretty close\nto this for most tests with an external tool, and that it would make\nthis sort of thing work for concurrent tests without needing to decide\nto concurrently test in advance.\n\n>> decide to stress test in advance, since we'd either flock() the trash\n>> dir, or just mktemp(1)-it.\n>\n> While 'mktemp' seems to be more portable than 'flock', it doesn't seem\n> to be portable enough; at least it's not in POSIX.>\n\nWe are already relying on stuff like mktemp() being reliable for\ne.g. the split index.\n\n> And in general I'd prefer deterministic trash directory names. If I\n> re-run a failed test after fixing the bug, then currently the trash\n> directory will be overwritten, and that's good. With 'mktemp's junk\n> in the trash direcory's name it won't be overwritten, and if my bugfix\n> doesn't work, then I'll start accumulating trash directories and won't\n> even know which one is from the last run.\n\nIn general you wouldn't need to set GIT_TEST_FLOCKED_TRASH_DIRS=1 and\nwould get the same monolithic trash names as now, and for something like\nthe flock() version it could use a job number as a suffix like your\npatch does.\n"},{"id":"364626","messageId":"20181205143919.GN30222@szeder.dev","threadId":"49946","inReplyTo":"87wooof4xm.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-05T14:39:19Z","receivedAt":"2018-12-05T14:39:26Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Dec 05, 2018 at 03:01:41PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> >> decide to stress test in advance, since we'd either flock() the trash\n> >> dir, or just mktemp(1)-it.\n> >\n> > While 'mktemp' seems to be more portable than 'flock', it doesn't seem\n> > to be portable enough; at least it's not in POSIX.>\n> \n> We are already relying on stuff like mktemp() being reliable for\n> e.g. the split index.\n\nThat's the mktemp() function from libc; I meant the 'mktemp' command\nthat we would use this early in 'test-lib.sh', where PATH has not been\nset up for testing yet.\n\n"},{"id":"364637","messageId":"87a7ljzqwn.fsf@evledraar.gmail.com","threadId":"49946","inReplyTo":"20181205143919.GN30222@szeder.dev","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-12-05T19:59:04Z","receivedAt":"2018-12-05T19:59:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Dec 05 2018, SZEDER Gábor wrote:\n\n> On Wed, Dec 05, 2018 at 03:01:41PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> >> decide to stress test in advance, since we'd either flock() the trash\n>> >> dir, or just mktemp(1)-it.\n>> >\n>> > While 'mktemp' seems to be more portable than 'flock', it doesn't seem\n>> > to be portable enough; at least it's not in POSIX.>\n>>\n>> We are already relying on stuff like mktemp() being reliable for\n>> e.g. the split index.\n>\n> That's the mktemp() function from libc; I meant the 'mktemp' command\n> that we would use this early in 'test-lib.sh', where PATH has not been\n> set up for testing yet.\n\nAh, if that ever becomes an issue it's a one-line test-tool wrapper.\n"},{"id":"364647","messageId":"20181205213625.GD19936@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181205103454.GJ30222@szeder.dev","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T21:36:26Z","receivedAt":"2018-12-05T21:36:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 05, 2018 at 11:34:54AM +0100, SZEDER Gábor wrote:\n\n> \n> Just a quick reply to this one point for now:\n> \n> On Wed, Dec 05, 2018 at 12:44:09AM -0500, Jeff King wrote:\n> > On Tue, Dec 04, 2018 at 05:34:57PM +0100, SZEDER Gábor wrote:\n> > > +\tjob_nr=0\n> > > +\twhile test $job_nr -lt \"$job_count\"\n> > > +\tdo\n> > > +\t\twait\n> > > +\t\tjob_nr=$(($job_nr + 1))\n> > > +\tdone\n> > \n> > Do we need to loop? Calling \"wait\" with no arguments should wait for all\n> > children.\n> \n> It should, but in dash, ksh, ksh93, Bash v4.2 or older it doesn't seem\n> to do so, at least not on my machine, and I get back my shell prompt\n> right after hitting ctrl-C or the first failed test, and then get the\n> progress output from the remaining jobs later.\n\nThat's weird. I run my stress script under dash with a single \"wait\",\nand it handles the failing case just fine. And switching to a single\n\"wait\" in your script works for me.\n\nBut the ^C case is interesting. Try running your script under \"sh -x\"\nand hitting ^C. The signal interrupts the first wait. In my script (or\nyours modified to use a single wait), we then proceed immediately to the\n\"exit\", and get output from the subshells after we've exited.\n\nBut in your script with the loop, the first wait is interrupted, and\nthen the _second_ wait (in the second loop iteration) picks up all of\nthe exiting processes. The subsequent waits are all noops that return\nimmediately, because there are no processes left.\n\nSo the right number of waits is either \"1\" or \"2\". Looping means we do\ntoo many (which is mostly a harmless noop) or too few (in the off chance\nthat you have only a single job ;) ). So it works out in practice.\n\nBut I think a more clear way to do it is to simply wait in the signal\ntrap, to cover the case when our original wait was aborted.\n\nI.e., something like this:\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9b7f687396..5e0c41626f 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -98,7 +98,7 @@ done,*)\n \tmkdir -p \"$(dirname \"$TEST_RESULTS_BASE\")\"\n \tstressfail=\"$TEST_RESULTS_BASE.stress-failed\"\n \trm -f \"$stressfail\"\n-\ttrap 'echo aborted >\"$stressfail\"' TERM INT HUP\n+\ttrap 'echo aborted >\"$stressfail\"; wait' TERM INT HUP\n \n \tjob_nr=0\n \twhile test $job_nr -lt \"$job_count\"\n@@ -128,12 +128,7 @@ done,*)\n \t\tjob_nr=$(($job_nr + 1))\n \tdone\n \n-\tjob_nr=0\n-\twhile test $job_nr -lt \"$job_count\"\n-\tdo\n-\t\twait\n-\t\tjob_nr=$(($job_nr + 1))\n-\tdone\n+\twait\n \n \tif test -f \"$stressfail\" && test \"$(cat \"$stressfail\")\" != \"aborted\"\n \tthen\n\nwhich behaves well for me in all cases.\n\n> Bash 4.3 or later are strange: I get back the shell prompt immediately\n> after ctrl-C as well, so it doesn't appear to be waiting for all\n> remaining jobs to finish either, but! I don't get any of the progress\n> output from those jobs to mess up my next command.\n\nInteresting. My bash 4.4 seems to behave the same as dash. It almost\nsounds like the SIGINT is getting passed to the subshells for you.\nProbably not really worth digging into, though.\n\nIn case anybody is playing with this and needed to simulate a stress\nfailure, here's what I used:\n\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex b6566003dd..a370cd9977 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -1171,4 +1171,11 @@ test_expect_success 'very long name in the index handled sanely' '\n \ttest $len = 4098\n '\n \n+test_expect_success 'roll those dice' '\n+\tcase \"$(openssl rand -base64 1)\" in\n+\tz*)\n+\t\treturn 1\n+\tesac\n+'\n+\n test_done\n"},{"id":"364649","messageId":"20181205215621.GE19936@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181205140106.GM30222@szeder.dev","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T21:56:21Z","receivedAt":"2018-12-05T21:56:25Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 05, 2018 at 03:01:06PM +0100, SZEDER Gábor wrote:\n\n> > > - Make '--stress' imply '--verbose-log' and discard the test's\n> > > standard ouput and error; dumping the output of several parallel\n> > > tests to the terminal would create a big ugly mess.\n> > \n> > Makes sense. My script just redirects the output to a file, but it\n> > predates --verbose-log.\n> > \n> > My script always runs with \"-x\". I guess it's not that hard to add it if\n> > you want, but it is annoying to finally get a failure after several\n> > minutes only to realize that your are missing some important\n> > information. ;)\n> > \n> > Ditto for \"-i\", which leaves more readable output (you know the\n> > interesting part is at the end of the log), and leaves a trash directory\n> > state that is more likely to be useful for inspecting.\n> \n> I wanted to imply only the bare minimum of options that are necessary\n> to prevent the tests from stomping on each other. Other than that I\n> agree and wouldn't run '--stress' without '-x -i' myself, and at one\n> point considered to recommend '-x -i' in the description of\n> '--stress'.\n\nYeah, it probably is the right thing to make the options as orthogonal\nas possible, and let the user decide what they want. I was just\nwondering if I could replace my \"stress\" script with this. But I think\nI'd still want it to remember to use a sane set of options (including\n\"--root\" in my case).\n\n> > > 'wait' for all parallel jobs before exiting (either because a failure\n> > > was found or because the user lost patience and aborted the stress\n> > > test), allowing the still running tests to finish. Otherwise the \"OK\n> > > X.Y\" progress output from the last iteration would likely arrive after\n> > > the user got back the shell prompt, interfering with typing in the\n> > > next command. OTOH, this waiting might induce a considerable delay\n> > > between hitting ctrl-C and the test actually exiting; I'm not sure\n> > > this is the right tradeoff.\n> > \n> > If there is a delay, it's actually quite annoying to _not_ wait; you\n> > start doing something in the terminal, and then a bunch of extra test\n> > statuses print at a random time.\n> \n> At least the INT/TERM/etc. handler should say something like \"Waiting\n> for background jobs to finish\", to let the user know why it doesn't\n> exit right away.\n\nThat seems reasonable. There's also no point in waiting for them to\nfinish if we know we're aborting anyway. Could we just kill them all?\n\nI guess it's a little tricky, because $! is going to give us the pid of\neach subshell. We actually want to kill the test sub-process. That takes\na few contortions, but the output is nice (every sub-job immediately\nsays \"ABORTED ...\", and the final process does not exit until the whole\ntree is done):\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9b7f687396..357dead3ff 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -98,8 +98,9 @@ done,*)\n \tmkdir -p \"$(dirname \"$TEST_RESULTS_BASE\")\"\n \tstressfail=\"$TEST_RESULTS_BASE.stress-failed\"\n \trm -f \"$stressfail\"\n-\ttrap 'echo aborted >\"$stressfail\"' TERM INT HUP\n+\ttrap 'echo aborted >\"$stressfail\"; kill $job_pids 2>/dev/null; wait' TERM INT HUP\n \n+\tjob_pids=\n \tjob_nr=0\n \twhile test $job_nr -lt \"$job_count\"\n \tdo\n@@ -108,10 +109,13 @@ done,*)\n \t\t\tGIT_TEST_STRESS_JOB_NR=$job_nr\n \t\t\texport GIT_TEST_STRESS_STARTED GIT_TEST_STRESS_JOB_NR\n \n+\t\t\ttrap 'kill $test_pid 2>/dev/null' TERM\n+\n \t\t\tcnt=0\n \t\t\twhile ! test -e \"$stressfail\"\n \t\t\tdo\n-\t\t\t\tif $TEST_SHELL_PATH \"$0\" \"$@\" >/dev/null 2>&1\n+\t\t\t\t$TEST_SHELL_PATH \"$0\" \"$@\" >/dev/null 2>&1 & test_pid=$!\n+\t\t\t\tif wait\n \t\t\t\tthen\n \t\t\t\t\tprintf >&2 \"OK %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n \t\t\t\telif test -f \"$stressfail\" &&\n@@ -124,16 +128,11 @@ done,*)\n \t\t\t\tfi\n \t\t\t\tcnt=$(($cnt + 1))\n \t\t\tdone\n-\t\t) &\n+\t\t) & job_pids=\"$job_pids $!\"\n \t\tjob_nr=$(($job_nr + 1))\n \tdone\n \n-\tjob_nr=0\n-\twhile test $job_nr -lt \"$job_count\"\n-\tdo\n-\t\twait\n-\t\tjob_nr=$(($job_nr + 1))\n-\tdone\n+\twait\n \n \tif test -f \"$stressfail\" && test \"$(cat \"$stressfail\")\" != \"aborted\"\n \tthen\n\n> > > +\t\t\t\telif test -f \"$stressfail\" &&\n> > > +\t\t\t\t test \"$(cat \"$stressfail\")\" = \"aborted\"\n> > > +\t\t\t\tthen\n> > > +\t\t\t\t\tprintf >&2 \"ABORTED %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n> > > +\t\t\t\telse\n> > > +\t\t\t\t\tprintf >&2 \"FAIL %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n> > > +\t\t\t\t\techo $GIT_TEST_STRESS_JOB_NR >>\"$stressfail\"\n> > > +\t\t\t\tfi\n> > \n> > Hmm. What happens if we see a failure _and_ there's an abort? That's\n> > actually pretty plausible if you see a failure whiz by, and you want to\n> > abort the remaining scripts because they take a long time to run.\n> > \n> > If the abort happens first, then the goal is to assume any errors are\n> > due to the abort.\n> \n> Yeah, that's why I added this ABORTED case. When I hit ctrl-C, a\n> couple of the tests tend to fail, but that's not the rare failure we\n> are hoping to find when stress testing, so printing FAIL would be\n> misleading. The test script exits with 1 all the same, so we can't\n> tell the difference, but since rare failures are, well, rare, this\n> failure is most likely due to the abort.\n\nWith the patch as posted I actually see most of them still say \"OK\",\nbecause the SIGINT does not make it to them (it would be a different\nstory if I killed the whole process group).\n\nBut with the modification above, I get \"ABORTED\" for most, which is what\nI'd want. And it should be pretty race-free, since the \"aborted\" file is\nput into place before triggering the signal cascade.\n\n> > If the fail happens first (which I think is the more likely case), then\n> > the abort handler will overwrite the file with \"aborted\", and we'll\n> > forget that there was a real failure. This works in my script because of\n> > the symlinking (if a real failure symlinked $fail to a directory, then\n> > the attempt to write \"aborted\" will just be a noop).\n> \n> OK, I think the abort handler should append, not overwrite, and then\n> the file's contents should be checked e.g. with a case statement\n> looking for *aborted*.\n> \n> Or use two separate files for signaling abort and test failure.\n\nTwo separate files seems a lot simpler to me.\n\n> > In my experience, outputting the failed log saves a step (especially\n> > with \"-i\"), since seeing the failed test and its output is often\n> > sufficient.\n> \n> I just don't like when test output, especially with '-x', is dumped on\n> my terminal :)\n\nDon't you then use your terminal to display the file? ;) (I'm kidding, I\nassume you load it in \"less\" or whatever, which is distinct).\n\nI wish this could be optional somehow, though. I really prefer the dump.\nI guess if I continue to use my script as a wrapper, it can dump for me\n(after digging in the $stressfail file, I guess).\n\n> > I'm also sad to lose the symlink to the failed trash directory, which\n> > saves cutting and pasting when you have to inspect it. But I guess we\n> > can't rely on symlinks everywhere.\n> \n> You can tab-complete most of the trash directory, and then there are\n> only those 1-2-3 digits of the job number that you have to type. That\n> worked out well enough for me so far (but I've only used it for a few\n> days, so...).\n> \n> We could rename the trash directory of the failed test to something\n> short, sweet, and common, but that could be racy in the unlikely case\n> that two tests fail at the same time. And I like the 'trash\n> directory.' prefix, because then 'make clean' will delete that one,\n> too, without modifying 't/Makefile' (which is also the reason for\n> adding the 'stress-N' suffix instead of a prefix, and putting the fail\n> file into the 'test-results' directory).\n\nWe could rename it to \"trash directory.xxx.failed\", which is at least\npredictable. That can even be done atomically, though I think it's a\nlittle tricky to do portably with shell tools. I guess I can continue to\ndo the symlink thing in my wrapper script.\n\n-Peff\n"},{"id":"364650","messageId":"20181205215958.GF19936@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181205122035.GL30222@szeder.dev","subject":"Re: [PATCH 2/3] test-lib-functions: introduce the 'test_set_port' helper function","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T21:59:59Z","receivedAt":"2018-12-05T22:00:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 05, 2018 at 01:20:35PM +0100, SZEDER Gábor wrote:\n\n> On Wed, Dec 05, 2018 at 12:17:09AM -0500, Jeff King wrote:\n> > On Tue, Dec 04, 2018 at 05:34:56PM +0100, SZEDER Gábor wrote:\n> > \n> > > Several test scripts run daemons like 'git-daemon' or Apache, and\n> > > communicate with them through TCP sockets. To have unique ports where\n> > > these daemons are accessible, the ports are usually the number of the\n> > > corresponding test scripts, unless the user overrides them via\n> > > environment variables, and thus all those tests and test libs contain\n> > > more or less the same bit of one-liner boilerplate code to find out\n> > > the port. The last patch in this series will make this a bit more\n> > > complicated.\n> > > \n> > > Factor out finding the port for a daemon into the common helper\n> > > function 'test_set_port' to avoid repeating ourselves.\n> > \n> > OK. Looks like this should keep the same port numbers for all of the\n> > existing tests, which I think is good.\n> \n> Not for all existing tests, though: t0410 and the 'git p4' tests do\n> get different ports.\n\nYeah, I should have said \"most\". The important thing is that they remain\nstatic and predictable for normal non-stress runs, which they do.\n\n-Peff\n"},{"id":"364660","messageId":"xmqqefavbj28.fsf@gitster-ct.c.googlers.com","threadId":"49946","inReplyTo":"20181205213625.GD19936@sigill.intra.peff.net","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-12-06T00:22:23Z","receivedAt":"2018-12-06T00:22:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> But the ^C case is interesting. Try running your script under \"sh -x\"\n> and hitting ^C. The signal interrupts the first wait. In my script (or\n> yours modified to use a single wait), we then proceed immediately to the\n> \"exit\", and get output from the subshells after we've exited.\n>\n> But in your script with the loop, the first wait is interrupted, and\n> then the _second_ wait (in the second loop iteration) picks up all of\n> the exiting processes. The subsequent waits are all noops that return\n> immediately, because there are no processes left.\n>\n> So the right number of waits is either \"1\" or \"2\". Looping means we do\n> too many (which is mostly a harmless noop) or too few (in the off chance\n> that you have only a single job ;) ). So it works out in practice.\n\nWell, if you time your ^C perfectly, no number of waits is right, I\nam afraid. You spawn N processes and while looping N times waiting\nfor them, you can ^C out of wait before these N processes all die,\nno?\n\n> But I think a more clear way to do it is to simply wait in the signal\n> trap, to cover the case when our original wait was aborted.\n\nSounds good.\n"},{"id":"364671","messageId":"20181206053525.GA29481@sigill.intra.peff.net","threadId":"49946","inReplyTo":"xmqqefavbj28.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-06T05:35:25Z","receivedAt":"2018-12-06T05:35:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 06, 2018 at 09:22:23AM +0900, Junio C Hamano wrote:\n\n> > So the right number of waits is either \"1\" or \"2\". Looping means we do\n> > too many (which is mostly a harmless noop) or too few (in the off chance\n> > that you have only a single job ;) ). So it works out in practice.\n> \n> Well, if you time your ^C perfectly, no number of waits is right, I\n> am afraid. You spawn N processes and while looping N times waiting\n> for them, you can ^C out of wait before these N processes all die,\n> no?\n\nEach \"wait\" will try to collect all processes, but may be interrupted by\na signal. So the correct number is actually \"1 plus the number of times\nthe user hits ^C\". I had assumed the user was just hitting it once,\nthough putting the wait into the trap means we do that \"1 plus\" thing\nanyway.\n\nI could also see an argument that subsequent ^C's should exit\nimmediately, but I think we are getting well into the realm of\nover-engineering.\n\n-Peff\n"},{"id":"364673","messageId":"xmqqva4788cw.fsf@gitster-ct.c.googlers.com","threadId":"49946","inReplyTo":"20181206053525.GA29481@sigill.intra.peff.net","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-12-06T06:41:51Z","receivedAt":"2018-12-06T06:42:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Each \"wait\" will try to collect all processes, but may be interrupted by\n> a signal. So the correct number is actually \"1 plus the number of times\n> the user hits ^C\".\n\nYeah and that is not bounded. It is OK not to catch multiple ^C\nthat races with what we do, so having ane extra wait in the clean-up\nprocedure after receiving a signal like you suggested would be both\ngood enough and the cleanest solution, I think.\n\nThanks.\n"},{"id":"364717","messageId":"20181206225601.GO30222@szeder.dev","threadId":"49946","inReplyTo":"20181205213625.GD19936@sigill.intra.peff.net","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-06T22:56:01Z","receivedAt":"2018-12-06T22:56:08Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Dec 05, 2018 at 04:36:26PM -0500, Jeff King wrote:\n> The signal interrupts the first wait.\n\nAh, of course. I'm ashamed to say that this is not the first time I\nforget about that...\n\n> > Bash 4.3 or later are strange: I get back the shell prompt immediately\n> > after ctrl-C as well, so it doesn't appear to be waiting for all\n> > remaining jobs to finish either, but! I don't get any of the progress\n> > output from those jobs to mess up my next command.\n> \n> Interesting. My bash 4.4 seems to behave the same as dash. It almost\n> sounds like the SIGINT is getting passed to the subshells for you.\n> Probably not really worth digging into, though.\n\nThe subshell does indeed get SIGINT. I don't know why that happens,\nto my understanding that should not happen.\n\n> In case anybody is playing with this and needed to simulate a stress\n> failure, here's what I used:\n> \n> diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\n> index b6566003dd..a370cd9977 100755\n> --- a/t/t0000-basic.sh\n> +++ b/t/t0000-basic.sh\n> @@ -1171,4 +1171,11 @@ test_expect_success 'very long name in the index handled sanely' '\n> \ttest $len = 4098\n> '\n> \n> +test_expect_success 'roll those dice' '\n> +\tcase \"$(openssl rand -base64 1)\" in\n> +\tz*)\n> +\t\treturn 1\n> +\tesac\n> +'\n\nWasteful :)\n\n test $(($$ % 42)) -ne 0\n\n"},{"id":"364721","messageId":"20181206231005.GP30222@szeder.dev","threadId":"49946","inReplyTo":"20181205215621.GE19936@sigill.intra.peff.net","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-06T23:10:05Z","receivedAt":"2018-12-06T23:10:12Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Dec 05, 2018 at 04:56:21PM -0500, Jeff King wrote:\n> Could we just kill them all?\n> \n> I guess it's a little tricky, because $! is going to give us the pid of\n> each subshell. We actually want to kill the test sub-process. That takes\n> a few contortions, but the output is nice (every sub-job immediately\n> says \"ABORTED ...\", and the final process does not exit until the whole\n> tree is done):\n\nIt's trickier than that:\n\n - Nowadays our test lib translates SIGINT to exit, but not TERM (or\n HUP, in case a user decides to close the terminal window), which\n means that if the test runs any daemons in the background, then\n those won't be killed when the stress test is aborted.\n\n This is easy enough to address, and I've been meaning to do this\n in an other patch series anyway.\n\n - With the (implied) '--verbose-log' the process killed in the\n background subshell's SIGTERM handler is not the actual test\n process, because '--verbose-log' runs the test again in a piped\n subshell to save its output:\n \n (GIT_TEST_TEE_STARTED=done ${TEST_SHELL_PATH} \"$0\" \"$@\" 2>&1;\n echo $? >\"$TEST_RESULTS_BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n\n That 'kill' kills the \"outer\" shell process.\n And though I get \"ABORTED...\" immediately and I get back my\n prompt right away, the commands involved in the above construct\n are still running in the background:\n\n $ ./t3903-stash.sh --stress=1\n ^CABORTED 0.0\n $ ps a |grep t3903\n 1280 pts/17 S 0:00 /bin/sh ./t3903-stash.sh --stress=1\n 1281 pts/17 S 0:00 tee -a <...>/test-results/t3903-stash.stress-0.out\n 1282 pts/17 S 0:00 /bin/sh ./t3903-stash.sh --stress=1\n 4173 pts/17 S+ 0:00 grep t3903\n\n I see this behavior with all shells I tried.\n I haven't yet started to think it through why this happens :)\n\n Not implying '--verbose-log' but redirecting the test script's\n output, like you did in your 'stress' script, seems to work in\n dash, ksh, and ks93, but not in Bash v4.3 or later, where, for\n whatever reason, the subshell get SIGINT before the SIGTERM would\n arrive.\n While we could easily handle SIGINT in the subshell with the same\n signal handler as SIGTERM, it bothers me that something\n fundamental is wrong here.\n Furthermore, then we should perhaps forbid '--stress' in\n combination with '--verbose-log' or '--tee'.\n \n\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 9b7f687396..357dead3ff 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -98,8 +98,9 @@ done,*)\n> \tmkdir -p \"$(dirname \"$TEST_RESULTS_BASE\")\"\n> \tstressfail=\"$TEST_RESULTS_BASE.stress-failed\"\n> \trm -f \"$stressfail\"\n> -\ttrap 'echo aborted >\"$stressfail\"' TERM INT HUP\n> +\ttrap 'echo aborted >\"$stressfail\"; kill $job_pids 2>/dev/null; wait' TERM INT HUP\n> \n> +\tjob_pids=\n> \tjob_nr=0\n> \twhile test $job_nr -lt \"$job_count\"\n> \tdo\n> @@ -108,10 +109,13 @@ done,*)\n> \t\t\tGIT_TEST_STRESS_JOB_NR=$job_nr\n> \t\t\texport GIT_TEST_STRESS_STARTED GIT_TEST_STRESS_JOB_NR\n> \n> +\t\t\ttrap 'kill $test_pid 2>/dev/null' TERM\n> +\n> \t\t\tcnt=0\n> \t\t\twhile ! test -e \"$stressfail\"\n> \t\t\tdo\n> -\t\t\t\tif $TEST_SHELL_PATH \"$0\" \"$@\" >/dev/null 2>&1\n> +\t\t\t\t$TEST_SHELL_PATH \"$0\" \"$@\" >/dev/null 2>&1 & test_pid=$!\n> +\t\t\t\tif wait\n> \t\t\t\tthen\n> \t\t\t\t\tprintf >&2 \"OK %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n> \t\t\t\telif test -f \"$stressfail\" &&\n> @@ -124,16 +128,11 @@ done,*)\n> \t\t\t\tfi\n> \t\t\t\tcnt=$(($cnt + 1))\n> \t\t\tdone\n> -\t\t) &\n> +\t\t) & job_pids=\"$job_pids $!\"\n> \t\tjob_nr=$(($job_nr + 1))\n> \tdone\n> \n> -\tjob_nr=0\n> -\twhile test $job_nr -lt \"$job_count\"\n> -\tdo\n> -\t\twait\n> -\t\tjob_nr=$(($job_nr + 1))\n> -\tdone\n> +\twait\n> \n> \tif test -f \"$stressfail\" && test \"$(cat \"$stressfail\")\" != \"aborted\"\n> \tthen\n> \n\n"},{"id":"364729","messageId":"20181207010310.GA21992@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181206225601.GO30222@szeder.dev","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-07T01:03:11Z","receivedAt":"2018-12-07T01:03:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 06, 2018 at 11:56:01PM +0100, SZEDER Gábor wrote:\n\n> > +test_expect_success 'roll those dice' '\n> > +\tcase \"$(openssl rand -base64 1)\" in\n> > +\tz*)\n> > +\t\treturn 1\n> > +\tesac\n> > +'\n> \n> Wasteful :)\n> \n> test $(($$ % 42)) -ne 0\n\nOh, indeed, that is much more clever. :)\n\n-Peff\n"},{"id":"364730","messageId":"20181207011425.GB21992@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181206231005.GP30222@szeder.dev","subject":"Re: [RFC PATCH 3/3] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-07T01:14:25Z","receivedAt":"2018-12-07T01:14:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 07, 2018 at 12:10:05AM +0100, SZEDER Gábor wrote:\n\n> On Wed, Dec 05, 2018 at 04:56:21PM -0500, Jeff King wrote:\n> > Could we just kill them all?\n> > \n> > I guess it's a little tricky, because $! is going to give us the pid of\n> > each subshell. We actually want to kill the test sub-process. That takes\n> > a few contortions, but the output is nice (every sub-job immediately\n> > says \"ABORTED ...\", and the final process does not exit until the whole\n> > tree is done):\n> \n> It's trickier than that:\n> \n> - Nowadays our test lib translates SIGINT to exit, but not TERM (or\n> HUP, in case a user decides to close the terminal window), which\n> means that if the test runs any daemons in the background, then\n> those won't be killed when the stress test is aborted.\n> \n> This is easy enough to address, and I've been meaning to do this\n> in an other patch series anyway.\n\nYeah, trapping on more signals seems reasonable (or just propagating INT\ninstead of TERM). I'm more worried about this one:\n\n> - With the (implied) '--verbose-log' the process killed in the\n> background subshell's SIGTERM handler is not the actual test\n> process, because '--verbose-log' runs the test again in a piped\n> subshell to save its output:\n\nHmm, yeah. The patch I sent earlier already propagates through the\nsubshell, so this is really just another layer there. I.e., would it be\nreasonable when we are using --verbose-log (or --tee) for the parent\nprocess to propagate signals down to child it spawned?\n\nThat would fix this case, and it would make things more predictable in\ngeneral (e.g., a user who manually runs kill).\n\nIt does feel like a lot of boilerplate to be propagating these signals\nmanually. I think the \"right\" Unix-y solution here is process groups:\neach sub-job should get its own group, and then the top-level runner\nscript can kill the whole group. We may run into portability issues,\nthough (setsid is in util-linux, but I don't know if there's an\nequivalent elsewhere; a small perl or C helper could do it, but I have\nno idea what that would mean on Windows).\n\n> (GIT_TEST_TEE_STARTED=done ${TEST_SHELL_PATH} \"$0\" \"$@\" 2>&1;\n> echo $? >\"$TEST_RESULTS_BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n> \n> That 'kill' kills the \"outer\" shell process.\n> And though I get \"ABORTED...\" immediately and I get back my\n> prompt right away, the commands involved in the above construct\n> are still running in the background:\n> \n> $ ./t3903-stash.sh --stress=1\n> ^CABORTED 0.0\n> $ ps a |grep t3903\n> 1280 pts/17 S 0:00 /bin/sh ./t3903-stash.sh --stress=1\n> 1281 pts/17 S 0:00 tee -a <...>/test-results/t3903-stash.stress-0.out\n> 1282 pts/17 S 0:00 /bin/sh ./t3903-stash.sh --stress=1\n> 4173 pts/17 S+ 0:00 grep t3903\n> \n> I see this behavior with all shells I tried.\n> I haven't yet started to think it through why this happens :)\n\nI think you get ABORTED because we are just watching for the\nparent-process to exit (and reporting its status). The fact that that\nthe subshell (and the tee) are still running is not important. That all\nmakes sense to me.\n\n> Not implying '--verbose-log' but redirecting the test script's\n> output, like you did in your 'stress' script, seems to work in\n> dash, ksh, and ks93, but not in Bash v4.3 or later, where, for\n> whatever reason, the subshell get SIGINT before the SIGTERM would\n> arrive.\n> While we could easily handle SIGINT in the subshell with the same\n> signal handler as SIGTERM, it bothers me that something\n> fundamental is wrong here.\n\nYeah, I don't understand the SIGINT behavior you're seeing with bash. I\ncan't replicate it with bash 4.4.23 (from Debian unstable).\n\n-Peff\n"},{"id":"364881","messageId":"20181209225628.22216-1-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181204163457.15717-1-szeder.dev@gmail.com","subject":"[PATCH v2 0/7] test-lib: add the '--stress' option to help reproduce occasional failures in flaky tests","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-09T22:56:21Z","receivedAt":"2018-12-09T22:56:40Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\nThis patch series tries to make reproducing rare failures in flaky\ntests easier: it adds the '--stress' option to our test library to run\nthe test script repeatedly in multiple parallel jobs, in the hope that\nthe increased load creates enough variance in the timing of the test's\ncommands that such a failure is eventually triggered.\n\n\nNotable changes since v1:\n\n - Made it more Peff-friendly :), namely it will now show the log of\n the failed test and will rename its trash directory.\n\n Furthermore, '--stress' will now imply '--verbose -x --immediate'.\n\n - Improved abort handling based on the discussion of the previous\n version. (As a result, the patch is so heavily modified, that\n 'range-diff' with default parameters consideres it a different\n patch; increasing the creation factor then results in one big ugly\n diff of a diff, so I left it as-is.)\n\n - Added a few new patches, mostly preparatory refactorings, though\n the first one might be considered a bugfix.\n\n - Other minor cleanups suggested in the previous discussion.\n\n\nv1: https://public-inbox.org/git/20181204163457.15717-1-szeder.dev@gmail.com/T/\n\n\nSZEDER Gábor (7):\n test-lib: translate SIGTERM and SIGHUP to an exit\n test-lib: parse some --options earlier\n test-lib: consolidate naming of test-results paths\n test-lib: set $TRASH_DIRECTORY earlier\n test-lib: extract Bash version check for '-x' tracing\n test-lib-functions: introduce the 'test_set_port' helper function\n test-lib: add the '--stress' option to run a test repeatedly under\n load\n\n t/README | 19 +++-\n t/lib-git-daemon.sh | 2 +-\n t/lib-git-p4.sh | 9 +-\n t/lib-git-svn.sh | 2 +-\n t/lib-httpd.sh | 2 +-\n t/t0410-partial-clone.sh | 1 -\n t/t5512-ls-remote.sh | 2 +-\n t/test-lib-functions.sh | 39 +++++++\n t/test-lib.sh | 227 +++++++++++++++++++++++++++++----------\n 9 files changed, 230 insertions(+), 73 deletions(-)\n\nRange-diff:\n-: ---------- > 1: 3a5c926167 test-lib: translate SIGTERM and SIGHUP to an exit\n-: ---------- > 2: 8eee8d7fba test-lib: parse some --options earlier\n1: f4bb53e676 ! 3: dd20ae5e9a test-lib: consolidate naming of test-results paths\n @@ -4,8 +4,9 @@\n \n There are two places where we strip off any leading path components\n and the '.sh' suffix from the test script's pathname, and there are\n - two places where we construct the filename of test output files in\n - 't/test-results/'. The last patch in this series will add even more.\n + four places where we construct the name of the 't/test-results'\n + directory or the name of various test-specific files in there. The\n + last patch in this series will add even more.\n \n Factor these out into helper variables to avoid repeating ourselves.\n \n @@ -15,22 +16,23 @@\n --- a/t/test-lib.sh\n +++ b/t/test-lib.sh\n @@\n - \texit 1\n - fi\n + \tesac\n + done\n \n +TEST_NAME=\"$(basename \"$0\" .sh)\"\n -+TEST_RESULTS_BASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$TEST_NAME\"\n ++TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n ++TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n +\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n - case \"$GIT_TEST_TEE_STARTED, $* \" in\n + if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n @@\n - \t# do not redirect again\n - \t;;\n - *' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*)\n + elif test -n \"$tee\" || test -n \"$verbose_log\" ||\n + test -n \"$valgrind\" || test -n \"$valgrind_only\"\n + then\n -\tmkdir -p \"$TEST_OUTPUT_DIRECTORY/test-results\"\n -\tBASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$(basename \"$0\" .sh)\"\n -+\tmkdir -p \"$(dirname \"$TEST_RESULTS_BASE\")\"\n ++\tmkdir -p \"$TEST_RESULTS_DIR\"\n \n \t# Make this filename available to the sub-process in case it is using\n \t# --verbose-log.\n @@ -48,8 +50,8 @@\n +\t echo $? >\"$TEST_RESULTS_BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n +\ttest \"$(cat \"$TEST_RESULTS_BASE.exit\")\" = 0\n \texit\n - \t;;\n - esac\n + fi\n + \n @@\n \n \tif test -z \"$HARNESS_ACTIVE\"\n @@ -58,7 +60,7 @@\n -\t\tmkdir -p \"$test_results_dir\"\n -\t\tbase=${0##*/}\n -\t\ttest_results_path=\"$test_results_dir/${base%.sh}.counts\"\n -+\t\tmkdir -p \"$(dirname \"$TEST_RESULTS_BASE\")\"\n ++\t\tmkdir -p \"$TEST_RESULTS_DIR\"\n \n -\t\tcat >\"$test_results_path\" <<-EOF\n +\t\tcat >\"$TEST_RESULTS_BASE.counts\" <<-EOF\n-: ---------- > 4: f3941077e6 test-lib: set $TRASH_DIRECTORY earlier\n-: ---------- > 5: fefbca96ee test-lib: extract Bash version check for '-x' tracing\n2: 9aec8662f9 ! 6: b4b6844a3e test-lib-functions: introduce the 'test_set_port' helper function\n @@ -27,7 +27,7 @@\n containing the digits 8 or 9 will trigger an error. Remove all\n leading zeros from the test numbers to prevent this.\n \n - Note that the Perforce tests are unlike the other tests involving\n + Note that the 'git p4' tests are unlike the other tests involving\n daemons in that:\n \n - 'lib-git-p4.sh' doesn't use the test's number for unique port as\n @@ -35,7 +35,7 @@\n \n - The port is not overridable via an environment variable.\n \n - With this patch even Perforce tests will use the test's number as\n + With this patch even 'git p4' tests will use the test's number as\n default port, and it will be overridable via the P4DPORT environment\n variable.\n \n @@ -161,7 +161,7 @@\n +\t\tBUG \"test_set_port requires a variable name\"\n +\tfi\n +\n -+\teval port=\\\"\\${$var}\\\"\n ++\teval port=\\$$var\n +\tcase \"$port\" in\n +\t\"\")\n +\t\t# No port is set in the given env var, use the test\n3: a5aa71f20c < -: ---------- test-lib: add the '--stress' option to run a test repeatedly under load\n-: ---------- > 7: 8470b55f65 test-lib: add the '--stress' option to run a test repeatedly under load\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364882","messageId":"20181209225628.22216-3-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181209225628.22216-1-szeder.dev@gmail.com","subject":"[PATCH v2 2/7] test-lib: parse some --options earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-09T22:56:23Z","receivedAt":"2018-12-09T22:56:42Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"'test-lib.sh' looks for the presence of certain options like '--tee'\nand '--verbose-log', so it can execute the test script again to save\nits standard output and error. This happens way before the actual\noption parsing loop, and the condition looking for these options looks\na bit odd, too. This patch series will add two more options to look\nout for, and, in addition, will have to extract these options' stuck\narguments (i.e. '--opt=arg') as well.\n\nAdd a proper option parsing loop to check these select options early\nin 'test-lib.sh', making this early option checking more readable and\nkeeping those later changes in this series simpler. Use a 'for opt in\n\"$@\"' loop to iterate over the options to preserve \"$@\" intact, so\noptions like '--verbose-log' can execute the test script again with\nall the original options.\n\nAs an alternative, we could parse all options early, but there are\noptions that do require an _unstuck_ argument, which is tricky to\nhandle properly in such a for loop, and the resulting complexity is,\nin my opinion, worse than having this extra, partial option parsing\nloop.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 53 +++++++++++++++++++++++++++++++--------------------\n 1 file changed, 32 insertions(+), 21 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9a3f7930a3..5577d9dc5a 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -71,13 +71,33 @@ then\n \texit 1\n fi\n \n+# Parse some options early, taking care to leave $@ intact.\n+for opt\n+do\n+\tcase \"$opt\" in\n+\t--tee)\n+\t\ttee=t ;;\n+\t-V|--verbose-log)\n+\t\tverbose_log=t ;;\n+\t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n+\t\tvalgrind=memcheck ;;\n+\t--valgrind=*)\n+\t\tvalgrind=${opt#--*=} ;;\n+\t--valgrind-only=*)\n+\t\tvalgrind_only=${opt#--*=} ;;\n+\t*)\n+\t\t# Other options will be handled later.\n+\tesac\n+done\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n-case \"$GIT_TEST_TEE_STARTED, $* \" in\n-done,*)\n-\t# do not redirect again\n-\t;;\n-*' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*)\n+if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n+then\n+\t: # do not redirect again\n+elif test -n \"$tee\" || test -n \"$verbose_log\" ||\n+ test -n \"$valgrind\" || test -n \"$valgrind_only\"\n+then\n \tmkdir -p \"$TEST_OUTPUT_DIRECTORY/test-results\"\n \tBASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$(basename \"$0\" .sh)\"\n \n@@ -94,8 +114,7 @@ done,*)\n \t echo $? >\"$BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n \ttest \"$(cat \"$BASE.exit\")\" = 0\n \texit\n-\t;;\n-esac\n+fi\n \n # For repeatability, reset the environment to known value.\n # TERM is sanitized below, after saving color control sequences.\n@@ -296,17 +315,6 @@ do\n \t\twith_dashes=t; shift ;;\n \t--no-color)\n \t\tcolor=; shift ;;\n-\t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n-\t\tvalgrind=memcheck\n-\t\tshift ;;\n-\t--valgrind=*)\n-\t\tvalgrind=${1#--*=}\n-\t\tshift ;;\n-\t--valgrind-only=*)\n-\t\tvalgrind_only=${1#--*=}\n-\t\tshift ;;\n-\t--tee)\n-\t\tshift ;; # was handled already\n \t--root=*)\n \t\troot=${1#--*=}\n \t\tshift ;;\n@@ -336,9 +344,12 @@ do\n \t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n \t\tfi\n \t\tshift ;;\n-\t-V|--verbose-log)\n-\t\tverbose_log=t\n-\t\tshift ;;\n+\t--tee|\\\n+\t-V|--verbose-log|\\\n+\t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind|\\\n+\t--valgrind=*|\\\n+\t--valgrind-only=*)\n+\t\tshift ;; # These options were handled already.\n \t*)\n \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n \tesac\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364883","messageId":"20181209225628.22216-2-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181209225628.22216-1-szeder.dev@gmail.com","subject":"[PATCH v2 1/7] test-lib: translate SIGTERM and SIGHUP to an exit","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-09T22:56:22Z","receivedAt":"2018-12-09T22:56:43Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Right now if a test script receives SIGTERM or SIGHUP (e.g., because a\ntest was hanging and the user 'kill'-ed it or simply closed the\nterminal window the test was running in), the shell exits immediately.\nThis can be annoying if the test script did any global setup, like\nstarting apache or git-daemon, as it will not have an opportunity to\nclean up after itself. A subsequent run of the test won't be able to\nstart its own daemon, and will either fail or skip the tests.\n\nInstead, let's trap SIGTERM and SIGHUP as well to make sure we do a\nclean shutdown, and just chain it to a normal exit (which will trigger\nany cleanup).\n\nThis patch follows suit of da706545f7 (t: translate SIGINT to an exit,\n2015-03-13), and even stole its commit message as well.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 0f1faa24b2..9a3f7930a3 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -476,7 +476,7 @@ die () {\n \n GIT_EXIT_OK=\n trap 'die' EXIT\n-trap 'exit $?' INT\n+trap 'exit $?' INT TERM HUP\n \n # The user-facing functions are loaded from a separate file so that\n # test_perf subshells can have them too\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364884","messageId":"20181209225628.22216-5-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181209225628.22216-1-szeder.dev@gmail.com","subject":"[PATCH v2 4/7] test-lib: set $TRASH_DIRECTORY earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-09T22:56:25Z","receivedAt":"2018-12-09T22:56:45Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"A later patch in this series will need to know the path to the trash\ndirectory early in 'test-lib.sh', but $TRASH_DIRECTORY is set much\nlater. Furthermore, the path to the trash directory depends on the\n'--root=' option, which, too, is parsed too late.\n\nMove parsing '--root=...' to the early option parsing loop, and set\n$TRASH_DIRECTORY where the other test-specific path variables are set.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 09c77cbd1b..ea1cd34013 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -85,6 +85,8 @@ do\n \t\tvalgrind=${opt#--*=} ;;\n \t--valgrind-only=*)\n \t\tvalgrind_only=${opt#--*=} ;;\n+\t--root=*)\n+\t\troot=${opt#--*=} ;;\n \t*)\n \t\t# Other options will be handled later.\n \tesac\n@@ -93,6 +95,12 @@ done\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n+test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n+case \"$TRASH_DIRECTORY\" in\n+/*) ;; # absolute path is good\n+ *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n+esac\n \n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n@@ -318,9 +326,6 @@ do\n \t\twith_dashes=t; shift ;;\n \t--no-color)\n \t\tcolor=; shift ;;\n-\t--root=*)\n-\t\troot=${1#--*=}\n-\t\tshift ;;\n \t--chain-lint)\n \t\tGIT_TEST_CHAIN_LINT=1\n \t\tshift ;;\n@@ -351,7 +356,8 @@ do\n \t-V|--verbose-log|\\\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind|\\\n \t--valgrind=*|\\\n-\t--valgrind-only=*)\n+\t--valgrind-only=*|\\\n+\t--root=*)\n \t\tshift ;; # These options were handled already.\n \t*)\n \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n@@ -1040,12 +1046,6 @@ then\n fi\n \n # Test repository\n-TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n-test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n-case \"$TRASH_DIRECTORY\" in\n-/*) ;; # absolute path is good\n- *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n-esac\n rm -fr \"$TRASH_DIRECTORY\" || {\n \tGIT_EXIT_OK=t\n \techo >&5 \"FATAL: Cannot prepare test area\"\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364885","messageId":"20181209225628.22216-6-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181209225628.22216-1-szeder.dev@gmail.com","subject":"[PATCH v2 5/7] test-lib: extract Bash version check for '-x' tracing","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-09T22:56:26Z","receivedAt":"2018-12-09T22:56:46Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Some of our test scripts can't be reliably run with '-x' tracing\nenabled unless executed with a Bash version supporting BASH_XTRACEFD\n(since v4.1), and we have a lengthy condition to disable tracing if\nsuch a test script is not executed with a suitable Bash version.\n\nMove this check out from the option parsing loop, so other options can\nimply '-x' by setting 'trace=t', without missing this Bash version\ncheck.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 37 +++++++++++++++++++------------------\n 1 file changed, 19 insertions(+), 18 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ea1cd34013..d55d158580 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -333,24 +333,7 @@ do\n \t\tGIT_TEST_CHAIN_LINT=0\n \t\tshift ;;\n \t-x)\n-\t\t# Some test scripts can't be reliably traced with '-x',\n-\t\t# unless the test is run with a Bash version supporting\n-\t\t# BASH_XTRACEFD (introduced in Bash v4.1). Check whether\n-\t\t# this test is marked as such, and ignore '-x' if it\n-\t\t# isn't executed with a suitable Bash version.\n-\t\tif test -z \"$test_untraceable\" || {\n-\t\t test -n \"$BASH_VERSION\" && {\n-\t\t test ${BASH_VERSINFO[0]} -gt 4 || {\n-\t\t\t test ${BASH_VERSINFO[0]} -eq 4 &&\n-\t\t\t test ${BASH_VERSINFO[1]} -ge 1\n-\t\t }\n-\t\t }\n-\t\t }\n-\t\tthen\n-\t\t\ttrace=t\n-\t\telse\n-\t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n-\t\tfi\n+\t\ttrace=t\n \t\tshift ;;\n \t--tee|\\\n \t-V|--verbose-log|\\\n@@ -373,6 +356,24 @@ then\n \ttest -z \"$verbose_log\" && verbose=t\n fi\n \n+if test -n \"$trace\" && test -n \"$test_untraceable\"\n+then\n+\t# '-x' tracing requested, but this test script can't be reliably\n+\t# traced, unless it is run with a Bash version supporting\n+\t# BASH_XTRACEFD (introduced in Bash v4.1).\n+\tif test -n \"$BASH_VERSION\" && {\n+\t test ${BASH_VERSINFO[0]} -gt 4 || {\n+\t test ${BASH_VERSINFO[0]} -eq 4 &&\n+\t test ${BASH_VERSINFO[1]} -ge 1\n+\t }\n+\t }\n+\tthen\n+\t\t: Executed by a Bash version supporting BASH_XTRACEFD. Good.\n+\telse\n+\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n+\t\ttrace=\n+\tfi\n+fi\n if test -n \"$trace\" && test -z \"$verbose_log\"\n then\n \tverbose=t\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364886","messageId":"20181209225628.22216-4-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181209225628.22216-1-szeder.dev@gmail.com","subject":"[PATCH v2 3/7] test-lib: consolidate naming of test-results paths","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-09T22:56:24Z","receivedAt":"2018-12-09T22:56:48Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"There are two places where we strip off any leading path components\nand the '.sh' suffix from the test script's pathname, and there are\nfour places where we construct the name of the 't/test-results'\ndirectory or the name of various test-specific files in there. The\nlast patch in this series will add even more.\n\nFactor these out into helper variables to avoid repeating ourselves.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 5577d9dc5a..09c77cbd1b 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -90,6 +90,10 @@ do\n \tesac\n done\n \n+TEST_NAME=\"$(basename \"$0\" .sh)\"\n+TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n+TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n@@ -98,12 +102,11 @@ then\n elif test -n \"$tee\" || test -n \"$verbose_log\" ||\n test -n \"$valgrind\" || test -n \"$valgrind_only\"\n then\n-\tmkdir -p \"$TEST_OUTPUT_DIRECTORY/test-results\"\n-\tBASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$(basename \"$0\" .sh)\"\n+\tmkdir -p \"$TEST_RESULTS_DIR\"\n \n \t# Make this filename available to the sub-process in case it is using\n \t# --verbose-log.\n-\tGIT_TEST_TEE_OUTPUT_FILE=$BASE.out\n+\tGIT_TEST_TEE_OUTPUT_FILE=$TEST_RESULTS_BASE.out\n \texport GIT_TEST_TEE_OUTPUT_FILE\n \n \t# Truncate before calling \"tee -a\" to get rid of the results\n@@ -111,8 +114,8 @@ then\n \t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \n \t(GIT_TEST_TEE_STARTED=done ${TEST_SHELL_PATH} \"$0\" \"$@\" 2>&1;\n-\t echo $? >\"$BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n-\ttest \"$(cat \"$BASE.exit\")\" = 0\n+\t echo $? >\"$TEST_RESULTS_BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\ttest \"$(cat \"$TEST_RESULTS_BASE.exit\")\" = 0\n \texit\n fi\n \n@@ -829,12 +832,9 @@ test_done () {\n \n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n-\t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n-\t\tmkdir -p \"$test_results_dir\"\n-\t\tbase=${0##*/}\n-\t\ttest_results_path=\"$test_results_dir/${base%.sh}.counts\"\n+\t\tmkdir -p \"$TEST_RESULTS_DIR\"\n \n-\t\tcat >\"$test_results_path\" <<-EOF\n+\t\tcat >\"$TEST_RESULTS_BASE.counts\" <<-EOF\n \t\ttotal $test_count\n \t\tsuccess $test_success\n \t\tfixed $test_fixed\n@@ -1040,7 +1040,7 @@ then\n fi\n \n # Test repository\n-TRASH_DIRECTORY=\"trash directory.$(basename \"$0\" .sh)\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n case \"$TRASH_DIRECTORY\" in\n /*) ;; # absolute path is good\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364888","messageId":"20181209225628.22216-7-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181209225628.22216-1-szeder.dev@gmail.com","subject":"[PATCH v2 6/7] test-lib-functions: introduce the 'test_set_port' helper function","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-09T22:56:27Z","receivedAt":"2018-12-09T22:56:49Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Several test scripts run daemons like 'git-daemon' or Apache, and\ncommunicate with them through TCP sockets. To have unique ports where\nthese daemons are accessible, the ports are usually the number of the\ncorresponding test scripts, unless the user overrides them via\nenvironment variables, and thus all those tests and test libs contain\nmore or less the same bit of one-liner boilerplate code to find out\nthe port. The last patch in this series will make this a bit more\ncomplicated.\n\nFactor out finding the port for a daemon into the common helper\nfunction 'test_set_port' to avoid repeating ourselves.\n\nTake special care of test scripts with \"low\" numbers:\n\n - Test numbers below 1024 would result in a port that's only usable\n as root, so set their port to '10000 + test-nr' to make sure it\n doesn't interfere with other tests in the test suite. This makes\n the hardcoded port number in 't0410-partial-clone.sh' unnecessary,\n remove it.\n\n - The shell's arithmetic evaluation interprets numbers with leading\n zeros as octal values, which means that test number below 1000 and\n containing the digits 8 or 9 will trigger an error. Remove all\n leading zeros from the test numbers to prevent this.\n\nNote that the 'git p4' tests are unlike the other tests involving\ndaemons in that:\n\n - 'lib-git-p4.sh' doesn't use the test's number for unique port as\n is, but does a bit of additional arithmetic on top [1].\n\n - The port is not overridable via an environment variable.\n\nWith this patch even 'git p4' tests will use the test's number as\ndefault port, and it will be overridable via the P4DPORT environment\nvariable.\n\n[1] Commit fc00233071 (git-p4 tests: refactor and cleanup, 2011-08-22)\n introduced that \"unusual\" unique port computation without\n explaining why it was necessary (as opposed to simply using the\n test number as is). It seems to be just unnecessary complication,\n and in any case that commit came way before the \"test nr as unique\n port\" got \"standardized\" for other daemons in commits c44132fcf3\n (tests: auto-set git-daemon port, 2014-02-10), 3bb486e439 (tests:\n auto-set LIB_HTTPD_PORT from test name, 2014-02-10), and\n bf9d7df950 (t/lib-git-svn.sh: improve svnserve tests with parallel\n make test, 2017-12-01).\n\nSigned-off-by: SZEDER Gábor \n---\n t/lib-git-daemon.sh | 2 +-\n t/lib-git-p4.sh | 9 +--------\n t/lib-git-svn.sh | 2 +-\n t/lib-httpd.sh | 2 +-\n t/t0410-partial-clone.sh | 1 -\n t/t5512-ls-remote.sh | 2 +-\n t/test-lib-functions.sh | 36 ++++++++++++++++++++++++++++++++++++\n 7 files changed, 41 insertions(+), 13 deletions(-)\n\ndiff --git a/t/lib-git-daemon.sh b/t/lib-git-daemon.sh\nindex f98de95c15..41eb1e3ae8 100644\n--- a/t/lib-git-daemon.sh\n+++ b/t/lib-git-daemon.sh\n@@ -28,7 +28,7 @@ then\n \ttest_skip_or_die $GIT_TEST_GIT_DAEMON \"file system does not support FIFOs\"\n fi\n \n-LIB_GIT_DAEMON_PORT=${LIB_GIT_DAEMON_PORT-${this_test#t}}\n+test_set_port LIB_GIT_DAEMON_PORT\n \n GIT_DAEMON_PID=\n GIT_DAEMON_DOCUMENT_ROOT_PATH=\"$PWD\"/repo\ndiff --git a/t/lib-git-p4.sh b/t/lib-git-p4.sh\nindex c27599474c..b3be3ba011 100644\n--- a/t/lib-git-p4.sh\n+++ b/t/lib-git-p4.sh\n@@ -53,14 +53,7 @@ time_in_seconds () {\n \t(cd / && \"$PYTHON_PATH\" -c 'import time; print(int(time.time()))')\n }\n \n-# Try to pick a unique port: guess a large number, then hope\n-# no more than one of each test is running.\n-#\n-# This does not handle the case where somebody else is running the\n-# same tests and has chosen the same ports.\n-testid=${this_test#t}\n-git_p4_test_start=9800\n-P4DPORT=$((10669 + ($testid - $git_p4_test_start)))\n+test_set_port P4DPORT\n \n P4PORT=localhost:$P4DPORT\n P4CLIENT=client\ndiff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh\nindex a8130f9119..f3b478c307 100644\n--- a/t/lib-git-svn.sh\n+++ b/t/lib-git-svn.sh\n@@ -13,6 +13,7 @@ fi\n GIT_DIR=$PWD/.git\n GIT_SVN_DIR=$GIT_DIR/svn/refs/remotes/git-svn\n SVN_TREE=$GIT_SVN_DIR/svn-tree\n+test_set_port SVNSERVE_PORT\n \n svn >/dev/null 2>&1\n if test $? -ne 1\n@@ -119,7 +120,6 @@ require_svnserve () {\n }\n \n start_svnserve () {\n-\tSVNSERVE_PORT=${SVNSERVE_PORT-${this_test#t}}\n \tsvnserve --listen-port $SVNSERVE_PORT \\\n \t\t --root \"$rawsvnrepo\" \\\n \t\t --listen-once \\\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex a8729f8232..e465116ef9 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -82,7 +82,7 @@ case $(uname) in\n esac\n \n LIB_HTTPD_PATH=${LIB_HTTPD_PATH-\"$DEFAULT_HTTPD_PATH\"}\n-LIB_HTTPD_PORT=${LIB_HTTPD_PORT-${this_test#t}}\n+test_set_port LIB_HTTPD_PORT\n \n TEST_PATH=\"$TEST_DIRECTORY\"/lib-httpd\n HTTPD_ROOT_PATH=\"$PWD\"/httpd\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex ba3887f178..0aca8d7588 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -480,7 +480,6 @@ test_expect_success 'gc stops traversal when a missing but promised object is re\n \t! grep \"$TREE_HASH\" out\n '\n \n-LIB_HTTPD_PORT=12345 # default port, 410, cannot be used as non-root\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \ndiff --git a/t/t5512-ls-remote.sh b/t/t5512-ls-remote.sh\nindex 32e722db2e..cd9e60632d 100755\n--- a/t/t5512-ls-remote.sh\n+++ b/t/t5512-ls-remote.sh\n@@ -260,7 +260,7 @@ test_lazy_prereq GIT_DAEMON '\n # This test spawns a daemon, so run it only if the user would be OK with\n # testing with git-daemon.\n test_expect_success PIPE,JGIT,GIT_DAEMON 'indicate no refs in standards-compliant empty remote' '\n-\tJGIT_DAEMON_PORT=${JGIT_DAEMON_PORT-${this_test#t}} &&\n+\ttest_set_port JGIT_DAEMON_PORT &&\n \tJGIT_DAEMON_PID= &&\n \tgit init --bare empty.git &&\n \t>empty.git/git-daemon-export-ok &&\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6b3bbf99e4..3746327bde 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1263,3 +1263,39 @@ test_oid () {\n \tfi &&\n \teval \"printf '%s' \\\"\\${$var}\\\"\"\n }\n+\n+# Choose a port number based on the test script's number and store it in\n+# the given variable name, unless that variable already contains a number.\n+test_set_port () {\n+\tlocal var=$1 port\n+\n+\tif test $# -ne 1 || test -z \"$var\"\n+\tthen\n+\t\tBUG \"test_set_port requires a variable name\"\n+\tfi\n+\n+\teval port=\\$$var\n+\tcase \"$port\" in\n+\t\"\")\n+\t\t# No port is set in the given env var, use the test\n+\t\t# number as port number instead.\n+\t\t# Remove not only the leading 't', but all leading zeros\n+\t\t# as well, so the arithmetic below won't (mis)interpret\n+\t\t# a test number like '0123' as an octal value.\n+\t\tport=${this_test#${this_test%%[1-9]*}}\n+\t\tif test \"${port:-0}\" -lt 1024\n+\t\tthen\n+\t\t\t# root-only port, use a larger one instead.\n+\t\t\tport=$(($port + 10000))\n+\t\tfi\n+\n+\t\teval $var=$port\n+\t\t;;\n+\t*[^0-9]*)\n+\t\terror >&7 \"invalid port number: $port\"\n+\t\t;;\n+\t*)\n+\t\t# The user has specified the port.\n+\t\t;;\n+\tesac\n+}\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364887","messageId":"20181209225628.22216-8-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181209225628.22216-1-szeder.dev@gmail.com","subject":"[PATCH v2 7/7] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-09T22:56:28Z","receivedAt":"2018-12-09T22:56:50Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Unfortunately, we have a few flaky tests, whose failures tend to be\nhard to reproduce. We've found that the best we can do to reproduce\nsuch a failure is to run the test repeatedly while the machine is\nunder load, and wait in the hope that the load creates enough variance\nin the timing of the test's commands that a failure is evenually\ntriggered. I have a command to do that, and I noticed that two other\ncontributors have rolled their own scripts to do the same, all\nchoosing slightly different approaches.\n\nTo help reproduce failures in flaky tests, introduce the '--stress'\noption to run a test script repeatedly in multiple parallel jobs until\none of them fails, thereby using the test script itself to increase\nthe load on the machine.\n\nThe number of parallel jobs is determined by, in order of precedence:\nthe number specified as '--stress=', or the value of the\nGIT_TEST_STRESS_LOAD environment variable, or twice the number of\navailable processors (as reported by the 'getconf' utility), or 8.\n\nMake '--stress' imply '--verbose -x --immediate' to get the most\ninformation about rare failures; there is really no point in spending\nall the extra effort to reproduce such a failure, and then not know\nwhich command failed and why.\n\nTo prevent the several parallel invocations of the same test from\ninterfering with each other:\n\n - Include the parallel job's number in the name of the trash\n directory and the various output files under 't/test-results/' as\n a '.stress-' suffix.\n\n - Add the parallel job's number to the port number specified by the\n user or to the test number, so even tests involving daemons\n listening on a TCP socket can be stressed.\n\n - Redirect each parallel test run's output to\n 't/test-results/$TEST_NAME.stress-.out', because dumping the\n output of several parallel running tests to the terminal would\n create a big ugly mess.\n\nFor convenience, print the output of the failed test job at the end,\nand rename its trash directory to end with the '.stress-failed'\nsuffix, so it's easy to find in a predictable path. If, in an\nunlikely case, more than one jobs were to fail nearly at the same\ntime, then print the output of all failed jobs, and rename the trash\ndirectory of only the last one (i.e. with the highest job number), as\nit is the trash directory of the test whose output will be at the\nbottom of the user's terminal.\n\nBased on Jeff King's 'stress' script.\n\nSigned-off-by: SZEDER Gábor \n---\n t/README | 19 +++++++-\n t/test-lib-functions.sh | 7 ++-\n t/test-lib.sh | 103 ++++++++++++++++++++++++++++++++++++++--\n 3 files changed, 123 insertions(+), 6 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 28711cc508..11ce7675e3 100644\n--- a/t/README\n+++ b/t/README\n@@ -186,6 +186,22 @@ appropriately before running \"make\".\n \tthis feature by setting the GIT_TEST_CHAIN_LINT environment\n \tvariable to \"1\" or \"0\", respectively.\n \n+--stress::\n+--stress=::\n+\tRun the test script repeatedly in multiple parallel jobs until\n+\tone of them fails. Useful for reproducing rare failures in\n+\tflaky tests. The number of parallel jobs is, in order of\n+\tprecedence: , or the value of the GIT_TEST_STRESS_LOAD\n+\tenvironment variable, or twice the number of available\n+\tprocessors (as shown by the 'getconf' utility),\tor 8.\n+\tImplies `--verbose -x --immediate` to get the most information\n+\tabout the failure. Note that the verbose output of each test\n+\tjob is saved to 't/test-results/$TEST_NAME.stress-.out',\n+\tand only the output of the failed test job is shown on the\n+\tterminal. The names of the trash directories get a\n+\t'.stress-' suffix, and the trash directory of the failed\n+\ttest job is renamed to end with a '.stress-failed' suffix.\n+\n You can also set the GIT_TEST_INSTALLED environment variable to\n the bindir of an existing git installation to test that installation.\n You still need to have built this git sandbox, from which various\n@@ -425,7 +441,8 @@ This test harness library does the following things:\n - Creates an empty test directory with an empty .git/objects database\n and chdir(2) into it. This directory is 't/trash\n directory.$test_name_without_dotsh', with t/ subject to change by\n- the --root option documented above.\n+ the --root option documented above, and a '.stress-' suffix\n+ appended by the --stress option.\n \n - Defines standard test helper functions for your scripts to\n use. These functions are designed to make all scripts behave\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 3746327bde..ee3435c6df 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1288,8 +1288,6 @@ test_set_port () {\n \t\t\t# root-only port, use a larger one instead.\n \t\t\tport=$(($port + 10000))\n \t\tfi\n-\n-\t\teval $var=$port\n \t\t;;\n \t*[^0-9]*)\n \t\terror >&7 \"invalid port number: $port\"\n@@ -1298,4 +1296,9 @@ test_set_port () {\n \t\t# The user has specified the port.\n \t\t;;\n \tesac\n+\n+\t# Make sure that parallel '--stress' test jobs get different\n+\t# ports.\n+\tport=$(($port + ${GIT_TEST_STRESS_JOB_NR:-0}))\n+\teval $var=$port\n }\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex d55d158580..e405191164 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -87,21 +87,110 @@ do\n \t\tvalgrind_only=${opt#--*=} ;;\n \t--root=*)\n \t\troot=${opt#--*=} ;;\n+\t--stress)\n+\t\tstress=t ;;\n+\t--stress=*)\n+\t\tstress=${opt#--*=} ;;\n \t*)\n \t\t# Other options will be handled later.\n \tesac\n done\n \n+TEST_STRESS_JOB_SFX=\"${GIT_TEST_STRESS_JOB_NR:+.stress-$GIT_TEST_STRESS_JOB_NR}\"\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n-TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n-TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n+TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME$TEST_STRESS_JOB_SFX\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME$TEST_STRESS_JOB_SFX\"\n test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n case \"$TRASH_DIRECTORY\" in\n /*) ;; # absolute path is good\n *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n esac\n \n+# If --stress was passed, run this test repeatedly in several parallel loops.\n+if test \"$GIT_TEST_STRESS_STARTED\" = \"done\"\n+then\n+\t: # Don't stress test again.\n+elif test -n \"$stress\"\n+then\n+\tif test \"$stress\" != t\n+\tthen\n+\t\tjob_count=$stress\n+\telif test -n \"$GIT_TEST_STRESS_LOAD\"\n+\tthen\n+\t\tjob_count=\"$GIT_TEST_STRESS_LOAD\"\n+\telif job_count=$(getconf _NPROCESSORS_ONLN 2>/dev/null) &&\n+\t test -n \"$job_count\"\n+\tthen\n+\t\tjob_count=$((2 * $job_count))\n+\telse\n+\t\tjob_count=8\n+\tfi\n+\n+\tmkdir -p \"$TEST_RESULTS_DIR\"\n+\tstressfail=\"$TEST_RESULTS_BASE.stress-failed\"\n+\trm -f \"$stressfail\"\n+\n+\tstress_exit=0\n+\ttrap '\n+\t\tkill $job_pids 2>/dev/null\n+\t\twait\n+\t\tstress_exit=1\n+\t' TERM INT HUP\n+\n+\tjob_pids=\n+\tjob_nr=0\n+\twhile test $job_nr -lt \"$job_count\"\n+\tdo\n+\t\t(\n+\t\t\tGIT_TEST_STRESS_STARTED=done\n+\t\t\tGIT_TEST_STRESS_JOB_NR=$job_nr\n+\t\t\texport GIT_TEST_STRESS_STARTED GIT_TEST_STRESS_JOB_NR\n+\n+\t\t\ttrap '\n+\t\t\t\tkill $test_pid 2>/dev/null\n+\t\t\t\twait\n+\t\t\t\texit 1\n+\t\t\t' TERM INT\n+\n+\t\t\tcnt=0\n+\t\t\twhile ! test -e \"$stressfail\"\n+\t\t\tdo\n+\t\t\t\t$TEST_SHELL_PATH \"$0\" \"$@\" >\"$TEST_RESULTS_BASE.stress-$job_nr.out\" 2>&1 &\n+\t\t\t\ttest_pid=$!\n+\n+\t\t\t\tif wait $test_pid\n+\t\t\t\tthen\n+\t\t\t\t\tprintf \"OK %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n+\t\t\t\telse\n+\t\t\t\t\techo $GIT_TEST_STRESS_JOB_NR >>\"$stressfail\"\n+\t\t\t\t\tprintf \"FAIL %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n+\t\t\t\tfi\n+\t\t\t\tcnt=$(($cnt + 1))\n+\t\t\tdone\n+\t\t) &\n+\t\tjob_pids=\"$job_pids $!\"\n+\t\tjob_nr=$(($job_nr + 1))\n+\tdone\n+\n+\twait\n+\n+\tif test -f \"$stressfail\"\n+\tthen\n+\t\techo \"Log(s) of failed test run(s):\"\n+\t\tfor failed_job_nr in $(sort -n \"$stressfail\")\n+\t\tdo\n+\t\t\techo \"Contents of '$TEST_RESULTS_BASE.stress-$failed_job_nr.out':\"\n+\t\t\tcat \"$TEST_RESULTS_BASE.stress-$failed_job_nr.out\"\n+\t\tdone\n+\t\trm -rf \"$TRASH_DIRECTORY.stress-failed\"\n+\t\t# Move the last one.\n+\t\tmv \"$TRASH_DIRECTORY.stress-$failed_job_nr\" \"$TRASH_DIRECTORY.stress-failed\"\n+\tfi\n+\n+\texit $stress_exit\n+fi\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n@@ -340,7 +429,8 @@ do\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind|\\\n \t--valgrind=*|\\\n \t--valgrind-only=*|\\\n-\t--root=*)\n+\t--root=*|\\\n+\t--stress|--stress=*)\n \t\tshift ;; # These options were handled already.\n \t*)\n \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n@@ -379,6 +469,13 @@ then\n \tverbose=t\n fi\n \n+if test -n \"$stress\"\n+then\n+\tverbose=t\n+\ttrace=t\n+\timmediate=t\n+fi\n+\n if test -n \"$color\"\n then\n \t# Save the color control sequences now rather than run tput\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"364895","messageId":"20181210013408.25577-1-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181209225628.22216-8-szeder.dev@gmail.com","subject":"[PATCH] fixup! test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-10T01:34:08Z","receivedAt":"2018-12-10T01:34:26Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"---\n\nErm. 'trace=t' must be set before checking whether the shell\nsupports BASH_XTRACEFD.\n\n t/test-lib.sh | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e405191164..3e9916b39b 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -446,6 +446,13 @@ then\n \ttest -z \"$verbose_log\" && verbose=t\n fi\n \n+if test -n \"$stress\"\n+then\n+\tverbose=t\n+\ttrace=t\n+\timmediate=t\n+fi\n+\n if test -n \"$trace\" && test -n \"$test_untraceable\"\n then\n \t# '-x' tracing requested, but this test script can't be reliably\n@@ -469,13 +476,6 @@ then\n \tverbose=t\n fi\n \n-if test -n \"$stress\"\n-then\n-\tverbose=t\n-\ttrace=t\n-\timmediate=t\n-fi\n-\n if test -n \"$color\"\n then\n \t# Save the color control sequences now rather than run tput\n-- \n2.20.0.rc2.156.g5a9fd2ce9c\n\n"},{"id":"365063","messageId":"20181211105750.GB8452@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181209225628.22216-2-szeder.dev@gmail.com","subject":"Re: [PATCH v2 1/7] test-lib: translate SIGTERM and SIGHUP to an exit","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-11T10:57:50Z","receivedAt":"2018-12-11T10:58:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Dec 09, 2018 at 11:56:22PM +0100, SZEDER Gábor wrote:\n\n> Right now if a test script receives SIGTERM or SIGHUP (e.g., because a\n> test was hanging and the user 'kill'-ed it or simply closed the\n> terminal window the test was running in), the shell exits immediately.\n> This can be annoying if the test script did any global setup, like\n> starting apache or git-daemon, as it will not have an opportunity to\n> clean up after itself. A subsequent run of the test won't be able to\n> start its own daemon, and will either fail or skip the tests.\n> \n> Instead, let's trap SIGTERM and SIGHUP as well to make sure we do a\n> clean shutdown, and just chain it to a normal exit (which will trigger\n> any cleanup).\n> \n> This patch follows suit of da706545f7 (t: translate SIGINT to an exit,\n> 2015-03-13), and even stole its commit message as well.\n\nNo wonder it was so nicely explained. ;)\n\nI think this is quite a reasonable thing to do. Since we're trying to\nclean up, in theory we would like to hook any signal death, but these\nthree are the common ones in practice. We handle QUIT and PIPE as well\nin our C code; the latter isn't an issue here. SIGQUIT is a possibility,\nI guess, but seems rather unlikely.\n\n-Peff\n"},{"id":"365064","messageId":"20181211110919.GC8452@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181209225628.22216-3-szeder.dev@gmail.com","subject":"Re: [PATCH v2 2/7] test-lib: parse some --options earlier","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-11T11:09:19Z","receivedAt":"2018-12-11T11:09:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Dec 09, 2018 at 11:56:23PM +0100, SZEDER Gábor wrote:\n\n> 'test-lib.sh' looks for the presence of certain options like '--tee'\n> and '--verbose-log', so it can execute the test script again to save\n> its standard output and error. This happens way before the actual\n> option parsing loop, and the condition looking for these options looks\n> a bit odd, too. This patch series will add two more options to look\n> out for, and, in addition, will have to extract these options' stuck\n> arguments (i.e. '--opt=arg') as well.\n> \n> Add a proper option parsing loop to check these select options early\n> in 'test-lib.sh', making this early option checking more readable and\n> keeping those later changes in this series simpler. Use a 'for opt in\n> \"$@\"' loop to iterate over the options to preserve \"$@\" intact, so\n> options like '--verbose-log' can execute the test script again with\n> all the original options.\n> \n> As an alternative, we could parse all options early, but there are\n> options that do require an _unstuck_ argument, which is tricky to\n> handle properly in such a for loop, and the resulting complexity is,\n> in my opinion, worse than having this extra, partial option parsing\n> loop.\n\nIn general, I'm not wild about having multiple option-parsing loops that\nskip the normal left-to-right parsing, since it introduces funny corner\ncases (like \"-foo --bar\" which should be the same as \"--foo=--bar\"\ninstead thinking that \"--bar\" was passed as an option).\n\nBut looking at what this is replacing:\n\n> -case \"$GIT_TEST_TEE_STARTED, $* \" in\n> -done,*)\n> -\t# do not redirect again\n> -\t;;\n> -*' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*)\n\nyour version is easily an order of magnitude less horrible. ;)\n\n> t/test-lib.sh | 53 +++++++++++++++++++++++++++++++--------------------\n> 1 file changed, 32 insertions(+), 21 deletions(-)\n\nThis looks good to me overall, though...\n\n> +# Parse some options early, taking care to leave $@ intact.\n> +for opt\n> +do\n> +\tcase \"$opt\" in\n> +\t--tee)\n> +\t\ttee=t ;;\n> +\t-V|--verbose-log)\n> +\t\tverbose_log=t ;;\n> +\t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n> +\t\tvalgrind=memcheck ;;\n> +\t--valgrind=*)\n> +\t\tvalgrind=${opt#--*=} ;;\n> +\t--valgrind-only=*)\n> +\t\tvalgrind_only=${opt#--*=} ;;\n> +\t*)\n> +\t\t# Other options will be handled later.\n> +\tesac\n> +done\n> [...]\n> +elif test -n \"$tee\" || test -n \"$verbose_log\" ||\n> + test -n \"$valgrind\" || test -n \"$valgrind_only\"\n\nNow that we've nicely moved the parsing up, would it make sense to put\nthe annotation for \"this option implies --tee\" with those options?\n\nI.e., set tee=t when we see --verbose-log, which keeps all of the\nverbose-log logic together?\n\n> @@ -336,9 +344,12 @@ do\n> \t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n> \t\tfi\n> \t\tshift ;;\n> -\t-V|--verbose-log)\n> -\t\tverbose_log=t\n> -\t\tshift ;;\n> +\t--tee|\\\n> +\t-V|--verbose-log|\\\n> +\t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind|\\\n> +\t--valgrind=*|\\\n> +\t--valgrind-only=*)\n> +\t\tshift ;; # These options were handled already.\n> \t*)\n\nIt's too bad there's not an easy way to selectively remove from the $@\narray (which would avoid duplicating this list here).\n\nThe best I could come up with is:\n\n-- >8 --\nfirst=t\nfor i in \"$@\"; do\n\ttest -n \"$first\" && set --\n\tfirst=\n\n\tcase \"$i\" in\n\t--foo)\n\t\techo \"saw foo\" ;;\n\t*)\n\t\tset -- \"$@\" \"$i\" ;;\n\tesac\ndone\n\nfor i in \"$@\"; do\n\techo \"remainder: $i\"\ndone\n-- 8< --\n\nbut I won't be surprised if there are portability problems with\nassigning $@ in the middle of a loop that iterates over it.\n\n-Peff\n"},{"id":"365065","messageId":"20181211111654.GD8452@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181209225628.22216-1-szeder.dev@gmail.com","subject":"Re: [PATCH v2 0/7] test-lib: add the '--stress' option to help reproduce occasional failures in flaky tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-11T11:16:54Z","receivedAt":"2018-12-11T11:17:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Dec 09, 2018 at 11:56:21PM +0100, SZEDER Gábor wrote:\n\n> This patch series tries to make reproducing rare failures in flaky\n> tests easier: it adds the '--stress' option to our test library to run\n> the test script repeatedly in multiple parallel jobs, in the hope that\n> the increased load creates enough variance in the timing of the test's\n> commands that such a failure is eventually triggered.\n> \n> Notable changes since v1:\n> \n> - Made it more Peff-friendly :), namely it will now show the log of\n> the failed test and will rename its trash directory.\n> \n> Furthermore, '--stress' will now imply '--verbose -x --immediate'.\n\n:) Thanks. I do sympathize with the notion of keeping orthogonal things\northogonal, but I hope that this will make the tool a little more\npleasant to use out of the box. We can always tweak the behavior later,\ntoo. It's not like this is something user-facing that we've promised as\na scripting interface.\n\n> - Improved abort handling based on the discussion of the previous\n> version. (As a result, the patch is so heavily modified, that\n> 'range-diff' with default parameters consideres it a different\n> patch; increasing the creation factor then results in one big ugly\n> diff of a diff, so I left it as-is.)\n\nYeah, this all looked good to me.\n\n> - Added a few new patches, mostly preparatory refactorings, though\n> the first one might be considered a bugfix.\n\nI left a few minor comments, but these all looked good to me.\n\n-Peff\n"},{"id":"365072","messageId":"20181211124245.GT30222@szeder.dev","threadId":"49946","inReplyTo":"20181211110919.GC8452@sigill.intra.peff.net","subject":"Re: [PATCH v2 2/7] test-lib: parse some --options earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-11T12:42:45Z","receivedAt":"2018-12-11T12:42:52Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Dec 11, 2018 at 06:09:19AM -0500, Jeff King wrote:\n> On Sun, Dec 09, 2018 at 11:56:23PM +0100, SZEDER Gábor wrote:\n> \n> > 'test-lib.sh' looks for the presence of certain options like '--tee'\n> > and '--verbose-log', so it can execute the test script again to save\n> > its standard output and error. This happens way before the actual\n> > option parsing loop, and the condition looking for these options looks\n> > a bit odd, too. This patch series will add two more options to look\n> > out for, and, in addition, will have to extract these options' stuck\n> > arguments (i.e. '--opt=arg') as well.\n> > \n> > Add a proper option parsing loop to check these select options early\n> > in 'test-lib.sh', making this early option checking more readable and\n> > keeping those later changes in this series simpler. Use a 'for opt in\n> > \"$@\"' loop to iterate over the options to preserve \"$@\" intact, so\n> > options like '--verbose-log' can execute the test script again with\n> > all the original options.\n> > \n> > As an alternative, we could parse all options early, but there are\n> > options that do require an _unstuck_ argument, which is tricky to\n> > handle properly in such a for loop, and the resulting complexity is,\n> > in my opinion, worse than having this extra, partial option parsing\n> > loop.\n> \n> In general, I'm not wild about having multiple option-parsing loops that\n> skip the normal left-to-right parsing, since it introduces funny corner\n> cases (like \"-foo --bar\" which should be the same as \"--foo=--bar\"\n> instead thinking that \"--bar\" was passed as an option).\n\nYeah, that's already an \"issue\" in the current implementation as well,\nthough there are no such options that require options as argument.\n\n> But looking at what this is replacing:\n> \n> > -case \"$GIT_TEST_TEE_STARTED, $* \" in\n> > -done,*)\n> > -\t# do not redirect again\n> > -\t;;\n> > -*' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*)\n\n\nAnyway, I had another crack at turning the current option parsing loop\ninto a for loop keeping $@ intact, and the results don't look all that\nbad this time. Note that this diff below only does the while -> for\nconversion, but leaves the loop where it is, so the changes are easily\nvisible. The important bits are the conditions at the beginning of\nthe loop and after the loop, and the handling of '-r'; the rest is\nmostly s/shift// and sort-of s/$1/$opt/.\n\nThoughts? Is it better than two loops? I think it's better.\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9a3f7930a3..efdb6be3c8 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -264,58 +264,65 @@ test \"x$TERM\" != \"xdumb\" && (\n \t) &&\n \tcolor=t\n \n-while test \"$#\" -ne 0\n+store_arg_to=\n+prev_opt=\n+for opt\n do\n-\tcase \"$1\" in\n+\tif test -n \"$store_arg_to\"\n+\tthen\n+\t\teval $store_arg_to=\\$opt\n+\t\tstore_arg_to=\n+\t\tprev_opt=\n+\t\tcontinue\n+\tfi\n+\tcase \"$opt\" in\n \t-d|--d|--de|--deb|--debu|--debug)\n-\t\tdebug=t; shift ;;\n+\t\tdebug=t ;;\n \t-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)\n-\t\timmediate=t; shift ;;\n+\t\timmediate=t ;;\n \t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n-\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG; shift ;;\n+\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG ;;\n \t-r)\n-\t\tshift; test \"$#\" -ne 0 || {\n-\t\t\techo 'error: -r requires an argument' >&2;\n-\t\t\texit 1;\n-\t\t}\n-\t\trun_list=$1; shift ;;\n+\t\tstore_arg_to=run_list\n+\t\tprev_opt=-r\n+\t\t;;\n \t--run=*)\n-\t\trun_list=${1#--*=}; shift ;;\n+\t\trun_list=${opt#--*=} ;;\n \t-h|--h|--he|--hel|--help)\n-\t\thelp=t; shift ;;\n+\t\thelp=t ;;\n \t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n-\t\tverbose=t; shift ;;\n+\t\tverbose=t ;;\n \t--verbose-only=*)\n-\t\tverbose_only=${1#--*=}\n-\t\tshift ;;\n+\t\tverbose_only=${opt#--*=}\n+\t\t;;\n \t-q|--q|--qu|--qui|--quie|--quiet)\n \t\t# Ignore --quiet under a TAP::Harness. Saying how many tests\n \t\t# passed without the ok/not ok details is always an error.\n-\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t; shift ;;\n+\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n \t--with-dashes)\n-\t\twith_dashes=t; shift ;;\n+\t\twith_dashes=t ;;\n \t--no-color)\n-\t\tcolor=; shift ;;\n+\t\tcolor= ;;\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n \t\tvalgrind=memcheck\n-\t\tshift ;;\n+\t\t;;\n \t--valgrind=*)\n-\t\tvalgrind=${1#--*=}\n-\t\tshift ;;\n+\t\tvalgrind=${opt#--*=}\n+\t\t;;\n \t--valgrind-only=*)\n-\t\tvalgrind_only=${1#--*=}\n-\t\tshift ;;\n+\t\tvalgrind_only=${opt#--*=}\n+\t\t;;\n \t--tee)\n-\t\tshift ;; # was handled already\n+\t\t;; # was handled already\n \t--root=*)\n-\t\troot=${1#--*=}\n-\t\tshift ;;\n+\t\troot=${opt#--*=}\n+\t\t;;\n \t--chain-lint)\n \t\tGIT_TEST_CHAIN_LINT=1\n-\t\tshift ;;\n+\t\t;;\n \t--no-chain-lint)\n \t\tGIT_TEST_CHAIN_LINT=0\n-\t\tshift ;;\n+\t\t;;\n \t-x)\n \t\t# Some test scripts can't be reliably traced with '-x',\n \t\t# unless the test is run with a Bash version supporting\n@@ -335,15 +342,21 @@ do\n \t\telse\n \t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n \t\tfi\n-\t\tshift ;;\n+\t\t;;\n \t-V|--verbose-log)\n \t\tverbose_log=t\n-\t\tshift ;;\n+\t\t;;\n \t*)\n-\t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n+\t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n \tesac\n done\n \n+if test -n \"$store_arg_to\"\n+then\n+\techo \"error: $prev_opt requires an argument\" >&2\n+\texit 1\n+fi\n+\n if test -n \"$valgrind_only\"\n then\n \ttest -z \"$valgrind\" && valgrind=memcheck\n\n"},{"id":"365499","messageId":"20181217214436.GB14251@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181211124245.GT30222@szeder.dev","subject":"Re: [PATCH v2 2/7] test-lib: parse some --options earlier","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-17T21:44:36Z","receivedAt":"2018-12-17T21:44:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 11, 2018 at 01:42:45PM +0100, SZEDER Gábor wrote:\n\n> > But looking at what this is replacing:\n> > \n> > > -case \"$GIT_TEST_TEE_STARTED, $* \" in\n> > > -done,*)\n> > > -\t# do not redirect again\n> > > -\t;;\n> > > -*' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*)\n> \n> \n> Anyway, I had another crack at turning the current option parsing loop\n> into a for loop keeping $@ intact, and the results don't look all that\n> bad this time. Note that this diff below only does the while -> for\n> conversion, but leaves the loop where it is, so the changes are easily\n> visible. The important bits are the conditions at the beginning of\n> the loop and after the loop, and the handling of '-r'; the rest is\n> mostly s/shift// and sort-of s/$1/$opt/.\n> \n> Thoughts? Is it better than two loops? I think it's better.\n\nIt certainly looks better to me. It also makes sense to me to validate\nthe options before forking/logging, though I suppose one could argue the\nopposite.\n\nI wonder why we didn't do it this way in the beginning (i.e., why the\ntee bits were all handled separately before the parsing phase). I guess\njust because we have to pass the options down to the sub-process.\n\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 9a3f7930a3..efdb6be3c8 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -264,58 +264,65 @@ test \"x$TERM\" != \"xdumb\" && (\n> \t) &&\n> \tcolor=t\n> \n> -while test \"$#\" -ne 0\n> +store_arg_to=\n> +prev_opt=\n> +for opt\n> do\n> -\tcase \"$1\" in\n> +\tif test -n \"$store_arg_to\"\n> +\tthen\n> +\t\teval $store_arg_to=\\$opt\n> +\t\tstore_arg_to=\n> +\t\tprev_opt=\n> +\t\tcontinue\n> +\tfi\n\nOK, so this is set for the unstuck options, which then pick up the\noption in the next loop iteration. That's perhaps less gross than my\n\"re-build the options with set --\" trick.\n\nA simple variable set is enough for \"-r\". In theory we could make this:\n\n if test -n \"$handle_unstuck_arg\"\n then\n\teval \"$handle_unstuck_arg \\$1\"\n fi\n ...\n\n -r)\n\thandle_unstuck_arg=handle_opt_r ;;\n\nand handle_opt_r() could do whatever it wants. But I don't really\nforesee us adding a lot of new options (in fact, given that this is just\nthe internal tests, I am tempted to say that we should just make it\n\"-r\" for the sake of simplicity and consistency. But maybe somebody\nwould be annoyed. I have never used \"-r\" ever myself).\n\n-Peff\n"},{"id":"365985","messageId":"20181230190419.GB6120@szeder.dev","threadId":"49946","inReplyTo":"20181217214436.GB14251@sigill.intra.peff.net","subject":"Re: [PATCH v2 2/7] test-lib: parse some --options earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:04:19Z","receivedAt":"2018-12-30T19:04:25Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Dec 17, 2018 at 04:44:36PM -0500, Jeff King wrote:\n> On Tue, Dec 11, 2018 at 01:42:45PM +0100, SZEDER Gábor wrote:\n\n> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> > index 9a3f7930a3..efdb6be3c8 100644\n> > --- a/t/test-lib.sh\n> > +++ b/t/test-lib.sh\n> > @@ -264,58 +264,65 @@ test \"x$TERM\" != \"xdumb\" && (\n> > \t) &&\n> > \tcolor=t\n> > \n> > -while test \"$#\" -ne 0\n> > +store_arg_to=\n> > +prev_opt=\n> > +for opt\n> > do\n> > -\tcase \"$1\" in\n> > +\tif test -n \"$store_arg_to\"\n> > +\tthen\n> > +\t\teval $store_arg_to=\\$opt\n> > +\t\tstore_arg_to=\n> > +\t\tprev_opt=\n> > +\t\tcontinue\n> > +\tfi\n> \n> OK, so this is set for the unstuck options, which then pick up the\n> option in the next loop iteration. That's perhaps less gross than my\n> \"re-build the options with set --\" trick.\n> \n> A simple variable set is enough for \"-r\". In theory we could make this:\n> \n> if test -n \"$handle_unstuck_arg\"\n> then\n> \teval \"$handle_unstuck_arg \\$1\"\n> fi\n> ...\n> \n> -r)\n> \thandle_unstuck_arg=handle_opt_r ;;\n> \n> and handle_opt_r() could do whatever it wants. But I don't really\n> foresee us adding a lot of new options\n\nYeah, I would refrain from making it too general and fancy with a\ncallback function for now, when there is only a single option that\ncould use it.\n\n> (in fact, given that this is just\n> the internal tests, I am tempted to say that we should just make it\n> \"-r\" for the sake of simplicity and consistency. But maybe somebody\n> would be annoyed. I have never used \"-r\" ever myself).\n\nI didn't even know what '-r' does...\n\nAnd I agree that changing it to '-r' would be the best, but this\npatch series is about adding '--stress', so changing how '-r' gets its\nmandatory argument (and potentially annoying someone) is beyond the\nscope, I would say.\n\n"},{"id":"365986","messageId":"20181230191629.3232-1-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181209225628.22216-1-szeder.dev@gmail.com","subject":"[PATCH v3 0/8] test-lib: add the '--stress' option to help reproduce occasional failures in flaky tests","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:16:21Z","receivedAt":"2018-12-30T19:16:41Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"To recap: this patch series tries to make reproducing rare failures in\nflaky tests easier: it adds the '--stress' option to our test library\nto run the test script repeatedly in multiple parallel jobs, in the\nhope that the increased load creates enough variance in the timing of\nthe test's commands that such a failure is eventually triggered.\n\nChanges since v2:\n\n - Most importantly, parse all options before handling '--tee',\n '--verbose-log' or '--stress'. This constitutes the bulk of the\n range diff.\n\n - Add a bit of sanity check for the argument of '--stress=123'.\n\n - Minor commit message updates.\n\nv2 was quickly followed up with a fixup! patch; the range-diff below\nincludes that fixup! already squashed in.\n\nPrevious versions:\n\n v2: https://public-inbox.org/git/20181209225628.22216-1-szeder.dev@gmail.com/T/#u\n v1: https://public-inbox.org/git/20181204163457.15717-1-szeder.dev@gmail.com/T/\n\nSZEDER Gábor (8):\n test-lib: translate SIGTERM and SIGHUP to an exit\n test-lib: parse options in a for loop to keep $@ intact\n test-lib: parse command line options earlier\n test-lib: consolidate naming of test-results paths\n test-lib: set $TRASH_DIRECTORY earlier\n test-lib: extract Bash version check for '-x' tracing\n test-lib-functions: introduce the 'test_set_port' helper function\n test-lib: add the '--stress' option to run a test repeatedly under\n load\n\n t/README | 19 ++-\n t/lib-git-daemon.sh | 2 +-\n t/lib-git-p4.sh | 9 +-\n t/lib-git-svn.sh | 2 +-\n t/lib-httpd.sh | 2 +-\n t/t0410-partial-clone.sh | 1 -\n t/t5512-ls-remote.sh | 2 +-\n t/test-lib-functions.sh | 39 +++++\n t/test-lib.sh | 360 ++++++++++++++++++++++++++-------------\n 9 files changed, 303 insertions(+), 133 deletions(-)\n\nRange-diff:\n1: 3a5c926167 = 1: 48a2b19218 test-lib: translate SIGTERM and SIGHUP to an exit\n2: 8eee8d7fba < -: ---------- test-lib: parse some --options earlier\n-: ---------- > 2: adc5e8a86e test-lib: parse options in a for loop to keep $@ intact\n-: ---------- > 3: 89748074db test-lib: parse command line options earlier\n3: dd20ae5e9a ! 4: 5a6d17f54a test-lib: consolidate naming of test-results paths\n @@ -16,8 +16,8 @@\n --- a/t/test-lib.sh\n +++ b/t/test-lib.sh\n @@\n - \tesac\n - done\n + \tverbose=t\n + fi\n \n +TEST_NAME=\"$(basename \"$0\" .sh)\"\n +TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n @@ -27,8 +27,8 @@\n # additionally to the file test-results/$BASENAME.out, too.\n if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n @@\n - elif test -n \"$tee\" || test -n \"$verbose_log\" ||\n - test -n \"$valgrind\" || test -n \"$valgrind_only\"\n + \t: # do not redirect again\n + elif test -n \"$tee\"\n then\n -\tmkdir -p \"$TEST_OUTPUT_DIRECTORY/test-results\"\n -\tBASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$(basename \"$0\" .sh)\"\n4: f6e56c91c4 ! 5: cd7626b782 test-lib: set $TRASH_DIRECTORY earlier\n @@ -15,15 +15,6 @@\n diff --git a/t/test-lib.sh b/t/test-lib.sh\n --- a/t/test-lib.sh\n +++ b/t/test-lib.sh\n -@@\n - \t\tvalgrind=${opt#--*=} ;;\n - \t--valgrind-only=*)\n - \t\tvalgrind_only=${opt#--*=} ;;\n -+\t--root=*)\n -+\t\troot=${opt#--*=} ;;\n - \t*)\n - \t\t# Other options will be handled later.\n - \tesac\n @@\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n @@ -37,26 +28,6 @@\n \n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n -@@\n - \t\twith_dashes=t; shift ;;\n - \t--no-color)\n - \t\tcolor=; shift ;;\n --\t--root=*)\n --\t\troot=${1#--*=}\n --\t\tshift ;;\n - \t--chain-lint)\n - \t\tGIT_TEST_CHAIN_LINT=1\n - \t\tshift ;;\n -@@\n - \t-V|--verbose-log|\\\n - \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind|\\\n - \t--valgrind=*|\\\n --\t--valgrind-only=*)\n -+\t--valgrind-only=*|\\\n -+\t--root=*)\n - \t\tshift ;; # These options were handled already.\n - \t*)\n - \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n @@\n fi\n \n5: 99ecd2902e ! 6: 90296ac776 test-lib: extract Bash version check for '-x' tracing\n @@ -2,23 +2,30 @@\n \n test-lib: extract Bash version check for '-x' tracing\n \n - Some of our test scripts can't be reliably run with '-x' tracing\n - enabled unless executed with a Bash version supporting BASH_XTRACEFD\n - (since v4.1), and we have a lengthy condition to disable tracing if\n - such a test script is not executed with a suitable Bash version.\n + One of our test scripts, 't1510-repo-setup.sh' [1], still can't be\n + reliably run with '-x' tracing enabled, unless it's executed with a\n + Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy\n + condition to check the version of the shell running the test script,\n + and disable tracing if it's not executed with a suitable Bash version\n + [2].\n \n Move this check out from the option parsing loop, so other options can\n imply '-x' by setting 'trace=t', without missing this Bash version\n check.\n \n + [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x',\n + 2018-02-24)\n + [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual\n + test scripts, 2018-02-24)\n +\n Signed-off-by: SZEDER Gábor \n \n diff --git a/t/test-lib.sh b/t/test-lib.sh\n --- a/t/test-lib.sh\n +++ b/t/test-lib.sh\n @@\n - \t\tGIT_TEST_CHAIN_LINT=0\n - \t\tshift ;;\n + \t--no-chain-lint)\n + \t\tGIT_TEST_CHAIN_LINT=0 ;;\n \t-x)\n -\t\t# Some test scripts can't be reliably traced with '-x',\n -\t\t# unless the test is run with a Bash version supporting\n @@ -38,10 +45,11 @@\n -\t\telse\n -\t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n -\t\tfi\n -+\t\ttrace=t\n - \t\tshift ;;\n - \t--tee|\\\n - \t-V|--verbose-log|\\\n +-\t\t;;\n ++\t\ttrace=t ;;\n + \t-V|--verbose-log)\n + \t\tverbose_log=t\n + \t\ttee=t\n @@\n \ttest -z \"$verbose_log\" && verbose=t\n fi\n6: dcf7d2a397 ! 7: a4a3e7cefa test-lib-functions: introduce the 'test_set_port' helper function\n @@ -178,7 +178,7 @@\n +\n +\t\teval $var=$port\n +\t\t;;\n -+\t*[^0-9]*)\n ++\t*[^0-9]*|0*)\n +\t\terror >&7 \"invalid port number: $port\"\n +\t\t;;\n +\t*)\n7: f27f2fc1d0 ! 8: b0dcff0d3f test-lib: add the '--stress' option to run a test repeatedly under load\n @@ -4,12 +4,12 @@\n \n Unfortunately, we have a few flaky tests, whose failures tend to be\n hard to reproduce. We've found that the best we can do to reproduce\n - such a failure is to run the test repeatedly while the machine is\n - under load, and wait in the hope that the load creates enough variance\n - in the timing of the test's commands that a failure is evenually\n - triggered. I have a command to do that, and I noticed that two other\n - contributors have rolled their own scripts to do the same, all\n - choosing slightly different approaches.\n + such a failure is to run the test script repeatedly while the machine\n + is under load, and wait in the hope that the load creates enough\n + variance in the timing of the test's commands that a failure is\n + evenually triggered. I have a command to do that, and I noticed that\n + two other contributors have rolled their own scripts to do the same,\n + all choosing slightly different approaches.\n \n To help reproduce failures in flaky tests, introduce the '--stress'\n option to run a test script repeatedly in multiple parallel jobs until\n @@ -37,19 +37,21 @@\n user or to the test number, so even tests involving daemons\n listening on a TCP socket can be stressed.\n \n - - Redirect each parallel test run's output to\n + - Redirect each parallel test run's verbose output to\n 't/test-results/$TEST_NAME.stress-.out', because dumping the\n output of several parallel running tests to the terminal would\n create a big ugly mess.\n \n For convenience, print the output of the failed test job at the end,\n and rename its trash directory to end with the '.stress-failed'\n - suffix, so it's easy to find in a predictable path. If, in an\n - unlikely case, more than one jobs were to fail nearly at the same\n - time, then print the output of all failed jobs, and rename the trash\n - directory of only the last one (i.e. with the highest job number), as\n - it is the trash directory of the test whose output will be at the\n - bottom of the user's terminal.\n + suffix, so it's easy to find in a predictable path (OTOH, all absolute\n + paths recorded in the trash directory become invalid; we'll see\n + whether this causes any issues in practice). If, in an unlikely case,\n + more than one jobs were to fail nearly at the same time, then print\n + the output of all failed jobs, and rename the trash directory of only\n + the last one (i.e. with the highest job number), as it is the trash\n + directory of the test whose output will be at the bottom of the user's\n + terminal.\n \n Based on Jeff King's 'stress' script.\n \n @@ -102,7 +104,7 @@\n -\n -\t\teval $var=$port\n \t\t;;\n - \t*[^0-9]*)\n + \t*[^0-9]*|0*)\n \t\terror >&7 \"invalid port number: $port\"\n @@\n \t\t# The user has specified the port.\n @@ -119,17 +121,42 @@\n --- a/t/test-lib.sh\n +++ b/t/test-lib.sh\n @@\n - \t\tvalgrind_only=${opt#--*=} ;;\n - \t--root=*)\n - \t\troot=${opt#--*=} ;;\n + \t\tverbose_log=t\n + \t\ttee=t\n + \t\t;;\n +\t--stress)\n +\t\tstress=t ;;\n +\t--stress=*)\n -+\t\tstress=${opt#--*=} ;;\n ++\t\tstress=${opt#--*=}\n ++\t\tcase \"$stress\" in\n ++\t\t*[^0-9]*|0*|\"\")\n ++\t\t\techo \"error: --stress= requires the number of jobs to run\" >&2\n ++\t\t\texit 1\n ++\t\t\t;;\n ++\t\t*)\t# Good.\n ++\t\t\t;;\n ++\t\tesac\n ++\t\t;;\n \t*)\n - \t\t# Other options will be handled later.\n + \t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n \tesac\n - done\n +@@\n + \ttest -z \"$verbose_log\" && verbose=t\n + fi\n + \n ++if test -n \"$stress\"\n ++then\n ++\tverbose=t\n ++\ttrace=t\n ++\timmediate=t\n ++fi\n ++\n + if test -n \"$trace\" && test -n \"$test_untraceable\"\n + then\n + \t# '-x' tracing requested, but this test script can't be reliably\n +@@\n + \tverbose=t\n + fi\n \n +TEST_STRESS_JOB_SFX=\"${GIT_TEST_STRESS_JOB_NR:+.stress-$GIT_TEST_STRESS_JOB_NR}\"\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n @@ -231,27 +258,3 @@\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n -@@\n - \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind|\\\n - \t--valgrind=*|\\\n - \t--valgrind-only=*|\\\n --\t--root=*)\n -+\t--root=*|\\\n -+\t--stress|--stress=*)\n - \t\tshift ;; # These options were handled already.\n - \t*)\n - \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n -@@\n - \tverbose=t\n - fi\n - \n -+if test -n \"$stress\"\n -+then\n -+\tverbose=t\n -+\ttrace=t\n -+\timmediate=t\n -+fi\n -+\n - if test -n \"$color\"\n - then\n - \t# Save the color control sequences now rather than run tput\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"365987","messageId":"20181230191629.3232-2-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-1-szeder.dev@gmail.com","subject":"[PATCH v3 1/8] test-lib: translate SIGTERM and SIGHUP to an exit","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:16:22Z","receivedAt":"2018-12-30T19:16:55Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Right now if a test script receives SIGTERM or SIGHUP (e.g., because a\ntest was hanging and the user 'kill'-ed it or simply closed the\nterminal window the test was running in), the shell exits immediately.\nThis can be annoying if the test script did any global setup, like\nstarting apache or git-daemon, as it will not have an opportunity to\nclean up after itself. A subsequent run of the test won't be able to\nstart its own daemon, and will either fail or skip the tests.\n\nInstead, let's trap SIGTERM and SIGHUP as well to make sure we do a\nclean shutdown, and just chain it to a normal exit (which will trigger\nany cleanup).\n\nThis patch follows suit of da706545f7 (t: translate SIGINT to an exit,\n2015-03-13), and even stole its commit message as well.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 0f1faa24b2..9a3f7930a3 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -476,7 +476,7 @@ die () {\n \n GIT_EXIT_OK=\n trap 'die' EXIT\n-trap 'exit $?' INT\n+trap 'exit $?' INT TERM HUP\n \n # The user-facing functions are loaded from a separate file so that\n # test_perf subshells can have them too\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"365988","messageId":"20181230191629.3232-3-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-1-szeder.dev@gmail.com","subject":"[PATCH v3 2/8] test-lib: parse options in a for loop to keep $@ intact","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:16:23Z","receivedAt":"2018-12-30T19:16:57Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"'test-lib.sh' looks for the presence of certain options like '--tee'\nand '--verbose-log', so it can execute the test script again to save\nits standard output and error, and to do so it needs the original\ncommand line options the test was invoked with.\n\nThe next patch is about to move the option parsing loop earlier in\n'test-lib.sh', but it is implemented using 'shift' in a while loop,\neffecively destroying \"$@\" by the end of the option parsing. Not\ngood.\n\nAs a preparatory step, turn that option parsing loop into a 'for opt\nin \"$@\"' loop to preserve \"$@\" intact while iterating over the\noptions, and taking extra care to handle the '-r' option's required\nargument (or the lack thereof).\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 77 ++++++++++++++++++++++++++++-----------------------\n 1 file changed, 42 insertions(+), 35 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9a3f7930a3..ed7267962f 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -264,58 +264,59 @@ test \"x$TERM\" != \"xdumb\" && (\n \t) &&\n \tcolor=t\n \n-while test \"$#\" -ne 0\n+store_arg_to=\n+prev_opt=\n+for opt\n do\n-\tcase \"$1\" in\n+\tif test -n \"$store_arg_to\"\n+\tthen\n+\t\teval $store_arg_to=\\$opt\n+\t\tstore_arg_to=\n+\t\tprev_opt=\n+\t\tcontinue\n+\tfi\n+\n+\tcase \"$opt\" in\n \t-d|--d|--de|--deb|--debu|--debug)\n-\t\tdebug=t; shift ;;\n+\t\tdebug=t ;;\n \t-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)\n-\t\timmediate=t; shift ;;\n+\t\timmediate=t ;;\n \t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n-\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG; shift ;;\n+\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG ;;\n \t-r)\n-\t\tshift; test \"$#\" -ne 0 || {\n-\t\t\techo 'error: -r requires an argument' >&2;\n-\t\t\texit 1;\n-\t\t}\n-\t\trun_list=$1; shift ;;\n+\t\tstore_arg_to=run_list\n+\t\t;;\n \t--run=*)\n-\t\trun_list=${1#--*=}; shift ;;\n+\t\trun_list=${opt#--*=} ;;\n \t-h|--h|--he|--hel|--help)\n-\t\thelp=t; shift ;;\n+\t\thelp=t ;;\n \t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n-\t\tverbose=t; shift ;;\n+\t\tverbose=t ;;\n \t--verbose-only=*)\n-\t\tverbose_only=${1#--*=}\n-\t\tshift ;;\n+\t\tverbose_only=${opt#--*=}\n+\t\t;;\n \t-q|--q|--qu|--qui|--quie|--quiet)\n \t\t# Ignore --quiet under a TAP::Harness. Saying how many tests\n \t\t# passed without the ok/not ok details is always an error.\n-\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t; shift ;;\n+\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n \t--with-dashes)\n-\t\twith_dashes=t; shift ;;\n+\t\twith_dashes=t ;;\n \t--no-color)\n-\t\tcolor=; shift ;;\n+\t\tcolor= ;;\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n-\t\tvalgrind=memcheck\n-\t\tshift ;;\n+\t\tvalgrind=memcheck ;;\n \t--valgrind=*)\n-\t\tvalgrind=${1#--*=}\n-\t\tshift ;;\n+\t\tvalgrind=${opt#--*=} ;;\n \t--valgrind-only=*)\n-\t\tvalgrind_only=${1#--*=}\n-\t\tshift ;;\n+\t\tvalgrind_only=${opt#--*=} ;;\n \t--tee)\n-\t\tshift ;; # was handled already\n+\t\t;; # was handled already\n \t--root=*)\n-\t\troot=${1#--*=}\n-\t\tshift ;;\n+\t\troot=${opt#--*=} ;;\n \t--chain-lint)\n-\t\tGIT_TEST_CHAIN_LINT=1\n-\t\tshift ;;\n+\t\tGIT_TEST_CHAIN_LINT=1 ;;\n \t--no-chain-lint)\n-\t\tGIT_TEST_CHAIN_LINT=0\n-\t\tshift ;;\n+\t\tGIT_TEST_CHAIN_LINT=0 ;;\n \t-x)\n \t\t# Some test scripts can't be reliably traced with '-x',\n \t\t# unless the test is run with a Bash version supporting\n@@ -335,14 +336,20 @@ do\n \t\telse\n \t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n \t\tfi\n-\t\tshift ;;\n+\t\t;;\n \t-V|--verbose-log)\n-\t\tverbose_log=t\n-\t\tshift ;;\n+\t\tverbose_log=t ;;\n \t*)\n-\t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n+\t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n \tesac\n+\n+\tprev_opt=$opt\n done\n+if test -n \"$store_arg_to\"\n+then\n+\techo \"error: $prev_opt requires an argument\" >&2\n+\texit 1\n+fi\n \n if test -n \"$valgrind_only\"\n then\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"365989","messageId":"20181230191629.3232-4-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-1-szeder.dev@gmail.com","subject":"[PATCH v3 3/8] test-lib: parse command line options earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:16:24Z","receivedAt":"2018-12-30T19:17:01Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"'test-lib.sh' looks for the presence of certain options like '--tee'\nand '--verbose-log', so it can execute the test script again to save\nits standard output and error. It looks for '--valgrind' as well, to\nset up some Valgrind-specific stuff. These all happen before the\nactual option parsing loop, and the conditions looking for these\noptions look a bit odd, too. They are not completely correct, either,\nbecause in a bogus invocation like './t1234-foo.sh -r --tee' they\nrecognize '--tee', although it should be handled as the required\nargument of the '-r' option. This patch series will add two more\noptions to look out for early, and, in addition, will have to extract\nthese options' stuck arguments (i.e. '--opt=arg') as well.\n\nSo let's move the option parsing loop and the couple of related\nconditions following it earlier in 'test-lib.sh', before the place\nwhere the test script is executed again for '--tee' and its friends.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 228 ++++++++++++++++++++++++++------------------------\n 1 file changed, 119 insertions(+), 109 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ed7267962f..fcc04afdff 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -71,13 +71,125 @@ then\n \texit 1\n fi\n \n+# Parse options while taking care to leave $@ intact, so we will still\n+# have all the original command line options when executing the test\n+# script again for '--tee' and '--verbose-log' below.\n+store_arg_to=\n+prev_opt=\n+for opt\n+do\n+\tif test -n \"$store_arg_to\"\n+\tthen\n+\t\teval $store_arg_to=\\$opt\n+\t\tstore_arg_to=\n+\t\tprev_opt=\n+\t\tcontinue\n+\tfi\n+\n+\tcase \"$opt\" in\n+\t-d|--d|--de|--deb|--debu|--debug)\n+\t\tdebug=t ;;\n+\t-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)\n+\t\timmediate=t ;;\n+\t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n+\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG ;;\n+\t-r)\n+\t\tstore_arg_to=run_list\n+\t\t;;\n+\t--run=*)\n+\t\trun_list=${opt#--*=} ;;\n+\t-h|--h|--he|--hel|--help)\n+\t\thelp=t ;;\n+\t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n+\t\tverbose=t ;;\n+\t--verbose-only=*)\n+\t\tverbose_only=${opt#--*=}\n+\t\t;;\n+\t-q|--q|--qu|--qui|--quie|--quiet)\n+\t\t# Ignore --quiet under a TAP::Harness. Saying how many tests\n+\t\t# passed without the ok/not ok details is always an error.\n+\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n+\t--with-dashes)\n+\t\twith_dashes=t ;;\n+\t--no-color)\n+\t\tcolor= ;;\n+\t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n+\t\tvalgrind=memcheck\n+\t\ttee=t\n+\t\t;;\n+\t--valgrind=*)\n+\t\tvalgrind=${opt#--*=}\n+\t\ttee=t\n+\t\t;;\n+\t--valgrind-only=*)\n+\t\tvalgrind_only=${opt#--*=}\n+\t\ttee=t\n+\t\t;;\n+\t--tee)\n+\t\ttee=t ;;\n+\t--root=*)\n+\t\troot=${opt#--*=} ;;\n+\t--chain-lint)\n+\t\tGIT_TEST_CHAIN_LINT=1 ;;\n+\t--no-chain-lint)\n+\t\tGIT_TEST_CHAIN_LINT=0 ;;\n+\t-x)\n+\t\t# Some test scripts can't be reliably traced with '-x',\n+\t\t# unless the test is run with a Bash version supporting\n+\t\t# BASH_XTRACEFD (introduced in Bash v4.1). Check whether\n+\t\t# this test is marked as such, and ignore '-x' if it\n+\t\t# isn't executed with a suitable Bash version.\n+\t\tif test -z \"$test_untraceable\" || {\n+\t\t test -n \"$BASH_VERSION\" && {\n+\t\t test ${BASH_VERSINFO[0]} -gt 4 || {\n+\t\t\t test ${BASH_VERSINFO[0]} -eq 4 &&\n+\t\t\t test ${BASH_VERSINFO[1]} -ge 1\n+\t\t }\n+\t\t }\n+\t\t }\n+\t\tthen\n+\t\t\ttrace=t\n+\t\telse\n+\t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n+\t\tfi\n+\t\t;;\n+\t-V|--verbose-log)\n+\t\tverbose_log=t\n+\t\ttee=t\n+\t\t;;\n+\t*)\n+\t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n+\tesac\n+\n+\tprev_opt=$opt\n+done\n+if test -n \"$store_arg_to\"\n+then\n+\techo \"error: $prev_opt requires an argument\" >&2\n+\texit 1\n+fi\n+\n+if test -n \"$valgrind_only\"\n+then\n+\ttest -z \"$valgrind\" && valgrind=memcheck\n+\ttest -z \"$verbose\" && verbose_only=\"$valgrind_only\"\n+elif test -n \"$valgrind\"\n+then\n+\ttest -z \"$verbose_log\" && verbose=t\n+fi\n+\n+if test -n \"$trace\" && test -z \"$verbose_log\"\n+then\n+\tverbose=t\n+fi\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n-case \"$GIT_TEST_TEE_STARTED, $* \" in\n-done,*)\n-\t# do not redirect again\n-\t;;\n-*' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*)\n+if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n+then\n+\t: # do not redirect again\n+elif test -n \"$tee\"\n+then\n \tmkdir -p \"$TEST_OUTPUT_DIRECTORY/test-results\"\n \tBASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$(basename \"$0\" .sh)\"\n \n@@ -94,8 +206,7 @@ done,*)\n \t echo $? >\"$BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n \ttest \"$(cat \"$BASE.exit\")\" = 0\n \texit\n-\t;;\n-esac\n+fi\n \n # For repeatability, reset the environment to known value.\n # TERM is sanitized below, after saving color control sequences.\n@@ -193,7 +304,7 @@ fi\n \n # Add libc MALLOC and MALLOC_PERTURB test\n # only if we are not executing the test with valgrind\n-if expr \" $GIT_TEST_OPTS \" : \".* --valgrind \" >/dev/null ||\n+if test -n \"$valgrind\" ||\n test -n \"$TEST_NO_MALLOC_CHECK\"\n then\n \tsetup_malloc_check () {\n@@ -264,107 +375,6 @@ test \"x$TERM\" != \"xdumb\" && (\n \t) &&\n \tcolor=t\n \n-store_arg_to=\n-prev_opt=\n-for opt\n-do\n-\tif test -n \"$store_arg_to\"\n-\tthen\n-\t\teval $store_arg_to=\\$opt\n-\t\tstore_arg_to=\n-\t\tprev_opt=\n-\t\tcontinue\n-\tfi\n-\n-\tcase \"$opt\" in\n-\t-d|--d|--de|--deb|--debu|--debug)\n-\t\tdebug=t ;;\n-\t-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)\n-\t\timmediate=t ;;\n-\t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n-\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG ;;\n-\t-r)\n-\t\tstore_arg_to=run_list\n-\t\t;;\n-\t--run=*)\n-\t\trun_list=${opt#--*=} ;;\n-\t-h|--h|--he|--hel|--help)\n-\t\thelp=t ;;\n-\t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n-\t\tverbose=t ;;\n-\t--verbose-only=*)\n-\t\tverbose_only=${opt#--*=}\n-\t\t;;\n-\t-q|--q|--qu|--qui|--quie|--quiet)\n-\t\t# Ignore --quiet under a TAP::Harness. Saying how many tests\n-\t\t# passed without the ok/not ok details is always an error.\n-\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n-\t--with-dashes)\n-\t\twith_dashes=t ;;\n-\t--no-color)\n-\t\tcolor= ;;\n-\t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n-\t\tvalgrind=memcheck ;;\n-\t--valgrind=*)\n-\t\tvalgrind=${opt#--*=} ;;\n-\t--valgrind-only=*)\n-\t\tvalgrind_only=${opt#--*=} ;;\n-\t--tee)\n-\t\t;; # was handled already\n-\t--root=*)\n-\t\troot=${opt#--*=} ;;\n-\t--chain-lint)\n-\t\tGIT_TEST_CHAIN_LINT=1 ;;\n-\t--no-chain-lint)\n-\t\tGIT_TEST_CHAIN_LINT=0 ;;\n-\t-x)\n-\t\t# Some test scripts can't be reliably traced with '-x',\n-\t\t# unless the test is run with a Bash version supporting\n-\t\t# BASH_XTRACEFD (introduced in Bash v4.1). Check whether\n-\t\t# this test is marked as such, and ignore '-x' if it\n-\t\t# isn't executed with a suitable Bash version.\n-\t\tif test -z \"$test_untraceable\" || {\n-\t\t test -n \"$BASH_VERSION\" && {\n-\t\t test ${BASH_VERSINFO[0]} -gt 4 || {\n-\t\t\t test ${BASH_VERSINFO[0]} -eq 4 &&\n-\t\t\t test ${BASH_VERSINFO[1]} -ge 1\n-\t\t }\n-\t\t }\n-\t\t }\n-\t\tthen\n-\t\t\ttrace=t\n-\t\telse\n-\t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n-\t\tfi\n-\t\t;;\n-\t-V|--verbose-log)\n-\t\tverbose_log=t ;;\n-\t*)\n-\t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n-\tesac\n-\n-\tprev_opt=$opt\n-done\n-if test -n \"$store_arg_to\"\n-then\n-\techo \"error: $prev_opt requires an argument\" >&2\n-\texit 1\n-fi\n-\n-if test -n \"$valgrind_only\"\n-then\n-\ttest -z \"$valgrind\" && valgrind=memcheck\n-\ttest -z \"$verbose\" && verbose_only=\"$valgrind_only\"\n-elif test -n \"$valgrind\"\n-then\n-\ttest -z \"$verbose_log\" && verbose=t\n-fi\n-\n-if test -n \"$trace\" && test -z \"$verbose_log\"\n-then\n-\tverbose=t\n-fi\n-\n if test -n \"$color\"\n then\n \t# Save the color control sequences now rather than run tput\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"365990","messageId":"20181230191629.3232-7-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-1-szeder.dev@gmail.com","subject":"[PATCH v3 6/8] test-lib: extract Bash version check for '-x' tracing","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:16:27Z","receivedAt":"2018-12-30T19:17:03Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"One of our test scripts, 't1510-repo-setup.sh' [1], still can't be\nreliably run with '-x' tracing enabled, unless it's executed with a\nBash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy\ncondition to check the version of the shell running the test script,\nand disable tracing if it's not executed with a suitable Bash version\n[2].\n\nMove this check out from the option parsing loop, so other options can\nimply '-x' by setting 'trace=t', without missing this Bash version\ncheck.\n\n[1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x',\n 2018-02-24)\n[2] 5fc98e79fc (t: add means to disable '-x' tracing for individual\n test scripts, 2018-02-24)\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 38 +++++++++++++++++++-------------------\n 1 file changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 2b88ba2de1..7d77a26d44 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -134,25 +134,7 @@ do\n \t--no-chain-lint)\n \t\tGIT_TEST_CHAIN_LINT=0 ;;\n \t-x)\n-\t\t# Some test scripts can't be reliably traced with '-x',\n-\t\t# unless the test is run with a Bash version supporting\n-\t\t# BASH_XTRACEFD (introduced in Bash v4.1). Check whether\n-\t\t# this test is marked as such, and ignore '-x' if it\n-\t\t# isn't executed with a suitable Bash version.\n-\t\tif test -z \"$test_untraceable\" || {\n-\t\t test -n \"$BASH_VERSION\" && {\n-\t\t test ${BASH_VERSINFO[0]} -gt 4 || {\n-\t\t\t test ${BASH_VERSINFO[0]} -eq 4 &&\n-\t\t\t test ${BASH_VERSINFO[1]} -ge 1\n-\t\t }\n-\t\t }\n-\t\t }\n-\t\tthen\n-\t\t\ttrace=t\n-\t\telse\n-\t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n-\t\tfi\n-\t\t;;\n+\t\ttrace=t ;;\n \t-V|--verbose-log)\n \t\tverbose_log=t\n \t\ttee=t\n@@ -178,6 +160,24 @@ then\n \ttest -z \"$verbose_log\" && verbose=t\n fi\n \n+if test -n \"$trace\" && test -n \"$test_untraceable\"\n+then\n+\t# '-x' tracing requested, but this test script can't be reliably\n+\t# traced, unless it is run with a Bash version supporting\n+\t# BASH_XTRACEFD (introduced in Bash v4.1).\n+\tif test -n \"$BASH_VERSION\" && {\n+\t test ${BASH_VERSINFO[0]} -gt 4 || {\n+\t test ${BASH_VERSINFO[0]} -eq 4 &&\n+\t test ${BASH_VERSINFO[1]} -ge 1\n+\t }\n+\t }\n+\tthen\n+\t\t: Executed by a Bash version supporting BASH_XTRACEFD. Good.\n+\telse\n+\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n+\t\ttrace=\n+\tfi\n+fi\n if test -n \"$trace\" && test -z \"$verbose_log\"\n then\n \tverbose=t\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"365991","messageId":"20181230191629.3232-8-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-1-szeder.dev@gmail.com","subject":"[PATCH v3 7/8] test-lib-functions: introduce the 'test_set_port' helper function","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:16:28Z","receivedAt":"2018-12-30T19:17:04Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Several test scripts run daemons like 'git-daemon' or Apache, and\ncommunicate with them through TCP sockets. To have unique ports where\nthese daemons are accessible, the ports are usually the number of the\ncorresponding test scripts, unless the user overrides them via\nenvironment variables, and thus all those tests and test libs contain\nmore or less the same bit of one-liner boilerplate code to find out\nthe port. The last patch in this series will make this a bit more\ncomplicated.\n\nFactor out finding the port for a daemon into the common helper\nfunction 'test_set_port' to avoid repeating ourselves.\n\nTake special care of test scripts with \"low\" numbers:\n\n - Test numbers below 1024 would result in a port that's only usable\n as root, so set their port to '10000 + test-nr' to make sure it\n doesn't interfere with other tests in the test suite. This makes\n the hardcoded port number in 't0410-partial-clone.sh' unnecessary,\n remove it.\n\n - The shell's arithmetic evaluation interprets numbers with leading\n zeros as octal values, which means that test number below 1000 and\n containing the digits 8 or 9 will trigger an error. Remove all\n leading zeros from the test numbers to prevent this.\n\nNote that the 'git p4' tests are unlike the other tests involving\ndaemons in that:\n\n - 'lib-git-p4.sh' doesn't use the test's number for unique port as\n is, but does a bit of additional arithmetic on top [1].\n\n - The port is not overridable via an environment variable.\n\nWith this patch even 'git p4' tests will use the test's number as\ndefault port, and it will be overridable via the P4DPORT environment\nvariable.\n\n[1] Commit fc00233071 (git-p4 tests: refactor and cleanup, 2011-08-22)\n introduced that \"unusual\" unique port computation without\n explaining why it was necessary (as opposed to simply using the\n test number as is). It seems to be just unnecessary complication,\n and in any case that commit came way before the \"test nr as unique\n port\" got \"standardized\" for other daemons in commits c44132fcf3\n (tests: auto-set git-daemon port, 2014-02-10), 3bb486e439 (tests:\n auto-set LIB_HTTPD_PORT from test name, 2014-02-10), and\n bf9d7df950 (t/lib-git-svn.sh: improve svnserve tests with parallel\n make test, 2017-12-01).\n\nSigned-off-by: SZEDER Gábor \n---\n t/lib-git-daemon.sh | 2 +-\n t/lib-git-p4.sh | 9 +--------\n t/lib-git-svn.sh | 2 +-\n t/lib-httpd.sh | 2 +-\n t/t0410-partial-clone.sh | 1 -\n t/t5512-ls-remote.sh | 2 +-\n t/test-lib-functions.sh | 36 ++++++++++++++++++++++++++++++++++++\n 7 files changed, 41 insertions(+), 13 deletions(-)\n\ndiff --git a/t/lib-git-daemon.sh b/t/lib-git-daemon.sh\nindex f98de95c15..41eb1e3ae8 100644\n--- a/t/lib-git-daemon.sh\n+++ b/t/lib-git-daemon.sh\n@@ -28,7 +28,7 @@ then\n \ttest_skip_or_die $GIT_TEST_GIT_DAEMON \"file system does not support FIFOs\"\n fi\n \n-LIB_GIT_DAEMON_PORT=${LIB_GIT_DAEMON_PORT-${this_test#t}}\n+test_set_port LIB_GIT_DAEMON_PORT\n \n GIT_DAEMON_PID=\n GIT_DAEMON_DOCUMENT_ROOT_PATH=\"$PWD\"/repo\ndiff --git a/t/lib-git-p4.sh b/t/lib-git-p4.sh\nindex c27599474c..b3be3ba011 100644\n--- a/t/lib-git-p4.sh\n+++ b/t/lib-git-p4.sh\n@@ -53,14 +53,7 @@ time_in_seconds () {\n \t(cd / && \"$PYTHON_PATH\" -c 'import time; print(int(time.time()))')\n }\n \n-# Try to pick a unique port: guess a large number, then hope\n-# no more than one of each test is running.\n-#\n-# This does not handle the case where somebody else is running the\n-# same tests and has chosen the same ports.\n-testid=${this_test#t}\n-git_p4_test_start=9800\n-P4DPORT=$((10669 + ($testid - $git_p4_test_start)))\n+test_set_port P4DPORT\n \n P4PORT=localhost:$P4DPORT\n P4CLIENT=client\ndiff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh\nindex a8130f9119..f3b478c307 100644\n--- a/t/lib-git-svn.sh\n+++ b/t/lib-git-svn.sh\n@@ -13,6 +13,7 @@ fi\n GIT_DIR=$PWD/.git\n GIT_SVN_DIR=$GIT_DIR/svn/refs/remotes/git-svn\n SVN_TREE=$GIT_SVN_DIR/svn-tree\n+test_set_port SVNSERVE_PORT\n \n svn >/dev/null 2>&1\n if test $? -ne 1\n@@ -119,7 +120,6 @@ require_svnserve () {\n }\n \n start_svnserve () {\n-\tSVNSERVE_PORT=${SVNSERVE_PORT-${this_test#t}}\n \tsvnserve --listen-port $SVNSERVE_PORT \\\n \t\t --root \"$rawsvnrepo\" \\\n \t\t --listen-once \\\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex a8729f8232..e465116ef9 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -82,7 +82,7 @@ case $(uname) in\n esac\n \n LIB_HTTPD_PATH=${LIB_HTTPD_PATH-\"$DEFAULT_HTTPD_PATH\"}\n-LIB_HTTPD_PORT=${LIB_HTTPD_PORT-${this_test#t}}\n+test_set_port LIB_HTTPD_PORT\n \n TEST_PATH=\"$TEST_DIRECTORY\"/lib-httpd\n HTTPD_ROOT_PATH=\"$PWD\"/httpd\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex ba3887f178..0aca8d7588 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -480,7 +480,6 @@ test_expect_success 'gc stops traversal when a missing but promised object is re\n \t! grep \"$TREE_HASH\" out\n '\n \n-LIB_HTTPD_PORT=12345 # default port, 410, cannot be used as non-root\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \ndiff --git a/t/t5512-ls-remote.sh b/t/t5512-ls-remote.sh\nindex 32e722db2e..cd9e60632d 100755\n--- a/t/t5512-ls-remote.sh\n+++ b/t/t5512-ls-remote.sh\n@@ -260,7 +260,7 @@ test_lazy_prereq GIT_DAEMON '\n # This test spawns a daemon, so run it only if the user would be OK with\n # testing with git-daemon.\n test_expect_success PIPE,JGIT,GIT_DAEMON 'indicate no refs in standards-compliant empty remote' '\n-\tJGIT_DAEMON_PORT=${JGIT_DAEMON_PORT-${this_test#t}} &&\n+\ttest_set_port JGIT_DAEMON_PORT &&\n \tJGIT_DAEMON_PID= &&\n \tgit init --bare empty.git &&\n \t>empty.git/git-daemon-export-ok &&\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6b3bbf99e4..4459bdda13 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1263,3 +1263,39 @@ test_oid () {\n \tfi &&\n \teval \"printf '%s' \\\"\\${$var}\\\"\"\n }\n+\n+# Choose a port number based on the test script's number and store it in\n+# the given variable name, unless that variable already contains a number.\n+test_set_port () {\n+\tlocal var=$1 port\n+\n+\tif test $# -ne 1 || test -z \"$var\"\n+\tthen\n+\t\tBUG \"test_set_port requires a variable name\"\n+\tfi\n+\n+\teval port=\\$$var\n+\tcase \"$port\" in\n+\t\"\")\n+\t\t# No port is set in the given env var, use the test\n+\t\t# number as port number instead.\n+\t\t# Remove not only the leading 't', but all leading zeros\n+\t\t# as well, so the arithmetic below won't (mis)interpret\n+\t\t# a test number like '0123' as an octal value.\n+\t\tport=${this_test#${this_test%%[1-9]*}}\n+\t\tif test \"${port:-0}\" -lt 1024\n+\t\tthen\n+\t\t\t# root-only port, use a larger one instead.\n+\t\t\tport=$(($port + 10000))\n+\t\tfi\n+\n+\t\teval $var=$port\n+\t\t;;\n+\t*[^0-9]*|0*)\n+\t\terror >&7 \"invalid port number: $port\"\n+\t\t;;\n+\t*)\n+\t\t# The user has specified the port.\n+\t\t;;\n+\tesac\n+}\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"365992","messageId":"20181230191629.3232-6-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-1-szeder.dev@gmail.com","subject":"[PATCH v3 5/8] test-lib: set $TRASH_DIRECTORY earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:16:26Z","receivedAt":"2018-12-30T19:17:05Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"A later patch in this series will need to know the path to the trash\ndirectory early in 'test-lib.sh', but $TRASH_DIRECTORY is set much\nlater. Furthermore, the path to the trash directory depends on the\n'--root=' option, which, too, is parsed too late.\n\nMove parsing '--root=...' to the early option parsing loop, and set\n$TRASH_DIRECTORY where the other test-specific path variables are set.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 41457d1dcf..2b88ba2de1 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -186,6 +186,12 @@ fi\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n+test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n+case \"$TRASH_DIRECTORY\" in\n+/*) ;; # absolute path is good\n+ *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n+esac\n \n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n@@ -1046,12 +1052,6 @@ then\n fi\n \n # Test repository\n-TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n-test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n-case \"$TRASH_DIRECTORY\" in\n-/*) ;; # absolute path is good\n- *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n-esac\n rm -fr \"$TRASH_DIRECTORY\" || {\n \tGIT_EXIT_OK=t\n \techo >&5 \"FATAL: Cannot prepare test area\"\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"365993","messageId":"20181230191629.3232-5-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-1-szeder.dev@gmail.com","subject":"[PATCH v3 4/8] test-lib: consolidate naming of test-results paths","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:16:25Z","receivedAt":"2018-12-30T19:17:07Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"There are two places where we strip off any leading path components\nand the '.sh' suffix from the test script's pathname, and there are\nfour places where we construct the name of the 't/test-results'\ndirectory or the name of various test-specific files in there. The\nlast patch in this series will add even more.\n\nFactor these out into helper variables to avoid repeating ourselves.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex fcc04afdff..41457d1dcf 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -183,6 +183,10 @@ then\n \tverbose=t\n fi\n \n+TEST_NAME=\"$(basename \"$0\" .sh)\"\n+TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n+TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n@@ -190,12 +194,11 @@ then\n \t: # do not redirect again\n elif test -n \"$tee\"\n then\n-\tmkdir -p \"$TEST_OUTPUT_DIRECTORY/test-results\"\n-\tBASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$(basename \"$0\" .sh)\"\n+\tmkdir -p \"$TEST_RESULTS_DIR\"\n \n \t# Make this filename available to the sub-process in case it is using\n \t# --verbose-log.\n-\tGIT_TEST_TEE_OUTPUT_FILE=$BASE.out\n+\tGIT_TEST_TEE_OUTPUT_FILE=$TEST_RESULTS_BASE.out\n \texport GIT_TEST_TEE_OUTPUT_FILE\n \n \t# Truncate before calling \"tee -a\" to get rid of the results\n@@ -203,8 +206,8 @@ then\n \t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \n \t(GIT_TEST_TEE_STARTED=done ${TEST_SHELL_PATH} \"$0\" \"$@\" 2>&1;\n-\t echo $? >\"$BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n-\ttest \"$(cat \"$BASE.exit\")\" = 0\n+\t echo $? >\"$TEST_RESULTS_BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\ttest \"$(cat \"$TEST_RESULTS_BASE.exit\")\" = 0\n \texit\n fi\n \n@@ -835,12 +838,9 @@ test_done () {\n \n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n-\t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n-\t\tmkdir -p \"$test_results_dir\"\n-\t\tbase=${0##*/}\n-\t\ttest_results_path=\"$test_results_dir/${base%.sh}.counts\"\n+\t\tmkdir -p \"$TEST_RESULTS_DIR\"\n \n-\t\tcat >\"$test_results_path\" <<-EOF\n+\t\tcat >\"$TEST_RESULTS_BASE.counts\" <<-EOF\n \t\ttotal $test_count\n \t\tsuccess $test_success\n \t\tfixed $test_fixed\n@@ -1046,7 +1046,7 @@ then\n fi\n \n # Test repository\n-TRASH_DIRECTORY=\"trash directory.$(basename \"$0\" .sh)\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n case \"$TRASH_DIRECTORY\" in\n /*) ;; # absolute path is good\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"365994","messageId":"20181230191629.3232-9-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-1-szeder.dev@gmail.com","subject":"[PATCH v3 8/8] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T19:16:29Z","receivedAt":"2018-12-30T19:17:07Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Unfortunately, we have a few flaky tests, whose failures tend to be\nhard to reproduce. We've found that the best we can do to reproduce\nsuch a failure is to run the test script repeatedly while the machine\nis under load, and wait in the hope that the load creates enough\nvariance in the timing of the test's commands that a failure is\nevenually triggered. I have a command to do that, and I noticed that\ntwo other contributors have rolled their own scripts to do the same,\nall choosing slightly different approaches.\n\nTo help reproduce failures in flaky tests, introduce the '--stress'\noption to run a test script repeatedly in multiple parallel jobs until\none of them fails, thereby using the test script itself to increase\nthe load on the machine.\n\nThe number of parallel jobs is determined by, in order of precedence:\nthe number specified as '--stress=', or the value of the\nGIT_TEST_STRESS_LOAD environment variable, or twice the number of\navailable processors (as reported by the 'getconf' utility), or 8.\n\nMake '--stress' imply '--verbose -x --immediate' to get the most\ninformation about rare failures; there is really no point in spending\nall the extra effort to reproduce such a failure, and then not know\nwhich command failed and why.\n\nTo prevent the several parallel invocations of the same test from\ninterfering with each other:\n\n - Include the parallel job's number in the name of the trash\n directory and the various output files under 't/test-results/' as\n a '.stress-' suffix.\n\n - Add the parallel job's number to the port number specified by the\n user or to the test number, so even tests involving daemons\n listening on a TCP socket can be stressed.\n\n - Redirect each parallel test run's verbose output to\n 't/test-results/$TEST_NAME.stress-.out', because dumping the\n output of several parallel running tests to the terminal would\n create a big ugly mess.\n\nFor convenience, print the output of the failed test job at the end,\nand rename its trash directory to end with the '.stress-failed'\nsuffix, so it's easy to find in a predictable path (OTOH, all absolute\npaths recorded in the trash directory become invalid; we'll see\nwhether this causes any issues in practice). If, in an unlikely case,\nmore than one jobs were to fail nearly at the same time, then print\nthe output of all failed jobs, and rename the trash directory of only\nthe last one (i.e. with the highest job number), as it is the trash\ndirectory of the test whose output will be at the bottom of the user's\nterminal.\n\nBased on Jeff King's 'stress' script.\n\nSigned-off-by: SZEDER Gábor \n---\n t/README | 19 ++++++-\n t/test-lib-functions.sh | 7 ++-\n t/test-lib.sh | 109 +++++++++++++++++++++++++++++++++++++++-\n 3 files changed, 130 insertions(+), 5 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 28711cc508..11ce7675e3 100644\n--- a/t/README\n+++ b/t/README\n@@ -186,6 +186,22 @@ appropriately before running \"make\".\n \tthis feature by setting the GIT_TEST_CHAIN_LINT environment\n \tvariable to \"1\" or \"0\", respectively.\n \n+--stress::\n+--stress=::\n+\tRun the test script repeatedly in multiple parallel jobs until\n+\tone of them fails. Useful for reproducing rare failures in\n+\tflaky tests. The number of parallel jobs is, in order of\n+\tprecedence: , or the value of the GIT_TEST_STRESS_LOAD\n+\tenvironment variable, or twice the number of available\n+\tprocessors (as shown by the 'getconf' utility),\tor 8.\n+\tImplies `--verbose -x --immediate` to get the most information\n+\tabout the failure. Note that the verbose output of each test\n+\tjob is saved to 't/test-results/$TEST_NAME.stress-.out',\n+\tand only the output of the failed test job is shown on the\n+\tterminal. The names of the trash directories get a\n+\t'.stress-' suffix, and the trash directory of the failed\n+\ttest job is renamed to end with a '.stress-failed' suffix.\n+\n You can also set the GIT_TEST_INSTALLED environment variable to\n the bindir of an existing git installation to test that installation.\n You still need to have built this git sandbox, from which various\n@@ -425,7 +441,8 @@ This test harness library does the following things:\n - Creates an empty test directory with an empty .git/objects database\n and chdir(2) into it. This directory is 't/trash\n directory.$test_name_without_dotsh', with t/ subject to change by\n- the --root option documented above.\n+ the --root option documented above, and a '.stress-' suffix\n+ appended by the --stress option.\n \n - Defines standard test helper functions for your scripts to\n use. These functions are designed to make all scripts behave\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 4459bdda13..92cf8f812c 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1288,8 +1288,6 @@ test_set_port () {\n \t\t\t# root-only port, use a larger one instead.\n \t\t\tport=$(($port + 10000))\n \t\tfi\n-\n-\t\teval $var=$port\n \t\t;;\n \t*[^0-9]*|0*)\n \t\terror >&7 \"invalid port number: $port\"\n@@ -1298,4 +1296,9 @@ test_set_port () {\n \t\t# The user has specified the port.\n \t\t;;\n \tesac\n+\n+\t# Make sure that parallel '--stress' test jobs get different\n+\t# ports.\n+\tport=$(($port + ${GIT_TEST_STRESS_JOB_NR:-0}))\n+\teval $var=$port\n }\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 7d77a26d44..16f74bd524 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -139,6 +139,19 @@ do\n \t\tverbose_log=t\n \t\ttee=t\n \t\t;;\n+\t--stress)\n+\t\tstress=t ;;\n+\t--stress=*)\n+\t\tstress=${opt#--*=}\n+\t\tcase \"$stress\" in\n+\t\t*[^0-9]*|0*|\"\")\n+\t\t\techo \"error: --stress= requires the number of jobs to run\" >&2\n+\t\t\texit 1\n+\t\t\t;;\n+\t\t*)\t# Good.\n+\t\t\t;;\n+\t\tesac\n+\t\t;;\n \t*)\n \t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n \tesac\n@@ -160,6 +173,13 @@ then\n \ttest -z \"$verbose_log\" && verbose=t\n fi\n \n+if test -n \"$stress\"\n+then\n+\tverbose=t\n+\ttrace=t\n+\timmediate=t\n+fi\n+\n if test -n \"$trace\" && test -n \"$test_untraceable\"\n then\n \t# '-x' tracing requested, but this test script can't be reliably\n@@ -183,16 +203,101 @@ then\n \tverbose=t\n fi\n \n+TEST_STRESS_JOB_SFX=\"${GIT_TEST_STRESS_JOB_NR:+.stress-$GIT_TEST_STRESS_JOB_NR}\"\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n-TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n-TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n+TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME$TEST_STRESS_JOB_SFX\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME$TEST_STRESS_JOB_SFX\"\n test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n case \"$TRASH_DIRECTORY\" in\n /*) ;; # absolute path is good\n *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n esac\n \n+# If --stress was passed, run this test repeatedly in several parallel loops.\n+if test \"$GIT_TEST_STRESS_STARTED\" = \"done\"\n+then\n+\t: # Don't stress test again.\n+elif test -n \"$stress\"\n+then\n+\tif test \"$stress\" != t\n+\tthen\n+\t\tjob_count=$stress\n+\telif test -n \"$GIT_TEST_STRESS_LOAD\"\n+\tthen\n+\t\tjob_count=\"$GIT_TEST_STRESS_LOAD\"\n+\telif job_count=$(getconf _NPROCESSORS_ONLN 2>/dev/null) &&\n+\t test -n \"$job_count\"\n+\tthen\n+\t\tjob_count=$((2 * $job_count))\n+\telse\n+\t\tjob_count=8\n+\tfi\n+\n+\tmkdir -p \"$TEST_RESULTS_DIR\"\n+\tstressfail=\"$TEST_RESULTS_BASE.stress-failed\"\n+\trm -f \"$stressfail\"\n+\n+\tstress_exit=0\n+\ttrap '\n+\t\tkill $job_pids 2>/dev/null\n+\t\twait\n+\t\tstress_exit=1\n+\t' TERM INT HUP\n+\n+\tjob_pids=\n+\tjob_nr=0\n+\twhile test $job_nr -lt \"$job_count\"\n+\tdo\n+\t\t(\n+\t\t\tGIT_TEST_STRESS_STARTED=done\n+\t\t\tGIT_TEST_STRESS_JOB_NR=$job_nr\n+\t\t\texport GIT_TEST_STRESS_STARTED GIT_TEST_STRESS_JOB_NR\n+\n+\t\t\ttrap '\n+\t\t\t\tkill $test_pid 2>/dev/null\n+\t\t\t\twait\n+\t\t\t\texit 1\n+\t\t\t' TERM INT\n+\n+\t\t\tcnt=0\n+\t\t\twhile ! test -e \"$stressfail\"\n+\t\t\tdo\n+\t\t\t\t$TEST_SHELL_PATH \"$0\" \"$@\" >\"$TEST_RESULTS_BASE.stress-$job_nr.out\" 2>&1 &\n+\t\t\t\ttest_pid=$!\n+\n+\t\t\t\tif wait $test_pid\n+\t\t\t\tthen\n+\t\t\t\t\tprintf \"OK %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n+\t\t\t\telse\n+\t\t\t\t\techo $GIT_TEST_STRESS_JOB_NR >>\"$stressfail\"\n+\t\t\t\t\tprintf \"FAIL %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n+\t\t\t\tfi\n+\t\t\t\tcnt=$(($cnt + 1))\n+\t\t\tdone\n+\t\t) &\n+\t\tjob_pids=\"$job_pids $!\"\n+\t\tjob_nr=$(($job_nr + 1))\n+\tdone\n+\n+\twait\n+\n+\tif test -f \"$stressfail\"\n+\tthen\n+\t\techo \"Log(s) of failed test run(s):\"\n+\t\tfor failed_job_nr in $(sort -n \"$stressfail\")\n+\t\tdo\n+\t\t\techo \"Contents of '$TEST_RESULTS_BASE.stress-$failed_job_nr.out':\"\n+\t\t\tcat \"$TEST_RESULTS_BASE.stress-$failed_job_nr.out\"\n+\t\tdone\n+\t\trm -rf \"$TRASH_DIRECTORY.stress-failed\"\n+\t\t# Move the last one.\n+\t\tmv \"$TRASH_DIRECTORY.stress-$failed_job_nr\" \"$TRASH_DIRECTORY.stress-failed\"\n+\tfi\n+\n+\texit $stress_exit\n+fi\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"365997","messageId":"20181230224453.GC6120@szeder.dev","threadId":"49946","inReplyTo":"20181230191629.3232-6-szeder.dev@gmail.com","subject":"Re: [PATCH v3 5/8] test-lib: set $TRASH_DIRECTORY earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T22:44:53Z","receivedAt":"2018-12-30T22:45:01Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Sun, Dec 30, 2018 at 08:16:26PM +0100, SZEDER Gábor wrote:\n> A later patch in this series will need to know the path to the trash\n> directory early in 'test-lib.sh', but $TRASH_DIRECTORY is set much\n> later. Furthermore, the path to the trash directory depends on the\n> '--root=' option, which, too, is parsed too late.\n> \n> Move parsing '--root=...' to the early option parsing loop, and set\n> $TRASH_DIRECTORY where the other test-specific path variables are set.\n\nSigh.\nAfter moving the whole option parsing loop, I should have updated this\ncommit message as well.\n\nHang on for a sec...\n\n\n> Signed-off-by: SZEDER Gábor \n> ---\n> t/test-lib.sh | 12 ++++++------\n> 1 file changed, 6 insertions(+), 6 deletions(-)\n> \n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 41457d1dcf..2b88ba2de1 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -186,6 +186,12 @@ fi\n> TEST_NAME=\"$(basename \"$0\" .sh)\"\n> TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n> TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n> +TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n> +test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n> +case \"$TRASH_DIRECTORY\" in\n> +/*) ;; # absolute path is good\n> + *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n> +esac\n> \n> # if --tee was passed, write the output not only to the terminal, but\n> # additionally to the file test-results/$BASENAME.out, too.\n> @@ -1046,12 +1052,6 @@ then\n> fi\n> \n> # Test repository\n> -TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n> -test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n> -case \"$TRASH_DIRECTORY\" in\n> -/*) ;; # absolute path is good\n> - *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n> -esac\n> rm -fr \"$TRASH_DIRECTORY\" || {\n> \tGIT_EXIT_OK=t\n> \techo >&5 \"FATAL: Cannot prepare test area\"\n> -- \n> 2.20.1.151.gec613c4b75\n> \n"},{"id":"365998","messageId":"20181230224822.18762-1-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230224453.GC6120@szeder.dev","subject":"[PATCH v3.1 5/8] test-lib: set $TRASH_DIRECTORY earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-12-30T22:48:22Z","receivedAt":"2018-12-30T22:48:30Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"A later patch in this series will need to know the path to the trash\ndirectory early in 'test-lib.sh', but $TRASH_DIRECTORY is set much\nlater.\n\nSet $TRASH_DIRECTORY earlier, where the other test-specific path\nvariables are set.\n\nSigned-off-by: SZEDER Gábor \n---\n\nUpdated the commit message, the diff is the same.\n\n t/test-lib.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 41457d1dcf..2b88ba2de1 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -186,6 +186,12 @@ fi\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n+test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n+case \"$TRASH_DIRECTORY\" in\n+/*) ;; # absolute path is good\n+ *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n+esac\n \n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n@@ -1046,12 +1052,6 @@ then\n fi\n \n # Test repository\n-TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n-test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n-case \"$TRASH_DIRECTORY\" in\n-/*) ;; # absolute path is good\n- *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n-esac\n rm -fr \"$TRASH_DIRECTORY\" || {\n \tGIT_EXIT_OK=t\n \techo >&5 \"FATAL: Cannot prepare test area\"\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366012","messageId":"CAPUEspirwDtbrnj2cNyEO8TuoRp_4ZhjkT0TRb_QgEO-Db8H-w@mail.gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-7-szeder.dev@gmail.com","subject":"Re: [PATCH v3 6/8] test-lib: extract Bash version check for '-x' tracing","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2018-12-31T17:14:22Z","receivedAt":"2018-12-31T17:14:36Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"slightly offtopic and definitely not meant to be relevant as a critic\non this specific change which I found otherwise fine, but just a heads\nup that this test itself is bash specific (since it uses arrays,\nagainst the recommendation in Documentation/CodingGuidelines) and\ntherefore breaks, at least in NetBSD as reported in :\n\n https://public-inbox.org/git/20181121224929.GD5348@jessie.local/\n\nCarlo\n"},{"id":"366064","messageId":"20190103045359.GC20047@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20181230190419.GB6120@szeder.dev","subject":"Re: [PATCH v2 2/7] test-lib: parse some --options earlier","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-01-03T04:53:59Z","receivedAt":"2019-01-03T04:54:02Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Dec 30, 2018 at 08:04:19PM +0100, SZEDER Gábor wrote:\n\n> > (in fact, given that this is just\n> > the internal tests, I am tempted to say that we should just make it\n> > \"-r\" for the sake of simplicity and consistency. But maybe somebody\n> > would be annoyed. I have never used \"-r\" ever myself).\n> \n> I didn't even know what '-r' does...\n\nI had to look it up, too. :)\n\n> And I agree that changing it to '-r' would be the best, but this\n> patch series is about adding '--stress', so changing how '-r' gets its\n> mandatory argument (and potentially annoying someone) is beyond the\n> scope, I would say.\n\nOK, I'm fine with that (though once we've built the infrastructure to\nhandle its unstuck form, I don't know if there's much point in changing\nit, so we can probably just let it live on forever).\n\n-Peff\n"},{"id":"366162","messageId":"20190105010859.11031-1-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20181230191629.3232-1-szeder.dev@gmail.com","subject":"[PATCH v4 0/8] test-lib: add the '--stress' option to help reproduce occasional failures in flaky tests","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-05T01:08:51Z","receivedAt":"2019-01-05T01:09:17Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"To recap: this patch series tries to make reproducing rare failures in\nflaky tests easier: it adds the '--stress' option to our test library\nto run the test script repeatedly in multiple parallel jobs, in the\nhope that the increased load creates enough variance in the timing of\nthe test's commands that such a failure is eventually triggered.\n\nThere is an issue with v3: by parsing all command line options before\nre-executing the test script for '--tee' or '--verbose-log', we not\nonly parse the options twice (which is harmless), but check the\nshell's version for untraceable test scripts twice as well.\nConsequently, the warning about ignoring '-x' are issued twice:\n\n $ ./t1510-repo-setup.sh -x --tee\n warning: ignoring -x; './t1510-repo-setup.sh' is untraceable without BASH_XTRACEFD\n warning: ignoring -x; './t1510-repo-setup.sh' is untraceable without BASH_XTRACEFD\n ok 1 - #0: nonbare repo, no explicit configuration\n <...>\n\nFurthermore, when TEST_SHELL_PATH is not /bin/sh but is set to Bash\nv4.1 or later, then the above command issues one warning as it's run\nwith /bin/sh from the shebang line, but then re-execs itself with Bash\nand does print the '-x' tracing output as well.\n\nThis version of the patch series fixes the above issue by checking the\nshell's version after '--tee' and '--verbose-log' re-execue the test\nscript. To do so it makes more sense to move the 'test-lib: extract\nBash version check for '-x' tracing' patch to the beginning of the\nseries. This move, however, made the range-diff quite noisy, hence\nthe interdiff below, as it describes the changes since v3 much better.\n\nIt's based on 'sg/test-bash-version-fix'.\n\n\nSZEDER Gábor (8):\n test-lib: translate SIGTERM and SIGHUP to an exit\n test-lib: extract Bash version check for '-x' tracing\n test-lib: parse options in a for loop to keep $@ intact\n test-lib: parse command line options earlier\n test-lib: consolidate naming of test-results paths\n test-lib: set $TRASH_DIRECTORY earlier\n test-lib-functions: introduce the 'test_set_port' helper function\n test-lib: add the '--stress' option to run a test repeatedly under\n load\n\n t/README | 19 +-\n t/lib-git-daemon.sh | 2 +-\n t/lib-git-p4.sh | 9 +-\n t/lib-git-svn.sh | 2 +-\n t/lib-httpd.sh | 2 +-\n t/t0410-partial-clone.sh | 1 -\n t/t5512-ls-remote.sh | 2 +-\n t/test-lib-functions.sh | 39 +++++\n t/test-lib.sh | 365 ++++++++++++++++++++++++++-------------\n 9 files changed, 308 insertions(+), 133 deletions(-)\n\nInterdiff:\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 3f0b0c07df..a1abb1177a 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -180,29 +180,6 @@ then\n \timmediate=t\n fi\n \n-if test -n \"$trace\" && test -n \"$test_untraceable\"\n-then\n-\t# '-x' tracing requested, but this test script can't be reliably\n-\t# traced, unless it is run with a Bash version supporting\n-\t# BASH_XTRACEFD (introduced in Bash v4.1).\n-\tif test -n \"$BASH_VERSION\" && eval '\n-\t test ${BASH_VERSINFO[0]} -gt 4 || {\n-\t test ${BASH_VERSINFO[0]} -eq 4 &&\n-\t test ${BASH_VERSINFO[1]} -ge 1\n-\t }\n-\t '\n-\tthen\n-\t\t: Executed by a Bash version supporting BASH_XTRACEFD. Good.\n-\telse\n-\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n-\t\ttrace=\n-\tfi\n-fi\n-if test -n \"$trace\" && test -z \"$verbose_log\"\n-then\n-\tverbose=t\n-fi\n-\n TEST_STRESS_JOB_SFX=\"${GIT_TEST_STRESS_JOB_NR:+.stress-$GIT_TEST_STRESS_JOB_NR}\"\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n@@ -322,6 +299,34 @@ then\n \texit\n fi\n \n+if test -n \"$trace\" && test -n \"$test_untraceable\"\n+then\n+\t# '-x' tracing requested, but this test script can't be reliably\n+\t# traced, unless it is run with a Bash version supporting\n+\t# BASH_XTRACEFD (introduced in Bash v4.1).\n+\t#\n+\t# Perform this version check _after_ the test script was\n+\t# potentially re-executed with $TEST_SHELL_PATH for '--tee' or\n+\t# '--verbose-log', so the right shell is checked and the\n+\t# warning is issued only once.\n+\tif test -n \"$BASH_VERSION\" && eval '\n+\t test ${BASH_VERSINFO[0]} -gt 4 || {\n+\t test ${BASH_VERSINFO[0]} -eq 4 &&\n+\t test ${BASH_VERSINFO[1]} -ge 1\n+\t }\n+\t '\n+\tthen\n+\t\t: Executed by a Bash version supporting BASH_XTRACEFD. Good.\n+\telse\n+\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n+\t\ttrace=\n+\tfi\n+fi\n+if test -n \"$trace\" && test -z \"$verbose_log\"\n+then\n+\tverbose=t\n+fi\n+\n # For repeatability, reset the environment to known value.\n # TERM is sanitized below, after saving color control sequences.\n LANG=C\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366163","messageId":"20190105010859.11031-2-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20190105010859.11031-1-szeder.dev@gmail.com","subject":"[PATCH v4 1/8] test-lib: translate SIGTERM and SIGHUP to an exit","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-05T01:08:52Z","receivedAt":"2019-01-05T01:09:18Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Right now if a test script receives SIGTERM or SIGHUP (e.g., because a\ntest was hanging and the user 'kill'-ed it or simply closed the\nterminal window the test was running in), the shell exits immediately.\nThis can be annoying if the test script did any global setup, like\nstarting apache or git-daemon, as it will not have an opportunity to\nclean up after itself. A subsequent run of the test won't be able to\nstart its own daemon, and will either fail or skip the tests.\n\nInstead, let's trap SIGTERM and SIGHUP as well to make sure we do a\nclean shutdown, and just chain it to a normal exit (which will trigger\nany cleanup).\n\nThis patch follows suit of da706545f7 (t: translate SIGINT to an exit,\n2015-03-13), and even stole its commit message as well.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex c34831a4de..4c3744cce4 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -476,7 +476,7 @@ die () {\n \n GIT_EXIT_OK=\n trap 'die' EXIT\n-trap 'exit $?' INT\n+trap 'exit $?' INT TERM HUP\n \n # The user-facing functions are loaded from a separate file so that\n # test_perf subshells can have them too\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366164","messageId":"20190105010859.11031-3-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20190105010859.11031-1-szeder.dev@gmail.com","subject":"[PATCH v4 2/8] test-lib: extract Bash version check for '-x' tracing","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-05T01:08:53Z","receivedAt":"2019-01-05T01:09:19Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"One of our test scripts, 't1510-repo-setup.sh' [1], still can't be\nreliably run with '-x' tracing enabled, unless it's executed with a\nBash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy\ncondition to check the version of the shell running the test script,\nand disable tracing if it's not executed with a suitable Bash version\n[2].\n\nMove this check out from the option parsing loop, so other options can\nimply '-x' by setting 'trace=t', without missing this Bash version\ncheck.\n\n[1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x',\n 2018-02-24)\n[2] 5fc98e79fc (t: add means to disable '-x' tracing for individual\n test scripts, 2018-02-24)\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 37 +++++++++++++++++++------------------\n 1 file changed, 19 insertions(+), 18 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 4c3744cce4..1f02e2e25b 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -317,24 +317,7 @@ do\n \t\tGIT_TEST_CHAIN_LINT=0\n \t\tshift ;;\n \t-x)\n-\t\t# Some test scripts can't be reliably traced with '-x',\n-\t\t# unless the test is run with a Bash version supporting\n-\t\t# BASH_XTRACEFD (introduced in Bash v4.1). Check whether\n-\t\t# this test is marked as such, and ignore '-x' if it\n-\t\t# isn't executed with a suitable Bash version.\n-\t\tif test -z \"$test_untraceable\" || {\n-\t\t test -n \"$BASH_VERSION\" && eval '\n-\t\t test ${BASH_VERSINFO[0]} -gt 4 || {\n-\t\t\t test ${BASH_VERSINFO[0]} -eq 4 &&\n-\t\t\t test ${BASH_VERSINFO[1]} -ge 1\n-\t\t }\n-\t\t '\n-\t\t }\n-\t\tthen\n-\t\t\ttrace=t\n-\t\telse\n-\t\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n-\t\tfi\n+\t\ttrace=t\n \t\tshift ;;\n \t-V|--verbose-log)\n \t\tverbose_log=t\n@@ -353,6 +336,24 @@ then\n \ttest -z \"$verbose_log\" && verbose=t\n fi\n \n+if test -n \"$trace\" && test -n \"$test_untraceable\"\n+then\n+\t# '-x' tracing requested, but this test script can't be reliably\n+\t# traced, unless it is run with a Bash version supporting\n+\t# BASH_XTRACEFD (introduced in Bash v4.1).\n+\tif test -n \"$BASH_VERSION\" && eval '\n+\t test ${BASH_VERSINFO[0]} -gt 4 || {\n+\t test ${BASH_VERSINFO[0]} -eq 4 &&\n+\t test ${BASH_VERSINFO[1]} -ge 1\n+\t }\n+\t '\n+\tthen\n+\t\t: Executed by a Bash version supporting BASH_XTRACEFD. Good.\n+\telse\n+\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n+\t\ttrace=\n+\tfi\n+fi\n if test -n \"$trace\" && test -z \"$verbose_log\"\n then\n \tverbose=t\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366165","messageId":"20190105010859.11031-4-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20190105010859.11031-1-szeder.dev@gmail.com","subject":"[PATCH v4 3/8] test-lib: parse options in a for loop to keep $@ intact","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-05T01:08:54Z","receivedAt":"2019-01-05T01:09:22Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"'test-lib.sh' looks for the presence of certain options like '--tee'\nand '--verbose-log', so it can execute the test script again to save\nits standard output and error, and to do so it needs the original\ncommand line options the test was invoked with.\n\nThe next patch is about to move the option parsing loop earlier in\n'test-lib.sh', but it is implemented using 'shift' in a while loop,\neffecively destroying \"$@\" by the end of the option parsing. Not\ngood.\n\nAs a preparatory step, turn that option parsing loop into a 'for opt\nin \"$@\"' loop to preserve \"$@\" intact while iterating over the\noptions, and taking extra care to handle the '-r' option's required\nargument (or the lack thereof).\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 78 +++++++++++++++++++++++++++------------------------\n 1 file changed, 42 insertions(+), 36 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 1f02e2e25b..3cf59a92f0 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -264,68 +264,74 @@ test \"x$TERM\" != \"xdumb\" && (\n \t) &&\n \tcolor=t\n \n-while test \"$#\" -ne 0\n+store_arg_to=\n+prev_opt=\n+for opt\n do\n-\tcase \"$1\" in\n+\tif test -n \"$store_arg_to\"\n+\tthen\n+\t\teval $store_arg_to=\\$opt\n+\t\tstore_arg_to=\n+\t\tprev_opt=\n+\t\tcontinue\n+\tfi\n+\n+\tcase \"$opt\" in\n \t-d|--d|--de|--deb|--debu|--debug)\n-\t\tdebug=t; shift ;;\n+\t\tdebug=t ;;\n \t-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)\n-\t\timmediate=t; shift ;;\n+\t\timmediate=t ;;\n \t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n-\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG; shift ;;\n+\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG ;;\n \t-r)\n-\t\tshift; test \"$#\" -ne 0 || {\n-\t\t\techo 'error: -r requires an argument' >&2;\n-\t\t\texit 1;\n-\t\t}\n-\t\trun_list=$1; shift ;;\n+\t\tstore_arg_to=run_list\n+\t\t;;\n \t--run=*)\n-\t\trun_list=${1#--*=}; shift ;;\n+\t\trun_list=${opt#--*=} ;;\n \t-h|--h|--he|--hel|--help)\n-\t\thelp=t; shift ;;\n+\t\thelp=t ;;\n \t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n-\t\tverbose=t; shift ;;\n+\t\tverbose=t ;;\n \t--verbose-only=*)\n-\t\tverbose_only=${1#--*=}\n-\t\tshift ;;\n+\t\tverbose_only=${opt#--*=}\n+\t\t;;\n \t-q|--q|--qu|--qui|--quie|--quiet)\n \t\t# Ignore --quiet under a TAP::Harness. Saying how many tests\n \t\t# passed without the ok/not ok details is always an error.\n-\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t; shift ;;\n+\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n \t--with-dashes)\n-\t\twith_dashes=t; shift ;;\n+\t\twith_dashes=t ;;\n \t--no-color)\n-\t\tcolor=; shift ;;\n+\t\tcolor= ;;\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n-\t\tvalgrind=memcheck\n-\t\tshift ;;\n+\t\tvalgrind=memcheck ;;\n \t--valgrind=*)\n-\t\tvalgrind=${1#--*=}\n-\t\tshift ;;\n+\t\tvalgrind=${opt#--*=} ;;\n \t--valgrind-only=*)\n-\t\tvalgrind_only=${1#--*=}\n-\t\tshift ;;\n+\t\tvalgrind_only=${opt#--*=} ;;\n \t--tee)\n-\t\tshift ;; # was handled already\n+\t\t;; # was handled already\n \t--root=*)\n-\t\troot=${1#--*=}\n-\t\tshift ;;\n+\t\troot=${opt#--*=} ;;\n \t--chain-lint)\n-\t\tGIT_TEST_CHAIN_LINT=1\n-\t\tshift ;;\n+\t\tGIT_TEST_CHAIN_LINT=1 ;;\n \t--no-chain-lint)\n-\t\tGIT_TEST_CHAIN_LINT=0\n-\t\tshift ;;\n+\t\tGIT_TEST_CHAIN_LINT=0 ;;\n \t-x)\n-\t\ttrace=t\n-\t\tshift ;;\n+\t\ttrace=t ;;\n \t-V|--verbose-log)\n-\t\tverbose_log=t\n-\t\tshift ;;\n+\t\tverbose_log=t ;;\n \t*)\n-\t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n+\t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n \tesac\n+\n+\tprev_opt=$opt\n done\n+if test -n \"$store_arg_to\"\n+then\n+\techo \"error: $prev_opt requires an argument\" >&2\n+\texit 1\n+fi\n \n if test -n \"$valgrind_only\"\n then\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366166","messageId":"20190105010859.11031-5-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20190105010859.11031-1-szeder.dev@gmail.com","subject":"[PATCH v4 4/8] test-lib: parse command line options earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-05T01:08:55Z","receivedAt":"2019-01-05T01:09:24Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"'test-lib.sh' looks for the presence of certain options like '--tee'\nand '--verbose-log', so it can execute the test script again to save\nits standard output and error. It looks for '--valgrind' as well, to\nset up some Valgrind-specific stuff. These all happen before the\nactual option parsing loop, and the conditions looking for these\noptions look a bit odd, too. They are not completely correct, either,\nbecause in a bogus invocation like './t1234-foo.sh -r --tee' they\nrecognize '--tee', although it should be handled as the required\nargument of the '-r' option. This patch series will add two more\noptions to look out for early, and, in addition, will have to extract\nthese options' stuck arguments (i.e. '--opt=arg') as well.\n\nSo let's move the option parsing loop and the couple of related\nconditions following it earlier in 'test-lib.sh', before the place\nwhere the test script is executed again for '--tee' and its friends.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 233 +++++++++++++++++++++++++++-----------------------\n 1 file changed, 124 insertions(+), 109 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 3cf59a92f0..14ccb60838 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -71,13 +71,102 @@ then\n \texit 1\n fi\n \n+# Parse options while taking care to leave $@ intact, so we will still\n+# have all the original command line options when executing the test\n+# script again for '--tee' and '--verbose-log' below.\n+store_arg_to=\n+prev_opt=\n+for opt\n+do\n+\tif test -n \"$store_arg_to\"\n+\tthen\n+\t\teval $store_arg_to=\\$opt\n+\t\tstore_arg_to=\n+\t\tprev_opt=\n+\t\tcontinue\n+\tfi\n+\n+\tcase \"$opt\" in\n+\t-d|--d|--de|--deb|--debu|--debug)\n+\t\tdebug=t ;;\n+\t-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)\n+\t\timmediate=t ;;\n+\t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n+\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG ;;\n+\t-r)\n+\t\tstore_arg_to=run_list\n+\t\t;;\n+\t--run=*)\n+\t\trun_list=${opt#--*=} ;;\n+\t-h|--h|--he|--hel|--help)\n+\t\thelp=t ;;\n+\t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n+\t\tverbose=t ;;\n+\t--verbose-only=*)\n+\t\tverbose_only=${opt#--*=}\n+\t\t;;\n+\t-q|--q|--qu|--qui|--quie|--quiet)\n+\t\t# Ignore --quiet under a TAP::Harness. Saying how many tests\n+\t\t# passed without the ok/not ok details is always an error.\n+\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n+\t--with-dashes)\n+\t\twith_dashes=t ;;\n+\t--no-color)\n+\t\tcolor= ;;\n+\t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n+\t\tvalgrind=memcheck\n+\t\ttee=t\n+\t\t;;\n+\t--valgrind=*)\n+\t\tvalgrind=${opt#--*=}\n+\t\ttee=t\n+\t\t;;\n+\t--valgrind-only=*)\n+\t\tvalgrind_only=${opt#--*=}\n+\t\ttee=t\n+\t\t;;\n+\t--tee)\n+\t\ttee=t ;;\n+\t--root=*)\n+\t\troot=${opt#--*=} ;;\n+\t--chain-lint)\n+\t\tGIT_TEST_CHAIN_LINT=1 ;;\n+\t--no-chain-lint)\n+\t\tGIT_TEST_CHAIN_LINT=0 ;;\n+\t-x)\n+\t\ttrace=t ;;\n+\t-V|--verbose-log)\n+\t\tverbose_log=t\n+\t\ttee=t\n+\t\t;;\n+\t*)\n+\t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n+\tesac\n+\n+\tprev_opt=$opt\n+done\n+if test -n \"$store_arg_to\"\n+then\n+\techo \"error: $prev_opt requires an argument\" >&2\n+\texit 1\n+fi\n+\n+if test -n \"$valgrind_only\"\n+then\n+\ttest -z \"$valgrind\" && valgrind=memcheck\n+\ttest -z \"$verbose\" && verbose_only=\"$valgrind_only\"\n+elif test -n \"$valgrind\"\n+then\n+\ttest -z \"$verbose_log\" && verbose=t\n+fi\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n-case \"$GIT_TEST_TEE_STARTED, $* \" in\n-done,*)\n-\t# do not redirect again\n-\t;;\n-*' --tee '*|*' --va'*|*' -V '*|*' --verbose-log '*)\n+if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n+then\n+\t: # do not redirect again\n+elif test -n \"$tee\"\n+then\n \tmkdir -p \"$TEST_OUTPUT_DIRECTORY/test-results\"\n \tBASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$(basename \"$0\" .sh)\"\n \n@@ -94,8 +183,35 @@ done,*)\n \t echo $? >\"$BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n \ttest \"$(cat \"$BASE.exit\")\" = 0\n \texit\n-\t;;\n-esac\n+fi\n+\n+if test -n \"$trace\" && test -n \"$test_untraceable\"\n+then\n+\t# '-x' tracing requested, but this test script can't be reliably\n+\t# traced, unless it is run with a Bash version supporting\n+\t# BASH_XTRACEFD (introduced in Bash v4.1).\n+\t#\n+\t# Perform this version check _after_ the test script was\n+\t# potentially re-executed with $TEST_SHELL_PATH for '--tee' or\n+\t# '--verbose-log', so the right shell is checked and the\n+\t# warning is issued only once.\n+\tif test -n \"$BASH_VERSION\" && eval '\n+\t test ${BASH_VERSINFO[0]} -gt 4 || {\n+\t test ${BASH_VERSINFO[0]} -eq 4 &&\n+\t test ${BASH_VERSINFO[1]} -ge 1\n+\t }\n+\t '\n+\tthen\n+\t\t: Executed by a Bash version supporting BASH_XTRACEFD. Good.\n+\telse\n+\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n+\t\ttrace=\n+\tfi\n+fi\n+if test -n \"$trace\" && test -z \"$verbose_log\"\n+then\n+\tverbose=t\n+fi\n \n # For repeatability, reset the environment to known value.\n # TERM is sanitized below, after saving color control sequences.\n@@ -193,7 +309,7 @@ fi\n \n # Add libc MALLOC and MALLOC_PERTURB test\n # only if we are not executing the test with valgrind\n-if expr \" $GIT_TEST_OPTS \" : \".* --valgrind \" >/dev/null ||\n+if test -n \"$valgrind\" ||\n test -n \"$TEST_NO_MALLOC_CHECK\"\n then\n \tsetup_malloc_check () {\n@@ -264,107 +380,6 @@ test \"x$TERM\" != \"xdumb\" && (\n \t) &&\n \tcolor=t\n \n-store_arg_to=\n-prev_opt=\n-for opt\n-do\n-\tif test -n \"$store_arg_to\"\n-\tthen\n-\t\teval $store_arg_to=\\$opt\n-\t\tstore_arg_to=\n-\t\tprev_opt=\n-\t\tcontinue\n-\tfi\n-\n-\tcase \"$opt\" in\n-\t-d|--d|--de|--deb|--debu|--debug)\n-\t\tdebug=t ;;\n-\t-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)\n-\t\timmediate=t ;;\n-\t-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)\n-\t\tGIT_TEST_LONG=t; export GIT_TEST_LONG ;;\n-\t-r)\n-\t\tstore_arg_to=run_list\n-\t\t;;\n-\t--run=*)\n-\t\trun_list=${opt#--*=} ;;\n-\t-h|--h|--he|--hel|--help)\n-\t\thelp=t ;;\n-\t-v|--v|--ve|--ver|--verb|--verbo|--verbos|--verbose)\n-\t\tverbose=t ;;\n-\t--verbose-only=*)\n-\t\tverbose_only=${opt#--*=}\n-\t\t;;\n-\t-q|--q|--qu|--qui|--quie|--quiet)\n-\t\t# Ignore --quiet under a TAP::Harness. Saying how many tests\n-\t\t# passed without the ok/not ok details is always an error.\n-\t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n-\t--with-dashes)\n-\t\twith_dashes=t ;;\n-\t--no-color)\n-\t\tcolor= ;;\n-\t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n-\t\tvalgrind=memcheck ;;\n-\t--valgrind=*)\n-\t\tvalgrind=${opt#--*=} ;;\n-\t--valgrind-only=*)\n-\t\tvalgrind_only=${opt#--*=} ;;\n-\t--tee)\n-\t\t;; # was handled already\n-\t--root=*)\n-\t\troot=${opt#--*=} ;;\n-\t--chain-lint)\n-\t\tGIT_TEST_CHAIN_LINT=1 ;;\n-\t--no-chain-lint)\n-\t\tGIT_TEST_CHAIN_LINT=0 ;;\n-\t-x)\n-\t\ttrace=t ;;\n-\t-V|--verbose-log)\n-\t\tverbose_log=t ;;\n-\t*)\n-\t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n-\tesac\n-\n-\tprev_opt=$opt\n-done\n-if test -n \"$store_arg_to\"\n-then\n-\techo \"error: $prev_opt requires an argument\" >&2\n-\texit 1\n-fi\n-\n-if test -n \"$valgrind_only\"\n-then\n-\ttest -z \"$valgrind\" && valgrind=memcheck\n-\ttest -z \"$verbose\" && verbose_only=\"$valgrind_only\"\n-elif test -n \"$valgrind\"\n-then\n-\ttest -z \"$verbose_log\" && verbose=t\n-fi\n-\n-if test -n \"$trace\" && test -n \"$test_untraceable\"\n-then\n-\t# '-x' tracing requested, but this test script can't be reliably\n-\t# traced, unless it is run with a Bash version supporting\n-\t# BASH_XTRACEFD (introduced in Bash v4.1).\n-\tif test -n \"$BASH_VERSION\" && eval '\n-\t test ${BASH_VERSINFO[0]} -gt 4 || {\n-\t test ${BASH_VERSINFO[0]} -eq 4 &&\n-\t test ${BASH_VERSINFO[1]} -ge 1\n-\t }\n-\t '\n-\tthen\n-\t\t: Executed by a Bash version supporting BASH_XTRACEFD. Good.\n-\telse\n-\t\techo >&2 \"warning: ignoring -x; '$0' is untraceable without BASH_XTRACEFD\"\n-\t\ttrace=\n-\tfi\n-fi\n-if test -n \"$trace\" && test -z \"$verbose_log\"\n-then\n-\tverbose=t\n-fi\n-\n if test -n \"$color\"\n then\n \t# Save the color control sequences now rather than run tput\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366167","messageId":"20190105010859.11031-6-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20190105010859.11031-1-szeder.dev@gmail.com","subject":"[PATCH v4 5/8] test-lib: consolidate naming of test-results paths","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-05T01:08:56Z","receivedAt":"2019-01-05T01:09:25Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"There are two places where we strip off any leading path components\nand the '.sh' suffix from the test script's pathname, and there are\nfour places where we construct the name of the 't/test-results'\ndirectory or the name of various test-specific files in there. The\nlast patch in this series will add even more.\n\nFactor these out into helper variables to avoid repeating ourselves.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 14ccb60838..5720292641 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -160,6 +160,10 @@ then\n \ttest -z \"$verbose_log\" && verbose=t\n fi\n \n+TEST_NAME=\"$(basename \"$0\" .sh)\"\n+TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n+TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n@@ -167,12 +171,11 @@ then\n \t: # do not redirect again\n elif test -n \"$tee\"\n then\n-\tmkdir -p \"$TEST_OUTPUT_DIRECTORY/test-results\"\n-\tBASE=\"$TEST_OUTPUT_DIRECTORY/test-results/$(basename \"$0\" .sh)\"\n+\tmkdir -p \"$TEST_RESULTS_DIR\"\n \n \t# Make this filename available to the sub-process in case it is using\n \t# --verbose-log.\n-\tGIT_TEST_TEE_OUTPUT_FILE=$BASE.out\n+\tGIT_TEST_TEE_OUTPUT_FILE=$TEST_RESULTS_BASE.out\n \texport GIT_TEST_TEE_OUTPUT_FILE\n \n \t# Truncate before calling \"tee -a\" to get rid of the results\n@@ -180,8 +183,8 @@ then\n \t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \n \t(GIT_TEST_TEE_STARTED=done ${TEST_SHELL_PATH} \"$0\" \"$@\" 2>&1;\n-\t echo $? >\"$BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n-\ttest \"$(cat \"$BASE.exit\")\" = 0\n+\t echo $? >\"$TEST_RESULTS_BASE.exit\") | tee -a \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\ttest \"$(cat \"$TEST_RESULTS_BASE.exit\")\" = 0\n \texit\n fi\n \n@@ -840,12 +843,9 @@ test_done () {\n \n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n-\t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n-\t\tmkdir -p \"$test_results_dir\"\n-\t\tbase=${0##*/}\n-\t\ttest_results_path=\"$test_results_dir/${base%.sh}.counts\"\n+\t\tmkdir -p \"$TEST_RESULTS_DIR\"\n \n-\t\tcat >\"$test_results_path\" <<-EOF\n+\t\tcat >\"$TEST_RESULTS_BASE.counts\" <<-EOF\n \t\ttotal $test_count\n \t\tsuccess $test_success\n \t\tfixed $test_fixed\n@@ -1051,7 +1051,7 @@ then\n fi\n \n # Test repository\n-TRASH_DIRECTORY=\"trash directory.$(basename \"$0\" .sh)\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n case \"$TRASH_DIRECTORY\" in\n /*) ;; # absolute path is good\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366168","messageId":"20190105010859.11031-7-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20190105010859.11031-1-szeder.dev@gmail.com","subject":"[PATCH v4 6/8] test-lib: set $TRASH_DIRECTORY earlier","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-05T01:08:57Z","receivedAt":"2019-01-05T01:09:25Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"A later patch in this series will need to know the path to the trash\ndirectory early in 'test-lib.sh', but $TRASH_DIRECTORY is set much\nlater.\n\nSet $TRASH_DIRECTORY earlier, where the other test-specific path\nvariables are set.\n\nSigned-off-by: SZEDER Gábor \n---\n t/test-lib.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 5720292641..0e9ac9118d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -163,6 +163,12 @@ fi\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n+test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n+case \"$TRASH_DIRECTORY\" in\n+/*) ;; # absolute path is good\n+ *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n+esac\n \n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n@@ -1051,12 +1057,6 @@ then\n fi\n \n # Test repository\n-TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n-test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n-case \"$TRASH_DIRECTORY\" in\n-/*) ;; # absolute path is good\n- *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n-esac\n rm -fr \"$TRASH_DIRECTORY\" || {\n \tGIT_EXIT_OK=t\n \techo >&5 \"FATAL: Cannot prepare test area\"\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366169","messageId":"20190105010859.11031-9-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20190105010859.11031-1-szeder.dev@gmail.com","subject":"[PATCH v4 8/8] test-lib: add the '--stress' option to run a test repeatedly under load","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-05T01:08:59Z","receivedAt":"2019-01-05T01:09:29Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Unfortunately, we have a few flaky tests, whose failures tend to be\nhard to reproduce. We've found that the best we can do to reproduce\nsuch a failure is to run the test script repeatedly while the machine\nis under load, and wait in the hope that the load creates enough\nvariance in the timing of the test's commands that a failure is\nevenually triggered. I have a command to do that, and I noticed that\ntwo other contributors have rolled their own scripts to do the same,\nall choosing slightly different approaches.\n\nTo help reproduce failures in flaky tests, introduce the '--stress'\noption to run a test script repeatedly in multiple parallel jobs until\none of them fails, thereby using the test script itself to increase\nthe load on the machine.\n\nThe number of parallel jobs is determined by, in order of precedence:\nthe number specified as '--stress=', or the value of the\nGIT_TEST_STRESS_LOAD environment variable, or twice the number of\navailable processors (as reported by the 'getconf' utility), or 8.\n\nMake '--stress' imply '--verbose -x --immediate' to get the most\ninformation about rare failures; there is really no point in spending\nall the extra effort to reproduce such a failure, and then not know\nwhich command failed and why.\n\nTo prevent the several parallel invocations of the same test from\ninterfering with each other:\n\n - Include the parallel job's number in the name of the trash\n directory and the various output files under 't/test-results/' as\n a '.stress-' suffix.\n\n - Add the parallel job's number to the port number specified by the\n user or to the test number, so even tests involving daemons\n listening on a TCP socket can be stressed.\n\n - Redirect each parallel test run's verbose output to\n 't/test-results/$TEST_NAME.stress-.out', because dumping the\n output of several parallel running tests to the terminal would\n create a big ugly mess.\n\nFor convenience, print the output of the failed test job at the end,\nand rename its trash directory to end with the '.stress-failed'\nsuffix, so it's easy to find in a predictable path (OTOH, all absolute\npaths recorded in the trash directory become invalid; we'll see\nwhether this causes any issues in practice). If, in an unlikely case,\nmore than one jobs were to fail nearly at the same time, then print\nthe output of all failed jobs, and rename the trash directory of only\nthe last one (i.e. with the highest job number), as it is the trash\ndirectory of the test whose output will be at the bottom of the user's\nterminal.\n\nBased on Jeff King's 'stress' script.\n\nSigned-off-by: SZEDER Gábor \n---\n t/README | 19 ++++++-\n t/test-lib-functions.sh | 7 ++-\n t/test-lib.sh | 109 +++++++++++++++++++++++++++++++++++++++-\n 3 files changed, 130 insertions(+), 5 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 28711cc508..11ce7675e3 100644\n--- a/t/README\n+++ b/t/README\n@@ -186,6 +186,22 @@ appropriately before running \"make\".\n \tthis feature by setting the GIT_TEST_CHAIN_LINT environment\n \tvariable to \"1\" or \"0\", respectively.\n \n+--stress::\n+--stress=::\n+\tRun the test script repeatedly in multiple parallel jobs until\n+\tone of them fails. Useful for reproducing rare failures in\n+\tflaky tests. The number of parallel jobs is, in order of\n+\tprecedence: , or the value of the GIT_TEST_STRESS_LOAD\n+\tenvironment variable, or twice the number of available\n+\tprocessors (as shown by the 'getconf' utility),\tor 8.\n+\tImplies `--verbose -x --immediate` to get the most information\n+\tabout the failure. Note that the verbose output of each test\n+\tjob is saved to 't/test-results/$TEST_NAME.stress-.out',\n+\tand only the output of the failed test job is shown on the\n+\tterminal. The names of the trash directories get a\n+\t'.stress-' suffix, and the trash directory of the failed\n+\ttest job is renamed to end with a '.stress-failed' suffix.\n+\n You can also set the GIT_TEST_INSTALLED environment variable to\n the bindir of an existing git installation to test that installation.\n You still need to have built this git sandbox, from which various\n@@ -425,7 +441,8 @@ This test harness library does the following things:\n - Creates an empty test directory with an empty .git/objects database\n and chdir(2) into it. This directory is 't/trash\n directory.$test_name_without_dotsh', with t/ subject to change by\n- the --root option documented above.\n+ the --root option documented above, and a '.stress-' suffix\n+ appended by the --stress option.\n \n - Defines standard test helper functions for your scripts to\n use. These functions are designed to make all scripts behave\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 4459bdda13..92cf8f812c 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1288,8 +1288,6 @@ test_set_port () {\n \t\t\t# root-only port, use a larger one instead.\n \t\t\tport=$(($port + 10000))\n \t\tfi\n-\n-\t\teval $var=$port\n \t\t;;\n \t*[^0-9]*|0*)\n \t\terror >&7 \"invalid port number: $port\"\n@@ -1298,4 +1296,9 @@ test_set_port () {\n \t\t# The user has specified the port.\n \t\t;;\n \tesac\n+\n+\t# Make sure that parallel '--stress' test jobs get different\n+\t# ports.\n+\tport=$(($port + ${GIT_TEST_STRESS_JOB_NR:-0}))\n+\teval $var=$port\n }\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 0e9ac9118d..a1abb1177a 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -139,6 +139,19 @@ do\n \t\tverbose_log=t\n \t\ttee=t\n \t\t;;\n+\t--stress)\n+\t\tstress=t ;;\n+\t--stress=*)\n+\t\tstress=${opt#--*=}\n+\t\tcase \"$stress\" in\n+\t\t*[^0-9]*|0*|\"\")\n+\t\t\techo \"error: --stress= requires the number of jobs to run\" >&2\n+\t\t\texit 1\n+\t\t\t;;\n+\t\t*)\t# Good.\n+\t\t\t;;\n+\t\tesac\n+\t\t;;\n \t*)\n \t\techo \"error: unknown test option '$opt'\" >&2; exit 1 ;;\n \tesac\n@@ -160,16 +173,108 @@ then\n \ttest -z \"$verbose_log\" && verbose=t\n fi\n \n+if test -n \"$stress\"\n+then\n+\tverbose=t\n+\ttrace=t\n+\timmediate=t\n+fi\n+\n+TEST_STRESS_JOB_SFX=\"${GIT_TEST_STRESS_JOB_NR:+.stress-$GIT_TEST_STRESS_JOB_NR}\"\n TEST_NAME=\"$(basename \"$0\" .sh)\"\n TEST_RESULTS_DIR=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n-TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME\"\n-TRASH_DIRECTORY=\"trash directory.$TEST_NAME\"\n+TEST_RESULTS_BASE=\"$TEST_RESULTS_DIR/$TEST_NAME$TEST_STRESS_JOB_SFX\"\n+TRASH_DIRECTORY=\"trash directory.$TEST_NAME$TEST_STRESS_JOB_SFX\"\n test -n \"$root\" && TRASH_DIRECTORY=\"$root/$TRASH_DIRECTORY\"\n case \"$TRASH_DIRECTORY\" in\n /*) ;; # absolute path is good\n *) TRASH_DIRECTORY=\"$TEST_OUTPUT_DIRECTORY/$TRASH_DIRECTORY\" ;;\n esac\n \n+# If --stress was passed, run this test repeatedly in several parallel loops.\n+if test \"$GIT_TEST_STRESS_STARTED\" = \"done\"\n+then\n+\t: # Don't stress test again.\n+elif test -n \"$stress\"\n+then\n+\tif test \"$stress\" != t\n+\tthen\n+\t\tjob_count=$stress\n+\telif test -n \"$GIT_TEST_STRESS_LOAD\"\n+\tthen\n+\t\tjob_count=\"$GIT_TEST_STRESS_LOAD\"\n+\telif job_count=$(getconf _NPROCESSORS_ONLN 2>/dev/null) &&\n+\t test -n \"$job_count\"\n+\tthen\n+\t\tjob_count=$((2 * $job_count))\n+\telse\n+\t\tjob_count=8\n+\tfi\n+\n+\tmkdir -p \"$TEST_RESULTS_DIR\"\n+\tstressfail=\"$TEST_RESULTS_BASE.stress-failed\"\n+\trm -f \"$stressfail\"\n+\n+\tstress_exit=0\n+\ttrap '\n+\t\tkill $job_pids 2>/dev/null\n+\t\twait\n+\t\tstress_exit=1\n+\t' TERM INT HUP\n+\n+\tjob_pids=\n+\tjob_nr=0\n+\twhile test $job_nr -lt \"$job_count\"\n+\tdo\n+\t\t(\n+\t\t\tGIT_TEST_STRESS_STARTED=done\n+\t\t\tGIT_TEST_STRESS_JOB_NR=$job_nr\n+\t\t\texport GIT_TEST_STRESS_STARTED GIT_TEST_STRESS_JOB_NR\n+\n+\t\t\ttrap '\n+\t\t\t\tkill $test_pid 2>/dev/null\n+\t\t\t\twait\n+\t\t\t\texit 1\n+\t\t\t' TERM INT\n+\n+\t\t\tcnt=0\n+\t\t\twhile ! test -e \"$stressfail\"\n+\t\t\tdo\n+\t\t\t\t$TEST_SHELL_PATH \"$0\" \"$@\" >\"$TEST_RESULTS_BASE.stress-$job_nr.out\" 2>&1 &\n+\t\t\t\ttest_pid=$!\n+\n+\t\t\t\tif wait $test_pid\n+\t\t\t\tthen\n+\t\t\t\t\tprintf \"OK %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n+\t\t\t\telse\n+\t\t\t\t\techo $GIT_TEST_STRESS_JOB_NR >>\"$stressfail\"\n+\t\t\t\t\tprintf \"FAIL %2d.%d\\n\" $GIT_TEST_STRESS_JOB_NR $cnt\n+\t\t\t\tfi\n+\t\t\t\tcnt=$(($cnt + 1))\n+\t\t\tdone\n+\t\t) &\n+\t\tjob_pids=\"$job_pids $!\"\n+\t\tjob_nr=$(($job_nr + 1))\n+\tdone\n+\n+\twait\n+\n+\tif test -f \"$stressfail\"\n+\tthen\n+\t\techo \"Log(s) of failed test run(s):\"\n+\t\tfor failed_job_nr in $(sort -n \"$stressfail\")\n+\t\tdo\n+\t\t\techo \"Contents of '$TEST_RESULTS_BASE.stress-$failed_job_nr.out':\"\n+\t\t\tcat \"$TEST_RESULTS_BASE.stress-$failed_job_nr.out\"\n+\t\tdone\n+\t\trm -rf \"$TRASH_DIRECTORY.stress-failed\"\n+\t\t# Move the last one.\n+\t\tmv \"$TRASH_DIRECTORY.stress-$failed_job_nr\" \"$TRASH_DIRECTORY.stress-failed\"\n+\tfi\n+\n+\texit $stress_exit\n+fi\n+\n # if --tee was passed, write the output not only to the terminal, but\n # additionally to the file test-results/$BASENAME.out, too.\n if test \"$GIT_TEST_TEE_STARTED\" = \"done\"\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366170","messageId":"20190105010859.11031-8-szeder.dev@gmail.com","threadId":"49946","inReplyTo":"20190105010859.11031-1-szeder.dev@gmail.com","subject":"[PATCH v4 7/8] test-lib-functions: introduce the 'test_set_port' helper function","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-05T01:08:58Z","receivedAt":"2019-01-05T01:09:29Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Several test scripts run daemons like 'git-daemon' or Apache, and\ncommunicate with them through TCP sockets. To have unique ports where\nthese daemons are accessible, the ports are usually the number of the\ncorresponding test scripts, unless the user overrides them via\nenvironment variables, and thus all those tests and test libs contain\nmore or less the same bit of one-liner boilerplate code to find out\nthe port. The last patch in this series will make this a bit more\ncomplicated.\n\nFactor out finding the port for a daemon into the common helper\nfunction 'test_set_port' to avoid repeating ourselves.\n\nTake special care of test scripts with \"low\" numbers:\n\n - Test numbers below 1024 would result in a port that's only usable\n as root, so set their port to '10000 + test-nr' to make sure it\n doesn't interfere with other tests in the test suite. This makes\n the hardcoded port number in 't0410-partial-clone.sh' unnecessary,\n remove it.\n\n - The shell's arithmetic evaluation interprets numbers with leading\n zeros as octal values, which means that test number below 1000 and\n containing the digits 8 or 9 will trigger an error. Remove all\n leading zeros from the test numbers to prevent this.\n\nNote that the 'git p4' tests are unlike the other tests involving\ndaemons in that:\n\n - 'lib-git-p4.sh' doesn't use the test's number for unique port as\n is, but does a bit of additional arithmetic on top [1].\n\n - The port is not overridable via an environment variable.\n\nWith this patch even 'git p4' tests will use the test's number as\ndefault port, and it will be overridable via the P4DPORT environment\nvariable.\n\n[1] Commit fc00233071 (git-p4 tests: refactor and cleanup, 2011-08-22)\n introduced that \"unusual\" unique port computation without\n explaining why it was necessary (as opposed to simply using the\n test number as is). It seems to be just unnecessary complication,\n and in any case that commit came way before the \"test nr as unique\n port\" got \"standardized\" for other daemons in commits c44132fcf3\n (tests: auto-set git-daemon port, 2014-02-10), 3bb486e439 (tests:\n auto-set LIB_HTTPD_PORT from test name, 2014-02-10), and\n bf9d7df950 (t/lib-git-svn.sh: improve svnserve tests with parallel\n make test, 2017-12-01).\n\nSigned-off-by: SZEDER Gábor \n---\n t/lib-git-daemon.sh | 2 +-\n t/lib-git-p4.sh | 9 +--------\n t/lib-git-svn.sh | 2 +-\n t/lib-httpd.sh | 2 +-\n t/t0410-partial-clone.sh | 1 -\n t/t5512-ls-remote.sh | 2 +-\n t/test-lib-functions.sh | 36 ++++++++++++++++++++++++++++++++++++\n 7 files changed, 41 insertions(+), 13 deletions(-)\n\ndiff --git a/t/lib-git-daemon.sh b/t/lib-git-daemon.sh\nindex f98de95c15..41eb1e3ae8 100644\n--- a/t/lib-git-daemon.sh\n+++ b/t/lib-git-daemon.sh\n@@ -28,7 +28,7 @@ then\n \ttest_skip_or_die $GIT_TEST_GIT_DAEMON \"file system does not support FIFOs\"\n fi\n \n-LIB_GIT_DAEMON_PORT=${LIB_GIT_DAEMON_PORT-${this_test#t}}\n+test_set_port LIB_GIT_DAEMON_PORT\n \n GIT_DAEMON_PID=\n GIT_DAEMON_DOCUMENT_ROOT_PATH=\"$PWD\"/repo\ndiff --git a/t/lib-git-p4.sh b/t/lib-git-p4.sh\nindex c27599474c..b3be3ba011 100644\n--- a/t/lib-git-p4.sh\n+++ b/t/lib-git-p4.sh\n@@ -53,14 +53,7 @@ time_in_seconds () {\n \t(cd / && \"$PYTHON_PATH\" -c 'import time; print(int(time.time()))')\n }\n \n-# Try to pick a unique port: guess a large number, then hope\n-# no more than one of each test is running.\n-#\n-# This does not handle the case where somebody else is running the\n-# same tests and has chosen the same ports.\n-testid=${this_test#t}\n-git_p4_test_start=9800\n-P4DPORT=$((10669 + ($testid - $git_p4_test_start)))\n+test_set_port P4DPORT\n \n P4PORT=localhost:$P4DPORT\n P4CLIENT=client\ndiff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh\nindex a8130f9119..f3b478c307 100644\n--- a/t/lib-git-svn.sh\n+++ b/t/lib-git-svn.sh\n@@ -13,6 +13,7 @@ fi\n GIT_DIR=$PWD/.git\n GIT_SVN_DIR=$GIT_DIR/svn/refs/remotes/git-svn\n SVN_TREE=$GIT_SVN_DIR/svn-tree\n+test_set_port SVNSERVE_PORT\n \n svn >/dev/null 2>&1\n if test $? -ne 1\n@@ -119,7 +120,6 @@ require_svnserve () {\n }\n \n start_svnserve () {\n-\tSVNSERVE_PORT=${SVNSERVE_PORT-${this_test#t}}\n \tsvnserve --listen-port $SVNSERVE_PORT \\\n \t\t --root \"$rawsvnrepo\" \\\n \t\t --listen-once \\\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex a8729f8232..e465116ef9 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -82,7 +82,7 @@ case $(uname) in\n esac\n \n LIB_HTTPD_PATH=${LIB_HTTPD_PATH-\"$DEFAULT_HTTPD_PATH\"}\n-LIB_HTTPD_PORT=${LIB_HTTPD_PORT-${this_test#t}}\n+test_set_port LIB_HTTPD_PORT\n \n TEST_PATH=\"$TEST_DIRECTORY\"/lib-httpd\n HTTPD_ROOT_PATH=\"$PWD\"/httpd\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex ba3887f178..0aca8d7588 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -480,7 +480,6 @@ test_expect_success 'gc stops traversal when a missing but promised object is re\n \t! grep \"$TREE_HASH\" out\n '\n \n-LIB_HTTPD_PORT=12345 # default port, 410, cannot be used as non-root\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \ndiff --git a/t/t5512-ls-remote.sh b/t/t5512-ls-remote.sh\nindex 32e722db2e..cd9e60632d 100755\n--- a/t/t5512-ls-remote.sh\n+++ b/t/t5512-ls-remote.sh\n@@ -260,7 +260,7 @@ test_lazy_prereq GIT_DAEMON '\n # This test spawns a daemon, so run it only if the user would be OK with\n # testing with git-daemon.\n test_expect_success PIPE,JGIT,GIT_DAEMON 'indicate no refs in standards-compliant empty remote' '\n-\tJGIT_DAEMON_PORT=${JGIT_DAEMON_PORT-${this_test#t}} &&\n+\ttest_set_port JGIT_DAEMON_PORT &&\n \tJGIT_DAEMON_PID= &&\n \tgit init --bare empty.git &&\n \t>empty.git/git-daemon-export-ok &&\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 6b3bbf99e4..4459bdda13 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1263,3 +1263,39 @@ test_oid () {\n \tfi &&\n \teval \"printf '%s' \\\"\\${$var}\\\"\"\n }\n+\n+# Choose a port number based on the test script's number and store it in\n+# the given variable name, unless that variable already contains a number.\n+test_set_port () {\n+\tlocal var=$1 port\n+\n+\tif test $# -ne 1 || test -z \"$var\"\n+\tthen\n+\t\tBUG \"test_set_port requires a variable name\"\n+\tfi\n+\n+\teval port=\\$$var\n+\tcase \"$port\" in\n+\t\"\")\n+\t\t# No port is set in the given env var, use the test\n+\t\t# number as port number instead.\n+\t\t# Remove not only the leading 't', but all leading zeros\n+\t\t# as well, so the arithmetic below won't (mis)interpret\n+\t\t# a test number like '0123' as an octal value.\n+\t\tport=${this_test#${this_test%%[1-9]*}}\n+\t\tif test \"${port:-0}\" -lt 1024\n+\t\tthen\n+\t\t\t# root-only port, use a larger one instead.\n+\t\t\tport=$(($port + 10000))\n+\t\tfi\n+\n+\t\teval $var=$port\n+\t\t;;\n+\t*[^0-9]*|0*)\n+\t\terror >&7 \"invalid port number: $port\"\n+\t\t;;\n+\t*)\n+\t\t# The user has specified the port.\n+\t\t;;\n+\tesac\n+}\n-- \n2.20.1.151.gec613c4b75\n\n"},{"id":"366245","messageId":"20190107084906.GA30186@sigill.intra.peff.net","threadId":"49946","inReplyTo":"20190105010859.11031-1-szeder.dev@gmail.com","subject":"Re: [PATCH v4 0/8] test-lib: add the '--stress' option to help reproduce occasional failures in flaky tests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-01-07T08:49:06Z","receivedAt":"2019-01-07T08:49:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jan 05, 2019 at 02:08:51AM +0100, SZEDER Gábor wrote:\n\n> To recap: this patch series tries to make reproducing rare failures in\n> flaky tests easier: it adds the '--stress' option to our test library\n> to run the test script repeatedly in multiple parallel jobs, in the\n> hope that the increased load creates enough variance in the timing of\n> the test's commands that such a failure is eventually triggered.\n\nThanks, I looked over this version (glossing over the bits that hadn't\nreally changed since v2) and it all looks good to me.\n\n-Peff\n"}]}