git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFC PATCH 6/6] test-lib: support running tests under valgrind in parallel

From
Thomas Rast <trast@inf.ethz.ch>
Date
May 16, 2013, 20:50 UTC
Message-ID
<1d22b4b7885ff6b27c0837608aeaacd20e36ed94.1368736093.git.trast@inf.ethz.ch>
In-Reply-To
<cover.1368736093.git.trast@inf.ethz.ch>

With the new --valgrind-parallel=<n> option, we support running the tests in a single test script under valgrind in parallel using 'n' processes.

This really follows the dumbest approach possible, as follows:
* We spawn the test script 'n' times, using a throw-away
  TEST_OUTPUT_DIRECTORY.  Each of the instances is given options that
  ensures that it only runs every n-th test under valgrind, but
  together they cover the entire range.
* We add up the numbers from the individual tests, and provide the
  usual output.

This is really a gross hack at this point, and should be improved. In particular we should keep the actual outputs somewhere more easily discoverable, and summarize them to the user.

Nevertheless, this is already workable and gives a speedup of more than 2 on a dual-core (hyperthreaded) machine, using n=4. This is expected since the overhead of valgrind is so big (on the order of 20x under good conditions, and a large startup overhead at every git invocation) that redundantly running the non-valgrind tests in between is not that expensive.

Signed-off-by: Thomas Rast <trast@inf.ethz.ch>
---
 t/test-lib.sh | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 55 insertions(+), 5 deletions(-)
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 55fa749..b4e81bc 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -204,6 +204,15 @@ do
 	--valgrind-only=*)
 		valgrind_only=$(expr "z$1" : 'z[^=]*=\(.*\)')
 		shift ;;
+	--valgrind-parallel=*)
+		valgrind_parallel=$(expr "z$1" : 'z[^=]*=\(.*\)')
+		shift ;;
+	--valgrind-only-stride=*)
+		valgrind_only_stride=$(expr "z$1" : 'z[^=]*=\(.*\)')
+		shift ;;
+	--valgrind-only-offset=*)
+		valgrind_only_offset=$(expr "z$1" : 'z[^=]*=\(.*\)')
+		shift ;;
 	--tee)
 		shift ;; # was handled already
 	--root=*)
@@ -217,7 +226,7 @@ do
 	esac
 done
 
-if test -n "$valgrind_only"
+if test -n "$valgrind_only" || test -n "$valgrind_only_stride"
 then
 	test -z "$valgrind" && valgrind=memcheck
 	test -z "$verbose" && verbose_only="$valgrind_only"
@@ -359,8 +368,10 @@ match_pattern_list () {
 }
 
 toggle_verbose () {
-	test -z "$verbose_only" && return
-	if match_pattern_list $test_count $verbose_only
+	test -z "$verbose_only" && test -z "$valgrind_only_stride" && return
+	if match_pattern_list $test_count $verbose_only ||
+		{ test -n "$valgrind_only_stride" &&
+		expr $test_count "%" $valgrind_only_stride - $valgrind_only_offset = 0 >/dev/null; }
 	then
 		exec 4>&2 3>&1
 	else
@@ -370,7 +381,7 @@ toggle_verbose () {
 
 toggle_valgrind () {
 	test -z "$GIT_VALGRIND" && return
-	if test -z "$valgrind_only"
+	if test -z "$valgrind_only" && test -z "$valgrind_only_stride"
 	then
 		GIT_VALGRIND_ENABLED=t
 		return
@@ -379,6 +390,10 @@ toggle_valgrind () {
 	if match_pattern_list $test_count $valgrind_only
 	then
 		GIT_VALGRIND_ENABLED=t
+	elif test -n "$valgrind_only_stride" &&
+		expr $test_count "%" $valgrind_only_stride - $valgrind_only_offset = 0 >/dev/null
+	then
+		GIT_VALGRIND_ENABLED=t
 	fi
 }
 
@@ -600,7 +615,10 @@ then
 	GIT_VALGRIND_MODE="$valgrind"
 	export GIT_VALGRIND_MODE
 	GIT_VALGRIND_ENABLED=t
-	test -n "$valgrind_only" && GIT_VALGRIND_ENABLED=
+	if test -n "$valgrind_only" || test -n "$valgrind_only_stride"
+	then
+		GIT_VALGRIND_ENABLED=
+	fi
 	export GIT_VALGRIND_ENABLED
 elif test -n "$GIT_TEST_INSTALLED"
 then
@@ -686,6 +704,38 @@ then
 else
 	mkdir -p "$TRASH_DIRECTORY"
 fi
+
+if test -n "$valgrind_parallel"
+then
+	for i in $(test_seq 1 $valgrind_parallel)
+	do
+		root="$TRASH_DIRECTORY/vgparallel-$i"
+		mkdir "$root"
+		TEST_OUTPUT_DIRECTORY="$root" \
+			${SHELL_PATH} "$0" \
+			--root="$root" --statusprefix="[$i] " \
+			--valgrind="$valgrind" \
+			--valgrind-only-stride="$valgrind_parallel" \
+			--valgrind-only-offset="$i" &
+		pids="$pids $!"
+	done
+	trap "kill $pids" INT TERM HUP
+	wait $pids
+	trap - INT TERM HUP
+	for i in $(test_seq 1 $valgrind_parallel)
+	do
+		root="$TRASH_DIRECTORY/vgparallel-$i"
+		eval "$(cat "$root/test-results/$(basename "$0" .sh)"-*.counts |
+			sed 's/^\([a-z][a-z]*\) \([0-9][0-9]*\)/inner_\1=\2/')"
+		test_count=$(expr $test_count + $inner_total)
+		test_success=$(expr $test_success + $inner_success)
+		test_fixed=$(expr $test_fixed + $inner_fixed)
+		test_broken=$(expr $test_broken + $inner_broken)
+		test_failed=$(expr $test_failed + $inner_failed)
+	done
+	test_done
+fi
+
 # Use -P to resolve symlinks in our working directory so that the cwd
 # in subprocesses like git equals our $PWD (for pathname comparisons).
 cd -P "$TRASH_DIRECTORY" || exit 1

-- 
1.8.3.rc2.393.g8636c0b
Previous: Phil HordNext: Jeff King
Message 21 of 44 in “--valgrind improvements”
  1. 0/6 --valgrind improvementsThomas Rast, May 16, 2013
  2. 1/6 test-lib: enable MALLOC_* for the actual testsThomas Rast, May 16, 2013
  3. Elia PintoMay 16, 2013
  4. Junio C HamanoMay 16, 2013
  5. 2/6 test-lib: refactor $GIT_SKIP_TESTS matchingThomas Rast, May 16, 2013
  6. Johannes SixtMay 17, 2013
  7. Thomas RastMay 17, 2013
  8. Junio C HamanoMay 17, 2013
  9. Thomas RastMay 17, 2013
  10. Junio C HamanoMay 17, 2013
  11. Johannes SixtMay 17, 2013
  12. 3/6 test-lib: verbose mode for only tests matching a patternThomas Rast, May 16, 2013
  13. Jeff KingMay 29, 2013
  14. Jeff KingMay 29, 2013
  15. Junio C HamanoMay 29, 2013
  16. 4/6 test-lib: valgrind for only tests matching a patternThomas Rast, May 16, 2013
  17. 5/6 test-lib: allow prefixing a custom string before "ok N" etc.Thomas Rast, May 16, 2013
  18. Phil HordMay 16, 2013
  19. Thomas RastMay 17, 2013
  20. Phil HordMay 17, 2013
  21. 6/6 test-lib: support running tests under valgrind in parallelThomas Rast, May 16, 2013
  22. Jeff KingMay 29, 2013
  23. 0/6 --valgrind improvementsThomas Rast, Jun 17, 2013
  24. 1/6 test-lib: enable MALLOC_* for the actual testsThomas Rast, Jun 17, 2013
  25. 2/6 test-lib: refactor $GIT_SKIP_TESTS matchingThomas Rast, Jun 17, 2013
  26. Johannes SixtJun 18, 2013
  27. Thomas RastJun 18, 2013
  28. 3/6 test-lib: verbose mode for only tests matching a patternThomas Rast, Jun 17, 2013
  29. Jeff KingJun 18, 2013
  30. Thomas RastJun 18, 2013
  31. 4/6 test-lib: valgrind for only tests matching a patternThomas Rast, Jun 17, 2013
  32. 5/6 test-lib: allow prefixing a custom string before "ok N" etc.Thomas Rast, Jun 17, 2013
  33. 6/6 test-lib: support running tests under valgrind in parallelThomas Rast, Jun 17, 2013
  34. Jeff KingJun 18, 2013
  35. 0/8 --valgrind improvementsThomas Rast, Jun 18, 2013
  36. 1/8 test-lib: enable MALLOC_* for the actual testsThomas Rast, Jun 18, 2013
  37. 2/8 test-lib: refactor $GIT_SKIP_TESTS matchingThomas Rast, Jun 18, 2013
  38. 3/8 test-lib: rearrange start/end of test_expect_* and test_skipThomas Rast, Jun 18, 2013
  39. Junio C HamanoJun 18, 2013
  40. 4/8 test-lib: self-test that --verbose worksThomas Rast, Jun 18, 2013
  41. 5/8 test-lib: verbose mode for only tests matching a patternThomas Rast, Jun 18, 2013
  42. 6/8 test-lib: valgrind for only tests matching a patternThomas Rast, Jun 18, 2013
  43. 7/8 test-lib: allow prefixing a custom string before "ok N" etc.Thomas Rast, Jun 18, 2013
  44. 8/8 test-lib: support running tests under valgrind in parallelThomas Rast, Jun 18, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.