threads / rfc / 27898

RFC patchtests: print failed test numbers at the end of the test run

Subject: [RFC PATCH] tests: print failed test numbers at the end of the test run

## tl;dr

9 messages between Jul 23, 2011 and Jul 26, 2011. Diffs are folded; open one to read it.

replies: 8people: 5as markdown or json

Jens Lehmann· Jul 23, 2011, 19:16 UTC · lore

On modern multi-core processors "make test" is often run in multiple jobs. If one of them fails the test run does stop, but the concurrently running tests finish their run. Finding out what test is broken involves a lot of scrolling. That gets even worse when the -i option is used.

If one or more tests failed, print a list of them before the test summary:
failed test(s): t1000 t6500

fixed 0 success 7638 failed 3 broken 49 total 7723

This makes it possible to just run the test suite with -i and collect all failed test scripts at the end for further examination.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---

Maybe I'm missing something completely obvious, but I always have a hard time finding out which test scripts did fail in a test run with -j30.

 t/aggregate-results.sh |   12 +++++++++++-
 1 files changed, 11 insertions(+), 1 deletions(-)
Show changes to t/aggregate-results.sh +11 −1
diff --git a/t/aggregate-results.sh b/t/aggregate-results.sh
index d206b7c..b8e929a 100755
--- a/t/aggregate-results.sh
+++ b/t/aggregate-results.sh
@@ -1,5 +1,6 @@
 #!/bin/sh

+failed_tests=
 fixed=0
 success=0
 failed=0
@@ -18,7 +19,12 @@ do
 		success)
 			success=$(($success + $value)) ;;
 		failed)
-			failed=$(($failed + $value)) ;;
+			failed=$(($failed + $value))
+			if test $value != 0; then
+				testnum=$(echo $file | cut -b 14-18)
+				failed_tests="$failed_tests $testnum"
+			fi
+			;;
 		broken)
 			broken=$(($broken + $value)) ;;
 		total)
@@ -27,6 +33,10 @@ do
 	done <"$file"
 done

+if [ -n "$failed_tests" ]; then
+	printf "\nfailed test(s):$failed_tests\n\n"
+fi
+
 printf "%-8s%d\n" fixed $fixed
 printf "%-8s%d\n" success $success
 printf "%-8s%d\n" failed $failed

-- 
1.7.6.346.g750efc
Junio C Hamano· Jul 24, 2011, 07:41 UTC · re: Jens Lehmann · lore

Re: [RFC PATCH] tests: print failed test numbers at the end of the test run

Jens Lehmann <Jens.Lehmann@web.de> writes:
> Maybe I'm missing something completely obvious, but I always have a hard
> time finding out which test scripts did fail in a test run with -j30.
I run "ls -d t/trash*" for that ;-)
> +			if test $value != 0; then
> +				testnum=$(echo $file | cut -b 14-18)
> +				failed_tests="$failed_tests $testnum"
> +			fi

Somehow "cut" here feels dirty, as it hardcodes a mysterious offset that cannot be guessed without knowing what string it is cutting from by looking at t/Makefile.

	testnum=$(expr "$file" : 'test-results/\(t[0-9]*\)-')
Jens Lehmann· Jul 24, 2011, 13:35 UTC · re: Junio C Hamano · lore

[PATCH v2] tests: print failed test numbers at the end of the test run

On modern multi-core processors "make test" is often run in multiple jobs. If one of them fails the test run does stop, but the concurrently running tests finish their run. It is rather easy to find out which test failed by doing a "ls -d t/trash*". But that only works when you don't use the "-i" option to "make test" because you want to get an overview of all failing tests. In that case all thrash directories are deleted end and the information which tests failed is lost.

If one or more tests failed, print a list of them before the test summary:
failed test(s): t1000 t6500

fixed 0 success 7638 failed 3 broken 49 total 7723

This makes it possible to just run the test suite with -i and collect all failed test scripts at the end for further examination.

Signed-off-by: Jens Lehmann <Jens.Lehmann@web.de>
---
Am 24.07.2011 09:41, schrieb Junio C Hamano:
Show 6 quoted lines
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> Maybe I'm missing something completely obvious, but I always have a hard
>> time finding out which test scripts did fail in a test run with -j30.
> 
> I run "ls -d t/trash*" for that ;-)

Yup. But that won't work when you use "-i" to continue to run all tests even when some fail, as all trash directories get deleted in the end. Or did I manage to overlook an option which stops that? ;-)

I rephrased the commit message to make it clear my patch only changes the behavior when "-i" is used and one or more tests fail, as the code in question isn't executed otherwise.

Show 10 quoted lines
>> +			if test $value != 0; then
>> +				testnum=$(echo $file | cut -b 14-18)
>> +				failed_tests="$failed_tests $testnum"
>> +			fi
> 
> Somehow "cut" here feels dirty, as it hardcodes a mysterious offset that
> cannot be guessed without knowing what string it is cutting from by
> looking at t/Makefile.
> 
> 	testnum=$(expr "$file" : 'test-results/\(t[0-9]*\)-')
Thanks, this version is using that expression.
 t/aggregate-results.sh |   12 +++++++++++-
 1 files changed, 11 insertions(+), 1 deletions(-)
Show changes to t/aggregate-results.sh +11 −1
diff --git a/t/aggregate-results.sh b/t/aggregate-results.sh
index d206b7c..60465fa 100755
--- a/t/aggregate-results.sh
+++ b/t/aggregate-results.sh
@@ -1,5 +1,6 @@
 #!/bin/sh

+failed_tests=
 fixed=0
 success=0
 failed=0
@@ -18,7 +19,12 @@ do
 		success)
 			success=$(($success + $value)) ;;
 		failed)
-			failed=$(($failed + $value)) ;;
+			failed=$(($failed + $value))
+			if test $value != 0; then
+				testnum=$(expr "$file" : 'test-results/\(t[0-9]*\)-')
+				failed_tests="$failed_tests $testnum"
+			fi
+			;;
 		broken)
 			broken=$(($broken + $value)) ;;
 		total)
@@ -27,6 +33,10 @@ do
 	done <"$file"
 done

+if [ -n "$failed_tests" ]; then
+	printf "\nfailed test(s):$failed_tests\n\n"
+fi
+
 printf "%-8s%d\n" fixed $fixed
 printf "%-8s%d\n" success $success
 printf "%-8s%d\n" failed $failed

-- 
1.7.6.346.g940def.dirty
Jeff King· Jul 24, 2011, 15:39 UTC · re: Jens Lehmann · lore

Re: [PATCH v2] tests: print failed test numbers at the end of the test run

On Sun, Jul 24, 2011 at 03:35:54PM +0200, Jens Lehmann wrote:
Show 11 quoted lines
> Am 24.07.2011 09:41, schrieb Junio C Hamano:
> > Jens Lehmann <Jens.Lehmann@web.de> writes:
> > 
> >> Maybe I'm missing something completely obvious, but I always have a hard
> >> time finding out which test scripts did fail in a test run with -j30.
> > 
> > I run "ls -d t/trash*" for that ;-)
> 
> Yup. But that won't work when you use "-i" to continue to run all tests
> even when some fail, as all trash directories get deleted in the end. Or
> did I manage to overlook an option which stops that? ;-)
I used to use:
  make -k -j30
  grep 'failed [^0]' test-results/*

but these days I use the "prove" tool, which has much nicer output. You can get Makefile support for it with:

  GIT_PROVE_OPTS = -j30
  DEFAULT_TEST_TARGET = prove
in your config.mak.
-Peff
Jens Lehmann· Jul 24, 2011, 19:21 UTC · re: Jeff King · lore

Re: [PATCH v2] tests: print failed test numbers at the end of the test run

Am 24.07.2011 17:39, schrieb Jeff King:
Show 18 quoted lines
> On Sun, Jul 24, 2011 at 03:35:54PM +0200, Jens Lehmann wrote:
> 
>> Am 24.07.2011 09:41, schrieb Junio C Hamano:
>>> Jens Lehmann <Jens.Lehmann@web.de> writes:
>>>
>>>> Maybe I'm missing something completely obvious, but I always have a hard
>>>> time finding out which test scripts did fail in a test run with -j30.
>>>
>>> I run "ls -d t/trash*" for that ;-)
>>
>> Yup. But that won't work when you use "-i" to continue to run all tests
>> even when some fail, as all trash directories get deleted in the end. Or
>> did I manage to overlook an option which stops that? ;-)
> 
> I used to use:
> 
>   make -k -j30
>   grep 'failed [^0]' test-results/*

Ah, now I understand how I managed to shoot myself in the foot when I tried that: I used "-i" instead of "-k" for make, so the test-results directory always was deleted after the tests ran.

Show 7 quoted lines
> but these days I use the "prove" tool, which has much nicer output. You
> can get Makefile support for it with:
> 
>   GIT_PROVE_OPTS = -j30
>   DEFAULT_TEST_TARGET = prove
> 
> in your config.mak.
Thanks, the output really is nicer and it even runs a bit faster! :-)
Sverre Rabbelier· Jul 25, 2011, 14:31 UTC · re: Jens Lehmann · lore

Re: [PATCH v2] tests: print failed test numbers at the end of the test run

Heya,
On Sun, Jul 24, 2011 at 21:21, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Ah, now I understand how I managed to shoot myself in the foot when I tried
> that: I used "-i" instead of "-k" for make, so the test-results directory
> always was deleted after the tests ran.

Regardless, I like your patch. I don't use prove, and I'm always afraid that when I run the test suite I'll find a test that failed with no way to find out which :).

-- 
Cheers,

Sverre Rabbelier
Nguyen Thai Ngoc Duy· Jul 26, 2011, 04:25 UTC · re: Jens Lehmann · lore

Re: [RFC PATCH] tests: print failed test numbers at the end of the test run

On Sun, Jul 24, 2011 at 2:16 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> On modern multi-core processors "make test" is often run in multiple jobs.
> If one of them fails the test run does stop, but the concurrently running
> tests finish their run.

Somewhat related (or not). I change something. I know it breaks things and want to know _all_ tests it breaks, but "make test" would stop early. Is there anyway to make it keep going through all tests even if some fails? "make -j<big number>" improves the situation but does not really solve it.

-- 
Duy
Jeff King· Jul 26, 2011, 05:42 UTC · re: Nguyen Thai Ngoc Duy · lore

Re: [RFC PATCH] tests: print failed test numbers at the end of the test run

On Tue, Jul 26, 2011 at 11:25:21AM +0700, Nguyen Thai Ngoc Duy wrote:
Show 10 quoted lines
> On Sun, Jul 24, 2011 at 2:16 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> > On modern multi-core processors "make test" is often run in multiple jobs.
> > If one of them fails the test run does stop, but the concurrently running
> > tests finish their run.
> 
> Somewhat related (or not). I change something. I know it breaks things
> and want to know _all_ tests it breaks, but "make test" would stop
> early. Is there anyway to make it keep going through all tests even if
> some fails? "make -j<big number>" improves the situation but does not
> really solve it.

Try "make -k", which will keep running rules that don't have failed dependencies (in this case, all of the tests are independent, so it will run all of them).

Or use "prove", which can do fancy things like putting the last-failed (so you can see the likely failures immediately and start to probe them while the rest of the suite runs).

-Peff
Jeff King· Jul 26, 2011, 06:10 UTC · re: Jeff King · lore

Re: [RFC PATCH] tests: print failed test numbers at the end of the test run

On Mon, Jul 25, 2011 at 11:42:36PM -0600, Jeff King wrote:
> Or use "prove", which can do fancy things like putting the last-failed
> (so you can see the likely failures immediately and start to probe them
> while the rest of the suite runs).

Oops, that should read "...putting the last-failed tests at the beginning".

-Peff

← back to recent threads