git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 6/6] Treat unexpectedly fixed known breakages more seriously

From
Adam Spiers <git@adamspiers.org>
Date
Sep 19, 2012, 18:49 UTC
Message-ID
<1348080591-16430-3-git-send-email-git@adamspiers.org>
In-Reply-To
<1348080591-16430-1-git-send-email-git@adamspiers.org>

Change color of unexpectedly fixed known breakages to bold red. An unexpectedly passing test indicates that the test code is somehow broken or out of sync with the code it is testing. Either way this is an error which is potentially as bad as a failing test, and as such is no longer portrayed as a pass in the output.

Signed-off-by: Adam Spiers <git@adamspiers.org>
---
 t/t0000-basic.sh | 30 ++++++++++++++++++++++++------
 t/test-lib.sh    | 13 +++++++++----
 2 files changed, 33 insertions(+), 10 deletions(-)
diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index 65f578f..ed44f7d 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -140,13 +140,31 @@ test_expect_success 'pretend we have fixed a known breakage' "
 	test_done
 	EOF
 	check_sub_test_lib_test passing-todo <<-\\EOF
-	> ok 1 - pretend we have fixed a known breakage # TODO known breakage
-	> # fixed 1 known breakage(s)
-	> # passed all 1 test(s)
+	> ok 1 - pretend we have fixed a known breakage # TODO known breakage vanished
+	> # 1 known breakage(s) vanished; please update test(s)
 	> 1..1
 	EOF
 "
 
+test_expect_success 'pretend we have fixed one of two known breakages (run in sub test-lib)' "
+	run_sub_test_lib_test partially-passing-todos \
+		'2 TODO tests, one passing' <<-\\EOF &&
+	test_expect_failure 'pretend we have a known breakage' 'false'
+	test_expect_success 'pretend we have a passing test' 'true'
+	test_expect_failure 'pretend we have fixed another known breakage' 'true'
+	test_done
+	EOF
+	check_sub_test_lib_test partially-passing-todos <<-\\EOF
+	> not ok 1 - pretend we have a known breakage # TODO known breakage
+	> ok 2 - pretend we have a passing test
+	> ok 3 - pretend we have fixed another known breakage # TODO known breakage vanished
+	> # 1 known breakage(s) vanished; please update test(s)
+	> # still have 1 known breakage(s)
+	> # passed all remaining 1 test(s)
+	> 1..3
+	EOF
+"
+
 test_expect_success 'pretend we have a pass, fail, and known breakage' "
 	test_must_fail run_sub_test_lib_test \
 		mixed-results1 'mixed results #1' <<-\\EOF &&
@@ -194,10 +212,10 @@ test_expect_success 'pretend we have a mix of all possible results' "
 	> #	false
 	> not ok 8 - pretend we have a known breakage # TODO known breakage
 	> not ok 9 - pretend we have a known breakage # TODO known breakage
-	> ok 10 - pretend we have fixed a known breakage # TODO known breakage
-	> # fixed 1 known breakage(s)
+	> ok 10 - pretend we have fixed a known breakage # TODO known breakage vanished
+	> # 1 known breakage(s) vanished; please update test(s)
 	> # still have 2 known breakage(s)
-	> # failed 3 among remaining 8 test(s)
+	> # failed 3 among remaining 7 test(s)
 	> 1..10
 	EOF
 "
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 7028ba8..b403e85 100755
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -278,7 +278,7 @@ test_failure_ () {
 
 test_known_broken_ok_ () {
 	test_fixed=$(($test_fixed+1))
-	say_color "" "ok $test_count - $@ # TODO known breakage"
+	say_color error "ok $test_count - $@ # TODO known breakage vanished"
 }
 
 test_known_broken_failure_ () {
@@ -373,13 +373,18 @@ test_done () {
 
 	if test "$test_fixed" != 0
 	then
-		say_color pass "# fixed $test_fixed known breakage(s)"
+		say_color error "# $test_fixed known breakage(s) vanished; please update test(s)"
 	fi
 	if test "$test_broken" != 0
 	then
 		say_color warn "# still have $test_broken known breakage(s)"
-		msg="remaining $(($test_count-$test_broken)) test(s)"
+	fi
+	if test "$test_broken" != 0 || test "$test_fixed" != 0
+	then
+		test_remaining=$(( $test_count - $test_broken - $test_fixed ))
+		msg="remaining $test_remaining test(s)"
 	else
+		test_remaining=$test_count
 		msg="$test_count test(s)"
 	fi
 	case "$test_failure" in
@@ -393,7 +398,7 @@ test_done () {
 
 		if test $test_external_has_tap -eq 0
 		then
-			if test $test_count -gt 0
+			if test $test_remaining -gt 0
 			then
 				say_color pass "# passed all $msg"
 			fi

-- 
1.7.12.147.g6d168f4
Previous: Adam SpiersNext: Junio C Hamano
Message 13 of 19 in “make test output coloring more intuitive”
  1. 0/6 make test output coloring more intuitiveAdam Spiers, Sep 19, 2012
  2. 1/6 Change the color of individual known breakagesAdam Spiers, Sep 19, 2012
  3. 2/6 Make 'not ok $count - $message' consistent with 'ok $count - $message'Adam Spiers, Sep 19, 2012
  4. Jeff KingSep 19, 2012
  5. Junio C HamanoSep 19, 2012
  6. Adam SpiersSep 19, 2012
  7. 3/6 Color skipped tests the same as informational messagesAdam Spiers, Sep 19, 2012
  8. 4/6 Refactor mechanics of testing in a sub test-libAdam Spiers, Sep 19, 2012
  9. Jeff KingSep 19, 2012
  10. Adam SpiersSep 19, 2012
  11. 4/6 Refactor mechanics of testing in a sub test-libAdam Spiers, Sep 19, 2012
  12. 5/6 Test the test framework more thoroughlyAdam Spiers, Sep 19, 2012
  13. 6/6 Treat unexpectedly fixed known breakages more seriouslyAdam Spiers, Sep 19, 2012
  14. Junio C HamanoSep 20, 2012
  15. Jeff KingSep 19, 2012
  16. Adam SpiersSep 19, 2012
  17. 5/6 Test the test framework more thoroughlyAdam Spiers, Sep 19, 2012
  18. 6/6 Treat unexpectedly fixed known breakages more seriouslyAdam Spiers, Sep 19, 2012
  19. Jeff KingSep 19, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.