[RFC PATCH] lib-test: show failed prereq was Re: [PATCH] t/lib-git.sh: fix ACL-related permissions failure

From: Fabian Stelzer <fs@gigacodes.de>
Date: Nov 12, 2021, 16:01 UTC
Message-ID: <20211112160101.xm7xi4474pgybrh4@fs>
In-Reply-To: <xmqqv916wh7t.fsf@gitster.g>

On 05.11.2021 12:11, Junio C Hamano wrote:

Show 18 quoted lines

>Adam Dinwoodie <adam@dinwoodie.org> writes:
>
>> This is probably a much broader conversation. I remember when I first
>> started packaging Git for Cygwin, I produced a release that didn't
>> have support for HTTPS URLs due to a missing dependency in my build
>> environment. The build and test suite all passed -- it assumed I just
>> wanted to build a release that didn't have HTTPS support -- so some
>> relatively critical function was silently skipped. I don't know how to
>> avoid that sort of issue other than relying on (a) user bug (or at
>> least missing function) reports and (b) folk building Git for
>> themselves/others periodically going through the output of the
>> configure scripts and the skipped subtests to make sure only expected
>> things get missed; neither of those options seem great to me.
>
>I agree with you that there needs a good way to enumerate what the
>unsatisfied prerequisites for a particular build are.  That would
>have helped in your HTTPS situation.
>

Sorry for not replying earlier. I've been sick the last couple of days and only slowly getting up to speed again. I will improve the prereq tests in a new commit in the other patch series still in progress that i'll shortly reroll.

As for the general prereq issue i ran into that as well during development. When you depend on other patches / a specific version of ssh-keygen for git I always have to remember to set the path correctly or the tests might silently be ignored by the missing prereq. Usually not a problem for single test runs, but when i run the full suite before sending something.

So, here's a simple rfc patch to maybe start with addressing this issue.

From 0e7e57e546ec969d31094405aecafd1b1f3cf4d8 Mon Sep 17 00:00:00 2001
From: Fabian Stelzer <fs@gigacodes.de>
Date: Fri, 12 Nov 2021 16:41:30 +0100
Subject: [RFC PATCH 1/2] test-lib: show failed prereq summary

Add failed prereqs to the test results. Aggregate and then show them with the totals.

Signed-off-by: Fabian Stelzer <fs@gigacodes.de>
---
 t/aggregate-results.sh | 12 ++++++++++++
 t/test-lib.sh          |  4 ++++
 2 files changed, 16 insertions(+)

diff --git a/t/aggregate-results.sh b/t/aggregate-results.sh
index 7913e206ed..ad531cc75d 100755
--- a/t/aggregate-results.sh
+++ b/t/aggregate-results.sh
@@ -6,6 +6,7 @@ success=0
 failed=0
 broken=0
 total=0
+missing_prereq=
 
 while read file
 do
@@ -30,10 +31,21 @@ do
 			broken=$(($broken + $value)) ;;
 		total)
 			total=$(($total + $value)) ;;
+		missing_prereq)
+			missing_prereq="$missing_prereq $value" ;;
 		esac
 	done <"$file"
 done
 
+if test -n "$missing_prereq"
+then
+	unique_missing_prereq=$(
+		echo $missing_prereq | tr -s "," | \
+		sed -e 's/ //g' -e 's/^,//' -e 's/,$//' -e 's/,/\n/g' \
+		| sort | uniq | paste -s -d ',')
+	printf "\nmissing prereq: $unique_missing_prereq\n\n"
+fi
+
 if test -n "$failed_tests"
 then
 	printf "\nfailed test(s):$failed_tests\n\n"
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 2679a7596a..472387afec 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -669,6 +669,8 @@ test_fixed=0
 test_broken=0
 test_success=0
 
+test_missing_prereq=
+
 test_external_has_tap=0
 
 die () {
@@ -1068,6 +1070,7 @@ test_skip () {
 		then
 			of_prereq=" of $test_prereq"
 		fi
+		test_missing_prereq="$missing_prereq,$test_missing_prereq"
 		skipped_reason="missing $missing_prereq${of_prereq}"
 	fi
 
@@ -1175,6 +1178,7 @@ test_done () {
 		fixed $test_fixed
 		broken $test_broken
 		failed $test_failure
+		missing_prereq $test_missing_prereq
 
 		EOF
 	fi

-- 
2.31.1



From d13d4c8ccbd832e1d62044b18b8b771f6586ee2a Mon Sep 17 00:00:00 2001
From: Fabian Stelzer <fs@gigacodes.de>
Date: Fri, 12 Nov 2021 16:43:18 +0100
Subject: [RFC PATCH 2/2] test-lib: introduce required prereq for test runs

Allows setting GIT_TEST_REQUIRE_PREREQ to a number of prereqs that must
succeed for this run. Otherwise the test run will abort.

Signed-off-by: Fabian Stelzer <fs@gigacodes.de>
---
 t/test-lib-functions.sh | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index eef2262a36..d65995cd15 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -669,6 +669,14 @@ test_have_prereq () {
 			satisfied_this_prereq=t
 			;;
 		*)
+			if ! test -z $GIT_TEST_REQUIRE_PREREQ
+			then
+				case ",$GIT_TEST_REQUIRE_PREREQ," in
+				*,$prerequisite,*)
+					error "required prereq $prerequisite failed"
+					;;
+				esac
+			fi
 			satisfied_this_prereq=
 		esac
 
-- 
2.31.1