git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 07/12] ci: optionally mark up output in the GitHub workflow

From
Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com>
Date
May 21, 2022, 22:18 UTC
Message-ID
<98059b94a881485f763efc54346e8b19103ee64b.1653171536.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.1117.v3.git.1653171536.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>

A couple of commands exist to spruce up the output in GitHub workflows: https://docs.github.com/en/actions/learn-github-actions/workflow-commands-for-github-actions

In addition to the `::group::<label>`/`::endgroup::` commands (which we already use to structure the output of the build step better), we also use `::error::`/`::notice::` to draw the attention to test failures and to test cases that were expected to fail but didn't.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/test-lib-functions.sh              |  4 +--
 t/test-lib-github-workflow-markup.sh | 50 ++++++++++++++++++++++++++++
 t/test-lib.sh                        |  5 ++-
 3 files changed, 56 insertions(+), 3 deletions(-)
 create mode 100644 t/test-lib-github-workflow-markup.sh
diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 93c03380d44..af4831a54c6 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -795,7 +795,7 @@ test_verify_prereq () {
 }
 
 test_expect_failure () {
-	test_start_
+	test_start_ "$@"
 	test "$#" = 3 && { test_prereq=$1; shift; } || test_prereq=
 	test "$#" = 2 ||
 	BUG "not 2 or 3 parameters to test-expect-failure"
@@ -815,7 +815,7 @@ test_expect_failure () {
 }
 
 test_expect_success () {
-	test_start_
+	test_start_ "$@"
 	test "$#" = 3 && { test_prereq=$1; shift; } || test_prereq=
 	test "$#" = 2 ||
 	BUG "not 2 or 3 parameters to test-expect-success"
diff --git a/t/test-lib-github-workflow-markup.sh b/t/test-lib-github-workflow-markup.sh
new file mode 100644
index 00000000000..d8dc969df4a
--- /dev/null
+++ b/t/test-lib-github-workflow-markup.sh
@@ -0,0 +1,50 @@
+# Library of functions to mark up test scripts' output suitable for
+# pretty-printing it in GitHub workflows.
+#
+# Copyright (c) 2022 Johannes Schindelin
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see http://www.gnu.org/licenses/ .
+#
+# The idea is for `test-lib.sh` to source this file when run in GitHub
+# workflows; these functions will then override (empty) functions
+# that are are called at the appropriate times during the test runs.
+
+start_test_output () {
+	test -n "$GIT_TEST_TEE_OUTPUT_FILE" ||
+	die "--github-workflow-markup requires --verbose-log"
+	github_markup_output="${GIT_TEST_TEE_OUTPUT_FILE%.out}.markup"
+	>$github_markup_output
+	GIT_TEST_TEE_OFFSET=0
+}
+
+# No need to override start_test_case_output
+
+finalize_test_case_output () {
+	test_case_result=$1
+	shift
+	case "$test_case_result" in
+	failure)
+		echo >>$github_markup_output "::error::failed: $this_test.$test_count $1"
+		;;
+	fixed)
+		echo >>$github_markup_output "::notice::fixed: $this_test.$test_count $1"
+		;;
+	esac
+	echo >>$github_markup_output "::group::$test_case_result: $this_test.$test_count $*"
+	test-tool >>$github_markup_output path-utils skip-n-bytes \
+		"$GIT_TEST_TEE_OUTPUT_FILE" $GIT_TEST_TEE_OFFSET
+	echo >>$github_markup_output "::endgroup::"
+}
+
+# No need to override finalize_test_output
diff --git a/t/test-lib.sh b/t/test-lib.sh
index bdb11e28eea..29640d107ca 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -204,6 +204,9 @@ parse_option () {
 	--write-junit-xml)
 		. "$TEST_DIRECTORY/test-lib-junit.sh"
 		;;
+	--github-workflow-markup)
+		. "$TEST_DIRECTORY/test-lib-github-workflow-markup.sh"
+		;;
 	--stress)
 		stress=t ;;
 	--stress=*)
@@ -1082,7 +1085,7 @@ test_start_ () {
 	test_count=$(($test_count+1))
 	maybe_setup_verbose
 	maybe_setup_valgrind
-	start_test_case_output
+	start_test_case_output "$@"
 }
 
 test_finish_ () {
-- 
gitgitgadget
Previous: Johannes Schindelin via GitGitGadgetNext: Johannes Schindelin via GitGitGadget
Message 92 of 98 in “ci: make Git's GitHub workflow output much more helpful”
  1. 0/9 ci: make Git's GitHub workflow output much more helpfulJohannes Schindelin via GitGitGadget, Jan 24, 2022
  2. 1/9 ci: fix code styleJohannes Schindelin via GitGitGadget, Jan 24, 2022
  3. 2/9 ci/run-build-and-tests: take a more high-level viewJohannes Schindelin via GitGitGadget, Jan 24, 2022
  4. Eric SunshineJan 24, 2022
  5. Johannes SchindelinJan 25, 2022
  6. 3/9 ci: make it easier to find failed tests' logs in the GitHub workflowJohannes Schindelin via GitGitGadget, Jan 24, 2022
  7. Ævar Arnfjörð BjarmasonJan 25, 2022
  8. 4/9 ci/run-build-and-tests: add some structure to the GitHub workflow outputJohannes Schindelin via GitGitGadget, Jan 24, 2022
  9. Phillip WoodFeb 23, 2022
  10. Johannes SchindelinFeb 25, 2022
  11. 5/9 tests: refactor --write-junit-xml codeJohannes Schindelin via GitGitGadget, Jan 24, 2022
  12. Ævar Arnfjörð BjarmasonJan 26, 2022
  13. 6/9 test(junit): avoid line feeds in XML attributesJohannes Schindelin via GitGitGadget, Jan 24, 2022
  14. 9/9 ci: call `finalize_test_case_output` a little laterJohannes Schindelin via GitGitGadget, Jan 24, 2022
  15. 8/9 ci: use `--github-workflow-markup` in the GitHub workflowJohannes Schindelin via GitGitGadget, Jan 24, 2022
  16. 7/9 ci: optionally mark up output in the GitHub workflowJohannes Schindelin via GitGitGadget, Jan 24, 2022
  17. Ævar Arnfjörð BjarmasonJan 26, 2022
  18. CI "grouping" within jobs v.s. lighter split-out jobs (was: [PATCH 0/9] ci: make Git's GitHub workflow output much more helpful)Ævar Arnfjörð Bjarmason, Jan 27, 2022
  19. Johannes SchindelinFeb 19, 2022
  20. Junio C HamanoFeb 20, 2022
  21. Johannes SchindelinFeb 20, 2022
  22. Ævar Arnfjörð BjarmasonFeb 21, 2022
  23. Johannes SchindelinFeb 22, 2022
  24. Ævar Arnfjörð BjarmasonFeb 20, 2022
  25. Johannes SchindelinFeb 22, 2022
  26. Ævar Arnfjörð BjarmasonFeb 22, 2022
  27. Phillip WoodFeb 23, 2022
  28. Ævar Arnfjörð BjarmasonFeb 25, 2022
  29. Johannes SchindelinFeb 25, 2022
  30. Junio C HamanoFeb 25, 2022
  31. Junio C HamanoFeb 26, 2022
  32. Junio C HamanoMar 1, 2022
  33. Junio C HamanoMar 1, 2022
  34. Johannes SchindelinMar 1, 2022
  35. Junio C HamanoMar 1, 2022
  36. Johannes SchindelinMar 1, 2022
  37. Junio C HamanoMar 1, 2022
  38. Johannes SchindelinMar 1, 2022
  39. win+VS environment has "cut" but not "paste"?Junio C Hamano, Mar 4, 2022
  40. Ævar Arnfjörð BjarmasonMar 4, 2022
  41. Johannes SchindelinMar 7, 2022
  42. Junio C HamanoMar 7, 2022
  43. Johannes SchindelinMar 9, 2022
  44. Ævar Arnfjörð BjarmasonMar 10, 2022
  45. Johannes SchindelinMar 7, 2022
  46. Junio C HamanoMar 7, 2022
  47. Phillip WoodMar 2, 2022
  48. Johannes SchindelinMar 7, 2022
  49. Junio C HamanoMar 7, 2022
  50. Ævar Arnfjörð BjarmasonMar 9, 2022
  51. Phillip WoodMar 7, 2022
  52. 0/9 ci: make Git's GitHub workflow output much more helpfulJohannes Schindelin via GitGitGadget, Mar 1, 2022
  53. 1/9 ci: fix code styleJohannes Schindelin via GitGitGadget, Mar 1, 2022
  54. 2/9 ci/run-build-and-tests: take a more high-level viewJohannes Schindelin via GitGitGadget, Mar 1, 2022
  55. 3/9 ci: make it easier to find failed tests' logs in the GitHub workflowJohannes Schindelin via GitGitGadget, Mar 1, 2022
  56. 4/9 ci/run-build-and-tests: add some structure to the GitHub workflow outputJohannes Schindelin via GitGitGadget, Mar 1, 2022
  57. 5/9 tests: refactor --write-junit-xml codeJohannes Schindelin via GitGitGadget, Mar 1, 2022
  58. 6/9 test(junit): avoid line feeds in XML attributesJohannes Schindelin via GitGitGadget, Mar 1, 2022
  59. 7/9 ci: optionally mark up output in the GitHub workflowJohannes Schindelin via GitGitGadget, Mar 1, 2022
  60. 8/9 ci: use `--github-workflow-markup` in the GitHub workflowJohannes Schindelin via GitGitGadget, Mar 1, 2022
  61. 9/9 ci: call `finalize_test_case_output` a little laterJohannes Schindelin via GitGitGadget, Mar 1, 2022
  62. Junio C HamanoMar 1, 2022
  63. Ævar Arnfjörð BjarmasonMar 2, 2022
  64. Johannes SchindelinMar 7, 2022
  65. Ævar Arnfjörð BjarmasonMar 7, 2022
  66. Junio C HamanoMar 7, 2022
  67. Ævar Arnfjörð BjarmasonMar 9, 2022
  68. Johannes SchindelinMar 9, 2022
  69. Junio C HamanoMar 9, 2022
  70. Ævar Arnfjörð BjarmasonMar 9, 2022
  71. Victoria DyeMar 25, 2022
  72. Ævar Arnfjörð BjarmasonMar 25, 2022
  73. Victoria DyeMar 25, 2022
  74. Johannes SchindelinMay 21, 2022
  75. Junio C HamanoMay 21, 2022
  76. Johannes SchindelinMay 22, 2022
  77. Junio C HamanoMay 22, 2022
  78. Johannes SchindelinMay 23, 2022
  79. Junio C HamanoMay 22, 2022
  80. Junio C HamanoMay 23, 2022
  81. Johannes SchindelinMay 23, 2022
  82. Ævar Arnfjörð BjarmasonMay 23, 2022
  83. Johannes SchindelinMay 23, 2022
  84. Ævar Arnfjörð BjarmasonMay 24, 2022
  85. 00/12 ci: make Git's GitHub workflow output much more helpfulJohannes Schindelin via GitGitGadget, May 21, 2022
  86. 01/12 ci: fix code styleJohannes Schindelin via GitGitGadget, May 21, 2022
  87. 02/12 tests: refactor --write-junit-xml codeJohannes Schindelin via GitGitGadget, May 21, 2022
  88. 03/12 test(junit): avoid line feeds in XML attributesJohannes Schindelin via GitGitGadget, May 21, 2022
  89. 04/12 ci/run-build-and-tests: take a more high-level viewJohannes Schindelin via GitGitGadget, May 21, 2022
  90. 05/12 ci: make it easier to find failed tests' logs in the GitHub workflowJohannes Schindelin via GitGitGadget, May 21, 2022
  91. 06/12 ci/run-build-and-tests: add some structure to the GitHub workflow outputJohannes Schindelin via GitGitGadget, May 21, 2022
  92. 07/12 ci: optionally mark up output in the GitHub workflowJohannes Schindelin via GitGitGadget, May 21, 2022
  93. 08/12 ci(github): skip the logs of the successful test casesJohannes Schindelin via GitGitGadget, May 21, 2022
  94. Ævar Arnfjörð BjarmasonMay 24, 2022
  95. 10/12 ci: use `--github-workflow-markup` in the GitHub workflowJohannes Schindelin via GitGitGadget, May 21, 2022
  96. 09/12 ci(github): avoid printing test case preamble twiceVictoria Dye via GitGitGadget, May 21, 2022
  97. 12/12 ci: call `finalize_test_case_output` a little laterJohannes Schindelin via GitGitGadget, May 21, 2022
  98. 11/12 ci(github): mention where the full logs can be foundJohannes Schindelin via GitGitGadget, May 21, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.