git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 06/12] t: prepare execution of potentially failing commands for `set -e`

From
Jeff King <peff@peff.net>
Date
Apr 18, 2026, 08:01 UTC
Message-ID
<20260418080111.GA3187789@coredump.intra.peff.net>
In-Reply-To
<xmqqtstb3rf1.fsf@gitster.g>
On Thu, Apr 16, 2026 at 07:34:26AM -0700, Junio C Hamano wrote:
Show 21 quoted lines
> Jeff King <peff@peff.net> writes:
> 
> > I'm still concerned that this approach is going to create extra friction
> > for test writers down the road. This series needed to clean up several
> > spots to avoid false positives, and some of the spots were non-trivial.
> >
> > Now that was the accumulated cruft of 20 years of test-writing, so it's
> > not clear to me how often new test-writers will run into this. But when
> > they do, I worry that it may be hard to even figure out what is going
> > on.
> >
> > But I've said as much in earlier rounds, and I'm not sure Junio agrees.
> > So we can note my dissent in the captain's log, and I can reserve the
> > right to told-you-so later if need be. ;)
> 
> The alternatigve to allow us to be sloppy is alluring from the point
> of view of a test writer in me.  But do we have an easy/canned way
> to run tests and see the unexpected failures outside test_expect_foo
> while ignoring all the noises from passing tests?  Perhaps running
> tests (with and without prove) while redirecting the standard output
> stream to /dev/null or something?

I took a stab at this. It's easy to do it hackily, but it proved a little trickier than I'd hoped to get something elegant.

My main goals were that you would still get good output to either the terminal or the test-results/*.out files (so we can't just redirect stderr for all runs), and that we wouldn't spend too much extra CPU (so just running the tests an extra time with stderr redirected is right out).

My first attempt was to teach test-lib.sh a mode where all of the test snippets are noops, stderr goes to a file in test-results/, and at the end of the script we complain if our stderr file is non-empty. And then we have a "test-lint-stderr" make target that runs each script in the special mode. That works, but:

  1. Quite a few scripts do stuff inside their test snippets that affect
     the environment, and then do stuff outside of a test snippet that
     depends on that. Some of it is questionable, like running git
     commands outside of test snippets to generate expected output. But
     there's reasonable stuff like t0012 generating the list of builtin
     inside a test snippet and expecting:
       while read builtin
       do
               test_expect_success "$builtin..." ...
       done <builtins
     So I think the test suite is a little too free-form for this kind
     of trickery.
  2. It's way too slow, anyway. Even with all of the snippets as noops,
     it still takes ~13s to run all of the scripts in parallel on my
     machine. The real test suite only takes ~90 seconds! There's
     probably low-hanging fruit to fix there, but it's still an order of
     magnitude off what I'd hope the cost would be.
So I think that's probably a dead end.

The next obvious choice is redirecting stderr via tee or similar, which is easy-ish. We already do it for --verbose-log, but we combine it with stdout there (and we turn on verbose mode for the tests, which is going to generate a bunch of extra output).

I think there's probably a way to do this with an extra tee for stderr. But actually...do we even need tee? The idea is that we are not expecting any output here, so if there is any, we'd bail and show it along with an error.

So I think we might able able to get by with just redirecting descriptor 2, and then making sure that test snippets stderr goes to the original stderr (which we already save as descriptor 7). And we catch only errors outside of the snippet.

Something like this seems to work:
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 1f7868c537..6657d56e52 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -728,7 +728,7 @@ then
 	exec 3>>"$GIT_TEST_TEE_OUTPUT_FILE" 4>&3
 elif test "$verbose" = "t"
 then
-	exec 4>&2 3>&2
+	exec 4>&7 3>&7
 else
 	exec 4>/dev/null 3>/dev/null
 fi
@@ -970,7 +970,7 @@ maybe_setup_verbose () {
 	test -z "$verbose_only" && return
 	if match_pattern_list $test_count "$verbose_only"
 	then
-		exec 4>&2 3>&2
+		exec 4>&7 3>&7
 		# Emit a delimiting blank line when going from
 		# non-verbose to verbose.  Within verbose mode the
 		# delimiter is printed by test_expect_*.  The choice
@@ -1052,7 +1052,7 @@ test_eval_ () {
 			test 1 = $trace_level_ && set +x
 			trace_level_=$(($trace_level_-1))
 		fi
-	} 2>/dev/null 4>&2
+	} 2>/dev/null 4>&7
 
 	if test "$test_eval_ret_" != 0 && want_trace
 	then
@@ -1234,6 +1234,14 @@ test_done () {
 		EOF
 	fi
 
+	if test -s "$TEST_RESULTS_BASE.stderr"
+	then
+		say_color >&5 error "FATAL: Unexpected output on stderr"
+		sed >&5 's/^/# /' "$TEST_RESULTS_BASE.stderr"
+		GIT_EXIT_OK=1
+		exit 1
+	fi
+
 	if test "$test_fixed" != 0
 	then
 		say_color error "# $test_fixed known breakage(s) vanished; please update test(s)"
@@ -1982,3 +1990,9 @@ test_lazy_prereq FSMONITOR_DAEMON '
 	git version --build-options >output &&
 	grep "feature: fsmonitor--daemon" output
 '
+
+# Now that test-lib setup is done, direct our stderr to a file, as we want to
+# catch and complain if anything ends up here. We can always access the
+# original via descriptor 7.
+mkdir -p "$TEST_RESULTS_DIR"
+exec 2>"$TEST_RESULTS_BASE.stderr"


If I stick a "test_expect_foobar" into a test script, it is found:

  $ ./t0001-init.sh
  ok 1 - plain
  [...]
  ok 102 - re-init reads matching includeIf.onbranch
  FATAL: Unexpected output on stderr
  # ./t0001-init.sh: 983: test_expect_foobar: not found

And if I run scripts with "-v" or "-x", that output still goes to the
terminal and does not trigger the stderr checker. Likewise with
--verbose-log, the log gets the verbose bits.

It _almost_ passes a full "make test", but there's one lingering
problem: random bits of code which want to output to stderr and bail,
and use "echo >&2" to do so. But with our redirect, that stderr goes to
the log file. t0000-basic notices this, because it is testing the test
suite itself. And it notices that:

  ./t0000-basic.sh --run=bogus

should complain to stderr, but now doesn't (it's swallowed by our log
file). We can fix that with;

diff --git a/t/test-lib.sh b/t/test-lib.sh
index 6657d56e52..aa6415b308 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -902,13 +902,13 @@ match_test_selector_list () {
 				if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
 				then
 					echo "error: $operation: invalid non-numeric in range" \
-						"start: '$orig_selector'" >&2
+						"start: '$orig_selector'" >&7
 					exit 1
 				fi
 				if expr "z${selector#*-}" : "z[0-9]*[^0-9]" >/dev/null
 				then
 					echo "error: $operation: invalid non-numeric in range" \
-						"end: '$orig_selector'" >&2
+						"end: '$orig_selector'" >&7
 					exit 1
 				fi
 				;;

but the same problem doubtless exists elsewhere. We could fix that
globally like this:

diff --git a/t/test-lib.sh b/t/test-lib.sh
index 6657d56e52..72d60d9dac 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -762,6 +762,15 @@ die () {
 	# test script run with '--immediate' fails, or when the user hits
 	# ctrl-C, i.e. when 'test_done' is not invoked at all.
 	test_atexit_handler || code=$?
+
+	# Dump the stderr log if we saw anything, since otherwise random
+	# errors will never make it to the user when we do not have
+	# a clean exit.
+	if test -s "$TEST_RESULTS_BASE.stderr"
+	then
+		cat >&7 "$TEST_RESULTS_BASE.stderr"
+	fi
+
 	if test -n "$GIT_EXIT_OK"
 	then
 		exit $code

It's a little weird because the stderr message is delayed until we cat
it out, but it makes sense. If we are bailing immediately, then the time
between "echo" and "cat" is small. And if we are not bailing, then that
is exactly the kind of bug our stderr log is trying to catch (and we
will complain during test_done).

I am not quite sure if this is an elegant solution or a terrible hack.

-Peff
Previous: Junio C HamanoNext: Junio C Hamano
Message 35 of 95 in “t: detect errors outside of test cases”
  1. 00/12 t: detect errors outside of test casesPatrick Steinhardt, Apr 13, 2026
  2. 01/12 t: prepare `test_match_signal ()` calls for `set -e`Patrick Steinhardt, Apr 13, 2026
  3. Junio C HamanoApr 13, 2026
  4. Patrick SteinhardtApr 14, 2026
  5. Junio C HamanoApr 14, 2026
  6. 02/12 t: prepare `test_must_fail ()` for `set -e`Patrick Steinhardt, Apr 13, 2026
  7. Junio C HamanoApr 13, 2026
  8. Patrick SteinhardtApr 14, 2026
  9. Jeff KingApr 14, 2026
  10. Junio C HamanoApr 14, 2026
  11. Patrick SteinhardtApr 15, 2026
  12. Jeff KingApr 16, 2026
  13. 03/12 t: prepare `stop_git_daemon ()` for `set -e`Patrick Steinhardt, Apr 13, 2026
  14. Junio C HamanoApr 13, 2026
  15. 04/12 t: prepare `git config --unset` calls for `set -e`Patrick Steinhardt, Apr 13, 2026
  16. Junio C HamanoApr 13, 2026
  17. 05/12 t: prepare conditional test execution for `set -e`Patrick Steinhardt, Apr 13, 2026
  18. Junio C HamanoApr 13, 2026
  19. 06/12 t: prepare execution of potentially failing commands for `set -e`Patrick Steinhardt, Apr 13, 2026
  20. Junio C HamanoApr 13, 2026
  21. Patrick SteinhardtApr 14, 2026
  22. Junio C HamanoApr 14, 2026
  23. Junio C HamanoApr 13, 2026
  24. Junio C HamanoApr 14, 2026
  25. Patrick SteinhardtApr 14, 2026
  26. Patrick SteinhardtApr 14, 2026
  27. Junio C HamanoApr 14, 2026
  28. Jeff KingApr 14, 2026
  29. Jeff KingApr 14, 2026
  30. Jeff KingApr 14, 2026
  31. Patrick SteinhardtApr 15, 2026
  32. Jeff KingApr 16, 2026
  33. Patrick SteinhardtApr 16, 2026
  34. Junio C HamanoApr 16, 2026
  35. Jeff KingApr 18, 2026
  36. Junio C HamanoApr 15, 2026
  37. 07/12 t: prepare `test_when_finished ()`/`test_atexit()` for `set -e`Patrick Steinhardt, Apr 13, 2026
  38. Junio C HamanoApr 13, 2026
  39. Patrick SteinhardtApr 14, 2026
  40. 08/12 t0008: silence error in subshell when using `grep -v`Patrick Steinhardt, Apr 13, 2026
  41. Junio C HamanoApr 13, 2026
  42. Patrick SteinhardtApr 14, 2026
  43. 09/12 t1301: don't fail in case setfacl(1) doesn't exist or failsPatrick Steinhardt, Apr 13, 2026
  44. 10/12 t6002: fix use of `expr` with `set -e`Patrick Steinhardt, Apr 13, 2026
  45. 11/12 t9902: fix use of `read` with `set -e`Patrick Steinhardt, Apr 13, 2026
  46. 12/12 t: detect errors outside of test casesPatrick Steinhardt, Apr 13, 2026
  47. Junio C HamanoApr 13, 2026
  48. Junio C HamanoApr 13, 2026
  49. Junio C HamanoApr 13, 2026
  50. 00/12 t: detect errors outside of test casesPatrick Steinhardt, Apr 16, 2026
  51. 01/12 t: prepare `test_match_signal ()` calls for `set -e`Patrick Steinhardt, Apr 16, 2026
  52. 02/12 t: prepare `test_must_fail ()` for `set -e`Patrick Steinhardt, Apr 16, 2026
  53. 03/12 t: prepare `stop_git_daemon ()` for `set -e`Patrick Steinhardt, Apr 16, 2026
  54. 04/12 t: prepare `git config --unset` calls for `set -e`Patrick Steinhardt, Apr 16, 2026
  55. 05/12 t: prepare conditional test execution for `set -e`Patrick Steinhardt, Apr 16, 2026
  56. 06/12 t: prepare execution of potentially failing commands for `set -e`Patrick Steinhardt, Apr 16, 2026
  57. 07/12 t: prepare `test_when_finished ()`/`test_atexit()` for `set -e`Patrick Steinhardt, Apr 16, 2026
  58. 08/12 t0008: silence error in subshell when using `grep -v`Patrick Steinhardt, Apr 16, 2026
  59. 09/12 t1301: don't fail in case setfacl(1) doesn't exist or failsPatrick Steinhardt, Apr 16, 2026
  60. 10/12 t6002: fix use of `expr` with `set -e`Patrick Steinhardt, Apr 16, 2026
  61. 11/12 t9902: fix use of `read` with `set -e`Patrick Steinhardt, Apr 16, 2026
  62. SZEDER GáborApr 16, 2026
  63. Junio C HamanoApr 16, 2026
  64. Patrick SteinhardtApr 17, 2026
  65. 12/12 t: detect errors outside of test casesPatrick Steinhardt, Apr 16, 2026
  66. Junio C HamanoApr 16, 2026
  67. 00/12 t: detect errors outside of test casesPatrick Steinhardt, Apr 20, 2026
  68. 01/12 t: prepare `test_match_signal ()` calls for `set -e`Patrick Steinhardt, Apr 20, 2026
  69. 02/12 t: prepare `test_must_fail ()` for `set -e`Patrick Steinhardt, Apr 20, 2026
  70. 03/12 t: prepare `stop_git_daemon ()` for `set -e`Patrick Steinhardt, Apr 20, 2026
  71. 04/12 t: prepare `git config --unset` calls for `set -e`Patrick Steinhardt, Apr 20, 2026
  72. 05/12 t: prepare conditional test execution for `set -e`Patrick Steinhardt, Apr 20, 2026
  73. 06/12 t: prepare execution of potentially failing commands for `set -e`Patrick Steinhardt, Apr 20, 2026
  74. 07/12 t: prepare `test_when_finished ()`/`test_atexit()` for `set -e`Patrick Steinhardt, Apr 20, 2026
  75. 08/12 t0008: silence error in subshell when using `grep -v`Patrick Steinhardt, Apr 20, 2026
  76. 09/12 t1301: don't fail in case setfacl(1) doesn't exist or failsPatrick Steinhardt, Apr 20, 2026
  77. 10/12 t6002: fix use of `expr` with `set -e`Patrick Steinhardt, Apr 20, 2026
  78. 11/12 t9902: fix use of `read` with `set -e`Patrick Steinhardt, Apr 20, 2026
  79. 12/12 t: detect errors outside of test casesPatrick Steinhardt, Apr 20, 2026
  80. Junio C HamanoApr 20, 2026
  81. Jeff KingApr 21, 2026
  82. Patrick SteinhardtApr 21, 2026
  83. 00/12 t: detect errors outside of test casesPatrick Steinhardt, Apr 21, 2026
  84. 01/12 t: prepare `test_match_signal ()` calls for `set -e`Patrick Steinhardt, Apr 21, 2026
  85. 02/12 t: prepare `test_must_fail ()` for `set -e`Patrick Steinhardt, Apr 21, 2026
  86. 03/12 t: prepare `stop_git_daemon ()` for `set -e`Patrick Steinhardt, Apr 21, 2026
  87. 04/12 t: prepare `git config --unset` calls for `set -e`Patrick Steinhardt, Apr 21, 2026
  88. 05/12 t: prepare conditional test execution for `set -e`Patrick Steinhardt, Apr 21, 2026
  89. 06/12 t: prepare execution of potentially failing commands for `set -e`Patrick Steinhardt, Apr 21, 2026
  90. 07/12 t: prepare `test_when_finished ()`/`test_atexit()` for `set -e`Patrick Steinhardt, Apr 21, 2026
  91. 08/12 t0008: silence error in subshell when using `grep -v`Patrick Steinhardt, Apr 21, 2026
  92. 09/12 t1301: don't fail in case setfacl(1) doesn't exist or failsPatrick Steinhardt, Apr 21, 2026
  93. 10/12 t6002: fix use of `expr` with `set -e`Patrick Steinhardt, Apr 21, 2026
  94. 11/12 t9902: fix use of `read` with `set -e`Patrick Steinhardt, Apr 21, 2026
  95. 12/12 t: detect errors outside of test casesPatrick Steinhardt, Apr 21, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.