Re: [PATCH] test-lib: catch misspelt 'test_expect_successo'

From: Jeff King <peff@peff.net>
Date: Mar 24, 2026, 19:35 UTC
Message-ID: <20260324193514.GA1870130@coredump.intra.peff.net>
In-Reply-To: <xmqqmrzxyu2h.fsf_-_@gitster.g>

On Tue, Mar 24, 2026 at 11:13:26AM -0700, Junio C Hamano wrote:

> In order to catch mistakes like misspelling "test_expect_success",
> we would like to eventually be able to run our test suite with the
> "-e" option on.

Using "-e" makes me very nervous, given all of its quirks. Granted, most of them are related to it _not_ kicking in when you'd want it to, but I worry it will create false positive/negative headaches.

In the past I've caught errors outside of the test snippet by noticing cruft on stderr. This is especially obvious if you use "prove", which captures stdout and gives a nice display (which the extra stderr then makes uglier).

I wonder if we could automate / formalize that. If we do this hacky patch on master:

diff --git a/t/Makefile b/t/Makefile
index ab8a5b54aa..f57180cc7b 100644
--- a/t/Makefile
+++ b/t/Makefile
@@ -79,7 +79,7 @@ prove: pre-clean $(TEST_LINT)
 	$(MAKE) clean-except-prove-cache
 
 $(T):
-	@echo "*** $@ ***"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)
+	echo "*** $@ ***"; '$(TEST_SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS) 2>$@.stderr
 
 $(UNIT_TESTS):
 	@echo "*** $@ ***"; $@

then:

  cd t
  make test
  for i in *.stderr; do test -s $i && echo $i; done

catches the problem in t4014 and nothing else. Note that it _doesn't_
work with --verbose-log, though, as that redirects stderr to stdout
(which is going to the log). It might be possible to do something
cleaner and more clever within test-lib.sh, though.

-Peff