From: Ævar Arnfjörð Bjarmason Date: Fri, 18 Mar 2022 00:33:56 GMT Subject: [PATCH 1/7] test-lib: add a "test_expect_todo", similar to "test_expect_failure" Message-ID: In-Reply-To: Add an alternative to the "test_expect_failure" function. Like "test_expect_failure" it will marks a test as "not ok ... TODO" in the TAP output, and thus document that it's a "TODO" test that fails currently. Unlike "test_expect_failure" it asserts that the tested code in exactly one manner, and not any other. We'll thus stop conflating e.g. segfaults with other sorts of errors, and generally be able to tell if our "expected to fail" tests start failing in new and unexpected ways. In more detail, the these problems of "test_expect_failure" are solved by this new function. * When "test_expect_failure" is used in combination with "test_{must,might}_fail" it will hide segfaults or abort() failures, such as failures due to LSAN. This is because we do do the right thing in those helpers, but they themselves are expected to be used within "test_expect_success" and return 1. The "test_expect_failure" can't differentiate this from a "real" failure. We could change "test_{must,might}_fail" to appropriately carry forward the status code to work around this specific issue, but doing so would be a large semantic change in the current test suite. * More generally, "test_expect_failure" by design doesn't test what does, but just asserts that the test fails in some way. For some tests that truly fail in unpredictable ways this behavior makes sense, but it's almost never what our tests want. We know we e.g. have a revision at HEAD~ instead of HEAD, and would like to know conflate that with a potential segfault or other behavior change. In summary, this new function behaves exactly like "test_expect_success", except that its "success" is then reported as a "not ok .. TODO". Let's convert a few "test_expect_failure" uses to the new "test_expect_todo". For previous discussion on this topic see [1] and [2], in particular the points by Junio that it's desired that the "test_expect_failure" output differentiate the TODO tests from "test_expect_success". 1. https://lore.kernel.org/git/87tuhmk19c.fsf@evledraar.gmail.com/ 2. https://lore.kernel.org/git/xmqqo824e145.fsf@gitster.g/ Signed-off-by: Ævar Arnfjörð Bjarmason --- t/README | 20 +++++++++++++++++++- t/t0000-basic.sh | 11 ++++++++--- t/t1060-object-corruption.sh | 4 ++-- t/t7815-grep-binary.sh | 4 ++-- t/test-lib-functions.sh | 28 ++++++++++++++++++++++++---- t/test-lib.sh | 32 ++++++++++++++++++++++++++++---- 6 files changed, 83 insertions(+), 16 deletions(-) diff --git a/t/README b/t/README index f48e0542cdc..e0aa8ebb0eb 100644 --- a/t/README +++ b/t/README @@ -805,10 +805,28 @@ see test-lib-functions.sh for the full list and their options. test_expect_success PERL,PYTHON 'yo dawg' \ ' test $(perl -E 'print eval "1 +" . qx[python -c "print 2"]') == "4" ' + - test_expect_todo []