Re: [RFC/PATCH] WIP: Report intra-test progress with TAP subtests

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Aug 11, 2010, 00:44 UTC
Message-ID: <20100811004425.GI2099@burratino>
In-Reply-To: <1281473829-2102-1-git-send-email-avarab@gmail.com>

Hi Ævar,

Ævar Arnfjörð Bjarmason wrote:

Show 10 quoted lines

>    ok 1 - A git test
>        ok 1 - doing test -f file
>        ok 2 - git commit ...
>        ok 3 - test_tick...
>        1..3
>    ok 2 - A git test
>    1..2
> 
> Here's an attempt at that, I've convented test_commit, test_merge and
> test_cmp to report intra-test progress.

What if a test uses none of those commands?

The ultimate conclusion of such an approach would be to use replacements for common commands like “git” and “cd”, making tests a lot harder to read and write for the uninitiated.

If we get peeks where the infrastructure has a chance to provide them, maybe it would be nice to allow explicitly inserting such checkpoints, too? Like:

	mv init init2 &&
	checkpoint &&
	git config -f .gitmodules submodule.example.url "$(pwd)/init2" &&
	checkpoint &&
	git config --remove-section submodule.example &&
	checkpoint &&
	[...]

while debugging.

However, I would rather see those inserted automatically. Two ideas without code to back them up:

1. Script parsing.

Yes, parsing shell command language is hard, but tests use only a subset of it. In particular the &&-chaining means it might be relatively easy to find where each command starts and ends.

The hard bits: “if”, “while”, and “for” flow control constructs.

The payoff of even partial progress in this would be very high, since it should make static analysis of &&-chaining possible.

2. Letting the shell trace for us.

When tests fail, I generally use

	sh -x ./failing-test.sh -v -i

and scroll back through the log to the important part.

Maybe the test harness could automate that a little, by using set -x at the beginning and set +x at the end of each test.

Hope that helps, Jonathan