Re: [RFC/PATCH] WIP: Report intra-test progress with TAP subtests
- From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
- Date
- Aug 11, 2010, 00:55 UTC
- Message-ID
- <AANLkTi=ygXFpRLR_Z7iEXX3THh+GT7P7X9o+NnaLajqs@mail.gmail.com>
- In-Reply-To
- <20100811004425.GI2099@burratino>
On Wed, Aug 11, 2010 at 00:44, Jonathan Nieder <jrnieder@gmail.com> wrote:
Show 16 quoted lines
> Hi Ævar, > > Ævar Arnfjörð Bjarmason wrote: > >> ok 1 - A git test >> ok 1 - doing test -f file >> ok 2 - git commit ... >> ok 3 - test_tick... >> 1..3 >> ok 2 - A git test >> 1..2 >> >> Here's an attempt at that, I've convented test_commit, test_merge and >> test_cmp to report intra-test progress. > > What if a test uses none of those commands?
Then it won't get any progress output.
> The ultimate conclusion of such an approach would be to use > replacements for common commands like “git” and “cd”, making tests a > lot harder to read and write for the uninitiated.
Yes, but there's a middle ground here. We use a lot of utility functions in the tests, adding tracing to these makes it easier to see what's going on, especially in large tests.
Show 13 quoted lines
> If we get peeks where the infrastructure has a chance to provide > them, maybe it would be nice to allow explicitly inserting such > checkpoints, too? Like: > > mv init init2 && > checkpoint && > git config -f .gitmodules submodule.example.url "$(pwd)/init2" && > checkpoint && > git config --remove-section submodule.example && > checkpoint && > [...] > > while debugging.
Yeah. I was going to add something like that if I got this working.
Show 13 quoted lines
> However, I would rather see those inserted automatically. > Two ideas without code to back them up: > > 1. Script parsing. > > Yes, parsing shell command language is hard, but tests use only > a subset of it. In particular the &&-chaining means it might > be relatively easy to find where each command starts and ends. > > The hard bits: “if”, “while”, and “for” flow control constructs. > > The payoff of even partial progress in this would be very high, > since it should make static analysis of &&-chaining possible.
Sounds like a good long term project, and shouldn't be so hard for the simple case.
Show 10 quoted lines
> 2. Letting the shell trace for us. > > When tests fail, I generally use > > sh -x ./failing-test.sh -v -i > > and scroll back through the log to the important part. > > Maybe the test harness could automate that a little, by using > set -x at the beginning and set +x at the end of each test.
The main use case for this for me is the smoke tester output, where you won't get access to the original machine (without asking). Having a few intra-test checkpoints helps you to see where things might have gone wrong.