From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Wed, 11 Aug 2010 00:55:49 GMT
Subject: Re: [RFC/PATCH] WIP: Report intra-test progress with TAP subtests
Message-ID: <AANLkTi=ygXFpRLR_Z7iEXX3THh+GT7P7X9o+NnaLajqs@mail.gmail.com>
In-Reply-To: <20100811004425.GI2099@burratino>

On Wed, Aug 11, 2010 at 00:44, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi Ævar,
>
> Ævar Arnfjörð Bjarmason wrote:
>
>>    ok 1 - A git test
>>        ok 1 - doing test -f file
>>        ok 2 - git commit ...
>>        ok 3 - test_tick...
>>        1..3
>>    ok 2 - A git test
>>    1..2
>>
>> Here's an attempt at that, I've convented test_commit, test_merge and
>> test_cmp to report intra-test progress.
>
> What if a test uses none of those commands?

Then it won't get any progress output.

> The ultimate conclusion of such an approach would be to use
> replacements for common commands like “git” and “cd”, making tests a
> lot harder to read and write for the uninitiated.

Yes, but there's a middle ground here. We use a lot of utility
functions in the tests, adding tracing to these makes it easier to see
what's going on, especially in large tests.

> If we get peeks where the infrastructure has a chance to provide
> them, maybe it would be nice to allow explicitly inserting such
> checkpoints, too?  Like:
>
>        mv init init2 &&
>        checkpoint &&
>        git config -f .gitmodules submodule.example.url "$(pwd)/init2" &&
>        checkpoint &&
>        git config --remove-section submodule.example &&
>        checkpoint &&
>        [...]
>
> while debugging.

Yeah. I was going to add something like that if I got this working.

> However, I would rather see those inserted automatically.
> Two ideas without code to back them up:
>
> 1. Script parsing.
>
> Yes, parsing shell command language is hard, but tests use only
> a subset of it.  In particular the &&-chaining means it might
> be relatively easy to find where each command starts and ends.
>
> The hard bits: “if”, “while”, and “for” flow control constructs.
>
> The payoff of even partial progress in this would be very high,
> since it should make static analysis of &&-chaining possible.

Sounds like a good long term project, and shouldn't be so hard for the
simple case.

> 2. Letting the shell trace for us.
>
> When tests fail, I generally use
>
>        sh -x ./failing-test.sh -v -i
>
> and scroll back through the log to the important part.
>
> Maybe the test harness could automate that a little, by using
> set -x at the beginning and set +x at the end of each test.

The main use case for this for me is the smoke tester output, where
you won't get access to the original machine (without asking). Having
a few intra-test checkpoints helps you to see where things might have
gone wrong.