Re: [PATCH 6/7] test-lib: make --verbose output valid TAP

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Mar 16, 2021, 09:10 UTC
Message-ID: <8735wvfoq8.fsf@evledraar.gmail.com>
In-Reply-To: <875z1zivlv.fsf@evledraar.gmail.com>

On Wed, Mar 10 2021, Ævar Arnfjörð Bjarmason wrote:

> On Tue, Mar 09 2021, SZEDER Gábor wrote:
>
>> On Tue, Mar 09, 2021 at 09:57:03PM +0100, Ævar Arnfjörð Bjarmason wrote:

[...]

Show 17 quoted lines

>>> Becuse the advantage of this series is that that sort of thing becomes
>>> really trivial without everything needing to be hardcoded in
>>> test-lib.sh, observe (this is with my series):
>>> 
>>>     
>>>     0 $ PERL5LIB=. prove -v --formatter=SZEDERVerboseLog ./t0201-gettext-fallbacks.sh :: --verbose-log
>>
>> Well, this doesn't look trivial at all, does it?  In fact, I consider
>> this unusably convoluted.
>
> ... I meant to say something closer to "does that output look
> ok?". Obviously we'd then make the --verbose-log run something like that
> under the hood.
>
> But in any case, I found it easier to just add this feature to my "tee"
> program than doing it with Perl's TAP libraries, i.e. something like
> this on top (will integrate it in an eventual re-roll):

FWIW I implemented this. I've got it at https://github.com/avar.git's avar/support-test-verbose-under-prove-2 if you want to take an early peek at it. It turned into a 50-some patch slog through refactoring various test-lib.sh code, so I'm seeing about how I submit it.

The end result is that -V now powered by a proper TAP-parsing tap-tee
program. So you'll have e.g. this working as before:
    
    $ ./t9004-example.sh -V
    ok 1 - decorate
    # passed all 1 test(s)
    1..1

But you can also tweak levels of verbosity to stdout (while retaining full logging) by suppling -V N to get N-level comments, as now indicated by N number of prefixed "#"'s:

   # -V0 is a special-case to get 0th level logging (couldn't think of a
   # better name), --verbose-log=0 also works
   $ ./t9004-example.sh -V0
   ok 1 - decorate
   1..1

At level 3 you start getting the source of passing tests (failing tests
are level 1, unexpected TODO tests level 2, trace level 4):
    
    $ ./t9004-example.sh -VVV 
    ok 1 - decorate
    ###
    ###     test-tool example-decorate
    ###
    # passed all 1 test(s)
    1..1

So obviously the devil's in the details, but it would be useful to know what you think about the approach before I try to roll this into submitted patches sooner than later.

I'm mainly interesetd in this for this providing building blocks for more advanced features down the road.

E.g. it would now be pretty easy to do something like run -x behind the scenes even if the user doesn't ask for it, and then when a test fail annotate the specific line that failed using fuzzy-matching on the -x output.

We could do that now, sort of, but once we've got unambiguous TAP start/end markers that can't be fooled it becomes trivial to write a function like get_trace_lines_for_nth_test(N-1) to get your own trace output for test N.