Re: [PATCH v2 0/3] Make the Git tests emit TAP format

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Jun 14, 2010, 22:10 UTC
Message-ID: <AANLkTinHrf_ZuuoMXlL8BFqm3UhRYxrV4t2Nmp5QNjrE@mail.gmail.com>
In-Reply-To: <m3631lp9oe.fsf@localhost.localdomain>

On Mon, Jun 14, 2010 at 21:49, Jakub Narebski <jnareb@gmail.com> wrote:

Show 20 quoted lines

> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>
>> All this series does is slightly adjust the raw text output of our
>> tests so that it conforms. to the TAP standard, i.e. instead of this:
>>
>>    $ ./t0005-signals.sh
>>    *   ok 1: sigchain works
>>    * passed all 1 test(s)
>>
>> We get this:
>>
>>    $ ./t0005-signals.sh
>>    ok 1 - sigchain works
>>    # passed all 1 test(s)
>>    1..1
>>
>> Changing the output format like this gives us the ability to run the
>> Git tests with any TAP tool (like prove(1)) at no extra cost. Every
>> other existing way of running the tests continues to work, it's just
>> easier for machines to read the output now.

I'll re-submit a version of the patch with a better commit message which addresses all of the below.

> This doesn't tell us if the result of running test suite with
> '--verbose' and/or with '--debug' changes, and if changes how?  Is it
> compatibile with TAP format so that TAP parsers understand it?

It just changes in that the lines that previously said e.g. "* ok 1: sigchain works" now say "ok 1 - sigchain works".

TAP parsers still understand it, since ignoring unknown garbage is part of the TAP standard.

> It doesn't tell us if (as I assume) after change git test suite still
> generates summary of tests in t/test-results/.  Do those results
> change, and if change then how?

The semantics of t/test-results don't change at all since that's aggregated internally by test-lib.sh guts that aren't changed in this series.

> It doesn't tell us if (as I assume) we still have the same color
> output as we had before, and what we should do to have the same color
> output e.g. from TAP tests in Perl.

The color output is still there if you run the tests under `make test' or manually.

Under TAP consumers like prove(1) the tests won't output color since the TAP Harness will set a dumb terminal.

That's a good thing, since we shouldn't emit terminal escape codes with TAP. The output could be used for e.g. a HTML page instead of a terminal.

Show 11 quoted lines

> +1
>
> In my opinion it is a good change, because there are many tools that
> understand TAP output format (and being able to output tests in Perl
> via standard Test::Simple / Test::More is also nice).  Those include
> e.g. Smolder and prove(1).
>
> It would be nice to have something like CPAN Testers
> (http://cpantesters.org) but for git test suite.  Junio would be able
> to submit new version of git and check if it breaks on some uncommon
> architecture...

There's been a lot of good changes to the CPAN Testers infrastructure lately. We could probably just set up a copy of their software.

Then anyone could run a smoker on some odd OS/architechture and send in test results.

> P.S. I wonder who was original designer of git tests output format,
> ans why it was chosen this way (and e.g. why not TAP)...

It was originally discussed here: http://thread.gmane.org/gmane.comp.version-control.git/2901/focus=3076

It seems that the people involved weren't aware of TAP at the time, at least it wasn't brought up.