From: Ævar Arnfjörð Bjarmason Date: Mon, 14 Jun 2010 22:10:23 GMT Subject: Re: [PATCH v2 0/3] Make the Git tests emit TAP format Message-ID: In-Reply-To: On Mon, Jun 14, 2010 at 21:49, Jakub Narebski wrote: > Ævar Arnfjörð Bjarmason   writes: > >> All this series does is slightly adjust the raw text output of our >> tests so that it conforms. to the TAP standard, i.e. instead of this: >> >>    $ ./t0005-signals.sh >>    *   ok 1: sigchain works >>    * passed all 1 test(s) >> >> We get this: >> >>    $ ./t0005-signals.sh >>    ok 1 - sigchain works >>    # passed all 1 test(s) >>    1..1 >> >> Changing the output format like this gives us the ability to run the >> Git tests with any TAP tool (like prove(1)) at no extra cost. Every >> other existing way of running the tests continues to work, it's just >> easier for machines to read the output now. I'll re-submit a version of the patch with a better commit message which addresses all of the below. > This doesn't tell us if the result of running test suite with > '--verbose' and/or with '--debug' changes, and if changes how?  Is it > compatibile with TAP format so that TAP parsers understand it? It just changes in that the lines that previously said e.g. "*   ok 1: sigchain works" now say "ok 1 - sigchain works". TAP parsers still understand it, since ignoring unknown garbage is part of the TAP standard. > It doesn't tell us if (as I assume) after change git test suite still > generates summary of tests in t/test-results/.  Do those results > change, and if change then how? The semantics of t/test-results don't change at all since that's aggregated internally by test-lib.sh guts that aren't changed in this series. > It doesn't tell us if (as I assume) we still have the same color > output as we had before, and what we should do to have the same color > output e.g. from TAP tests in Perl. The color output is still there if you run the tests under `make test' or manually. Under TAP consumers like prove(1) the tests won't output color since the TAP Harness will set a dumb terminal. That's a good thing, since we shouldn't emit terminal escape codes with TAP. The output could be used for e.g. a HTML page instead of a terminal. > +1 > > In my opinion it is a good change, because there are many tools that > understand TAP output format (and being able to output tests in Perl > via standard Test::Simple / Test::More is also nice).  Those include > e.g. Smolder and prove(1). > > It would be nice to have something like CPAN Testers > (http://cpantesters.org) but for git test suite.  Junio would be able > to submit new version of git and check if it breaks on some uncommon > architecture... There's been a lot of good changes to the CPAN Testers infrastructure lately. We could probably just set up a copy of their software. Then anyone could run a smoker on some odd OS/architechture and send in test results. > P.S. I wonder who was original designer of git tests output format, > ans why it was chosen this way (and e.g. why not TAP)... It was originally discussed here: http://thread.gmane.org/gmane.comp.version-control.git/2901/focus=3076 It seems that the people involved weren't aware of TAP at the time, at least it wasn't brought up.