Re: [PATCH] test-lib: save test counts across invocations

From: Jeff King <peff@peff.net>
Date: Sep 1, 2011, 18:45 UTC
Message-ID: <20110901184554.GA18737@sigill.intra.peff.net>
In-Reply-To: <1314902334.5371.17.camel@umgah.localdomain>

On Thu, Sep 01, 2011 at 02:38:54PM -0400, Alex Vandiver wrote:

Show 11 quoted lines

> On Thu, 2011-09-01 at 12:38 -0400, Jeff King wrote:
> > Hmm. What happens when we're wrong? Does our eye-candy just print
> > something non-sensical like "13/12", or does prove actually care that we
> > run the right number of tests?
> 
> prove very much does care -- having a mismatch between the number of
> tests planned and the number of tests run is an error in the testfile,
> and is reported as such in big red text.  This is because stating how
> many tests you plan to run gives prove a way (in addition to the exit
> status) to know if the test stopped prematurely, so all mismatches
> between plan and actual test counts are reported as testfile failures.

Thanks. I suspected something like that, but was too lazy to look. :)

Given that our methods for automatically determining the number of tests are so flaky, and that prove will treat it so seriously, it doesn't seem worth pursuing to me.

We already handle the premature abort case by trapping exit from the shell before the script calls test_done. So I don't think that is a feature of prove that we particularly care about.

> As far as I know prove doesn't have a way to print the estimated time
> remaining, though using the contents of the .prove file (if you ran
> prove --state=save) to guess it wouldn't be all that hard of a change.

That would be a neat feature. In practice, I know about how many tests there are total (~7500), and how long it takes to run on my system (~60 seconds), so I can do the math myself. Still, a little more eye candy couldn't hurt. ;)

If I underestand the code correctly, we could even write our own custom "formatter" for git and use it via "prove --formatter".

-Peff