Re: Prove "Tests out of sequence" Error

From: Jeff King <peff@peff.net>
Date: Oct 21, 2016, 08:20 UTC
Message-ID: <20161021082035.xad6wfny5i6wtshh@sigill.intra.peff.net>
In-Reply-To: <CAGZ79kZo3LdcRmrjQTAvgx=H6U2tdjASK3qv5A2K5J2HQ2NvSw@mail.gmail.com>

On Thu, Oct 20, 2016 at 11:10:39PM -0700, Stefan Beller wrote:

Show 11 quoted lines

> > on TravisCI I see these weird "Tests out of sequence" errors with prove
> > and they seem to not go away. I assume the reason that they not go away
> > is that the ".prove" file is carried over from on build to another (but I can't
> > look into this file on TravisCI).
> >
> > Has anyone an idea where these errors might come from?
> >
> > ------------------------------------------------------------------------
> > t5547-push-quarantine.sh                         (Wstat: 0 Tests: 5 Failed: 0)
> 
> push quarantine is a new thing made by Jeff 2 weeks ago, IIRC.

Yes, but I do not see how it can trigger this:

> >   Parse errors: Tests out of sequence.  Found (2) but expected (3)
> >                 Tests out of sequence.  Found (3) but expected (4)
> >                 Tests out of sequence.  Found (4) but expected (5)
> >                 Bad plan.  You planned 4 tests but ran 5.

The TAP output from one of our tests should look something like:

  ok 1 - subject one
  ok 2 - subject two
  ok 3 - subject three
  ok 4 - subject four
  # passed all 4 test(s)
  1..4

the "plan" is the bit at the end. That looks like $test_count accidentally got incremented by one and we generated something like:

  ok 1 - subject one
  ok 3 - subject two
  ok 4 - subject three
  ok 5 - subject four
  1..4

which would explain the "out of sequence" errors as well as the "planned 4 but ran 5".

But I do not see how the test script could screw that up. The counting is handled entirely by the harness in test-lib.sh.

Nor do I see how a stale .prove file could matter. It does not store information about the test plan at all. E.g., here is the entry from mine for t5547:

  t5547-push-quarantine.sh:
    elapsed: 0.0762169361114502
    gen: 1
    last_pass_time: 1477037708.741
    last_result: 0
    last_run_time: 1477037708.741
    last_todo: 0
    seq: 437
    total_passes: 1

Puzzling.

-Peff