patchtest-lib: save test counts across invocations

Under 'prove' we can get progress status for tests running in parallel. However we never told it the total number of tests in a file in advance, and thus the progress only showed the tests already executed.

Save the number of tests run ($test_count) in a file under test-counts/. Then when sourcing test-lib.sh the next time, compare the timestamps. If the counts file is older than the test, discard. Otherwise use the count that we saved and give prove the test plan ("1..N") up front.

This results in 'make prove' giving progress output like

  ===(    2884;121   8/12   24/147  1/8  1/3  )============================

if you have already run the tests before.

Prerequisite changes can mean that a whole test group is skipped (e.g. NO_SVN_TESTS=1). We thus need to be somewhat careful to only emit the "full" plan once we know we're not going to $skip_all.

t9700 needs special treatment on top of $test_external_has_tap because the latter can only be set once we know that the external test will run. If a prerequisite fails, we still need to emit the plan.

The Makefile changes are required because we want to keep that test-count subdirectory unless the *user* invokes 'make clean', but we previously ran the latter ourselves after every successful test run.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---

Sparked by a discussion on G+. I think this is the "simple" approach. The "cute" approach would be to let test-lib.sh define test_* as test-counting dummies once, source the test script itself (avoiding the sourcing loop with test-lib) to count what it does, then do the real work.

 t/.gitignore        |    1 +
 t/Makefile          |    9 ++++++---
 t/t9700-perl-git.sh |    1 +
 t/test-lib.sh       |   27 +++++++++++++++++++++++++--
 4 files changed, 33 insertions(+), 5 deletions(-)

Thomas Rast <trast@student.ethz.ch> writes:

... which is this logic ...

I think the patch is cute, but I however do not think this is sufficient to catch prerequisite changes, unfortunately. I'd rather leave the total unknown than giving incorrect numbers.

On Thu, Sep 01, 2011 at 03:08:45PM +0200, Thomas Rast wrote:

Hmm. What happens when we're wrong? Does our eye-candy just print something non-sensical like "13/12", or does prove actually care that we run the right number of tests?

I don't think the "cute" approach will ever be accurate. Deciding whether to run later tests sometimes depends on the results of earlier tests, in at least two cases:

  1. Some tests find out which capabilities the system has, and set
     prerequisites. You need to actually run those tests, not make them
     counting dummies.

  2. Some tests create state that we then iterate on. For example, I
     think the mailinfo tests do something like:

        test_expect_success 'split' '
                git mailsplit -o patches mbox
        '
        for i in patches/*; do
          test_expect_success "check patch $i" '
                  git mailinfo $i >output
                  ...
          '
        done

      You'd get an inaccurate count if you didn't actually run the
      mailsplit command.

Anyway, this whole thing is a cute idea, and I do love eye candy, but I wonder if it's worth the complexity. All this is telling us is how far into each of the scripts it is. But we have literally hundreds of test scripts, all with varying numbers of tests of varying speeds, and you're probably running 16 or more at one time. So it doesn't tell you what you really want to know, which is: how soon will the test suite probably be done running.

-Peff

On Thu, 2011-09-01 at 12:38 -0400, Jeff King wrote:

prove very much does care -- having a mismatch between the number of tests planned and the number of tests run is an error in the testfile, and is reported as such in big red text. This is because stating how many tests you plan to run gives prove a way (in addition to the exit status) to know if the test stopped prematurely, so all mismatches between plan and actual test counts are reported as testfile failures.

As far as I know prove doesn't have a way to print the estimated time
remaining, though using the contents of the .prove file (if you ran
prove --state=save) to guess it wouldn't be all that hard of a change.
 - Alex

On Thu, Sep 01, 2011 at 02:38:54PM -0400, Alex Vandiver wrote:

Thanks. I suspected something like that, but was too lazy to look. :)

Given that our methods for automatically determining the number of tests are so flaky, and that prove will treat it so seriously, it doesn't seem worth pursuing to me.

We already handle the premature abort case by trapping exit from the shell before the script calls test_done. So I don't think that is a feature of prove that we particularly care about.

That would be a neat feature. In practice, I know about how many tests there are total (~7500), and how long it takes to run on my system (~60 seconds), so I can do the math myself. Still, a little more eye candy couldn't hurt. ;)

If I underestand the code correctly, we could even write our own custom "formatter" for git and use it via "prove --formatter".

-Peff

Jeff King wrote:

I guess you're right.  Let's drop this, then.