Re: [PATCH] test-lib: add ability to cap the runtime of tests

From: Stefan Beller <sbeller@google.com>
Date: Jun 5, 2017, 19:03 UTC
Message-ID: <CAGZ79kZn+bjtqBYTWuYqfsKdA=eWreNHQPCLVDGiS_8977VcTA@mail.gmail.com>
In-Reply-To: <CACBZZX5FP_jxXaT+NW8g2JqH89iYajHPjHhxCj=_vWnkxZ=rYQ@mail.gmail.com>

Show 12 quoted lines

> That's never going to be a problem on a less beefy machine with
> --state=slow,save, since the 30s test is going to be long over by the
> time the rest of the tests run.
>
> Cutting down on these long tail tests allows me to e.g. replace this:
>
>     git rebase -i --exec '(make -j56 all && cd t && prove -j56 <some
> limited glob>)'
>
> With a glob that runs the entire test suite, with the rebase only
> taking marginally longer in most cases while getting much better test
> coverage than I'd otherwise bother with.

I wonder if this functionality is rather best put into prove?

Also prove doesn't know which tests are "interesting", e.g. if you were working on interactive rebase, then you really want the longest test to be run in full?

And this "judge by time, not by interest" doesn't bode well with me.

I have a non-beefy machine such that this particular problem doesn't apply to me, but instead the whole test suite takes just long to run.

For that I reduce testing intelligently, i.e. I know where I am working on, so I run only some given tests (in case of submodules I'd go with "prove t74*") which would also fix your issue IIUC?