Re: [PATCH] ci: avoid running the test suite _twice_

From: Jeff King <peff@peff.net>
Date: Nov 13, 2023, 18:49 UTC
Message-ID: <20231113184909.GB3838361@coredump.intra.peff.net>
In-Reply-To: <pull.1613.git.1699894837844.gitgitgadget@gmail.com>

On Mon, Nov 13, 2023 at 05:00:37PM +0000, Johannes Schindelin via GitGitGadget wrote:

> This is a late amendment of 19ec39aab54 (ci: stop linking the `prove`
> cache, 2022-07-10), fixing a bug that had been hidden so far.

We don't seem to have that commit in Junio's tree; it is only in git-for-windows.

Not that we should not fix things if they are broken, but I am trying to understand if git/git is experiencing the same bug. It sounds like not yet, though from looking at 19ec39aab54, I would expect to get these doubled runs any time we store the prove state. But maybe without that commit our state-file symlink is going somewhere invalid, and prove fails to actually store anything?

Show 12 quoted lines

> But starting with that commit, we run `prove` _twice_ in CI, and with
> completely different sets of tests to run. Due to the bug, the second
> invocation re-runs all of the tests that were already run as part of the
> first invocation. This not only wastes build minutes, it also frequently
> causes the `osx-*` jobs to fail because they already take a long time
> and now are likely to run into a timeout.
> 
> The worst part about it is that there is actually no benefit to keep
> running with `--state=slow,save`, ever since we decided no longer to
> try to reuse the Prove cache between CI runs.
> 
> So let's just drop that Prove option and live happily ever after.

Yes, I think this is the right thing to do regardless. If we are not saving the state to use between two related runs, there is no point storing it in the first place.

I do have to wonder, though, as somebody who did not follow the unit-test topic closely: why are the unit tests totally separate from the rest of the suite? I would think we'd want them run from one or more t/t*.sh scripts. That would make bugs like this impossible, but also:

  1. They'd be run via "make test", so developers don't have to remember
     to run them separately.

  2. They can be run in parallel with all of the other tests when using
     "prove -j", etc.

-Peff