Re: [PATCH] tests: add a special setup where prerequisites fail

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: May 14, 2019, 12:37 UTC
Message-ID: <nycvar.QRO.7.76.6.1905141434040.44@tvgsbejvaqbjf.bet>
In-Reply-To: <87lfz9wexs.fsf@evledraar.gmail.com>

Hi Ævar,

On Tue, 14 May 2019, Ævar Arnfjörð Bjarmason wrote:

Show 15 quoted lines

> On Tue, May 14 2019, Johannes Schindelin wrote:
>
> > What would you think about a mode where random test cases are skipped?
> > It would have to make sure to provide a way to recreate the problem,
> > e.g. giving a string that defines exactly which test cases were
> > skipped.
> >
> > I am *sure* that tons of test scripts would fail with that, and we
> > would probably have to special-case the `setup` "test cases", and we
> > would have to clean up quite a few scripts to *not* execute random
> > stuff outside of `test_expect_*`...
>
> I think it would be neat, but unrelated to and overkill for spotting the
> practical problem we have now, which is that we *know* we skip some of
> this now on some platforms/setups due to prereqs.

I understand, but I am still worried that this is a lot of work for an incomplete fix.

For example, the t7600-merge.sh test script that set off this conversation has two prereqs that are unmet on Windows: GPG and EXECKEEPSPID. On Azure Pipelines' macOS agents, it is only GPG that is unmet. So switching off all prereqs would not help macOS with e.g. a bug where the GPG test cases are skipped but the EXECKEEPSPID test case is not.

Ciao, Dscho