Re: Stochastic bisection support

From: Chris Torek <chris.torek@gmail.com>
Date: Nov 20, 2021, 07:54 UTC
Message-ID: <CAPx1GvfT36SvJ6Lwf1-2KUebVXkCkNvYTUw=FU+dHBy76VN5RQ@mail.gmail.com>
In-Reply-To: <nycvar.QRO.7.76.6.2111191731400.63@tvgsbejvaqbjf.bet>

On Fri, Nov 19, 2021 at 8:51 AM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 11 quoted lines

> An interesting problem, for sure!
>
> It is slightly related to a scenario that has been described to me
> recently: in a gigantic project whose full test suite is too large to run
> with every Pull Request, where tests are more likely to become flaky
> rather than simply break, a stochastic CI regime was introduced where a
> semi-random subset of the test suite is run with every CI build. That team
> also came up with the concept of attaching confidences as you describe.
>
> I only had time to look at the first patch closely so far. I hope to find
> more time next week to review further.

I only scanned for obvious items myself, but the idea of a probabilistic test is indeed interesting.

I do wonder why you (Jan Kara, not Dscho :-) ) used fixed-point arithmetic here though.

Chris