Re: [PATCH] CI: add SANITIZE=[address|undefined] jobs

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Jul 28, 2022, 23:10 UTC
Message-ID: <220729.8635ekkerp.gmgdl@evledraar.gmail.com>
In-Reply-To: <YuL7EotrIpnOn5BT@coredump.intra.peff.net>

On Thu, Jul 28 2022, Jeff King wrote:

Show 27 quoted lines

> On Thu, Jul 28, 2022 at 06:54:37PM +0200, Ævar Arnfjörð Bjarmason wrote:
> [...]
>> We also leave a lot of CI performance on the table by e.g. doing "chain
>> lint" in every single test run (except Windows), there *are* platform
>> edge-cases there like with SANITIZE=address, but I wonder if we should
>> just declare it good enough to do it in 1-2 jobs.
>
> I'd be fine with that, but I think chain lint isn't actually that
> expensive. The original in-shell bits are super cheap. The extra
> sed process is measurable, but I think I blunted the worst of it in the
> 2d86a96220 (t: avoid sed-based chain-linting in some expensive cases,
> 2021-05-13).
>
> Still, that patch should make it easy to time things just by setting
> GIT_TEST_CHAIN_LINT_HARDER=0 in various jobs. It does seem to buy ~100s
> of CPU time per test run on my Linux box. That's not a lot in the grand
> scheme, but perhaps adds up. And I could believe it's much worse on
> Windows. Maybe worth seeing how it performs in the actual CI
> environments.
>
>> Ditto TEST_NO_MALLOC_CHECK=1 & --no-bin-wrappers, but we can think about
>> all of those some other time....
>
> I'd be surprised if the malloc checking itself is all that expensive,
> though it does look like we call getconf and expr once per test there
> for setup. We could almost certainly hoist that out and call it once per
> script.

I posted some benchmarks for these a while ago: https://lore.kernel.org/git/220405.86k0c3lt2l.gmgdl@evledraar.gmail.com/

YMMV, but these make a big difference for some tests, although less for a whole run, as you point out disabling chain lint least impact.

For me a full test suite run is:

* TEST_NO_MALLOC_CHECK=1 --no-bin-wrappers --no-chain-lint:
	real    1m55.600s
	user    8m48.039s
	sys     3m17.776s
* [no opts, TEST_NO_MALLOC_CHECK=0 & --bin-wrappers && --chain-lint are on]
	real    2m18.126s
	user    10m40.230s
	sys     4m7.152s

So far from the >2x speedup you can get with some tests, but a big difference nonetheless.