Re: [PATCH 2/3] ci: avoid bare "gcc" for osx-gcc job

From: Jeff King <peff@peff.net>
Date: May 17, 2024, 08:19 UTC
Message-ID: <20240517081909.GB1517321@coredump.intra.peff.net>
In-Reply-To: <ZkXX5MlN3EbaMhNG@tanuki>

On Thu, May 16, 2024 at 11:54:44AM +0200, Patrick Steinhardt wrote:

> That's certainly the biggest part, yeah. But I have been hitting lots of
> compiler-dependent behaviour. This is mostly in the area of bugs though,
> where for example toolchain A may initialize variables on the stack to
> all zeroes while toolchain B does not.

I've definitely run into differing runtime outcomes for undefined behavior stuff like that. But in my experience most of that is consistently found by ASan/UBSan (which we do run in CI these days).

It's possible there are cases that those sanitizers don't catch but that cause differing behavior. But I can't think of one off the top of my head where that has happened.

Show 5 quoted lines

> I'm not sure whether this is sufficient reason on its own to warrant
> testing with several toolchains. But we can easily combine this with
> additional tuning knobs. Two separate test jobs with GCC and Clang are
> comparatively boring. But if we make it GCC+SHA1 and Clang+SHA256 then
> it becomes more interesting.

Yeah. Combining orthogonal properties into a single job lets us cover both (for the common case of success on both) with less CPU. But:

  - it can sometimes be hard to figure out which of the properties was
    responsible for a failure. That was the very subject of the thread I
    referenced earlier, where "linux-gcc" was "use gcc" and also "set
    lots of knobs".

  - they might not actually be orthogonal. If you care about checking
    runtime behavior in the output of two compilers, then that _could_
    manifest only in the sha256 code. Or as you get into more
    properties, they may overlap in other ways. I think reftable+sha256
    is an interesting (eventual) combo to test on top of reftable+sha1.

-Peff