Re: [PATCH 2/3] ci: avoid bare "gcc" for osx-gcc job

From: Patrick Steinhardt <ps@pks.im>
Date: May 16, 2024, 09:54 UTC
Message-ID: <ZkXX5MlN3EbaMhNG@tanuki>
In-Reply-To: <20240516071930.GB83658@coredump.intra.peff.net>

On Thu, May 16, 2024 at 03:19:30AM -0400, Jeff King wrote:

Show 25 quoted lines

> On Sat, May 11, 2024 at 07:21:28PM +0200, Patrick Steinhardt wrote:
> 
> > On Fri, May 10, 2024 at 03:47:39PM -0700, Junio C Hamano wrote:
> > > Jeff King <peff@peff.net> writes:
> > [snip]
> > > > [1] Another quirk is that we run the whole test suite for both
> > > >     compilers, which is probably overkill. The main value in comparing
> > > >     gcc vs clang is that we don't use any constructs that the compiler
> > > >     complains about. It's _possible_ for there to be a construct that
> > > >     the compiler does not notice but which causes a runtime difference
> > > >     (say, undefined behavior which happens to work out on one compiler),
> > > >     but I think we're again hitting diminishing returns.
> > > 
> > > Yeah, that is a very good point.
> > 
> > On Linux, we have the "pedantic" job that runs Fedora and only compiles
> > the sources with DEVOPTS=pedantic without running any of the tests. We
> > could do the same on macOS.
> 
> Yeah, I think the infrastructure is there (looks like just resetting
> $run_tests). We probably could stand to use it in more places. E.g., is
> there even value in running the tests for linux-gcc and linux-clang?
> It's _possible_ for there to be a run-time difference in the compiler
> outputs, but we may be hitting diminishing returns. The main value I
> think is just seeing what the compilers complain about.

That's certainly the biggest part, yeah. But I have been hitting lots of compiler-dependent behaviour. This is mostly in the area of bugs though, where for example toolchain A may initialize variables on the stack to all zeroes while toolchain B does not.

I guess this is mostly a question of defaults though, and I think it is partially influenced by the overall toolchain environment as configured by my distro. Especially hardening options are for example quite likely to lead to different behaviour.

I'm not sure whether this is sufficient reason on its own to warrant testing with several toolchains. But we can easily combine this with additional tuning knobs. Two separate test jobs with GCC and Clang are comparatively boring. But if we make it GCC+SHA1 and Clang+SHA256 then it becomes more interesting.

So I think dropping the compiler coverage completely is rather pointless because we already run multiple different jobs per platform anyway. But we should investigate whether we can cleverly combine those so that we do not need a separate jobs just to test a specific compiler.

Patrick

Show 9 quoted lines

> But I dunno. This thread argues there is value in running the tests with
> the separate compiler:
> 
>   https://lore.kernel.org/git/pull.266.git.gitgitgadget@gmail.com/
> 
> which I guess would argue for doing the same for osx-clang and osx-gcc
> (if the latter continues to exist).
> 
> -Peff