Re: Coverity, was Re: What's cooking in git.git (Oct 2021, #02; Wed, 6)

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Aug 19, 2022, 11:22 UTC
Message-ID: <3896n74p-0r16-866o-r668-70q6pos078n9@tzk.qr>
In-Reply-To: <Yvw87DJOOu+/jG6o@coredump.intra.peff.net>

Hi Peff,

On Tue, 16 Aug 2022, Jeff King wrote:

Show 19 quoted lines

> On Tue, Aug 16, 2022 at 11:05:48AM +0200, Johannes Schindelin wrote:
>
> > > It sounds like Taylor is volunteering to set up the Coverity side for
> > > git.git, and I can help him with getting those COVERITY_* variables into
> > > the GitHub environment.
> >
> > Given the challenges with Coverity (false positives, lack of support on
> > Synopsys' side, severely limited access to the reports), and given the
> > renewed efforts by OSTIF that focus not on Coverity but on CodeQL, I am
> > in favor of abandoning the idea to integrate Coverity in our GitHub
> > workflow.
> >
> > Regarding CodeQL, I am still uncertain what level of integration we will
> > end up with, and the contacts I am working with are currently all on
> > vacation, but I am confident that we will have an easier time going
> > forward with static analysis using CodeQL instead of Coverity.
>
> OK. I haven't been that impressed with CodeQL for C so far, but it may
> be getting better.

If your lack of being impressed stems from CodeQL's default suite catching very, very few issues, I agree with you.

The reason why I am more hopeful about CodeQL than about Coverity is that I now have a contact to work with (although they are currently on vacation, so I get to work on other things for now, including `merge-ort` and trying to integrate `mimalloc` in Git for Windows). And that is one more contact who is willing to work with me than I ever had on the Coverity side of things.

And apparently CodeQL's default settings optimize for reducing false positives in an attempt to avoid scaring potential users away.

But I have credible assurances that CodeQL has many more checks in store that simply need to be enabled in order to catch way more issues, at the expense of risking more false positives.

Show 11 quoted lines

> I certainly would be happier with a system that made it easier to
> display and share reports.
>
> Coverity does have a lot of false positives, but I've at least been able
> to pick useful fixes out of them (especially because it is good about
> saying "here are 5 new things to look at"). I've been continuing to
> build my private branch with it, so we'll see if it turns up anything
> useful. I do agree that inflicting it on ordinary users may be
> counter-productive (I often have to stare really hard to understand why
> its false positives are false, and that is not something I would wish
> on, say, a GGG user).

Don't get me wrong, I do not plan on dropping the Coverity builds of the Git for Windows project's `main` branch at https://dev.azure.com/git-for-windows/git/_build?definitionId=35.

As you probably recall, I specifically looked through the Coverity report in advance of v2.37.0 and tried to address some of the issues in https://lore.kernel.org/git/pull.1264.git.1655336146.gitgitgadget@gmail.com/, with mixed results (mainly because of some time constraints on my side, combined with your willingness to help fix some issues with my patches, as well as René's, Taylor's and Junio's excellent assistance).

However, the sheer amount of false positives, with intentional issues being a close second (in test helpers, we are not all that strict about releasing memory, for example), makes it a tough sell to ask more than just a few very dedicated contributors to have a look at the reports.

With CodeQL, I am optimistic that we can get it to a point where the burden can be carried by a larger group of people, with the help of enthusiastic CodeQL experts, which also means that the reports have a bigger chance at making Git safer.

Ciao, Dscho