Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI

From: Jeff King <peff@peff.net>
Date: Jul 16, 2021, 18:09 UTC
Message-ID: <YPHLUsXsWO+JYS5X@coredump.intra.peff.net>
In-Reply-To: <877dhqxqbt.fsf@evledraar.gmail.com>

On Fri, Jul 16, 2021 at 04:46:12PM +0200, Ævar Arnfjörð Bjarmason wrote:

Show 24 quoted lines

> > and so on. That requires two new features in test-lib.sh:
> >
> >   - making a GIT_TEST_RUN variable that is the opposite of GIT_TEST_SKIP
> >     (instead of just the command-line --run).
> >
> >   - adding GIT_TEST_{RUN,SKIP}_FROM variables to read the values from a
> >     file rather than the environment (I suppose the caller could just
> >     stuff the contents into the variable, but I expect that test-lib.sh
> >     may want to pare down the entries that do not even apply to the
> >     current script for the sake of efficiency in checking each test).
> >
> > That infrastructure would then be applicable to other cases, too. Or
> > even just useful for using another list (or no list at all) when you
> > are looking at whether other tests are leak-free or not.
> 
> I've included a mechanism for whitelisting specific globs, the idea was
> not to have that be too detailed, but we'd e.g. get to the point of t00*
> or whatever passing.
> 
> Anything that's a lot more granular than that is doing to suck,
> e.g. exposing teh GIT_TEST_SKIP and --run features. of specific test
> numbers, now you need to count your tests if you add one in the middle
> of one of those, and more likely you won't test under the mode and just
> see it in CI.

I think you can do the same level of skipping with GIT_TEST_SKIP, though. My argument was just that adding a new mechanism does not make sense when we already have one. I.e., running:

  GIT_SKIP_TESTS='
    t[123456789]*
    t0[^0]*
    t00[^016]*
    t000[469]
    t001[2459]
    t006[0248]
  ' make SANITIZE=leak test

works already to do the same thing. The only thing we might want is a nicer syntax (e.g., to allow positive and negative patterns, or to read from a file). But that would benefit all users of GIT_SKIP_TESTS, not just people interested in leaks.

> It also means everything works by default, you get an appropriate notice
> from prove(1), and even if you run one test manually it'll skip, but
> emit a message saying you can set the env var to force its run.

With GIT_SKIP_TESTS you obviously don't get a message saying "try skipping this test" when it fails. :) But IMHO that is not that big a deal. You'll get a test failure with good LSan output. If you are working on expanding leak-checker coverage, you already know about your options for skipping. If you're adding a new test that leaks, you might consider fixing the leak (though not always, if it's far from code you're touching).

-Peff