Re: [PATCH v2 1/4] tests: add a test mode for SANITIZE=leak, run it in CI

> On Fri, Jul 16, 2021 at 04:46:12PM +0200, Ævar Arnfjörð Bjarmason wrote:
>
>> > and so on. That requires two new features in test-lib.sh:
>> >
>> >   - making a GIT_TEST_RUN variable that is the opposite of GIT_TEST_SKIP
>> >     (instead of just the command-line --run).
>> >
>> >   - adding GIT_TEST_{RUN,SKIP}_FROM variables to read the values from a
>> >     file rather than the environment (I suppose the caller could just
>> >     stuff the contents into the variable, but I expect that test-lib.sh
>> >     may want to pare down the entries that do not even apply to the
>> >     current script for the sake of efficiency in checking each test).
>> >
>> > That infrastructure would then be applicable to other cases, too. Or
>> > even just useful for using another list (or no list at all) when you
>> > are looking at whether other tests are leak-free or not.
>> 
>> I've included a mechanism for whitelisting specific globs, the idea was
>> not to have that be too detailed, but we'd e.g. get to the point of t00*
>> or whatever passing.
>> 
>> Anything that's a lot more granular than that is doing to suck,
>> e.g. exposing teh GIT_TEST_SKIP and --run features. of specific test
>> numbers, now you need to count your tests if you add one in the middle
>> of one of those, and more likely you won't test under the mode and just
>> see it in CI.
>
> I think you can do the same level of skipping with GIT_TEST_SKIP,
> though. My argument was just that adding a new mechanism does not make
> sense when we already have one. I.e., running:
>
>   GIT_SKIP_TESTS='
>     t[123456789]*
>     t0[^0]*
>     t00[^016]*
>     t000[469]
>     t001[2459]
>     t006[0248]
>   ' make SANITIZE=leak test
>
> works already to do the same thing. The only thing we might want is a
> nicer syntax (e.g., to allow positive and negative patterns, or to read
> from a file). But that would benefit all users of GIT_SKIP_TESTS, not
> just people interested in leaks.

>> It also means everything works by default, you get an appropriate notice
>> from prove(1), and even if you run one test manually it'll skip, but
>> emit a message saying you can set the env var to force its run.
>
> With GIT_SKIP_TESTS you obviously don't get a message saying "try
> skipping this test" when it fails. :) But IMHO that is not that big a
> deal. You'll get a test failure with good LSan output. If you are
> working on expanding leak-checker coverage, you already know about your
> options for skipping. If you're adding a new test that leaks, you might
> consider fixing the leak (though not always, if it's far from code
> you're touching).