Re: Funny test flake failure: t7518-ident-corner-cases.sh, test 1

From: Jeff King <peff@peff.net>
Date: Oct 16, 2020, 20:00 UTC
Message-ID: <20201016200031.GA3355643@coredump.intra.peff.net>
In-Reply-To: <CABPp-BHQK-AtAJa6V1mtu6_jq3oeUeL_0p00RCRNA6Jw3DpJSg@mail.gmail.com>

On Fri, Oct 16, 2020 at 12:29:26PM -0700, Elijah Newren wrote:

Show 19 quoted lines

> The issue comes from this test code:
> 
>        (
>                sane_unset GIT_AUTHOR_EMAIL &&
>                GIT_AUTHOR_NAME= &&
>                test_must_fail git commit --allow-empty -m foo 2>err &&
>                test_i18ngrep ! null err
>        )
> 
> The last line expects 'null' to NOT be found in err, unfortunately,
> the file err contained the line:
> 
> fatal: empty ident name (for
> <runner@fv-az128-670.gcliasfzo2nullsdbrimjtbyhg.cx.internal.cloudapp.net>)
> not allowed
> 218
> 
> Note that 'null' appears as a substring of the domain name, found
> within 'gcliasfzo2nullsdbrimjtbyhg'.

Heh. That's an amusing find. Good detective work, and sorry I created the problem.

Show 6 quoted lines

> +# confirm that we do not segfault _and_ that we do not say "(null)", as
> +# glibc systems will quietly handle our NULL pointer
> 
> Should we tighten the test to check for "(null)" instead of "null", or
> should we do something else?  Or just ignore it as it is somewhat
> unlikely that anyone ever hits this flake again?

I think it's worth tightening the test. The "(null)" phrasing is pretty common, and it's really the best we can do. The chance that somebody is on a platform that neither segfaults not prints "(null)", _and_ that they introduce a regression there seems pretty low. And in comparison, you already wasted time tracking down a false negative. Let's make sure that doesn't happen again.

I wish there was an environment variable or something we could set in the test suite to convince glibc to actually segfault on NULL (because it _will_ segfault on other platforms, and we're rather catch such things sooner rather than later).

Another option, I guess, is that we probably _do_ run the tests on such a platform (IIRC BSD will segfault, so macos probably does). So we could just remove the grep entirely from this test and let any regression get caught there.

Of the two, I think I have a slight preference for matching (null), but I don't feel strongly.

-Peff