Re: [PATCH 00/15] tests: don't ignore "git" exit codes

From: Derrick Stolee <derrickstolee@github.com>
Date: Mar 3, 2022, 02:02 UTC
Message-ID: <76a1ff22-3eb0-08fb-5aa9-e612ee5b522f@github.com>
In-Reply-To: <cover-00.15-00000000000-20220302T171755Z-avarab@gmail.com>

On 3/2/2022 12:27 PM, Ævar Arnfjörð Bjarmason wrote:

Show 9 quoted lines

> This series fixes issues where we ignored the exit code of "git" due
> to it being on the LHS of a pipe, or because we interpolated its
> output with $() in a "test" construct, or had missing &&- chains in
> helper functions etc.
> 
> This series is not made by string-replacing things in our test suite,
> if it was it would be much larger. These are all tests I've seen real
> hide real failures under SANITIZE=leak, either on current "master", or
> in combination with various local leak fixes I've got unsubmitted.

My first reaction was to check that Subham was on the CC line (yes) because they have been working in this space, too. Your focus on examples that break SANITIZE=leak is appreciated so there is room for that project.

> In cases where I was starting to fix a pattern in a file I'd fix the
> rest of the file if it was easy, but otherwise these are all cases
> where I ran SANITIZE=leak, had a test pass, but having ASAN_OPTIONS
> log to a file revealed that we had memory leaks within that test.

Neat trick.

The patches in this series clearly do the right transformations to expose these errors at the appropriate time. The only time I got hung up was patch 11 where test_expect_failure was swapped for test_expect_success (while also dropping a test_must_fail inside the test). The double-negative confused me at first, but in the end the patch works as-is.

Show 11 quoted lines

> As an aside we still have various potential issues with hidden
> segfaults etc. in the test suite after this that are tricked to solve,
> because:
> 
>  * Our tests will (mostly) catch segfaults and abort(), but if we
>    invoke a command that invokes another command it needs to ferry the
>    exit code up to us.
> 
>  * run-command.c notably does not do that, so for e.g. "git push"
>    tests where we expect a failure and an underlying "git" command
>    fails we won't ferry up the segfault or abort exit code.

Perhaps run-command.c could auto-exit for certain well-known error codes that could only happen on certain kinds of failures (segfault, for example). A simple die() might be something that is expected to be handled by the top-level command in some cases.

Show 6 quoted lines

>  * We have gitweb.perl and some other perl code ignoring return values
>    from close(), i.e. ignoring exit codes from "git rev-parse" et al.
> 
>  * We have in-tree shellscripts like "git-merge-one-file.sh" invoking
>    git commands, and if they fail returning "1", not ferrying up the
>    segfault or abort() exit code.

These are more involved and harder to evaluate. Add them to the pile of projects for new contributors?

Thanks, -Stolee