Re: [PATCH 0/4] meson: parse TAP output generated by our tests

> 
> 
> On 23/05/2025 16:58, Junio C Hamano wrote:
>> Patrick Steinhardt <ps@pks.im> writes:
>>
>>> ... The problem is that we have a test that unexpectedly
>>> passes on macOS:
>>>
>>>     ▶  868/1023 - git grep .fi a                             UNEXPECTEDPASS
>>>
>>> The test in question is this one:
>>>
>>>     test_expect_failure !CYGWIN 'git grep .fi a' '
>>>         git grep .fi a
>>>     '
>>>
>>> The test passes if '.' matches a NUL byte, which we expect to only
>>> happen on Cygwin. 064eed36c7f (config.mak.uname: only set NO_REGEX on
>>> cygwin for v1.7, 2025-04-17) mentions that this behaviour was probably
>>> imported from FreeBSD, which makes me wonder whether macOS eventually
>>> also inherited the same code given its BSD lineage.
>>
>> Yup, I was wondering about the same thing.  Thanks for confirming.
>> It is unfortunate that we have blanket USES_BSD_REGEXP prerequisite
>> ;-)
>>
>>> I think we probably want something like the below patch to fix this. We
>>> could also have a prereq, but that prereq would look almost the exact
>>> same as the test. It does make me question the value of the test itself
>>> as the behaviour is completely platform specific.
> 
> Yep, I very nearly sent that patch with a hunk which deleted that
> test. In the end, I decided to play it more conservatively. :)
> 
>> Curious.
>>
>> Don't we run the same set of tests on macOS without Meson?  the
>> exact same test must be passing unexpectedly.  Why do we see the
>> complaint on only osx-meson job without osx-{clang,reftable,gcc}
>> jobs?
> 
> Yes, for about a decade, git built on cygwin with the system regex library
> would have shown an 'unexpected pass' on this test. However, unless you
> look for it, you wouldn't know; make and prove are quite happy with the
> situation, since it is an unexpected _pass_:
> 
>   $ git diff
>   diff --git a/t/t7815-grep-binary.sh b/t/t7815-grep-binary.sh
>   index b7d83f9a5d..3bd91da970 100755
>   --- a/t/t7815-grep-binary.sh
>   +++ b/t/t7815-grep-binary.sh
>   @@ -63,7 +63,7 @@ test_expect_success 'git grep ile a' '
>           git grep ile a
>    '
>    
>   -test_expect_failure !CYGWIN 'git grep .fi a' '
>   +test_expect_failure 'git grep .fi a' '
>           git grep .fi a
>    '
>    
>   $ cd t
>   $ ./t7815-grep-binary.sh
>   ok 1 - setup
>   ok 2 - git grep ina a
>   ok 3 - git grep -ah ina a
>   ok 4 - git grep -I ina a
>   ok 5 - git grep -c ina a
>   ok 6 - git grep -l ina a
>   ok 7 - git grep -L bar a
>   ok 8 - git grep -q ina a
>   ok 9 - git grep -F ile a
>   ok 10 - git grep -Fi iLE a
>   ok 11 - git grep ile a
>   not ok 12 - git grep .fi a # TODO known breakage
>   ok 13 - grep respects binary diff attribute
>   ok 14 - grep --cached respects binary diff attribute
>   ok 15 - grep --cached respects binary diff attribute (2)
>   ok 16 - grep revision respects binary diff attribute
>   ok 17 - grep respects not-binary diff attribute
>   ok 18 - setup textconv filters
>   ok 19 - grep does not honor textconv
>   ok 20 - grep --textconv honors textconv
>   ok 21 - grep --no-textconv does not honor textconv
>   ok 22 - grep --textconv blob honors textconv
>   # still have 1 known breakage(s)
>   # passed all remaining 21 test(s)
>   1..22
>   $ echo $?
>   0
>   $ 
> 
> Again, prove doesn't care:
> 
>   $ prove t7815-grep-binary.sh
>   t7815-grep-binary.sh .. ok    
>   All tests successful.
>   Files=1, Tests=22,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.09 cusr  0.13 csys =  0.25 CPU)
>   Result: PASS
>   $ echo $?
>   0
>   $ 

> When I tested the patch (without !CYGWIN) with 'meson test' it didn't care
> either - that was *before* Patrick's recent patch to make meson parse TAP
> output. ;)
> 
> Question: should meson (or indeed prove) fail the test because of an
> unexpected _pass_?