Re: Git Test Coverage Report (Tuesday, Sept 25)

From: Derrick Stolee <stolee@gmail.com>
Date: Sep 26, 2018, 10:43 UTC
Message-ID: <b46d6363-1709-968e-105a-3f4e8a77155e@gmail.com>
In-Reply-To: <d92f18e2-0bbc-e36b-123a-3ed3f44cf418@gmail.com>

On 9/25/2018 5:12 PM, Ben Peart wrote:

Show 34 quoted lines

>
>
> On 9/25/2018 2:42 PM, Derrick Stolee wrote:
>> In an effort to ensure new code is reasonably covered by the test 
>> suite, we now have contrib/coverage-diff.sh to combine the gcov 
>> output from 'make coverage-test ; make coverage-report' with the 
>> output from 'git diff A B' to discover _new_ lines of code that are 
>> not covered.
>>
>> This report takes the output of these results after running on four 
>> branches:
>>
>>          pu: 80e728fa913dc3a1165b6dec9a7afa6052a86325
>>
>>         jch: 0c10634844314ab89666ed0a1c7d36dde7ac9689
>>
>>        next: 76f2f5c1e34c4dbef1029e2984c2892894c444ce
>>
>>      master: fe8321ec057f9231c26c29b364721568e58040f7
>>
>> master@{1}: 2d3b1c576c85b7f5db1f418907af00ab88e0c303
>>
>> I ran the test suite on each of these branches on an Ubuntu Linux VM, 
>> and I'm missing some dependencies (like apache, svn, and perforce) so 
>> not all tests are run.
>>
>> I submit this output without comment. I'm taking a look especially at 
>> my own lines to see where coverage can be improved.
>>
>
> Thanks for driving this.  I think it provides an interesting view into 
> new code and how well it is being tested.  In an effort to make this 
> as useful as possible, we should be looking to eliminate as much noise 
> as possible otherwise people will stop looking at it.

Thanks for helping identifying the noise.

Show 11 quoted lines

>
> I looked at the lines that came from my patches and most if not all of 
> them are only going to be executed by the test suite if the correct 
> "special setup" option is enabled.  In my particular case, that is the 
> option "GIT_TEST_INDEX_THREADS=<n>" as documented in t/README.
>
> I suspect this will be the case for other code as well so I wonder if 
> the tests should be run with each the GIT_TEST_* options that exist to 
> exercise uncommon code paths with the test suite.  This should prevent 
> false positives on code paths that are actually covered by the test 
> suite as long as it is run with the appropriate option set.

This is a bit tricky to do, but I will investigate. For some things, the values can conflict with each other (GIT_TEST_SPLIT_INDEX doesn't play nicely with other index options, I think). For others, we don't have the environment variables in all versions yet, as they are still merging down.

Show 7 quoted lines

>
> I realize it would take a long time to run the entire test suite with 
> all GIT_TEST_* variables so perhaps they can only be tested "as 
> needed" (ie when patches add new variables in the "special setups" 
> section of t/README).  This should reduce the number of combinations 
> that need to be run while still eliminating many of the false positive 
> hits.

This is something to think about. For my own thoughts, I was thinking of trying to run it when we see large blocks of code that are uncovered and obviously because of environment variables. This is what I was thinking when I saw your and Duy's commits in the output. I'll see if I can re-run the suite using GIT_TEST_INDEX_THREADS=2 and GIT_TEST_INDEX_VERSION=4.

Thanks, -Stolee