From: Derrick Stolee Date: Wed, 26 Sep 2018 10:43:14 GMT Subject: Re: Git Test Coverage Report (Tuesday, Sept 25) Message-ID: In-Reply-To: On 9/25/2018 5:12 PM, Ben Peart wrote: > > > On 9/25/2018 2:42 PM, Derrick Stolee wrote: >> In an effort to ensure new code is reasonably covered by the test >> suite, we now have contrib/coverage-diff.sh to combine the gcov >> output from 'make coverage-test ; make coverage-report' with the >> output from 'git diff A B' to discover _new_ lines of code that are >> not covered. >> >> This report takes the output of these results after running on four >> branches: >> >>          pu: 80e728fa913dc3a1165b6dec9a7afa6052a86325 >> >>         jch: 0c10634844314ab89666ed0a1c7d36dde7ac9689 >> >>        next: 76f2f5c1e34c4dbef1029e2984c2892894c444ce >> >>      master: fe8321ec057f9231c26c29b364721568e58040f7 >> >> master@{1}: 2d3b1c576c85b7f5db1f418907af00ab88e0c303 >> >> I ran the test suite on each of these branches on an Ubuntu Linux VM, >> and I'm missing some dependencies (like apache, svn, and perforce) so >> not all tests are run. >> >> I submit this output without comment. I'm taking a look especially at >> my own lines to see where coverage can be improved. >> > > Thanks for driving this.  I think it provides an interesting view into > new code and how well it is being tested.  In an effort to make this > as useful as possible, we should be looking to eliminate as much noise > as possible otherwise people will stop looking at it. Thanks for helping identifying the noise. > > I looked at the lines that came from my patches and most if not all of > them are only going to be executed by the test suite if the correct > "special setup" option is enabled.  In my particular case, that is the > option "GIT_TEST_INDEX_THREADS=" as documented in t/README. > > I suspect this will be the case for other code as well so I wonder if > the tests should be run with each the GIT_TEST_* options that exist to > exercise uncommon code paths with the test suite.  This should prevent > false positives on code paths that are actually covered by the test > suite as long as it is run with the appropriate option set. This is a bit tricky to do, but I will investigate. For some things, the values can conflict with each other (GIT_TEST_SPLIT_INDEX doesn't play nicely with other index options, I think). For others, we don't have the environment variables in all versions yet, as they are still merging down. > > I realize it would take a long time to run the entire test suite with > all GIT_TEST_* variables so perhaps they can only be tested "as > needed" (ie when patches add new variables in the "special setups" > section of t/README).  This should reduce the number of combinations > that need to be run while still eliminating many of the false positive > hits. This is something to think about. For my own thoughts, I was thinking of trying to run it when we see large blocks of code that are uncovered and obviously because of environment variables. This is what I was thinking when I saw your and Duy's commits in the output. I'll see if I can re-run the suite using GIT_TEST_INDEX_THREADS=2 and GIT_TEST_INDEX_VERSION=4. Thanks, -Stolee