Re: [PATCH 6/6] t/README: A new section about test coverage

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Jul 24, 2010, 23:32 UTC
Message-ID: <20100724233232.GB6374@burratino>
In-Reply-To: <AANLkTim7v199lMKNEhOALx_38ChIIgvD4oERQJU5SNdv@mail.gmail.com>

Ævar Arnfjörð Bjarmason wrote:

>> Ævar Arnfjörð Bjarmason wrote:

>>> + - Check the test coverage for your tests. See the "Test coverage"
>>> +   below.

[...]

Show 14 quoted lines

> What I was going for here is that you should try to make sure that the
> code you're adding is covered by tests by running the coverage tests.
> 
> I.e. if I add a new function "blah" to git-whatever which is
> implemented by the "do_blah" function checking if every line of
> "do_blah" is covered is an excellent indicator of whether that code is
> being exhaustively tested, as opposed to just superficially tested.
> 
> In most cases a low test coverage counts is telling about the overall
> quality of the tests.
> 
> But, the wording can probably be improved. Do you have a suggestion
> for the above intent compressed into a sentence or two? I can't come
> up with anything right now.

What I meant is that when developing a new feature, I think paying too much attention to coverage numbers is a very dangerous thing.

It produces two hazards: too many tests and too few tests.

 - too many tests because when I write my "do_blah" function
   that is about a case no one cares about in practice, to write
   artificial tests to exercise would actually be to do harm.

 - too few tests because if I focus on testing all the code I
   just wrote, then I am very unlikely to include tests for the
   cases I did /not/ write code for.  Some important cases are
   just easy; we should still test them because that will help
   if the code is ever refactored later.  Some important cases
   may be just not implemented yet; test cases for them are
   very helpful indeed to readers and future implementers.

In other words, I would rather that when writing tests, authors would forget about the implementation for a moment and just think about what a user wants to do.

That said:

The rest of the time, checking test coverage does provide a very good indication about what features might not be well tested yet. So it is still a good way to decide where to /start/ writing tests.

Plus it’s great fun to look at. :)

Thanks, Jonathan