Re: [PATCH/RFC/GSOC] make git-pull a builtin

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Mar 21, 2015, 17:35 UTC
Message-ID: <4388b6e4005c2872aa7f5f83024f021d@www.dscho.org>
In-Reply-To: <CACRoPnQp-3RGAPmoo1aM2KUA-phi138J2o82FwP9annM+xvOXA@mail.gmail.com>

Hi Paul,

On 2015-03-21 14:23, Paul Tan wrote:

> Thanks for the review, though I would like to work on the proposal now
> before the deadline passes :)

That makes sense.

Show 31 quoted lines

> On Thu, Mar 19, 2015 at 1:52 AM, Matthieu Moy
> <Matthieu.Moy@grenoble-inp.fr> wrote:
>> Paul Tan <pyokagan@gmail.com> writes:
>>
>>> Ideally, I think the solution is to
>>> improve the test suite and make it as comprehensive as possible, but
>>> writing a comprehensive test suite may be too time consuming.
>>
>> time-consuming, but also very beneficial since the code would end up
>> being better tested. For sure, a rewrite is a good way to break stuff,
>> but anything untested can also be broken by mistake rather easily at any
>> time.
>>
>> I'd suggest doing a bit of manual mutation testing: take your C code,
>> comment-out a few lines of code, see if the tests still pass, and if
>> they do, add a failing test that passes again once you uncomment the
>> code.
> 
> Maybe code coverage tools could help here so we only need to focus on
> the code paths that are untested by the test suite. At the minimum,
> all of the non-trivial code paths in both the shell script and the
> converted builtin must be covered by tests. This should help to
> eliminate most sources of breakages. Anything further than that would
> require an experienced understanding of all the possible important
> inputs to be tested, which I personally feel would make the project
> quite tedious.
> 
> I see git already has gcov support. For shell scripts, maybe kcov[1]
> could be used. With some slight code changes, I managed to generate a
> report for the git-pull tests[2] which should at least provide a good
> starting point for how the tests can be improved.

While it is often a tempting idea to make test suites as thorough as possible, there lies a true danger herein. True war story: in one of the projects I was involved in, the test suite grew to a size that one complete run lasted two weeks. Yes, that is fourteen days. Needless to say: this test suite was run rarely. How useful is a test suite that is run rarely? More useful than a non-existent one, to be sure, but it is still more of a burden than a boon.

Now, on Windows the test suite takes almost three hours to run. This really, really slows down development.

So while we are not yet at the "too large to be useful state", I would caution against trying to get there.

Instead, I would really like to focus on the *usage*. Calling `git grep "git pull" t/` should give you an idea what usage of `git pull` is already tested. It should be pretty easy to come up with a list of *common* use cases, and if any of them are not covered, adding tests for them is simple and straight-forward, too.

Ciao, Johannes