From: Derrick Stolee <stolee@gmail.com>
Date: Mon, 04 Jun 2018 11:30:07 GMT
Subject: Re: [PATCH v3 06/20] commit-graph: add 'verify' subcommand
Message-ID: <8d0165f3-6e0c-4799-6042-b0570b529b6e@gmail.com>
In-Reply-To: <86d0x8rimu.fsf@gmail.com>

On 6/2/2018 5:19 PM, Jakub Narebski wrote:
> Derrick Stolee <stolee@gmail.com> writes:
>> Do we have a way to run individual steps of the test suite? I am
>> unfamiliar with that process.
> The t/README describes three such ways in "Skipping Tests" section:
>
> - GIT_SKIP_TESTS environment variable, which can either can match the
>    "t[0-9]{4}" part to skip the whole test, or t[0-9]{4} followed by
>    ".$number" to say which particular test to skip
>
> - For an individual test suite --run could be used to specify that
>    only some tests should be run or that some tests should be
>    excluded from a run (the latter with '!' prefix).
>
> - 'prove' harness can also run individual tests; one of more useful
>    options is --state, which for example would allow to run only failed
>    tests with --state=failed,save ... if the tests were independent.
>
>> Adding the complexity of storing a copy of the commit-graph file for
>> re-use in a later test is wasted energy right now, because we need to
>> run the steps of the test that create the repo shape with the commits
>> laid out as set earlier in the test. This shape changes as we test
>> different states of the commit-graph (exists and contains all commits,
>> exists and doesn't contain all commits, etc.)
> I think we can solve most of the problem by separating validation tests
> (which all or almost all use the same commit-graph file) and other test;
> putting them in different test scripts.  This means that the more
> complicated case would be limited to the subset of tests.
>
> Anyway, if the setup stages are clearly separated and clearly marked as
> such, we would be able to at least manually skip tests, or manually run
> only a subset of tests.
>
> Test independence is certainly something nice to have, but as the git
> testsuite is not in best shape wrt this, it is not a requirement.

I'm all for making the test suite better. In this case, I will hold that 
for a later series (that is entirely focused on that feature) as I 
expect we will want to discuss the correct pattern in detail.

Thanks,
-Stolee