Re: [RFC PATCH] Makefile: new prove target for running the tests with TAP

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Oct 14, 2010, 09:33 UTC
Message-ID: <AANLkTinDq4FOZd7v5C8HZ7S81xoO6nJZTEe4+-9AO8HQ@mail.gmail.com>
In-Reply-To: <4CB6CB97.7040009@drmicha.warpmail.net>

On Thu, Oct 14, 2010 at 09:21, Michael J Gruber <git@drmicha.warpmail.net> wrote:

Show 40 quoted lines

> Ævar Arnfjörð Bjarmason venit, vidit, dixit 14.10.2010 11:05:
>> On Thu, Oct 14, 2010 at 08:53, Michael J Gruber
>> <git@drmicha.warpmail.net> wrote:
>>> Introduce a new make target "prove" which runs the tests via "prove" (or
>>> $(PROVE) if set) so that one does not have to cd around any more. One
>>> can simply use "make prove" for "make test" or go wild as in:
>>>
>>> GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' GIT_PROVE_OPTS="-j9 -v" GIT_TEST_OPTS="--verbose --debug" make prove
>>>
>>> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
>>> ---
>>> RFC also because of lack of doc, and:
>>>
>>> Currently, one can do
>>>
>>> make -C t t6010-merge-base.sh
>>>
>>> or even
>>>
>>> make -C t t601*
>>>
>>> which is cool but undocumented. If we want this with prove it requires more
>>> effort, or shoving prove into the $(T) target rule which is doable, of course.
>>> I'm just wondering whether it's accepted to introduce a specific rule for prove
>>> at all.
>>
>> I like the rationale behind this, but fwiw. I already had a patch that
>> was ejected for this:
>>
>>     http://article.gmane.org/gmane.comp.version-control.git/146566
>>
>> Quoth Junio:
>>
>>     "Besides, "make -j15 test" from the toplevel already runs the
>>     tests in parallel.  I don't see much point in this change."
>>
>
> Sorry for the kind of dupe then. Though I can understand that the fixed
> options back then didn't find Junio's approval. Maybe it's different
> now, when it's completely interchangeable with make test?

Maybe. I try not to decipher him too much :)

But a patch for this that'd be acceptable would be great. Good that you're working on it.

Show 12 quoted lines

>> But I'd like to have it so that I could `make && make test_harness`
>> and get readable test output.
>>
>> But we should probably be using t/harness to run them, not
>> prove(1). Then we would make the test_harness target respect the
>> HARNESS_OPTIONS variable, which I e.g. have set to HARNESS_OPTIONS=j9
>> on my system.
>
> Didn't know about that one either. What does t/harness
> differently/better from/than prove? Does it take the same options? Is
> t/harness respecting the SHELL_PATH for the tests? Running a test under
> make test and make prove/harness needs to be completely equivalent.

t/harness is just using the library that prove(1) uses directly. We already use it for "make smoke". The advantage is that you can run custom setup code there, e.g. this is from perl.git's own t/harness:

    my $jobs = $ENV{TEST_JOBS};
    my ($rules, $state, $color);
    if ($ENV{HARNESS_OPTIONS}) {
        for my $opt ( split /:/, $ENV{HARNESS_OPTIONS} ) {
            if ( $opt =~ /^j(\d*)$/ ) {
                $jobs ||= $1 || 9;
            }
            elsif ( $opt eq 'c' ) {
                $color = 1;
            }
            else {
                die "Unknown HARNESS_OPTIONS item: $opt\n";
            }
        }
    }

So if we expanded t/harness to use that and added a "test_harness" target we could make it grok the TEST_JOBS and HARNESS_OPTIONS environment variables.

The output would be the same (see "make smoke"), and we can make it respect SHELL_PATH.