Re: What's cooking in git.git (Jan 2020, #04; Wed, 22)

From: SZEDER Gábor <szeder.dev@gmail.com>
Date: Jan 23, 2020, 14:16 UTC
Message-ID: <20200123141626.GB6837@szeder.dev>
In-Reply-To: <xmqqftg6671g.fsf@gitster-ct.c.googlers.com>

On Wed, Jan 22, 2020 at 10:06:19PM -0800, Junio C Hamano wrote:

Show 26 quoted lines

> SZEDER Gábor <szeder.dev@gmail.com> writes:
> 
> > On Wed, Jan 22, 2020 at 02:18:05PM -0800, Junio C Hamano wrote:
> >> * yz/p4-py3 (2020-01-15) 14 commits
> >>  - ci: also run linux-gcc pipeline with python3.5 environment
> >
> > I still think that this last patch needs to be reworked before this
> > series is merged any further.
> >
> > The only Python script we have is 'git p4', so the Python version is
> > only relevant for 'git p4' tests ('t98*'), while the rest of Git and
> > the test suite couldn't care less [1].  This patch, however, not only
> > builds Git and runs the full test suite for each of the two Python
> > versions, but, worse, runs the full test suite _twice_ for each, first
> > as a "regular" test run and then again with all the GIT_TEST_* knobs
> > enabled.  Consequently, it adds ~50mins to every build's runtime.
> >
> > That's just too wasteful.
> 
> Thanks for a reminder.  Yes, I do recall you raised the above point
> and I agree with the assessment.
> 
> What's the ideal endgame wrt the tests?  Build with Py$N and run
> full test suite once, and run full test suite again with the unusual
> knobs enabled, which is what is done without this series, plus build
> with Py(5-$N) and run and run only t98?? tests?

Running the 'linux-clang' job with Python 2 and the 'linux-gcc' job with Python 3 would be the simplest and cheapest, I'd think. We'd only need to add the appropriate 'PYTHON_PATH=...' to out MAKEFLAGS. As far as Travis CI is concerned, their Xenial image (i.e. the Linux image we're using) comes with both 'python2' and 'python3' in PATH, at versions v2.7 and v3.5, with the former being the default.

Perhaps we could do the same with the OSX Clang and GCC jobs as well, dunno. Travis CI's OSX image, too, comes with both 'python2' and 'python3' in PATH, though Python 3 is already at v3.7, but still v2.7 is the default.

(Note that 'contrib/svn-fe/svnrdump_sim.py' is not added to SCRIPT_PYTHON in our Makefile, so it is not affected by the PYTHON_PATH we'd set in MAKEFLAGS, and it's shebang line remains '#!/usr/bin/env python'.)

I think the choice of compiler to build Git and the choice of Python version to run 'git p4' are completely independent, and it's not worth to check all their possible combinations.

Furthermore, to re-run only a subset of the test suite with 'prove' we'd need to tweak our $GIT_PROVE_OPTS, because the '--state=' option that we have in there, will cause us troubles:

  $ cd t/
  # I have the prove state file from the last full test run:
  $ ls -l .prove
  -rw-r--r-- 1 szeder szeder 188758 Jan 23 14:02 .prove
  # Let's try to run only a 'git p4' test script with the default
  # '--state=' option from our $GIT_PROVE_OPTS:
  $ make DEFAULT_TEST_TARGET=prove \
    GIT_PROVE_OPTS=--state=failed,slow,save T=t9800-git-p4-basic.sh 
  rm -f -r 'test-results'
  *** prove ***
  t9001-send-email.sh ................................ 23/?
  <... snip ...>
  # Uh-oh, it proceeds to run all the test scripts that are recorded
  # in the prove state file, i.e. the whole test suite.
  
  # Now let's try that without the '--state=' option:
  $ make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS= T=t9800-git-p4-basic.sh 
  rm -f -r 'test-results'
  *** prove ***
  t9800-git-p4-basic.sh .. ok    
  All tests successful.
  Files=1, Tests=21, 12 wallclock secs ( 0.02 usr  0.01 sys +  3.46 cusr
  2.04 csys =  5.53 CPU)
  Result: PASS

I couldn't find a set of 'prove' options that allow us to run slowest tests first, but run only the tests that are explicitly specified as cmdline arguments.

So we'd need to tweak how $GIT_PROVE_OPTS is used in our CI builds to drop the '--state=' option but still keep all other options ('--jobs'!) when re-running only the 'git p4' tests.