From: Jeff King <peff@peff.net>
Date: Thu, 13 Jul 2017 16:58:41 GMT
Subject: Re: [PATCH v1 0/4] Teach 'run' perf script to read config files
Message-ID: <20170713165840.e5cdw7pa2m6haaen@sigill.intra.peff.net>
In-Reply-To: <20170713065050.19215-1-chriscool@tuxfamily.org>

On Thu, Jul 13, 2017 at 08:50:46AM +0200, Christian Couder wrote:

> Goal
> ~~~~
> 
> Using many long environment variables to give parameters to the 'run'
> script is error prone and tiring.
> 
> We want to make it possible to store the parameters to the 'run'
> script in a config file. This will make it easier to store, reuse,
> share and compare parameters.

Because perf-lib is built on test-lib, it already reads
GIT-BUILD-OPTIONS. And the Makefile copies several perf-related values
into it, including GIT_PERF_MAKE_OPTS and GIT_PERF_REPEAT_COUNT. So you
can already do:

  echo 'GIT_PERF_REPEAT_COUNT = 10' >>config.mak
  echo 'GIT_PERF_MAKE_OPTS = CFLAGS="-O2" DEVELOPER=1' >>config.mak
  make
  cd t/perf
  ./run <versions-and-scripts>

I suspect there are still a lot of things that could be made easier with
a config file, so I'm not against the concept. Your example here:

> [perf "with libpcre"]
>         makeOpts = DEVELOPER=1 CFLAGS='-g -O0' USE_LIBPCRE=YesPlease
> [perf "without libpcre"]
>         makeOpts = DEVELOPER=1 CFLAGS='-g -O0'

is a lot more compelling. But right now the perf suite is not useful at
all for comparing two builds of the same tree. For that, I think it
would be more useful if we could define a tuple of parameters for a run.
One of which could be the tree we're testing. Build opts are another.
Tested repository is another. And then we'd fill in a table of results
and let you slice up the table by any column (e.g., compare times for
runs against a single tree but with differing build options).

So then I think your config file primarily becomes about defining the
properties of each run. I'm not sure if it would look like what you're
starting on here or not.

-Peff