Re: [PATCH v1 0/4] Teach 'run' perf script to read config files

From: Jeff King <peff@peff.net>
Date: Jul 13, 2017, 16:58 UTC
Message-ID: <20170713165840.e5cdw7pa2m6haaen@sigill.intra.peff.net>
In-Reply-To: <20170713065050.19215-1-chriscool@tuxfamily.org>

On Thu, Jul 13, 2017 at 08:50:46AM +0200, Christian Couder wrote:

Show 9 quoted lines

> Goal
> ~~~~
> 
> Using many long environment variables to give parameters to the 'run'
> script is error prone and tiring.
> 
> We want to make it possible to store the parameters to the 'run'
> script in a config file. This will make it easier to store, reuse,
> share and compare parameters.

Because perf-lib is built on test-lib, it already reads GIT-BUILD-OPTIONS. And the Makefile copies several perf-related values into it, including GIT_PERF_MAKE_OPTS and GIT_PERF_REPEAT_COUNT. So you can already do:

  echo 'GIT_PERF_REPEAT_COUNT = 10' >>config.mak
  echo 'GIT_PERF_MAKE_OPTS = CFLAGS="-O2" DEVELOPER=1' >>config.mak
  make
  cd t/perf
  ./run <versions-and-scripts>

I suspect there are still a lot of things that could be made easier with a config file, so I'm not against the concept. Your example here:

> [perf "with libpcre"]
>         makeOpts = DEVELOPER=1 CFLAGS='-g -O0' USE_LIBPCRE=YesPlease
> [perf "without libpcre"]
>         makeOpts = DEVELOPER=1 CFLAGS='-g -O0'

is a lot more compelling. But right now the perf suite is not useful at all for comparing two builds of the same tree. For that, I think it would be more useful if we could define a tuple of parameters for a run. One of which could be the tree we're testing. Build opts are another. Tested repository is another. And then we'd fill in a table of results and let you slice up the table by any column (e.g., compare times for runs against a single tree but with differing build options).

So then I think your config file primarily becomes about defining the properties of each run. I'm not sure if it would look like what you're starting on here or not.

-Peff