Re: [PATCH v1 0/4] Teach 'run' perf script to read config files

From: Jeff King <peff@peff.net>
Date: Jul 14, 2017, 08:05 UTC
Message-ID: <20170714080544.3nzndfx3p2yaxmac@sigill.intra.peff.net>
In-Reply-To: <CAP8UFD1FXPtzHF1J5ZfcsmMiHrfXQOd=qEghJQp3eFYsVd-nxQ@mail.gmail.com>

On Fri, Jul 14, 2017 at 08:27:53AM +0200, Christian Couder wrote:

Show 6 quoted lines

> The whole thing seems really complex to me though. And this makes me
> think that people might want to specify different GIT-BUILD-OPTIONS
> and config.mak files to be used when running perf tests, so that the
> results from perf tests can easily be reproduced later even if they
> have changed their build options in their development Git repo in the
> meantime.

I agree with the complexity. The general idea is that your currently built HEAD is a snapshot in time of options. But that doesn't have to be so, and laying out the options in a config file does seem like an improvement.

There is another implicit dependency, though: the set of (and exact content of) the tests depends on your HEAD, too. So if I do:

  git checkout v2.5.0
  cd t/perf
  ./run v2.0.0 v2.1.0

I might get different results if I replace "v2.5.0" in the first command with some other version, because the content of the tests will be different. I'm not sure how to account for that in storing results. Most of the time the version of the tests you ran is not going to be interesting. But it can be a source of confusing discrepancies if a test subtly changed between two runs. It probably happens infrequently enough that it's not worth worrying about.

Show 6 quoted lines

> So perhaps the config file should make it possible to specify a
> directory where all the build files (GIT-BUILD-OPTIONS, config.mak,
> config.mak.autogen and config.status) that should be used should be
> taken. And then it could also let people change some variables to
> override what is in those files which is needed to run perf tests with
> different parameters.

That sounds reasonable. I think you could ditch GIT-BUILD-OPTIONS entirely. It's only needed to pull in GIT_PERF variables that would be better served by being in the config in the first place.

-Peff