Re: [PATCH v1 0/4] Teach 'run' perf script to read config files

From: Junio C Hamano <gitster@pobox.com>
Date: Jul 13, 2017, 18:29 UTC
Message-ID: <xmqqeftkxkax.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <20170713165840.e5cdw7pa2m6haaen@sigill.intra.peff.net>

Jeff King <peff@peff.net> writes:

Show 13 quoted lines

> Because perf-lib is built on test-lib, it already reads
> GIT-BUILD-OPTIONS. And the Makefile copies several perf-related values
> into it, including GIT_PERF_MAKE_OPTS and GIT_PERF_REPEAT_COUNT. So you
> can already do:
> ...
> But right now the perf suite is not useful at
> all for comparing two builds of the same tree.
> For that, I think it
> would be more useful if we could define a tuple of parameters for a run.
> One of which could be the tree we're testing. Build opts are another.
> Tested repository is another. And then we'd fill in a table of results
> and let you slice up the table by any column (e.g., compare times for
> runs against a single tree but with differing build options).

Yeah, I think we saw this discussed in not-so-distant past, for which we want a good solution, and it might be the case that such a solution can be made easier to use with a separate configuration file (which this topic may or may not be used as a building block).

> So then I think your config file primarily becomes about defining the
> properties of each run. I'm not sure if it would look like what you're
> starting on here or not.

Yeah, I suspect that the final shape that defines the matrix might have to become quite a bit different.