Re: [PATCH] p5310: stop timing non-bitmap pack-to-disk

From: Jeff King <peff@peff.net>
Date: Mar 26, 2020, 11:12 UTC
Message-ID: <20200326111201.GA4193632@coredump.intra.peff.net>
In-Reply-To: <cf64c47a-481e-3802-3ce6-a2a583cd7c75@gmail.com>

On Thu, Mar 26, 2020 at 06:48:37AM -0400, Derrick Stolee wrote:

Show 12 quoted lines

> >   - the point of the t/perf suite is to find performance regressions,
> >     and it won't help with that. We don't compare the numbers between
> >     two tests (which the perf suite has no idea are even related), and
> >     any change in its numbers would have nothing to do with bitmaps.
> 
> This does make me think if there is a way to adjust "./run" to test two
> different config settings or command-line options instead of just two
> different build versions. Perhaps something like
> 
>  ./run "HEAD -c core.commitGraph=true" "HEAD -c core.commitGraph=false" -- p4200-line-log.sh
> 
> But that's just musing on my part.

I think that's the right direction, but it would require tests knowing how to make use of those extra parameters. What I'd _really_ like is a way to parameterize a whole bunch of things:

 - test repo
 - git version
 - particular config settings

do runs with whatever combinations of parameters you choose, and then create tables showing differences across various settings of a parameter. Some parameters would be automatically set up, but test scripts would have to advertise other parameters they know about.

I don't think it's conceptually hard, but the existing perf code is pretty hacked-together and relies on the filesystem for storage. So the notion that we'd only ever compare different versions runs deep; that's how the on-disk storage is organized.

Show 5 quoted lines

> > So let's just drop it. It's not useful and is adding minutes to perf
> > runs.
> 
> I agree with your reasoning. This is not a critical path for clients,
> and all servers should be using bitmaps.

Yeah, I should have noted that we're not actually testing the performance of a stock repack anywhere else. It wouldn't belong in p5310, but in theory it might be of interest in another script. But I agree for the reasons you gave that it's not all that interesting a timing.

-Peff