Re: [PATCH v3 0/3] Add a JSON Schema for trace2 events

From: SZEDER Gábor <szeder.dev@gmail.com>
Date: Jul 25, 2019, 11:18 UTC
Message-ID: <20190725111803.GL20404@szeder.dev>
In-Reply-To: <cover.1564009259.git.steadmon@google.com>

On Wed, Jul 24, 2019 at 04:06:50PM -0700, Josh Steadmon wrote:

Show 11 quoted lines

> This is a proof of concept series that formalizes the structure of trace2 event
> output using JSON-Schema [1].
> 
> It provides a validator (written in Go) that verifies the events in a given
> trace2 event output file match the schema. I am happy to rewrite this validator
> in some other language, provided that the language has a JSON-Schema library
> supporting at least draft-04.
> 
> It runs the validator as part of the CI suite (it increase the runtime
> by about 15 minutes). It tests that the trace output of "make test"
> conforms to the schema.

I don't like this approach. 15 minutes is a lot, and spending that much time on JSON schema validation in every CI build is overkill and (IMO unacceptably) wasteful. I mean, the test suite involves e.g. thousands of 'git (add|commit|checkout|reset|...)' executions to set up the test cases, but surely it's not necessary to validate the trace output of every single one of them.

> I would appreciate any feedback on better ways to integrate the
> validator into the CI suite.

How about adding a test script dedicated to JSON schema validation, which runs only as many git commands as needed to cover all trace2 events.