git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: A "why TAP?" manifesto

From
Junio C Hamano <gitster@pobox.com>
Date
Mar 24, 2022, 17:26 UTC
Message-ID
<xmqqczibjm7v.fsf@gitster.g>
In-Reply-To
<220324.8635j7nyvw.gmgdl@evledraar.gmail.com>
Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
Show 6 quoted lines
> == It "can be parsed"
> == Human readable
> == It's trivial to produce
> == It's more flexible
> == It doesn't matter if nobody else uses it
> == Everyone uses it
I agree with all of what you said above.

But do non-human readers really want to read output from an "--immediate" session? Don't they rather see the whole thing?

The only "Huh?" I had with the original patch that started the thread was that TAP output currently does not work well with the "--verbose" option, and I've never run our tests with the "--immediate" option without the "--verbose" option to see where and how the first breakage happens and what is left behind in the trash directory, i.e. to bootstrap an interactive debugging session.

But not being useful for the use case of human reader running interactively to get the leftover state does not mean improvement for other use cases is not welcome.

Show 23 quoted lines
> I think if you did the legwork of trying to make those formats represent
> our "--verbose -x" output in a machine-readable way and tried to
> roundtrip that parsed output (which I have done for TAP) that you'd find
> it somewhere between "much, much harder" and "impossible" to do the same
> for those other formats in the context of our test suite.
>
> So yes, I agree that there's a lack of focus here, and we should put
> more wood behind fewer arrows in terms of making our test output
> machine-parsable.
>
> The [2] I mentioned above shows the sorts of wins we can get from that,
> i.e. emitting *only* the lines of test output relevant to the specific
> failures we had.
>
> That's really useful, and something you inherently can't do with the
> sort of approach you're going for in your [1] series.
>
> At least not without buffering the whole thing & parsing it, at which
> point you're back to square #1 in terms of fixing the sorts of issues
> noted in "Handling the long-tail of machine-readability" above.
>
> 1. https://lore.kernel.org/git/pull.1117.git.1643050574.gitgitgadget@gmail.com/
> 2. https://lore.kernel.org/git/220302.86mti87cj2.gmgdl@evledraar.gmail.com/
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 4 of 6 in “test-lib: have --immediate emit valid TAP on failure”
  1. test-lib: have --immediate emit valid TAP on failureÆvar Arnfjörð Bjarmason, Mar 23, 2022
  2. Johannes SchindelinMar 24, 2022
  3. A "why TAP?" manifesto (was: [PATCH] test-lib: have --immediate emit valid TAP on failure)Ævar Arnfjörð Bjarmason, Mar 24, 2022
  4. Junio C HamanoMar 24, 2022
  5. Ævar Arnfjörð BjarmasonMar 28, 2022
  6. Taylor BlauMar 24, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.