Re: [PATCH v2 0/9] ci: make Git's GitHub workflow output much more helpful

From: Junio C Hamano <gitster@pobox.com>
Date: May 21, 2022, 23:05 UTC
Message-ID: <xmqq35h2cwrm.fsf@gitster.g>
In-Reply-To: <nycvar.QRO.7.76.6.2205212307090.352@tvgsbejvaqbjf.bet>

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

Show 19 quoted lines

>> * print the verbose logs only for the failed test cases (to massively cut
>>   down on the size of the log, particularly when there's only a couple
>>   failures in a test file with a lot of passing tests).
>
> That's an amazingly simple trick to improve the speed by a ton, indeed.
> Thank you for this splendid idea!
>
>> * skip printing the full text of the test in 'finalize_test_case_output'
>>   when creating the group, i.e., use '$1' instead of '$*' (in both passing
>>   and failing tests, this information is already printed via some other
>>   means).
>>
>> If you wanted to make sure a user could still access the full failure logs
>> (i.e., including the "ok" test results), you could print a link to the
>> artifacts page as well - that way, all of the information we currently
>> provide to users can still be found somewhere.
>
> That's a good point, I added that hint to the output (the link is
> unfortunately not available at the time we print that advice).

https://github.com/git/git/runs/6539786128 shows that all in-flight topics merged to 'seen', except for the ds/bundle-uri-more, passes the linux-leaks job. The ds/bundle-uri-more topic introduces some leaks to commands that happen to be used in tests that are marked as leak-checker clean, making the job fail.

Which makes a great guinea pig for the CI output improvement topic.

So, I created two variants of 'seen' with this linux-leaks breakage. One is with the js/ci-github-workflow-markup topic on this thread. The other one is with the ab/ci-github-workflow-markup topic (which uses a preliminary clean-up ab/ci-setup-simplify topic as its base). They should show the identical test results and failures.

And here are their output:

 - https://github.com/git/git/runs/6539835065
 - https://github.com/git/git/runs/6539900608

If I recall correctly, the selling point of the ab/* variant over js/* variant was that it would give quicker UI response compared to the former, but other than that, both variants' UI are supposed to be as newbie friendly as the other.

When I tried the former, it reacted too poorly to my attempt to scroll (with mouse scroll wheel, if it makes a difference) that sometimes I was staring a blank dark-gray space for a few seconds waiting for it to be filled by something, which was a bit jarring experience. When I tried the latter, it didn't show anything to help diagnosing the details of the breakage in "run make test" step and the user needed to know "print test failures" needs to be looked at, which I am not sure is an inherent limitation of the approach. After the single extra click, navigating the test output to find the failed steps among many others that succeeded was not a very pleasant experience.

Those who are interested in UX experiment may want to visit these two output to see how usable each of these is for themselves.

Thanks.