Re: ab/cmake-nix-and-ci, was Re: [PATCH] test-lib.sh: discover "git" in subdirs of "contrib/buildsystems/out"

From: Junio C Hamano <gitster@pobox.com>
Date: Dec 9, 2022, 03:48 UTC
Message-ID: <xmqqk0311blt.fsf@gitster.g>
In-Reply-To: <oq7p2776-po8p-r9s0-82o2-o77so874n419@tzk.qr>

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Junio, maybe you could clarify your take on this? As project lead, it is
> your decision to define how Git uses Continuous Builds, and how the
> project handles failed CI runs.

I have pretty much been with what Peff and Taylor said in the thread already ever since we added CMake support to help Windows/VS folks. I agree with you that we do not need to run it for Linux or macOS, and if the promised/hoped-for benefit, i.e. that running them on non-Windows build would uncover issues that are common across the platform and help Windows, is not something that is likely to materialize, I'd prefer to see our resources (CI time and developer attention) not spent on that.

I do not think "how the project handles filed CI runs" is a very big issue. I often ignore partial failures (e.g. "winVS(n) test job triggered rate limit") and the only annoyance I feel is that such a temporary failure contribute one more message to my trash mailbox, and I can learn to do the same for a test that marked as failed due to linux-cmake-ctest job. I expect that regular contributors are doing the same pretty much.

How blocking is a CI failure for drive-by contributors who use GGG? While I do not necessarily value drive-by contributions as much as you do, if such "an unimportant failure we can ignore" discourages those coming from GGG route, that would be unfortunate, exactly because they may not have contributed anything to the failures. This is not just cmake-ctest, but the leak checking job where a new use of a tool that is known to be leaky in a test can turn a test that has been passing to fail. If we can mark failures in selected jobs as non-blocking, we definitely should do so.

Between keeping and marking linux-cmake-ctest as non-blocking, and removing it altogether, I am inclined to say that I'd favor the latter for the reasons I explained earlier in this message. But to help casual contributors coming via GGG, we would anyway need to (1) allow submitting even with failing tests, and (2) tell them that it is OK to do so. Which means it is not the end of the world, from the point of view of helping casual developers, if we had kept these brittle CI jobs like linux-cmake-ctest and linux-leaks.