Re: [PATCH] test-lib.sh: discover "git" in subdirs of "contrib/buildsystems/out"

From: Jeff King <peff@peff.net>
Date: Dec 2, 2022, 23:10 UTC
Message-ID: <Y4qF3iHW2s+I0yNe@coredump.intra.peff.net>
In-Reply-To: <221202.86sfhxg2ng.gmgdl@evledraar.gmail.com>

On Fri, Dec 02, 2022 at 05:40:34PM +0100, Ævar Arnfjörð Bjarmason wrote:

Show 22 quoted lines

> > [1] While our CI helps the MSVC job runs CMake manually, performs an
> > in-tree build and does not use ctest. In contrast a user running the 
> > MSVC GUI does not run CMake themselves, ends up with an out-of-tree
> > build and runs the tests with ctest.
> 
> I don't run Windows by choice, and I'm not interested in running a
> propriterary IDE (VS) either.
> 
> The main reason I'm working on this series is that while we as a project
> are happy to support proprietary OS's, it hasn't been a requirement for
> participation that you need to buy a copy of Windows, OSX, AIX, HP/UX or
> whatever to submit patches.
> 
> Of course we have platform-specific code. but this CMake component is
> unique in how invasive it is.
> 
> It's easy to e.g. stay away from the OSX-specific code in
> compat/fsmonitor/*darwin*.[ch], or generally speaking the
> Windows-specific C code.
> 
> But for CMake it's become a hard requirenment for many changes, even
> though it's a contrib/ component.

I have similar feelings to you here. Back when cmake support was introduced, I explicitly wanted it to be something for people who cared about it, but that wouldn't bother people who didn't use it:

  https://lore.kernel.org/git/20200427200852.GC1728884@coredump.intra.peff.net/

I stand by that sentiment, but it seems to have crept up as a required thing to deal with, and that is mostly because of CI. Using cmake in CI is good for telling developers when a change they make has broken cmake. But it also makes cmake their problem, and not the folks interested in cmake.

Now maybe attitudes have changed, and I am out of date, and cmake support is considered mature and really important (or maybe nobody even agreed with me back then ;) ). But if not, should we consider softening the CI output so that cmake failures aren't "real" failures? That seems drastic and mean, and I don't like it. But it's the root of the issue, IMHO.

As a side note, this isn't the only such instance of this problem. Two other things to think about:

  - You mentioned darwin fsmonitor code. And it's true that you can
    largely ignore it if you don't touch it. But every once in a while
    you get bit by it (e.g., enabling a new compiler warning which
    triggers in code you don't compile on your platform, and now you
    have to guess-and-check the fix with CI). This sucks, but is kind of
    inevitable on a cross-platform system. I think the issue with cmake
    is that because it's basically duplicating/wrapping the Makefile, it
    _feels_ unnecessary to people on platforms with working make, and
    triggers more frequently (because changes to the rest of the build
    system may break cmake in subtle ways).

  - I'd actually put the leak-checking CI in the same boat. It's a good
    goal, and one I hope we work towards. But it feels like the current
    state is not very mature, and people often end up wrestling with CI
    to deal with failures that they didn't even introduce (e.g., adding
    a new test that happens to run a Git program that has an existing
    leak, and now you are on the hook for figuring out why the existing
    "passes leaks" annotation is wrong).

    My original hope is that we would introduce leak-checking tooling
    that people interested in leaks could use, and other people could
    ignore until we reached a leak-free state. Because it's in CI it
    means that people get notified of new leaks in code they write
    (which is good, and helps people interested in leaks), but it also
    means they have to deal with the immature state.

I'm not necessarily proposing to drop the leaks CI job here. I'm mostly philosophizing about the greater problem. In the early days of Git, the cross-platform testing philosophy was: somebody who cares will test on that platform and write a patch. If they don't, how important could it be? With CI that happens automatically and it becomes everybody's problem, which is a blessing and a curse.

-Peff