Re: [PATCH] Makefile: add and use the ".DELETE_ON_ERROR" flag

From: Jeff King <peff@peff.net>
Date: Jun 29, 2021, 02:26 UTC
Message-ID: <YNqE8BIRF6NeYQcd@coredump.intra.peff.net>
In-Reply-To: <87fsx6xn0b.fsf@evledraar.gmail.com>

On Fri, Jun 25, 2021 at 11:49:14AM +0200, Ævar Arnfjörð Bjarmason wrote:

Show 24 quoted lines

> > Ah, right. Thanks for refreshing me.
> >
> > TBH, I don't find this that serious a problem. Your compile will fail.
> > But is rebuilding in the middle of a test run something it's important
> > to support seamlessly? It seems like a bad practice in the first place.
> 
> Yeah I think so, and I think it's good practice, it enabled development
> workflows that you and Junio clearly don't use (which is fine), but
> which are useful to others. For me it would result in more total
> testing, and earlier catching of bugs, not less.
> 
> Quoting an earlier mail of yours[1]:
> 
>     I think having the test suite loudly complain is a good way to
>     remind you that you have not in fact run the whole suite on a given
>     build.
> 
> It's useful as you're programming you save/compile, and have the tests
> run in a loop in the background, and are alerted if they start
> failing.
> 
> That's not really possible with git currently without having that loop
> continually push to another workdir, making it work in one checkout
> helps for some workflows.

I actually _do_ have a workflow like this, but as you might guess, it involves a separate workdir. I have a loop going in one terminal waiting for new commits, which it then checks out on a detached HEAD and tests.

So I'm definitely sympathetic to this kind of continuous testing. My questioning was really about doing it in a hacky way where you're not actually sure what's been tested and what hasn't. If you're not willing to test just committed states, then it gets a lot harder (OTOH, I'd expect a ton of spurious failures there as you have half-finished states). But if you are, then I think making sure you've tested each commit fully has huge value, because then you can cache the results.

I'm not sure if the "only test committed states" thing is a deal-breaker for you or not. If not, the script I use is at:

  https://github.com/peff/git/blob/meta/ci

which is really just a glorified infinite loop with some inotify hackery. It relies on this program to do the caching:

  https://github.com/mhagger/git-test

A nice extra thing you can do is use the same cache with "git rebase -x" as a final check of each patch in a series before you send it out. If the CI loop was running continuously in the background, then it's just a noop of "yes, we already tested this".

Show 5 quoted lines

> Yes it could allow you to run format-patch and send-email while you're
> 50% through a full loop, or not just run the full tests then, but at
> some point I think we've got to assume some basic competency from
> people. We also have CI running the full tests, and I could have just
> run tests 0000-5000, compiled, and then run 5001-9999.

Yeah, I can see the view that running the test suite as a basic sanity check may have value, if it's backed by more careful testing later (and certainly while I'm developing, I wouldn't hesitate to run a subset of the test suite to see how my work is progressing).

My main point was that I don't see much reason to do work to make that kind of continuous "make test" work with simultaneous recompiles and test-runs, if we can encourage people to do it more robustly with a single compile-and-test-run loop. Maybe adding in the extra workdir there makes it too heavy-weight? (Certainly my "ci" script is overkill, but it seems like a loop of "reset to the current branch tip, compile, run" in a worktree would be the minimal thing).

-Peff