Re: [PATCH 0/8] t7900: untangle test dependencies

From: Kristoffer Haugsbakk <code@khaugsbakk.name>
Date: Oct 17, 2023, 20:14 UTC
Message-ID: <8cd788dc-7d16-4cfd-9f70-7889dcaa7199@app.fastmail.com>
In-Reply-To: <xmqqbkcxhvf9.fsf@gitster.g>

On Tue, Oct 17, 2023, at 21:59, Junio C Hamano wrote:

Show 10 quoted lines

> It is kind-of surprising that with only 8 patches you can reach such
> a state, but ...
>
>> # The tests that used to depend on each other should still pass
>> # when run together
>> ./t7900-maintenance.sh --quiet --run=setup,30,31 &&
>
> ... this puzzles me.  What does it mean for tests to "depend on each
> other"?  Does this mean running #31 with or without running #30 runs
> under different condition and potentially run different things?

What I mean is that some preceding test has a side-effect that a test depends on. Or that the test depends on some test *not* having done something; patch 9/8 changes `maintenance.auto config option` to delete and init the repository since it depends on the preceding tests *not* having run `git maintenance register`, since that turns off the default `true` value of `maintenance.auto`.

(Maybe those last meta-tests with combining tests like number 30 and 31 was a bit silly.)

Show 8 quoted lines

> One might argue that, in an ideal world, our tests should work when
> any non-setup tests are omitted (so, instead of $i above, you'll
> have an arbitrary subsequence of 1..42 and your tests still pass),
> and it may be a worthy goal, but at the same time, it may be a bit
> impractical, as setting things up is costly, but what you can do in
> the common "setup" will be very small.  Or you'll have so much
> "recovering from damage" in test_when_finished for each test that
> makes such untangling of dependencies too costly.

I don't know what the policy is. :) My motivation was that I was working on something else which seemed to break the suite, then I tried to reduce the tests that were run to get rid of the noise (`--verbose`), but then it got confusing because I didn't know if I had really broken some tests myself or if more tests would start failing by only running a subset of them.

That last patch 9/8 deals with what I discovered when I added two tests before `maintenance.auto config option`; the test started failing, which made me think that my changes might have some side-effect on what the test is testing. But it was just an invisible dependency on `git maintenance register` *not* having been run in the whole suite up until that point.

Cheers