Re: [PATCH] tests: add an optional test to test git-annex

From: Junio C Hamano <gitster@pobox.com>
Date: May 17, 2017, 10:19 UTC
Message-ID: <xmqqbmqrwzu1.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <CACBZZX4Jppr7ht7m444EjW4CDYX5CMvnxtStH4bF=A19TYKcZg@mail.gmail.com>

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

Show 11 quoted lines

>> Well, it is one thing to place git-annex under CI to make sure its
>> latest and greatest works together well with our latest and greatest
>> (and it may be something we want to see happen), but driving its
>> tests from our testsuite sounds like a tail wagging the dog, at
>> least to me.
>
> To me this is just a question of:
>
> * Is it the case that git-annex tests for a lot of edge cases we don't
> test for: Yes, probably. As evidenced by them spotting this
> regression, and not us.

And I'd encourage them to keep doing so.

Show 6 quoted lines

> * We can (and should) add a test for the specific breakage we caused
> in 2.13.0, but that's no replacement for other things annex may be
> covering & we may be missing which'll catch future breakages.
>
> * It's a pretty established practice to test a library (git) along
> with its consumers (e.g. annex) before a major release.

I am not so sure about the division of labor. What you are advocating would work _ONLY_ if we test with a perfect & bug-free version of the consumers. If they are also a moving target, then I do not think it is worth it. After all, we are *not* in the business of testing these consumers.

Unless I misunderstood you and you were saying that we freeze a version, or a set of versions, of customer that is/are known to pass their own tests, and test the combination of that frozen version of the customer with our daily development. If that is the case, then I would agree that we are using their test to test us, not them. But I somehow didn't get that impression, hence my reaction.