Re: [PATCH] t1092: use GIT_PROGRESS_DELAY for consistent results

From: Derrick Stolee <stolee@gmail.com>
Date: May 24, 2021, 20:38 UTC
Message-ID: <e35b5217-362f-fc4f-7396-08218a9abf63@gmail.com>
In-Reply-To: <YKwMU13DOtTIgaeP@google.com>

On 5/24/21 4:28 PM, Jonathan Nieder wrote:

Show 17 quoted lines

> Hi,
> 
> Derrick Stolee wrote:
> 
>> The t1092-sparse-checkout-compatibility.sh tests compare the stdout and
>> stderr for several Git commands across both full checkouts, sparse
>> checkouts with a full index, and sparse checkouts with a sparse index.
>> Since these are direct comparisons, sometimes a progress indicator can
>> flush at unpredictable points, especially on slower machines. This
>> causes the tests to be flaky.
> 
> Hm, I think this test strategy is going to be fundamentally flaky
> regardless: Git doesn't intend to guarantee any kind of stability in
> the exact stderr output it writes.
> 
> Could the tests be made to check more semantically meaningful
> information such as "git ls-files -s" output instead?

The test is comparing the same exact Git command just with different configurations. Any change to what Git writes to stderr should be consistent across these, unless there is an explicit reason why it would behave differently across these options (for example, saying "You are in a sparse checkout" in 'git status').

There are no expectations that stderr is stable across versions of Git. These tests don't add friction to developers making new features or changing the error messages that appear over stderr. It's just that these tests should catch any unintended inconsistency across these modes.

Thanks, -Stolee