From: Junio C Hamano <gitster@pobox.com>
Date: Sun, 03 Oct 2021 17:33:10 GMT
Subject: Re: [PATCH 3/9] test-mergesort: add test subcommand
Message-ID: <xmqqee9210a1.fsf@gitster.g>
In-Reply-To: <850aa059-61d9-0eba-5809-e0c27a19dfb4@web.de>

René Scharfe <l.s.r@web.de> writes:

> Repeatable tests are not essential (the original paper didn't mention
> seeding), but shouldn't be much trouble to implement and would simplify
> comparisons across versions, systems and among testers.
>
> The only downside I can think of is that it may perhaps also simplify
> over-fitting, i.e. I might find micro-tweaks that only work for our
> specific rand() sequence and then misinterpret them as general
> improvements..

Yup, I think you summarized the pros-and-cons nicely.