Re: [PATCH v5 2/2] tests(mingw): avoid very slow `mingw_test_cmp`

> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> When Git's test suite uses `test_cmp`, it is not actually trying to
> compare binary files as the name `cmp` would suggest to users familiar
> with Unix' tools, but the tests instead verify that actual output
> matches the expected text.
>
> On Unix, `cmp` works well enough for Git's purposes because only Line
> Feed characters are used as line endings. However, on Windows, while
> most tools accept Line Feeds as line endings, many tools produce
> Carriage Return + Line Feed line endings, including some of the tools
> used by the test suite (which are therefore provided via Git for Windows
> SDK). Therefore, `cmp` would frequently fail merely due to different
> line endings.
>
> To accommodate for that, the `mingw_test_cmp` function was introduced
> into Git's test suite to perform a line-by-line comparison that ignores
> line endings. This function is a Bash function that is only used on
> Windows, everywhere else `cmp` is used.
>
> This is a double whammy because `cmp` is fast, and `mingw_test_cmp` is
> slow, even more so on Windows because it is a Bash script function, and
> Bash scripts are known to run particularly slowly on Windows due to
> Bash's need for the POSIX emulation layer provided by the MSYS2 runtime.
>
> The commit message of 32ed3314c104 (t5351: avoid using `test_cmp` for
> binary data, 2022-07-29) provides an illuminating account of the
> consequences: On Windows, the platform on which Git could really use all
> the help it can get to improve its performance, the time spent on one
> entire test script was reduced from half an hour to less than half a
> minute merely by avoiding a single call to `mingw_test_cmp` in but a
> single test case.
>
> Learning the lesson to avoid shell scripting wherever possible, the Git
> for Windows project implemented a minimal replacement for
> `mingw_test_cmp` in the form of a `test-tool` subcommand that parses the
> input files line by line, ignoring line endings, and compares them.
> Essentially the same thing as `mingw_test_cmp`, but implemented in
> C instead of Bash. This solution served the Git for Windows project
> well, over years.
>
> However, when this solution was finally upstreamed, the conclusion was
> reached that a change to use `git diff --no-index` instead of
> `mingw_test_cmp` was more easily reviewed and hence should be used
> instead.
>
> The reason why this approach was not even considered in Git for Windows
> is that in 2007, there was already a motion on the table to use Git's
> own diff machinery to perform comparisons in Git's test suite, but it
> was dismissed in https://lore.kernel.org/git/xmqqbkrpo9or.fsf@gitster.g/
> as undesirable because tests might potentially succeed due to bugs in
> the diff machinery when they should not succeed, and those bugs could
> therefore hide regressions that the tests try to prevent.
>
> By the time Git for Windows' `mingw-test-cmp` in C was finally
> contributed to the Git mailing list, reviewers agreed that the diff
> machinery had matured enough and should be used instead.
>
> When the concern was raised that the diff machinery, due to its
> complexity, would perform substantially worse than the test helper
> originally implemented in the Git for Windows project, a test
> demonstrated that these performance differences are well lost within the
> 100+ minutes it takes to run Git's test suite on Windows.