Re: [RFC PATCH v3 1/1] unit tests: Add a project plan document

> In our current testing environment, we spend a significant amount of
> effort crafting end-to-end tests for error conditions that could easily
> be captured by unit tests (or we simply forgo some hard-to-setup and
> rare error conditions).Describe what we hope to accomplish by
> implementing unit tests, and explain some open questions and milestones.
> Discuss desired features for test frameworks/harnesses, and provide a
> preliminary comparison of several different frameworks.
>
> Signed-off-by: Josh Steadmon <steadmon@google.com>
> Coauthored-by: Calvin Wan <calvinwan@google.com>
> ---

> diff --git a/Documentation/technical/unit-tests.txt b/Documentation/technical/unit-tests.txt
> new file mode 100644
> index 0000000000..dac8062a43
> --- /dev/null
> +++ b/Documentation/technical/unit-tests.txt
> @@ -0,0 +1,141 @@
> += Unit Testing
> +
> +In our current testing environment, we spend a significant amount of effort
> +crafting end-to-end tests for error conditions that could easily be captured by
> +unit tests (or we simply forgo some hard-to-setup and rare error conditions).
> +Unit tests additionally provide stability to the codebase and can simplify
> +debugging through isolation. Writing unit tests in pure C, rather than with our
> +current shell/test-tool helper setup, simplifies test setup, simplifies passing
> +data around (no shell-isms required), and reduces testing runtime by not
> +spawning a separate process for every test invocation.
> +
> +We believe that a large body of unit tests, living alongside the existing test
> +suite, will improve code quality for the Git project.
> +
> +== Definitions
> +
> +For the purposes of this document, we'll use *test framework* to refer to
> +projects that support writing test cases and running tests within the context
> +of a single executable. *Test harness* will refer to projects that manage
> +running multiple executables (each of which may contain multiple test cases) and
> +aggregating their results.
> +
> +In reality, these terms are not strictly defined, and many of the projects
> +discussed below contain features from both categories.
> +

> +== Choosing a framework & harness
> +
> +=== Desired features
> +
> +==== TAP support
> +
> +The https://testanything.org/[Test Anything Protocol] is a text-based interface
> +that allows tests to communicate with a test harness. It is already used by
> +Git's integration test suite. Supporting TAP output is a mandatory feature for
> +any prospective test framework.
> +
> +==== Diagnostic output
> +
> +When a test case fails, the framework must generate enough diagnostic output to
> +help developers find the appropriate test case in source code in order to debug
> +the failure.
> +
> +==== Parallel execution
> +
> +Ideally, we will build up a significant collection of unit tests cases, most
> +likely split across multiple executables. It will be necessary to run these
> +tests in parallel to enable fast develop-test-debug cycles.
> +
> +==== Vendorable or ubiquitous
> +
> +If possible, we want to avoid forcing Git developers to install new tools just
> +to run unit tests. So any prospective frameworks and harnesses must either be
> +vendorable (meaning, we can copy their source directly into Git's repository),
> +or so ubiquitous that it is reasonable to expect that most developers will have
> +the tools installed already.
> +
> +==== Maintainable / extensible
> +
> +It is unlikely that any pre-existing project perfectly fits our needs, so any
> +project we select will need to be actively maintained and open to accepting
> +changes. Alternatively, assuming we are vendoring the source into our repo, it
> +must be simple enough that Git developers can feel comfortable making changes as
> +needed to our version.
> +
> +==== Major platform support
> +
> +At a bare minimum, unit-testing must work on Linux, MacOS, and Windows.
> +
> +==== Lazy test planning
> +
> +TAP supports the notion of _test plans_, which communicate which test cases are
> +expected to run, or which tests actually ran. This allows test harnesses to
> +detect if the TAP output has been truncated, or if some tests were skipped due
> +to errors or bugs.
> +
> +The test framework should handle creating plans at runtime, rather than
> +requiring test developers to manually create plans, which leads to both human-
> +and merge-errors.
> +
> +==== Skippable tests
> +
> +Test authors may wish to skip certain test cases based on runtime circumstances,
> +so the framework should support this.
> +
> +==== Test scheduling / re-running
> +
> +The test harness scheduling should be configurable so that e.g. developers can
> +choose to run slow tests first, or to run only tests that failed in a previous
> +run.
> +
> +==== Mock support
> +
> +Unit test authors may wish to test code that interacts with objects that may be
> +inconvenient to handle in a test (e.g. interacting with a network service).
> +Mocking allows test authors to provide a fake implementation of these objects
> +for more convenient tests.
> +
> +==== Signal & exception handling
> +
> +The test framework must fail gracefully when test cases are themselves buggy or
> +when they are interrupted by signals during runtime.
> +
> +==== Coverage reports
> +
> +It may be convenient to generate coverage reports when running unit tests
> +(although it may be possible to accomplish this regardless of test framework /
> +harness support).